AMERICAN STATISTICAL ASSOCIATION

                                     FALL MEETING OF

                             AMERICAN STATISTICAL ASSOCIATION

                                         WITH THE

                            ENERGY INFORMATION ADMINISTRATION

                                          DAY 2

                                     Washington, D.C.

                                 Friday, October 19, 2007

                                                                 2

              1    PARTICIPANTS:

              2       NAGARAJ K. NEERCHAL, Chair

                      Department of Mathematics and Statistics

              3       University of Maryland

              4    MEMBERS:

              5       EDWARD A. BLAIR

                      University of Houston

              6

                      CUTLER CLEVELAND

              7       Center for Energy and Environmental Studies

              8       JAE EDMONDS

                      Pacific Northwest National Laboratory

              9

                      MOSHE FEDER

             10       Research Triangle Institute International

             11       WALTER W. HILL

                      St. Mary's College of Maryland

             12

                      NEHA KHANNA

             13       Department of Economics, Binghamton University

             14       EDWARD KOKKELENBERG

             15    OTHERS:

             16       GEORGE LADY, Consultant to SMG

                      PAUL HOLTBERG, OIAF

             17       WILLIAM WATSON, CNEAF

                      MARGOT ANDERSON

             18       JANICE LENT

                      THOMAS RUTHERFORD

             19       FRED MAYES

                      EDWARD BLAIR

             20       KARA NORMAN

                      JOHN WEYANT

             21       RAMESH DANDEKAR

                      PRESTON McDOWNEY

             22       KOBI PLAT

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                 3

              1    PARTICIPANTS (Cont'd):

              2       JOEL DOUGLAS

                      SHAWNA WAUGH

              3       GUY CARUSO, Administrator, EIA

                      HOWARD GRUENSPECHT

              4       DR. NANCY KIRKENDALL, SMG

                      NICHOLAS JOHNSON

              5

              6

              7

              8

              9

             10

             11

             12

             13                       *  *  *  *  *

             14

             15

             16

             17

             18

             19

             20

             21

             22

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                 4

              1                   P R O C E E D I N G S

              2                                         (8:52 a.m.)

              3              MR. NEERCHAL:  I think a couple of

              4    things we need to do before we start.  Those

              5    who were not here yesterday, during

              6    introductions can you please introduce

              7    yourself.  State your name and your

              8    affiliation.  If you are in the audience

              9    please come to the mike.

             10              MR. LADY:  I am George Lady and I

             11    am a consultant to SMG.

             12              MR. WATSON:  Good morning.  I am

             13    Bill Watson with CNEAF, quality control.

             14              SPEAKER:  Good morning, I am Jay

             15    Kornasie (?) with Statistics Methods Group.

             16              MR. JOHNSON:  I am Nicholas Johnson

             17    with the web staff on the National Energy

             18    Information Center.

             19              MR. HOLTBERG:  I am Paul Holtberg.

             20    I am with OIAF.

             21              MR. NEERCHAL:  Thank you.  And I

             22    think I want to remind you that if you have

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                 5

              1    not signed your name and e-mail and details

              2    on the sign-up sheet outside on the desk,

              3    please do so, just to keep record of who has

              4    attended and sort of getting back to you if

              5    necessary.  And with that I think we'll start

              6    the first presentation today by Paul Holtberg

              7    and William Watson.

              8              MR. HOLTBERG:  All right, good

              9    morning.  The Annual Energy Outlook is one of

             10    only four hard copy publications which is

             11    still published by EIA.  And to understand

             12    the process that leads the release of the

             13    AEO, what I tend to do is review the quality

             14    control process that we go through for the

             15    AEO.

             16              Now since the AEO is a major EIA

             17    publication, it involves one of the most

             18    complete quality control processes of any of

             19    the EIA publications.  So some will be done

             20    to a lesser extent than what I am showing you

             21    here on a generic basis.

             22              Again the AEO is one of four annual

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                 6

              1    products produced by the EIA office of

              2    Integrated Analysis and Forecasting.  The

              3    others are the International Energy Outlook,

              4    the voluntary report on greenhouse gases, and

              5    the emissions of greenhouse gas, the

              6    inventory report.  This is just the cover of

              7    the AEO just to put it in reference if you've

              8    seen the publication itself.

              9              Now, the AEO is produced using the

             10    national energy modeling system which is

             11    summarized here.  I don't intend to discuss

             12    the model itself.  I am just putting this up

             13    here for a reference.  But if you have

             14    questions, I'd be happy to try to answer

             15    them.

             16              And before I get into the nuts and

             17    bolts, I wanted to pose some questions I'd

             18    like the committee to keep in mind as I go

             19    through my presentation.  First, is the

             20    external review process sufficient?  Do we

             21    need to expand the number, or type of

             22    reviewers included in the process?  Are there

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                 7

              1    ways in which the review process could be

              2    streamlined?  As I'll point our later, it

              3    takes a great deal of time the way it is done

              4    right now.

              5              Are there other aspects of the AEO

              6    that need to be reviewed, which we are not

              7    touching on?  And how do we balance that need

              8    to provide an independent projection and

              9    analysis with stakeholders' inputs, which we

             10    may or may not agree with?

             11              And we try to balance five major

             12    objectives in producing the AEO.  First, we

             13    want to ensure that the AEO addresses areas

             14    of interest to stakeholders.  They want it to

             15    have some relevancy.  The AEO is an EIA

             16    product not an OIAF product, so we want to

             17    make sure the projections include input from

             18    all of EIA.

             19              The projections include data that

             20    EIA reports as historical time series and

             21    other publications.  We want to make sure

             22    that the data reported in the AEO was

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                 8

              1    reasonably consistent with the reported EIA

              2    historical data.  The models need to be

              3    regularly reviewed to make sure they are

              4    consistent with current thought, policy,

              5    market trends, and technology, so that they

              6    provide an unbiased projection.  And we want

              7    to make sure that the models and results are

              8    completely documented for use as a reference.

              9              Now, the AEO production process

             10    itself is a year round process.  In fact it

             11    takes a little more than a year to actually

             12    complete.  The data and model development

             13    takes place roughly between January and

             14    September of a calendar year.  The projection

             15    development and analysis takes place between

             16    August and roughly October, sometimes a

             17    little later, with the projection development

             18    process focused during the early months of

             19    that period and the analysis during the later

             20    months of that period.

             21              The publication and documentation

             22    processes are completed between roughly

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                 9

              1    October and April.  Again, that slips a

              2    little bit depending on the year.  And that

              3    process includes multiple documents.  We have

              4    what's called the AEO Retrospective Review,

              5    which is completed between August and

              6    January, which looks at the AEO relative to

              7    another year of actual historical data and

              8    looks for the biases or errors within the AEO

              9    forecast itself, so it's really looking back

             10    one year.  It's done with a retrospective,

             11    the previous AEO.

             12              The AEO Summary Publication, which

             13    is the cover I showed you before, which is

             14    completed between October and roughly

             15    February.  And the methodology reports are

             16    which are completed between January and April

             17    and sometimes a little bit later, depending

             18    on where we are.  There's a series of a 12 or

             19    13 different methodology reports.

             20              Now, the AEO Quality Control

             21    Process is applied at all steps of the

             22    process.  We do a reassessment of newly

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                10

              1    available historical data, which is used as

              2    an input to update the model's relationships

              3    and to develop assumptions.  We regularly

              4    review and revise the models to add

              5    additional modeling capability.

              6              The preliminary projection results

              7    are regularly reviewed for many weeks before

              8    they are actually finalized and put out as an

              9    EIA product.  And all publications and

             10    documentations of both the projection results

             11    and the models are carefully reviewed before

             12    they are released.

             13              Now, the process involve employees,

             14    both external and internal stakeholders in

             15    the process.  The external reviews are both

             16    formal, for example, surveys, peer review

             17    process, working groups, independent expert

             18    reviews, as well as the conference

             19    presentations themselves, and informal,

             20    questions we receive from users after the

             21    publication is out, as well as independent

             22    reviews and articles which appear around the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                11

              1    course of the year.

              2              Now, the internal reviews are

              3    completed at multiple levels within EIA.  For

              4    example, at the OIAF office level by staff

              5    and management, by other EIA offices through

              6    the category-run review process to

              7    participation in the working groups I

              8    mentioned, participation in the development

              9    of assumptions and models, the direct review

             10    of draft of the projection results as well as

             11    a review of the documentation.

             12              We also have participation by EIA

             13    senior management, through multiple

             14    presentations of key assumptions as well as

             15    projection results.

             16              Now, the timing of the input to the

             17    quality control process varies.  For example,

             18    much of the external input, particularly the

             19    informal input, is after the fact.  Users are

             20    reacting to published results.  So that type

             21    of review tends to impact the following

             22    year's AEO.  It's really a following process.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                12

              1              However, that's not always the

              2    case.  Some of the external review includes

              3    as I said the independent expert reviews,

              4    working groups, which are completed during

              5    the development process or in some cases

              6    before we begin the development process for

              7    certain model pieces.  The internal input

              8    tends to coincide with the process, and is

              9    incorporated into the AEO prior to release.

             10              Now, the next few slides I'll step

             11    through the quality control process for each

             12    major step in the AEO, starting with the data

             13    update process.

             14              Now, quality control during the

             15    data update process is predominantly done

             16    internally.  As new historical data becomes

             17    available it reviewed internally by OIAF

             18    analyst and management for consistency with

             19    prior data series, so we can identify things

             20    that are changing for whatever reason.

             21              Some data for example, data

             22    characterizing technologies, is many times

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                13

              1    developed through third party studies.  And

              2    that data is typically reviewed by OIAF

              3    staff, again the working groups as well as

              4    external experts in some circumstances.  We

              5    circulate reports coming out of those

              6    contracts, and ask for a comment.

              7              Now, we try to make consistency

              8    with the EIA Short-Term Energy Outlook.  The

              9    STEO projection is reviewed by OIAF analysts

             10    and management for consistency, relative to

             11    what the forecast would do for the same

             12    years.  And that's not just done after the

             13    STEO is completed.  It involves discussion

             14    with the STEO analyst during the production

             15    process prior to STEO coming out.

             16              Major exogenous assumptions such as

             17    world oil prices or even macro economic

             18    growth are complimented by a Delphi approach

             19    involving external experts as well as the EIA

             20    offices, management, and staff.  What we are

             21    trying to do here is ensure that the major

             22    assumptions reflect a broad perspective, as

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                14

              1    opposed to just a parochial, a very narrow

              2    perspective internal to OIAF.

              3              And turning to the Model Update

              4    Process, the need for model updates is

              5    identified through independent expert

              6    reviews, customer comments received on the

              7    previous AEO, and of course internal analyst

              8    review of energy market trends.  A lot of the

              9    information comes right out of that.

             10              The proposed model updates are

             11    discussed with working groups, which are

             12    composed of a mix of internal and external

             13    stakeholders at the beginning of the AEO

             14    process and with OIAF management.  Once the

             15    updates are completed they are reviewed by

             16    the analysts, OIAF management, and the

             17    working groups again at a later time.

             18              The OIAF managers also hold weekly

             19    run reviews with the analyst to track model

             20    progress.  We do those in three sub groups

             21    stretched out over about three hours once a

             22    week.  So it's a pretty extensive process and

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                15

              1    we do that for about four months.

              2              Model updates completed by

              3    contractors are reviewed by the analyst, OIAF

              4    management, and again in those working group

              5    meetings.  Major updates are those that are

              6    particularly sensitive include a review by

              7    EIA's most senior management, for example,

              8    last year we did a lot of work in the area of

              9    ethanol and we talked directly with senior

             10    management before those that was finally

             11    incorporated.  The model updates are also

             12    reviewed by SMG, as part the documentation

             13    review.

             14              And finally we conduct an external

             15    peer review that includes questions on the

             16    modeling, which feed into the next year's

             17    AEO.  The external review are composed of

             18    typically 10 big users of the AEO who are

             19    familiar with the models and projections,

             20    outside users typically, but we try to make

             21    it across all types of people -- labs,

             22    industry, up on the Hill, DOE offices -- and

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                16

              1    the membership changes every year.  We try to

              2    rotate the people who are on those groups,

              3    who look at the forecast.

              4              And next is the projection analysis

              5    quality control process.  Again OIAF managers

              6    meet weekly to discuss projection progress to

              7    identify issues and priorities.  A weekly run

              8    review is again had with the analyst to

              9    discuss draft results, to identify issues and

             10    track projection progress.  It's again

             11    formatted in to the same breakout in terms of

             12    three subgroups for about three hours once a

             13    week.

             14              Now, the initial meetings are not

             15    breakout meetings for that group.  They are

             16    done as one large joint session.  And that

             17    allows the analyst to provide comments

             18    outside their work area, so I can have the

             19    residential analyst saying something about

             20    what's going on at commercial or industrial

             21    areas.

             22              The joint meetings also alert

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                17

              1    analysts to changes that may impact their

              2    efforts.  The guys modeling electric

              3    generation want to know if the natural gas

              4    prices are going to be much higher or much

              5    lower, because it directly impacts what their

              6    numbers are going to look like and the mix of

              7    capacities is going to look like.

              8              Now, early in the process the

              9    deputy administrator is briefed on the

             10    preliminarily results for key variables, for

             11    example, natural gas prices.  And the

             12    preliminarily projection results are

             13    discussed with the working groups.  This is

             14    the second time we are meeting with the

             15    working groups to basically allow input prior

             16    to finalization of the numbers.

             17              We also brief all the EIA offices

             18    on preliminarily projection results prior to

             19    completion, to get their input.  The

             20    administrator and deputy administrator are

             21    briefed on the preliminarily projection

             22    results prior to completion to get input.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                18

              1    This occurs very late in the process, but

              2    with enough time for us to do something with

              3    the results if there is an issue.

              4              And again the external peer review

              5    includes questions on the AEO projections

              6    themselves, not just the modeling which

              7    provides input to future AEOs.  And that

              8    retrospective review which I mentioned

              9    earlier measures the error in the AEO

             10    projections over time, looking back on

             11    previous AEOs.

             12              It's published every year, after

             13    completion of the AEO, and it provides

             14    information on the magnitude as well as the

             15    direction of error and it uses it in the

             16    input when making changes for the next, the

             17    future AEOs.

             18              Now, the quality control process

             19    applied during the publication writing

             20    process is multifaceted, and is probably the

             21    most detailed of all of the quality control

             22    processes.  The major analysis topics

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                19

              1    included in the Issues in Focus, which is one

              2    major section of the AEO, and the Legs and

              3    Regs, which is another major section, are

              4    developed using customer input collected

              5    through surveys as well as other contacts

              6    over the course of the year.

              7              The final topics are approved by

              8    OIAF management and then the deputy

              9    administrator.  Outlines for all major

             10    sections are completed and approved by OIAF

             11    management prior to any writing beginning.

             12    Drafts of the sections are again reviewed and

             13    approved by the relevant division director,

             14    as well as later by OIAF management itself.

             15              And the publication drafts, once we

             16    get to that stage, are professionally edited

             17    four times during the production process,

             18    following comment at different points in the

             19    review.  The drafts are reviewed three times

             20    by the deputy administrator and one time by

             21    the administrator prior to finalization.

             22              Now, all sections of the AEO are

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                20

              1    reviewed or QAed by OIAF management staff

              2    that were not involved in writing the

              3    section.  That involves checking all data to

              4    ensure that it is correct, and actually

              5    matches the projection results, which

              6    sometimes you'd be surprised, and reviewing

              7    the writing to ensure that it's readable and

              8    arguments are logical.

              9              All the major AEO publication

             10    pieces go through the EIA category one

             11    process that involves a detailed review by

             12    the other EIA offices.  Following that review

             13    a category one response memo is prepared and

             14    that discusses how the comments were

             15    addressed.

             16              For example, one is done for each

             17    of the major pieces, the Issues on Focus, the

             18    Legs and Regs, the Market Trends.  The Market

             19    Trends document, the memo going back will

             20    typically be anywhere as from 50 to 70 pages,

             21    which has simple answers of what we do with

             22    each individual comment as they're listed.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                21

              1    They can be very, very extensive.

              2              The external peer review process

              3    includes questions on the quality and

              4    appropriateness of the AEO summary

              5    publications, and that inputs into the next

              6    AEO.

              7              And of course no matter how much of

              8    this we do there's still some errors that

              9    appear during the year, and we prepare an

             10    erratum and post it online as errors are

             11    found during the course of the year.

             12              The last aspect of the quality

             13    control process addresses the methodology

             14    documentation.  All of the methodology

             15    documentation is revised by the modelers to

             16    reflect updates and input from SMG reviews

             17    from the previous year or even years, the

             18    methodology drafts or reviews by the

             19    responsible division director.  This draft's

             20    final methodology documentation is reviewed

             21    and approved by the OIAF division director

             22    prior to release.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                22

              1              And the completed methodology

              2    documentation is sent to SMG for review on a

              3    cycle that results in the review of all

              4    sections over roughly a three-year time

              5    period.

              6              That summarizes the quality control

              7    process.  It's an extensive process and I

              8    hope that it leads you to assume that the

              9    analysis that we are putting out is actually

             10    heavily reviewed, and it works.

             11              It involves both internal and

             12    external participants.  It is applied at all

             13    levels of the production process.  It

             14    provides input to both the current and future

             15    AEOs.

             16              But one drawback, I think, of the

             17    process as it's currently done is it's very

             18    time consuming and resource-intensive.  So

             19    you have to weigh that cost/benefit that's

             20    going on.

             21              For example, the quality control

             22    process for writing the AEO summary involved

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                23

              1    over 80 working days last year, and that's

              2    working days.

              3              And the last slide once again poses

              4    some questions I'd like the committee to

              5    address.  Again, first, is the external

              6    review process sufficient?  I think you know

              7    my answer, but I'll wait for yours.

              8              Do we need to expand the number or

              9    type of reviewers included in the process?

             10    Are there ways in which the review process

             11    could be streamlined?  Are there other

             12    aspects of the AEO that need to be reviewed,

             13    either more extensively or in a different

             14    way?

             15              And how do we balance that need to

             16    provide an independent projection which is

             17    what the AEO is, with stakeholders' inputs

             18    which we may or may not agree with?  We can't

             19    satisfy every single comment we get, because

             20    they are many times conflicting.

             21              And that's it.  I'd be happy to

             22    answer any clarifying questions if anyone has

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                24

              1    any, if we have time.

              2              Okay.  Thank you.

              3              MR. KOKKELENBERG:  Thank you.  I

              4    think I -- do I start off?

              5              MR. NEERCHAL:  No, I think we have

              6    another presentation.

              7              MR. KOKKELENBERG:  Oh, I am sorry.

              8              MS. KHANNA:  Except that by the

              9    time we hear Bill's talk you'll be looking at

             10    his questions, not Paul's questions.  So if

             11    you have some input --

             12              MR. HOLTBERG:  I could do it any

             13    way you want to do it.

             14              MR. NEERCHAL:  Yes, because you are

             15    also listed as a discussant.

             16              MR. KOKKELENBERG:  Well, that's the

             17    point.  In the sequence here, but if Bill

             18    feels that a --

             19              MS. KHANNA:  Why don't we have --

             20              MR. KOKKELENBERG:  It's up to you

             21    people to make the decision.

             22              MS. KHANNA:  It just seems like it

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                25

              1    makes more sense to have -- well, if anybody

              2    has any particular questions for Paul, but

              3    then have the discussion of Paul's --

              4              MR. KOKKELENBERG:  Okay.  I did

              5    have one particular question and that was

              6    specifically I haven't looked at the,

              7    basically what you call the retrospective

              8    review.  Can you say a just a few sentences

              9    about how good or accurate the Annual Energy

             10    Outlook is?

             11              And I know you forecast hundreds,

             12    if not thousands of variables under dozens of

             13    different scenarios, so it's a challenge to

             14    try to summarize that in a few sentences, but

             15    perhaps you reflected on that, and can do

             16    that.

             17              MR. HOLTBERG:  Let me correct you

             18    on one thing first, and I try to say this in

             19    every audience I can.  The AEO is a

             20    projection, not a forecast.

             21              MR. KOKKELENBERG:  Okay.

             22              MR. HOLTBERG:  And why that's

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                26

              1    important is because by definition we do not

              2    assume or try to forecast new policy or new

              3    changes.

              4              MR. KOKKELENBERG:  No, I understand

              5    that.

              6              MR. HOLTBERG:  And so when you are

              7    looking back at older AEOs it's somewhat

              8    undermined if that policy is changed over

              9    time.  That said and done, when we look at it

             10    usually we are the furthest off in terms of

             11    energy prices.

             12              MR. KOKKELENBERG:  You mean that's

             13    where the error is the highest.

             14              MR. HOLTBERG:  That's where you see

             15    the highest errors, okay?

             16              We are much better on economic

             17    growth, we are actually much better on energy

             18    consumption.  And we are much better from the

             19    supply to many extents, which means those

             20    prices tend to be offsetting.  They tend to

             21    be moving in the same direction, the error

             22    tends to be moving in the same direction.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                27

              1              But nonetheless it does, it also

              2    shows the original retrospective review was

              3    done when we switched over to the NEMS model.

              4    And it has shown a marked improvement with a

              5    newer modeling structure than what we had

              6    previously been doing, so we are starting at

              7    a point around there.  However, there are

              8    still errors that are fairly substantial.

              9    Okay?

             10              MR. NEERCHAL:  I think we'll move

             11    in to the next topic.

             12              MR. FEDER:  I was just wondering,

             13    if you can tell us what kind of criticism do

             14    you get from internal and external reviewers?

             15              MR. HOLTBERG:  All kinds.

             16                   (Laughter)

             17              MR. HOLTBERG:  No, I mean,

             18    seriously, it really does go the gamut,

             19    because you have people who have vested

             20    interests in what they are commenting on, and

             21    they'd like to see results go in one

             22    direction or another.  And so you are getting

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                28

              1    comments which are based on what's the near

              2    term doing and applying it to a long term

              3    projection.

              4              You get more academically-based

              5    comments where people are spending some time

              6    and thinking about the statistics and the

              7    trends and what's going on.

              8              They stretch, you know, pretty much

              9    across the board.  And so it's really hard

             10    for me to go, you know, on that.

             11              It really has to do with are we

             12    meeting certain trends that people see out

             13    there.  For example, are we reflecting

             14    improvements in efficiency, adoption of

             15    renewables, a lot of criticism based on the

             16    issue of carbon.

             17              Since there is no regulation out

             18    there in the AEO we do not assume carbon

             19    emissions regulation.  However, we do a lot

             20    of side cases and analysis, where we do look

             21    at potential policy on carbon, which we do

             22    point people to.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                29

              1              But that's a common criticism.  I

              2    did a presentation last week where it's the

              3    only thing I heard.

              4              Okay?  Thank you, I'll be happy to

              5    answer and I'll be here.

              6              MR. NEERCHAL:  There will be more

              7    questions later on.  And I am sure they will

              8    want a discussion.

              9              MR. WATSON:  Good morning.  Is

             10    everybody able to hear me okay?

             11              I am pleased to be able to spend

             12    time with you and look forward to the

             13    comments and suggestions provided by the

             14    committee.  What I am going to be doing here

             15    this morning in the next 15 minutes will be

             16    to give you a quick overview of the coal data

             17    surveys and the QA processes that we use to

             18    look at the data.

             19              I am also going to pose several

             20    questions to the committee and solicit their

             21    answers to those questions.  First, a little

             22    bit of an overview of our program.  We do

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                30

              1    have four surveys on the coal side of our

              2    data collection effort.  And here is some

              3    data or information about those surveys.

              4              I am going to be emphasizing the

              5    IDC, the Internet Data Collection

              6    implementation of our surveys, because that's

              7    where we really get good payoff in terms of

              8    QA, and I'll try to emphasize that as I go

              9    through the presentation.

             10              Roughly speaking these surveys in

             11    total represent about 4,500 individual survey

             12    forms that come back and have to be processed

             13    on an annual basis.  And you can see the IDC

             14    implementation was early as 2003, and then a

             15    little bit later on some of the other

             16    surveys.

             17              Here's some additional data about

             18    that process, as background information.  I

             19    am not going to be spending a lot of time

             20    talking about the 6A survey, because this

             21    next cycle will be the last for that.  And I

             22    am not going to be talking very much about

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                31

              1    the 5, which is a very small survey that goes

              2    to the coke plants in the U.S., and there are

              3    only 19 of those.

              4              So, most of my remarks will be

              5    directed primarily at the 7A and at the 3.

              6    The 7A -- we went by it very quickly -- is a

              7    data survey that goes to all U.S. coal mines.

              8    And the three 3 survey is the survey that

              9    goes to all U.S. manufacturing plants that

             10    utilize coal.

             11              I would call your attention to the

             12    column that has the percent IDC, and the 7,

             13    we are around 74 percent at this point.  The

             14    6, we are around 64.  The 3, we've been over

             15    95 percent from the implementation of the IDC

             16    back in the beginning of 2004.  And there was

             17    a direct coalition between IDC rate and the

             18    success of our QA processes.

             19              What do we mean by QA in this

             20    situation?  There are really two parts to it.

             21    One is, is the frame or the list of

             22    respondents that are supposed to be reporting

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                32

              1    on this survey a complete list?  How do we

              2    know that?  How do we know we have complete

              3    coverage of everybody that should be in the

              4    survey?

              5              And then after that, the next major

              6    issue here is what strategies and processes

              7    do we have to assure data accuracy when we

              8    receive data from our respondents?

              9              Here is a little bit of information

             10    about the frame process.  We do have

             11    thresholds, cut-off thresholds.  If you're

             12    below that threshold, you don't have to

             13    report on the survey.  Those thresholds are

             14    set according to industry standards and block

             15    out very few firms, and over time, of course,

             16    because of economies of scale, and

             17    consolidation, the thresholds have even less

             18    effect on who is required to submit data.

             19              We do have to establish a base

             20    frame, identify the people that are supposed

             21    to be in the list of respondents, and we have

             22    to revise that over time.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                33

              1              Two of our surveys, the 6 and 7,

              2    you could say that they are self-maintained

              3    in the sense that both of them depend largely

              4    upon frame information that we receive from

              5    the Mine Safety and Health Administration.

              6              Every coal mine in the U.S. legally

              7    is required to establish an MSHA ID and

              8    register with Mine Safety and Health

              9    Administration, and we use that information

             10    to keep our respondent list up to date in

             11    that case.

             12              In the case of the 3, which goes to

             13    manufacturing firms, we definitely have a

             14    problem there.  We are challenged to keep

             15    that frame maintained and I'll come back to

             16    that point a little bit later.

             17              Okay, here is some information

             18    about what we do with our data survey in

             19    terms of both gathering it from the

             20    respondents and internal checking that we do.

             21              We spend a lot of time reviewing

             22    the instructions on our data survey forms,

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                34

              1    and we take feedback from our respondents as

              2    a basis for those revisions.  If anything is

              3    confusing, obviously that needs to be

              4    changed, so there's a lot of common sense

              5    approach here to keep things moving in a good

              6    direction.

              7              We're also constantly working to

              8    make our Internet data collection interface

              9    user-friendly and to simplify the process in

             10    order to minimize human error.  So we have a

             11    lot of dropdown lists there, standardized

             12    names, those kinds of things that help our

             13    respondents accurately respond to the request

             14    for the data.

             15              The two things that are really,

             16    really powerful with Internet data collection

             17    are the next two items, and they are really

             18    respondent-side processes.  A respondent to

             19    one of our surveys must resolve all of the

             20    imbalances that could occur inside the form.

             21              For example, some of our forms

             22    require information on stocks, consumption,

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                35

              1    receipts and there is a balance equation

              2    involved.  Namely, in that case, your

              3    beginning stocks plus your receipts minus

              4    your consumption plus adjustment to stocks

              5    must be equal to ending stocks.  And if

              6    that's not satisfied by the data that you

              7    entered, you have to resolve that before you

              8    can submit your data survey to us.

              9              The other is automated edit flag

             10    checks.  We have certain ranges on certain

             11    data.  For example, prices.  Your price must

             12    be within a certain percentage of the last

             13    period price and an edit flag goes up, the

             14    respondent is challenged.

             15              They can resolve that by entering

             16    corrected data or they can put in an override

             17    comment explaining why the data falls outside

             18    the range.  And so we have for the most part,

             19    almost complete editing on the respondent

             20    side here.

             21              Now, what do we do on the federal

             22    side?  Because we don't necessarily want to

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                36

              1    just accept everything that our respondents

              2    give to us.  So we have a process that's

              3    wrapped up in a thing called the Respondent

              4    Contact Record Tool, and as the forms come to

              5    us and we see them electronically, everything

              6    that carries an edit flag will be a record in

              7    this tool and it will have a default status

              8    of pending.

              9              The next thing that happens is, our

             10    contractor goes through the pending records

             11    and looks at the comment that the respondent

             12    has put in about that edit flag and makes a

             13    quick judgment as to whether or not that's a

             14    reasonable comment.  If it is, they hit a

             15    "fast acceptance," a standard message is

             16    recorded into the data bank and the status

             17    then is changed to "Completed."  And that

             18    would include every IDC form submitted with

             19    an override comment.

             20              Of course at the beginning of the

             21    process that's mainly what we're getting.

             22    We're just getting IDC forms that show up in

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                37

              1    this part of our system as a result of an

              2    override comment.

              3              Now, we don't stop there.  The

              4    federal staff actually reviews those

              5    fast-acceptance cases and identifies forms

              6    for recheck, because our federal staff has

              7    specialized information about the industry

              8    that probably exceeds that of our contractor

              9    in many cases.

             10              So they have to impose their

             11    judgment about whether or not those

             12    fast-accepted cases are good or not, and we

             13    just iterate around that process for a while,

             14    and we don't close our surveys until all the

             15    statuses are set equal to "completed," which

             16    means they've gone through this two-step

             17    process.

             18              Here's a level 3 check that we do

             19    after all of that's over, which is built

             20    around the idea that sometimes outliers will

             21    reveal problems with the data.  So once all

             22    of the data are into the system and we have

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                38

              1    cleared everything in terms of giving

              2    everything a "completed" status, we run a

              3    bunch of checks, data field by data field

              4    doing sorts from highest to lowest.

              5              And we're looking at the top and

              6    the bottom of that list, at those outliers

              7    specifically, because sometimes somebody can

              8    make a mistake and here is an example.

              9              On some of our forms the

             10    respondents are asked to put in the percent

             11    of sulfur by weight, and they're supposed to

             12    put a "4" as 4 percent.  But if they get a

             13    lab report back and it's got 0.04, they might

             14    overlook that and put 0.04 into the data

             15    form.  They're challenged and they put in a

             16    comment that says that's the value I got from

             17    the lab.

             18              The contractor sees that, that's an

             19    acceptable response and they accept it.  Our

             20    federal staff looks at what the contractor

             21    did with it.  They react in the same way.  We

             22    see that when we look at the outliers and we

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                39

              1    catch that, and we catch a small percentage

              2    of data this way that are inaccurate as a

              3    result of problems such as the one that I've

              4    just given you as an example.

              5              The other part of the level 3 is,

              6    we also search all of the fields for null

              7    entries, because again, even though

              8    everything is reviewed twice before we even

              9    get to this stage, through a variety of

             10    routes something can sneak through and often

             11    does.  Not many, but some do, because of the

             12    comment override feature of the system.

             13              And we also do a bunch of balance

             14    checks.  Here is a page 1, page 2, where a

             15    part of the form upfront has total receipts,

             16    and page 2 has some detail broken out by

             17    receipts, but the sum of the receipts on the

             18    page 2 should be equal to the page 1

             19    receipts.

             20              That is flagged to the respondent

             21    as well, but if they put in an override

             22    comment and we don't catch it, and the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                40

              1    contractor doesn't catch it, and we don't

              2    catch the contractor, here is another place

              3    where we can catch that kind of thing.  So

              4    these are sort of backup systems to try to

              5    get the data straightened out as best we can.

              6              All of this is guided by what we

              7    call a checklist, and this is a document

              8    which can run something like five pages or

              9    so, and it provides a timeline, and all of

             10    our survey managers follow the checklist, and

             11    they're supposed to, as the process unfolds,

             12    put in comments electronically.

             13              Of course it's a Word document so

             14    they can get in there and say, "I've gotten

             15    to this stage today" and "This is the problem

             16    we're having" or "We ought to spend some time

             17    working over here in this area" and so on.

             18    And so that documents issues and problems

             19    that we have as we work along.

             20              They also have the ability to

             21    recommend changes, and then they're required

             22    to sign this at the end of the reporting

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                41

              1    cycle and submit to our CNEAF QA officer.

              2              And we do go back and look at this

              3    checklist document on a regular basis to

              4    refine that process.  The other thing we do

              5    is we hold weekly, biweekly status review

              6    meetings.  There we're looking mainly at

              7    response rate charts and cleanup progress

              8    charts, and then we have an end-of-cycle

              9    wrap-up review, what worked, what didn't

             10    work, what needs to be fine-tuned.

             11              And here are some data on some of

             12    our surveys.  This is the 2007, quarter 1,

             13    EIA survey to manufacturing firms.  The

             14    broken line shows you the response rate.  The

             15    due date for this survey is 30 days after the

             16    reporting cycle starts and you can see we're

             17    right at 85 percent.  By that time we've

             18    received 85 percent of the surveys.

             19              Because of the fact that we have a

             20    very high IDC rate here, our staff is able to

             21    clean these up also very rapidly as we move

             22    along here.  So the percent clean of

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                42

              1    everything received is well above 90 percent

              2    from day one, because we're cleaning right

              3    away.  This is a good example of what you can

              4    do with an IDC system when you have a high

              5    IDC enrollment.

              6              Contrast that with the 7A which

              7    goes to coal mines.  This is the latest

              8    survey.  This survey is due 60 days after the

              9    start of the reporting cycle, and at that

             10    point we had about 55 percent of the people

             11    reporting back in, but notice that the

             12    cleaning is delayed and stretched out a lot

             13    and doesn't actually get to 100 percent until

             14    the end.  And we're working on that, so I'm

             15    going to come to that in a minute.

             16              This gives you some sort of overall

             17    picture of what happens here and supports the

             18    two graphs that we just saw before.  So for

             19    example on the 3, where we have about 2,000

             20    survey forms every year, we have 920 roughly

             21    of these edit flags.  This is representative

             22    of what happens every cycle.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                43

              1              Only about 250 of those actually

              2    require a check-back with the respondent, by

              3    which I mean we actually have to get on the

              4    phone and we have to send e-mail.  Now, there

              5    are multiple flags for a particular site.  So

              6    this doesn't represent 250 separate people

              7    that have to be called, it represents roughly

              8    a 125 people.  That's a very manageable

              9    number.

             10              Then you come to the 7, where

             11    because of the low IDC enrollment, a lot of

             12    our forms come to us as hard copy forms, and

             13    we enter those forms ourselves.  And when

             14    they blow off edit flags, the respondent is

             15    not sitting there dealing with that.  We

             16    immediately have to schedule time to get on

             17    the phone with the respondent and clean that

             18    up.

             19              So a lot of the edit flags in that

             20    case require respondent contact.  That

             21    represents roughly about a 1,000 telephone

             22    calls and given the size of our staff, that's

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                44

              1    challenging.  That's a lot of human resource.

              2              The 6 is a little bit better in

              3    that sense, the respondent contact is more or

              4    less in a manageable size, but we could

              5    easily cut that in half because we have

              6    discovered recently that we have a bunch of

              7    un-needed edit flags there and if we got rid

              8    of those, why, it would cut these data in

              9    half, and we're in the process of doing that

             10    currently.

             11              Okay.  Here are micro-strategies to

             12    improve quality, and number one, get that IDC

             13    enrollment high, hopefully above 95 percent.

             14    We're there on the 3.  We need to do that on

             15    the 7.  Why?  Because of the powerful feature

             16    of the self-editing by the respondent, and on

             17    our side it just lets us do a lot of fast

             18    compilations of the edit flag and a fast

             19    review and resolution of the pending edits.

             20    And it allows time where we haven't spent

             21    very much time at this point, I have to say,

             22    on cross-survey comparisons, where we were

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                45

              1    able to do that and there are some

              2    opportunities for doing that.

              3              But at this point we're pretty much

              4    tasked with our current resources just to

              5    deal with everything else outside of

              6    cross-survey comparisons.

              7              In terms of macro-strategies, here

              8    I come back to the EIA-3, the survey that

              9    goes to the U.S. manufacturing firms.  We

             10    have to rely upon industry newsletters and

             11    reports to identify new coal users.

             12              We tried looking at certain state

             13    level and federal level inventories on

             14    environmental regulations, because after all

             15    this is coal burning and coal is highly

             16    regulated on the environmental side.  We

             17    tried to see if we could get information to

             18    keep to maintain our frame.  We haven't had

             19    great luck there, I have to say.  Maybe some

             20    of you have some ideas about that.

             21              And then we can do some

             22    cross-reference to some of our other surveys.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                46

              1    We've done that, and that's helped us a

              2    little bit.

              3              But that feeds into this business

              4    about aggregate balance.  For the U.S., if

              5    all the data were completely accurate, the

              6    sum of production imports and stock change

              7    should be equal to consumption plus exports.

              8    Or to say that a different way, supply should

              9    be equal to demand.  Turns out that supply in

             10    the aggregate here in our data series for

             11    coal routinely exceeds demand.

             12              And so we need to go further to

             13    figure out who we're missing on the

             14    consumption and export side.  My view is that

             15    it's probably the EIA-3.  We've got some coal

             16    users there that are not in that frame that

             17    need to be there.

             18              There could also be an issue with

             19    whether or not the export data which we get

             20    from the Commerce Department is totally

             21    accurate.  So we need to expend some

             22    resources to look into those two areas, to

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                47

              1    help us on the frame side.

              2              So here are the areas that I've

              3    identified that I thought would be things

              4    that possibly the Committee could help us

              5    with.  What weaknesses do you see in our QA

              6    process?  What can we learn about data survey

              7    methods from academic sources?  We've not

              8    spend very much time looking at the

              9    literature on data survey methods.  Maybe we

             10    ought to do that.

             11              If you have recommendations I'd

             12    certainly like to receive them from you.  Are

             13    there best practices we should know and

             14    follow?

             15              There is a problem here of being

             16    parochial.  You know, we're sort of stuck in

             17    our offices and we're busy all the time just

             18    trying to get through all those data series

             19    on a week-by-week basis, so really we don't

             20    get a lot of exposure to what's going on

             21    outside of EIA.  And maybe people outside EIA

             22    are doing things that we should be aware of

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                48

              1    and could follow.

              2              And in general, how can we assure

              3    continued quality improvement?  We do have

              4    some layers of QA that we impose on the

              5    system, but maybe there are other things that

              6    we should be looking at as well.

              7              Thank you very much.  I'll take any

              8    clarifying questions as you have.

              9              MR. NEERCHAL:  So shall we do the

             10    same thing?  If there is any specific for

             11    Bill's presentation, we entertain that

             12    question right now before we go on to the

             13    discussions.

             14

             15              MR. KOKKELENBERG:  I have one on

             16    your supply-demand.  When you're looking at

             17    coal consumers, are they pretty much power

             18    plants and other energy producers that are

             19    producing, say, electricity or steam --

             20    burning?

             21              MR. WATSON:  Right.  Well, we --

             22              MR. KOKKELENBERG:  Or more

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                49

              1    specifically, there are users for coal that

              2    don't involve that.

              3              MR. WATSON:  True.

              4              MR. KOKKELENBERG:  And a quick

              5    example, for example the foundry molding

              6    sands use a lot of coal, or used to, I don't

              7    know if they do today.

              8              MR. WATSON:  What was your example

              9    again, please?

             10              MR. KOKKELENBERG:  Foundry molding

             11    sand?  That sand is generally a mix of many

             12    things for a mold that is a cast mold?

             13              MR. WATSON:  Oh, okay.

             14              MR. KOKKELENBERG:  And coal is one

             15    of the ingredients, or used to be one of the

             16    ingredients in it, back in the technology I

             17    knew.

             18              MR. WATSON:  Right.  We do have

             19    foundries in our EIA-3.  We have people who

             20    make the furnaces for kilns, which is a

             21    little bit different than what you're talking

             22    about, so that might be something that we

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                50

              1    might want to look at.

              2              But in general, the consumption

              3    that I'm talking about here is not just the

              4    consumption that we survey on the coal

              5    program.  It includes the electric utility

              6    consumption of coal.

              7              MR. KOKKELENBERG:  Yeah.

              8              MR. WATSON:  But in addition to

              9    that we have manufacturing, direct

             10    manufacturing using coal.  But there is a

             11    small amount of coal that is used -- well,

             12    actually five to ten million tons of coal are

             13    used in commercial and institutional

             14    facilities such as military bases and state

             15    hospitals, universities.  There are many

             16    universities that burn coal.  I'm surprised

             17    you're not having demonstrations on your

             18    campus.  Maybe that's next year.  It's just

             19    around the corner I think.

             20              MR. KOKKELENBERG:  -- technical and

             21    begin that process of use —————————————— that

             22    things will do and --

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                51

              1              MR. WATSON:  Yeah.  So we survey

              2    those as well, and for a while we had a

              3    category called Ag, representing agricultural

              4    construction and mining, but I kept saying to

              5    the staff, "When do you think the last time

              6    we burned coal in a tractor was?"  So that

              7    amount of coal has dwindled to practically

              8    zero because we only have coal going into

              9    providing heat for greenhouses, that kind of

             10    thing.

             11              MR. KOKKELENBERG:  Yeah.  This

             12    wasn't a heating usage, this was -- the thing

             13    I'm familiar with is the people who

             14    manufactured the sand or blended the sand

             15    that went to foundries or sold this to

             16    foundries and it consisted of a number of

             17    organics, one of which was coal.  And they

             18    brought it in car-load blocks, or, you know,

             19    hopper cars.

             20              MR. WATSON:  Okay, yeah.  That

             21    specialty use I was not familiar with, so I

             22    thank you.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                52

              1              MR. KOKKELENBERG:  Yeah, and well,

              2    it may be being picked up somewhere, I'm just

              3    saying there will be others like it, and

              4    again, I was --

              5              MR. WATSON:  There are a couple of

              6    others like that that I'm familiar with.

              7              MR. KOKKELENBERG:  Okay.

              8              MR. GRUENSPECHT:  How big is the

              9    gap?

             10              MR. KOKKELENBERG:  That's the next

             11    question.  How big is the gap?

             12              MR. GRUENSPECHT:  How big is the

             13    supply-demand gap?

             14              MR. WATSON:  Oh.  It's somewhere in

             15    the order of 5 to 10 million tons a year.

             16    But it's always on one side.

             17              MR. GRUENSPECHT:  Right.  That's

             18    half a percent, roughly?  Half percent to one

             19    percent?

             20              MR. WATSON:  Yeah, it's a very

             21    small number, but ideally of course what you

             22    would hope for would be, sometimes positive,

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                53

              1    sometimes negative.  I would rather see that

              2    pattern, obviously, than always positive.

              3    Because that means that, systematically,

              4    you're overlooking something.

              5              MR. KOKKELENBERG:  Yeah, that's

              6    about 500 unit trains worth of coal.

              7              MR. WATSON:  It's a small amount.

              8    It's not much, but --

              9              MR. KOKKELENBERG:  500 hundred unit

             10    trains impresses me as a big amount of coal.

             11              MR. WATSON:  Uh-huh.

             12              MR. KOKKELENBERG:  Yeah.  I'm sure

             13    that much is not going into the foundries.

             14              MR. WATSON:  Yeah, I think we are

             15    going to look more carefully at commercial

             16    and institutional.  We may not have good

             17    coverage there, so we may be able to pick up

             18    a couple million tons or so there, and maybe

             19    a couple of million tons on exports.

             20              And then after that because of the

             21    sort of phasing of when you collect the data,

             22    sometimes of course our data may be out of

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                54

              1    sync with the electric power data and so on

              2    and so you should bounce up and down around

              3    zero, sometimes plus, sometimes negative,

              4    that's sort of the pattern that we would like

              5    to see here.

              6              MR. FEDER:  You mentioned

              7    identifying your users through newsletters

              8    and bulletins and things like that.  Is it

              9    possible you might be missing some new users?

             10              MR. WATSON:  True.  Some recent new

             11    users are people that are building these

             12    ethanol plants in the Midwest and they're

             13    using coal for heat in the process there.

             14    And we do get those, we do get notices of

             15    those in industry newsletters and we do have

             16    people who are looking through the

             17    newsletters for that purpose.

             18              But that's an ad hoc procedure.

             19    Maybe there's a newsletter that we're not

             20    looking at that we should be, that would

             21    identify those.  So that is an issue whether

             22    or not we are looking widely enough to get

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                55

              1    those new people.

              2              MR. NEERCHAL:  Is there a business

              3    database that kind of captures, you know,

              4    lots of businesses like Dun and Bradstreet

              5    and things like that?  Will they be useful

              6    for this purpose?

              7              MR. WATSON:  I'll put it down on my

              8    list and see if we --

              9              MR. NEERCHAL:  I mean they have --

             10    essentially it is like a list of all

             11    businesses which can employ at least a

             12    certain number of employees and so on, I

             13    think, I don't know whether --

             14              MR. KOKKELENBERG:  There used to be

             15    something called Thompson's, but I don't know

             16    if it still exists, or if that's appropriate,

             17    but I don't think they had listed inputs.

             18              MR. NEERCHAL:  No, no, they -- I

             19    mean, at least you can get the addresses, see

             20    whether there is some obvious category that

             21    you're --

             22              MR. WATSON:  Yeah, there may be

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                56

              1    something in the way in which they identify

              2    themselves which might be an indicator that

              3    they could be burning coal.

              4              MR. KOKKELENBERG:  But that leads

              5    to the question that the census of

              6    manufacturers conducted by the Bureau of

              7    Census, maybe the -- I'm not sure if BLS does

              8    it for them or who does it for them, but have

              9    you coordinated your frames with theirs?

             10              MR. WATSON:  We have, but it's only

             11    a limited effort, because the data there --

             12    for example, we can compare our data

             13    aggregated up to a state level against that

             14    data.  But then if we identify a difference

             15    they're unable to tell us, because the

             16    confidentiality, anything about the specific

             17    firms that might be involved.  So it's sort

             18    of an unsatisfactory thing except that it

             19    gives us a little bit of a clue perhaps on

             20    where to look in terms of a geographic

             21    location.

             22              MS. KIRKENDALL:  We actually had

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                57

              1    the Census Bureau, we gave them our frame and

              2    they matched to the -- whatever data they had

              3    matched with, and I think they found 60 or 70

              4    percent agreement.  Maybe we're a little

              5    closer than that on volumes because of course

              6    you get things that don't match that are

              7    really both there.

              8              And the problem is they couldn't

              9    really tell you very much about what you were

             10    missing.  I think they love it because they

             11    can fix their frames.  But all they would

             12    tell us, they could tell us where we were off

             13    in terms of SIC codes, and tell us where we

             14    were at the geographic level.

             15              MR. WATSON:  And state level too --

             16    yeah, geographic.

             17              MR. KOKKELENBERG:  They can tell

             18    you you're getting warm.

             19              MS. KIRKENDALL:  Yeah, so.

             20              MR. WATSON:  Yeah, and there's a

             21    big lag there too, at least in the survey

             22    that Nancy's talking about.  It's called the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                58

              1    NECS and it's only done every 4 years, so

              2    we're doing every quarter and so we're pretty

              3    current with this EIA-3 and so we're actually

              4    retrospectively looking to see how we're

              5    doing compared to their latest survey which

              6    they do every 4 years and then has about a

              7    two year lag before we actually get to take a

              8    look.

              9              MR. KOKKELENBERG:  Well, the only

             10    other thing I can think of is going to some

             11    of these coal distributors or sellers and

             12    saying, "Could you look over our list to see

             13    if there is somebody big that we're missing"?

             14    And that's just asking for voluntary

             15    collaboration.

             16              MR. WATSON:  Yeah.  The 6A survey

             17    that we run, that talks about or surveys

             18    distribution, so it goes to all U.S. coal

             19    mines.  Actually more accurately it goes to

             20    the coal companies that have coal mines in

             21    each of the coal-producing states and so

             22    they're able to report on all the coal mines

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                59

              1    in a given state, and we asked them who do

              2    they distribute their coal to.

              3              We stopped asking them probably

              4    about 10 years ago the information about who

              5    was purchasing the coal.  So we only know it

              6    goes to EG -- electric generation -- or to

              7    manufacturing, to export, and so forth, that

              8    kind of consumer type.  We don't know

              9    anything specific.

             10              In order to do something like that

             11    we have to go to go to OMB and ask

             12    authorization to do that.  So, it's not

             13    anything that is easily done.

             14              I agree with your point.  That

             15    would have been a beautiful way actually to

             16    have cleaned this up, and we did have

             17    internal discussion about this three years

             18    ago when we did our form 3 authorization, but

             19    there really wasn't much interest among

             20    people above me -- put it that way -- in

             21    doing that.

             22              MR. FEDER:  Okay.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                60

              1              MR. KOKKELENBERG:  Thank you.

              2              MR. NEERCHAL:  We'll move on to the

              3    ASA discussion part.  Moshe you want to go?

              4              MR. FEDER:  I guess we're

              5    addressing both presentations.  With respect

              6    to the first one, it's hard for me, as a

              7    humble statistician who doesn't have much

              8    understanding of the substantive matter, to

              9    do it without seeing any numbers, and I was

             10    wondering, there was a mention of

             11    retrospective look at the historical data and

             12    I guess, even though I appreciate the

             13    difference between projections and forecasts,

             14    is there a way to look at those differences

             15    between the projections and the actual

             16    numbers that historically can be obtained?

             17              Maybe in some future meeting, we

             18    could look at this and see how well the AEO

             19    is performing.  It could be easier for

             20    statisticians to look at it.  I think overall

             21    the process is very impressive.  Input is

             22    being solicited from stakeholders.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                61

              1    Internally and externally people are

              2    providing a lot of input.

              3              The speaker was wondering if the

              4    processes may be too intensive.  Hard to

              5    judge, but definitely no stone has been left

              6    unturned.  There's a lot of good thinking

              7    that's going there, and that looks very

              8    impressive, but again, without seeing actual

              9    numbers it's hard to judge.

             10              And I ask also about what criticism

             11    they have received and I understand that you

             12    would get, especially from academics, some

             13    people questioning the assumptions so I'm not

             14    worried about it, it looks like the process

             15    is good.  That's all I have to say.  Ed

             16    probably could say much more.

             17              MR. KOKKELENBERG:  I could speak

             18    more.  Do I say more, is the question.  I

             19    think, in general, we have seen that Paul has

             20    given us a good overview of a process that is

             21    very thoughtful and very reflective and very,

             22    very professional in turning out the annual

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                62

              1    energy outlook.  I would recommend or hope

              2    that other agencies emulate the thoroughness.

              3              Having said that, I wonder if

              4    perhaps there's too many internal reviews

              5    that are looking over everybody else's

              6    shoulder and perhaps too many meetings

              7    associated with the actual reviewing, or

              8    perhaps too frequent.  Each type of meeting

              9    may be essential, but do they have to do it

             10    every week?  Or something like that, and that

             11    might help streamline the process.  But there

             12    is going to be a lot of reluctance, I think,

             13    on the part of administrators to shorten that

             14    too much because of the hazard of coming out

             15    with a serious glitch.

             16              The second thing I have to ask then

             17    is, well, I go back to how good is the

             18    result, as the result presented is pretty

             19    good.  But it's got to be compared to what?

             20              So when you come down to saying

             21    prices are the most difficult thing they have

             22    the projections on, I guess my first question

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                63

              1    would be, well, what's the accuracy of the

              2    other projections that are made by others,

              3    particularly for the long term?  And

              4    secondly, how does the accuracy of the

              5    short-term energy outlook price projections?

              6    How accurate is that compared to others that

              7    might be officially out there?

              8              Prices are going to be the hardest

              9    thing, I think, to project, more because of

             10    this difference between the idea that this is

             11    not a forecast, this is a projection, and

             12    policies do change and therefore prices that

             13    are actually realized do change.

             14              Markets react to the annual energy

             15    outlook and they say, "Well."  And so they

             16    take that action which would kind of undo

             17    price projections if they are perhaps not

             18    favorable to the economic actor.  So they

             19    protect themselves.

             20              So that one is going to be a hard

             21    one to handle because you can't put policies

             22    in there, except by scenarios, and you've got

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                64

              1    enough scenarios as it is, I think, in the

              2    annual energy outlook.

              3              When you ask if the external review

              4    process is sufficient, I'm not sure.  I can't

              5    answer that because I'm not sure what the

              6    types of external reviewers are, the number

              7    of them, or the depth they are asked to go

              8    into.  And so that answers or doesn't answer

              9    or it fails to answer the first and second

             10    questions.  Streamlining, again the internal

             11    review process is very, very extensive.

             12              Finally, I would come back to this

             13    balance question that you have.  I'm not sure

             14    who these other stakeholders are, how you

             15    identify them, and how do you evaluate their

             16    analysis or projections.  And everybody is

             17    making a projection about energy nowadays.

             18    How do you filter out the ones that you don't

             19    consider?

             20              Don't you think you are putting too

             21    many in?  You should be proud of the

             22    projections you do and it is nice to say that

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                65

              1    there are other competent projections out

              2    there.  But you don't have to include the

              3    other universe.  The interested user will

              4    find it, particularly if they've got some

              5    particular cause they're espousing.

              6              SPEAKER:  And we could ————————

              7    some oil.  The oil market is into the action,

              8    so it is not surprising projections are up

              9    significantly whereas financial gain is much

             10    more by divesting markets ——————————————————

             11    to see what happened to natural gas estimates

             12    ———————————————————————— oil projections.

             13              And if I may make a very quick

             14    comment, I know in some of these survey

             15    results earlier, maybe in our last session,

             16    there was some concern about whether or not

             17    this volume was used or picked out ——————————

             18    ———————————— were less likely to use

             19    ——————————————————.  I'm not sure.

             20              MR. NEERCHAL:  Walter, go ahead.

             21              MR. HILL:  You said that gas is

             22    more predictable than domestic.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                66

              1              SPEAKER:  If we guess the

              2    natural --

              3              MR. HILL:  Do you think --

              4              SPEAKER:  ————————————————————

              5    ——————————————————————.

              6              MR. HILL:  It is hard to predict.

              7    It is --

              8              MR. NEERCHAL:  Neha.

              9              MS. KHANNA:  Yes.  I must reason

             10    that AEO is one the most important policy

             11    documents that comes out from the EIA and I

             12    actually very much appreciate the internal

             13    review process that was highlighted by the

             14    presenter because I think the fact that

             15    groups that are not working in that

             16    particular sector, unless you actually get to

             17    read it or talk about it I think is very

             18    important because it ensures consistency

             19    across groups and it actually ensures

             20    communication across different groups of

             21    people within the EIA.  And I think that's

             22    actually very important to the overall

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                67

              1    quality of what the EIA puts out.

              2              And the question was how to

              3    streamline the process.  I would strongly

              4    recommend not abridging it, that in keeping

              5    the internal review process, even though you

              6    were talking about possibly, you know, having

              7    it less frequent.

              8              I mean, I don't care about the

              9    number of hours as long as the quality of the

             10    internal review is maintained.  If you can do

             11    that with reducing the number of hours,

             12    great, but a backlog is really important.

             13              The one question I had about the

             14    coal mine service MNA was, you said that is

             15    very low IDC enrollment and that was part of

             16    the reason why it was dramatically ————————

             17    ————————————————————————————.  So the

             18    question is why is there such a limited

             19    ——————————————————.

             20              MR. WATSON:  It is a matter of

             21    poor --

             22              MS. KIRKENDALL:  Bill, come up to

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                68

              1    the table so we'll get it on-mike.  And Paul,

              2    maybe you should come up to the table too.

              3              MR. HOLTBERG:  Okay.

              4              MR. WATSON:  So several factors.

              5    One has to do with the fact that the frame

              6    there is established by the coal mines that

              7    are enrolled with the Mine Safety and Health

              8    Administration.  So when we start our survey

              9    in January, the Mine Safety and Health

             10    Administration, it is itself -- which is a

             11    quarterly survey -- surveying mines for the

             12    fourth quarter in that January-February-March

             13    period.

             14              And, roughly speaking, every year

             15    in the U.S. coal industry, mining industry,

             16    these are the smaller firms, not the big

             17    ones.  Mines come and go.  We have about 200

             18    to 300 mines out of a base of 1,500 that come

             19    in and leave every year.  And so a good part

             20    of those we don't know about until we get

             21    that last slug of data from the Mine Safety

             22    and Health Administration.  So that's part of

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                69

              1    the answer there.

              2              So what we are doing this year is

              3    we are going to meet with MSHA.  They are

              4    able to survey all these 1,500 people on a

              5    quarterly basis within 75 days.  Last year,

              6    it took us 210 days to do it.  So we are kind

              7    of anxious to see what they are doing that we

              8    are not doing, and it could just simply be

              9    they have a lot more resources.  We don't

             10    know yet.  We haven't gone and visited them

             11    yet.  We're going to do that.

             12              And we are also looking for ways to

             13    get from them very early information about

             14    the new people that are joining the frame at

             15    that point.  So that's a part of the

             16    response.

             17              The other part is, we just need to

             18    ask our resources ourselves to push harder on

             19    improving that, and we can do that now.  We

             20    know who was in the frame last year because

             21    if they weren't IDC, we did get a form from

             22    them.  So we can go back and identify them.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                70

              1              And then there is a third reason.

              2    Some small operators in the 21st century

              3    don't know anything about the Internet.  In

              4    order to file IDC, you have to have an

              5    Internet connection.  So there is always

              6    going to be a small number of people who will

              7    only send you a form by fax or by mail.

              8              But if you have any other ideas

              9    why, please.

             10              MR. NEERCHAL:  John.

             11              MR. WEYANT:  Actually, through our

             12    own studies in ICCC and other venues, I spent

             13    a lot of time thinking about scenarios.  I

             14    actually like to start with some fundamental

             15    impressions.

             16              It seems to me that the job of

             17    producing these scenarios is -- just the way

             18    it gets structured is kind of an impossible

             19    task on the one hand.  On the other hand, it

             20    is probably the most useful thing EIA does

             21    that maybe anybody does in the analytic

             22    community.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                71

              1              So let me explain a little bit more

              2    about that.  You know, if you think about

              3    what you are doing, kind of the first rule of

              4    decision theory is be clear about your

              5    objectives and why you are doing the analysis

              6    and stuff before you do anything else.  That

              7    also carries through, I think, to scenario

              8    design.  Because often scenario design is an

              9    element of some analysis.

             10              Now, the problem that you have here

             11    which makes it an almost impossible task is

             12    you have everybody wanting you to do

             13    everything all of the time.  I don't have to

             14    tell you that.  So I think on Paul's issue

             15    regarding balancing the independent

             16    productions of analysis with stakeholder

             17    inputs, I think the way I come out on that is

             18    you're providing useful information, probably

             19    not credible information, not obtainable

             20    anywhere else, in pieces -- you know,

             21    elasticities, supply schedules, baseline

             22    data, benchmark data for baseline -- but you

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                72

              1    can't expect to be moving that information

              2    out in a way that suits the needs of

              3    everybody out there who is trying to make a

              4    corporate decision or a trade association

              5    decision or EPA, or whatever.

              6              The reason I go through this, which

              7    seems kind of academic and abstract, is I

              8    think that's kind of the reality and I think

              9    sometimes I forget and we forget.  So I have

             10    people come up to me and say, wow, there is

             11    an R&D analysis out there that somebody else

             12    has done and they are using that darn EIA

             13    baseline projections.  I'm sure you hear this

             14    all the time.  And I go, jeeze, you know,

             15    you're in one industry, one part of one

             16    industry with one particular objective,

             17    usually with lots of politics involved, and

             18    you want EIA to do the scenario that's the

             19    perfect fulcrum for doing R&D assessment for

             20    that.  Just think about how much comes out of

             21    that.

             22              So I think it's helpful and

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                73

              1    probably psychologically good for you guys to

              2    kind of think about, you know, this filling

              3    up half of all of those glasses without ever

              4    feeling unfulfilled because you haven't

              5    filled up everybody's glasses completely at

              6    every turn of the screw.

              7              So I would absolutely not keep

              8    doing this kind of work and I would interface

              9    with the outside world in a way that's

             10    pragmatic and realistic.

             11              Now, on the workload, it does seem

             12    for both of you guys that it seems like a lot

             13    of work.  I mean, it seems like that's got to

             14    occupy a lot of the total resources here.  So

             15    there are two things one could think of

             16    doing.  One is, I think Ed started on, kind

             17    of not doing things, maybe even the whole

             18    outlook.  Is it legally possible to, say, do

             19    an AEO every other year, and an IEO in the

             20    odd years, or something like that?  It may

             21    not be legally possible.

             22              And actually even in things like

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                74

              1    the ICCC, which has a 5- or 6-year cycle, a

              2    bunch of us did a critique after four rounds

              3    that said, first of all, you ought to spend

              4    years just assessing how the process has gone

              5    after four rounds.  Secondly, things don't,

              6    even on that cycle, this kind of long-term

              7    issue, things don't move that fast.  So you

              8    may be better off leaving more time.  And

              9    it's actually parallel in a way, so people

             10    can do more new work to be assessed.

             11              I mean, there does seem like, from

             12    an outsider's point of view, when you get

             13    these peaks of demand for service reports,

             14    and you have all these other processes going

             15    on simultaneously, that it's like people have

             16    three or four jobs at the same time.

             17              So I don't know.  It's probably

             18    worth considering, but may be hard to do, to

             19    cut down on frequency of issuing these major,

             20    major reports.

             21              And I actually wonder how much

             22    things actually do change when they're put in

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                75

              1    place.  If you had actually only historically

              2    done that every 2 years, there would be an

              3    outcry from the user community and would you

              4    feel like you kind of missed something or you

              5    didn't keep up with what was going on?

              6              On the internal review, you guys

              7    both talked a lot about that.  It does seem

              8    extremely detailed and well done.  I suspect

              9    though that we can't really think of what we

             10    would leave out.

             11              So we could turn it around and ask

             12    you, what do you think is kind of not

             13    worthwhile in the sense of not leading to

             14    very much revision, changes, qualification of

             15    the numbers as you go on?

             16              Is there anything that you would

             17    leave out or do every other year?  Or you

             18    know, you could actually rotate and say, this

             19    round we're going to really focus on this

             20    one, and in the next round we're really going

             21    to -- and you could actually chew that up

             22    according to where you think the real

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                76

              1    problems are.

              2              Finally on the price thing, I think

              3    the other thing besides the policy

              4    uncertainty is expectations of behavioral

              5    change, I think, affect prices more,

              6    especially changes in expectation.  Again,

              7    I've dabbled a little bit in stock market

              8    analysis so what you see is, if you change

              9    your mind about the future a lot overnight,

             10    that makes you go buy stocks or sell stocks.

             11              This is kind of just a follow-on to

             12    what Ed said.  You could say, oh, it's not

             13    just next year that seems to be changing a

             14    little bit, but that gives me a whole

             15    different idea about what's going to happen

             16    for the next 10 years, and so I'm going to

             17    adjust my capital stock and make new

             18    investments.

             19              And I think that shows up because

             20    of the momentum in the system and the

             21    fixidity of the capital stock in prices

             22    sooner.  So I think that is, as Walter

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                77

              1    pointed out, when you've got international

              2    players and the OPEC quotas and things like

              3    that that makes it ever more difficult.

              4              It's again one of those things like

              5    the people running away from the bear and the

              6    guy stops to put his sneakers on and the

              7    other guy says, "You think you can outrun the

              8    bear?"  And he says, "No, I just need to

              9    outrun you."

             10                   (Laughter)

             11              MR. WEYANT:  Standards revision

             12    should not be perfection, I think it should

             13    be -- you know, you could look at some of the

             14    other groups out there, but again they all

             15    have their own issues.  So I think you should

             16    keep doing what you're doing and fend off the

             17    forces of the dark side.

             18                   (Laughter)

             19              MR. WEYANT:  So think it through.

             20    What you do so well is kind of balance all

             21    these interests, and I think one thing I

             22    would do if I was you, which you probably do

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                78

              1    do, is when people complain in these ways,

              2    say, hey, I'm not going to do your job for

              3    you.  We're providing information you can

              4    pick up.  It's kind of what I do with my

              5    little low academic level, is I pick up stuff

              6    that's useful for my particular analysis.  I

              7    said, well, I don't expect you guys to do

              8    scenarios that are absolutely perfect for,

              9    you know, evaluating hydrogen vehicles.  And

             10    that's what they have actually done.

             11              You know, there is no way I'm going

             12    to go and say, well, you know, he does a

             13    terrible job, because I have an analysis job

             14    that's focused on some specific thing and you

             15    don't give me the perfect baseline for me.

             16    You just got to do a "come on, now" kind of

             17    response.

             18              MR. NEERCHAL:  You were wondering

             19    if it should be done once every other year

             20    and I was thinking, is once a year frequent

             21    enough?

             22                   (Laughter)

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                79

              1              MR. NEERCHAL:  So it partially

              2    depends on the stakeholders, do they still --

              3              MR. WEYANT:  Yes, yes.  It actually

              4    -- there is a loss of continuity of sorts.

              5    There may be just different time phasings --

              6              MR. NEERCHAL:  ——————————————————

              7    ————————————————————.

              8              MR. WEYANT:  If you did it every

              9    half year it would be necessary to do it

             10    every half year.  The more you did it the

             11    more you --

             12                   (Laughter)

             13              MR. NEERCHAL:  Absolutely.

             14              MR. WEYANT:  If you started out

             15    doing it every other year, doing it every

             16    other year would probably be fun.

             17              MR. NEERCHAL:  I think what John

             18    said, you take every other year as the review

             19    year, that may be a good way to kind of phase

             20    in that, you know, step-down.

             21              MR. WEYANT:  Yeah, but what we

             22    strongly recommend and I perceive, because

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                80

              1    usually that the response is, oh, but things

              2    changed in the policy that, you know, that

              3    negotiations are going on, we've got to hit

              4    the meetings, you know, we've got to -- that

              5    kind of drives the process.

              6              But you could do -- in their

              7    context, you could do expert, I forget what

              8    they said, they have a whole bunch of titles

              9    for special reports on hot important issues,

             10    like gyrations in oil prices.

             11              Meaning you could just focus on the

             12    new stuff every year and either do only the

             13    new stuff or only really audit the control

             14    and push on the things that seem most

             15    different from the previous ones.

             16              I don't know how you feel, because

             17    it may be that you feel that, you know, every

             18    write-on there is a lot of new stuff that

             19    comes out, and therefore you really need to

             20    do a comprehensive.

             21              Or those of you involved in it may

             22    feel like things don't change that much and

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                81

              1    you're going to have to go through a lot of,

              2    you know, rolling up your sleeves and digging

              3    in in areas where very little change has

              4    taken place.

              5              And you kind of know that before

              6    you start on account of that.  It's always

              7    hard in these processes to get flexibility

              8    into it, but with more flexibility it would

              9    actually I think make the workload a little

             10    bit more manageable.

             11              I was going to say the problem is

             12    when things in the energy field are very

             13    hectic, that's when people want you do it

             14    more often and that's when it's usually more

             15    necessary to do it more often.  You know, oil

             16    prices are changing dramatically, you have to

             17    do it.  And, you know, natural gas prices are

             18    changing, you have to do it.

             19              MR. CONKLIN:  But that's what

             20    you've got all the service reports for --

             21              MR. WEYANT:  And we've got six

             22    service reports at the same time.  Now, a

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                82

              1    decade ago, there was probably an opportunity

              2    to do it less frequently.  I'm sure we didn't

              3    take the opportunity to do it less frequently

              4    back then, but if you probably missed a year

              5    back then you could have probably not been

              6    too missed.

              7              Right now, it's a pretty difficult

              8    time because of the volatility in energy

              9    markets along with the policy analyses that

             10    are being considered in the Congress as well

             11    as our efforts to try and get on the track of

             12    a new energy modeling system.

             13              So we're being hard pressed on a

             14    few different fronts.

             15              MR. CONKLIN:  I didn't even bring

             16    that one up, because that's -- and obviously

             17    that's the whole audit control system --

             18              MR. WEYANT:  So what we do try and

             19    do, you know, from a modeling perspective, is

             20    there are areas where we don't have to focus

             21    a lot of resources in some years.  Like, for

             22    the past two years we haven't focused a lot

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                83

              1    of resources in the electric power sector

              2    because the model before that has been pretty

              3    well developed.  We've spent a lot of

              4    resources for a number of years to do it.

              5    Right now, we're going to have to put a lot

              6    of resources into our oil and gas modeling

              7    efforts, and we are.  Hopefully they'll begin

              8    to bear fruit next year or the year after.

              9              MR. CONKLIN:  It's a tough balance.

             10              MR. WEYANT:  Yeah.  That's on

             11    account of limited resources -- which I

             12    pushed for continuously.

             13              MR. CONKLIN:  Yeah.  But we can

             14    recommend that to the --

             15              MR. KOKKELENBERG:  Well, I'm going

             16    to build on this idea a little bit, and I'm

             17    trying to -- I'm struggling for the right

             18    words, but the area of, say, should we worry

             19    about ethanol or what is the hot topic is one

             20    thing, but in terms of streamlining the

             21    process, one might look at the process level.

             22              And I think John said this, sort

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                84

              1    of, or perhaps Moshe.  The question of a

              2    reviewer making extensive comments versus a

              3    reviewer who pretty much finds no problems or

              4    issues, whether they are internal or

              5    external, might be identified as a way to cut

              6    down on the process, not the areas that are

              7    investigated, but the process of review.

              8              So if there is a reviewer who only

              9    occasionally -- some element that only

             10    occasionally finds some things to correct,

             11    that reviewer may be used less frequently

             12    than the weekly or the monthly or even for

             13    the entire AEO.

             14              That might be one way to help

             15    streamline the process.  Other than that I

             16    think I have to go back to what Neha said.

             17    It's one of the flagship documents, and

             18    whatever changes you do make should not

             19    weaken a flagship but strengthen it.  That's

             20    all.

             21              MR. HOLTBERG:  One question I ask

             22    you on that though, and I think John said

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                85

              1    this as well, part of the process of review,

              2    processes of for example working groups, at

              3    some times you may have a reviewed error or a

              4    particular meeting we don't get that much

              5    ——————————.  But it also serves the function

              6    of telling you what you're doing.  There is

              7    an information transfer that's going on.  It

              8    has value, even if it's not getting the

              9    comments about it.  That's something you have

             10    to balance and I don't know how that works.

             11              MS. KHANNA:  Well, that's exactly

             12    what I said, that the internal review is

             13    actually really important.

             14              MR. CONKLIN:  A lot of times at

             15    those meetings too is if you don't provide

             16    the rationale for what you're going to do, it

             17    comes back at you in criticism, whereas if

             18    it's provided ahead of time they understand

             19    it and it's no longer a criticism.

             20              MR. HOLTBERG:  That may actually

             21    cut off some of those --

             22              MR. CONKLIN:  Right, it cuts off

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                86

              1    some criticisms.

              2              MR. NEERCHAL:  Thank you.  Thank

              3    you.  This was a good group discussion we

              4    had.  We are running a little bit behind as

              5    you know.

              6              MS. KIRKENDALL:  All right.  Well,

              7    this will be a fairly quick presentation

              8    because I'm particularly interested in your

              9    comments and recommendations.

             10              Back in the early '90s when NEMS

             11    was first developed, the credit assessor

             12    office of the statistics and methods group

             13    ran a model quality audit program.  And this

             14    was probably particularly important because

             15    NEMS was new and as part of that we did three

             16    different steps.

             17              We did independent expert reviews

             18    of the model methodology.  We still have an

             19    independent expert review process that goes

             20    on so that's something that we intend to

             21    keep.

             22              We also had a contractor help doing

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                87

              1    sensitivity analysis and other specific

              2    analysis of model performance.

              3              And then the last step involved

              4    verifying documentation, a little bit like

              5    what we're doing for documentation reviews

              6    right now that Paul talked about.

              7              But we also did model archival, so

              8    the statistics and methods group looked at

              9    the documentation, and we looked at the

             10    archive model, we made sure they were

             11    consistent and then we archived it.  And now

             12    we don't think that that's particularly

             13    important because the model archival is a

             14    more automatic process.  So that led to what

             15    we're doing now.

             16              So the question's what should we do

             17    this time.  NEMS is going to be revised and

             18    improved and changed and we would like to

             19    have a value-added process with our model

             20    quality reviews.

             21              So some of the things that we

             22    thought about that we like -- and I've

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                88

              1    consulted with John on this -- hopefully we

              2    can come up with something that everybody

              3    likes.

              4              One thing that we helped with is

              5    establishing the workshop that I talked about

              6    yesterday that's on December 11th.  They've

              7    commissioned papers on topics of particular

              8    interest.  We're having outside experts come

              9    in and present their white papers and then

             10    we'll have a discussion with internal people

             11    and some outside people also.

             12              And that will feed into what we

             13    decide to do.  So that's a nice format for

             14    getting outside input.  At least I think it

             15    is.

             16              We also would plan on doing

             17    independent expert reviews of component

             18    design reports for the various modules as

             19    they are produced as part of that.

             20              When we were talking to John and

             21    other folks who have worked on modeling, they

             22    particularly think that we need to have

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                89

              1    practical reviews.

              2              Many times the people who do the

              3    reviews are a little bit more theoretical

              4    than we really need.  We need to make sure we

              5    get people who could give us inputs that we

              6    can act on.

              7              And ways of doing that are to try

              8    to build our list of reviewers to find even

              9    more good people to engage in the review

             10    process and we can probably tighten up the

             11    contract we have with them to try to make

             12    sure that they focus on practical issues.

             13              In the past we did a sensitivity

             14    analysis through a contractor and this time

             15    we've got George, and you'll hear about

             16    George Lady's project in the next

             17    presentation.

             18              He is doing a lot of work on

             19    sensitivity analysis of the current methods

             20    and modules.  And people on my staff --

             21    Lawrence here, and Preston -- are going to be

             22    working with George to learn how he is doing

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                90

              1    the sensitivity analysis.  So I'm hoping that

              2    over time we'll have in-house staff who can

              3    help with regular sensitivity analyses of new

              4    models as they come up.

              5              The other thing we do are model

              6    documentation reviews and that one's always

              7    interesting.  One question that came up in

              8    our discussions is how can we make our

              9    documentation useful to others.  We have some

             10    documentation that's really good and people

             11    find very useful and other documentation

             12    that's not quite so good, and it's just not

             13    clear.  What is a good systematic approach

             14    for trying to improve our documentation

             15    overall?

             16              So here are the questions.

             17              Do you have other ideas about what

             18    EIA's model quality audit program should

             19    entail?

             20              What else should EIA do to get

             21    outside expert advice about model quality?

             22              Do you have ideas about building a

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                91

              1    list of external reviewers?

              2              And then finally the issue of

              3    documentation.  What needs to be done?  How

              4    do we figure out what needs to be done to

              5    improve our documentation?  Once we know that

              6    then maybe we can act on it.

              7              And that's it.

              8              MR. NEERCHAL:  Now, we are back on

              9    time I think.  It's time for committee

             10    discussion.  I want to start off with a

             11    question for Nancy.

             12              I think you have annual NEMS model,

             13    a NEMS conference, or something like that.

             14    And I always think that if I have any problem

             15    that I am looking for any idea that I can,

             16    you know, use, I sometimes turn it into a

             17    take-home exam for my students.

             18                   (Laughter)

             19              MR. NEERCHAL:  And I think you will

             20    be amazed with the kind of things they come

             21    up with.  Of course, you know, I give them

             22    credit if they come up with something.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                92

              1              And I am wondering if you can

              2    somehow, you know, turn this into a question

              3    that you can maybe make into a student

              4    competition, or something like that, in the

              5    model context, and get some really new people

              6    to think about the problem.

              7              MR. FEDER:  Actually, that reminds

              8    me of something.  I just went last week to a

              9    conference in Minneapolis of syndromic

             10    surveillance.  I'll tell you why I'm

             11    mentioning this in a minute, but this is all

             12    about detection of outbreaks looking at some

             13    data such as giving over-the-counter

             14    purchases, ER admissions, things like that.

             15    And they actually put on have a competition.

             16    If you go to syndromic.org for univaried (?)

             17    analysis of the data, it's a timed series,

             18    and if you can, I guess, be the best, the

             19    victor would win the prize.

             20              So that's a way to do it, is to

             21    make a competition.  Put some EIA series or

             22    some special data and let students do the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                93

              1    work for you.  And maybe, I know this sounds

              2    sensational, with a NISS.  Maybe it's another

              3    way to do it.

              4              MR. NEERCHAL:  I think you will be

              5    amazed at what they think up.  I mean, they

              6    think totally out of the box.

              7              The good thing about being young.

              8              MR. KOKKELENBERG:  On identifying

              9    or building a list of external reviewers, I'm

             10    going with the National Academy.  I would

             11    talk to people over there because in the

             12    process of building their panels and their

             13    committees, they consider a very large number

             14    of people for auditing, kind of focussing it

             15    down to six or ten people that they have.

             16              And people at either CNSTAD or

             17    maybe in other areas of the National Academy,

             18    they have lists of people that they're

             19    willing to share, that they've considered for

             20    areas that are relevant to analysis.

             21              I was amazed at how efficient they

             22    are at gathering data on individuals, both in

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                94

              1    academics and non-academic areas.  And they

              2    have perhaps expertise in this.  I'm not

              3    quite sure how they get that, but they've got

              4    it.  So that might add to your reviewer list.

              5              In the case of one of the other

              6    questions you had, the sensitivity analysis,

              7    I think much of that should be really done

              8    inside because with the way computers work

              9    nowadays, you can sit there and say, what if

             10    I tweaked a variable, and in a few minutes,

             11    depending on how complex your model is, and

             12    how many other ramifications there are to

             13    this, you can get an answer, and you can do

             14    your sensitivity analysis very, very rapidly

             15    for limited trials.

             16              So that is probably easier to do

             17    internally than to have an external effort on

             18    that one, because the external effort,

             19    they've got to gather all this modeling,

             20    gather the data, get the idea of where you

             21    want the sensitivity analysis, and you can

             22    just sit in front of the computer and do it

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                95

              1    yourself pretty quick.

              2              MR. WEYANT:  The National Academy

              3    is a great idea.  And perhaps we could help a

              4    little bit with a couple of networks at the

              5    time involved in ICCC and EIA that we're --

              6    I'd have to review, because he hit a key

              7    issue here, because I've been involved in a

              8    few of these efforts and sometimes I'm

              9    surprised that they work much better, based

             10    on the people, I mean, and the state of the

             11    art.  They work much better than I thought,

             12    and sometimes much worse than I thought.

             13              And I think that it's kind of --

             14    and maybe contracts helps you do this, but

             15    it's kind of clearly and forcefully defining

             16    and maybe picking people who are willing to

             17    take advice.  What the mission is --

             18    academics often will say, well, it's the way

             19    you're doing that, but the perfect model

             20    would look like this, and then of course the

             21    first thing that happens is they say, but

             22    there is no data.  We have the data, we have

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                96

              1    to do that.  So there is a lot of that

              2    finessing that goes on.

              3              Actually when I was working with

              4    Bill Hogan years ago, people used to come in

              5    and say, these models are off-guard, didn't

              6    you know.  And we would say, well, what's the

              7    problem?  And they'd say, well, you don't do

              8    this and that and the other thing, and the

              9    person would usually say these models are not

             10    very good and useful, and he would actually

             11    just look at them and say, compared to what?

             12    And then they're going, what?  What do you

             13    mean by that?

             14              What he meant was, you know, the

             15    models are only useful in comparison with

             16    your next best thing, which in those days,

             17    and still to some extent this day, is just

             18    purely winging it.

             19              So there is kind of that standard

             20    of comparison thing, but I think in the

             21    external reviews, you really have to say, you

             22    know, supposing we were doing a brand new

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                97

              1    model, which we are, how would you go about

              2    it?  And whatever constraints you think

              3    actually are practical constraints, you have

              4    to say, well, we can't hire -- we have this

              5    thing in California that's supposed to do all

              6    the analysis with the new rules.

              7              So they were told, well, you don't

              8    have any economists, it's all engineers, but

              9    we'll give you 50 new slots.  Eighteen months

             10    later, the budget has only been there for two

             11    new people.

             12              So you could say that, you know, we

             13    have this mix of people we have, this is the

             14    kind of things that we feel we could do, as

             15    opposed to leaving everything wide open as if

             16    you had, you know, ten years and $100 million

             17    to do new model development.

             18              But I think from my side of this,

             19    that's actually very important for the

             20    Committee to fully recognize that and place

             21    what their task is as opposed to something

             22    that's more obsolete.  Because I do think you

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                98

              1    get a lot of non-pragmatic advice from people

              2    who assume that the data isn't there.  Well,

              3    it should be there, but then you've got to go

              4    get it.  If we had figured this out or

              5    decided this 20 years ago then we would have

              6    the data to build a better model that you'll

              7    see can do the kind of metrics or whatever

              8    that would be most useful.

              9              MR. NEERCHAL:  Yeah.

             10              MR. CONKLIN:  I was going to build

             11    on something John said, and what we try and

             12    do is -- and you know, it's one thing to

             13    throw this information out to anybody who

             14    wants to comment on it, but if it's much more

             15    relevant if I get energy modelers, serious

             16    energy modelers like the people that John

             17    worked with and other energy people that do

             18    this to provide comments.  Because we get a

             19    lot of criticism from people that basically

             20    wing it.  And you know, you have a, you

             21    haven't matched my perceptions today as they

             22    exist.  So what --

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                                99

              1              MR. WEYANT:  Yeah, I might change

              2    my -- I haven't thought about this much or I

              3    might change my mind tomorrow, but I'm not

              4    going to admit that.

              5              MR. CONKLIN:  And so our first

              6    response to that was, well, what of our

              7    assumptions do you disagree with?  And, you

              8    know, we've got volumes of assumptions and

              9    documentation to start, but they don't want

             10    to deal with that.  We just don't agree with

             11    your answers, more likely.

             12              So for them to be useful to us, how

             13    do we improve model documentation?  We've got

             14    to actually go to people that read our model

             15    documentation and try and do their modeling

             16    based on that or get rid of those --

             17              MR. WEYANT:  Well, good luck.  One

             18    strong point of the EIA is documentation so

             19    one reason people always want to use it as a

             20    benchmark for what they do is it's -- back on

             21    this internal review thing -- it's viewed as

             22    probably the only, which is frustrating for

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               100

              1    me, because I'm trying to learn about the

              2    other models.  I don't know how you deal with

              3    looking at other people's documentations, but

              4    it's not as good as yours is.

              5              MR. CONKLIN:  Maybe we have to

              6    bring them in and kind of talk to them.

              7              MR. WEYANT:  Yeah.

              8              MR. NEERCHAL:  So obviously you are

              9    looking for a few thorough reviewers rather

             10    than a whole --

             11              MR. WEYANT:  You know, the white

             12    paper is another good strategy, if you can

             13    get somebody who is very thoughtful to talk

             14    about it, maybe in stages, what you could do

             15    around the current architectural review mark.

             16              And then you don't have to kind of

             17    take it onboard as a very forceful, you know,

             18    be-all and end-all of recommendations to

             19    completely change course over the next year.

             20    I think that's got to be frustrating from you

             21    guys' point of view, unless you're going to

             22    do a new model, to have people essentially

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               101

              1    say, start this whole part of the model from

              2    scratch.

              3              MS. KHANNA:  One thing which maybe

              4    you do insist -- and I don't know, so it's

              5    very rude -- but when you go to seek an

              6    external review, I mean, what you want to do

              7    is present very specific questions.  And then

              8    think about how you come to this Committee.

              9              Sometimes the best responses you

             10    get from us is when the question is very

             11    specific to a particular presentation, not a

             12    very broad question.  Because then you're

             13    going to have people just go off on a tangent

             14    in whichever direction they want.

             15              So if you actually have an external

             16    review you sort of make them focus on a very

             17    tiny grid of modeling.  Here is where we are

             18    unsatisfied, for this reason.  Here is where

             19    we would like to end up.  How would you

             20    suggest we get there?  And I think you might

             21    get a lot more out of it.

             22              MR. CONKLIN:  I think we always go

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               102

              1    into it with that as a goal.  I'm not exactly

              2    sure if we actually are able to achieve that.

              3    And we work with Nancy's group closely in

              4    these independent expert reviews.

              5              Sometimes in the case of specific

              6    technologies we are very successful.  We have

              7    a need to know about a coal to liquids

              8    technology so we approach people that have

              9    experience in that area and we get some very

             10    good information.

             11              When it gets less useful is when,

             12    you know, we're having a problem forecasting

             13    petroleum product prices.  You know, could

             14    you look at the parts of our model that do

             15    that and help us, you know, get smarter

             16    about?  Then it's a much weaker situation.

             17              MR. NEERCHAL:  Anything else?  So I

             18    think we should we get into a break as we --

             19              MR. KOKKELENBERG:  Can I add one

             20    comment?  I think you mentioned, you wanted

             21    practical advice as opposed to theoretical.

             22    I think one of the things that over the years

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               103

              1    I've noticed is, and I'm not sure this speaks

              2    to that issue.

              3              The issue is that a number of the

              4    presentations I have seen, in terms of model

              5    building, are a bit ad hoc.  And they may be

              6    the best way to go about these things because

              7    the theoretical underpinnings may not be too

              8    solid or it may be difficult in applying

              9    because we don't have the data, the ideal

             10    model we want.

             11              But I think it's a concern that an

             12    outside reviewer or this group may have is

             13    that data mining and ad hocery are perhaps

             14    pejoratives, but that's basically the way one

             15    skeptically looks at some of these studies --

             16    that well, we found that the number of

             17    bananas on sale at the local market really is

             18    a good forecaster of the energy prices.  And

             19    we kind of scratch our heads.

             20              So you're going to have to have

             21    some theory in some of these things.

             22              But I want to underscore what John

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               104

              1    said, which was that the ideal model, they

              2    require data we don't have.  And so you do

              3    have to look at the number of ————————————

              4    ————————————————.

              5              MR. WEYANT:  ————————————————————.

              6              MR. KOKKELENBERG:  Well, okay.

              7              MR. WEYANT:  But it might just

              8    work.

              9                   (Laughter)

             10              MS. KIRKENDALL:  If we're smart

             11    about it then it's possible.  I mean, your

             12    modeling needs ———————————— and data

             13    requirements which would be an ideal way for

             14    this to unroll too.

             15              MR. KOKKELENBERG:  Well, one might

             16    wonder if the number of bananas really is

             17    good at helping you forecasting, what are you

             18    missing in the theory that ties bananas to

             19    the forecast, whatever those bananas are?

             20              MR. WEYANT:  We're counting

             21    tortillas that are tied into bio-field --

             22              MR. KOKKELENBERG:  Right.  You

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               105

              1    know, why is tortilla prices more --

              2              MR. WEYANT:  -- who would have

              3    expected that?

              4              MR. KOKKELENBERG:  -- and it ties

              5    to costs.

              6              MS. KHANNA:  And I think this is

              7    just another commentary which is when you

              8    talk about prices, especially oil prices, I

              9    mean, I don't think you can expect that to

             10    predict oil prices really accurately.

             11              MR. CONKLIN:  Well, you and I can

             12    agree on that.  There is a lot of people out

             13    there that do expect us to be able to

             14    accurately predict world oil prices,

             15    petroleum product prices.

             16              MR. FEDER:  But if you're normally

             17    modeling, you try to get in as many things as

             18    you can and show that the residual is quite

             19    low, so that proves the predictability but it

             20    is possible putting some predictors that will

             21    do better than just linking it or just banana

             22    prices.  What you have to do is reduce it to

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               106

              1    purely random noises, victims to the stock

              2    market or anything else.

              3              MS. KHANNA:  Oh, I'm not sure that

              4    I'm totally successful on the stock market

              5    either.

              6              MR. FEDER:  But those who know

              7    don't reveal it.

              8              MS. KHANNA:  Oh especially if you

              9    can --

             10              MR. FEDER:  I know I won't reveal

             11    it to you, Nancy.

             12              MS. KHANNA:  Especially if you're

             13    trying to predict 30 years out.

             14                   (Laughter)

             15              MR. FEDER:  That means that the

             16    residual is indeed white noise, which means

             17    you don't have the crystal ball but then the

             18    critics cannot say, but I can do better,

             19    because you can't and if you could, you

             20    wouldn't be talking to me.  That's what I'm

             21    saying.  If they criticize you they don't

             22    criticize you because it's -- it's because

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               107

              1    there is something you can still do beyond,

              2    but you just can't wait.

              3              MR. CONKLIN:  Well, but also to go

              4    beyond that, in terms of the model, the thing

              5    that drives these models besides oil prices

              6    is really time.  And if you can build

              7    strength on those things that are very, very

              8    important, and then allow for a range of

              9    different oil prices, lines that are

             10    important but you can't build a strong

             11    forecast, and you do that in the annual

             12    energy outlook.  I think that's one of its

             13    strengths.  You do allow for different

             14    scenarios for oil prices, high and low growth

             15    economy and those are the things that drive

             16    the model.  Counting those is probably the

             17    best you're going to be able to do without

             18    the crystal ball.

             19              MR. NEERCHAL:  Thank you.  And

             20    that's a lot of input, I think.

             21              MS. KIRKENDALL:  And if you have

             22    other ideas, let me know.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               108

              1              MR. NEERCHAL:  And I think we're

              2    going to head into the break and I feel

              3    better if I say that I'm being dragged into a

              4    meeting -- not quite literally -- back in the

              5    campus at 12:00 noon, so I have a very able

              6    acting chair for the home stretch here when

              7    you come back.

              8              MS. KIRKENDALL:  So what time do we

              9    come back?

             10              MR. NEERCHAL:  I think we are

             11    breaking about 15 minutes later.  We should

             12    be back by 10:50.

             13                   (Recess)

             14              MR. BLAIR:  We're going to have an

             15    update report from George Lady on NEMS

             16    Forecast Evaluation and my understanding is

             17    that Neha and Walt will fill in as

             18    discussants.

             19              MR. LADY:  This is very much a work

             20    in progress and in some sense the questions

             21    to the Committee which are supposed to

             22    structure how the presentations go, were

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               109

              1    already asked last time.  And so I'm really

              2    going to report on what has been done since

              3    then.  And as I'll go along, some of the

              4    major decisions that reflect what I'm doing

              5    are questions.  And the Committee therefore

              6    has a chance to agree with what was proposed

              7    to do last time or give me suggestions.  So

              8    this is very open-ended.

              9              The overview of the project is to

             10    work through the methodological and other

             11    considerations that will support SMG,

             12    bringing about a quality control product,

             13    conceivably on an annual basis, which is

             14    designed to confront the accuracy not only of

             15    NEMS but ultimately the short term forecasts

             16    as well to present the differences between

             17    the forecasts and the eventual realizations

             18    of the variables.

             19              And on top of that, try to

             20    understand how differences in the conditional

             21    assumptions made at the time the forecasts

             22    were prepared end up explaining the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               110

              1    differences between the forecasts and the

              2    actual events.

              3              Here is sort of a laundry list of

              4    the sort of things that would be at issue.

              5    Particular ones that I am worrying about at

              6    this point, in terms of methodology

              7    development, are the parts of the conditional

              8    variables that are inherent to the models.

              9              There are other considerations

             10    which are probably in many cases just as

             11    important, such as changes in laws and

             12    regulations, accidents or one-time events,

             13    and changes in the definitions of the data

             14    series that can make a big difference between

             15    what eventually is offered as the actual

             16    outcome and what was forecast.

             17              This is not beyond the scope of the

             18    overall project but it is beyond the scope of

             19    what I am actually dealing with at this

             20    point.  Further, of the model-specific

             21    influences -- conditional variables, if you

             22    will -- I'm focusing for the moment on the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               111

              1    ones that would be relevant to the shorter

              2    time horizon for NEMS.  So for the regression

              3    analyses that I will report on later, I'm

              4    only looking at projections three to eight

              5    years ahead with NEMS.

              6              Longer term, you have other issues

              7    having to do particularly with technology and

              8    so forth which is for the moment outside of

              9    what I'm concentrating on.  So specifically,

             10    the influences that I'm going to be looking

             11    at are going to be things like the weather,

             12    prices, and the macro economic variables that

             13    drive the energy forecasts.  Things such as

             14    technology, which are obviously important,

             15    I'm setting aside for the moment and that

             16    will be in the next phase, so to speak.

             17              An important question is how to do

             18    this.  And there are basically three ways to

             19    do it, in my opinion at least.  The best way,

             20    and the Committee comments last time proposed

             21    this as the best way for the model-specific

             22    conditional variables, would be to save the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               112

              1    model and then when the time comes rerun the

              2    model with the actual values cleverly, one at

              3    a time, changed to differentially pick out

              4    how the actual weather and so forth would

              5    have hopefully improved the forecast if it

              6    had been known.

              7              For NEMS at least this is trickier

              8    than you think.  With the short term model, I

              9    believe we will do that.  That is my

             10    intention.  But for NEMS, to keep NEMS around

             11    in order to run it later, is not trivial.

             12              For example, I was helped by OIAF.

             13    I should say, by the way as an aside,

             14    anything I report on today, OIAF was

             15    absolutely supportive of me in a wonderful

             16    way.  I could never have done anything unless

             17    they had been very patient with me.  I was

             18    part of the problem.  I had a learning curve.

             19              But the truth of the matter is that

             20    NEMS itself as a computer program also

             21    depends on the development environment in

             22    which it runs.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               113

              1              For example, Microsoft has a new

              2    operating system called Vista.  One of the

              3    operating platforms that NEMS runs in is not

              4    supported by Vista.  So when I bought the new

              5    current version which is, part of the problem

              6    was not my learning curve -- although I have

              7    to admit that that was part of the problem --

              8    but the truth was that the software

              9    environment that was on my new computer was

             10    not like the software environment on OIAF's

             11    computers.  This is for the last year's

             12    version of the model.

             13              So the short story is, in my

             14    opinion, keeping NEMS around for a decade and

             15    trying to run it when the time comes is very

             16    hard to do and I'm not even sure that as a

             17    practical matter that it can be done.

             18              Another thing you could do, which

             19    is number three, is just simply save the NEMS

             20    solutions that were made for the AEO and for

             21    special studies and try to pick out of them

             22    the kinds of sensitivities that you want.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               114

              1    This I think is not possible, because there

              2    is just not a wide enough range of variation.

              3              So, as of April, and the discussion

              4    with the Committee, I'd launched myself on a

              5    project to acquire in New Jersey a version of

              6    NEMS and then to run sensitivity analyses

              7    that would support the creation of a

              8    database, that would allow me to pick out the

              9    sensitivities of the particular conditional

             10    variables that I was looking for.  And I did

             11    it.  God bless OIAF.  They absolutely made it

             12    possible.

             13              So that I, now, in New Jersey, can

             14    run any NEMS scenario I want to as long as

             15    I'm not using the integrating portion of the

             16    model.  So I can pick out for the structural

             17    supply and demand pieces the sensitivities to

             18    the major influences.  And that is what I

             19    have done.

             20              The idea of the method is to

             21    artfully design a set of NEMS solutions that

             22    will enable the sensitivities that you are

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               115

              1    trying to identify to be picked out through a

              2    regression analysis of the solution data.

              3              Accordingly, just to go through the

              4    list of what I did, first thing to do, of

              5    course, is to find what the roster of

              6    variables is.  And OAIF has two very nice

              7    documentation resources.  One, they have a

              8    fine summary of NEMS which accounts for the

              9    way the variables are passed within the model

             10    and they have a data dictionary.

             11              So again I had a learning curve,

             12    but the resources were right there and I

             13    could see, for each of the modules that I

             14    wanted to work with, how to go at getting the

             15    variables.

             16              First thing I did was to look at

             17    how austere versus comprehensive these

             18    regression specifications would have to be.

             19    I was focusing on the demand modules

             20    particularly.

             21              And the singular question that I

             22    had to answer was whether or not electricity

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               116

              1    demand was a function of many different

              2    prices or just electricity's price.  And here

              3    is an example of the way it works.  So here

              4    it is for the commercial sector.  The demand

              5    elasticities for distillate, natural gas, and

              6    electricity, which were the three fuels that

              7    I modeled for that sector, and as you can

              8    see, the own elasticity is just what it

              9    should be.  It is robust and the right sign.

             10    The cross elasticities, except way out at the

             11    end for natural gas, are basically

             12    inconsequential.

             13              So for my goal of looking at the

             14    three- to eight-year time span for the NEMS

             15    solution, having looked at this, I felt here

             16    and elsewhere that in modeling sectoral

             17    demand I could ignore the the cost-price

             18    effects because they were so small.

             19              Which in terms of then designing

             20    the number of solutions, saved me a lot of

             21    work because, as the Committee proposed I do

             22    last time, I should use something like a

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               117

              1    Latin Hypercube Design and do all of the

              2    possible combinations and variations, and

              3    that gets to be very expensive if the number

              4    of variables, the explanatory variables, is

              5    large.

              6              So the specification that I used

              7    was extremely austere.  But in the

              8    sensitivity runs that I made going into this,

              9    I feel it can be defended.  And what it

             10    amounts to is that compared to the current

             11    Air Analysis that OAIF reports which does

             12    things by fuel, this approach right now is

             13    disaggregating on the consumption side by

             14    sector and then further for each sector

             15    disaggregating with respect to two or three

             16    different influences.  So the level of detail

             17    is going up quite substantially, in terms of

             18    what is being tracked.

             19              So for most of the fuel demands,

             20    the upper equation was exactly the one I used

             21    for the regression.  I've had the own price,

             22    some obvious drivers, such as floor space, in

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               118

              1    the case of the commercial sector demand, and

              2    then all of the infrastructure which is in

              3    NEMS and which in the end might be broken

              4    out, particularly for longer time frames.  I

              5    let that be and used the lagged endogenous

              6    variable as the instrument that would keep

              7    track of this kind of variation.

              8              MR. WATSON:  D is a constant there?

              9    I mean --

             10              MR. LADY:  Pardon me?

             11              MR. WATSON:  D is just a constant.

             12    It is not a difference operator?

             13              MR. LADY:  T?  T is just time.

             14              MR. WATSON:  D.

             15              MR. LADY:  Oh, D is just a

             16    constant.  I'm sorry.  Let me look -- here we

             17    go.  No, D would be the coefficient of the

             18    lag.

             19              MR. WATSON:  Okay.

             20              MR. LADY:  Yeah.  And there is a

             21    constant term A there.

             22              Anyway here is an example of one of

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               119

              1    the regressions and later we can look at the

              2    data that were used for this but one of my

              3    messages to the Committee is that, as far as

              4    I can see, it is working like mad.  These

              5    are, at least from an economist's standpoint,

              6    who is used to, how shall we say it, not

              7    necessarily the tightest of fists in terms of

              8    data analysis, this is just fine.  So the

              9    significance of the marginals and the overall

             10    fifth, given R squared, are very, very good.

             11              Further, the feedback through the

             12    lagged endogenous variable, which gives you

             13    an inner temporal effect for changes, worked

             14    very well and I could compare it to the model

             15    output.  And the regression and the model

             16    output particularly really fit very well.  So

             17    within the limits of what was attempted, this

             18    has been very successful in my opinion.

             19              Let me just give you a sense of

             20    what I was thinking when I was looking

             21    residential sector electricity for a second.

             22    One of the problems of course is to wonder,

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               120

              1    once you have your explanatory variables

              2    identified, what kind of functional

              3    relationship should you posit when you go

              4    ahead and try to fit the model with this

              5    approximation.

              6              I worked with NEMS solutions maybe

              7    for a decade and have found that it is pretty

              8    linear with some notable exceptions.  So what

              9    I did was I went ahead and I ran some

             10    sensitivity tests to check the relationship

             11    between how each of the exogenous variables

             12    individually would affect the forecast,

             13    versus when I changed them all at the same

             14    time.

             15              So if you look at this output, the

             16    first column shows you for residential

             17    electricity demand how the singular change in

             18    the price variable affected the demand.

             19              Then the second column shows you

             20    how the singular change in the household

             21    variable affected demand.

             22              And in the third column I just

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               121

              1    added them up.

              2              Then I ran a case where I changed

              3    both of those inputs at the same time.  And

              4    the next column is the difference there.  And

              5    then I took the ratio and as that is close to

              6    one, the relationship appears linear.  And as

              7    you can see it is close to one.  So this made

              8    me not so fearful of using a very simple

              9    specification.

             10              Then, in keeping with the

             11    Committee's advice last time -- look down

             12    here at the bottom -- I ran all possible

             13    combinations where I had a single high/low

             14    variation defined for both the price and the

             15    driver, in this case households.

             16              And since I only had the two

             17    explanatory variables, I could do a complete

             18    Latin Hypercube specification -- which the

             19    Committee said to do, so I did it.  And the

             20    results as you see were quite compelling.

             21    Again, the fits, all of these regressions

             22    are, from an economist's standpoint at least,

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               122

              1    of the very highest quality.

              2              And then quickly to brag more, we

              3    can look at the summary for electricity.  And

              4    there it is for the three demand sectors that

              5    I model electricity for, and the number in

              6    parenthesis underneath of the elasticity is

              7    the T statistic for the associated

              8    coefficient.  And as you can see, everything

              9    is quite significant and the fit is quite

             10    tight.

             11              Yes, ma'am?

             12              MS. KHANNA:  Just a quick

             13    clarification.  How did you compute the

             14    elasticity for that --

             15              MR. LADY:  The elasticity is the

             16    coefficient from the linear regression

             17    multiplied through by the ratio of the means

             18    of the explanatory variable --

             19              MS. KHANNA:  Ah, so it is the --

             20              MR. LADY:  Right.  So it is taken

             21    at the means of the database for the forecast

             22    data.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               123

              1              MS. KHANNA:  How did you compute

              2    the standard error of that elasticity?

              3              MR. KOKKELENBERG:  That's the

              4    standard error of the coefficient --

              5              MS. KHANNA:  Oh, that is the

              6    standard -- no, that is --

              7              MR. KOKKELENBERG:  Well, the T

              8    statistic is --

              9              MR. LADY:  The T statistic is just

             10    the standard error divided into the

             11    coefficient one.

             12              MS. KHANNA:  No, but the --

             13              MR. KOKKELENBERG:  That is the T

             14    statistic from the elasticity.

             15              MR. LADY:  No, it is T statistic

             16    for the coefficient.  However, this was

             17    mentioned last time.  I at a push of a button

             18    can run it in log form, where the coefficient

             19    will be the elasticity and it really doesn't

             20    make very much difference.  In other words,

             21    this is the T statistic for the coefficient

             22    but it is not misleading as to the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               124

              1    significance of the marginal that you are

              2    measuring.  If I had done this in logs, the

              3    elasticity would have been about the same and

              4    the T statistic would have been about the

              5    same.  At least in the times I looked.

              6              Finally, we can admire the results

              7    for the ten data series that I looked at.  In

              8    order to do this, in addition to the AEO 2007

              9    Base Case to model ten fuels, with sectoral

             10    distinction, so in the sector fuel

             11    combinations, there were ten examples that I

             12    went through.  I had to run 94 scenarios for

             13    NEMS.

             14              And one of the things that came out

             15    of this is that this project if it comes to

             16    be established in SMG as an annual activity,

             17    is really another pair of hands and set of

             18    eyes, as a diagnostic resource to NEMS.

             19              In other words, there is no way I

             20    think you could run a hundred scenarios,

             21    changing just a few things, and not find

             22    something in NEMS that would be worthy of

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               125

              1    communicating back to the modelers to deserve

              2    inspection, if not remedy.

              3              And so it was here.  There were a

              4    number of things I found.  Some of the things

              5    I found I didn't understand was because I

              6    didn't understand, but some of the things I

              7    found that I didn't understand was because

              8    they weren't understandable, and the modelers

              9    were thankful and made changes.

             10              So that this project besides

             11    bringing together I think in the end a

             12    credential for the forecast, because I

             13    believe when the conditional variable's

             14    influences on the differences is measured and

             15    reported you will find that the forecasting

             16    systems are dong very well.  Much better than

             17    just a nominal comparison without this kind

             18    of analysis associated with it.

             19              So, that is my report.  I'm using

             20    an austere specification because at this

             21    point the project is aiming at a rather near-

             22    term part of the NEMS projection period.  For

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               126

              1    the austere specification that I used every

              2    thing seems to be fine.  There are no

              3    problems as far as I can see.

              4              Downstream, the idea is to take the

              5    regression results and to substitute in them

              6    the actual variables.  To pick out the

              7    differences as associated with failure to

              8    forecast the weather, forecast price and

              9    forecast economic activity.  So the

             10    Committee's advice and the basic idea of the

             11    project I think is proceeding well.

             12              MR. BLAIR:  I think we are

             13    scheduled to discuss at this time --

             14              MS. KHANNA:  I actually don't have

             15    much to say because -- I mean, there isn't

             16    much to say about this, except you know, you

             17    ran your regressions, you got really amazing

             18    fits, which, I mean, given your model, you

             19    are going to mag the variables at 21 years of

             20    data.

             21              I'm not surprised you get the

             22    results that you do.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               127

              1              MR. LADY:  Well, the lag give you a

              2    good fit but the marginals are good as well.

              3              MS. KHANNA:  Right.  So the one

              4    question -- I actually have questions more

              5    than comments, really.  Maybe my questions, I

              6    hope to understand and then provide some

              7    advice as well.  What purpose does this final

              8    regression actually run?  What do you hope to

              9    learn from it?  I mean, if it hadn't fit as

             10    well, what would happen?  What would it tell

             11    you?

             12              MR. GRUENSPECHT:  You talked about

             13    running 100 regressions and --

             14              MR. LADY:  Hundred --

             15              MR. GRUENSPECHT:  -- and make

             16    some --

             17              MR. LADY:  One hundred scenarios.

             18              MR. GRUENSPECHT:  Hundred scenarios

             19    and finding some, I think you said you found

             20    some anomalous --

             21              MR. LADY:  Yes.

             22              MR. GRUENSPECHT:  So maybe that

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               128

              1    would be, you know, as part of answering this

              2    question, what the regressions tell you, you

              3    know, maybe talk a little bit more about what

              4    those were.

              5              MR. LADY:  Well, the process of

              6    running the regressions does provide a

              7    diagnostic resource to the modeler.  So I

              8    don't know if I should tell stories on OIAF,

              9    just exactly, but one of the things in NEMS

             10    is an accounting for the weather.  But this

             11    is obviously there for general reasons and it

             12    is not there for the purpose of predicting a

             13    cold winter in the year 2025.

             14              But to make it work, in other

             15    wards, we need to figure out how to change

             16    the weather variable.  I did predict a cold

             17    winter in the year 2025.  And the model did

             18    something very strange.  And it was a simple

             19    problem that was instantly fixed.

             20              If you look back at some of the

             21    cost elasticities, they are the wrong sign.

             22    But they are small.  You wonder why that is.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               129

              1    So there is a diagnostic feature to this.

              2              As far as the regressions,

              3    themselves, why do you do that?  The main

              4    reason is to give you later a way to evaluate

              5    the influence of the actual outcomes for the

              6    conditional variables compared to what was

              7    assumed.

              8              So six years from now, we can look

              9    at a forecast made last year, and then we can

             10    put in the actual weather, the actual price,

             11    the actual household measure and see how

             12    those different conditional variables when

             13    done accurately would have hopefully made the

             14    forecast closer.

             15              MS. KHANNA:  Without actually

             16    rerunning the whole model?

             17              MR. LADY:  No, exactly.  In fact,

             18    my belief is you can't rerun the whole model.

             19    You can't -- you won't be here eight years

             20    from now.  It is just not going to happen.

             21              In addition, in my opinion, if you

             22    do this year to year, you are going to have

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               130

              1    another way to understand how the model is

              2    changing.  And sometimes you can see changes,

              3    across model verges and regressions, which

              4    again look anomalous.  And in the past there

              5    have been diagnostic results.  In other

              6    words, the elasticity has changed in a way

              7    that just doesn't make any sense.  And that

              8    was tracked down.

              9              Further, for NEMS, the structural

             10    elasticities, as the economist views them,

             11    the elasticities of supply and demand to

             12    their major influences, are implicit to the

             13    model.

             14              And to run the regressions brings

             15    out an understanding of the way the model is

             16    modeling the world.  So the regressions give

             17    you a way to do your conditional analysis on

             18    errors, it allows you to keep track of how

             19    the model is changing.  The process of

             20    generating the data might give you some

             21    diagnostic help.  And then you have a way of

             22    talking about the model which at least from

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               131

              1    the economist's standpoint, is much more

              2    clear and intuitive.  All of which I think is

              3    good.

              4              MS. KHANNA:  Okay, well, I guess, I

              5    don't have much to say on this.  So I will

              6    just stop right here.

              7              MR. HILL:  Yeah, that story you

              8    told initially was basically instructive

              9    about Microsoft Vista not being able to

             10    handle the older model.  I think there might

             11    be a review, some place maybe, this must be

             12    before it came out.  And it is really, as I

             13    said, it is a little bit different so you

             14    might put it down.  So it obviously can do

             15    things that the old version can't do.  But I

             16    am wondering if you are happy with what it

             17    can do?

             18              MR. LADY:  Actually, we had a

             19    certain amount of humor, if that is the right

             20    word.  OIAF was being very helpful to me.

             21    And they explained that one reason that they

             22    were being helpful was because I was

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               132

              1    encountering the problems they were going to

              2    encounter when they upgraded their platforms.

              3    So they let me suffer so that they could get

              4    it straight.

              5              MR. HILL:  ——————————————————————

              6    ————————————————————————————————————————.  I

              7    have got a colleague who prefers smaller

              8    regression models to a larger and structural

              9    equation models, and what we call -- it would

             10    predict -- what would you call it?  ————————

             11    ———————————————— certainly, and you can see

             12    that when you are applying this single

             13    regression model and it does quite well.  And

             14    it is consistent with, you know, with what

             15    one of my colleagues says should happen.  And

             16    I remember a discussion the last time, about

             17    running a model by putting in the explanatory

             18    variables after you know their value so that

             19    you -- you look into -- you don't know now,

             20    what --

             21              MR. LADY:  Right.

             22              MR. HILL:  -- the weather will be

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               133

              1    next year.  But next year, a year from now,

              2    you know what the weather was.

              3              MR. LADY:  Exactly.

              4              MR. HILL:  So it is good to see

              5    that.  And your constant in modeling was

              6    austere -- yeah, that is a —————— word in

              7    terms of -- but I like that.  Now, I was

              8    wondering what -- a quick comment -- whether

              9    or not they are, off the cuff, whether or not

             10    some of the variables you think are sensitive

             11    to being austere, but obviously you can check

             12    -- check the sensitivity -- is that --

             13              MR. LADY:  To check

             14    sensitivities --

             15              MR. HILL:  If you think the model

             16    is very sensitive to some item, like the

             17    price of bananas and then you can sort of

             18    look at that as one of the variables and see

             19    whether or not that --

             20              MR. LADY:  Yes.  In fact, as was

             21    mentioned earlier, when you have the model on

             22    your desk, and you have a hot flash of

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               134

              1    interest in something, you can just push a

              2    button and you can find out what the model

              3    will do.  So you can investigate many

              4    sensitivities, which I did, because I had to

              5    in the end conclude what would be the

              6    regression specification to use.  And

              7    sometimes it is counterintuitive.

              8              You know, economic activity is

              9    going to drive the demand variable.  But just

             10    particularly which measure of that actually

             11    in the model does the job is sometimes not

             12    immediately obvious.  You have to tinker a

             13    little bit.  But in any case, yes, if you

             14    wanted to see how bananas' prices would do,

             15    you could do it.

             16              MR. HILL:  —————————————— older

             17    data —————————— indexed to inflation ——————

             18    you'd come back, some time later.  And

             19    hopefully it will bring in knowledge to the

             20    common desktop --

             21              MR. LADY:  Right.

             22              MR. HILL:  -- that's up to -- it

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               135

              1    really looks -- it really looks like you are

              2    getting incredibly good results.  Yes,

              3    against your model, very impressive.

              4              MR. LADY:  I'm very pleased.  I

              5    think it is working the way I hoped it would

              6    work.

              7              MR. BLAIR:  Thank you, Walter.  If

              8    I can take the liberty, Chair's liberty, to

              9    ask a quick question?  The regression, you

             10    know, you're doing this, essentially, I

             11    guess, the sensitivity analysis aside, the

             12    forms of regressions you used, once you lock

             13    in on those, this is over what period of

             14    time?  This is, you know, and they serve sort

             15    of a near-term pattern thing.

             16              MR. LADY:  For this particular

             17    exercise, I pooled the solution data for the

             18    projections for the year 2010 through 2015.

             19              MR. BLAIR:  I guess what I am

             20    wondering about is, you know, we run an

             21    analysis, lock in on the regressions, do we

             22    have exposure to the, you know, the sensitive

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               136

              1    and the regressions themselves changing over

              2    time?  And having only broad, limited,

              3    compensation scenarios, you know, a limited

              4    number of regression parameters, so you just

              5    can't get back to where we want to be five

              6    years from now?

              7              MR. LADY:  The time frames of the

              8    regressions can be whatever you want.  But

              9    the solutions themselves, the spirit of this

             10    is that in the end, what you do initially is

             11    what you've got.  So this, of course, would

             12    suggest trying to create the biggest database

             13    you can.

             14              MR. BLAIR:  Yeah.  Exactly.

             15              MR. KOKKELENBERG:  I just have a

             16    question, actually.  When the NEMS came out

             17    of its shell, 20, 10 years or 15 years ago,

             18    what kind of computers was this run on?  Was

             19    this run on desktops?

             20              MR. LADY:  I believe the desktop

             21    version and I'm not an OIAF person.  But I

             22    know what is available on the Internet, which

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               137

              1    I understand to be limited to desktop

              2    versions.  So I think it goes back to 2001 or

              3    2000.  Before that it was not desktop.

              4    Here's somebody who actually knows, who can

              5    tell us.

              6              MS. BEDICKER:  This is Erin

              7    Bedicker.  I modeled the commercial sector.

              8    Originally, NEMS was run on the mainframe

              9    here, correct.

             10              MR. KOKKELENBERG:  Well, this is a

             11    complex model with all kinds of elements

             12    coming in from a variety of offices.  And it

             13    really is almost a, what used to be a

             14    mainframe problem.  Probably still is, in

             15    many ways, where you have a bunch of priests

             16    and servants and priestesses who know the

             17    mainframe, and run it consistently through

             18    time.  When you migrate to PCs, you are at

             19    the whims of people like Microsoft who decide

             20    a new gadget is going to be interesting and

             21    you just undermine the ability to take those

             22    extremely complex problems and simply rerun

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               138

              1    them on the newer equipment.  And you've

              2    encountered that.

              3              I don't know what the solution for

              4    that is.  I mean, I am not saying you should

              5    go back to an old 4381 IBM machine and

              6    enshrine it, and keep it running it -- but

              7    this is a huge problem, because Microsoft, as

              8    soon as you get this thing solved, assuming

              9    you can get it solved in the next couple of

             10    years, Microsoft will have called Ladysmith

             11    or something.  And then, you know, it will be

             12    an entirely new version.  And you won't be

             13    able to evaluate the progress.

             14              MR. HILL:  I'm sure you can be

             15    happy with older software.

             16              SPEAKER:  I don't think so.

             17              MR. KOKKELENBERG:  But it may not

             18    run on a newer platform.  That is the problem

             19    with these things.  You know, maybe it should

             20    be run on Unix.  I don't know.  But the issue

             21    is that it is really hard to find somebody

             22    who can even write Fortran nowadays.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               139

              1              SPEAKER:  That's the only thing

              2    that I can do.

              3                   (Laughter)

              4              MR. LADY:  Well, my thought is that

              5    if you are artful and energetic, you can

              6    create a large database of solutions.  Now,

              7    admittedly questions will eventuate where you

              8    didn't anticipate them, but still you can

              9    probably make a good guess, and a lot of

             10    things you are going to want to know later

             11    and you can archive the solutions and

             12    Microsoft can't do anything about that, in

             13    other words.  You have --

             14              MR. KOKKELENBERG:  That may be.  I

             15    think the issue though, is that you -- you do

             16    have to spend a lot of time if you change

             17    equipment and platforms and software, trying

             18    to emulate something that was run five or ten

             19    years ago.  And there --

             20              SPEAKER:  And that is how we are

             21    trying to beat them.

             22              MR. KOKKELENBERG:  I know, and that

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               140

              1    should be the issue.  The issue should be how

              2    good is the model not how -- you are not a

              3    software designer.  And you had to become

              4    one, to some extent.

              5              MS. KIRKENDALL:  Actually, it is --

              6    it is --

              7              MR. KOKKELENBERG:  And I don't have

              8    a solution for it.  Just an observation.

              9              MS. KIRKENDALL:  And it is sort of

             10    interesting, because you know, we've had this

             11    standard that we are going to make archives

             12    of all of our models, and we are faithfully

             13    doing that.  Only problem is you can't run

             14    them after a point in time because of

             15    technology.  The hardware/software changes.

             16    So, now what?

             17              MR. GRUENSPECHT:  Well, I've

             18    actually pulled -- you know, I don't know if

             19    guys will be actually be able to run it, but

             20    I actually pulled some very detailed

             21    information from 1996, actually, because

             22    somebody proposed -- it was very interesting.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               141

              1    We got into a fight with the -- not a fight,

              2    we don't fight because we are non-combatant,

              3    but we got into a dispute about the -- there

              4    was a claim that would be a tremendous amount

              5    of cellulosic ethanol by 2017, say.  And we

              6    said, no, we really don't think so.  And

              7    someone said, well, gee, you know, how much

              8    ethanol did you think there'd be in 2006, you

              9    know, and trying to say ten years before

             10    2006.  That's why it was 1996.  And you know,

             11    but the number published in the, you know, in

             12    the AEO was something like 2 million and it

             13    was really 6 million.  So this was supposed

             14    to be a, you know, a see how good you guys

             15    are?  Which, you know, certainly, you can

             16    always do that.

             17              But it turned out that the real

             18    thing to look at was how much total ethanol

             19    plus MTBE we thought would be used in 2006.

             20    Because there was a policy change, going back

             21    to another discussion that we had this

             22    morning, that kind of ruled out MTBE as an

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               142

              1    oxygenate, which was sort of economically

              2    preferred but politically and environmentally

              3    perhaps not economically preferred.

              4              And in the published book, you

              5    couldn't go back and see that.  And we had to

              6    go back and look at into one of these, you

              7    know, table 2085 which actually has this kind

              8    of detail.  And we pulled it up and it was

              9    like, you know, five and a half, you know,

             10    million -- a billion gallons or something of

             11    ethanol plus MTBE.  It was very close to the

             12    actual number that was used.  Again, not

             13    always the case that you are very close.

             14              We kind of pointed out that, well,

             15    you know, we were petty close on -- you know,

             16    you focused on this ethanol thing, but there

             17    was a policy change and in fact, if you look

             18    at oxygenate, you know, it was pretty much,

             19    again, following the theme of this morning,

             20    prices may be really bad but the quantities

             21    tend to be pretty good.

             22              And so like, this line of argument

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               143

              1    within the executive branch kind of, you

              2    know, imploded with that observation.

              3              But, you know, it did require going

              4    back to look at what we did ten years ago,

              5    and it was -- I don't know if we would

              6    actually been able to run it, but it was

              7    fortunate that we were able to pull up not

              8    only what is published but you can archive

              9    more than what you put in the book.  You got

             10    to archive everything, you know.

             11              But you are right.  If we had to do

             12    a side case with a ten year ago model, I

             13    don't know how difficult that would be.  In

             14    fact, that would be very hard.  But this is a

             15    real issue that came up in a real policy

             16    discussion.  And it was pretty important.  So

             17    it's a hard --

             18              MR. BLAIR:  John, do you have a

             19    comment?

             20              MR. WEYANT:  Let's see, a couple of

             21    comments.  First, I think it is a great

             22    project.  It sounds like you started with

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               144

              1    kind of model validation in mind, and then

              2    along the way, model verification,

              3    assessment, different types of debugging

              4    insights all came in, and actually at the

              5    end, you are talking like there might be even

              6    a different way to do modeling.  That this is

              7    kind of the simple model, maybe using a

              8    simple model calibrated into the big model.

              9              One of my pet themes, again, moving

             10    on the EIA stuff like everybody else is how

             11    to do uncertainty analysis.  I'm sure that is

             12    the last thing you want to hear because then

             13    it is even more important to come to grips

             14    with what the issue is.  But you did use this

             15    kind of information as a kind of a

             16    springboard.  You know, the first thing you

             17    would want to do is a sensitivity analysis, I

             18    think.

             19              There always is this fear that EIA

             20    and everybody else who does baseline

             21    workouts, it's tuned, and either if they know

             22    or don't know that if you change some key

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               145

              1    driver a little bit, the whole world changes

              2    and all your investments that looked good in

              3    the one world don't look good in that world.

              4    It is usually paranoia but it sounds like

              5    you've got this brain mapping capability

              6    here.  So I think this is a great project in

              7    many dimensions.

              8              On the validation, per se, there is

              9    a guy at Berkley named Max Auffhammer, I

             10    don't know if you have seen his stuff.  He

             11    has been trying to do EIA -- I wonder anybody

             12    here knows his work?  It's I think funded by

             13    the UC Energy Center.  But this is kind of

             14    from the outside, just doing basic, how well

             15    have the forecasts done.  And I think maybe

             16    some playing it off against some SIPO models

             17    and so on.  So I think that it would be good

             18    to -- in fact, I will send you all the --

             19    probably can send you PDFs and I know some of

             20    the people in his group.

             21              One issue I think that's important

             22    for the EIA is what to do with this stuff.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               146

              1    So far, I think, we have been talking like

              2    it's all internal.  It's kind of another

              3    internal review piece.

              4              But I just wonder, you know,

              5    worrying, and I go way back worrying about,

              6    you know, good modeling practice.  If some of

              7    this stuff should actually be published

              8    alongside of the AEO or as a separate

              9    document, I would find it even from the point

             10    of view of a journal editor, kind of a great

             11    thing to have in a journal.

             12              Even what you have already done,

             13    let alone some of the things you are going to

             14    do in the next couple of days.

             15              Again, it probably helps,

             16    selectively maybe, with model credibility

             17    that we have done.  We have done simple

             18    sensitivities and even some not so simple

             19    sensitivities, and strategically sorted

             20    through, just as George has presented.  So

             21    that for me is kind of an issue of how much

             22    of that.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               147

              1              As a person that worries a lot

              2    about model validation and usefulness and so

              3    on, it seems like this would have been a

              4    great thing.  It actually would be great to

              5    force every ———————— team to do something

              6    like this.

              7              MR. BLAIR:  May I ask a question,

              8    John?

              9              MR. WEYANT:  Sure.

             10              MR. BLAIR:  That is, do I correctly

             11    understand you to be saying that the work

             12    that George is doing and in effect the range

             13    of solutions that he calculates, which are

             14    now available for EIA itself to go back and

             15    check and say, now that we can plug in the

             16    real values, we've actually got a solution

             17    kind of related to that boundary line model?

             18    That there be published in effect a document

             19    so that people like yourself --

             20              MR. WEYANT:  Yes.  That is what I

             21    was doing --

             22              MR. BLAIR:  -- with proper finances

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               148

              1    to do this --

              2              MR. WEYANT:  I said, well, I don't

              3    really like this scenario for my purposes.  I

              4    would like one that is kind of in between

              5    here and a little bit outside the range

              6    there.  This would be a much better way than

              7    the way that they actually do it.  Just at

              8    that level.  So but I think it also would

              9    have a credibility validation benefit.

             10              MR. LADY:  Well, I think it would

             11    be a credential for EIA, because there is

             12    some recently somewhat critical articles on

             13    the projections which I think are unfair.  I

             14    mean, how can you be held accountable for the

             15    weather and all of these sorts of things?

             16              MR. WEYANT:  Yeah, I hear these all

             17    the time.  So that is part of my concern.

             18              MR. LADY:  And I'm sure that the --

             19    at least as far as I can --

             20              MR. WEYANT:  I think most of them

             21    are unfair.

             22              MR. LADY:  As far as I can tell,

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               149

              1    when you are seeing how the conditional

              2    variables, had they been correctly projected,

              3    affect the outcome, NEMS looks good.

              4              MR. BLAIR:  One thing that strikes

              5    me is that if I think of myself as a

              6    potential commercial user, if these -- can I

              7    call them solutions? -- if these solutions

              8    were available, I'd start making my own bets

              9    on the, you know, on the input variables and

             10    say, wow, you know, I trust his modeling.  I

             11    just don't like his starting point, or I just

             12    don't like EIA's starting point.  I'm going

             13    to play it out a little bit differently here.

             14              MR. WEYANT:  One thing I just

             15    though of that I meant to mention before and

             16    I know in the old days there was a lot of

             17    pressure on the key drivers and we've already

             18    all decided economic drivers are critically

             19    important there.  Is there still an

             20    influence?  I mean, back -- let me not even

             21    give you how many years ago was the old days,

             22    but it used to be that the CEA gave you

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               150

              1    optimistic GDP projections, which probably is

              2    to be expected.  This is before Howard was

              3    at --

              4              MR. GRUENSPECHT:  I don't know if I

              5    was involved in that subject or not.

              6              MR. WEYANT:  But then, as I

              7    understood it, EIA was forced by political

              8    factors to include those in here.  They said

              9    -- I understand that that case is much, much

             10    better now.  Is that true?

             11              MR. GRUENSPECHT:  We rely -- I

             12    don't know the whole, because I have only

             13    been here about four years, but my -- I don't

             14    think we were really ever forced to you know,

             15    to adopt the Administration's macro outlook.

             16              I think my sense of the macro

             17    outlook, more as a user than before I came

             18    here, was that we actually carried a macro

             19    outlook for a long time that until the mid

             20    '90s, kind of we had for a productivity

             21    growth and labor force growth, kind of as

             22    drivers, and I think, you know, GDP growth

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               151

              1    was a little long-term, I think headed down

              2    to like the two percent area.  And then we

              3    had the very strong economic scenario in the

              4    late '90s and really, you know, most of this

              5    recent, I guess, half of this decade as well.

              6    And then I think -- so I don't know what you

              7    would have changed.  Paul would maybe know

              8    better about 1998, you know, 1999 or 2000 --

              9              MR. WEYANT:  You see, that is

             10    predating my time.

             11              MR. GRUENSPECHT:  Also, too.  Well,

             12    so would us.  Tony maybe can do macro but

             13    whatever, I mean, we had a -- you know, the

             14    economic growth, as I'm saying it was a

             15    period of time when the economic, you know,

             16    long term average over, annual average over

             17    25 years was like 2.1 or 2.2 percent.

             18              And then after some period of time

             19    in the late '90s, I think it got up above or

             20    at 3 percent.  You know, average annual

             21    growth.  And now, I think last year, that

             22    dropped a little bit.  I think this coming

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               152

              1    year, it is going to drop some more.  And

              2    that those have big effects.

              3              MR. WEYANT:  Well, nobody remembers

              4    this as -- this -- that this was a problem.

              5    It is probably not the problem now.

              6              MR. LADY:  This issue is why there

              7    became an EIA, in part.  Prior to that,

              8    unfortunately, I'm so old that nothing

              9    happened before me.

             10              MR. WEYANT:  But in the early

             11    years --

             12              MR. LADY:  But they used to --

             13              MR. WEYANT:  I bet I could document

             14    it.  In the early days there was this funny

             15    thing --

             16              MR. LADY:  Before EIA, the

             17    administration could talk about goals and you

             18    couldn't project the macro forecasts that was

             19    inconsistent with the goals.

             20              MR. WEYANT:  Yeah, but I think even

             21    in the early days of EIA there was this, I

             22    don't know --

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               153

              1              MR. GRUENSPECHT:  There was one EIA

              2    administrator, I think, who was dismissed

              3    over something.

              4              MR. WEYANT:  I don't want to get in

              5    to that.  But the clever thing was, this went

              6    on for a couple of years and people thought

              7    they were really on top of us. What they put

              8    into the AEI they completely offset.  So they

              9    made an assumption about energy exogenous,

             10    energy used per unit of GDP, that brought it

             11    down to that more normal forecast.

             12              I think the argument that was used,

             13    it wasn't so much just pure political

             14    pressure.  It was kind of the not-my-job

             15    thing.  That you guys don't do -- your main

             16    function isn't macro so you should listen to

             17    macro guys.  And actually I do think that you

             18    guys are both right.

             19              That in talking this through with

             20    Ron earlier, what actually happened is

             21    people, other legitimate unbiased assessments

             22    like BEA and groups that do independent

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               154

              1    economic workouts, were irritated.  When that

              2    actually ended the nonsense pretty quickly --

              3    I think that is right.  And I think it was in

              4    part EIA's rule to allowed that to -- to --

              5    that problem to not come up.  But once it

              6    came up, they did get it resolved and did

              7    find --

              8              Is there anything else like that,

              9    that you feel at EIA that we are forced to

             10    take, you know, reserve estimates or -- these

             11    are things that come up in our studies over

             12    time.  That you really got economic wrong

             13    because you believe the CEA or actually some

             14    international -- we did a lot of

             15    international studies so we have this kind of

             16    orderly correct economic outlook stuff for

             17    oil reserve numbers, you have to use the

             18    USGS.  It will have to have the -- were there

             19    any things like that that people feel -- ?

             20              MR. GRUENSPECHT:  Probably not.  I

             21    think George is right that this has been a

             22    kind of the ethic of EIA.  Maybe in the early

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               155

              1    days, when there weren't experts in those

              2    areas, these things got, it really was not my

              3    job, but then it turns out, the reason I

              4    thought of this is that this turns out to be

              5    the most important thing.  And it's a little

              6    bit worse than weather.  It is kind of oil

              7    reserves or economic -- you know things that

              8    are close enough.  But it what seems to have

              9    happened with EAI is that they have to

             10    recognize that when you have people who are

             11    either become experts in those areas or they

             12    go out and find out who the true unbiased --

             13              SPEAKER:  Correct me if I'm

             14    wrong --

             15                   (Multiple speakers off-mike)

             16              SPEAKER:  -- where for example,

             17    most of a large quantity is supplied on line

             18    it is not supported by resource-based number

             19    that is in there.  And we know that there is

             20    going to be a multiyear delay in getting the

             21    number and we are on a schedule, doing this

             22    once a year, there is a process for convening

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               156

              1    an expert panel or expert opinion.

              2              SPEAKER:  Excuse me, just step to

              3    the microphone so that it can be recorded

              4    properly.

              5              MR. BLAIR:  Let me -- yeah, let me

              6    suggest in the interest of time that it be

              7    repeated that we are just having an

              8    elaboration around this idea that the inputs

              9    into the model, if you get scenarios which

             10    show variations in model outcomes and those

             11    are published, those could be enormously

             12    useful to some people and be useful all by

             13    themselves because you might just think it

             14    depends on an elaboration on that.  Any other

             15    comments?

             16              Thank you very much, George.  I

             17    think everybody will agree that it was very

             18    interesting stuff, right?  The -- oh, I'm

             19    sorry.  Feel free.

             20              SPEAKER:  Oh, so you didn't know

             21    about that.

             22              MR. GRUENSPECHT:  You didn't know

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               157

              1    about it.  No one told him.

              2                   (Laughter)

              3              MR. GRUENSPECHT:  Uh oh.  Well,

              4    something for you.  As those of you who were

              5    at dinner last night know, Bill Weinig, who's

              6    played a large role --

              7              MR. WEINIG:  I think I should get

              8    out of here.

              9              MR. GRUENSPECHT:  -- you know, in

             10    the work of this Committee, is leaving EIA

             11    and this is a Department of Energy

             12    Exceptional Service Award.  It says, "for his

             13    extraordinary stewardship of the American

             14    Statistical Association's Committee on Energy

             15    Statistics.  To run its work with the

             16    Committee has substantially enhanced the

             17    quality and usefulness of EIA's statistical

             18    and analytical products."  And it is signed

             19    by Guy F. Caruso, Administrator, EIA.

             20              Bill?

             21                   (Applause)

             22              MR. WEINIG:  Done?  Thank you.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               158

              1              MR. GRUENSPECHT:  Anyway we thought

              2    we would do it here because --

              3              MR. WEINIG:  Excellent.

              4              MR. GRUENSPECHT:  And I think --

              5    you know, honestly, but you didn't know.

              6                   (Laughter)

              7              MR. BLAIR:  Well, we have --

              8              MR. GRUENSPECHT:  It is your

              9    meeting, not our meeting.  So better to get

             10    up and do something like that.  It wasn't on

             11    the agenda but it was -- it was just hot off

             12    the -- the plate is still warm.

             13              MR. BLAIR:  We have two remaining

             14    agenda items.  First is the invitation to

             15    public comments, so if any member of the

             16    public would care to step forward and comment

             17    at this time?

             18              MS. KHANNA:  Can I do that?  I'm

             19    soon to be public --

             20              MR. BLAIR:  Absolutely.

             21              MS. KHANNA:  I just wanted to say

             22    that this is my last meeting and I just

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               159

              1    wanted to thank everyone here for a really

              2    good time.  I hope I contributed more than I

              3    learnt.

              4              I have to say I learnt a lot by

              5    being on this Committee.  And I know there

              6    are a lot of people who I never see but

              7    actually keep this up and running.  So I

              8    would like just to acknowledge them and thank

              9    them.  And also all the other people who come

             10    from various departments.  It is actually

             11    really nice to meet folks at the EIA who have

             12    been just names on the Internet for me,

             13    sometimes.  So thank you all very much.

             14              MS. KIRKENDALL:  Thank you.

             15              MR. GRUENSPECHT:  Thank you.

             16              MR. RADICH:  I'm Tony Radich from

             17    OAIF.  And I've got a question for George,

             18    that I think was brought up the last time,

             19    but I don't know that we ever got an answer

             20    to.

             21              NEMS is a deterministic model.  Yet

             22    you are running regressions on it and

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               160

              1    performing statistical hypothesis tests.

              2    Where is the unknown component coming in

              3    here?

              4              MR. LADY:  Well, in my jargon what

              5    this is amounting to is so-called comparative

              6    statics of the individual components.  So I'm

              7    trying to pick out, in the end for both the

              8    supply and demand modules there, the marginal

              9    impacts of important inputs.  The big

             10    integrating feature in NEMS is prices which I

             11    am treating as exogenous rather than

             12    indogenious, since I'm dealing with the

             13    structural, rather than the reduced form,

             14    sort of concept.

             15              So when you look at the projections

             16    subject to the limits of first order linear

             17    approximation, you can determine the degree

             18    to which the weather and the prices and floor

             19    space and things of that sort, had they been

             20    assumed correctly when the forecast was

             21    prepared, would have changed the projection

             22    that was made.  I'm sorry --

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               161

              1              MR. RADICH:  That's simply a use of

              2    the -- you're essentially estimating a

              3    reduced form, a very reduced form of NEMS.

              4    And so that has to do with the coefficient

              5    estimates.  But I'm concerned about how you

              6    are testing your coefficient estimates.  You

              7    have to have -- I agree that you have to have

              8    some kind of test of marginal effects but why

              9    assume that it is following a T distribution?

             10    There's a -- it is standard assumption when

             11    you have a component that you acknowledge is

             12    unknown but what are you saying we don't know

             13    here about NEMS?  I mean, people built this

             14    entire thing.

             15              MR. LADY:  Well, price elasticity

             16    of demand for electricity in the commercial

             17    sector is implicit to the way that module

             18    processes a lot of data.  But as it works out

             19    that particular influence, say, is reasonably

             20    linear.  And so if you generate a proper

             21    collection of solutions where you've

             22    disturbed that input, then the linear

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               162

              1    regression gives you an approximation of how

              2    NEMS functions with respect to that

              3    influence, which can then be later used to

              4    see had you gotten the electricity price

              5    correct, which I don't think is a fear of

              6    requirement for NEMS, you would have had your

              7    forecast be closer.  So I'm not -- in other

              8    words, what you are doing is approximating

              9    from NEMS solutions, the underlying structure

             10    of the model.

             11              MR. RADICH:  Yes.  I see that part.

             12              MR. LADY:  What is it that you are

             13    not?  I don't understand.  I'm not against

             14    you, I just don't understand what your issue

             15    that you are drawing is.

             16              MR. RADICH:  I don't know where

             17    there is a random component is so I don't

             18    know that a statistical hypothesis test is

             19    valid.

             20              MR. LADY:  It is a matter of fit.

             21    NEMS is deterministic.  And as I understand

             22    it, from the Committee, that something on the

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               163

              1    order of a Latin Hypercube design is a proper

              2    way to disturb inputs and then fit something

              3    that approximates what's going on in the

              4    models.

              5              So that is what I did.  It seems to

              6    work very well.

              7              MS. KIRKENDALL:  The randomization

              8    comes from the design of the input variables,

              9    how you change the input variables.  And that

             10    helps to make sure that your output variables

             11    will have some random distribution around two

             12    fixed values --

             13              MR. LADY:  Before you --

             14              Mr. WATSON:  If I understand the

             15    question, the sense is to have something like

             16    maybe if you have 50 states, and from that 50

             17    states you are trying to predict.  So you

             18    have to have the full sample.  There is no

             19    way ever that you estimate it all.  But then

             20    you compute a T test.  Well, in some you

             21    shouldn't do that because there is no

             22    uncertainty.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               164

              1              MR. RADICH:  Yeah.  That's -- if

              2    you are --

              3              Mr. WATSON:  That is what we are --

              4              MR. RADICH:  -- if you're just

              5    looking at a set of NEMS results but I

              6    think --

              7              MR. WATSON:  Oh, okay.  I'll have

              8    to look it up.  I understand your question.

              9              MR. RADICH:  Well, Nancy's comment

             10    indicates that your -- before you do your

             11    regressions, you're actually generating --

             12    I'm generating a bunch of outputs based on --

             13    on -- no, wait a minute.  You'd have to have

             14    done the regression first to generate those

             15    outputs, don't you?  Or are you -- are you

             16    actually --

             17              MR. LADY:  No, I looked at NEMS

             18    documentation and, not to my great surprise,

             19    discovered that the commercial models, say,

             20    receives from NEMS the generation, prices and

             21    macro economic variables.  And I could run

             22    the commercial module myself.  And I

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               165

              1    determined that the cost-price elasticities

              2    were reasonably small with respect to, say,

              3    electricity consumption as projected by that

              4    module.  And I found that the electricity

              5    price and floor space and allowed indogenious

              6    variable to pick up all of the infrastructure

              7    technology that that module processes, would

              8    give you good approximations of what the

              9    model would project.

             10              And so I ran nine solutions where I

             11    disturbed price and floor space and all

             12    possible combinations up and down, pooled

             13    that data, and then fit the regression to it.

             14    And my claim is that if you ran NEMS with the

             15    changes in those variables that you specify,

             16    that that regression equation would give you

             17    a good approximation of what the commercial

             18    module would tell you about electricity

             19    consumption.

             20              MR. RADICH:  Okay.  I think that

             21    that answers my question.

             22              MR. BLAIR:  The regression is

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               166

              1    actually affecting that approximation?

              2              MR. LADY:  Yeah.  So later, if you

              3    compare the projection and you discover that

              4    it was high, but you realized that

              5    electricity prices were much higher in fact

              6    than was projected, then you use the

              7    regression to see, had you projected the

              8    higher prices, how that would have influenced

              9    consumption, presumably you will see that

             10    actually NEMS was pretty good.  But the

             11    conditional variable was mis-forecast, not

             12    the indogenious variable.

             13              MR. RADICH:  Thank you.

             14              Mr. WATSON:  The first order simply

             15    comes up because the assumption that you have

             16    a different population in a particular

             17    sample.  And from the sample, you can make

             18    —————————————————————————————— but here,

             19    there is not -- you have the entire

             20    population, it is -- sometimes it is

             21    incorrect because ————————————————————

             22    whatever this 15-years behind population.  So

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               167

              1    it is -- in some sense it is a ——————————————

              2    question for me.  —————————————————————— you

              3    shouldn't have to go to ————————————————————.

              4    There are people -- I have colleagues who say

              5    that the same thing should be done for

              6    ————————————————————————.

              7              MS. KIRKENDALL:  Well, I think he

              8    said what I was going to say, so that is

              9    fine.

             10              MR. BLAIR:  Thank you very much.

             11    Our last agenda item is to discuss possible

             12    topics for the next, 2008 meeting.  And if

             13    anybody wants to bring anything up, certainly

             14    there is the possibility of e-mailing ideas

             15    to Nagaraj.

             16              MR. WEYANT:  Oh, the one -- I

             17    haven't seen this previously, and I don't

             18    know if this had been addressed, but just,

             19    this tricky thing is, how are you dealing

             20    with archives?  And in either a scenario

             21    variable or how -- whatever you are doing,

             22    however you're doing it.  I mean, that just

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               168

              1    seems to be the most rapidly evolving thing.

              2    I didn't know when I left ————————————————

              3    project, but I don't know what your schedule

              4    is.

              5              MS. KIRKENDALL:  I think George is

              6    planning to come to most of the next SMG

              7    meetings to keep you appraised he is up to.

              8              MR. GRUENSPECHT:  Yeah, one thing

              9    in terms of oil, there is actually a white

             10    paper, a service report without a client but

             11    we think it is motivated by GAO and by

             12    General Accounting Office's report on deep

             13    oil and a recent National Petroleum Council

             14    study and we are in fact hopefully doing a

             15    paper that actually tries to discuss those

             16    issues in, like, a useful way.

             17              It is actually focused really

             18    beyond the horizon of NEMS but it's going to

             19    try to see if the way we are looking at it

             20    will, you know, make sense.  And that might

             21    be an extra topic for this group.  Actually,

             22    I mean, that --

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               169

              1              MR. WEYANT:  The other one that is

              2    kind of like that is climate change.

              3              MR. GRUENSPECHT:  Right.

              4              MR. WEYANT:  Are you getting enough

              5    service for your request and are you

              6    interested in --

              7              MR. GRUENSPECHT:  We're getting

              8    more than we want there.

              9              MR. WEYANT:  Are you interested in

             10    advice on that now?  It's just beginning, so

             11    it will be interesting.  I mean, a real big

             12    challenge for -- you know, this is NEMS -- is

             13    how this had come up at a couple of meetings,

             14    but again, I don't know ———————— —————— the

             15    meeting is, I think, with the very good

             16    reason that we don't, you know, the base

             17    cases don't assume policy change or forecast

             18    policy change.

             19              But the thing that I find

             20    challenging is we don't -- we seem to go so

             21    far in our model, our modeling approach, that

             22    the existence of a policy issue -- I'm not

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               170

              1    talking about any specific, you know, X will

              2    probably be implemented in the year Y -- but

              3    just, you know, if we had two parallel

              4    worlds, one in which if you walked up to

              5    people and said climate change, they had

              6    never heard of it as an issue, and another

              7    one where it is an issue, and you know,

              8    possibly, in those two worlds behavior is

              9    different but in NEMS those two worlds it is

             10    not different.

             11              So it isn't a question of not, you

             12    know, why do you assume that there will be

             13    likely legislations, but this is the reason I

             14    think for us not to do that.  But how we deal

             15    with something being out there is a very

             16    tricky and hard business for us.  And I think

             17    in one way we can get out while, you know,

             18    you are forecasting policy and we are not, to

             19    get at some of the differences, but some

             20    these differences on -- since I know that if

             21    we look at the world without climate change

             22    as an issue and the world with climate change

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               171

              1    as an issue, at least I think the way we put

              2    together NEMS, we get the same type of

              3    baseline cases.  And that is a tricky

              4    business.

              5              MR. GRUENSPECHT:  The only thing I

              6    know -- well, actually a few things, but the

              7    one I know best is the NERG (?) stuff, where

              8    they have this single uncertainty single

              9    resolution of uncertainty kind of thing to

             10    look at policy and then they say, well in the

             11    real world, factors in the economic system

             12    are actually already hedging against climate

             13    policy.  What are -- what is --

             14              MR. WEYANT:  We don't know how we

             15    represent that.

             16              MR. GRUENSPECHT:  What is a

             17    reasonable assumption about what probability

             18    they assign to what target and they could do

             19    it backwards.  Of course, that is a very

             20    simple model with expectations and

             21    uncertainty.  But that is -- I think that is

             22    a hard problem for everyone.

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               172

              1              This is actually one of the big

              2    debates on the ICCC scenarios.  What do you

              3    mean by based on this scenario, given that

              4    some countries have explicit climate policy,

              5    some are in progress, lot of it is being

              6    implemented, some aren't.  It is actually

              7    felt that that is a huge source of

              8    uncertainly and what people call baselines,

              9    because they make different assumptions.

             10    Sometimes they know they are making different

             11    assumptions.  Sometimes they actually don't.

             12              MR. WEYANT:  Maybe that is an

             13    interesting issue.  Maybe not to go to

             14    something more elaborate than that.  But just

             15    have somebody talk about that issue.

             16              MR. BLAIR:  Any comments on that,

             17    anybody?  Any ideas?

             18              MR. KOKKELENBERG:  One thought was

             19    that we've given some thoughts to the design

             20    of the webpage.  And the key in economic

             21    indicators.  And I'd scratch the progress

             22    report on how it goes to elements are coming

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               173

              1    along.  I'd love to hear if we found where

              2    their coal is going.

              3              MR. GRUENSPECHT:  That may be

              4    something that is resource limitation.  Or

              5    limiting investigation.  But the question is

              6    how crucial is that amount of rollover

              7    missing.

              8              MR. KOKKELENBERG:  Right.

              9              MR. GRUENSPECHT:  You raised that

             10    half of one percent.

             11              MR. KOKKELENBERG:  Yeah, is that --

             12    is that a key?  Is the coal market that

             13    sensitive?  Then half of one percent in some

             14    things is an extremely important thing

             15    because the way the market is traded and

             16    swings prices. But it may not be in coal.

             17              MR. BLAIR:  Any more thoughts?  In

             18    that case, other ideas can go to Nagaraj.  We

             19    will stand adjourned.

             20                   (Whereupon, the MEETING was

             21                   adjourned at 11:55 a.m.)

             22

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382

                                                               174

              1

              2

              3

              4

              5                       *  *  *  *  *

              6

              7

              8

              9

             10

             11

             12

             13

             14

             15

             16

             17

             18

             19

             20

             21

             22

                                   BETA COURT REPORTING

                                   www.betareporting.com

                      202-464-2400                       800-522-2382