UNITED STATES DEPARTMENT OF ENERGY















                  MEETING OF THE AMERICAN STATISTICAL ASSOCIATION

                   (ASA) COMMITTEE ON ENERGY STATISTICS WITH THE

                      ENERGY INFORMATION ADMINISTRATION (EIA)













                                  Washington, D.C.

                             Thursday, October 28, 2004





                                                              2


           1    PARTICIPANTS:

           2       F. JAY BREIDT

           3       NICOLAS HENGARTNER

           4       JOHNNY BLAIR

           5       MARK BURTON

           6       MOSHE FEDER

           7       BARBARA FORSYTH

           8       NEHA KHANA

           9       NAGARAJ K. NEERCHAL

          10       SUSAN M. SEREIKA

          11       RANDY R. SITTER

          12       HOWARD BRADSHER-FREDRICK

          13       ROBERT RUTCHIK

          14       NANCY KIRKENDALL

          15       PRESTON McDOWNEY

          16       GUY CARUSO

          17       TOM BROENE

          18       HENRY S. BROOKS

          19       BRENDA COX

          20       GRACE SUTHERLAND

          21       SHAWNA WAUGH

          22




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                              3


           1    PARTICIPANTS (CONT'D):

           2       TOM LORENZ

           3       PHILLIP TSENG

           4       JOHN WOOD

           5       HOWARD GRUENSPECHT

           6       WILLIAM WEINIG

           7       INDUJIT KUNDRA

           8       JOE SEDRANSK

           9       KAREN NORMAN

          10

          11

          12                       *  *  *  *  *

          13

          14

          15

          16

          17

          18

          19

          20

          21

          22




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                              4


           1                      C O N T E N T S

           2    AGENDA:                                    PAGE

           3       Greetings, Presentations and               5
                   Remarks
           4
                   Updates for the Committee                 34
           5
                   Proposal for Analysis of Collected        62
           6       Data

           7       A Customer Evaluation of the Short        83
                   Term Energy Outlook (STEO)
           8
                   Frames: Progress on and Lessons          126
           9       Learned from Frames Evaluations

          10       The EIA Short Term Regional Electricity  134
                   Model: Capabilities and Data
          11       Requirements

          12       Natural Gas Production, Frames,          219
                   Samples and Estimation
          13
                   Methods for Assessing NEMS Solution      227
          14       Data for Interpretive and Diagnostic
                   Purposes
          15
                   Introduction to Program Assessment       238
          16       Rating Tool (PART) Program Evaluation

          17       External Evaluations of Survey Programs  245

          18       External Evaluations of Forecasting      306
                   and Models
          19
                   How the ASA Energy Committee Might Help  317
          20       in Program Evaluation under PART?

          21

          22                       *  *  *  *  *




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                              5


           1                   P R O C E E D I N  G S

           2                                             (8:30 a.m.)

           3              CHAIRMAN BREIDT:  Okay, I think

           4    we're ready to begin.  Welcome to the fall

           5    meeting of the American Statistical

           6    Association Committee on energy statistics

           7    with the Energy Information Administration.

           8    This meeting is being held under the

           9    provision of the Federal Advisory Committee

          10    Act.  This is an ASA not an EIA committee,

          11    which periodically provides advice to EIA.

          12    The meeting is open to the public and public

          13    comments are welcome.  Time will be set aside

          14    for comments at the end of each morning and

          15    the afternoon sessions.  Written comments are

          16    welcome and may be sent to either ASA or EIA.

          17              All attendees including guests and

          18    EIA employees should sign the register in the

          19    hall and should include their e-mail

          20    addresses.  Restrooms are at the end of the

          21    hall towards the back of this room, back that

          22    way, a fountain is in the same corridor on




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                              6


           1    the way, telephones in this room share a

           2    single number.  It is, if you're interested

           3    202-586-3071.

           4              Kathleen Wert with ASA meetings

           5    department is here.  She was out at the front

           6    desk and is available for committee member

           7    questions on expense reimbursements.  Tara

           8    Stull was the former committee meetings

           9    liaison, and you may know that she has moved

          10    to a new job in Boston where she'll be

          11    working for the Boston Museum of Natural

          12    History.  And if you were up late last night

          13    you may have seen pictures of her revelling.

          14    So in commenting each participant is asked to

          15    speak toward a microphone, we have a

          16    transcriber to appreciate that and committee

          17    members and speakers at the head table need

          18    to speak clearly and into a microphone, Bill

          19    always writes that for me, "clearly and into

          20    a microphone".

          21              These microphones are reasonably

          22    sensitive so you may not need to really lean




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                              7


           1    in, they should be able to pick it up.  And

           2    speakers are asked to use a  microphone,

           3    which is available here, and Bill can help

           4    out with that.  So now I'd like each of us to

           5    introduce ourselves.  So, we'll begin with

           6    the committee members and then we'll go out

           7    into the audience.  Please speak into a

           8    microphone and give your name and your

           9    affiliation.  So my name is Jay Breidt, I am

          10    with Colorado State University Department of

          11    Physics and can we start here.

          12              MR. McDOWNEY:  Preston Mc Downey,

          13    EIR, SMG.

          14              DR. NEERCHAL:  Nagaraj Neerchal,

          15    UMBC, Math department.

          16              MS. KHANNA:  Neha Khanna, State

          17    University of New York, Birghamton, Economics

          18    and Environmental Studies.

          19              MS. Forsyth:  Burhara Forsyth from

          20    Westate.

          21              DR. FEDER:  Meshe Feder, Research

          22    Triangle Institute, Research and statistics




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                              8


           1    division.

           2              DR. BURTON:  Mark Burton,

           3    University of Tennessee, Knoxville center for

           4    Transportation Research.

           5              DR. HENGARTNER:  Nick Hengartner,

           6    Los Alamo, National Laboratory Statistical

           7    Science Group.

           8              MR. CARUSO:  Guy Caruso,

           9    Administrator, Energy Information

          10    Administration.

          11              MS. KIRKENDALL:  Nancy Kirkendall,

          12    Statistics and Methods Group.

          13              CHAIRMAN BREIDT:  Okay we'll go

          14    back to the audience and can we use the

          15    microphone please?  Yeah I'm pointing at you

          16    -- yeah or maybe Bill, you want to do this.

          17              MR. WEINIG:  This'll be fine.

          18              CHAIRMAN BREIDT:  Ok I'm sorry.

          19              MR. WEINIG:  One last member of the

          20    EIA, I'm Bill Weinig with the Energy

          21    Information Administration.

          22              MR. TSENG:  I'm Phillip Tseng, EIA.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                              9


           1              MS. HOJJATI:  Behjat Hojjati,

           2    Energy Information Administration.

           3              MS. WAUGH:  Shawna Waugh,

           4    Statistics and Methods Group.

           5              MR. HOUGH:  Richard Hough, US

           6    Census Bureau.

           7              MS. HAITOT:  Vicki Haitot, US

           8    Census Bureau.

           9              MS. SNAPP:  Eva Snapp, US Census

          10    Bureau.

          11              MS. LOFTIN:  Lacey Loftin, US

          12    Census Bureau.

          13              MR. BOURNAZIAN:  Jake Bournazian,

          14    EIA.

          15              MR. HSEN:  Paul Hsen, Census

          16    Bureau.

          17              MR. LORENZ:  Tom Lorenz, EIA.

          18              MS. BLESSING:  Colleen Blessing,

          19    EIA.

          20              MS. MILLER:  Renee Miller, EIA.

          21              MR. SEDRANSK:  Joe Sedransk, CWRU

          22    and EIA.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             10


           1              MR. RUTCHIK:  Bob Rutchik, EIA.

           2              MR. WOOD:  John Wood, EIA.

           3              MR. WEINIG:  Just pass the

           4    microphone over.

           5              MS. JENNINGS:  Alethia Jennings,

           6    EIA.

           7              MR. FREEDMAN:  Stan Freedman, EIA.

           8              MS. TAYLOR:  Yvonne Taylor, EIA.

           9              MR. JOOSTEN:  Jim Joosten, EIA.

          10              MR. BRADSHER-FREDRICK:  Howard

          11    Bradsher-Fredrick, EIA.

          12              MR. WOOD:  John Wood, EIA, Reserve

          13    and Production tables.

          14              MS. SPENCER:  Lynda Spencer, Kema

          15    Incorporated.

          16              MS. FRENCH:  Carol French, EIA.

          17              MR. ZHANG:  Bin Zhang, EIA.

          18              MR. MILLER:  Herb Miller, EIA.

          19              CHAIRMAN BREIDT:  Okay.  Bill is

          20    checking the halls for anyone else.  Okay,

          21    it's now time for my favorite announcement,

          22    for your information Nancy Kirkendall is the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             11


           1    designated Federal officer for the advisory

           2    committee.  In this capacity Dr.  Kirkendall

           3    may chair, but must attend each meeting and

           4    she is authorized to adjourn the meeting if

           5    she determines this to be in the public

           6    interest.  She must approve all meeting of

           7    the advisory committee and every agenda; also

           8    she may designate or substitute in her

           9    absence.

          10              Okay, so we have a very interesting

          11    agenda today, we have a number of breakout

          12    sessions.  This is one of the most technical

          13    agendas I think we've seen in a long time.

          14    So there's a lot of statistical and

          15    econometric meet here, so this should be a

          16    lot of fun.  There are a total of 10 breakout

          17    sessions and the first session this morning,

          18    which is not a break out session, plenary

          19    session.

          20              We'll begin the briefing with EIA

          21    Administrator Guy Caruso, on EIA's general

          22    state of affairs and then Nancy Kirkendall




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             12


           1    the director of EIAs Statistics and Methods

           2    Group will tell us about EIA work since the

           3    spring 2004 meeting and the advice we

           4    provided.

           5              Lunch for the committee, presenters

           6    and invited guests will be on the first floor

           7    at 12:15.  It will be in the usual room.  And

           8    this evening we have reservation for dinner

           9    at the Aroma Indian Restaurant 1919 "I"

          10    Street Northwest.  You can take the blue line

          11    or the orange line to get there.  It's pretty

          12    close to the hotel GWU Inn that we're staying

          13    at.  Our reservations are at 6, so can I see

          14    a show of hands for who will be attending the

          15    dinner.

          16              Okay, and tomorrow morning

          17    breakfast for the committee will be here

          18    again beginning at about 8 and we'll resume

          19    at 8:30 tomorrow in this room.  So I guess

          20    that's it.  Now it's my pleasure to recognize

          21    Guy Caruso, Administrator of the Energy

          22    Information Administration.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             13


           1              MR. CARUSO:  Thank you Jay and good

           2    morning everyone and special welcome to ASA

           3    committee members and I want to first start

           4    by telling you how much we appreciate the

           5    work that you have done and now my fifth

           6    meeting since I've been here and I think

           7    we've found this advice extremely valuable,

           8    and as Jay said today's agenda I think shows

           9    the kind of specific linkage to the work that

          10    we've been doing that I'll talk a little bit

          11    about and how important it can be and in the

          12    work of EIA and in the I think the accuracy

          13    and the timeliness of our information and the

          14    analysis.

          15              So thank you once again and I will

          16    say a little bit more about Jay specifically,

          17    at the end, because as you known this is his

          18    last session as the chair and a so.  Let me

          19    start by talking a little but about what's

          20    been going on at EIA since our last meeting

          21    in spring.  As you know every year EIA is

          22    asked to do winter fuels outlook and this




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             14


           1    year the attention on the outlook for this

           2    winter was particularly acute with rising oil

           3    and gas prices and it fairly is -- as we'll

           4    talk about the interest not only in market

           5    but on the hill and the public has been

           6    higher than ever.

           7              We're going to talk a bit about

           8    confidentiality.  I know this committee has

           9    had a number of briefings on by Jay and Nancy

          10    in the past and that's continuing to play out

          11    with respect to the statistical agencies and

          12    guidelines from OMB etcetera as well as more

          13    general data access that we talked about in

          14    this committee.

          15              Budgeting is always an issue, of

          16    course to be able to achieve the results we

          17    hope to and I'll bring you up to date on

          18    where we are and the fiscal year 2005 budget

          19    and we've reached the committee last meeting

          20    on our strategic plan which we came to

          21    agreement on as organization and individual

          22    offices since we last met and bring you up to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             15


           1    date a little bit on the progress towards

           2    that.

           3              Winter fuels as I mentioned you

           4    know, enormous interest not only because of

           5    the type of markets both in oil and gas but

           6    going into a political -- a presidential

           7    election there's even more interest in what

           8    we do than ever.  As we do our monthly

           9    forecasting and put out our weekly data these

          10    are all areas, which this committee has

          11    directly or indirectly had some influence

          12    over and will continue to.

          13              In today's meeting we're going to

          14    talk about some of the surveys that feed into

          15    this Short term energy outlook and

          16    specifically some of the results and the

          17    regional modeling of this short term energy

          18    outlook and the customer survey of the short

          19    term energy outlook which we'll be talking

          20    about in fact in the next session of the

          21    breakout session.

          22              That just shows in this slide the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             16


           1    specific sessions we'll be having today and

           2    tomorrow that feed directly into the

           3    methodology and results of our short term

           4    energy outlook and so once again a very

           5    topical very timely, very much appreciated

           6    the advice we've gotten from you on our

           7    survey's and on our frames and in particular

           8    on the -- for the modeling of the short term

           9    energy outlook which continues to be enhanced

          10    particularly on the regionalization aspects

          11    of that.  Dave Costello will be here to talk

          12    about that.  I think that's later on today or

          13    tomorrow.

          14              Another one which really started

          15    two years ago was our desire to more directly

          16    collect monthly natural gas production data

          17    and John Wood and his people in Dallas have

          18    been working with this committee for, in fact

          19    from the first meeting that I've been here --

          20    Five meetings now though and finally we have

          21    -- we hope some good news and that is that

          22    OMB has approved the natural gas production




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             17


           1    survey forum 914 in Iowa.

           2              There'll be a session later on

           3    today that John will be talking to you about

           4    it -- specifically about what we're doing on

           5    that survey and what we hope to achieve and I

           6    can assure you that the work that this

           7    committee has done has directly led to the

           8    development of that survey and we were hoping

           9    for more timely and more accurate natural gas

          10    production data, that's our goal and I know

          11    that the winters of '02-'03 and '03-'04 are

          12    when natural gas prices spike so much there

          13    was an increase in political concern over

          14    natural gas and this will be, we hope, an

          15    important response from EIA.

          16              On the budget side the congress

          17    went on recess without having passed the

          18    budget in our case, an interior budget, where

          19    our budget is an interior committee.

          20              So we're on a continuing resolution

          21    for fiscal year '05 at the same spending

          22    level of '04 which is always a challenge




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             18


           1    because we were expecting a 5% increase going

           2    from 81 million to 85 million this year.  So

           3    we're having to operate at a lower pace of

           4    spending than we would've liked and that

           5    directly affects some of the things including

           6    the natural gas survey that I just mentioned

           7    but, we will actually have a session led by

           8    Howard Gruenspecht tomorrow morning to talk

           9    about that and get your advice on operating

          10    in this kind of tight budget scenario.

          11              CIPSEA as you have been -- as this

          12    has been on the agenda a number of times

          13    here, continues to be in effect and we are

          14    now working with OMB on the specific

          15    guidelines for the agencies like our own,

          16    like EIA, who are not directly, mentioned in

          17    CIPSEA, but really come under it and we're

          18    working very hard to tighten up security of

          19    the data and we are doing things like

          20    training and other areas of -- to basically

          21    to improve the security of our data.  One of

          22    the areas we've talked about in this




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             19


           1    committee is granting access to researchers

           2    and we've started a inter- office team to

           3    look into this issue, Nancy Kirkendall is

           4    leading that and we've also have had meetings

           5    with a number of other statistical agencies

           6    and work with them on getting advice and

           7    ideas on how they do it and some of the

           8    agencies are represented here, that we work

           9    with -- on that.  We will continue to work --

          10    to come up with a way that we think can

          11    satisfy the needs of researchers but also

          12    meet the requirements under CIPSEA and other

          13    confidentiality pledges.

          14              The last meeting we talked a little

          15    bit about how sensitive some of the EIA data

          16    has become with respect to the financial

          17    institutions and markets in general and it's

          18    -- I think it's interest is heightened more

          19    since then because several financial

          20    instruments have been launched on NYMEX and

          21    other for which EIA data is directly being

          22    used for either derivatives or auctions and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             20


           1    therefore when our weekly data for either oil

           2    or natural gas comes out, Wednesday or

           3    Thursday morning, there's considerable amount

           4    of writing on it and therefore a lot more

           5    interest there than there's ever been, by the

           6    media, by the financial houses and the

           7    public.

           8              So this sort of feeds back to the

           9    issue of confidentiality and need to even

          10    have greater security of our data, which you

          11    know I think EIA has not been subject to the

          12    kind of scrutiny that other statistical

          13    agencies that produce the principal economic

          14    indicators such as Census BLA and BLS have

          15    dealt with this for decades whereas it's kind

          16    of new for EIA and we're, again working with

          17    all of those agencies to work on data

          18    safeguards and I think that this slide really

          19    speaks to that.

          20              Certainly we need to prevent the

          21    unauthorized release of data and we're

          22    finding that the scrutiny is when there are




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             21


           1    billions, or certainly hundreds of millions

           2    of dollars riding on what that number is each

           3    week.  We're finding consequences that is

           4    very, very close scrutiny by as I mentioned,

           5    financial institutions but also the media,

           6    which uses our data to inform their

           7    customers.

           8              One of the technical issues which

           9    isn't directly an issue for this committee is

          10    that our website is being utilized

          11    intensively during those few moments just

          12    when the data is to be released at 10:30 on

          13    Wednesdays and Thursday mornings and many of

          14    the consumers are using robots to -- the term

          15    of it is pinging our system, hundreds and

          16    thousands of times per second and that has

          17    created a bandwidth problem for us in terms

          18    of speed at which our customers can acquire

          19    the data and so we're working on that as well

          20    but that's really more of an IT issue but

          21    clearly one of the consequences of this

          22    heightened interest in EIAs data.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             22


           1              The other thing, Jay Edmonds isn't

           2    here, he's had a close interest in the

           3    greenhouse gas collection -- data collection

           4    issue, that is the guidelines for collecting

           5    emissions data, voluntary reporting of

           6    Greenhouse Gases has moved forward and is

           7    close to being finalized at OMB for final

           8    approval right now, but it will have a direct

           9    impact on EIA because we will be required to

          10    collect a great deal more of this greenhouse

          11    gas emission data than under the current

          12    system.  Talked about the strategic plan

          13    goals last time.

          14              These are the 3 main goals and

          15    there are a number of sub-elements under this

          16    and perhaps the quality of data is one that

          17    this committee has dealt mostly with and

          18    that's -- the next slide talks about some of

          19    the things that the -- this committee has

          20    done on the survey quality assessments and

          21    then again the issue of the quality of our

          22    analysis and the sufficiency of the frames




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             23


           1    which we'll talk about directly this morning.

           2              And the 3rd goal, which is, more of

           3    having to do with the motivation and the

           4    leadership quality of EIA, motivation of the

           5    staff and productivity.  We have -- there

           6    have been a couple of developments one, we

           7    did a survey, an organizational assessments

           8    survey here in EIA in May and June of this

           9    year and Howard and I are very pleased with

          10    the results which are shown in this slide,

          11    comparatively we've improved in terms of the

          12    staff's agreeing or strongly agreeing with

          13    the statement that the management provides

          14    quality and continues improvement by creating

          15    and modeling in the organization of vision

          16    values and quality.

          17              I think you know -- this is fairly

          18    very positive development and we know we can

          19    still do better and we're looking for some of

          20    the issues, that we're going to get on our

          21    agenda -- today -- again to lead to further

          22    improvements in those areas, in both of the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             24


           1    goals, one and goal three.  The other thing

           2    on goal one is we did a -- we participated in

           3    American Customer Satisfaction Index surveys.

           4    It's called ACSI on our website and for the

           5    first time I believe.  Is that accurate

           6    Colleen? is that the first time we've ever

           7    participated in ACSI?

           8              MS. BLESSING:  No we did it -- we

           9    participated in 2000.

          10              MR. CARUSO:  2000, okay.  But we

          11    hadn't done it for several years and we did

          12    it again this year.  And the ACSI score of

          13    72, which we thought was very very pleased.

          14    It was a bit higher than the Government

          15    average and -- but I think as important were

          16    some of the areas where we thought that we

          17    could use to improve our website, which is

          18    becoming increasingly used and in just the 2

          19    years or so that I've been here.

          20              And what we've done about improving

          21    the website is to appoint a web champion,

          22    John Weiner and John is taking that task on




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             25


           1    and we're certainly looking at ways on

           2    improving the navigation and other issues

           3    that the ACSI survey pointed out and

           4    certainly it will be coming to this committee

           5    in future to ask for further assistance and

           6    advice from you.

           7              So that's a very quick run through

           8    of some of the things that we've been doing

           9    here at EIA and the events that have been

          10    driving our agenda here in the last six

          11    months and looking forward to again getting

          12    the advice and guidance from this committee

          13    during the next day and half.  So thanks once

          14    again Jay, I'd be happy to answer any

          15    questions now or at any time during the next

          16    two days.

          17              CHAIRMAN BREIDT:  Okay, thank you.

          18              MR. CARUSO:  Are there any

          19    questions?

          20              CHAIRMAN BREIDT:  What is a web

          21    champion, is that like a drug czar?

          22              MR. CARUSO:  Yeah, we have a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             26


           1    committee within EIA, from each of the

           2    offices that looks on their specific products

           3    that goes on the web and in effect John is

           4    going to be the coordinator of that group.

           5    But it sort of, yeah it's kind of more of,

           6    we're trying to give it a little more, let's

           7    say imprimature, more like a czar you're

           8    right about that.  You know, if there aren't

           9    any questions let me take this moment this

          10    moment to more formally thank Jay Breidt who

          11    has been the chairman since last year -- how

          12    long has it been?

          13              CHAIRMAN BREIDT:  Two years.

          14              MR. CARUSO:  Two years.  That went

          15    by fast, and on the committee itself for six

          16    years.  I have a letter from Secretary

          17    Abraham, which I'll read and then Nancy has a

          18    certificate, which she'll present to you.

          19              "Dear Dr. Breidt, Thank you for

          20    your outstanding service as committee chair

          21    and as a member of the American Statistical

          22    Association Committee on energy statistics.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             27


           1    You have contributed substantially to the

           2    effectiveness of the committee during the

           3    past six years.  Your leadership in guiding

           4    the committee and you're continuing examples

           5    of vision, encouragement and sound judgment

           6    have kept the committee productive and on

           7    course.  Your insightful comments and

           8    technical discussions have helped the Energy

           9    Information Administration in many technical

          10    areas during the past 6 years.

          11              In particular we appreciate your

          12    advice on selecting estimation procedures

          13    using relative standard errors, editing and

          14    evaluating EIA survey testing and methods.

          15    We thank you for your vision and help, and

          16    hope that we may continue to call upon you

          17    from time to time for advice and assistance.

          18    Sincerely, Spencer Abraham, Secretary of

          19    Energy." Thanks Jay.

          20              MS. KIRKENDALL:  And before you

          21    disappear we also have a certificate and this

          22    is from Guy and myself thanking you for your




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             28


           1    work with the committee.  Oh I guess I'm

           2    supposed to present next.

           3              CHAIRMAN BREIDT:  So it's my

           4    pleasure to introduce Nancy Kirkendall.

           5              MS. KIRKENDALL:  Okay well Guy went

           6    through updates on a lot of the activities

           7    that we had in the last meeting and I'm going

           8    to sweep up with just a few additional ones.

           9    There are 4 topics that I'll talk about.

          10    Electric power, the revision policy and

          11    survey testing methods and then estimation of

          12    stock of other oils.  So on electric power we

          13    had a big session in the last meeting where

          14    Doug talked about transmission papers, he'll

          15    be here later, that paper is -- I think it's

          16    been approved by Howard, it's about ready to

          17    come out of the front office and they're

          18    trying to collect names of people who would

          19    like to have a hard copy.

          20              It is going to be posted on the web

          21    so you can always look at it there.  But if

          22    you're interested in a copy of the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             29


           1    publication let us know and we can make sure

           2    that you get one.

           3              We also talked about the focus

           4    groups that Howard Bradsher-Fredrick and Phil

           5    participated in, that also illuminated our

           6    data needs on transmission.  And the electric

           7    power package, they actually did use the

           8    information that we learned from the focus

           9    groups and transmission paper.  I think your

          10    advice was that we needed to do these things

          11    a little bit more quickly and coordinate them

          12    a little bit better and of course, that's

          13    good advice.

          14              But it did make an impact on the

          15    questionnaire and right now the electric

          16    power package, it received a number of

          17    questions from the industry, they're not sure

          18    they would like to report transmission

          19    outages to EIA.  And so I guess EIA is in the

          20    process of coming up with it's response back

          21    to OMB.  Of course that was one of the

          22    recommendations from Doug's paper.  I guess




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             30


           1    that was all on that one.

           2              All right revision policy -- we

           3    talked about revision policy that was Alethia

           4    Jennings and Renee Miller.  Just to remind

           5    you the goal of -- we have a standard that

           6    says that we shouldn't revise our data too

           7    much and I think that the committee wasn't

           8    sure that -- they thought that was

           9    particularly important.  But we presented our

          10    new policy on revising data that we had

          11    previously called final, trying to make it

          12    more accurate and the collaboration that goes

          13    on within EIA to agree on changes like that.

          14              And you saw the proposal was fine,

          15    it is being implemented now and there've been

          16    a number of changes that have been made that

          17    way.  Stan Freedman and Bob Rutchik talked

          18    about their survey testing methods.  We just

          19    went through what we think are good

          20    approaches on testing and we ask for your

          21    assessment.  Committee recommendations are on

          22    the next slide and we're basically trying to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             31


           1    follow your advice since then -- next slide

           2    -- Stan has presented this paper in Amsterdam

           3    and has gotten some good comments on it and

           4    approval of the methods and he's also been

           5    invited by Lynda Carlson who of course

           6    started this whole cognitive testing approach

           7    within EIA to conduct a workshop on our

           8    methods at her office and she also wants us

           9    to work with her going doing cognitive

          10    testing and some of their surveys.

          11              So there will be some cross-

          12    fertilization.  She also has some people who

          13    have done cognitive testing, so we can learn

          14    a little bit, they can learn a little bit,

          15    that and might be a good collaboration.

          16              We talk about -- the project was to

          17    come up with a better way of estimating

          18    weekly stocks of other oil.  Basically this

          19    is -- we have a weekly survey that gives us

          20    the inventories of the major products and

          21    other oils is sort of the rest, however, we

          22    also like to publish it total.  So we need an




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             32


           1    estimate for the other, so that we can get a

           2    good estimate in total.  There was a problem

           3    last year when stocks -- our estimate rather,

           4    oil was pretty bad and so this has been a

           5    project to try to do better.

           6              The major improvement has been

           7    caused by collecting propane data, so now

           8    what we're estimating is much smaller than it

           9    used to be and that was the thing that had

          10    the biggest change during that problem time

          11    period.  We're still looking at ways of doing

          12    a better job, estimating the residual -- next

          13    slide -- I think you're recommendation was

          14    that we needed to collaborate better because

          15    we have many efforts in EIA that look at

          16    estimating stocks of other oils including our

          17    short term energy outlook folks as well as

          18    the petroleum people.

          19              So we're still working on this, we

          20    have come up with a couple of our best

          21    regression based methods and now we're doing

          22    some comparisons and now we're working with




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             33


           1    the other program officers to see what we can

           2    come up with and that is basically what I

           3    have as an update.  Anybody have questions

           4    about anything?  Anything either Guy or I

           5    forgot to talk about that you were

           6    particularly remembering as an issue from

           7    last time.  Okay, thank you.

           8              CHAIRMAN BREIDT:  Thanks Nancy.

           9    Okay we're now ready to move to break out

          10    sessions and if the committee members take a

          11    look at the break-out sessions.  You might

          12    notice that if you are not Barb then you

          13    always go to the same place.  So if you're

          14    upstairs you always stay upstairs, if you go

          15    downstairs you always stay downstairs, except

          16    Barb who is the only exception --

          17              MS. FORSYTH:  I need to work out.

          18              CHAIRMAN BREIDT:  Okay.  Last time

          19    we had some complicated fractional factorial

          20    and people were -- complaining about that, so

          21    this is a little simpler.  So let's go ahead

          22    and reconvene in breakout.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             34


           1                   (Recess)

           2              MR. RUTCHIK:  Good morning, I'm Bob

           3    Rutchik.  I'm of Statistics and Methods Group

           4    of EIA.  I want to welcome you to the third

           5    breakout session.  In the past year and a

           6    half that we've had on EIA 920 combined heat

           7    and power plant reports.  The previous two

           8    sessions were in April of last year and that

           9    was on the EIA 920 pre-survey design visits.

          10              In last October a year ago we had

          11    another session on proposed respondent level

          12    and aggregate level edits for the EIA-920.

          13    The purpose of this session is to get your

          14    feedback, -- the committee's feedback on

          15    EIA's plan to evaluate the effectiveness of

          16    the form EIA-920 which has been collecting

          17    data since January of this year.  We picked

          18    the EIA-920 to apply this program to, that

          19    I'll be discussing, because we at EIA have

          20    extensive experience in the survey's design

          21    and testing.  We are very familiar with it

          22    and very comfortable with it.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             35


           1              This is the plan on which I'm going

           2    to be asking your advice -- for which I'm

           3    going to be asking the committees advice.

           4              As you can see there are four

           5    criteria, which we have proposed to use to

           6    evaluate the EIA- 920.  These criteria are

           7    timeliness of response, ability to provide

           8    data, accuracy of response and reduction in

           9    processing time for the EIA.  And shortly I

          10    will go over measures to operationalize each

          11    of these four criteria and I'll also be

          12    asking your advice on the measures to

          13    operationalize each.  We also hope to use

          14    this proposal as a template to evaluate other

          15    surveys, which EIA has applied it's survey

          16    design and testing program to do something

          17    that Nancy talked about earlier.

          18              So EIA will not just have a testing

          19    and design program, we will have a testing,

          20    design and evaluation program.  Now before I

          21    get to the measures for each of the criteria,

          22    I just want to review the EIA-920 with you.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             36


           1    The EIA-920 combines a heat and power plant

           2    report, collects data from combined heat and

           3    power plant facilities such as, papers mills,

           4    refineries, steel mills and food processing

           5    plants and so on.  We collect data on four

           6    variables.  One is total fuel consumed by the

           7    CHP facility, the second is fuel consumed to

           8    generate electricity in that CHP facility,

           9    third is electrical generation and fourth is

          10    fossil fuel stock, coal and distillate oil.

          11              The major focus of the survey is

          12    fuel consumed to generate electricity.  If

          13    the respondent has understood the survey, the

          14    EIA-920, the reported quantity for fuel

          15    consumed to generate electricity should be

          16    less than total fuel consumed because,

          17    basically the CHP plant should be using it's

          18    fuel for two purposes.  One, to generate

          19    electricity and the other for it's

          20    manufacturing process, such as drying paper.

          21              Two other things about the EIA-920

          22    that I just forgot to mention is that it is a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             37


           1    monthly form of a sample selected from all

           2    CHP facilities and I think there are probably

           3    about 600 -- and that could be wrong -- CHP

           4    facilities in this country.  There's also an

           5    annual form that's completed by CHP

           6    facilities that are not selected to respond

           7    monthly.  Okay, now into the variables that

           8    we plan to use, or hope to use - - to

           9    operationalize the four criteria that I

          10    presented earlier.  The first criteria, is

          11    timeliness and under timeliness we want to

          12    measure that or evaluate that by more

          13    respondents reporting within the reporting

          14    period.

          15              What I mean by more respondents is

          16    that before the EIA-920, EIA used to collect

          17    data from CHP plants on the form EIA-906,

          18    which is power plant reports, now that

          19    EIA-906 collects from regulated utilities, so

          20    add more respondents to the EIA-906, and two

          21    fewer respondents getting reporting

          22    extensions, EIA would get phone calls from




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             38


           1    respondents saying, no I can't fill this up

           2    by the 15th of the month can you give me an

           3    extra 5 days or 10 days or so on.  And also

           4    I've been using the word "we" a lot here, I

           5    think that's a good sign to mention that on

           6    this project I'm working with my colleagues

           7    Tom Broene, Stan Freedman, Guy Caruso and

           8    Karen Norman, all of Statistics and Methods

           9    Group.

          10              The 2nd criteria that -- in our

          11    plan is ability to provide data and we have a

          12    number of measures that we plan to use to

          13    evaluate that.  One is response rate, and

          14    again, the response rates -- you know, higher

          15    than let's say then was on the EIA-906,

          16    that's just one example.

          17              Now there is reporting errors,

          18    pardon me.  Let's say that there's a problem

          19    with a particular variable -- all right, in

          20    the EIA-920, let's say that instead of

          21    recording just fossil fuel stocks such as

          22    coal -- you know, plants are recording




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             39


           1    something in storage that is not a fossil

           2    fuel.  So there may be a problem with the

           3    instructions or the survey itself not being

           4    clear.  The 3rd thing we have is analysis of

           5    comments, you'll notice at the bottom of the

           6    first page of the EIA-920 the section of

           7    comments from respondents, so we're going to

           8    analyze those respondents and also their

           9    comments that from EIA staff, that is put in

          10    the file for the EIA-920 so we're going to

          11    look at that too, and also down the line

          12    after we've -- you know, looked at response

          13    rate, reporting errors and all sorts of

          14    comments and other things we're going to talk

          15    to the EIA-920 processing staff.

          16              We're going to ask them a lot of

          17    questions but main -- basically things like,

          18    what you think is going right with this

          19    survey, what do you think the problem area

          20    is, what's the bulk of your pullbacks on,

          21    what's the bulk of questions that you get

          22    from respondents, what can we do to improve




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             40


           1    the survey and so forth, a lot of things.

           2    And then, after that probably one of the last

           3    things we're going to do is, we're going to

           4    follow-up on respondents and we're going to

           5    ask them question as basic as you think this

           6    survey is an improvement over the survey

           7    EIA-906, do you understand what EIA wants in

           8    question 2, that's a key question there, fuel

           9    used to generate electricity and several

          10    other questions.

          11              Okay we have, two other criteria

          12    and the 3rd one is accuracy of response.  One

          13    thing we're going to do statistically is

          14    compare data on the EIA-920 to data collected

          15    on the EIA-906, the power plant report.  This

          16    is a little bit of a pitfall if it's here,

          17    because on the EIA-906, EIA collected data

          18    from CHP plants for something called useful

          19    thermal output, which is a fuel used for

          20    manufacturing and other processes and combine

          21    heat and power plants.

          22              From useful thermal EIA then




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             41


           1    calculated fuel used to generate electricity,

           2    as the difference between useful thermal

           3    output and total fuel consumed.  It was EIAs

           4    wariness about the accuracy of use of thermal

           5    output in the calculated fuel used to

           6    generate electricity now but that led to the

           7    design and implementation of EIA-920, so the

           8    comparison there is a little bit fraught as

           9    one of my colleagues suggested yesterday.

          10              Two, we're going to try to do this

          11    also this act by comparison with sub groups

          12    in the EIA-920 by prime mover steam turbine

          13    configuration.  There's different type of

          14    turbines, there's steam turbines, gas

          15    turbines and other types of setups.  And also

          16    by industry is there a difference between

          17    paper mills versus food processors, or

          18    plastic manufacturers versus refineries and

          19    so forth.  And finally, our final criteria in

          20    the measures we've have for that, is

          21    reduction in processing time for EIA.

          22              We're going to look at -- you know,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             42


           1    has there been a decrease in respondent call

           2    backs and in other words that would be kind

           3    of an indication if there's been a decrease

           4    in call backs that respondents are having --

           5    are understanding the survey and in other

           6    words they're filling in more items than,

           7    let's say the EIA-906 and this will also

           8    indicate that maybe the EIA is saving

           9    resources on the EIA-920.

          10              MR. RUTCHIK:  So those are 4

          11    criteria and on these criteria we just listed

          12    out, we have measures we want to use

          13    operationalize that criterion.  We've only

          14    done research so far on ability to provide

          15    data and as I said there is preliminary

          16    analysis and that's a very good phrase.  We

          17    looked at January to May data.  We have

          18    looked at about 140 respondents of the

          19    approximately 300 responses that EIA got for

          20    that period.  We looked at respondent in the

          21    EIA comments, then we compared the question.

          22              One, total fuel consumption data to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             43


           1    question two data, that's data -- fuel use to

           2    generate electricity data.  The comments

           3    provided very little systematic information.

           4    There is nothing in the comments that just

           5    leaked out at you to say, this is going right

           6    with the survey, this is going wrong, and

           7    kind of all over the board and illuminated

           8    very little.

           9              The question -- two to question one

          10    comparison though did yield to this breakdown

          11    of the 140 or so responses that we looked

          12    for.  Seventy four respondents recorded a

          13    fuel use to generate electricity at the CHP

          14    plant was less then total consumption, in

          15    other words, these people seem to have got

          16    the basic concept of the survey.  There were

          17    six to eight respondents that did not seem to

          18    get the concept of the survey.  For most of

          19    these, they report their fuel use to generate

          20    electricity was equally to their total fuel

          21    consumption.

          22              There were some that did not report




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             44


           1    consumption at all just before -- no data at

           2    all, just had some other problems with their

           3    responses.  There is also some other

           4    findings.  I don't want to put findings in

           5    quote.  From our analysis a look at these 140

           6    responses.

           7              One is on waste heat fuel

           8    consumption not reported.  Let me explain

           9    waste heat is.  In combined heat in power

          10    plants, some of them have what's called

          11    combined cycle technology.  And you are

          12    getting the full brunt of my engineering

          13    knowledge with this.  Combined cycle

          14    technology, in that fuel is, let's take coal,

          15    is put into in a boiler churns(?) out turbine

          16    electricity is generated.  Then a process of

          17    generating electricity, steam is produced, at

          18    that time the plant CHP facility can do one

          19    or two things with that steam, they can admit

          20    it into the atmosphere.

          21              Hence the phrase "waste heat," they

          22    don't need it, they are not going to use it,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             45


           1    it's waste, or they can recycle the steam and

           2    use it, let's say as part of their

           3    manufacturing processes, you know, to

           4    manufacture the candy or again use my

           5    favorite one, drive the paper in the paper

           6    mill.  Some are designed as this cognitive of

           7    any views, we knew that the plants would have

           8    problems reporting fuel consumption for waste

           9    heat.

          10              These are waste heat, we really do

          11    not consume fuel for waste heating because

          12    the production has already been done at the

          13    first part of the cycle, so therefore they

          14    would report zero fuel consumption for waste

          15    heat, and that's what happened when we look

          16    at the responses for the combined cycle

          17    points in the 140 or so that we looked at, I

          18    think we did about maybe about a dozen

          19    combined cycle facilities of the 140 and none

          20    of them I think reported any fuel consumed

          21    with a waste heat.

          22              The second kind of finding was a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             46


           1    larger respondents appeared to report

           2    correctly, what I mean by this is unless (?)

           3    we have, let's say a large food processing

           4    company or a large paper company and they

           5    have eight facilities reporting eight or nine

           6    -- 8 separate responses and as I looked at

           7    them and Stan looked at them we see that

           8    let's say six reported correctly, they

           9    reported their fuel use to generate

          10    electricity with less then total fuel.

          11              Two, total fuel was equal to fuel

          12    use to generate electricity.  So three

          13    quarters of the people in that company saying

          14    that they got the idea of the survey, and it

          15    also shows that this decentralized reporting

          16    you know, the reports are coming to EIA or

          17    EIA-920, it is not coming from the EIA from

          18    the central headquarters, but from the

          19    individual facilities.

          20              And the final two things is that

          21    our pre-survey design respondents reported

          22    correctly in the 140 or so, there were about




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             47


           1    four pre-survey design people that we have

           2    visited and talked to and our education

           3    efforts seemed to work, they seemed to get

           4    the hang of the survey.  There were about 6

           5    or 7 cognitive respondents in this group, I

           6    think was half and half on whether they got

           7    the idea of survey or did not.

           8              Now I can come up with a facile of

           9    rationale for why it was this way with the

          10    cognitive respondents versus the pre-survey

          11    design respondents and I will.  In the

          12    cognitive respondents, we let's say visited

          13    you know, just once facility out of let's say

          14    3 or 4 of that company and that one facility

          15    filled it out correctly but the others that

          16    we didn't visit would not fill it out

          17    correctly, so again direct education may

          18    work.

          19              Okay, so that's the only research

          20    we've done on this so far.  And these are the

          21    questions I have for the committee.  Are the

          22    criteria, the full criteria that EAI has




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             48


           1    proposed for the evaluation at the ENI-20,

           2    are these fully valid?  Are the other

           3    criteria that EIA can use?  And finally, are

           4    the measures to operationalize each of the

           5    four criteria -- are there criteria -- are

           6    there measures other than the ones we

           7    proposed that we can use still?

           8              In other words, is there something

           9    under ability to provide data that we have

          10    not listed that we can use.  Again to sum up

          11    you know, the thrust of the question, is EIA

          12    on the right track to assessing survey

          13    effectiveness?  Thank you very much and open

          14    it up for the committee.  Nicolas?

          15              DR. HENGARTNER:  I have a question

          16    that is not addressing any of these issues

          17    but I'm really thankful for you to have

          18    provided that test form because the first

          19    time I see what you were talking about, and

          20    that I'm reading at the top and the sentence

          21    says, report actual value.

          22              MR. RUTCHIK:  Very good.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             49


           1              DR. HENGARTNER:  These are not

           2    available, report estimates the value, now if

           3    I go down on the phone I see nowhere an

           4    indication that the value you are collecting

           5    are either real or estimate.  That is an

           6    issue.  So that goes to the accuracy and the

           7    ability to answer because if we don't know if

           8    the numbers that we are getting are real or

           9    estimate, then we have a problem.  And there

          10    is also no ability to correct as the fact

          11    that you have an estimate as to make the

          12    value in January and you get the real numbers

          13    in February do you resubmit the form?  What's

          14    happening?

          15              MR. RUTCHIK:  Okay, this may not

          16    fully answer your question, and that's

          17    something I left out.  When we did a

          18    pre-survey design as it is in, they were not

          19    cognitive interviews, the respondents told us

          20    that it could measure directly fuel used to

          21    generate electricity.  It goes into the right

          22    nature of the configuration of the CHP




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             50


           1    plants.

           2              But he said, through I'm backing

           3    out of steam pressures and other things, that

           4    they could provide us with an estimation.  I

           5    know this direct -- addressed it directly,

           6    only question two but that's what they told

           7    that stuff.  What we were getting most of the

           8    time in question two, if you used to generate

           9    electricity are, you know, estimates,

          10    evaluations done by the facility themselves

          11    and as a matter of fact this was the just as

          12    the talk of the first session we had on this

          13    in April 2003.  So I don't know that really

          14    answered your question but these are some

          15    questions -- calculations are right.

          16              MR. BLAIR:  May I follow up --

          17              MR. RUTCHIK:  Yes, sure.

          18              MR. BLAIR:  To the next question

          19    and that is, do you know from talking with

          20    the respondents sort of what procedures they

          21    use for estimation and do you agree with how

          22    they are coming up with their estimates?




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             51


           1              MR. RUTCHIK:  No, not at this

           2    point, I don't know.  But that would be a

           3    good idea to use if, you know, we do

           4    follow-ups.

           5              MR. BLAIR:  I think I mean just

           6    about the second letter.  If you are going to

           7    continue these visits in what you described

           8    as education that if you know sort of how

           9    people are coming up with these estimates

          10    that that could be some part of what you

          11    continue to collect the information on and

          12    also maybe build into part of your education

          13    networks.

          14              MR. RUTCHIK:  But it is fun, I

          15    think it's more basic one that you want to

          16    know sort of which way it is estimated --

          17              DR. HENGARTNER:  And the danger now

          18    is that using question two, those estimated

          19    value or how sophisticated, they already

          20    estimated that as - - evaluated through a

          21    survey by saying, well, hold the consumption

          22    and the electrical estimate and consumption




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             52


           1    producing -- well, not asking the price, they

           2    could be the same.  Well, you have to be a

           3    little careful using those values to evaluate

           4    how well the respondents are understanding

           5    the form.  As you are measuring out the

           6    specification of the respondents, we need to

           7    have as you say, calculating the pressure and

           8    we are getting a specific way beyond

           9    cognitive.

          10              MR. RUTCHIK:  Yeah.  Anybody else

          11    has anything they want to ask?  John, I'm

          12    sorry, John.

          13              MR. BLAIR:  I was just going to

          14    come back to the criteria number, response

          15    rate which is obviously important but

          16    describe sort of separately and I wonder if

          17    you are putting together sort of the unit

          18    response rate and the item response rate, so

          19    that for each of the variable each of the

          20    questions that you can compute an individual

          21    rate which is the unit response rate times

          22    the item response rate.  And that is what




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             53


           1    really tells you I think what you have for

           2    each of your data for instance.

           3              MR. RUTCHIK:  Good idea.  I think

           4    you should --

           5              MS. WAUGH:  I'm looking at -- is

           6    this on?

           7              MR. RUTCHIK:  Yeah.

           8              SPEAKER:  Can you tell me who you

           9    are?

          10              MS. WAUGH:  Shawna Waugh from EAI,

          11    I'm looking at some of the theory if you will

          12    about survey talks about three factors,

          13    timing, accuracy and cost.  And it appears

          14    that the EIA processing time might be related

          15    to cost but I didn't actually see any

          16    variable that would evaluate cost

          17    effectiveness and that might be something to

          18    also attempt operation on.

          19              MR. RUTCHIK:  Thank you.  Okay is

          20    there anybody else have any comments,

          21    suggestions because all we got so far have

          22    been very, very helpful.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             54


           1              MR. BLAIR:  Yes, maybe if you go

           2    back for a bit to -- you talked about the

           3    comments that you receive --

           4              MR. RUTCHIK:  Yeah.

           5              MR. BLAIR:  And they were kind of

           6    all over the place and there was no pattern.

           7    I guess sort of two things and one is if I

           8    get comments all over the place and the

           9    comments seem sensible that worries me a bit

          10    because, it sound like then that are

          11    problems.

          12              MR. RUTCHIK:  Well, what I meant by

          13    that I probably wasn't very clear, there

          14    didn't seem to be any great mass or number of

          15    comments on any one thing.  I think I did

          16    something where I did like a frequency chart,

          17    a bar chart of frequency.  You know, when I

          18    did various categories and have two or three

          19    on this, two or three on that, three of four

          20    on this, three or four on that, there wasn't

          21    something like let say 140 respondents and

          22    not all those 140 -- well actually I looked




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             55


           1    at the whole 300 actually, not everybody

           2    makes a comment, so those who made comments,

           3    there wasn't somehow but say 15 or 20 or 25

           4    saying that we don't understand question two.

           5              MR. BLAIR:  Right.

           6              MR. RUTCHIK:  That's what I meant.

           7              MR. BLAIR:  Sure, but you also

           8    suggest that you did some sort at least

           9    informal coding of the comments in order to

          10    --

          11              MR. RUTCHIK:  Yeah, informal would

          12    be a good word here.

          13              MR. BLAIR:  Okay, if now that you

          14    are getting that was it 68 out 140 or

          15    something, 142 that provided comments?

          16              MR. RUTCHIK:  No, I don't know how

          17    many provided comments, we didn't count that

          18    up.

          19              MR. BLAIR:  Oh that was different

          20    --

          21              MR. RUTCHIK:  That was a --

          22              MR. BLAIR:  People who didn't get




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             56


           1    it.

           2              MR. RUTCHIK:  Yeah, there was a

           3    question one versus two comparison.  We

           4    didn't really count up how many people, of

           5    responses -- respondents provided comments.

           6              MR. BLAIR:  Oh, okay, well and this

           7    is kind of stupid(?) and you certainly want

           8    to count that and depending on how many that

           9    we have may be able to do some more formal

          10    coding and track over time you know if there

          11    are some differences but, how useful that is

          12    depends on sort of how many comments you got

          13    and if you got five comments you know what

          14    can you do, but if you had 50 you know that's

          15    something, something else.  So if you have

          16    some substantial numbers then I would you

          17    know, look at developing some sort of a

          18    coding for it.

          19              MR. HOUGH:  My name is Rick Hough

          20    from Census Bureau, you guys investigated the

          21    use of electronic reporting for your response

          22    related issues?




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             57


           1              MR. RUTCHIK:  I wish somebody here

           2    from the program office was here, but I think

           3    the survey is reported electronically.  It's

           4    not a mail in survey, respondents go to a web

           5    site, they access the survey, they fill it

           6    out and send it back electronically, I mean

           7    that's a very welcome and a clear way of

           8    explaining it but is it on an electronic

           9    reporting system.

          10              MR. HOUGH:  Okay.

          11              MR. BLAIR:  I don't want to drag

          12    this out, but, a question on the expert

          13    review.  You mentioned that the survey staff

          14    with the people and I wonder if there might

          15    be a larger, of course I'm actually not sure

          16    what the survey says but includes and that

          17    includes the people that are responsible for

          18    EEO, the processing or people analysis with

          19    the data you know for the users but you might

          20    consider sort of to know exactly who is that

          21    that are appropriate for this expert

          22    revealing, maybe something part of you should




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             58


           1    make it different information from the

           2    processing staff, that you get from the data

           3    collecting staff.

           4              MR. RUTCHIK:  Anybody else,

           5    anything you wish to add.  I want to thank

           6    you first for participating and listening and

           7    if you don't have anymore comments and if you

           8    do please come forward I just want to thank

           9    you because of the comments we got were very,

          10    very clear in substance.  Thank you.

          11              SPEAKER:  Good job.

          12              CHAIRMAN BREIDT:  Okay, I guess we

          13    will get started with the summary of the

          14    break out session.  So first Johnny Blair

          15    will summarize the proposal for announces of

          16    collected data break out issues.

          17              MR. BLAIR:  EIA has begun

          18    evaluation of redesign form 920 and in our

          19    session we are basically commenting on the

          20    evaluation plan and essentially comments were

          21    very positive.  There were three questions

          22    that were posed to the committee, the first




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             59


           1    one was, are the evaluation criteria the

           2    right ones?  There were four general criteria

           3    which are timeliness of response to the

           4    survey, ability to provide the data that is

           5    requested, the accuracy of response, the

           6    accuracy of those data and reduction in

           7    processing time for the EIA.

           8              And in general we thought yes,

           9    those were appropriate criteria but we had

          10    some comments, which I will come to later on

          11    how to apply them.  The second question to

          12    the committee was, are there other criteria

          13    that EIA can use and there were two points

          14    that were raised here, the first which was

          15    raised actually by someone in the audience

          16    was that cost is not explicitly measured in

          17    the criteria.  There are some indirect

          18    measures such as processing time, but we felt

          19    that more could be done to track cost

          20    directly in the track changing cost due to

          21    the new form.

          22              The second is that the form has on




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             60


           1    page 2 interesting instruction, which is

           2    report actual values if these are not

           3    available, we put estimated values.  And a

           4    comment on this was that it would be useful

           5    to know item by item for each respondent

           6    which items are actual and which items are

           7    estimated and so the suggestion was to make a

           8    minor modification in the forms so that

           9    respondents can indicate when they are

          10    putting in estimated values and more over in

          11    the follow-up with respondents which is also

          12    a part of the evaluation plan to look at how

          13    those estimates are arrived at, how are the

          14    estimates being constructed and whether EIA

          15    agrees in general with the estimation

          16    procedures and if not what suggestion that

          17    they can make to improve them and as part of

          18    their education, ongoing education of

          19    respondents, what can they do to improve

          20    those estimates.

          21              The third question to the committee

          22    was what, what measures can be use to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             61


           1    operationalize the criteria and just a couple

           2    of comments here, one connected to -- related

           3    to the expert review which is one procedure

           4    that used right now.  There is an expert

           5    review by the survey staff and we suggested

           6    that that might be expanded to include other

           7    people such as the processing staff, data

           8    analysts and more broadly other users to

           9    participate in that expert review since

          10    expert reviews are generally the low cost and

          11    quick way to get response.

          12              There should be really marginal

          13    additional effort to expand the pool of

          14    experts, they do their review.  And finally

          15    on that response rate which is obviously an

          16    important criterion, really unit response

          17    rate was focused on and some mention of item

          18    response rate but the suggestion was that

          19    item by item is question by question of that

          20    the product that goes is really what one

          21    should be looking at, the product of unit and

          22    item response rate for each question to see




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             62


           1    sort of what you're really getting at the

           2    question level.

           3              And those are the notes that I have

           4    which actually probably took as long as our

           5    session did, which very quickly but if any of

           6    the other committee members want to correct

           7    or add anything to what I've said, now is the

           8    time.

           9              CHAIRMAN BREIDT:  Okay any comments

          10    or question for Johnny?  Okay thanks, Johnny.

          11    And now I will summarize the secondary break

          12    out session which is a customary evaluation

          13    of the Short Term Energy Outlook STEO, this

          14    is based on a random sample of 500 email

          15    addresses from list of about 4,000 email

          16    addresses.  These are self-selected, it's a

          17    list -- so these are people who have chosen

          18    to repeat new updates monthly and these are

          19    thought to be fairly representative of the

          20    core users of the STEO so.

          21              From that email list, you have a

          22    logistically straightforward means of




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             63


           1    sampling and there is a lot of automation

           2    that can be built into it which is a nice

           3    feature.  The run for about 25% unit response

           4    rate and one of the questions had to do with

           5    response rate.  Howard Bradsher-Fredrick

           6    described the results of the survey,

           7    generally showing pretty much satisfaction

           8    with the STEO, it's fairly consistent for

           9    results for other products particularly the

          10    Annual Energy Outlook and the International

          11    Energy Outlook and you might recall from the

          12    April meeting those were reported on the

          13    basis of this sample, the satisfaction

          14    survey, in that case was a survey of NEMS

          15    conference register, is that right.

          16              So let's see, one of the questions

          17    on the survey that was emailed to people was

          18    whether or not that respondent would agree to

          19    follow up for more detailed questions, so

          20    again there is stage of self-selection.  You

          21    have the self-selected list of -- you have

          22    those who agree to participate in the survey




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             64


           1    and then further you have further self-

           2    selection on those who might be willing to

           3    follow it up.  And still with all this

           4    self-selection going on, you get something

           5    that is not necessarily representative of all

           6    STEO users uses or even necessarily of all

           7    STEO core users but probably something useful

           8    in terms of customer satisfaction.  You get a

           9    useful first look.

          10              The questions for the committee

          11    were, how often should customer surveys of

          12    this type be conducted?  Would once a year be

          13    too frequent or not and is the response rate

          14    adequate and then questions on other

          15    suggestions.  So on the first question, how

          16    often should customer surveys of this type be

          17    administered?  The committee I think was in

          18    general agreement, you didn't want to do it

          19    every year sending out to everyone that would

          20    be unduly annoying, so it was thought that

          21    one thing you could do is to implement some

          22    kind of rotating panel where you might




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             65


           1    revisit some people so that you can track

           2    change over time if there were changes that

           3    were of interest, you do want to revisit some

           4    of the same folks and a possible selection

           5    for part of that core panel could be some of

           6    these people who have self selected to hear

           7    the survey again.

           8              Beyond that you can have a rotating

           9    component and since there is 4,000 names on

          10    the list and that gets somewhat refreshed

          11    over time as people add and drop, you can

          12    rotate through the email addresses for the

          13    sample as a 500 only over eight years so

          14    that's not really unduly burdensome.  The

          15    second question has to do with the response

          16    rate, it was 25.9% and would that be adequate

          17    and again spent some what you mean by

          18    adequate, this really isn't going to be a

          19    representative sample of all STEO users again

          20    but it should be a useful sample in terms of

          21    trying to get some information on customer

          22    satisfaction.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             66


           1              So the list service is self

           2    selected, the respondents are self selected

           3    after four ways of email follow- up and still

           4    the thinking is that you should be able to

           5    get some useful information from those who

           6    want to be heard, those who are actually

           7    taking the time to respond to the survey are

           8    those who really have an opinion and this

           9    should give you useful first look at the kind

          10    of those core users.

          11              But you get something that's sort

          12    of actionable where you might be able to work

          13    on the STEO.  The performance of STEO you

          14    want to go back to those core uses who added

          15    strong opinion and actually find out exactly

          16    what is it that they have questions or

          17    concerns about.  And it seems that they do

          18    have a number of mechanisms for giving

          19    feedback to EIA on this product but through

          20    the survey mechanism they do have this thing

          21    where they can self select to be followed up

          22    and nothing has been done with that yet but




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             67


           1    it could be, so that can be very useful to

           2    jot down with that follow-up to get some

           3    discussion and specifics like model

           4    components and things like that.

           5              As far as increasing response rate,

           6    there are things that you can do but with

           7    multiple ways of email follow-up it's not

           8    going to do the trick.  At some point you

           9    again have to change the mode of data

          10    collection if you really want to increase the

          11    response rate and the other possibility is

          12    that, eventually there is going to be these

          13    non respondents that are going to respond no

          14    matter how many times you contact them to

          15    this kind of conventional mode follow-up, you

          16    might sub sample them and then make some

          17    efforts to get out responses from them, may

          18    with a simplified questionnaire.

          19              So and maybe giving them some

          20    incentive to respond, I don't know.  Then the

          21    other question was other suggestions and the

          22    discussion kind of centered around other




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             68


           1    modes of entry, sort of other entry points

           2    for data collection and these included some

           3    discussion of the American Consumer

           4    Satisfaction Index ACSI which is a pop under,

           5    must be careful not to say pup up.  But pop

           6    under methodology that is sitting there on

           7    the web page after you close you browsers so

           8    it has -- this has caused technical problems

           9    where people have a dozen of these open or

          10    something and they are irritating and the

          11    other concern is that there is a sense of

          12    frame bias because the more technological

          13    savvy users have blocked all their pop ups

          14    and pop unders and everything else.

          15              So it's maybe not as useful as

          16    other means of getting that data but it's

          17    good to have for customer satisfaction not

          18    necessarily the case that you need

          19    representation, you need to give people the

          20    opportunity of have their voices heard in

          21    different mechanism, two different

          22    mechanisms.  So let's see and one of the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             69


           1    other possible mechanisms discussed was sort

           2    of an invitation on the web page where you

           3    were invited to participate in survey.  Again

           4    self-selection but again potentially giving

           5    you useful information that could be followed

           6    up on.  So that was kind of the being focus

           7    of our discussion I think.  Is there any

           8    comments from people who are there to follow

           9    up on that question.

          10              DR. FEDER:  Jay when you said

          11    rotating panel, are those panels going to be

          12    interviewed more than once?

          13              CHAIRMAN BREIDT:  Yes --

          14              DR. FEDER:  Okay.

          15              CHAIRMAN BREIDT:  I think of a core

          16    panel that would be interviewed year after

          17    year and the others would sort of rotate out.

          18              DR. FEDER:  I have one small

          19    question, is that, what about new users is,

          20    there any attention going to be paid to new

          21    users?

          22              CHAIRMAN BREIDT:  Well, the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             70


           1    interest through the new users who are core

           2    users who self select and get in on the list

           3    there you know, can get in there but I guess

           4    it would be a good idea to try to get those

           5    into the core as well, refresh that panel,

           6    that didn't really come off -- there is a

           7    concern about new users to kind of get a

           8    lower level being more frustrated with

           9    products and things like that, so it's a

          10    definitely interesting sub group.  Any other?

          11              MS. KHANNA:  Yeah, I think on the

          12    clarification point.  One of the things that

          13    we talked about is the (?)N.  We are talking

          14    about listing this up under -- approach was

          15    that -- even that would not get rid of self

          16    selection or even the bias in the selection

          17    because people who bookmark the pages would

          18    not necessarily receive pop under or the

          19    invitation, the letter that we sort of talked

          20    about --

          21              CHAIRMAN BREIDT:  Right.

          22              MS. KHANNA:  Having invitation but




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             71


           1    even there there was going to be some issues

           2    of bias even the potential sample that you

           3    got --

           4              CHAIRMAN BREIDT:  Right.

           5              MS. KHANNA:  Apart from

           6    self-selection.

           7              CHAIRMAN BREIDT:  And that just

           8    reminds me something that -- I think that

           9    statistics is something like 10,000 hits on

          10    the STEO page and 4,000 users on the lists

          11    there and those are different population,

          12    they overlap to some extent but this

          13    particular survey is really aimed at those

          14    core uses who have self selected to

          15    participate in this email collection.

          16              MR. BLAIR:  Yeah, just a comment on

          17    the official response rate and customer

          18    satisfaction survey that there is some --

          19    there is some evidence in the literature in

          20    fact that indicates that that non

          21    respondents, excuse me, the respondents to

          22    customer satisfaction surveys tend to have a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             72


           1    more positive rating than non respondents

           2    which seems somewhat kind of imperative but,

           3    in fact the explanation or at least that's

           4    been positive, is that respondents, our

           5    sample members look at doing the survey as

           6    kind of a favor for the agency doing the

           7    survey, so there is a kind of issue of

           8    reciprocity, so if they have a positive

           9    feeling towards the agency or whomever is

          10    doing the survey, there is someone more

          11    likely to respond and those who are not so --

          12    this is something I think that you know,

          13    maybe useful to keep in mind when you're

          14    dealing with this low response surveys.

          15              CHAIRMAN BREIDT:  One other thing

          16    we came is that -- the earlier respondents

          17    were grumpier, so as they went along -- Are

          18    there other questions or comments?

          19              DR. HENGARTNER:  Do you also

          20    compile complaints?  I mean customer

          21    satisfaction you know, you don't want to get

          22    people mad.  You don't want them to complain,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             73


           1    I mean as long as they are grumpy that's all

           2    right but as long as they don't complain.

           3              MS. KIRKENDALL:  There is a comment

           4    box on the website, that is people are

           5    invited, they send in a comment if they want

           6    to.

           7              DR. HENGARTNER:  And is it

           8    reviewed?

           9              MS. KIRKENDALL:  Colleen wasn't

          10    reviewing it every month and sends them out

          11    to whoever -- those who have seen it.  But I

          12    think she does you know I think she said

          13    that, that's what she does with it.

          14              CHAIRMAN BREIDT:  Yes.

          15              DR. HENGARTNER:  There are always

          16    going to keep complaining.  The idea is to

          17    keep the rate as low as possible or to avoid

          18    -- notice most things fight in time and

          19    that's one way one could actually monitor the

          20    ongoing quality.

          21              MS. FORSYTH:  For expecting contact

          22    as well.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             74


           1              CHAIRMAN BREIDT:  Yes, because you

           2    know, people are going to grumble, grumble,

           3    grumble -- and then it goes up and that's

           4    when you have to be worried about.  So let

           5    them have the monitoring of the time, which

           6    seems to be not too hard in conception --

           7              MS. KIRKENDALL:  Well, it's sort of

           8    interesting because I think by telling

           9    business every month but I don't know that

          10    I've seen that summary that she puts out

          11    monthly or anything that she would -- and

          12    distributes a number of complaints that she

          13    gets over time related to the same.

          14              MS. FORSYTH:  And it might mean

          15    that some complaints that are specific to a

          16    particular product or a particular, I don't

          17    know -- a page or whatever that hearing about

          18    them for another page people who manage other

          19    contracts or other pages might be able to

          20    manage the content -- the distribution of

          21    that at least some kind of content might also

          22    be --




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             75


           1              MS. KIRKENDALL:  We should have our

           2    new web champion to look at it.

           3              MS. FORSYTH:  Yeah.

           4              CHAIRMAN BREIDT:  Nagaraj?

           5              DR. NEERCHAL:  Well, another point

           6    he is going to bring up is that you know, one

           7    aspect of customer satisfaction is to letting

           8    all the customers know that you are improved

           9    -- you are trying to improve and I think one

          10    thinks that might be useful is based on the

          11    survey or even as of your normal way of doing

          12    things, you come up with improvement, send a

          13    general email to the list saying that you

          14    have improved the service.

          15              You know, so because many people

          16    who are not responding, they may not even

          17    have noticed the problem but the fact that

          18    you saw a problem and you have improved it, I

          19    think that is already kind of improves your

          20    scoring in their book.

          21              CHAIRMAN BREIDT:  Okay, so we are

          22    on time or pretty much on time to go to next




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             76


           1    break up session.  Can we reconvene there?

           2                   (Recess)

           3              MS. SUTHERLAND:  Good morning I'm

           4    Grace Sutherland welcome to the session on

           5    assessing EIA frames and update on EIA wide

           6    projects.  This will be an update from the

           7    spring meeting as to what we've been doing

           8    and where we are headed.  For a brief

           9    background, in the spring EIA presented to

          10    the committee the activity this has been

          11    involved in up to that point, it included

          12    checking respondent list, comparing aggregate

          13    data, examining supply to sufficient balances

          14    and comparing price data volumes.

          15              In response the committee suggested

          16    that EIA ask known establishments to identify

          17    others within the same market to calculate

          18    propensity scores by post stratifying

          19    potentially using census data.  To apply the

          20    principles of dual system estimation to

          21    available frame data, to obtain as much

          22    information from census without disclosing




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             77


           1    sensitive data in order to identify missing

           2    respondents.

           3              That EIA might forego using

           4    balancing items for measuring coverage -- the

           5    balancing items with least favorable message

           6    of assessing frames.  In response to the

           7    committee's suggestion with regard to asking

           8    establishments to identify others within

           9    their same market, EIA had tried that and

          10    respondents have been reluctant to provide

          11    EIA with their customer's information.

          12    However we have had success in obtaining

          13    names of the buyers or competitors.  For

          14    instance our electric power surveys has used

          15    this technique.  The EIA is not currently

          16    preferring the sampling approach but may

          17    consider it again in the future.

          18              With regards to fine principles of

          19    dual system estimation to available frame

          20    data, EIA is in favor of this whenever

          21    possible and have applied this principle, for

          22    example EIA compared the EEIA electricity




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             78


           1    renewable frame with the National Renewable

           2    Energy Laboratory frame.  We also now have an

           3    EIA wide interoffice team with evaluating

           4    frame sufficiency.  Because of the difficulty

           5    in coming up with quantitative assessments

           6    for frame sufficiency for all surveys, EI is

           7    pursuing a qualitative approach.

           8              The team is using the information

           9    gathered previously on it's survey and is not

          10    only gathering additional information and

          11    will look at frame stability over a longer

          12    period of time.  We currently have just the

          13    past year or ongoing frame, and we don't know

          14    whether or not surveys are using comparable

          15    lists and if they are not, why?  Whether

          16    there reason are such constrains or legal

          17    issues?

          18              And at the spring meeting in 2005,

          19    you will be hearing an update from that

          20    team's effort.  Up next you will hear Howard

          21    Bradsher-Fredrick of the EIA who will

          22    summarize the dual system estimation study.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             79


           1    The EIA electricity renewable frame and the

           2    National Renewable Energy Laboratory Frame.

           3    And then Shawana Waugh of the EIA will

           4    describe the collaboration between the Census

           5    Bureau and EIA regarding the evaluating of

           6    several EIA frames and manufacturing sector.

           7    And last you will hear from Tom Lorenz of EIA

           8    who will describe what he has learnt by using

           9    data from EIA's petroleum surveys both

          10    monthly and annual to edit MECS data.  MECS

          11    stands for Manufacturing Energy Consumption

          12    Survey.  Although Tom's analysis was an

          13    editing project, the frames he will be able

          14    to use, his analysis in effecting the frames.

          15    And at the end of Tom's presentation, the EIA

          16    would like a feedback from committee.  But

          17    first we will hear from Howard

          18    Bradsher-Fredrick.

          19              MR. BRADSHER-FREDRICK:  Okay I

          20    think my slides are coming up.  Okay, I'm

          21    going to discuss the application of the dual

          22    system principles to compute a hypothetical




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             80


           1    estimate of the number of frame elements

           2    missing from both the National Renewable

           3    Energy Lab data frame and I mean I will refer

           4    to this as NREL and the EIA renewable frame.

           5    As Grace mentioned this analysis was

           6    requested by the committee at the last

           7    meeting, last spring.

           8              NREL collects data from secondary

           9    sources and generates a nameplate capacity

          10    from renewable energy generating plants, the

          11    NREL list is publicly available and EIA

          12    collects data on renewable plants but it was

          13    a 1 megawatt cut off.  So we have eliminated

          14    the NREL facility below the cut off from the

          15    study.  This exercise was intended to show

          16    how the dual system principles might be

          17    applied in a real setting if these

          18    assumptions were to hold, I won't get to the

          19    assumptions until the end.

          20              First I'll go through some of the

          21    computations.  So excuse me, we are going to

          22    proceed to estimate the number of facilities




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             81


           1    missing from both frames if the assumptions

           2    were to hold.  So we look at the total data,

           3    you notice that the NREL is just 5,146

           4    facilities, the EIA frame had 4,960 and if we

           5    break this down into the components, we have

           6    in the NREL list and in the EIA frame we

           7    4,932 in common and we have in the EIA frame

           8    28 that are not in the NREL list and we had

           9    214 in the NREL list that were not in the EIA

          10    frame.

          11              So we had these three sales

          12    together.  We get a total of 5,174 known

          13    facilities.  So we want to try to figure out,

          14    well, how many are we likely to have missed

          15    in both.  We can sell for X with this by

          16    making a few assumptions.  So we assume

          17    independence, we have the probability of NREL

          18    error times the probability of an EIA error

          19    times the total number of facilities, we do

          20    the mathematics here, we get 1.2 missing

          21    elements.

          22              We can also apply that principle to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             82


           1    the various fuel strata that also have

           2    information on.  In this case in fact we

           3    looked at all the fuels but all of them came

           4    out to be zero except in the case of

           5    agricultural residuals.  In that case NREL

           6    showed 27, EIA showed 33, and you will notice

           7    here we have a higher error rate because we

           8    have not -- there is a percentage not as many

           9    as common, 19 in both, 14 that are not in the

          10    NREL, but in the EIA frame, we have 8 that

          11    are in NREL and not in the EIA frame and we

          12    have X being those that we are trying to

          13    figure out which are those that are not in

          14    both is an estimate.

          15              Again we can do the mathematics

          16    here and we get 2.7 missing.  Again applying

          17    the same assumptions and those assumptions

          18    that were essentially using here that there

          19    are two necessary ones and I was really

          20    seeing that neither of these hold.  The

          21    frames need to be independently assembled and

          22    at times that seems not to be the case




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             83


           1    because NREL uses all secondary sources and

           2    we know they also use the EIA as a source.

           3              And the missing frame information

           4    should be random rather than systematic and

           5    this appears not to be the case, I mean at

           6    least in terms of timber residues.  We have a

           7    140, other 214 of the missing elements in the

           8    EIA frame.  The 65% being in all one fuel, so

           9    this appears to be system error.

          10              MS. SUTHERLAND:  Okay, up next it

          11    is going to be Shawana Waugh who is going to

          12    talk about the collaboration between census

          13    and the EIA.

          14              MS. WAUGH:  Good morning.  I will

          15    be talking about the five frames that are

          16    being evaluated by the Census Bureau and they

          17    are all in the manufacturing sector and I'd

          18    like to start actually by introducing Rick

          19    Hough from Census Bureau.  He is the next

          20    survey manager and he was actually -- if you

          21    like to go ahead and stand up Rick.  He is

          22    the person who will be working on this




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             84


           1    analysis.

           2              I'll be covering the purpose of the

           3    frames evaluation.  I'll be reintroducing you

           4    to the five frame surveys here, EIA(?) that

           5    is being evaluated and then talking about the

           6    methodology that we propose to use as well as

           7    applying it to both EIA 5 which is COKE

           8    plants and the EIA 63A which is solar thermal

           9    panels and then I'll talk a little bit about

          10    this schedule in terms of what is next.

          11              The purpose of the frame evaluation

          12    is basically to look at the coverage and to

          13    do this in a quantitative way.  Also the

          14    surveys from the coal, nuclear, electric and

          15    alternative fuel program office, and again

          16    there are five of them.  And also as part of

          17    the analysis we're going to be looking at

          18    differences between the EIA and the census

          19    frame, as well as the forms that will affect

          20    coverage and the way in which we present the

          21    coverage.

          22              Because you are looking at volume




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             85


           1    and so there may be some differences in the

           2    question wording on the forms that would

           3    impact volumes that are published separately

           4    by EIA and census.  And also looking at

           5    characteristics of establishments that are

           6    missing from our census bureau frame.  There

           7    are two surveys in the coal area, two on the

           8    renewable and one in electric power.

           9              The two coal surveys, the EIA3 is

          10    -- reports consumption of coal by

          11    manufacturing plants and the EIA5 reports

          12    consumption of coal as well as production of

          13    coal, coke by coke plants.  And the renewable

          14    energy surveys include the 63A, which collect

          15    information from manufacturers or photo

          16    thermal collectors and the 63B, which

          17    collects information from manufacturers who

          18    produce photo voltaic cells and modules.  And

          19    the electric power survey, which actually,

          20    they discussed at the previous session, will

          21    include the sub set respondent to a report

          22    for combined heat and power plants and that's




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             86


           1    on the aerial frame of the EIA6.

           2              This displays the 5 frames again,

           3    which are in the left column as well as the

           4    census frames that we will be evaluating them

           5    with.  And it will either be the economic

           6    census and the census of manufacturers or it

           7    will be the current industrial report and

           8    then in each case we are going to be looking

           9    at either an input or an output and today I

          10    will be talking specifically these solar

          11    collectors where we are going to be looking

          12    at the dollar value for value of shipments as

          13    an output to determine coverage by volume.

          14              And again there will be other

          15    variables used depending on the key variables

          16    of the survey and I've also displayed here

          17    the numbers that are on the frame and three

          18    of that we are currently working on are 562A

          19    and 63B, all have a small population around

          20    20 and 21 intake of the coke plant.  The

          21    manufacturing plants, there are eastward 2002

          22    507 and the electric power plants there are




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             87


           1    actually more on the frame than what I've

           2    shown here but we are going to be looking at

           3    the 625 combined heat and power plants and

           4    then also looking at the independent power

           5    producers which are more like an electric

           6    utility but may also be manufacturing.

           7              There are a number of steps and

           8    methodologies.  The first one is to actually

           9    conduct the matching and then that will be

          10    followed by the analysis.  And during the

          11    matching we've identified the establishment

          12    from the EIA frame and also the threshold.

          13    So in some cases we may have a smaller

          14    population than what the census bureau has

          15    because, for example on the electric power

          16    surveys we only have people who are producing

          17    1 megawatt of electricity to report.  Then

          18    those establishments will be matched to the

          19    economic census bureaus business register.

          20              And again we are taking a very

          21    broad stroke here by looking at across the

          22    economic census which would include both




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             88


           1    those establishments whose primary activity

           2    is in the manufacturing sector which makes 31

           3    to 33 as well as those that might be in

           4    another sector.

           5              In the third step we are going to

           6    be looking at detailed product code

           7    information which is actually a 10 digit

           8    product code and that information is only

           9    available for establishments which are in the

          10    manufacturing sector.  So when we look at the

          11    volume of information that will be for

          12    establishments in the manufacturing sector.

          13    And when we ascertain establishments that do

          14    not match and possibly making some

          15    corrections or adjustments.

          16              The findings, we'll summarize the

          17    results, the tabular results for count as

          18    well as volume and I'll be showing you those

          19    tables in a moment.  And we also hope to

          20    summarize some of the differences between the

          21    frames including differences in

          22    classifications of establishments and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             89


           1    boundaries of establishments, and

           2    characteristics of missing establishments for

           3    example in the count table which we hope to

           4    be able to produce -- at the state or

           5    regional level we may be able to identify

           6    establishments that are missing with, in a

           7    specific region or state, the size of an

           8    establishment and the type of establishment.

           9    And the ability to provide this information

          10    will be limited by the census bureaus

          11    restrictions on data disclosure.

          12              I'm going to walk you through a

          13    couple of examples here.  The first one is

          14    matching for the EIA5 frame, which is the

          15    coke plant.  And again we've identified the

          16    coke plant on the EIA frame and matched those

          17    establishments to establishments on the

          18    economic census using the business register.

          19    And then we have identified for those

          20    establishments which produce petroleum coke

          21    and cold coke.

          22              There's two types of cokes there's




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             90


           1    petroleum coke and cold coke and we are only

           2    interested in the cold coke.  And most of

           3    these establishments are either in 1 or 2

           4    six-digit NAICS codes.  They are either in

           5    the 324, which is related to petroleum

           6    industry, but they are not petroleum industry

           7    they are coke plants that produce petroleum

           8    or coke products which is related to the

           9    petroleum industry.  Or they are in the steel

          10    and iron mills which is 331111.  And then

          11    again there are these 10 digit product codes

          12    that are associated with the outputs of cold

          13    coke and --

          14              For the 63 age cover the solar

          15    thermal Collectors we've identified the

          16    establishments and also matched those to the

          17    business register and again they have a 10

          18    digit product code, which is the solar

          19    thermal collector and the research has been

          20    assigned to identify establishments that do

          21    not match and to reconcile those differences.

          22              In addition for the 63A we've also




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             91


           1    looked at producing tables that would display

           2    the results both by count and by volume

           3    coverage and this example provides or show if

           4    you will for what we intend to produce we've

           5    just, -- for each of the states we know the

           6    number of establishments that produce solar

           7    thermal collectors.  We'll identify those

           8    that match on the two frames and then

           9    individually those that do not match on the

          10    economic census separately from those on EIA.

          11              And then perhaps the most important

          12    one is to identify the percent coverage which

          13    will if you'll notice the formula done, it

          14    will be the number that match on the economic

          15    census divided by the number that are in

          16    scope on the economic census.

          17              The next table is a volume, is

          18    intended to display the volume coverage and

          19    again it would be for the 63A and small

          20    tables would be done for the other surveys.

          21    At this level we anticipate that either we'll

          22    be able to produce a national or potentially




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             92


           1    a portal and this would be a dollar amount.

           2    Again we will be displaying what information

           3    we are able to find that matches on both

           4    surveys as well as the differences between

           5    the two and producing a percent of coverage.

           6              The one difference between this

           7    table and the previous one is that I

           8    mentioned that the 10 digit code the census

           9    is, the establishments which are in the

          10    manufacturing sector will be able to give a

          11    percent which will act, may in some cases be

          12    a sub set of those establishments that were

          13    actually able to match because if an

          14    establishment is outside the manufacturing

          15    sector then there would not be 10 digit

          16    product code information available for that

          17    establishment.

          18              The proposed reports done for each

          19    of these surveys will be discussion of the

          20    matching, the tables that indicate the

          21    percent coverage both by count and by volume,

          22    and then identifying similar indifferences




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             93


           1    between the surveys.  And again these

           2    differences could be due to the purpose of

           3    the survey or the frame, for example there

           4    may be differences among how respondents

           5    report on joint ventures.

           6              On an EIA frame for example there

           7    may be actually be 2 respondents for 1

           8    establishment and on the census that may be

           9    done slightly differently.  So there may be

          10    reasons for differences in the counts or in

          11    the way the volume or information is

          12    collected.  And also in terms of the

          13    questionnaire designed, there may be some

          14    reasons for differences in the volumes that

          15    are actually collected and we need to look at

          16    that a little further.

          17              Characteristics of the

          18    establishments missing, again we're going to

          19    look at location and hopefully size and type

          20    of establishment and that will then be

          21    helpful in terms of being able to take some

          22    action to improve EIA frames.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             94


           1              The schedule is that by December of

           2    2004 we hope to have the finalized results

           3    for the three small surveys, the solar panel

           4    and the coke plants that I mentioned earlier

           5    as well as the photo voltaic cells and

           6    modules and by July of 2005 have the final

           7    results for the EIA3 which covers the

           8    manufacturing sector -- manufacturing

           9    establishments that consume coal and the

          10    combined heat and power plants in the

          11    electric power survey.

          12              SPEAKER:  Okay, last up is as we

          13    come around we are going to talk about the

          14    result in surveys and the MECS data.

          15              MR. LORENZ:  Good morning everyone.

          16    I've been working on a project for the past

          17    couple of months now where I've been

          18    reconciling EIA 810 and 820 information 2002

          19    MECS.  I just want to acknowledge a few

          20    people Rick, Vikki, Lucy and Eva for helping

          21    me out with collecting some of this

          22    information.  I also want to thank Preston




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             95


           1    for doing the slides for me this morning.

           2    You know, I just want to give you a quick

           3    background on some -- on the three surveys.

           4    MECS is a quadrennial survey of manufacturing

           5    industries that's based on NAICS and we have

           6    3 survey forms one of which is specifically

           7    designed for the refineries and we have it

           8    setup for the refineries because we don't

           9    want any duplication of the other refining

          10    surveys such as consumption data of the

          11    petroleum based products and for any other

          12    sources, with any other, any other energy

          13    sources we don't collect -- we collect

          14    everything for stock use.

          15              The EIA 10 they track the beginning

          16    through the ending monthly balances of not

          17    only crude oil but also the other petroleum

          18    based products that are made at the refinery.

          19              The EIA 20 is an annual survey that

          20    collects the current and next year

          21    predictions of the refining capacities.  It

          22    also collects previous year purchase fuel




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             96


           1    data, which is helpful for us on the MECS.

           2    Now you might ask why we are doing this.  We

           3    are doing this because since the MECS has

           4    been fielded since 1985 and there have been

           5    differences in the petroleum based fuel

           6    consumption data, primarily in the 810 and

           7    the MECS.

           8              And this year is the first year

           9    that we have had an agreement between the EIA

          10    and census, initially as Grace mentioned it

          11    was for MECS purposes, but we also decided

          12    that it was a good time to look out at the

          13    micro level data to see if we could find out

          14    why there are some of these discrepancies.

          15    Another reason that we are doing this is

          16    because refining is a very important part of

          17    the manufacturing industry, it accounts for

          18    approximately 30% of the total energy use in

          19    the manufacturing sector in the US.

          20              Just I will give a bit about, about

          21    what we did on the project.  Look at on the 2

          22    phases.  First phase was an overall




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             97


           1    assessment where we compared the active

           2    refinery from the MECS to the active and the

           3    inactive refineries in the 810 and the 820

           4    frame.  Phase 2 I mean I call it a

           5    discrepancy resolution.  We contacted

           6    approximately refineries starting with the

           7    larger ones and ending with the smaller ones.

           8    Once we found a discrepancy we would contact

           9    the MECS responded identify the discrepancies

          10    to them and have them deal with the

          11    discrepancy with their EIA 810 or 820 contact

          12    at the refinery internally and then report

          13    back to us what the resolution was.

          14              I'm just going through results very

          15    quickly, in phase 1, 96 refineries out of the

          16    116 matched between the 2 groups so that's

          17    approximately 83% which isn't too bad.  As

          18    you noticed there is about 20 refinery

          19    difference there and in 6 of those they were

          20    in the MECS frame but they were not in the

          21    EIA 810 820 frame.

          22              One of them was, we identified as




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             98


           1    being out of business.  The other 5 were

           2    identified as wax and lubricating oils were

           3    the primary shipments, which is very

           4    important because that identifies a certain

           5    group, that the 810 and 820 might not

           6    necessarily collect for.  Two of those 5

           7    checked the refining and non-refining

           8    operations on our survey.  And in phase 2 as

           9    you can see the first two bullets, the

          10    petroleum based consumption fuels really did

          11    not match up all that well and those are the

          12    fuels that are on the 810 survey.

          13              There's 3 that, there are three

          14    petroleum based fuels that matched up fairly

          15    well after response errors were corrected.

          16    Now those response errors were not only MECS

          17    but also response errors that were identified

          18    as being response errors on the 810 that had

          19    already been corrected since 2002.  And the

          20    purchase fuel consumption data from the 820

          21    matched with the MECS very well so.  Just in

          22    over all there's I think two different




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                             99


           1    difference, first the coverage is things that

           2    aren't necessarily, that aren't that are okay

           3    to have, they are okay to have the coverage

           4    differences with the MECS and the 810 820

           5    frames.

           6              First the kinds of establishments

           7    we collect for, such as those I just

           8    mentioned a few minutes ago.  The waxes and

           9    lubricating oils as being a primary

          10    shipments. I think a lot of it has to do with

          11    the fact that MECS uses the NAICS to fully

          12    establish a base.  NAICS has a very rigid

          13    structure for classifying establishments that

          14    the 810 and the 820 don't have.  Another

          15    coverage difference is that the EIA10 does

          16    not cover some of the on site fuel use,

          17    that's not used in the refining process and

          18    what I'm speaking of is the diesel and motor

          19    gas which is used in front side vehicles,

          20    used in heavy machinery, pumps, compressors

          21    and things of that nature that isn't

          22    necessarily used in the refining process.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            100


           1              The thing that I think we have to

           2    concentrate more on is the data differences,

           3    again the next purchase fuel data seems to be

           4    okay, it seems to match up fairly well with

           5    the 820.  However the petroleum based fuel

           6    consumption data on the MECS it doesn't match

           7    up very well with the 810.  And one of the

           8    things that we have found was that there is

           9    only three refineries that had the same

          10    responded between the MECS and the 810, so I

          11    think there is some inconsistency and some

          12    interpretation when a couple, more than one

          13    person is filling out two different surveys,

          14    so we have to look into that a little bit

          15    more.

          16              Now what can we do about it, the

          17    potential next step, I mean potential because

          18    this all depends upon budgetary provisions

          19    and such and if we can do it to not, we still

          20    have to look into it.  What we'd like is,

          21    what we are looking into it is possibly

          22    drawing the sample earlier so we can verify




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            101


           1    some contact information, possibly sending

           2    out you now like a post card to verify that

           3    information and get better contacts.  Bless

           4    you.

           5              Another thing that we can do is, is

           6    the MECS and the EIA 810 and 820 people can

           7    hopefully work together to improve some of

           8    the in- consistencies of the data.  And what

           9    I mean is we can reference each others

          10    surveys were appropriate so that we can

          11    possibly identify that for example the MECS

          12    can say this should be equivalent to the 810

          13    survey if that's where it's needed.  And

          14    again if you have any suggestions, if we can

          15    do anything differently I'd be happy to hear

          16    it and any questions you might have for me.

          17              MS. SUTHERLAND:  Thank you, sir.

          18    At this time we would appreciate any comments

          19    the committee might like to share about our

          20    committee's activities.  You have any

          21    questions.

          22              DR. HENGARTNER:  I'm glad you are




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            102


           1    doing the dual system estimation.  It's nice

           2    to see because I mean it's a good atrocity,

           3    it's very dark, always assumptions that go

           4    into these things and the independent

           5    assumptions as to how saying yours, at most

           6    at this tenures.

           7              Now what is also true is that very

           8    often when the census dept cant know that,

           9    when you do the real system estimation is

          10    stratified and you do it on the stratified

          11    thing and hopefully the assumptions on the

          12    stratified might be better.  Now the other

          13    thing that I just want to point out is that

          14    I've tried to do the math myself also and

          15    I've got slightly different questions, which

          16    is normal, I tend to make mistakes --

          17              MR. BRADSHER-FREDRICK:  Well it

          18    depends upon the denominator, what one really

          19    wants to be used as well what is the total

          20    thing and I use the total number of

          21    facilities, they I think I just go around and

          22    I have used different numbers than previous




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            103


           1    to go around, so there may be discrepancy.

           2              DR. HENGARTNER:  Those small things

           3    would address -- if I can sit down with you

           4    on a one to one basis so we can go over, it

           5    doesn't make a big difference, but if we are

           6    going to use the methods we might as well try

           7    to be sure that it works.  So these were the

           8    two comments they had on the dual systems.

           9              The other thing that occurred to me

          10    as the last presentation on using the census

          11    having two forms with different answers, that

          12    is always something that I like

          13    inconsistencies because it reveals something,

          14    it reveals something that mainly we can learn

          15    about the errors.

          16              I don't know which one of the two

          17    is right, but if I learn that you know their

          18    inconsistency, people make mistakes that

          19    might inform me on knowing on how much faith

          20    I should have with the individual numbers on

          21    the other forms which I haven't subscripted.

          22    Because I mean both people might have done




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            104


           1    the same mistake as well.

           2              MR. LORENZ:  Well we found that

           3    too, we found that --

           4              DR. HENGARTNER:  So there are ways

           5    you can use that to enlighten yourself about

           6    the error rate in the response, and although

           7    it is a good thing to try to eliminate them

           8    afterwards, having at least a way of

           9    calculating those error rates is interesting.

          10    And it should be taken into account since, in

          11    the quality of the data that this we're

          12    collecting and that is something that

          13    something we don't have the occasion of doing

          14    very often, asking the same data twice and

          15    getting different answers.  So this is

          16    interesting in that sense and those are

          17    things that you can learn from it.  So I --

          18              MR. LORENZ:  It's wonderful.

          19              DR. HENGARTNER:  Okay.

          20              DR. FEDER:  I just want to make one

          21    brief comment on what Nick said that he likes

          22    to see inconsistencies I mean that's what's




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            105


           1    good, I think Barbara probably knows more

           2    about the liability studies, when you repeat

           3    a survey or a testing instrument specifically

           4    to see how consistent the answers would be.

           5    So here you have something, which wasn't

           6    maybe planned to be this way.  But it still

           7    gives you data.  Of course the instruments

           8    are slightly different so one has to try to

           9    understand if this is indeed because of

          10    uncertainty on behalf of the respondent or

          11    may be it's something within the

          12    administration of the question that causes

          13    the discrepancy.

          14              MS. KIRKENDALL:  Just a comment on

          15    the dual system installation.  Everybody

          16    knows the replies that Shawna provided on

          17    what we're going to get in terms of the count

          18    from a comparison -- we should be able to run

          19    through that.

          20              DR. HENGARTNER:  Oh, yes.  I mean

          21    as I said, these are estimates and like any

          22    estimates I mean, it's as good, it's as good




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            106


           1    a guess as I could give you otherwise.

           2              MS. KIRKENDALL:  At least when

           3    we've comparing the census bureau referring

           4    to the EIA frame was pretty independent.  At

           5    least I know that first assumption that could

           6    be violated.

           7              DR. HENGARTNER:  Yes.  It

           8    stratifies then you can even take into

           9    account will show you for example it's one

          10    industry where everybody is missing, like the

          11    lumber industry so that, that's what I

          12    think -- if we know something like the

          13    different spectral.  And that's it.  I'm glad

          14    you're doing it.

          15              MS. KIRKENDALL:  Actually what we

          16    didn't talk so much about I think, as Grace

          17    mentioned it, as we had the EIA team, that is

          18    one person from each office and they are

          19    charged with going through all of the

          20    information that we could find on all of our

          21    frames, which is a very good chore to put

          22    together and also a big chore to go through




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            107


           1    that evaluation.  And they are trying to

           2    figure out you know what can I say about each

           3    frame do they think it looks good and say

           4    it's sufficient or whether it's not

           5    sufficient or you just not know.

           6              We are looking at right now we

           7    don't really know.  You may you may have some

           8    thoughts but you might not choose for the

           9    classification done since you know you're

          10    going to get answers sometime later, early,

          11    more input and this is sometimes very hard to

          12    come by.  I am hopeful for that exercise

          13    because it will be the first time we've ever

          14    taken a big broad look at all of our frames,

          15    and see what they all look like.

          16              DR. HENGARTNER:  The other thing I

          17    was surprised to learn that it's hard to get

          18    information at times, for competitors it's

          19    easy.  I mean it's not what I expected.  Well

          20    I mean I didn't think about it.  Using that

          21    information also to hope to evaluate the

          22    frames if that can be done that's good.  Now




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            108


           1    again I mean I don't know if it's practical

           2    yet.

           3              MS. KIRKENDALL:  Yes, I know it

           4    depends on where we find problems in the

           5    frames.  If we have questions about the

           6    quality of the frame or we have some insight

           7    to what the issues than that might be or not.

           8    That will be something we'll have to look at.

           9              DR. FEDER:  What is NREL? Who is

          10    doing NREL?

          11              MR. BRADSHER-FREDRICK:  This is The

          12    National Renewable Energy Lab and they have

          13    some thing on their website that's all

          14    generators in the country that operate on

          15    renewable fuel and they call them REPIS

          16    database.

          17              DR. FEDER:  Are they government or

          18    what are they?

          19              MR. HSEN:  They are one

          20    organization, the DOE.

          21              MS. KIRKENDALL:  He's not getting

          22    that recorded.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            109


           1              DR. FEDER:  And MECS is done by

           2    census right?

           3              MR. LORENZ:  MECS is done by the

           4    EIA.

           5              MS. KIRKENDALL:  Oh no, MECS is an

           6    EIA survey but it is conducted for us by the

           7    census bureau.

           8              MR. LORENZ:  Okay thank you, that I

           9    forgot.

          10              MS. KIRKENDALL:  So we both are

          11    very interested in MECS and the quality of

          12    MECS.

          13              MS. WAUGH:  And we have two MECS

          14    surveys.

          15              MR. BLAIR:  I just had a question

          16    on the dual system estimation of what

          17    variable or criteria that is used to compare

          18    an item on one frame to an item, an element

          19    on one frame to an element on the other

          20    frame?

          21              MR. BRADSHER-FREDRICK:  Actually it

          22    was done visually.   I did all of it




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            110


           1    personally, it took about 2 months and it was

           2    done in the year 2002.  And you just had to

           3    try to match names.  Naming conventions

           4    weren't the same, I mean I looked at such

           5    things as, when the generator first started

           6    operating, you know some of the variables

           7    that we have and they have in common.  So I

           8    think we were pretty accurate.

           9              MR. BLAIR:  Right.  And since you

          10    did it personally I'm sure it was -- but in

          11    general one think to keep in mind especially

          12    if you start doing this with large numbers

          13    that you can't do personally by hand, is the

          14    possibility of misclassification errors,

          15    because this depends on you know getting

          16    these things in the right cells.

          17              And you know you want to think

          18    about sort of what criterion, what criteria

          19    you are using to match and if there might be

          20    multiple criteria level that will improve

          21    your, you know classification.  And again

          22    whenever we are estimating -- and I think




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            111


           1    that's when you don't want to overload.

           2              MR. BRADSHER-FREDRICK:  Yes,

           3    especially if we automate some of it --

           4              MR. BLAIR:  Exactly.

           5              SPEAKER:  Even if it's automated I

           6    would encourage you to the ones that are not

           7    in common.  In one frame but not the other,

           8    each one just look at them visually.

           9              SPEAKER:  Yes.

          10              SPEAKER:  Because these are the

          11    ones that are going to match in terms of

          12    errors, and sometimes those naming

          13    conventions might change a little bit to

          14    know, if all those things might just show

          15    little differences, that a human looking at

          16    it says Oh No clearly they are distinct.

          17              SPEAKER:  Yes.

          18              SPEAKER:  So the ones that are

          19    classified probably they match, they are

          20    probably correct, they are the ones that I

          21    would look at visually.  And that would speed

          22    up things.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            112


           1              MR. BRADSHER-FREDRICK:  Yes and

           2    there was a visual checking on all of them.

           3              MS. KIRKENDALL:  Well you'll

           4    certainly hear about, I guess we didn't have

           5    too many specific questions for you, this is

           6    mostly an update session.  Probably at the

           7    next meeting we can give you a better summary

           8    of where we are, we should have heard at

           9    least on the small surveys from the census

          10    bureau and we should know from our team what

          11    the over all assessment looks like.  And then

          12    we would be trying to figure out what to do

          13    with it, or we are going to have a good

          14    challenge.

          15              MS. WAUGH:  I just want to ask for

          16    some guidance.  If we come up with an

          17    increased percentage coverage by volume what

          18    would you think would be the position.

          19              MS. KIRKENDALL:  Good question.

          20              MS. FORSYTH:  Well a 100.

          21              MS. WAUGH:  Could we have 20 or 21

          22    establishments and then the other two have




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            113


           1    like 500 or a 1000.  So to the larger thing,

           2    is the difference "percentage" that would be

           3    commissioned from surveys that only have 20.

           4              MR. BLAIR:  On the larger ones I

           5    would go back to the comment that I made in a

           6    different context and that is the

           7    stratification to look, to see not just the

           8    percentage but sort of why we should be more

           9    concerned if you are missing a bunch of

          10    people or in a particular type then you will

          11    sort of with the overall because the nature

          12    of the size or these have been effective

          13    estimates and that's what you are interested

          14    in.

          15              And so you know they are not all

          16    created equal, so I don't know, it's just

          17    simple as saying what's a good percentage and

          18    what's a bad percentage by sort of what's the

          19    being typed on your estimation.

          20              MS. WAUGH:  Okay.

          21              MS. KIRKENDALL:  At one time Tom

          22    said that there are 83% match and that's not




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            114


           1    really good.  But there were some of those

           2    that were really 60% match and those that

           3    were really a question that were not online

           4    and the others were whether they were

           5    inactive or not with this issue.

           6              MR. HOUGH:  I think the main

           7    results of those issues, I don't believe

           8    there's anything for designing surveys.  The

           9    six that we identified were in the realm of

          10    refineries, they were small, so even if you

          11    make an effort to go out and find them, I

          12    don't believe it's going to give that extra

          13    data to the point where I don't know if

          14    there's a benefit to undertaking that type of

          15    exercise.

          16              MR. BROOKS:  As part of the A20 we

          17    don't classify them as refineries, it's

          18    really lubrication plants.  You may want to

          19    add it into your census.

          20              MR. HOUGH:  They didn't get the

          21    definition.

          22              MR. BROOKS:  From the standard




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            115


           1    point of the Department Of Energy, they are

           2    not refineries.  They take what the

           3    refineries produce, which we call as base

           4    stocks and purify them or filter them and

           5    package them.

           6              MR. LORENZ:  Which is one of the

           7    definitions.

           8              MR. BROOKS:  But they don't, they

           9    don't distill crude oil.

          10              MS. KIRKENDALL:  I think what it

          11    means for the EIA is we just, we may want to

          12    be a little more careful of what we say.  I

          13    don't think it means that we want to add them

          14    to our frame but it may mean we want to be

          15    more careful in our verbal description of

          16    what we had discuss.

          17              MR. HOUGH:  We are going through

          18    the same process what Shawn talked about also

          19    and to respond in the same fashion and frame.

          20    The ultimate question that I'm asking is, I

          21    believe it was discussed a second ago, there

          22    are response errors obviously that goes down




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            116


           1    to a 10 digit level of detail.  The first

           2    question that I have to ask when I identified

           3    differences in the two frames is if the

           4    product value that was reported to the census

           5    balanced or is it a response error from a

           6    respondent where they are actually

           7    manufacturing a product that should not be

           8    classified as a solar thermal collector

           9    bridge.

          10              So there is a kind of a

          11    verification going on at first site, I did

          12    exactly what I believed -- I looked at the

          13    differences and I go to the web and I

          14    researched what the company is doing and then

          15    look at what the company is telling us they

          16    are doing and making sure that that remains

          17    accurate because what we found especially in

          18    the reconciliation of the MECS data is any

          19    time you send a question out, it is open to

          20    interpretation from the person who is

          21    responding and a lot of the differences in

          22    what Tom talked about in my belief goes right




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            117


           1    to that interpretation question, and actually

           2    some of the respondents actually told us

           3    that.

           4              They were using, one person used

           5    this method of calculation or in context,

           6    this person used this method of calculation

           7    for content, we need to go talk about it and

           8    figure out which one we should be using to

           9    report on both.  So there are, there is this

          10    verification stage I think we have and you

          11    know all surveys contain some level of

          12    response error and any time you have an

          13    ability to compare data, which should be

          14    comparable.

          15              You need to make sure that you are

          16    not comparing apples and oranges and that you

          17    are comparing apples and apples.  So there

          18    is, the benefit is there in both ways here,

          19    if there has some response errors that was

          20    identified and corrections have been made to

          21    the census database and the results that will

          22    be given back to the EIA will be from a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            118


           1    tabulation that is done after those

           2    corrections are made.  So the processing is

           3    something as we have just suggested that it

           4    is on going please.

           5              SPEAKER:  Looks like we are fairly

           6    early but it sounds like we've run out of

           7    things to talk about.  Summary is supposed to

           8    happen at 11:55, so there is another 15

           9    minutes.

          10              CHAIRMAN BREIDT:  Okay, I guess we

          11    are ready to summarize our breakout sessions

          12    and Barb Forsyth will go first.

          13              MS. FORSYTH:  I guess I'll begin by

          14    summarizing what the session was on, and we

          15    talked about frame evaluation and a series of

          16    frame evaluation and data comparison test

          17    that EIA has been working on, some of it we

          18    talked about in our spring meeting and this

          19    was an update of some of those or a follow-up

          20    to some recommendations that you know of our

          21    spring meeting, so we had re-presentation,

          22    one on a relatively small dual frame




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            119


           1    estimation activity and a second on clearance

           2    or working progress comparing three center

           3    surveys with five EIA surveys in terms of

           4    their coverages and also in terms of the

           5    estimates they produced.

           6              And then a third comparing the MECS

           7    which is a survey of manufacturing industries

           8    with two EIA surveys, which I believe focus

           9    on refineries and all three of the papers we

          10    heard had frame evaluation components and two

          11    of them also had some survey comparison data

          12    component, so both of those, I think speaks

          13    the interest of the folks around the table.

          14    In terms of the dual frame estimation paper

          15    on -- the dual frame estimation suggested

          16    there are relatively few elements missing

          17    from the EIA frame or frame, frames and sub-

          18    frames, I guess that were compared and so

          19    that was promising.

          20              EIA noted that there are some

          21    relatively strong assumptions required for

          22    the dual system estimation and we talked a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            120


           1    little bit about that in our follow-up.  In

           2    terms of the work in progress comparing the

           3    three central surveys and the five EIA

           4    surveys, again that paper is focusing both on

           5    coverage and their plans -- they talk mostly

           6    about their plans, are to produce assessments

           7    and coverage both in terms of counts of

           8    missing elements and also in terms of the

           9    volumes of, I guess, products represented

          10    volumes of shipments, represented by those

          11    missing elements, so that's a nice double

          12    look at coverage.

          13              And then the third paper look at

          14    coverage and got a pretty high percent of

          15    matches across the MECS with the two EIA

          16    refinery surveys and then they were able to

          17    look in more detail at some data differences

          18    between the two surveys and some differences

          19    between the elements in one set of frames and

          20    not in the other, so that they could draw

          21    some conclusions about possible definitions

          22    or other reasons for the lack of coverage.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            121


           1    So, that I think, those are the papers we

           2    talked about and then the questions to the

           3    committee were in general comments.  And,

           4    yeah, we have some general comments.

           5              First of all there were

           6    enthusiastic reception for the dual frame

           7    assessment in particular but I think for the

           8    task of evaluating frames in general, I think

           9    the committee generally was enthusiastic

          10    about all three of these activities.  And we

          11    noted that, especially for the dual frame

          12    estimation, some of the assumptions maybe

          13    tenuous but there are ways of working around

          14    some of those potential risks to the

          15    estimation, for example by stratifying and

          16    looking at coverage issues within strata in

          17    the surveys and that affords several

          18    advantages.

          19              First, you can determine whether

          20    there are systematic holes in any of the

          21    frames and those of course are important and

          22    under similar point the magnitude of coverage




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            122


           1    error maybe less important than the

           2    systemeticity of coverage error and

           3    stratification can help you assess the latter

           4    systemeticity of it and maybe give you a way

           5    of interpreting the magnitude of coverage

           6    error they would get.

           7              So, that the assumptions are

           8    tenuous, doesn't really worry us, we can

           9    stratify and begin to address it, it worries

          10    us of course but there are tools for getting

          11    around it and those tools will, will get you

          12    other benefits as well.  We also noted that

          13    there is a real benefit to comparing a range

          14    of frames and so the three papers that we

          15    talked about looked at frames as very

          16    different kinds of different kinds of

          17    relationship among themselves and looking at

          18    that range of comparison gives you a sense

          19    for how the frames fit in together and can

          20    give you a tool for looking at, perhaps

          21    logical reasons why there are missing

          22    elements from some frames, so in particular




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            123


           1    the census frame, for example are relatively

           2    independent of some of the EIA frames and so

           3    they provide a way of perhaps interpreting

           4    results you find with other potentially more

           5    correlated frame.

           6              We raised the question or we talked

           7    about matching criteria, it's an important

           8    feature of frame evaluation and when the

           9    frame evaluation is relatively small,

          10    humanized and do some of their matching and

          11    we thought there was a real benefit to having

          12    experts involved in the matching process.  As

          13    the process -- it requires more automation,

          14    it's important to introduce a lot of matching

          15    criteria, that way you can reduce the

          16    likelihood of failure to match when there

          17    should be a correct match.  You can avoid

          18    misclassification here, is another word.

          19              So, it's important as you automate

          20    the processes to make sure you have a lot of

          21    matching criteria, so that was one point

          22    made.  And when you're matching automatically




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            124


           1    or when you automated the matching process,

           2    it's particularly important to examine those

           3    places where you get mismatches and look at

           4    those -- if you can involve an expert, that

           5    is one place where an expert is likely to be

           6    most useful because changes in convention,

           7    for example in coding conventions or naming

           8    conventions may produce mismatches --

           9    automated mismatches that in fact an expert

          10    will be able to determine are not real

          11    mismatches but they are in fact

          12    classification errors.

          13              Let's see -- so I think -- I hope

          14    that captures most of what we said about the

          15    frame, I was scribbling pretty frantically.

          16    I think there was also -- this is kind of

          17    interesting, there was also enthusiastic

          18    reception for the inconsistent - -

          19    inconsistencies in data found across surveys,

          20    so we did probe to see some inconsistencies

          21    in data and I think the reason is an

          22    important one that that inconsistencies can




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            125


           1    tell you a lot about your data and I think

           2    that's the point that at least some of us

           3    made pretty strongly that inconsistencies

           4    give you way of evaluating the quality of

           5    your data and for tracking down reasons for

           6    inconsistencies that are potentially

           7    actionable since you might be able to do

           8    something about.

           9              So if there are data that -- if

          10    there are definitions, elements are used in

          11    different surveys or in the way they're

          12    interpreted by different respondents to

          13    different surveys, those are the things that

          14    you can act on once you've identified them,

          15    so the search for inconsistencies across

          16    surveys that should yield similar results, it

          17    seems to offer(?) -- is a fruitful activity.

          18    So, I don't think there was a whole lot of

          19    negative votes although I may have missed

          20    something but I think the committee was

          21    pretty enthusiastic about all of the work

          22    that we were giving.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            126


           1              CHAIRMAN BREIDT:  Okay, any quick

           2    questions or comments?

           3              MS. FORSYTH:  About addition?

           4              CHARIMAN BREIDT:  Okay, let's turn

           5    it over to Mark to summarize the second

           6    phase.

           7              DR. BURTON:  Okay, we were in a

           8    session where we were presented with the --

           9    about the description then and some

          10    preliminary results of the multi-regional

          11    electricity equilibrium model that Phil and

          12    Dave are in the process of developing.  This

          13    is an extraordinarily ambitious project, the

          14    current models are national in scope and

          15    don't really allow us to answer a number of

          16    other important questions.

          17              The models that they were in the

          18    process of developing, if is in fact

          19    successful will add materially to the ability

          20    to evaluate things like summer gasoline

          21    prices within specific regions or whether

          22    fuel prices within specific regions, so that




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            127


           1    it really can't add a great deal to our

           2    ability to anticipate what's coming,

           3    essentially there is -- it's an equilibrium

           4    model with a supply and demand sector, the

           5    demand side of the process is estimated on a

           6    monthly basis using some time series

           7    econometrics techniques that incorporate both

           8    economic impacts or economic drivers as well

           9    as time series components that they capture a

          10    sort of technical nature of electricity

          11    demand, it's done for 13 separate regions.

          12    We spent a fair amount time talking about why

          13    those regional definitions are as they are.

          14              The monthly -- the econometric

          15    monthly estimates are then allocated to daily

          16    and then even hourly demand estimates within

          17    each regions, so that what they're trying to

          18    do is track the demand for electricity and

          19    then the corresponding use over the course of

          20    each hour of each day in order to look for a

          21    variety of different impacts.

          22              So, the supply side of the model is




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            128


           1    developed based on both base load and

           2    available peak load generation, there are

           3    certain facilities that they have generated

           4    power continuously and then the electricity

           5    -- the quantity of electricity available has

           6    increased as necessary based on the costs

           7    dispatching other facilities and in that way

           8    it captures the relative fuel prices because

           9    again a facility is dispatched based on it's

          10    cost and so that if natural gas is

          11    particularly expensive right now, our gas

          12    power plant would be more slowly dispatched

          13    and plant fueled a different way.

          14              It does provide an equilibrium for

          15    each region, for each part of the day and it

          16    allows the ability to engage in a number of

          17    simulations.  Now, the first thing that they

          18    described was a simulation without any sort

          19    of transmission constraint where they simply

          20    allow for electricity to flow from one region

          21    to another based on the cost of generating

          22    that power and the demand within each region.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            129


           1              And that then serves as kind of a

           2    baseline result -- against which more complex

           3    simulations can be compared and by more

           4    complex simulations, I prefer to adding in

           5    transmission constraints that can at least

           6    constraint the amount of power that's traded

           7    between one region and another, so that we

           8    can -- to the extent that we can identify

           9    these transmission constraints and we can

          10    also then quantify the impacts that they are

          11    having on a variety of different outputs or

          12    outcomes.

          13              This model is extraordinarily rich

          14    in what it may allow to happen, we can look

          15    at the equilibrium power supplies and the

          16    quantity of electricity that's being produced

          17    and is consumed within the overall nation or

          18    within the specific regions, we can identify

          19    imbalances, we can simulate the effects of

          20    changing fuel prices -- we can make the

          21    summer warmer or cooler and see how things

          22    change.  It allows the prediction of imports




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            130


           1    and exports within regions of electricity.

           2    If complete and functional, this will be an

           3    extraordinarily powerful tool.

           4              Now, I do this with modeling, not

           5    in this context and I know how challenging

           6    this is and so it's not surprising that the

           7    committee had some question and some things

           8    that we wanted more information about.  There

           9    were some question about how it is that they

          10    actually measure and then implement the

          11    transmission constraints.  I think that

          12    that's an area where we all have to admit it

          13    right now that there is a great deal of

          14    uncertainty.

          15              There was some concern that prices

          16    fall represented on a monthly basis within

          17    the econometric estimation are included or

          18    not, they sort of didn't survive the process

          19    any further in the daily or hourly demand

          20    representation.  There were questions about

          21    whether or not there had been any work to

          22    measure the variability of the outputs, well,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            131


           1    and that sort of gets it at, I think one of

           2    the most basic issues that we don't really

           3    know at this point how well this model

           4    performs and moreover it's sort of hard to

           5    have the data to make the comparisons to to

           6    assess that model performance and that kind

           7    of leads us to -- to one of the things that

           8    actually I think arose -- one of the issues

           9    that arose on the elevator --

          10              Part of the problem is that some of

          11    the data necessary to make really good

          12    assessments is proprietary data that's not

          13    immediately available and the question was

          14    raised whether or not if given the potential

          15    usefulness of this model, whether or not it

          16    might be worth expenditure of EIA resources

          17    to go out and actually collect that data

          18    directly, if that in fact is doable itself, I

          19    will have time between the committee and

          20    lunch, so I will stop, if there are other

          21    people that were -- that were downstairs and

          22    have things to add, I'm sure I didn't capture




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            132


           1    everything.

           2              DR. FEDER:  I don't like to be

           3    between the committee and lunch but I would

           4    like to add -- started by saying that the

           5    model at the baseline is used to evaluate

           6    transmission constraints and then it added --

           7    what's so special about the transmission

           8    constraints that make it done this way.  It's

           9    like very hard to model transmission

          10    constraints.

          11              DR. BURTON:  I'm going to let

          12    either Phil or Dave -- if he's around to

          13    answer that, it's --

          14              MR. TSENG:  I think this importance

          15    of transmission constraint, actually it's

          16    what reflected in the end use electricity

          17    prices, so if a region cannot get cheap

          18    electricity from other regions, then it would

          19    have to be generated within the region like

          20    in Florida, for example, if we cannot cheap

          21    electricity from other regions, then the

          22    price of electricity would be higher.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            133


           1              DR. FEDER:  My question was about

           2    modeling difficulties not transmission, I

           3    mean why not just put everything in the model

           4    including the transmission constraints.

           5              MR. TSENG:  Okay, part of the

           6    reason is, if we don't solve everything

           7    computationally -- that  issue and in terms

           8    of how do you know a region is experiencing

           9    transmission constraints and so we need to

          10    actually look at the supply curve within each

          11    region and then look at the cost of structure

          12    and then see how they would be dispatched and

          13    then compare that with the transmission

          14    constraint, see if it's binding.

          15              DR. BURTON:  Well, I think that by

          16    doing the unconstrained and constrained gives

          17    you some very rough estimate of the potential

          18    benefits of additional investments in

          19    transmission facilities.

          20              DR. FEDER:  Good point.

          21              CHAIRMAN BREIDT:  Okay, so I think

          22    now it would be good time to invite --




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            134


           1              DR. FEDER:  We were talking.

           2              CHAIRMAN BREIDT:  You're still

           3    talking, I thought you're finished, I thought

           4    I jumped in there, so is there any further

           5    public comment.

           6              CHAIRMAN BREIDT:  Okay, so we're

           7    adjourned for lunch and note that from the

           8    agenda that when we come back, we come

           9    directly back to the breakout sessions.

          10                   (Recess)

          11              MS. KIRKENDALL:  Well, I think we

          12    should get started.  We're five minutes late

          13    and we've got a number of people to talk, so

          14    why don't I introduce Preston McDowney.  This

          15    session is about our new natural gas

          16    production survey.  It's supposed to -- it's

          17    been approved by O&B, it's supposed to begin

          18    in January and Preston.  This is sort of a

          19    history and where we are and survey and

          20    methodology.

          21              MR. McDOWNEY:  Good afternoon, I'm

          22    Preston McDowney of Statistics and Methods




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            135


           1    group and today I'm presenting the simulation

           2    results of the probability, sampling for

           3    EIA's monthly natural gas production survey.

           4    I'm going to start the presentation by

           5    recounting Inderjit Kundra's spring 2004

           6    presentation today as a committee, I will

           7    follow that with the simulation, the steps

           8    with the simulation project and then results

           9    of the simulation and then I will introduce

          10    KMB.

          11              At this time I'd like to drive the

          12    committee's attention to a hand out that has

          13    been placed in front of them, this hand out

          14    has supplemental slides to my presentation,

          15    more information about some of the slides at

          16    the end of presentation.  At the last spring

          17    meeting -- and to introduce the committee to

          18    the 9/14 event, the purpose, if you like,

          19    monthly production of natural gas in The

          20    United States.  We proposed a part budget,

          21    pushed aside sample, the goals were to

          22    estimate the total US production of the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            136


           1    co-efficient and variation of the 1% and

           2    regional production of CV as 5%.

           3              The regions that we were looking at

           4    were taxes in the federal gulf, Louisiana, in

           5    Mexico, Oklahoma, Wyoming, and other includes

           6    all of the other states, excluding Alaska.

           7    Presume optimal allocation formula, we used

           8    this formula to -- like I have the samples

           9    itself.  Using that formula we calculated

          10    that we needed 176 operators -- natural gas

          11    operators to get the 1% CV at the national

          12    level.  There were 176 operators, 95

          13    particular, operators produced 82% of the

          14    total natural gas in United States.

          15              However to get the 5% CV in other

          16    regions, you have to go up to 358 operators,

          17    211 of those operators being within a region

          18    comprised of a lot of companies that were

          19    included 17, in one region and another region

          20    and the account promoting state reported 70

          21    groups, it reduced survey by about 109

          22    operators.  And we are extremely happy with




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            137


           1    this sample survey size, we actually -- I'll

           2    go back -- it actually increased the CV in

           3    federal gulf to 1% CV and hand out -- there's

           4    -- calculation for how many operators it

           5    takes to go from 1% to -- from the 5% to 1%

           6    in each region and we had some operators who

           7    actually went ahead and did that and the ASA

           8    committee didn't have a problem with the

           9    approach and so we decided to test the

          10    samples out in a simple method, to

          11    simulation.

          12              Before we just stimulate we have to

          13    cerate a pseudo frame and I will talk more

          14    about the pseudo frame in the status to

          15    follow but after that frame was selected we

          16    split certain group for each region

          17    individually and then most of the operators

          18    were -- remained the same throughout the

          19    simulation process.  After the group was

          20    selected we start to use PPS sampling to

          21    sample the remaining operators that were

          22    left.  After we started 70% of PPS sample




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            138


           1    operators, the estimates -- the total natural

           2    gas production of the pseudo frame using

           3    variations of four different estimation

           4    procedures and I'll talk about those

           5    estimation procedures later.  The pseudo

           6    frame consisted of operators that we had from

           7    the year 2000 EIA-Sample and on the 2002

           8    EIA-23 frame and EIA-23 is the annual natural

           9    gas survey.

          10              And this was done to adjust, not

          11    having a complete frame for the year 2000 and

          12    the two year difference reflects the

          13    approximate time it takes to prepare a

          14    complete frame for a given year, so we go to

          15    select sample for the 2005 survey -- the 2004

          16    data won't be available yet to at least 2003

          17    data to select the sample from.  The four

          18    estimation procedures, the first was the PPS

          19    size estimator and this is used as the

          20    inverse of the kind of ---- selected as the

          21    weight and to make estimations.

          22              Next, regression estimators are




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            139


           1    provided least square estimator, which is the

           2    classic ratio estimator and there's more

           3    information about each of the estimation

           4    procedures in the supplemental slides.  For

           5    the weighted least squares and the ordinary

           6    least squares estimators, we have two methods

           7    to make an estimation.  The first used only

           8    the non certain companies in estimation

           9    procedure, the second method used both the

          10    certain and non certain companies in the

          11    estimation procedures.

          12              So both the big guys and the middle

          13    guys were used to estimate for the non sample

          14    population.  The both methods we tried to do

          15    it from variations, the first variation is

          16    all of the operators in the given method, the

          17    second variation removes out layers in

          18    estimation procedure and use internal --

          19    utilize residuals to determine out layers,

          20    the third method, remove out layers and

          21    overly influential variables and use (?)DFS

          22    to determine the influence of variables




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            140


           1    observation and information on out layer

           2    removal and agent's observations are in

           3    subsequent slides.  Also the simulation

           4    themselves use the program called ARIS(?)to

           5    simulate the main samples and also to

           6    calculate summary statistics across all of

           7    the simulation.

           8              We're going to present on example,

           9    there was 100,000 samples were simulated.  We

          10    started with smaller samples of a 100 and a

          11    1000 but as you prepare one sample -- one

          12    simulation of a 100 samples to another

          13    simulation of 100 samples, we got slightly

          14    different results for the number of samples

          15    to get an equilibrium where you get

          16    consistent answers with consistent summaries.

          17              Here's just a sample of what the

          18    output of the simulations are.  We're looking

          19    at the percent of the estimated total of the

          20    various estimated procedures, all of these

          21    you know are using -- all of the operators

          22    were not removing any out layers or




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            141


           1    influential variables, the PPS estimators

           2    assessed gave us the lowest rule average, the

           3    lowest, 1% error were given, very large range

           4    of errors with 90% of the errors being

           5    between negative 8-1/2 and almost 4%.  The

           6    regression estimators have a much tighter

           7    error band but all of the estimators give us

           8    a negative bias, the estimators themselves

           9    gives us the desired CV of 1% that we're

          10    looking for in the beginning of the

          11    simulation project.

          12              Everything about this is due to the

          13    dynamic natural gas industry itself.  And in

          14    conclusion that the PPS certainly didn't give

          15    us the desired accuracy that we're looking

          16    for.  We didn't get the CV's that we were

          17    looking for and the estimation procedures

          18    with negative biases and in the second lowest

          19    bias there's summary statistics for, we take

          20    the out layers out and we take the

          21    influential variables out, I didn't put it in

          22    the presentation, that doesn't change the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            142


           1    results that much at all.

           2              This did get the same overall bias

           3    and not the CV's, which we are looking for.

           4    Because of all this we decided to use a cut

           5    off sample, almost 90% coverage at the US

           6    level and if there's no burning questions,

           7    I'm going to concede the remaining of my time

           8    to the distinguished gentleman from Texas to

           9    talk about the 90% cut off sample.

          10              MR. WOOD:  Are there any burning

          11    questions?

          12              DR. SITTER:  I have a couple.  In

          13    your -- I just want to clarify couple of

          14    things if you don't mind.

          15              DR. SITTER:  Sorry, you used a

          16    bunch of different estimators when you did

          17    your beta hats, what happened to the unequal

          18    probability of selection sampling or did you

          19    use different simulations to them?

          20              MR. McDOWNEY:  We used a different

          21    simulation, we do other lot of things to this

          22    -- we stimulated a lot of different things,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            143


           1    so couldn't present everything that we did,

           2    but nothing give us the actual.

           3              DR. SITTER:  So how did you select

           4    your samples when you were using beta hats

           5    here as the regression features?

           6              MR. McDOWNEY:  The sample session,

           7    it sounds to me the same because each set --

           8    for each sample we used the different beta

           9    hats to --

          10              DR. SITTER:  All right.

          11              MR. McDOWNEY:  Estimate.

          12              DR. SITTER:  So that means for each

          13    sample you selected with unequal probability,

          14    PPS?

          15              MR. McDOWNEY:  Yes.  For each

          16    sample.

          17              DR. SITTER:  And what happened to

          18    the weights, the unequal probability

          19    selection in your beta hats?  Were they in

          20    there?

          21              MR. McDOWNEY:  Yes they were in

          22    there.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            144


           1              MS. KIRKENDALL:  No, the weights

           2    are not in there the company data were used

           3    only, but the probability of selection was

           4    not used when computer failed.

           5              DR. SITTER:  Okay.  So these beta

           6    hats are not correct.

           7              MR. McDOWNEY:  It wasn't.

           8              DR. SITTER:  They are not unbiased,

           9    so don't expect them to be unbiased.  So if

          10    you're going to use an unequal probability of

          11    selection PPS and you want to use a

          12    regression estimator, you need fill the

          13    weights in to your Beta hat.  Which can be

          14    done with the -- you could've used the ratio

          15    estimator for example which is one of the

          16    beta hats you used, but you need the -- you

          17    need the 1/pi(i)'s and the numerator and

          18    denominator for that to be a proper

          19    estimator, approximate them by an estimator.

          20              MR. McDOWNEY:  Okay, your second

          21    question?

          22              DR. SITTER:  Second question is




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            145


           1    related to -- so this gets worse when you

           2    talk about your out layer and influential

           3    stuff, because now your actually believing a

           4    model where the beta hats is actually not the

           5    rating anymore, so I don't think those over

           6    influential things are going to be right

           7    either.  And then when you did and one other

           8    question related to that maximum you got for

           9    the PPS sample.

          10              MR. McDOWNEY:  Uh-huh

          11              DR. SITTER:  You use the Horowitc

          12    Thompson estimator, that means you basically

          13    use this, your estimator the sum of the

          14    y(i)'s/pi(i) or pi(i) is the inclusions of

          15    the ----

          16              MR. McDOWNEY:  Yes.  Something

          17    similar to that.

          18              DR. SITTER:  Something similar to

          19    that.  So, that can have problems, it's

          20    called Famous Bassus elephant example.  You

          21    probably should instead use a ratio estimator

          22    and that maximum would probably disappear,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            146


           1    that is, you take the sum of the

           2    y(i)'s/pi(i)'s you divide by the sum of the

           3    1/pi(i)'s and then you multiply by N, it's a

           4    ratio estimator because the kind of bottom is

           5    actually an estimate of N and so -- I know it

           6    sounds strange, and a lot of people don't

           7    realize, but there are situations where

           8    estimating a known value is actually better.

           9              DR. FEDER:  It's one way of

          10    calibration.

          11              DR. SITTER:  Correct, So this is

          12    only going to happen in a few cases where you

          13    get a bad sample, but you're doing 10,000

          14    simulations, you are almost guaranteed to get

          15    a few and probably those maximums are

          16    therefore not really a problem, in other

          17    words they can be fixed.  And I think that

          18    you make at your some of your CV's, now, of

          19    course your sample sides calculation were

          20    based on SRS.  You're not doing SRS, so you

          21    can't really use those of the CV's, so I just

          22    want to make sure what's right before you




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            147


           1    looked on.  I have one other question that

           2    it's slipped my mind -- I'm trying to

           3    recapture.

           4              DR HENGARTNER:  Just a

           5    clarification when you calculated those

           6    things that are the aggression, you did it

           7    only -- you excluded the certain percent,

           8    correct?

           9              MR. McDOWNEY:  In one method, yes.

          10              MS. KIRKENDALL:  In one method --

          11              DR. SITTER:  That's the right

          12    one(?) we did.

          13              MR. McDOWNEY:  That's how I did.

          14    The first way first.

          15              DR. HENGARTNER:  So it is, certain

          16    example needs to be excluded and -- because

          17    you exclude the certainty sample you see the

          18    actual CVs larger on the estimated project,

          19    but then when you added it again --

          20              DR. SITTER:  Then you get the gain.

          21              DR. HENGARTNER:  Then you get the

          22    gain.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            148


           1              DR. SITTER:  Yes.

           2              DR. HENGARTNER:  Right -- Okay.

           3              DR. SITTER:  But one other

           4    question, you said you used the random

           5    generator to generate samples, was this

           6    without replacement, PPS sample you know with

           7    the with PPS sample.

           8              MR. McDOWNEY:  Essentially without

           9    replacement with -- you get region

          10    individually you know and then we made sure

          11    what's the result expected on one region, we

          12    removed it from being ---- in the rest of the

          13    regions, and then we take their collection

          14    especially almost certainly with that.

          15              DR. SITTER:  So how did you

          16    calculate the selection probabilities?

          17              MR. McDOWNEY:  That question I

          18    understand, how do I fit.

          19              DR. SITTER:  Well --

          20              MR. McDOWNEY:  You're trying to

          21    place it to each region individually.

          22              DR. SITTER:  Now- -- but what I




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            149


           1    mean is that -- if it's PPS without

           2    replacement sample that's not trivial to do.

           3    And when I say that it's not the number you

           4    divide by is not the PPS anymore.  It can't

           5    be doubled.

           6              MR. McDOWNEY:  Yes.

           7              DR. SITTER:  So did you, were you

           8    able to --

           9              MR. McDOWNEY:  Yeah we adjusted for

          10    the sample itself changing.

          11              DR. SITTER:  Oh, you did so?

          12              MR. McDOWNEY:  Yes.

          13              DR. SITTER:  How did you do that,

          14    there's many methods -- different methods of

          15    security?

          16              MR. McDOWNEY:  I essentially

          17    physically did it and that, I was modeling it

          18    in Excel, it's all in the Excel program.

          19              DR. SITTER:  Okay, maybe we can

          20    talk offline.

          21              MR. McDOWNEY:  Offline okay.

          22              DR. SITTER:  Okay, thanks.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            150


           1              MR. McDOWNEY:  Any other questions?

           2              DR. FEDER:  I just want to pick up

           3    on what Randy said about the Hycar(?)

           4    estimator when you divide by sum of The

           5    English probability of inclusion which by

           6    itself is an estimate of the population size

           7    and that's one way of calibration, when you

           8    could just caliber to the population size but

           9    if you have other information, you can

          10    calibrate on this, it will improve the

          11    procedures.  And I Randy's comment was about

          12    difficulty in calculating the pi(i) assuming

          13    that --

          14              DR. SITTER:  No, I was asking how,

          15    right?  I mean, I did not understand about --

          16              DR. FEDER:  I'm not sure what's the

          17    method the PPS use, but if you use, I don't

          18    know -- one of this, I think it should stop

          19    there.  Now has building some of those

          20    procedures.  I'm just -- I was curious as to

          21    what they did.

          22              MR. KUNDRA:  Oh, I think --




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            151


           1              DR. SITTER:  Systematically.

           2              DR. FEDER:  So you get the correct

           3    inclusion probability to --

           4              DR. SITTER:  Right, that's fine,

           5    thanks.

           6              DR. FEDER:  You can also, from the

           7    simulation check that you get right inclusive

           8    and --

           9              MR. McDOWNEY:  Any further

          10    questions?  Okay, John.

          11              MR. WOOD:  So after doing all that

          12    we decided to use a 90% cut off and SMG

          13    recommended it to reserves on production

          14    data, we thought it is a hell of a good idea

          15    because the difficulty of maintaining and

          16    understanding the -- how the frame that you

          17    draw all these sample for portable size which

          18    in fact our original recommendation is, you

          19    know, just a tremendous burden especially as

          20    I think our catch phrase now is the Natural

          21    Gas industry is very dynamic and the

          22    relationship of the large operators to the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            152


           1    small operators is always in flux.

           2              One of the first line, if you

           3    noticed that we in our title you know our

           4    plan is to use a 90% cut off, sample as the

           5    primary data said that draw of the study.

           6    Now, I want to repeat started off with

           7    objectives that we needed to improve the

           8    quality and timeliness of the EIA's natural

           9    gas production estimates, and in fact we

          10    wanted to have releasable numbers, 60 days

          11    after the end of a production month, and we

          12    wanted those to be within 1% of the national

          13    level and from 1 to 5% in the six specific

          14    areas Texas, federal gulf of Mexico,

          15    Oklahoma, Wyoming, New Mexico and Louisiana

          16    and those that are rank border and 10%of the

          17    US over 48 production.

          18              Texas is the largest producing

          19    state with about 30% of the US total.  And

          20    next slide, now --

          21              MR. WOOD:  I think that I would

          22    like to entertain questions and even maybe




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            153


           1    answer them during the talk, because some of

           2    this is relatively complex and complicated

           3    and it's rather difficult to layout.

           4    Basically how we intend to make the estimates

           5    in a 15-minute period and when we practice

           6    this with the SMG, Nancy had some questions

           7    starting about the 3rd slide that she kind of

           8    wanted answered then.  And I think it's a

           9    fairly good way to do it.

          10              But in particular we're going to

          11    show the data and show it behaves and show

          12    how we expect to make the actual estimates

          13    from the sample and we going to know what you

          14    think about it.  In particular, the

          15    production distribution by operators is an

          16    extremely skewed distribution with the top 5

          17    operators having 29 percent of the US

          18    production and it's similar in each area.  So

          19    the impact of the large operators or overtly

          20    influential operators or however we chose to

          21    identify them is a very important subject.

          22    The, again in Texas also the, next slide,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            154


           1    what's there -- well we can hardly see it

           2    here okay.  That was a test.  In any case the

           3    top 5 operators in Texas have about 25% of

           4    the total production in Texas.

           5              Then, in fact then, if we have non-

           6    response from these large operators or

           7    apparent errors from these operators, you

           8    know, there are several ways to handle it and

           9    we certainly like to have your opinions on

          10    how to do so.  Or there are some algorithms

          11    that we've tested that identify overtly

          12    influential operators and how should we deal

          13    with these when making estimates.

          14              And one of the ways is that if we

          15    do not have to use the same group of the

          16    sample operators to estimate the non- sample

          17    production, in each month.  And now we'll go

          18    over that several times as we go through the

          19    presentation.  Well, the next slide.  Now

          20    this is from our original file where we just

          21    took the 300 top operators from 2002.  And

          22    again the top 5 operators in the lower 48 had




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            155


           1    29% of the production.  And the top 14

           2    operators had 51 percent, the top 53 -- and

           3    this is out of 15,000 or so operators, the

           4    top 53 had 75% and 223 operators in this

           5    particular sample had 9% of the production.

           6    And so the other 15000 had 10 percent.

           7              Next slide.  Now this is about what

           8    we intend to do with respect to sample

           9    selection.  We have built another frame,

          10    which is essentially the same operators and

          11    essentially the same rank order, but it is

          12    built from the thing we're actually going to

          13    survey, the gross production that the

          14    operators produce.  And we've built these

          15    from commercially available data sets and

          16    therefore we set up a history of the

          17    production of every operator, by month, from

          18    1997 through 2003 and into 2004.

          19              And you get rather similar results

          20    if you rank these operators by production

          21    size and you pick the first 222, you've got

          22    90% of the lower 48 production.  If you then




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            156


           1    go and look at the individual areas you've

           2    got 98% coverage in the Federal offshore Gulf

           3    of Mexico, one of the critical areas to

           4    monitor.  You've got 93% coverage in New

           5    Mexico and you have 97% coverage in Wyoming.

           6    And that seems like adequate coverage.  We

           7    then went to Texas and for the rest of the

           8    operators holding a cut off at 222, we ranked

           9    them by size of production in Texas and you

          10    had what, 32 operators and you have 90%

          11    coverage in Texas.

          12              Then we ranked the Oklahoma

          13    production and you have to add 36 operators

          14    and you have 90% coverage in Oklahoma.  And

          15    then ranking the Louisiana operators that

          16    haven't already been selected, we ended up

          17    adding 4 and we have 90% coverage in

          18    Louisiana.  And then if you want 90 percent

          19    coverage of the other states, excluding

          20    Alaska, you have to go all the way up to 337

          21    and we don't think that's probably worth the

          22    respondent burden or the processing load.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            157


           1    And so if we do what we plan, we will pick a

           2    sample of approximately 284 operators.  And

           3    that gives 90% lower 48 coverage and over 90%

           4    coverage in the 6 areas that we wanted to

           5    specifically monitor.

           6              DR. HENGARTNER:  John, may I ask a

           7    question?

           8              MR. WOOD:  Yes.

           9              DR. HENGARTNER:  Can you flashback

          10    the -- I just need to understand.  So you

          11    take the production of every producer, you

          12    order them, then you take the sum, correct.

          13              MR. WOOD:  Right.  Rank order by

          14    their average annual production during a full

          15    calendar year.

          16              DR. HENGARTNER:  Okay.  So this is

          17    going to give you a function of that

          18    cumulative sum.

          19              MR. WOOD:  Uh-huh.

          20              DR. HENGARTNER:  Okay.  Now I have

          21    some misgivings.  So between 222 and 254 I

          22    had about 32, right.  Between 254 and 280,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            158


           1    that's about the same thing, you get 34 more

           2    or less, same distance.

           3              MR. WOOD:  Uh-huh.

           4              DR. HENGARTNER:  Then I would

           5    expect the difference, let's take Oklahoma,

           6    the 8317 goes up to 8325.  I would have

           7    expected that to be larger that difference

           8    than between 90 and 83, right.  Because if

           9    your rank order --

          10              MS. KIRKENDALL:  Yes but they rank,

          11    in that one they are using the rank in

          12    Louisiana.

          13              DR. HENGARTNER:  For Oklahoma, or

          14    --

          15              MR. WOOD:  No, in Oklahoma we

          16    started here, and you add, you stop and you

          17    take the 254 operators.  Many of which, in

          18    fact most of which operate in every one of

          19    these areas.  And then you start with the one

          20    that would have, you know, the next operator

          21    that hasn't already been selected and you

          22    rank them by the size of their production in




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            159


           1    Oklahoma only --

           2              DR. HENGARTNER:  In Oklahoma only,

           3    is that --

           4              MR. WOOD:  And then you added 36 to

           5    get from 83% coverage to 90%.

           6              DR. HENGARTNER:  But what I'm

           7    confused about is if I take the 222 and add

           8    32 to get 254 in Oklahoma, those -- these are

           9    rank orders in Oklahoma.

          10              MS. KIRKENDALL:  No, no going --

          11              MR. WOOD:  But most of the large

          12    operators in Oklahoma have already been

          13    selected.

          14              DR. HENGARTNER:  Okay, then these

          15    are nationally ranked.

          16              MS. KIRKENDALL:  Well no, you start

          17    nationally rank and you do a 90% sample.

          18    Then you go to the next area you're

          19    interested in, you rank that area, and you

          20    select the companies that give you the

          21    percentage in that area.  They also

          22    contribute to other areas but you pick them




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            160


           1    based on that area.  So then you go to

           2    another area and you rank the companies that

           3    contribute to that area that you picked in.

           4              DR. FEDER:  Because they operate in

           5    more than one region, right.

           6              MS. KIRKENDALL:  They operate in

           7    more than one region.

           8              MR. WOOD:  I'll tell that.  The

           9    largest operators in Oklahoma are selected in

          10    that first 222.

          11              MR. BLAIR:  So that column that

          12    shows number of operators, that justifies

          13    nationally, doesn't apply to each of the --

          14    each of these regions.

          15              MR. WOOD:  Right, those are --

          16    that's a unique count.

          17              MR. BLAIR:  Oh, okay.

          18              MR. WOOD:  That you know, if

          19    they're in 5 districts they're not counted 5,

          20    that is the national, you know, you may

          21    count, you know.

          22              MR. BLAIR:  Right.  But the way the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            161


           1    table is set up it would, in first looking at

           2    it, you think that the number of operators

           3    applies to each of those columns, but it

           4    doesn't --

           5              MR. WOOD:  It doesn't.

           6              SPEAKER:  Correct.

           7              DR. SITTER:  I'm glad I asked the

           8    question because I got it all wrong --

           9              MR. WOOD:  Well, that was why my

          10    sincere invitation that you know, if we are

          11    not on the same page here, we're not going to

          12    do much better later.

          13              DR. SITTER:  And you decided this

          14    ordering on the report?

          15              MR. WOOD:  Yes.  Now there are few

          16    ways to actually get a few less operators

          17    that I think - -

          18              MS. KIRKENDALL:  Importance is

          19    based on the size of the region, like for

          20    Texas it's one of the --

          21              MR. WOOD:  Yeah, this is --

          22              DR. SITTER:  Got you.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            162


           1              MR. WOOD:  But the actual minimum

           2    number of people, you know you start out with

           3    the one with the least coverage and do it

           4    first and rank them back through and it you

           5    can do a few, but it's not worth the trouble

           6    of explaining it.

           7              DR. SITTER:  Got you.

           8              MR. WOOD:  So what we intend to do

           9    then is pick these out in a calibration year

          10    period and again this is based on each

          11    operators annual average production during a

          12    year, for example the year of 2000.  And then

          13    sample those operators and the estimation

          14    year would generally have up to a 2- year

          15    lag.  So we might be sampling those operators

          16    and try to estimate the year 2002, from the

          17    people we selected in the calibration year of

          18    2000.

          19              Because there's a long lag between

          20    the preliminary data and actually knowing

          21    what the final total is.  So if we go back

          22    now to the year 2000, we know what the total




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            163


           1    is and we know what the operation or what the

           2    production of each operator in each of those

           3    significant estates.  And now when you go and

           4    look at these, again if you plot by month

           5    across the year like 1997, this is the group

           6    of operators who had 90% on an annual basis

           7    of the 1997 production in Texas.

           8              And you notice as soon as you start

           9    moving into succeeding years, they start

          10    representing a smaller percentage of the

          11    total.  And in the year 2000, where we have

          12    run some detail tests we're going to show in

          13    a minute, it's the same thing.  There's some

          14    significant variation month to month, but

          15    this group is picked on the operators that

          16    had 90% of the production in the year 2000,

          17    on an annualized basis.

          18              And again in 2000 as soon as you

          19    get into succeeding months, the percentage of

          20    the production represented by those operators

          21    starts declining, and if you get into the,

          22    well let's just finish in January of '03




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            164


           1    here, you're down where, you know it's, they

           2    represent, they've only got about 87% of the

           3    production.  So you can't use just a linear

           4    multiplier times of production of the sample

           5    operators and expect to have anything, but a

           6    low biased answer and you saw all the

           7    techniques that Preston demonstrated.

           8              And again you see the same behavior

           9    in 2002.  Next slide.  This if for the Gulf

          10    of Mexico and this group had 95% and you

          11    notice that the, you're up over 98% of the

          12    survey sample.  And again you see the same

          13    type of behavior though.  After this

          14    calibration year of '99, the percentage

          15    represented by that group, the 90% group in

          16    '99 is you'd unless steadily -- say steadily

          17    declining except there's some relatively

          18    abrupt changes too.

          19              The Gulf of Mexico in particular,

          20    large operators can come in and not have it

          21    produced the year before and then be a major

          22    player, because they bring on a major field




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            165


           1    in the off shore Gulf of Mexico.  Next slide.

           2    Now again, we're going to have a some types

           3    of model, where we had picked operators in

           4    the calibration year and in Texas in 2002,

           5    that would be 203 operators that had 90% of

           6    the production.  And then this curve, we

           7    normalized by dividing by that so that they

           8    have a relative 100 in the year 2000.

           9              Those operators then have declining

          10    production and they only have around 97% of

          11    what they started with.  They've lost about

          12    3%.  Those -- and that's 203 operators, the

          13    top 90% group.  The data that we're trying to

          14    estimate is for the people in the 10 percent

          15    group.  Now that's 5473 operators and they

          16    had 10 percent of the production.  But their

          17    behavior is very different.  And in fact they

          18    fairly rapidly increased and they had about

          19    17% by the end of 2002, 2-year lag to their

          20    shares of production.

          21              DR. HENGARTNER:  Now, you told me

          22    the dynamic situations there, operators




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            166


           1    coming and going.

           2              MR. WOOD:  Correct.

           3              DR. HENGARTNER:  Do you add --

           4              MR. WOOD:  And increasing and

           5    declining.

           6              DR. HENGARTNER:  Yes, so in the

           7    remaining 10%, do you add a new operator that

           8    are generated, that might be very long --

           9              MR. WOOD:  That is such an

          10    excellent question.  Could I have the next

          11    slide?

          12              SPEAKER:  Okay.

          13              MR. WOOD:  Or maybe go one more

          14    forward.  One more.  You can split them into

          15    2 groups.  In the bottom 10% there are

          16    actually 4,158 operators that were producing

          17    in the year 2000.  In the next 2 years, they

          18    added about 12% to their total, which is 1.2%

          19    of the total production.  They're also,

          20    during the 2 years after 2000, a 139

          21    operators, that were not producing at all.

          22    And they had at the end of the year 2002,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            167


           1    they had gone up to 0.7 percent.  Between

           2    them they'd gone up to about 2 percent.  So

           3    the bulk of it was actually from the

           4    operators that were in fact producing and you

           5    know, since they didn't have as far to go to

           6    zero, they preferentially increased, and then

           7    there's actually an amazingly complex topic

           8    of why the large operators would tend to be

           9    declining and the small ones tend to be

          10    increasing.  But part of it is that is these

          11    new companies that are coming on.  And you

          12    know in part, some of those could be a split

          13    of 2 other companies, and we can track that

          14    behavior over time although it's rather

          15    cumbersome.

          16              MS. KIRKENDALL:  So of the 139

          17    companies, you got them on the list, but are

          18    they on your list as -- were they on your

          19    list producing zero, are these companies that

          20    sort of shows up on the list as a mistake or

          21    an error, can you tell?

          22              MR. WOOD:  You can do a gedanken




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            168


           1    experiment and say you didn't know about

           2    them.  But when you know, it might be 2001 or

           3    later when you actually do this calibration

           4    for the year 2000.  So in all likelihood when

           5    we're, let's say sitting out here about

           6    January of 2002, we would've known that most

           7    of these operators had started production.

           8              MS. KIRKENDALL:  You would know

           9    that they are new names on the list that way

          10    you could --

          11              MR. WOOD:  Yes.  But you know, we

          12    wouldn't have known, how much they would, you

          13    know they might have been just one month of

          14    production let's say, and you wouldn't have

          15    you know like annual data for them, or plant

          16    history.

          17              MS. KIRKENDALL:  You might -- you

          18    know, how do you find out if they've started

          19    producing, these are tiny companies in 2001.

          20              MR. WOOD:  They appear in a

          21    production list of -- usually we go, the

          22    states, most of the states, let me just have




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            169


           1    it.

           2              MS. KIRKENDALL:  You don't have

           3    2001 data?

           4              MR. WOOD:  In 2002, in the -- by

           5    the, we have probably the majority by far of

           6    the production in Texas for 2001, as we start

           7    into the first quarter of 2002.  It's just

           8    not complete.

           9              MS. KIRKENDALL:  Uh-huh.

          10              MR. WOOD:  In the first quarter of

          11    2001, we'd have excellent estimates of the

          12    total and relatively good coverage.

          13              MS. KIRKENDALL:  But these tend to

          14    be little guys at least at the beginning of--

          15              MR. WOOD:  Yeah.  It's just you

          16    don't know it close enough to really

          17    calibrate your, you can estimate relatively

          18    accurately, but you don't have the data in

          19    hand, everyone's actual production report,

          20    the full set of all operators to add them all

          21    up and say that's the number.  You can

          22    estimate that number pretty well, and in fact




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            170


           1    one of the things we might well want to do is

           2    try to use a one-year lag and not a two-year

           3    lag.

           4              MS. KIRKENDALL:  That might be

           5    better if you could.

           6              MR. WOOD:  But, you know again, the

           7    people that had zero production in 2000, and

           8    there was a 139 of them that appeared in 2001

           9    or 2002 and they -- they were pushing toward

          10    1% of the total by the end of 2002.

          11              DR. SITTER:  Did you look to see if

          12    it's just some sort of random process, that

          13    is, do they disappear and reappear again or

          14    are these just brand new people?

          15              MR. WOOD:  They appear mostly and

          16    then stay a while.  Some appear and go away.

          17    Now by the way it's the oil and gas industry

          18    and the reason that there are so many small

          19    gas producers is that almost every oil

          20    producer also produces gas.  It is produced

          21    directly with the oil and so they can have a

          22    profitable operation producing oil and then




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            171


           1    they get some natural gas also which goes

           2    into the total.  And that type of gas

           3    represents about 20% of the U.S production

           4    total.

           5              DR. SITTER:  But you don't have

           6    instances where they go away and come back.

           7              MR. WOOD:  Actually we do intend,

           8    and in fact have spent an entertaining

           9    afternoon or two putting up all the data of

          10    all the operators by month for a 7 year

          11    period and in fact watch the many of the

          12    large operators declining down and many of

          13    the intermediate size and small operators

          14    inclining as they become large operators.

          15    And so you can in fact there is data to that

          16    effect and there is aggregate data to that

          17    affect.  You want to back up 2 slides.

          18              Again, there are four curves here.

          19    One of them is the 90% sample and for 2000 in

          20    Texas, and you notice it starts losing

          21    percentage share as you go forward in time,

          22    it's relatively smooth and predictable.  This




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            172


           1    particular run, this right here, the light

           2    blue line is the first quartile of

           3    production, which is also the top 5

           4    operators.  And you notice they went from a

           5    relative 100 down to like 92.  They lost

           6    about 8 percent of their production in a

           7    2-year period.

           8              And this is one of the reasons why

           9    EIA's data was often criticized by energy

          10    analysts who look out their window and see

          11    the Hudson river.  Or even those who'd look

          12    out of their window and see the Charles

          13    river, that they assume they had some

          14    publicly available information on the large

          15    operators and if it was declining then all of

          16    the production was declining and obviously

          17    that isn't necessarily true.  Now, this curve

          18    here again, is the one we're trying to

          19    estimate.

          20              The ones that started out on the

          21    10% group and it's increasing, the full 90%

          22    group is declining, the first quartile was




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            173


           1    declining rapidly.  This group here, is the

           2    90% group with the top 5 operators taken out

           3    of it.  And that again represents about 65%

           4    of the production data in Texas and it's

           5    behavior, it increases slightly a percent or

           6    two and then is relatively stable across the

           7    2-year period.  And it's an example of one of

           8    the sets of data, if you chose you could use

           9    to predict the data's that we're estimating

          10    the non-sampled operators.

          11              DR. HENGARTNER:  Now, just to get

          12    this right again, there's 10% of non-samples.

          13              MR. WOOD:  Correct.

          14              DR. HENGARTNER:  That has been

          15    normalized with 10% of total production.

          16              MR. WOOD:  During the calibration

          17    year.

          18              DR. HENGARTNER:  Yes.  So although

          19    it looks like it goes up 15%, it actually

          20    goes up only 11/2% of total production.

          21              MR. WOOD:  Correct yeah, and in

          22    fact it does go up -- it increases.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            174


           1              DR. FEDER:  Yes, that's important.

           2              MR. WOOD:  But it increases.  You

           3    know again it increases.  It's normalized

           4    production.  That production increased by

           5    15%, which represents 1.5 percent of the

           6    total.

           7              DR. HENGARTNER:  Yes.

           8              MR. WOOD:  But again, if you know

           9    in account for that, and then the stuff from

          10    the new operators, you're missing 2% of the

          11    data and it's hard to make a 1% accuracy

          12    estimate.  Now, if we look at the next slide.

          13    Again here's the data we're trying to

          14    estimate, the yellow line.  And in particular

          15    the top 15 operators, 50% of the production

          16    is the dark line, and again it declines about

          17    5% over the two-year period.  The top 15 --

          18    the top 15 operators.

          19              Now the purple line is, and we

          20    started with 203, so the next 188, that we're

          21    sampling, is the purple line which, you know,

          22    it has more variation in it than the line




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            175


           1    which had about a 198 operators in it, you

           2    know, 10 more operators.  But you know again,

           3    at least they didn't decline.  And then other

           4    groups on the line are none that even got

           5    close to replicating the actual behavior of

           6    the 10% group.  Next, and next.  Now, you can

           7    write down a relatively general model, that

           8    we can always get the total in some month by

           9    summing the production from all the

          10    operators.

          11              In Texas that's, you know, roughly

          12    5000.  You can break that into 3 groups, and

          13    the first sum here is the sum of the --

          14    production sum of the operators used to

          15    estimate the non-surveyed operators.  The

          16    second sum is any of the survey group that

          17    you choose not to use their data in the

          18    estimating module and the third group is the

          19    sum of the non- surveyed operators.  Now this

          20    is a -- you know a critical concept

          21    especially if we're planning to run

          22    algorithms to check if they are people that,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            176


           1    may be an error, or overtly influential

           2    operators or outliners or something and we

           3    may need the flexibility of actually changing

           4    the group we used to model the non-surveyed

           5    operators potentially every month, and you

           6    know this general scheme is flexible enough

           7    to do that.

           8              From those operators that we're

           9    going to use to estimate the non- surveyed

          10    operators, we're going to multiply them by a

          11    function and get an estimate of the

          12    non-surveyed operators.  Now, in your fantasy

          13    world, it would be nice if we could just take

          14    your sample which at one point in time

          15    represented 90% of the production and divide

          16    it by 0.9 and 2 years later get a good

          17    estimate for the total.  But we've just seen

          18    how the data behaves and that will always

          19    lead to low biasing.  Well, in all the data

          20    we looked at it leads to low bias.

          21              Next slide.  So what we've been

          22    tried is you start out with the ratio for the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            177


           1    calibration year which is basically 0.11, if

           2    you use a 90% sample it's 1/0.9.  And then as

           3    we saw, that there's a slope to the

           4    relationship between the 10 percent group and

           5    any of the survey groups, and we -- you know

           6    fit and the data for 2 years after the

           7    calibration year and determined what the fit

           8    parameter Am would be, the next slide.  And

           9    then you know you -- you multiply by one plus

          10    this R and you get the sum of the operators

          11    that you surveyed and used in the estimation

          12    process.

          13              The estimate of the 10 percent you

          14    did not sample and the sum of those operators

          15    that you sampled, but you did not use them to

          16    estimate the 10 percent group.  Now this is

          17    the errors of the, percent errors in the

          18    sample in the calibration run using the, you

          19    know, the '99, starting with the '99

          20    calibration year.  The red bars are plus and

          21    minus half a percent.  So at least in the

          22    calibration fits the errors are rather small.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            178


           1    But again these are monthly and so there is

           2    some variation month to month.

           3              Next slide.  And of course it has

           4    to get a little more complicated.  You

           5    noticed in '97, this parameter is 0.0026, and

           6    in '98 it's 0.0053, and then 0.0074 and

           7    0.0089, 0.0012 basically.  So the Am also

           8    have the slope over time.  Next slide.  This

           9    is a fit of that.  The R2 is 0.9887, you

          10    could hope that this is a trend that might be

          11    projectable for a year, but not count on.

          12    Now if you have no slope component, you even

          13    with the Am parameter in there, that adjust

          14    roughly for the you know, changing

          15    relationship between the 10 percent group and

          16    the 90 percent group.

          17              It's lagging 2 years, because you

          18    calibrated against 2-year-old data and the Am

          19    actually had a slope.  So you know, that also

          20    leads to a bias, although, you know,

          21    basically we're in the 1% range for Texas and

          22    you'd hope we'd be well in the 1% range for




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            179


           1    the nation.  If you put in the slope

           2    correction, then you know, your, none of the

           3    monthly errors are greater than plus or minus

           4    5% and they're running effectively without

           5    bias.  Yes.

           6              DR. SITTER:  What data do you use

           7    to estimate the Am?

           8              MR. WOOD:  The we had a data set,

           9    that is all of the monthly production of all

          10    of the operators, starting in January '97

          11    through and is complete through 2002.  It's

          12    relatively complete through 2003 and some

          13    2004 data.  So this is a history of the

          14    actual production as it unfolded.

          15              DR. SITTER:  Okay.  So I'm

          16    predicting for January 2003, I have to

          17    estimate or which you just taken to be like

          18    1/90, and then you got an estimate Am.  So

          19    what data did they use for the Am, that the

          20    estimated got that point through January?

          21              MR. WOOD:  The one that I got when

          22    I went back and did the process 2 years




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            180


           1    before.

           2              DR. SITTER:  So you -- so you took

           3    2001 data.

           4              MR. WOOD:  Yes.

           5              DR. SITTER:  But.

           6              MR. WOOD:  We used '99 calibration

           7    year, estimated 2001 --

           8              DR. SITTER:  Oh I see, I see.

           9              MR. WOOD:  And then you start with

          10    2001 calibration year and use that Am.  So it

          11    was calibrated on data with almost a 2-year

          12    lag.

          13              DR. SITTER:  All right.  I got it.

          14              MR. WOOD:  Right.

          15              DR. SITTER:  So you use like '99 to

          16    2001 to get your Am --

          17              MR. WOOD:  Uh-huh.

          18              DR. SITTER:  And then took the

          19    whole relationship back.

          20              MR. WOOD:  Correct.  Then calibrate

          21    it against 2001 and used that Am, the one

          22    done with the 1997 calibration year and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            181


           1    projected 2003.

           2              DR. SITTER:  You assumed that the

           3    change in this year between the 90% and the

           4    10% is sort of stationary over time.  They

           5    change is stationary.

           6              MR. WOOD:  Uh-huh.

           7              MR. WOOD:  And if you want to draw

           8    back a slide, you know, that is the change

           9    over time, of the Am.  This is Am as a

          10    function of time starting in well you know,

          11    it would be '97 calibration year, '99

          12    estimation year and that's the first Am.

          13    Could you drop back one more slide and you

          14    know, here are the way they are laid out,

          15    the, you know, 0.026 and it doubled to 0.53

          16    and it's basically going up by that 0.026.

          17    So it has a relatively linear slope over

          18    those years.

          19              Again there's no guarantee that

          20    persists into the future.  All right.  Top

          21    four of about three slides.  No, one back.

          22    This one was done with the first quartile of




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            182


           1    production in Texas taken out, the top 5

           2    operators taken out.  And if you remember

           3    from the production plots, the top 5

           4    operators were losing the, had the greatest

           5    declining slope and they had a, you know, 25%

           6    of the production.  And so with no

           7    correction, this series and the other one

           8    went down to I guess, more than 1%.

           9              This series is running you know,

          10    more like a what, 0.4 percent average low

          11    bias and then again you can put a slope

          12    component in there and it's the purple line.

          13    Now as Nancy pointed out when she looked at

          14    this, that led to a slightly larger error

          15    here, than you had when you used the full 90%

          16    sample.

          17              MS. KIRKENDALL:  This month --

          18    you're just, you're using the data for those

          19    5 companies, it's just you're not using those

          20    5 companies to estimate further -- the little

          21    guys.

          22              MR. WOOD:  Correct.  And then, you




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            183


           1    know again that is, you always use the

           2    surveyed data.  But you either just add it

           3    in, as it'd be nice to know what the 5

           4    largest companies actually produce, and

           5    secondly you can use any subset of the

           6    surveyed operators to calibrate all these

           7    parameters and then project which are 10% or

           8    what starts out as 10% is the non-surveyed

           9    operators, that component that you're not

          10    surveying.

          11              Now all of these are generally

          12    relatively good.  You know, it kind of seems

          13    like a nice idea if you don't give up a lot

          14    of potential large errors to have something

          15    that at least, the plan is not to have a

          16    biased estimate.  Now there are small

          17    variations to all the states data, but all of

          18    the states tended to have the same type of

          19    behavior.  You pick a set of operators that

          20    represented 90% of the production at a given

          21    calibration year and they rapidly or slowly

          22    or sometimes radically lose production share




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            184


           1    over time.

           2              And now the next slide.  You know,

           3    there's obvious questions here.  Do we want

           4    to just basically use the full 90%, why does

           5    this appear to give better answers than, it's

           6    like sampling proportional to size, etcetera.

           7    Should we have a biased or picking some group

           8    to use and only changing if we have to, like

           9    leave the top 5 operators out and unless

          10    there's a real problem if you use, and Texas

          11    would be the other 188 operators, oh, 198

          12    operators.  If we don't get some of the very

          13    large operators to report, how should we

          14    impute for them.  And that's potentially the

          15    simplest question.  Yes.

          16              DR. FEDER:  Actually I'm just

          17    wondering if calculation of standard errors

          18    of the estimates won't be hard.  Because in

          19    addition to the use of the server error and

          20    there's going to be a component in the error

          21    of the Am and also the R, and I don't know

          22    what -- what you do when you publish




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            185


           1    estimates, what do you publish, what kind of

           2    precision indicators you do for that?  I know

           3    what EIA is doing and I should've done my

           4    homework, which haven't, but it's just

           5    another component.  I still think that the

           6    message makes a lot of sense to me.  I'm just

           7    wondering what you're going to do about

           8    standard errors.

           9              MR. WOOD:  You want to know -- you

          10    know what I'm going to do about it.  I'm

          11    going to let Nancy worry about it.

          12              DR. FEDER:  I meant you in the

          13    plural sense.

          14              MR. WOOD:  You'd be -- is that, you

          15    know, there is an obvious thing you can do

          16    and that's go back and look at a 5 year

          17    history and say --

          18              DR. FEDER:  Okay, that's a good

          19    answer.

          20              MR. WOOD:  You know, 99 out of a

          21    100 times it was less than half a percent

          22    error and we don't -- and we'll let you know




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            186


           1    if the situation changed a lot.

           2              DR. FEDER:  Yeah, I mean your

           3    method is based on the assumption, which is a

           4    logical one, the past is indicative of the

           5    future and yeah, you can use that to estimate

           6    the server error.

           7              MR. WOOD:  Yeah, the model be,

           8    actual formal estimation of the errors is a

           9    much more difficult problem than calibrating

          10    the model.  And it just --

          11              DR. FEDER:  That's always the case.

          12              DR. SITTER:  I think I read in some

          13    of the stuff that we have here that there

          14    were more reasons for this approach being

          15    attractive than just the simulation study, is

          16    that correct?

          17              MS. KIRKENDALL:  The simulation

          18    study made us worry about the frames data, we

          19    go in -- the reason why you have that really

          20    weird or some of the really weird estimates

          21    from the PPS sample, and the PPS estimator

          22    was because there are some huge changes that




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            187


           1    go on in the industry.  Some companies that

           2    have a small weight one year and huge -- a

           3    small volume one year and huge production the

           4    next or vice versa.  And the way they do

           5    their frames and their update based on the

           6    EIA-23 which is a survey they're on, it's an

           7    annual survey.  They spend a lot of time

           8    cleaning the data for the big companies.

           9              So the data available within the

          10    frame for the big companies are likely to be

          11    pretty good.  The ones for the smaller

          12    companies are fairly not quite as good

          13    because they never get the scrutiny that the

          14    big ones do, because there's so much more

          15    impact on the total.

          16              DR. SITTER:  Now, you made this

          17    claim that stimulation problems were due to

          18    some changes in the weights.

          19              MS. KIRKENDALL:  Well, the changes

          20    in the industry, companies that would be huge

          21    one year and smaller next or vice versa.

          22    Since they wont be --




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            188


           1              DR. SITTER:  Do you think you're

           2    able to go in and verify that that's what was

           3    causing the PPS to --

           4              MS. KIRKENDALL:  A couple of

           5    things, once they gave those maximum errors,

           6    you could get one or two companies that are

           7    in the sample that had big changes over the

           8    time.

           9              DR. SITTER:  Nancy, what this is

          10    suggesting is that we should do the PPS and

          11    it's by John's method to get the weights to

          12    change over time, like the 5 largest ones,

          13    their weights or inclusion in the PPS should

          14    go down?

          15              MS. KIRKENDALL:  Well, they would

          16    have been in the certainty group for the PPS

          17    sample anyway.

          18              DR. SITTER:  Your point I

          19    understand.  So you're saying that you think

          20    -- you know the weights you selected them

          21    with, correct?

          22              MS. KIRKENDALL:  Well, you know the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            189


           1    past data, yeah, you've got our IQ.

           2              DR. SITTER:  All right.  So what

           3    you're talking about is you got weights that

           4    were large or then you selected the "Y" and

           5    the "Y" wasn't as large as you anticipated or

           6    vice versa.  But that's only going to effect

           7    the variability, not the bias.

           8              MS. KIRKENDALL:  No, but it does

           9    make for the occasional large errors.  That's

          10    what gives you the maximum and minimum error,

          11    minimum error, there weren't many minimum

          12    errors they were mostly maximum error that's

          13    on PPS estimator.

          14              DR. SITTER:  I'll be curious to see

          15    what happens with Hayek estimator as opposed

          16    to Horwitz- Thompson in that case.  It should

          17    be easy enough to calculate.

          18              MS. KIRKENDALL:  What is the Hayek

          19    estimator?

          20              DR. SITTER:  It's the one where you

          21    instead of just --

          22              MS. KIRKENDALL:  With the ratio?




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            190


           1              DR. SITTER:  It's basically a ratio

           2    estimator.  So you use the Horwitz-Thompson

           3    estimator, which is what you're already

           4    doing.  You divide by the sum of the

           5    reciprocal weight, reciprocal inclusion

           6    probabilities and multiply by the total

           7    sample -- by total population, so, so it's a

           8    ratio on the population size.

           9              MS. KIRKENDALL:  Right.

          10              DR. SITTER:  But it should be less

          11    sensitive to that kind of things and that

          12    problem may disappear.  Now, I think you have

          13    other --

          14              MS. KIRKENDALL:  I think it would

          15    be easy enough to add that to the simulation.

          16              DR. SITTER:  Yeah, it should be.

          17    It's just a different estimator absolutely.

          18    Also if you're really putting some emphasis

          19    on the simulation results this should really

          20    be done right and those datahats are all

          21    right.  They're not meaningful.

          22              MS. KIRKENDALL:  You can compute it




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            191


           1    in the other way to.

           2              DR. SITTER:  Yeah, you should

           3    compute in the other way and see how it

           4    works.  Okay, so that's to comment on,

           5    because I wasn't sure how much of this is

           6    operational.  There are operational

           7    advantages to not playing with those small

           8    ones as well, as you mentioned they're harder

           9    to deal with.  So I just want to point that

          10    if it's two things that you're looking at it,

          11    that are influencing what you're deciding to

          12    do, you should make sure about simulation

          13    results, because I don't think they're really

          14    indicative.  They may have problems, I don't

          15    know, they may even have other problems.

          16              MS. KIRKENDALL:  Well I mean, the

          17    other question about the simulations results,

          18    which were used to ----.  And now we've got

          19    data that we could get from John, that he has

          20    with the whole population.

          21              MR. WOOD:  Correct, and --

          22              MS. KIRKENDALL:  2000, the whole




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            192


           1    population in 2002.  So we could really do a

           2    good simulation with that.

           3              DR. SITTER:  Now you're still going

           4    to run into problems.  But I don't know how

           5    much you're -- how you're updating your

           6    training over the years, because, okay in the

           7    cut off sample that property you're seeing

           8    there is going to translate into the fact

           9    that the guys that are in your certainty --

          10    that are your certainty groups are not going

          11    to be the certainty group you would

          12    necessarily want over time.

          13              MS. KIRKENDALL:  I have to select a

          14    sample every year.

          15              DR. SITTER:  That you call your

          16    certainty group.

          17              MS. KIRKENDALL:  Yes.

          18              DR. SITTER:  And there are

          19    operational disadvantages to that.  All

          20    right, but if you're doing that then they,

          21    this -- you see, I think I can explain why

          22    this happens.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            193


           1              MR. WOOD:  Which happens, what

           2    happened?

           3              DR. SITTER:  This property of the

           4    90% cut off guys decreasing.

           5              MR. WOOD:  And what do you -- can

           6    you --

           7              DR. SITTER:  I'm just

           8    hypothesizing, okay.  Suppose we have some,

           9    we have process that's random, it's

          10    fluctuating, that is the big operators are

          11    producing, not producing more or less each

          12    year.  And other operators are coming in and

          13    going out, a birth and death process as well,

          14    and -- at any given point in time, I take the

          15    top group.  I guarantee, if I now focus on

          16    that top group, it's production will end.

          17              DR. HENGARTNER:  It's called

          18    regression towards mediocrity.

          19              DR. SITTER:  Regression towards

          20    mediocrity.  You have to have them.  It's not

          21    something that's going like this, you've got

          22    a group that's going like this and you picked




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            194


           1    them.

           2              SPEAKER:  No.

           3              DR. SITTER:  Of course, the guys,

           4    if you look at who's sitting on top, then the

           5    point up there the 90% is staying the same

           6    but the players are different.  But if you

           7    look at a player he's going like this.  So at

           8    some point, at this point you pick this group

           9    of players.  So now guaranteed all the guys

          10    in those players are now going to go like

          11    this, right?  So that -- that same group is

          12    not going to stay the same group, it's -- it

          13    can't, you're picking the maximum, you're

          14    picking it when it's at it's peak, it's going

          15    to go down.  Okay, so.

          16              MR. WOOD:  This is one of the

          17    slides that I think is in your handout.

          18              DR. SITTER:  Yes.  I think I

          19    understand it better now.

          20              MR. WOOD:  And in fact you know,

          21    this is let's say the 90% group in the year

          22    2001, and you know, it's the same set of




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            195


           1    operators, you know who had 86% 4 years

           2    before.  Their really percent went up and

           3    peaked in this year and then came down.

           4              DR. SITTER:  You're going to see

           5    that in every case.

           6              MR. WOOD:  And you do and in fact

           7    that was -- the first, this is the first one

           8    we plotted and looked at and said, you know,

           9    I can make a general statement about that.

          10              DR. SITTER:  Yes, I think that's

          11    true.  I think you could, you know if you --

          12    you know if it were the same model, the

          13    random noise that's going on there, then you

          14    could probably figure out what that cycle is,

          15    I mean or have an idea of what that slope is.

          16    What I'm saying is, it's the process of

          17    deciding to do a cut off sample in this

          18    context, which is causing this pattern that

          19    you're seeing.

          20              MS. KIRKENDALL:  Well, but when we

          21    think about doing a cut off sample we will

          22    look at it every year and update it.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            196


           1              DR. SITTER:  That's --

           2              MS. KIRKENDALL:  So yes we do have

           3    them.

           4              DR. SITTER:  Yeah, you -- I know

           5    you'll update the group but the data you're

           6    using to do your estimates is still going to

           7    be from 2 years before, because you don't

           8    have the data.

           9              MS. KIRKENDALL:  Right.  So we'll

          10    be off by about 2 percent.

          11              DR. SITTER:  Yeah, exactly what

          12    John is - -

          13              MR. WOOD:  The data might be, but

          14    the estimate doesn't have to be.

          15              MS. KIRKENDALL:  Right.

          16              DR. SITTER:  No I mean in -- I'm

          17    sure there are, I mean --

          18              MR. WOOD:  I mean as for the

          19    estimate.

          20              DR. SITTER:  No but if the

          21    stationarity you're seeing is because of the

          22    sort of process -- and you're picking out a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            197


           1    maximum group, so that maximum group has some

           2    stationarity.

           3              MS. KIRKENDALL:  Because it's

           4    stationary ready.

           5              DR. SITTER:  What -- what is it any

           6    non-stationary process you're going to get a

           7    -- let me go.  I think that it's stationary,

           8    you know, other wise you wouldn't get good

           9    estimates.

          10              MS. KIRKENDALL:  Well the fact that

          11    you had the adjustments you made in, the

          12    changes over time, which is indicative of the

          13    --

          14              MR. WOOD:  And if you had great

          15    selections at a given time, you'd have to

          16    build something like that in or build in the

          17    absolute, you have to build in the time

          18    change of the distribution.

          19              DR. SITTER:  What I mean is this

          20    that, I think that if you take a stationary

          21    process and you pick out a maximum group,

          22    that maximum group is going to have a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            198


           1    downward trend, it has to.

           2              MS. KIRKENDALL:  Oh yeah, it has

           3    to.

           4              DR. SITTER:  So I'm not talking

           5    about, I'm talking, you're saying that the

           6    slope of that line means it's not stationary,

           7    yeah, in a certain sense.  That's not what I

           8    was speaking of actually.

           9              MR. WOOD:  No, I mean that this is

          10    a general phenomenon.

          11              DR. SITTER:  Yes.

          12              MR. WOOD:  It's that you take the

          13    top group of a --

          14              DR. SITTER:  Not sure which would

          15    be, I'm not sure whether what, if you sat me

          16    down and said should I do something like this

          17    or should I do PPS sampling, from what I've

          18    seen I can't tell because the PPS isn't

          19    correct.

          20              MS. KIRKENDALL:  But we can always

          21    do a better job on simulation.  We want to do

          22    that in a --




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            199


           1              DR. SITTER:  Yes.

           2              MS. KIRKENDALL:  And in the

           3    estimators have talked about that, given all

           4    those.  I think the effort is gone into

           5    developing the stimulation and this will be

           6    an easy ---- come to that.

           7              DR. SITTER:  But the advantage, the

           8    cost of course would be sampling some smaller

           9    --

          10              MS. KIRKENDALL:  Right.

          11              DR. SITTER:  Firms.

          12              MS. KIRKENDALL:  And that's, and I

          13    think you would probably end up with a bigger

          14    sample presuming it's under that making the

          15    sample a bit smaller than we thought we're

          16    going to have --

          17              DR. SITTER:  I don't know if that's

          18    --

          19              MS. KIRKENDALL:  We'll get an idea

          20    when we do the stimulation.  We were thinking

          21    closer to 350 companies before we -- because

          22    we were sort of moving from the 5% of the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            200


           1    regional level up to a higher level of

           2    accuracy, wanting a higher level of accuracy

           3    for each region.  So we were at about 250, I

           4    don't think anybody was thinking we were

           5    going to be able to get much lower than that

           6    and with the cut off sample it was like 300,

           7    setting on a smaller sample means that we can

           8    actually maybe do the work, bigger companies

           9    are more cooperative and they are more happy

          10    about, less unhappy about sending in the

          11    data.

          12              MS. NORMAN:  Also there is a

          13    downward trend that looks like it stayed

          14    somewhat relatively stable for at least a

          15    year.  It will completely plummet the second

          16    you hit that 90% group.

          17              MR. WOOD:  By the way it is a

          18    practicable thing to do, to actually go in

          19    and project groups of the operators that are

          20    on an upward trajectory and those are on a

          21    downward trajectory and there are some

          22    tremendous things many of which might be




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            201


           1    subjects of controversial papers of, why are

           2    those large operators declining so fast when

           3    a bunch of intermediate size and small size

           4    operators are actually increasing their

           5    production and it is the behavior of those

           6    firms compared to the behavior of other firms

           7    which is a different kind of problem but one

           8    addressable in policy?

           9              MS. KIRKENDALL:  But where we are

          10    now on this, I think the surveys been

          11    approved for January, to begin with January

          12    data and we can implement, assuming we have

          13    the money to do it but we can implement this

          14    methodology, it doesn't mean that we have to

          15    stay with it forever but we need something we

          16    can do in January.  Now we continue to look

          17    at it and revise this next time, so, I'm sure

          18    you'll hear more about it.

          19              MR. WOOD:  So yeah, there is a

          20    question of is there a sampling procedure

          21    which it actually making it more efficient,

          22    less operator with the same reliability




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            202


           1    survey and we were going to look at that,

           2    maybe, could --

           3              MS. KIRKENDALL:  I'm not sure you

           4    are going to get few operators in a survey, I

           5    think if you use random -

           6              MR. WOOD:  There would actually be

           7    more.

           8              MS. KIRKENDALL:  It could end up

           9    being more, because that -- it's certainly

          10    something new.

          11              MR. WOOD:  And then there is the

          12    level of effort question, you know, we're

          13    dancing around few tens of percent error in

          14    individual stake, where we have been, people

          15    have in the past said we're off by 4% or 5 or

          16    6 or 7 or 8 --

          17              MS. KIRKENDALL:  We did that for

          18    years.

          19              MR. WOOD:  And this is well within

          20    our target that we started out and it appears

          21    to be actually very straight forward in

          22    application and relatively easy to explain,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            203


           1    and in effect they are, I would suspect that

           2    there are the modifications of the sampling

           3    proportional to size, would take some fairly

           4    experienced people about a year to sort out

           5    because now I have questions about them, some

           6    of the processes we used also, you know just

           7    how, you know, just the way some of the

           8    assumptions or approximations were made in

           9    the sampling proportional to size and you

          10    know, we'd actually use different things, you

          11    know, just before -- here we've got a process

          12    that is probably getting us and --

          13              MS. KIRKENDALL:  We understand it

          14    --

          15              MR. WOOD:  We could spend a lot of

          16    time -

          17              MS. KIRKENDALL:  We'll see if we

          18    can do it.

          19              MR. WOOD:  No, not that a graduate

          20    student couldn't profitable do this but --

          21              DR. SITTER:  Well I think there are

          22    other things to do just as sensible enough.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            204


           1    And one comment is, you know, your hand has a

           2    slope and it looks petty --

           3              MR. WOOD:  Yes.

           4              DR. SITTER:  Did you try putting a

           5    quadratic term in there because linearing the

           6    slope where your slope should really be a

           7    quadratic term in the original, like one plus

           8    AMP --

           9              MR. WOOD:  I can show you these

          10    written down in my notes here, yes and

          11    exponential and no and this is more, actually

          12    I kind of like this, that it's not a

          13    constant, in fact the ratio itself is a

          14    variable and then the fifth parameter inside

          15    the ratio is also a variable.  It's a very

          16    straightforward way of presenting it, after

          17    you've done that, then you can go back and

          18    set it with the quadratic and get the same

          19    results.  And in fact, you get essentially

          20    the same thing even if you start fitting the

          21    quadratic term.

          22              CHAIRMAN BREIDT:  Okay, so are we




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            205


           1    ready to state the summary.  I think you're

           2    up Randy.

           3              DR. SITTER:  That's ridiculous, you

           4    know everybody else had twenty minutes to

           5    prepare.  Give me a break.  Every other

           6    session finished early and I guess I'm

           7    blamed.

           8              CHAIRMAN BREIDT:  You asked the

           9    last question.

          10              DR. SITTER:  I asked the last

          11    question.  All right, well I'll give it my

          12    best shot.  Okay, so this was on monthly

          13    natural gas production survey and how to go

          14    about doing it.  We started with the recap of

          15    2004, where the ASA committee proposed using

          16    a PPS sample to this situation and some

          17    investigation had been done since the

          18    stimulation study initially.  The stimulation

          19    study used psuedo frame based on the EIA-23

          20    and we tried using optimum allocation of

          21    sample size for stratified SRS as a basis for

          22    choosing the sample sizes and then ran a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            206


           1    simulation study using the psuedo population

           2    to evaluate different estimation procedures.

           3              The samples were selected PPS

           4    without replacement and different estimators

           5    were considered.  Some of the estimators, one

           6    of the estimators was PPS Horwitz Thompson

           7    estimator and I guess -- another were

           8    versions of regression estimator regressing

           9    on two years previous data essentially, it

          10    was regression through the origin I believe

          11    just a beta hat multiplier, either with

          12    weighted least square or ordinary least

          13    squares also considered keeping the certainty

          14    units in the calculations of beta hat or not,

          15    so there were some questions by the committee

          16    on this, I say the committee primary that was

          17    me I guess, on the estimators themselves.

          18              The results of the stimulations

          19    suggested that the regression estimators were

          20    negatively biased and the PPS estimators

          21    though having no bias and behaving okay

          22    didn't have the CV it was supposed to, and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            207


           1    had some extreme cases, which would be --

           2              The comment from the committee was

           3    that probably the Horwitz Thompson PPS

           4    estimator wasn't the best one to look at

           5    though, of course it's in the literature and

           6    it's unbiased and there's nothing really

           7    wrong with it but it does have this property

           8    that if you get a, you can have some very

           9    badly behaved samples so that might be the

          10    result.  That might be the cause or problem

          11    cases that were observed and that the

          12    negative bias of the beta hat is probably due

          13    to the fact that the sampling we turned out

          14    included in the estimation procedure.

          15              When you do do them I suggest that

          16    the beta hat or the committee suggested as

          17    well that you should not really include

          18    certainty in calculation in your beta hat

          19    because you don't if the relationship is the

          20    same.  So the simulation stated, there was

          21    some question about it, about some, they can

          22    easily be done again to investigate whether




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            208


           1    those questions have any bearing on the

           2    decisions or any bearing on conclusions about

           3    the various estimators, they may or may not.

           4              They were other reasons that PPS

           5    sampling is not that attractive so it may be

           6    that even if committees concerns are

           7    realized, they still may decide that this

           8    isn't the best, necessarily the best way to

           9    go.  So then investigation using a 90% cut

          10    off sample, that is take the just essentially

          11    sample with certainty, the producers which

          12    could be attributed to 90% of the production

          13    two years previously when looking at today,

          14    problem then is that, of course if that 90%

          15    production group remains constant over time

          16    then in other words they could constantly be

          17    attributed to 90% of production then it would

          18    be reasonably simple multiplier to take care

          19    of the non-sample and in this case a very

          20    small number of the five to ten thousand of

          21    producers attributed for the 90% of

          22    production.  So operationally it is




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            209


           1    attractive.

           2              The problem is, is that when you

           3    pull the 90% out in say 2001, by 2003 they no

           4    longer are attributable to 90% of production.

           5    That pattern seems to occur over time and it,

           6    the worst comment from the committee that

           7    that's probably natural, your picking at a

           8    maximum group over a period of time, almost

           9    certainly that group will not remain the

          10    maximum group because of random perturbation

          11    of the process of relying.

          12              So I think that the estimation

          13    method that was suggested is basically to

          14    calibrate that, since there seems to be a

          15    general reasonably fit by a linear trend, I

          16    guess overtime, the percentage of the

          17    production that could be attributed to the

          18    maximum, the 90% group over the following two

          19    years.  One could actually model it and it

          20    performs reasonably well.  There were some

          21    questions asked of the committee, so I've

          22    been sort of answering them as we go.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            210


           1    Questions were asked about comments for data

           2    and approach to modeling or estimating the

           3    non-sampled portion.

           4              Some of the recommendations on the

           5    stimulation study follow that, some questions

           6    about the procedure that seems to be the cut

           7    off, 90% cut off was how are you going to

           8    estimate variation.  The answer was by

           9    looking at the window of time and seeing how

          10    it actually did perform when you get the data

          11    in so, that was about the only suggestion.

          12    There was a question by the committee about

          13    the inclusion probability in the stimulation

          14    study and how they were calculated and the

          15    answer was that the PPS sample in the

          16    stimulation was actually taken systematically

          17    and I think there was the other questions

          18    were related to outliers and influential

          19    observations and the committee did not get to

          20    address that issue.

          21              CHAIRMAN BREIDT:  Thanks Randy, any

          22    quick questions or comments.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            211


           1              MR. KUNDRA:  Can I answer?

           2              CHAIRMAN BREIDT:  Okay.

           3              MR. KUNDRA:  I think the problem

           4    with PPS was that the frame which was used to

           5    select the PPS sample, when they went to the

           6    next year, most of the, I don't have any

           7    problem with the certain sample, the problem

           8    will be non-certain sample because most of

           9    the units were not found in the following

          10    year.  The problem is lying with the frame

          11    because the units which have been selected in

          12    the using the frame 2000, if you go to 2001

          13    or 2002, you don't find those operators over

          14    there.

          15              So that's one of the problems

          16    because that's why they were getting some bad

          17    results and so for the variances concerned

          18    because you could not substitute, or you

          19    could not make any sort of assessment for

          20    those varied floors or patterns which are

          21    missing from the, for the year which the

          22    materials were use.  That's one of the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            212


           1    reasons the frame is not very stable.

           2              DR. SITTER:  But that wouldn't

           3    arise itself in the stimulation study?

           4              MR. KUNDRA:  I don't know.  I'm

           5    talking about the PPS sample.

           6              DR. SITTER:  No, I understand that

           7    but that --

           8              MR. KUNDRA:  It will rise in the

           9    simulation also too.

          10              DR. SITTER:  It arose in the

          11    stimulation and was a cause in the

          12    simulation?  It's quite a bit different.

          13              MS. KIRKENDALL:  Well except in a

          14    simulation he had, I mean, there were, zeros

          15    weren't in there.  We took --

          16              DR. SITTER:  Yeah, you took out all

          17    the unmatched.

          18              MR. KUNDRA:  We took out all the

          19    unmatched.

          20              MS. KIRKENDALL:  We took the

          21    outmatched out, so that --

          22              DR. SITTER:  So I think what the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            213


           1    committee is saying is, is that the

           2    simulation study is one aspect of why you may

           3    want to decide on two things.  Stimulation

           4    study needs to be done, something's need to

           5    be done with it, to make sure that you're

           6    properly evaluating that aspect.  But it's

           7    not the only aspect of whether you will or

           8    won't decide to go to this sampling frame,

           9    and it maybe that the operational things such

          10    as a change in frame may overwhelm the

          11    efficiency issues.

          12              MR. KUNDRA:  That is one of the

          13    main things.

          14              CHAIRMAN BREIDT:  Thanks Joe.

          15              MR. SEDRANSK:  I'll just try to

          16    reduce some confusion.  There should've been

          17    a line on the stimulation results that had a

          18    line under the PPS estimator.  All of the

          19    remaining ones were model- based or were

          20    intended to be model-based estimators, so in

          21    fact the fact that they didn't do well, just

          22    that the model didn't fit terribly well,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            214


           1    rather than anything about the selection and

           2    that has a lot to do with what Indar just

           3    said.

           4              I looked with an intern over the

           5    summer at these constructed data sets, they

           6    do have an underlying linear regression

           7    through the origin but they are very bumpy,

           8    really bumpy.  Ones at the lower end and

           9    especially in the middle, which we're really

          10    amazed at, so basically what I think we're

          11    seeing what the, you know, sort of poor

          12    performance for all those weighted least

          13    square estimates is in fact, that if you

          14    really wanted to perceive that way you need

          15    to more, I'll use the word a more

          16    sophisticated estimator.

          17              Now I wouldn't continue doing this,

          18    I was talking to John at lunch and John is

          19    concerned about this data set, you know, the

          20    one that we can, you know sort of the data

          21    that was used for it and I think this is a

          22    little bit suspect, in other words, the data




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            215


           1    set that was used for the stimulation is

           2    questionable but basically a confusion was

           3    that the ones below, the reason I said that

           4    there should be a line, is all those weighted

           5    least squares were intended as model-based

           6    estimators and --

           7              DR. SITTER:  so there were, their

           8    performance was evaluated as model based?

           9              MR. SEDRANSK:  No, no, no, they

          10    were, no they were evaluated as, I mean, they

          11    were given a rigorous, a more rigorous test

          12    to the sets.  But they weren't -

          13              DR. SITTER:  The thing is that, if

          14    you, you know, if you're getting stimulated

          15    biases and the stimulation is based on the

          16    repeated sample --

          17              MR. SEDRANSK:  Sample, that's

          18    right, yeah they are.

          19              DR. SITTER:  Then the bias is, the

          20    bias are actually measured in --

          21              MR. SEDRANSK:  That's absolutely

          22    right.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            216


           1              DR. SITTER:  Right.

           2              MR. SEDRANSK:  So that's correct.

           3              DR. SITTER:  Yeah.  In fact and one

           4    of the interpretations was that they were no

           5    good because they were biased.

           6              MR. SEDRANSK:  Yeah, the other

           7    question which maybe Preston can ask, I'm not

           8    too sure we got too many of those really

           9    small units, this is related to Indar's

          10    comment, I'm not too sure we got too many of

          11    those small units.  Maybe it's PPS sample, so

          12    in fact it wouldn't have fixed this problem

          13    up if you know if that group is different

          14    than the top.  That I can't answer, do you

          15    know Preston off hand what percentage, you

          16    know, how many small units we got?

          17              MR. McDOWNEY:  I can't answer that

          18              MR. SEDRANSK:  Yeah I know, anyway

          19    that's all.

          20              MS. KIRKENDALL:  There probably

          21    weren't too many --

          22              MR. McDOWNEY:  No.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            217


           1              MS. KIRKENDALL:  Because it went to

           2    other EIA-23 survey, which provided the data

           3    for 2000 and we matched to the EIA-23.

           4              MR. SEDRANSK:  And we matched to

           5    the 23 frame in 2002, so I --

           6              MS. KIRKENDALL:  The frame would

           7    have had more small companies so we matched

           8    the data from the EIA-23 and that was more

           9    big companies.

          10              MR. WOOD:  It also reported a

          11    couple of thousand.

          12              SPEAKER:  Let me finish.

          13              MS. KIRKENDALL:  Okay.

          14              CHAIRMAN BREIDT:  Let me jump in

          15    here now, or we'll go too long, you guys can

          16    continue your break out session in the break

          17    if you like.  So I need to ask those who did

          18    not introduce themselves this morning to go

          19    ahead and do so now, so Bill will pass around

          20    the mike and if you could just give your name

          21    and your affiliation please.

          22              MR. WEINIG:  Might I also add that




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            218


           1    if you are sitting under a speaker this thing

           2    will squeal like the dickens so, if that's

           3    the case, please use the other mike.

           4              SPEAKER:  That for example is a

           5    speak.

           6              DR. SITTER:  Randy Sitter, Simon

           7    Frazer University.

           8              MR. WEINIG:  Johnny?

           9              MR. BLAIR:  Johnny Blair, Abt

          10    Associates.

          11              MR. MERRRIAM:  I'm Rob Merriam,

          12    from Z INC.

          13              MS. COX:  Brenda Cox, from

          14    Battelle.

          15              MR. KUNDRA:  Indujit Kundra, from

          16    SMG.

          17              MR. LANEY:  George Laney from

          18    Temple.

          19              MS. NORMAN:  Karen Norman, SMG.

          20              MR. WILSON:  Nathan Wilson, EIA.

          21              MR. TITA:  Michael Tita, SAIC.

          22              MS. OLSON:  Irene Olson, EIA.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            219


           1              MS. ULREY:  Peri Ulrey, NGSA.

           2              MS. CARROLL:  Carolyn Carroll, Stat

           3    Tech.

           4              MR. CASTLEBURY:  Jay Castlebury,

           5    EIA.

           6              MR. LU:  Rueypyng Lu, EIA.

           7              MR. WOOD:  While they are waiting,

           8    John Wood.

           9              MR. HALE:  Doug Hale, EIA.

          10              MR. MAGNUS:  Howard Magnus, EIA.

          11              MR. TRAPPMANN:  Bill Trappmann,

          12    EIA.

          13              MR. KASS:  Roy Kass, EIA.

          14              MS. GEISERT:  Lynn Geisert Z INC.

          15              CHAIRMAN BREIDT:  Okay, so I think

          16    that went around.  Now I introduce Neha

          17    Khanna to summarize the other breakout

          18    session.

          19              MS. KHANNA:  Thank you.  Our

          20    session focused on methods to assess the

          21    solution that the NEMS models produces and

          22    one of the issues of the NEMS is that it




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            220


           1    produces an enormous amount of output data so

           2    you'll have to, was it understood there are

           3    200 papers of results that span several

           4    hundred pages and part of the question is,

           5    how do you analyze the outputs in a quick but

           6    rigorous manner and that was the focus of our

           7    session.

           8              Professor Lady (?), what he did,

           9    was to write -- wrote a graphics interface to

          10    facilitate comparisons of solutions and he

          11    used three different diagnostic checks as

          12    well.  These were, what he calls it, his

          13    "monotonicity" checks to see if that, the

          14    number of types solutions trajectory's

          15    actually crossed each other, he used -- he

          16    used the forecast error to see for example,

          17    if you were forecasting to solve the intent

          18    value, if you forecasted it from 1998, you

          19    should have much greater variability than if

          20    you forecast it from 2004, to see again is

          21    NEMS producing you know, as forecast is

          22    getting more consistent as you get close to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            221


           1    the time and then he also to looks to see,

           2    just how variable the forecast were in

           3    general and what he found was, after doing

           4    that he took a large, he took the NEMS

           5    solution and actually tried to approximate

           6    the NEMS solution by a simple linear

           7    regression model to try and you know, can we

           8    get a handle for how the solution have

           9    changed over different runs of the NEMS model

          10    as there is pieces of assumption of margin

          11    for change to what time it's different

          12    solutions.

          13              And what he found was in fact, that

          14    was a reasonable first proclamation of the

          15    NEMS solution and that many of the results

          16    could be represented by linear

          17    approximations, also I was thinking that

          18    couple of cases you did use columnar

          19    regressions because there was a great

          20    non-linearity involved.  The basic questions

          21    for the committee there was, I think four or

          22    five of the questions, we had fixed this,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            222


           1    that we decided some of them says the same

           2    thing.

           3              The basic question is, is this a

           4    sensible way of approximating the NEMS

           5    solution and I understand, my feeling as the

           6    committee said it depends on what, what

           7    you're trying to use the solutions for.  If

           8    you are trying to see are NEMS solutions

           9    really changing over different runs of the

          10    models yes, it seems to be a good way because

          11    it sees some sort of blips in the estimated

          12    cooperation of your regression.  You could

          13    look at why do those blips occur, so if

          14    that's what you want to do, this seems to be

          15    a sensible approach and we have some common

          16    sense on how to improve that and I'll come to

          17    those in a minute, and the other side was if

          18    you want to see are NEMS results generally

          19    sensible, in the sense of you know, all the

          20    results that NEMS is providing you, there are

          21    comparable with like say results coming out

          22    from some other independent models, then you




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            223


           1    might want to impose on more economic type

           2    structural model on the NEMS solutions to see

           3    you know, the kind of results you'd get for

           4    example, the price elasticity coming out of

           5    NEMS are they or if it is in them, are they

           6    really sets of comparable with or consistent

           7    with other price elasticity elsewhere?

           8              So there were some that directly

           9    went to the question of you know, should we

          10    look at goodness of fit or should we only

          11    look at the possibility of specifications and

          12    then there the committee felt it depends on

          13    what you are trying to do.  As far as trying

          14    to improve on some of these, trying to

          15    improve upon this approach, at first

          16    committee felt that one of the things that we

          17    could do was, to develop, develop confidence

          18    intervals around predications and see if the

          19    new predictions from the NEMS fell within the

          20    confidence span then we're pretty much okay,

          21    if it fell outside the confidence span then

          22    well, this is something interesting and we




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            224


           1    need to look into it and I should say

           2    actually at the outset, all of this was done

           3    to demonstrate it.

           4              There is still work in progress and

           5    when you find the results that "differ," it

           6    should excite interest, it should not

           7    necessarily mean that there's something wrong

           8    in NEMS because there could be a fundamental

           9    change in, a fundamental and reasonable

          10    change in an assumption that drives the

          11    different results, so this is the just an

          12    attempt to find out what's going on and, is

          13    something changing?  We've also felt that the

          14    NEMS approximation should be "stand alone",

          15    or free standing so that anyone who is using

          16    NEMS and wanted to analyze the results could

          17    do it and we didn't need sort of an expert to

          18    sit by you to know, how to actually analyze

          19    or to approximate the NEMS solution.

          20              It was also felt that some of the

          21    scenarios that it maybe worthwhile thinking

          22    about running some NEMS scenarios, which are




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            225


           1    maybe now not been run but keeping in mind

           2    that we are going to use something like this

           3    in the near approximation to analyze the NEMS

           4    result in the future.  So you know, because

           5    of the great heterogeneity and sanguinity

           6    issues involved in these models, it may be

           7    nice to have certain drives or shift that

           8    will help us to identify some of those

           9    equations and that maybe something worth

          10    considering and we had some discussion about

          11    that.

          12              We also felt that actual data

          13    become available to overlay the NEMS

          14    projections with the actual data to see how

          15    we were doing.  In terms of uncovering

          16    programming errors or inappropriate

          17    solutions, one member from in audience

          18    suggested that we should look at feedback

          19    groups and I believe more and more that is

          20    actually being done, so that was a good

          21    thing.  There was one question that we didn't

          22    actually address, you all just nodded and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            226


           1    then left it, and that was what procedures

           2    would be appropriate for using NEMS

           3    approximation to facilitate convergence of

           4    the NEMS -- and we talked about how the

           5    linear approximations might be used to

           6    suggest starting value for NEMS but we didn't

           7    really say anything beyond that, it's just

           8    that I said, "yes, that's good, it seems

           9    reasonable".

          10              Another thing that came out of the

          11    linear approximation to NEMS was, it could be

          12    used in fact to partition out the forecasting

          13    areas into what part of the those areas are

          14    actually due to just differences in forecast

          15    and what part of those areas are actually due

          16    to just some difference in exogenous

          17    variables, so if you have a NEMS projection

          18    you use, you actually put in the actual data

          19    that came into the regression model and then

          20    see how much of it remains unexplained.

          21              So you could actually figure out,

          22    what was the source of the differences and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            227


           1    that would be useful tool to use, I believe

           2    that's that the summary to, the kind of free

           3    wheeling discussions I'm sure I've left out

           4    some pieces here and if somebody wants to add

           5    to it, I'll be very happy.

           6              CHAIRMAN BREIDT:  Maybe just one

           7    quick comment on the convergence issue, it's

           8    kind of like it wasn't so critical to have --

           9              MS. KHANNA:  Oh, Yes.

          10              CHAIRMAN BREIDT:  Good starting

          11    values.

          12              MS. KHANNA:  Thank you.

          13              CHAIRMAN BREIDT:  Any questions or

          14    comments.  Okay so I won't need much time for

          15    that.  I'll just turn it over to Nancy, who

          16    will give an introduction to Program

          17    Assessment Rating Tool, PART program

          18    evaluation.

          19              SPEAKER:  Why don't we do this and

          20    then we take a break.

          21              SPEAKER:  Yes.

          22              MS. KIRKENDALL:  Okay, this is an




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            228


           1    introduction basically to the rest of the

           2    afternoon and I'm going to give a sort of

           3    introduction to the break out sessions that

           4    we have scheduled, we've kind of stuck it in

           5    because we figured people wouldn't understand

           6    what we are really talking about in the break

           7    out sessions, if we didn't give you this

           8    little introduction.  There's a new tool, the

           9    Performance Assessment-Rating Tool is a new,

          10    well it's three years old.  It's used by the

          11    Office Management Budget.  There're

          12    completing it for all federal agencies.  We

          13    will actually, we, EIA were actually

          14    fortunate that we weren't hit with having to

          15    do it for the first two years, this is the

          16    third year of the program and we did have to

          17    do it this year, so this was our first year

          18    of introduction to this PART program.

          19              Other statistical agencies have

          20    been evaluated before we were and of course I

          21    wouldn't be too is trying to figure out how

          22    to actually implement this, so even though




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            229


           1    BOS and BEA have gotten good ratings in the

           2    past, not everybody does and most have been

           3    for education statistics got a bad rating one

           4    year because the person who did the, was

           5    interviewed, didn't take it seriously and

           6    didn't answer the questions and didn't

           7    prepare.  But other than being serious about

           8    it and they post the results on the website,

           9    so you can look and see what everybody, the

          10    scores everybody is getting, so it can be

          11    viewed in the website there one way.

          12              There are four areas in PART,

          13    everybody is scored under these four areas

          14    and the first one is on program, purpose and

          15    design as they show up there, EIA did very

          16    well on that so that doesn't identify any

          17    issues for us.  The second one is, strategic

          18    planning and performance measures.  We didn't

          19    do as well there, partly because we had just

          20    changed our performance measures OMB agreed

          21    that we had better measures that we were

          22    aiming for now, but they couldn't get us full




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            230


           1    credit for it because we didn't actually have

           2    baseline values and results yet.

           3              The third area of program

           4    management again, EIA did quite well, so we

           5    don't have any particular concerns there and

           6    on program results and accountability is an

           7    area that we're struggling with, we didn't do

           8    very well with that one either.  So the

           9    challenges I've got listed, none of them is

          10    quite as much of a challenge and our managers

          11    are new, so we don't have results yet well

          12    you know, over time that will change and

          13    we'll have results but the last one is, one

          14    that's more serious and that is that we don't

          15    have any external evaluations.

          16              So the evaluation question shows up

          17    in two places in PART.  In fact it shows up

          18    in that first area that we did not do well

          19    and in the second area that we did not do

          20    well in.  In the first area they say they

          21    want, regular independent evaluations and

          22    they should be used to support program




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            231


           1    improvement, assess effectiveness and

           2    relevance and when we answered the question

           3    requirement to the ASA committee and the fact

           4    that we talked to you when we tried to get

           5    advice on our technical problems and then we

           6    try to use the advice to fix things.

           7              We told them about our independent

           8    extra review program, we've had lots of

           9    things reviewed by independent experts and we

          10    try to improve things that way and all of our

          11    models are in the public domain and they were

          12    unconvinced and they said, "no".  So we're

          13    actually and well this one lives in the area

          14    where performance measures are important too

          15    and so they really couldn't give us anything

          16    because those are preliminary.

          17              In the second one, under program

          18    results and accountability, this is a very

          19    similar question and they actually refer you

          20    back to the first one.  They say you want

          21    independent evaluations indicate program is

          22    effect and it is achieving results.  So we




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            232


           1    talked about the American Customer

           2    Satisfaction Index, we talked to you about

           3    that earlier today.  We've done small studies

           4    of the impact of the EIA's data on markets,

           5    you've seen charts where our weekly data are

           6    released and the price on the market goes

           7    haywire.

           8              And so they gave us a small extend

           9    on that, so it's better than a "no" but it's

          10    not really great.  And the big criticism is

          11    that, we don't have independent evaluations

          12    of our entire program.  So this refers back

          13    to the question, we'll need to come up with a

          14    way of doing independent program evaluation.

          15    Independent means mostly outside of EIA.

          16    Although you could say that there is some

          17    independence in my office if we do an

          18    evaluation of a program office, they still

          19    might not think that's quite as independent

          20    as you like but we can certainly be involved

          21    to some extent.

          22              And the purpose of these




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            233


           1    evaluations should be to identify program

           2    improvements.  The more defectiveness appears

           3    in many places in their, in their discussions

           4    so we need to somehow assess effectiveness,

           5    we need to assess relevance and we need to

           6    identify program impact.  On identifying

           7    impact, the guidelines suggest that you

           8    should do this is the guidelines was prepared

           9    by OMB.  The only idea they have is to do

          10    random controlled trials.  I haven't got a

          11    clue how you would do that to effect, to

          12    estimate the impact of a data program.  But

          13    they say, you can look for the data from

          14    someone, give to others I guess.

          15              Anyway the guidance also

          16    acknowledges that this is not always feasible

          17    but it doesn't really give you much of an

          18    effort of a idea of what you could do and so

          19    that we have scheduled for you this

          20    afternoon, after the break is that, Brenda

          21    Cox is working under contract with us and she

          22    is trying to develop a way of doing a program




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            234


           1    evaluation that perhaps we could give to a

           2    contractor or at one point there was a

           3    discussion at the interagency council on

           4    statistical policy and national agricultural

           5    statistic service and the Director suggested

           6    that maybe statistical agencies could

           7    evaluate each other, so you could have a team

           8    of people from agencies that could go and do

           9    evaluations for other agencies and the only

          10    thing about that is that, you have to have

          11    some kind of a template.

          12              You'd have to be able to give them,

          13    what exactly are you going to do and then

          14    maybe they could go off and do it.  Sometimes

          15    we're still and now this is of interest more

          16    than to just EIA because all government

          17    agencies have to answer these questions, so

          18    we're not the only ones struggling with it.

          19    So Brenda is going to talk about her efforts

          20    to start looking at developing a template,

          21    where an approach that could be used to

          22    evaluate survey programs.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            235


           1              Then we were thinking in particular

           2    about families of survey because in EIA we

           3    talk about our petroleum marketing package,

           4    it goes out to find out basically about

           5    prices in the petroleum area, to evaluate the

           6    markets but we have our petroleum supply

           7    package.  It's a family of survey that

           8    captures petroleum supply information.  The

           9    natural gas program gets information about

          10    natural gas.  So we have these little you

          11    know, groups or packages of programs and it

          12    seems like, you know, it might be worthwhile

          13    to assess families of programs and you could

          14    call that a program evaluation.

          15              So that's what Brenda is going to

          16    talk about and then Doug Hale is going to be

          17    meeting the other session.  He's going to be

          18    asking about, how can we get extra

          19    evaluations for models and analysis and of

          20    course he's starting from what we know, we

          21    know about the independent expert review

          22    program and so on and then afterwards we're




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            236


           1    going to come back together and have a more

           2    detailed talk to see if there's anyway the

           3    that you guys, the energy committee could

           4    help us do program evaluation.

           5              So if there is something that you

           6    could do, that we could say, "see we got this

           7    evaluation or this input from our ASA energy

           8    committee" and that's a recommendation that

           9    we could count as external program evaluation

          10    information, so it's a question, how do you

          11    phrase the question, what questions do you

          12    ask, now truly we have to make it clear to

          13    everybody that you're using what you learn,

          14    but that's okay.  That's what makes it useful

          15    to give the advice right.

          16              DR. SITTER:  You said that BOS and

          17    the Bureau of sensors got good scores, what

          18    are they doing?

          19              MS. KIRKENDALL:  Well, they were

          20    early in the program and things are changing

          21    over time.  They are much more stringent now

          22    than they were last year and the year before




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            237


           1    but one of the things that's been sighted as

           2    a good external program evaluation was the

           3    Voskin initiative that worked with the CPI,

           4    for the bureau of labor statistics.  There

           5    was, I guess, they were five academics who

           6    made some specific recommendations that was

           7    charted by the Senate, I guess, to look at

           8    the CPI and that's a very important indicator

           9    used in many government programs and they

          10    made various specific recommendations as to

          11    what BLS should do to fix the CPI.

          12              And they actually went ahead and

          13    they have implemented many of those changes.

          14    I think that was, that ended up in

          15    recommending that they expand the service

          16    sector.  I could be wrong about that but --

          17              DR. SITTER:  But, I've been here

          18    for five years, we make recommendations like

          19    that all the time.

          20              MS. KIRKENDALL:  Well, you know and

          21    maybe you do and then the question is, how do

          22    we package it.  You know because there are




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            238


           1    other pieces that's just, we are reporting to

           2    OMB on this and we were not evidently

           3    convincing enough in what we said.

           4              DR. NEERCHAL:  A quick question,

           5    you know, is "program" an official unit or

           6    when they did the program evaluation, do they

           7    mean like an office by office or department

           8    by department, what do they have in mind?

           9              MS. KIRKENDALL:  That's a good

          10    question.  I think for their purposes, EIA

          11    sells out one program evaluation, with the

          12    energy data program.  I think for our

          13    purposes, we don't have to, I don't think you

          14    can get anything useful out of looking at

          15    something that big but you can look at the

          16    pieces.  That's why I thought you know, we

          17    could focus on various survey programs or our

          18    analysis program or maybe a long-term

          19    forecasting program.  So, Tom do you have any

          20    input, Tom Broene is, was intimately involved

          21    in putting together our PART report and

          22    working with OMB through this exercise.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            239


           1              MR. BROENE:  It has to cover the

           2    entire agency and so I mean, it's got to be

           3    systematic, it's got to be recurring and you

           4    have to actually use the results and have

           5    proof of that, so I mean they can report on

           6    the CPI, this minute things like, how did you

           7    go waiting to take our different type of

           8    average and everything from -- spend more

           9    time pricing, features of cellular phones,

          10    lifetime pricing, the characteristics of

          11    bananas and other things on the servicing --

          12    they covered everything and it's five guys,

          13    did in 18 months.

          14              MS. KIRKENDALL:  But CPI is not a

          15    BOS.

          16              MR. BROENE:  Right, that's true, I

          17    don't know what they did for the rest of BOS.

          18              DR. NEERCHAL:  The reason I am

          19    saying that I think, that many of these

          20    academics here will recognize this thing, you

          21    know, we have to go through something called

          22    program review.  External panel will come and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            240


           1    look at this statistic, the department,

           2    typically department by department and we go

           3    through, they have a 7 year cycle office and

           4    this being initiated you know, officially

           5    this is coming all the way from the board of

           6    regions but somebody on campus, from Dean's

           7    office is usually in charge of it and I think

           8    that independence aspect is really assured

           9    because the Dean's office is involved in

          10    inviting the outside panel, even though you

          11    suggest the panel, they decide and they

          12    correspond with them, that, that should, that

          13    makes them feel comfortable that they are

          14    really independent, even though we know who's

          15    coming and so on right, so it seems to me

          16    that if, that is my followed question would

          17    be, how close the OMB wants to be involved

          18    with the evaluation process because if you do

          19    an evaluation on your own, like you are doing

          20    an intent right now, they don't feel that it

          21    is independent enough.

          22              It maybe that they wanted more




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            241


           1    closer role, you know, and they want to

           2    closely monitor it and make sure that it's

           3    independent, you know, and that's why if it

           4    is and I'm really surprised that they are

           5    coming down to the CPI you know, how you

           6    know, they are drilling down all the way --

           7    I'm really surprised that they have that kind

           8    of time to, you had your programs that is

           9    left, surprise.

          10              MS. KIRKENDALL:  Well now OMB was

          11    not involved with that and now it's a long

          12    time ago too, but that's what BLS can cite

          13    and an external program evaluation you know,

          14    and OMB thinks that was fine, because you

          15    know, it was chartered by the Senator,

          16    whoever, so chartered by an external body.

          17              DR. NEERCHAL:  I think that the

          18    independence, maybe that is the assurance and

          19    somehow they want that to be you know,

          20    demonstrably independent, right.

          21              MS. KIRKENDALL:  Uh-huh, Neha.

          22              MS. KHANNA:  Yes, I think you




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            242


           1    actually hit the nail on the head when you

           2    talked to him and used the word strategy.  I

           3    think a lot of what this committee does

           4    should constitute an independent review and

           5    maybe part of your responsibility should just

           6    be, you know, over the twenty or thirty years

           7    this committee has been in existence to chart

           8    out some of the recommendations, major

           9    recommendations that have come out of this

          10    committee and actually effected the way a

          11    particular program or the EIA as a whole has

          12    operated or you know, done data collection or

          13    is this modeling everything, what better --

          14              MS. KIRKENDALL:  Actually I'm

          15    wonder whether Cal's history that he put

          16    together might have some information, Cal

          17    Kent did that history of the ASA committee

          18    and we can take and look and see if there is

          19    anything we can steal, yeah and --

          20              DR. HENGARTNER:  Los Alamos peer

          21    gets reviewed once a year but the one thing

          22    that I think, that what you wanted




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            243


           1    independent and the one thing that we can

           2    always looked at is, essentially publication,

           3    the things that re out in the open I mean if

           4    you're software's out in the open, if data

           5    gets in the open, it gets peer reviewed,

           6    that's independence review, if it's wrong,

           7    well, the customers are going to let you

           8    know.  It is true.

           9              DR. BURTON:  Absolutely.

          10              DR. HENGARTNER:  And what I don't

          11    understand is how OMB doesn't get that.  I

          12    mean because I think, if everything is in the

          13    open you have to still the review

          14    publications, that's more than any other

          15    places you have and it's mainly a matter of

          16    convincing OMB that, this is how things work,

          17    more than trying to glamour up something

          18    else.

          19              MS. KIRKENDALL:  Well I agree that

          20    especially based on this discussion we can do

          21    more on packaging what we have already have

          22    and showing it and maybe we'll learn better




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            244


           1    how to say, how to describe what we're doing

           2    in a right way, that would be a bit more

           3    convincing, but there are other things that

           4    you know, when I was talking to Brenda about

           5    this is external survey evaluation thing, but

           6    we've haven't done for example, with our

           7    family of surveys, is to really look from an

           8    independent point of view and see how they

           9    fit together you know, does it make sense

          10    that you know, we have exact the same survey

          11    goes overtime or the different people answer

          12    different questions the same way, is there a

          13    better way of structuring the surveys and

          14    still capturing the information we need.

          15              We do that every three years when

          16    we put the forms clearance parents package

          17    together but we don't always step back and

          18    take a fresh look at it because we know how

          19    we've done it and it's hard to see it a

          20    different way, when you know this is how

          21    you've done it, so it could be that some of,

          22    a good outside look at how we operationalize




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            245


           1    some of our programs, would be helpful, it

           2    might provide some new insights, you know, so

           3    that I can see that you might be able to get

           4    some benefits from a good outside assessment

           5    occasionally, if you can find the right way

           6    to ask the questions which is other

           7    challenge.

           8              MS. KIRKENDALL:  Okay.

           9              CHAIRMAN BREIDT:  We are about 20

          10    minutes into fifteen-minute break, so

          11    everybody can just grab our snacks and head

          12    to the break out session directly.  -- 4a

          13              MS. KIRKENDALL:  Then we need just

          14    start the session really 10 minutes late I

          15    guess.  I'd like to introduce Brenda Cox.

          16    She's working with us to come up with a

          17    survey program evaluation process.  So

          18    Brenda.

          19              MS. COX:  Thank you.  I think this

          20    is a joint endeavor between myself and Nancy.

          21    We've been talking about what do you do to

          22    evaluate a survey program.  I should say this




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            246


           1    research is being conducted under a contract

           2    that Battelle has with Z Inc. and we

           3    appreciate it, Rob ---- and other Z Inc.

           4    people are here as well.  This is coming as

           5    the performance assessment rating tool, this

           6    is my interpretation of what that does.

           7              It encourages rigorous assessments

           8    of government programs.  That doesn't mean,

           9    are you doing okay, it means can you do

          10    better.  That would be the way that I would

          11    say.  So you may be doing a fine job, but

          12    there may be some little way in which you can

          13    improve what you're doing.  So we continue to

          14    boost the quality of government programs, and

          15    it provided the impetus to this

          16    investigation, in which I'm going to design

          17    templates that can be used to evaluate

          18    individual surveys, but more importantly

          19    government programs that are composed of a

          20    family of surveys.

          21              EIA does a number of surveys, a

          22    large number of surveys.  They tend to be




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            247


           1    small.  It's not like the agencies for

           2    Healthcare Research and Quality, which does

           3    the math or that cost tens of millions of

           4    dollars.  Whether they -- their programs tend

           5    to be smaller surveys that are, that put

           6    together in programs of surveys.  Now the

           7    objective of what we're going, what I'm going

           8    to be doing is first to develop a survey

           9    evaluation template and that part has been

          10    done, and then go from there to develop a

          11    survey program evaluation template.  And a

          12    survey program is a collection of surveys on

          13    a common topic that is conceptually linked.

          14    The petroleum marketing survey is one

          15    example.

          16              And we're going to actually test

          17    the concept of the survey evaluation template

          18    that's been developed on the 11 surveys that

          19    compose the petroleum marketing program.  And

          20    then the next step will be to test the

          21    concept of the survey program evaluation

          22    template on that same program.  Now, why am I




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            248


           1    not feeling anything?

           2              SPEAKER:  It's once again --

           3              MS. COX:  Oh, it's been brought in

           4    one at a time.  Okay, so the -- the

           5    components of the survey evaluation template

           6    and I should say the paper that I submitted

           7    has the actual template in the appendix.  So

           8    it's the -- it's the template as it exists

           9    now, it is a work in progress.  Since we're

          10    filling this out almost like a form that

          11    would be completed, we start with the name, a

          12    brief description of the survey, the survey

          13    objectives.  Now the survey objectives here

          14    have to be what's driving the design and that

          15    means you have to say what domains, the

          16    domains for which you need separate

          17    estimation.

          18              If this is a survey that has a

          19    strong geographic component and you want

          20    separate estimation for each of the regions

          21    and divisions of the United States, you have

          22    to say so.  Because the design will be,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            249


           1    that's part of evaluating the design.  The

           2    target population and this is -- it has to be

           3    extremely explicit because you actually

           4    review is this an appropriate target

           5    population for a study.  Some people

           6    misunderstand what a target population is, so

           7    they define the target population as the

           8    population that it's easy to survey as

           9    opposed to the population that people are

          10    actually going to make inferences about.

          11              So for a telephone survey they may

          12    define the population to be people who can

          13    hear.  Well those are who you survey, but

          14    that's not the target population because

          15    generally you want to make inferences to the

          16    entire population of the United States.

          17    There's also the sampling frame and it's

          18    characteristics.  We look at the quality of

          19    the frame like, coverage of the target

          20    population, whether there is overlap,

          21    multiplicity, what's done to really create

          22    the sampling frame and make it the best




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            250


           1    sampling frame.

           2              The sample design it says, I could

           3    tell when I came in that you all were looking

           4    at, that this group was looking at sample

           5    design characteristics particularly for EIA

           6    surveys which tend to be economic surveys for

           7    which some particular company can drive an

           8    estimate because of their very size.  And

           9    even if they're not driving the estimate for

          10    the nation as a whole, they may be driving

          11    the estimate for the northeast for instance,

          12    because of their size.  Other components of

          13    the survey evaluation template, the way I've

          14    defined it, I should say, this is what I've

          15    defined, I think I would look at if I'm

          16    evaluating a survey.

          17              And I should also say that I tried

          18    to think of every possible thing you could

          19    look at.  Some of them may not be appropriate

          20    for every survey, but it was just trying to

          21    get everything out there, that you would look

          22    at or consider.  Sample allocation is another




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            251


           1    issue and that you were addressing in fact

           2    this morning for the survey.  It's very

           3    common to create certain selections for very

           4    large businesses.  Data collection, we need

           5    to look at the aspects of data collection to

           6    see, if it's both rigorous and efficient in

           7    the way it's being done.  Editing and

           8    implication is another issue for business

           9    surveys.

          10              There's a feeling that surveys that

          11    -- economic surveys added the data too much.

          12    Now, I'm not talking EIA here, I'm saying

          13    across economic surveys there are people who

          14    say, we do too much editing.  And for

          15    economic surveys the amount of editing that's

          16    done is usually a great deal greater than for

          17    person surveys.  Part of the reason is, the

          18    data items tend to be interrelated and you

          19    want consistency.

          20              If you're doing a balance sheet,

          21    you want your assets to be equal to your

          22    liabilities plus equity.  So there's certain




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            252


           1    relationships that need to hold.  Weighting,

           2    we need to look at the weighting process to

           3    see, first is it following the sample design.

           4    Secondly, how is it adjusting for

           5    non-response, because non-response happens in

           6    all of their surveys, and is it appropriately

           7    capturing the best, doing the best job of

           8    minimizing non-response biased and the

           9    effective under-coverage.  So we look at

          10    things like post-stratification, non-response

          11    adjustments etcetera.

          12              For the data analysis, this is

          13    looking at the way the data are used.  And it

          14    could be used as input for models, like the

          15    NEMS model, it could be used in data reports,

          16    etcetera.  So looking at the data analysis to

          17    say how are the data being used, there might

          18    be a question of how they should be used, but

          19    also I find the data analysis informs me to

          20    as to what the sample design really should

          21    be.  I looked at a survey recently where,

          22    it's like one out of every 2 tables and all




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            253


           1    of their reports had a row or a column based

           2    upon years since degree or age.  In other

           3    words two variables that were closely related

           4    to the age of the person.  This was a

           5    population survey.

           6              Although age was available, and

           7    years since degree were available for use in

           8    the sampling process, they weren't used.  So

           9    that tells you something.  Finally, the

          10    summary of findings, because you want to do

          11    more in the template than just evaluate the

          12    survey and say what are you doing.  What's

          13    good about what you're doing, what should you

          14    improve?  Now I've emphasized and I think

          15    it's very important what's good about what's

          16    being done.  The people who -- the directors,

          17    the managers of these programs, you need to

          18    know this is something you're doing right and

          19    you need to continue doing it.  This is a

          20    good idea.

          21              They also though need to know areas

          22    where improvements could be made.  It might




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            254


           1    be a redefinition of the stratification

           2    variables, extra sorting that might be done

           3    to give more control over particular

           4    variables.  Problems that you're seeing with

           5    respect to the weighting procedures,

           6    whatever, it needs to be there.  And the fact

           7    that you're suggesting an area for

           8    improvement doesn't mean that the survey is

           9    bad.  It just means, it's not 100% perfect

          10    and there's usually always areas for

          11    improvement.  So that's kind of, where we're

          12    going with the survey evaluation template.

          13    Now, when we were discussing how to evaluate

          14    a program I said, I don't see how you can

          15    evaluate a program until you evaluate the

          16    individual surveys that are forming this

          17    program.

          18              Because you have to see the

          19    component parts and then the next step is how

          20    do these fit together.  So we began with the

          21    summary evaluation of each survey and then go

          22    to the survey evaluation template, which




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            255


           1    would be used to profile each the survey.

           2    Now the individual survey template is

           3    available in the paper I wrote.  The survey

           4    evaluation template has not been done yet.

           5              Some of the thoughts for the survey

           6    evaluate -- for the program evaluation

           7    template is to look at issues like, which

           8    respondents report to which survey and what

           9    data requests are made of them.  The reason

          10    for doing this is because even though the

          11    populations may look really different you

          12    could be going to the same companies over and

          13    over again and imposing slightly different or

          14    maybe even the same questions to them.  So

          15    you really need to look at the issue of the

          16    respondents for each individual survey.

          17              And this can be -- this can be a

          18    special issue for economic surveys because

          19    large businesses tend to be taken for every

          20    survey.  You want to identify unnecessary

          21    burden on survey respondents across survey

          22    components and whether specific




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            256


           1    subpopulations are overburdened.  And you

           2    want to determine if the survey program needs

           3    restructuring.  Restructuring might be needed

           4    because you're capturing data in repetitive

           5    ways or you're missing some component that

           6    you should be collecting data on.  Or your

           7    time sequence needs improvement.  There's

           8    different ways that this could happen.

           9              Other areas that could be looked

          10    at, is to determine how the data across

          11    surveys are used in analyses and presented to

          12    the public.  And again this is to give you a

          13    sense for what you should be doing.  Not only

          14    for data analysis but what you should be

          15    doing in your design and reporting.  For

          16    instance the use made of the data, may be

          17    such that it's used for, very timely needs.

          18              For that tells you, you need to

          19    look at the time dimension of the survey.  We

          20    want to delaminate the relationship cross

          21    design between the data reporting and the

          22    sample design.  So we need to look just to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            257


           1    see what's going on across these surveys.

           2    Then there is the issue of synergy across

           3    surveys.  I'll give you an example from

           4    another federal program.  This is the

           5    scientists and engineers statistical data

           6    system, and I've worked on that for a number

           7    of years.

           8              About 10 years ago, they were

           9    evaluated by the committee on national

          10    statistics.  And one of the recommendations

          11    that the committee gave them was you do three

          12    surveys of scientists and engineers.  You do

          13    a survey of doctoral recipients, you do a

          14    survey of recent college grads, and you do a

          15    survey of college grads that are not so

          16    recent.  And they said, you don't put these

          17    surveys together.  You really cover the

          18    entire population of scientists and

          19    engineers, but you're using different

          20    questionnaires and you are not allowing for

          21    their integration.  So one of their strong

          22    recommendations is redesigning surveys so




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            258


           1    that they can be integrated to create one

           2    unified database.  And the National Science

           3    Foundation has done that.

           4              So that's an example of building

           5    synergy across surveys.  The system now is so

           6    much better than the part because before the

           7    college, the doctoral recipients was only US

           8    earned doctoral recipients.  But by bringing

           9    in the other survey of not so recent grads

          10    which included foreign graduates, foreign

          11    doctorates they could represent the whole

          12    doctorate population.  I'm sorry.  Okay.

          13    Another thing that we thought -- I thought of

          14    is that some data items maybe being collected

          15    repetitively across surveys.  We want to look

          16    at the relationship between question ready

          17    and consistency, from EIA's and the

          18    respondent's point of view.

          19              The whole point of designing

          20    questionnaires is to get as close to the way

          21    the respondent is thinking, their records,

          22    what they have available to them.  Decide if




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            259


           1    there is a better design for forms and

           2    instructions.  I think any economic program

           3    has to look at the issue of confidentiality

           4    protection, because the data that they're

           5    collecting is confidential business

           6    information.  And in fact the respondent's

           7    willingness to respond in part is related to

           8    how well confidentiality protection is being

           9    applied.  And that means looking at the

          10    actual data reports too, because some surveys

          11    are -- some businesses are so big that if you

          12    report for a certain entity, you may be

          13    reporting really for them.

          14              If you're a rice producer, you may

          15    be taking care, you may, could be producing

          16    95% of all the rice in Louisiana.  So if you

          17    report for Louisiana, you're reporting for

          18    them and you're revealing their information.

          19    We need to look at response rates across

          20    surveys and to see are there particular

          21    subgroups that are subject greater response,

          22    is there a need to work on the issue of




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            260


           1    response and what might be done to improve

           2    response.  We need to determine if the data

           3    are reasonably timely and understandable to

           4    users.  I said recently timely, because users

           5    often would like to have the data a day after

           6    we collect it.

           7              MS. KIRKENDALL:  Or before --

           8              MS. COX:  Or before -- for the

           9    program it is critical that we document the

          10    strengths of the program and the desirable

          11    data elements already in place.  Again this

          12    is reinforcing that -- it's reinforcing the

          13    behavior you want to see.  But truthfully you

          14    -- we're dealing with professionals who are

          15    trying to do a good job.  And you're

          16    assessing it and they want to come back and

          17    here -- here's all the ways in which you've

          18    done a good job.  That's a fair thing.  We

          19    need to specify areas for improvement and

          20    specific recommendations for improvement and

          21    then suggest enhancements that would increase

          22    the utility of the data across surveys.  Now




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            261


           1    I have questions for the committee.  I do

           2    have something that I can distribute, that I

           3    was looking at, I don't have a slide for

           4    this, but I do have copies of the slides as

           5    well as notes, but the slide wasn't prepared

           6    in time to be distributed.  I think there's

           7    enough for everyone.  I could use one copy of

           8    that.

           9              I should say that this is very

          10    much, this project only started fairly

          11    recently and so the first phase is completed

          12    which is the survey evaluation template, but

          13    not the program evaluation template.  But for

          14    the program evaluation template, another

          15    thing that I started thinking about is the

          16    theory on the value of information and that's

          17    what I'm -- I gave you, attributes of

          18    information value.

          19              And I think this really captures a

          20    lot than we want to look at, the reporting

          21    cycles, in other words, how often are these

          22    data collected.  Partly how often you need to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            262


           1    collect data relates to how swiftly things

           2    change as well as how much money you have.

           3    Sufficiency, that means are you collecting

           4    the data that's needed, are you collecting it

           5    from enough people, that you can make

           6    meaningful use of these data?

           7              Understandability, that's looking

           8    at the end products in particular but it

           9    could be looking at the questionnaire, saying

          10    to what extent are users able to understand,

          11    what we're giving them.  Freedom from bias,

          12    this is pretty critical in many different

          13    ways, we need to be concerned about non-

          14    response bias, under-coverage bias,

          15    measurement bias due to misunderstanding of

          16    the questions we're being asked.  Reporting

          17    delay, that's the delay between when we

          18    collect the data versus when we get the data

          19    out.  And I think every survey needs to look

          20    and say, can we get it, how -- what can we do

          21    to speed up the process, is there something

          22    we can do?  Reliability, I would also put




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            263


           1    accuracy there too.

           2              Part of this is how dependable are

           3    the data we're producing.  People actually

           4    make financial decisions based upon EIA data.

           5    Decision relevance, are we giving them the

           6    data that they need, in other words, that

           7    looks at the content of what's being

           8    prepared, cost efficiency.  EIA is not alone

           9    among federal agencies to have at best a

          10    budget that stays flat for several years.

          11    And sometimes it's reduced, so we always have

          12    to look at the cost efficiency of what we're

          13    doing.

          14              The comparability in economic

          15    surveys and a frequent event is that your

          16    users will have access to data sets, that you

          17    don't have access, that you may or may not

          18    have access to and they'll compare your

          19    results to their data set.  I was at the USDA

          20    during a time when there was a number of

          21    complaints about the, the estimates of young

          22    calves being born.  And their estimate was




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            264


           1    way too large.  And the way they ascertained

           2    it was based upon certain sales and things

           3    like that.

           4              So they had numbers that should be

           5    related to what USDA was reporting and those

           6    numbers didn't agree.  So there are data sets

           7    for which comparisons can be made and the

           8    agencies tend to be held accountable for

           9    that.  Quantitativeness, that relates to the

          10    extent to which the data can be quantified

          11    and the degree of aggregation.  Now against

          12    economic surveys we have to look very

          13    carefully at aggregation because the identity

          14    of the respondent can be revealed through too

          15    much aggregation, too little aggregation,

          16    excuse me.

          17              And in fact there users tend to

          18    want a great deal of geographic understanding

          19    of the data, which is a very big driver of

          20    many aspects.  It's the one thing they want

          21    that's very difficult to provide, because

          22    they want data at a very low level.  So




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            265


           1    that's just something new that I've been

           2    looking at most recently.

           3              For the committee with respect to

           4    the survey evaluation template which was

           5    included as an appendix in the paper to the

           6    committee, the questions I have is, are there

           7    additional items which should be included in

           8    the survey evaluation template?  Do some

           9    items need to be modified?  Is this okay for

          10    evaluation of an individual survey and then

          11    any suggestion for important components that

          12    should be included in the program evaluation

          13    template, the template for the entire

          14    collection of surveys.  And that's it, so I

          15    should bring the -- oh, I lost it all.  I

          16    don't think I can get back.  I'm not moving

          17    at this.

          18              SPEAKER:  Okay, so looking at --

          19              MS. COX:  Did everyone get a copy

          20    of this slide that one in one by the way.

          21              DR. FEDER:  I think every survey

          22    should include some QA procedures that are




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            266


           1    conducted continuously including some things,

           2    I can tell you, I'm not that familiar with

           3    how the operation of the EIA service, maybe

           4    some of what, I'm going to say it's

           5    inappropriate, in this context you have to

           6    modify them.  But we for instance look for

           7    interviewer falsification, verification that

           8    the interview took place.  I know it's

           9    probably inappropriate here.

          10              You look at the date, the time to

          11    complete a questionnaire, again maybe not

          12    appropriate here but you can collect when you

          13    make a telephone interview information, how

          14    long it takes.  You look at outliers, an

          15    outlier is both a calling back to check if

          16    everything, you know, checks out okay.  In

          17    addition if the survey is conducted over

          18    three months, one month into the survey, I

          19    would look at the estimates, partial as they

          20    may be, with maybe incomplete waiting just to

          21    see is there a trend, am I doing something

          22    wrong?




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            267


           1              We discovered the problem was one

           2    of the questions, because it was administered

           3    to very young individuals who misinterpreted

           4    the question.  I can actually, I might as

           5    well give you the example the SM, were you

           6    enrolled in school last year?  And they saw

           7    this the action of signing up for school, not

           8    being at school.  So if that was discovered

           9    simply by looking at the estimate.  So an

          10    ongoing monitoring of the results,

          11    verification, you send a call back, thank you

          12    very much for your participation.  If this is

          13    in error lets us know, call this toll free

          14    number, is another way of checking that

          15    everything works okay.  Again I don't know,

          16    well I think part of it was covered, but I'm

          17    not sure fully if questionnaire testing,

          18    we've spoken about here, I'm not sure, but

          19    it's something else that gives what to do.

          20              MS. COX:  Now that is a fair idea,

          21    I don't think, it's exclusive there.

          22              DR. FEDER:  When I was back in




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            268


           1    Canada, actually this kind of offers this

           2    course and many conferences open to public on

           3    questionnaire design and it's astounding how

           4    bad the results maybe if the questionnaire is

           5    not, because we are professionals.  We know

           6    exactly what a question means and we expect

           7    the answers to match that and often not the

           8    case.

           9              MS. KIRKENDALL:  Johnny?

          10              MR. BLAIR:  Yeah, first of all, I

          11    just want to make the observation that I

          12    think that there is a, to use your word, a

          13    synergy between this template and EIA

          14    procedures, at what they're actually doing.

          15    Since I expect you're, not going to put

          16    something in the template for evaluation

          17    that, you know, that you're not doing.  And

          18    --

          19              MS. COX:  No, I wouldn't say that.

          20              MR. BLAIR:  Well, I would hope not

          21    anyway.  But --

          22              MS. KIRKENDALL:  Well, your




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            269


           1    starting from what you think should be in the

           2    template, without looking at any surveys.

           3              MR. BLAIR:  No, I -- I'm just going

           4    mad.

           5              MS. KIRKENDALL:  It remains to be

           6    seen.

           7              MR. BLAIR:  No, I -- I guess I

           8    wasn't clear.  But after you see the

           9    template, I think that you're probably not

          10    going to pass it on into use until you see

          11    how you're going to address the things that

          12    are in it.  But one thing that I see in this,

          13    what I see is related to what most of it is

          14    some, this notion of quality assurance and so

          15    forth, that I think it's important enough to

          16    have it as a separate item, a quality

          17    assurance plan, that would include, not just

          18    -- I mean the kinds of things that were

          19    mentioned, some things that you already have

          20    in here, for example, the kinds of things you

          21    do to check that the weights are correct and

          22    so forth.  But I think it's important enough




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            270


           1    to have as a separate item, as a separate

           2    document --

           3              MS. COX:  To be evaluated.

           4              MR. BLAIR:  Right.

           5              MS. COX:  Yeah.

           6              MR. BLAIR:  Yes, to be evaluated.

           7    And that would include things like, you know,

           8    is there a re-interview program or is there

           9    something that's done periodically to do

          10    validation of some of the data, that is, you

          11    know a kind of comprehensive, you know,

          12    quality assurance plan.  I think that there

          13    should be a separate item.  Another item that

          14    I think and this is kind of related to that

          15    but I think it should be there, is something

          16    about user documentation.  You know, what is

          17    it that's provided with the data sets,

          18    whether online or in whatever form for the

          19    users.  Some assessment of that, and the

          20    third thing I mentioned that, again it's sort

          21    of implied but I think it's not exclusively

          22    here, it's sort of the people's side




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            271


           1    interviewer training, supervision, those

           2    kinds of things, going to give some of that,

           3    you know, what overlap it would be in the

           4    quality, the quality assurance plan.  Now,

           5    one other question if I may to follow on,

           6    many -- I'm not sure exactly how it's

           7    intended, that the template will be used by

           8    the outside review or the outside expert.

           9              Is this just provided as a document

          10    to say you know, there are just some things

          11    to keep in mind, or is there something more

          12    specific in terms of what instruction is

          13    given to the expert reviewers or the outside

          14    evaluators or whatever they are called and

          15    understanding the summery of findings that

          16    you want back, but it's -- it were intended

          17    to be as much guidance as is given or is it

          18    -- is it intended that it be left fairly open

          19    for them to decide what they get back to you

          20    or is that something you're going to be doing

          21    more of?

          22              MS. COX:  My intention was that




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            272


           1    this would be all different things that you

           2    could look at.  I was thinking on myself,

           3    critically evaluating a survey.  And what I

           4    would look at and think about, now something

           5    are very particular in there, they don't

           6    apply to all surveys.  Like some are male

           7    surveys, they'd have a different way, than a

           8    face-to-face survey.  But it was all the

           9    different things that you would look at if

          10    you were concerned about evaluating the

          11    quality of the survey.  You said something

          12    that was interesting.  I would hope that this

          13    template would be distributed to everyone

          14    doing the survey.  Because first of all they

          15    should be documenting the kind of things that

          16    are in that template and I think the very

          17    process of documentation can identify

          18    problems.

          19              MR. BLAIR:  Sure.

          20              MS. COX:  So that I would think

          21    that one way the evaluation template can

          22    help, is it tells people what to look at and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            273


           1    think about and it spotlights the problem

           2    even before they're evaluated they can say,

           3    oh wait a minute, you know, this is talking

           4    about this, I think we've got a little

           5    problem here we need to address.  So it's

           6    almost like giving someone a cheat sheet, but

           7    it's a cheat sheet that will allow them to

           8    address problems and know that something is a

           9    problem even before an outsider has to say it

          10    is a problem.

          11              MS. KIRKENDALL:  The other thing

          12    that, thinking about how it might by used is,

          13    if we're going to hire somebody to do an

          14    external evaluation or somehow otherwise

          15    we're talking of doing it.  It's going to be

          16    cheaper if we can send them documentation

          17    that has all the information they would need

          18    to do the survey.  And I'm not sure EIA

          19    documentation is in that stage yet, so we'll

          20    find out.  The other way they can do it of

          21    course is come in and talk to everybody, then

          22    try to ferret out the information that way or




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            274


           1    some combination of the two.  So if it's

           2    based on documentation, then you like to have

           3    your documentation cover all these pieces.

           4              MR. BLAIR:  And I guess one piece

           5    of that, just responding to that would be a

           6    methodology report and which is different

           7    from this user documentation and responsive

           8    data files that I would have there, I would

           9    want an outside reviewer to evaluate.  Now,

          10    the methodology report has in it what it

          11    should have the collection of details.

          12              DR. HENGARTNER:  Let me start by

          13    saying that, in my opinion EIA is ahead of

          14    the curve here.  Because I mean we've been

          15    talking about data quality, data continuity

          16    --

          17              SPEAKER:  Continuity.

          18              DR. HENGARTNER:  Thank you, there

          19    are some words I just can't say.  We've been

          20    discussing that I mean for years now, I mean

          21    this is really not a new subject then, it is

          22    also, I'm a little bit worried about




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            275


           1    something.  You know, what I'm worried about

           2    is that we're going to have these evaluations

           3    and we've got to look at the ways of listing

           4    them, oh no this data here has a high rate of

           5    missing than that one, so that's a bad study.

           6    When in fact we know you know that, in some

           7    areas the data is much much harder to get

           8    than in other areas.  And so I -- and the

           9    problem is clearly that we don't know what

          10    level of achievement we're looking for,

          11    right?

          12              You say, it is to improve the data

          13    collection and so forth, of course, what is

          14    achievable?  I mean, it's like safety and

          15    security, I mean we talked about that thing

          16    yearly, all the time.  Their incidence all

          17    the time is what we're striving for but

          18    things happen, and same thing and they say

          19    that we're trying to do the best possible but

          20    we need to know what is realistic that --

          21    that's been, what are the bench marks?  And

          22    having a template without the bench marker,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            276


           1    knowing what to expect is very hard because

           2    it might actually be disheartening to people

           3    when they receive the report.  And things

           4    that are missing on the practical level, we

           5    don't talk about customers directly.

           6              At EIA, each time we talk about

           7    data quality, we always go back, and there's

           8    lot of customers, why do we do it in the

           9    first place?  And yes it's touched a little

          10    bit by survey objectives, but I would

          11    actually list it, who's using it and why are

          12    they using it.  There is also something else

          13    that I would put in is, we can talk about for

          14    example the missingness, I mean to say, to

          15    talk about timeliness, missingness and so

          16    forth.  But the other thing that I would urge

          17    you to put into the document is the cost

          18    benefit.

          19              How much is the cost to reducing

          20    the -- to rate of missing it?  Yes you can

          21    reduce it, but it's going to cost so much

          22    that it's not worthwhile.  We are going to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            277


           1    make a decision based on that.  And so having

           2    this explicit view will somehow lessen the

           3    comparison, because they say we don't know

           4    the benchmarks, but knowing how much it

           5    costs, that is going to be, the OMB will love

           6    that, I think.  Yes, there's something else

           7    also, yes, you mentioned the overburden,

           8    usually when you take a program, like let's

           9    say, natural gas or petroleum, the EIA tries

          10    to map the whole spectrum from producer to

          11    consumer.

          12              And because the producers, unless

          13    the consumers specifically know the

          14    transporters, you don't tend to sample the

          15    same people.  So you are talking about,

          16    within each program to look at redundancy.

          17    My fear is that actually the redundancy will

          18    come by looking across programs.  That means,

          19    you look at the producers, right.  And you

          20    ask, you're going to ask about petroleum and

          21    then you're going to knock on their doors for

          22    electricity and then you're going to on this,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            278


           1    and you see how that, there is going to be

           2    the repetition that I think there is, I mean

           3    Nancy is going to correct me on this.

           4              MS. KIRKENDALL:  Actually you're

           5    right, and they know it.  For example, I mean

           6    we had our refinery surveys, we've had a lot

           7    of surveys in the filling and supply area,

           8    they also respond on the electric power

           9    survey as a combined power plant.  A

          10    different label, but they're in there, at

          11    least a lot of them are.

          12              DR. HENGARTNER:  Yeah, absolutely

          13    and so that's where the redundancy would come

          14    in.  And so by evaluating them program by

          15    program, you actually miss exactly what

          16    you're trying to review.  So --

          17              MS. KIRKENDALL:  Brenda has talked

          18    about it, we're starting, you look at

          19    families are pro- reactive, but once you do

          20    that, I think you can start looking at

          21    combined programs, so it will be you know --

          22              SPEAKER:  Well, the whole EIA.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            279


           1              MS. KIRKENDALL:  The whole EIA.

           2    People say that there are similarities in

           3    natural gas and electric power, there should

           4    be some commonalities, other than totally

           5    different offices, they do nothing the same.

           6    And as far, you know, look at those two

           7    together, oh they would be some benefits we'd

           8    probably find.

           9              DR. HENGARTNER:  I think the, just

          10    last point, I think the confidentiality that

          11    shouldn't stand also, I mean for the whole

          12    EIA.  I mean how do I -- how do we ensure

          13    confidentiality, this is so important.  And I

          14    know you stressed it, but I think this is,

          15    that's the reason why I mean, we're getting

          16    the data in the first place.  And if we

          17    cannot guarantee that then we are not doing

          18    our jobs.

          19              DR. FEDER:  And it's the law.

          20              DR. HENGARTNER:  And it's the law.

          21              MS. COX:  Yes.

          22              DR. HENGARTNER:  The bottom line is




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            280


           1    we've been talking about this all along.  So

           2    I'm not sure what this is going to bring new

           3    to the process and that's an open question, I

           4    don't know.

           5              DR. FEDER:  I wanted to second what

           6    Johnny said about implementation and I'd like

           7    to recommend that this be an item in every of

           8    the evaluation that the level of

           9    documentation of the survey is part of the

          10    items which should be checked at, and also

          11    what really you're talking about, something

          12    that that would call data consistency, like

          13    the calves and the number of calves actually

          14    born versus purchased.  And in fact sometimes

          15    you get similar estimates from different

          16    surveys that don't match and I remember again

          17    back in Canada where the ultimate citizen

          18    pointed out a difference between two

          19    estimates and their follow up of that

          20    uncovered a certain estimation issue that was

          21    discovered.  So I would call it data

          22    consistency.  I -- and there's been good data




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            281


           1    consistency because in surveys I think

           2    there's another thing called consistency of

           3    data, yeah I guess, that's how we call it.

           4    Data consistency, it's at a program level not

           5    at the survey level.  Of course sometimes

           6    within the surveys too.

           7              MS. KIRKENDALL:  Go ahead.

           8              MR. BLAIR:  Nicholas mentioned that

           9    you don't have, in many cases sort of

          10    benchmarks for what you should do to -- what

          11    you do have is a baseline and I think it's

          12    useful to look at some of these measures,

          13    response rate is an obvious one item, you

          14    know, response rate of what we see over time.

          15    I don't know what the plan is or how

          16    frequently these evaluations would be done,

          17    but one measure is sort of you know, are you

          18    getting better, are you sort of you know,

          19    holding your own, are you getting worse, are

          20    you holding your own but for higher cost, you

          21    know, those kinds of things I think can be

          22    looked at.  And because you know this is




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            282


           1    probably, you know, the more elaborate it

           2    becomes, the more expensive it's going to

           3    become, time consuming.

           4              And you may not do this as

           5    frequently as you would like, but what might

           6    be worth looking into this is sort of partial

           7    evaluation.  That is if you may, you know,

           8    you do this is first time and it's this, you

           9    know, massive effort and also you get lots of

          10    good information and you identify some things

          11    that are you know candidates for improvement

          12    to follow up the evaluation, you know, and

          13    doesn't have to miss, I mean it can be the

          14    whole thing again, but it could also be a

          15    follow-up on those particular points that

          16    were identified and I think again, that would

          17    something, if OMB ever remembers looking at

          18    improvement but, you know, but things that

          19    could be, you know, like sort of --

          20              MS. KIRKENDALL:  And the other

          21    thing, just remind you, we talked last time

          22    about our survey process that Tom was trying




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            283


           1    to measure and Shawna was working on that

           2    too.  And so that there are lots of different

           3    ways of getting evaluations -- about hearing

           4    form our program managers what they're doing

           5    and then Tom and Shawna went towards them to,

           6    sometimes a little more in there but -- so

           7    they're very similar and this is the outside

           8    view and how do we bring it all together and

           9    make sense out of the whole thing.

          10              DR. SITTER:  I guess we're

          11    consistent.  I only wrote down three things

          12    during the talk, one was cost efficiency, one

          13    was document, decisions and trade offs and

          14    the other one is which hasn't really been

          15    talked about is redundancy, is still very

          16    much related to issues, data consistency

          17    issue.  If you evaluate too much and decide

          18    that you're sort of being redundant, you can

          19    get rid of all of your redundancy, which is

          20    not a good thing.

          21              MS. KIRKENDALL:  Not really.

          22              DR. SITTER:  Not necessarily.  On




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            284


           1    the issue of reducing costs and

           2    documentation, documenting, I think the

           3    natural gas discussion is a good example that

           4    happened this afternoon.  The way that this

           5    worked in the past may not be -- have been

           6    very useful for you.  I mean, we make a

           7    recommendation you do a simulation and say,

           8    we don't like your recommendation, we're

           9    going to do something else.  We come back and

          10    say, they didn't do the simulation very well,

          11    then you come back and say well there're

          12    other reasons we didn't do it, but

          13    documenting that decision, the trade off that

          14    you're making is important.

          15              That is in the end of the day it

          16    comes down to cost, simplicity, data

          17    availability, data quality, sometimes, you

          18    know, being able to get good quality data,

          19    you know, your data quality from the big

          20    producers might be much much better and that

          21    by using those as an estimate for the guys

          22    know you're going to have trouble, I mean,




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            285


           1    it's a good trade off to me.  But that may

           2    not appear anyway.  Other than the minutes of

           3    this meeting it says, it's not, maybe not a

           4    good thing for you to do.  And I think that

           5    that's why, so an external reviewer comes in

           6    and you know depending on what they look at,

           7    they can say that you're ignoring, but I mean

           8    that would be easy enough to say, and that's

           9    just not true.

          10              MS. KIRKENDALL:  Right.

          11              DR. SITTER:  I mean it's a process,

          12    right, we're part of that process.

          13              MS. KIRKENDALL:  Yes.

          14              DR. SITTER:  And I think

          15    documenting those kind of trade off's is

          16    really important.  Because when you look at

          17    these things sort of first glance, everybody

          18    is going to have the same thing, your going

          19    to look at, say you decided on this 90% cut

          20    on survey, and they're trying to evaluate

          21    whether that's a good thing.  The things that

          22    are going to run through their minds will be




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            286


           1    the things running through our minds, why did

           2    you do this, why did you do that.

           3              If none of that appears there, if

           4    we seriously consider doing this and that,

           5    and you were even advised to do this and that

           6    and you knew the reason that we didn't do

           7    that.  I think that it doesn't take much, but

           8    especially the really obvious one, a report

           9    like, you know, why didn't you try to get a

          10    better response rate and why you decided to

          11    have a outlined this way or you felt it was

          12    okay and I'm sure you do many of them but --

          13              MS. KIRKENDALL:  Oh, we could

          14    definitely get better in documenting and I

          15    think the idea  -- one of the things that we

          16    are doing now that we started doing at least

          17    for a couple of years is we do have a summary

          18    of each session with the ASA committee and we

          19    say what we talked about, what your

          20    recommendations were and what we did about

          21    it.  And it's not very expensive but at least

          22    that went on.  And maybe that in some




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            287


           1    situations we need to be more careful and

           2    this natural gas discussion I think it's one.

           3    I think we need to fine-tune our simulations,

           4    so that we have something else to point to.

           5              DR. SITTER:  I think that another

           6    thing about, despite how much funds we have.

           7    I think that over the five years I've been

           8    here, the way that it's viewed by you as a --

           9    an external review process has increased.  I

          10    think that you think of it more that way.

          11    But still it's certainly not as formal as a

          12    formal review process would be.  Now I'm only

          13    having a year left.  But you know, things

          14    like, you're getting -- doing summaries now,

          15    they weren't being done before.

          16              Writing up documentation and

          17    attaching them to the surveys, that are being

          18    talked about, so the things that would make,

          19    you know, make it a truly more independent

          20    external view.  I think in some cases it's

          21    happening, in other cases not -- over the

          22    pipeline.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            288


           1              MS. KIRKENDALL:  Right.  I think

           2    you're right that most of our, right now what

           3    we're doing is we're bringing things to you

           4    in an early stage and we actually, we -- I

           5    mean you -- we use your advise, you know,

           6    next week.  And so there has been an impact

           7    on what we do, but it isn't well documented

           8    probably because of the fact that it's so

           9    early.  So maybe one of the things we need to

          10    do is pretty close the loop at some point,

          11    once we talked about things and we have a

          12    final product and then have a discussion of

          13    that because we all agree this was a good

          14    prospect and we have to get good results,

          15    that's something.

          16              DR. HENGARTNER:  Two meetings ago

          17    we had a discussion about how this committee

          18    works, namely what mention that the EPA also

          19    has advisory committee that they write

          20    reports, big reports.  And that of course,

          21    you can point to that and say, aha, that we

          22    have been reviewed, we have a report.  It is




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            289


           1    our fault that, I mean I don't want to write

           2    a report.  And I'm going to be honest here,

           3    but it is a fact that if you would be writing

           4    reports, then there would be something to

           5    point to in saying, yes this -- we are a body

           6    that independently reviewed the EIA on a

           7    regular basis.

           8              DR. SITTER:  There are ways to do

           9    that, but  for example each one of us

          10    summarize -- type that up and send it to the

          11    person that will read it, make sure that they

          12    tape read what's been said -- you know, you

          13    have one person who's identified as the

          14    person and so far that can be done reasonably

          15    easily and it's not a big onerous deal, all

          16    right, it's just an e-mail to read -- you

          17    know.

          18              MS. KIRKENDALL:  Okay, if it's not

          19    that long.

          20              DR. SITTER:  No you just -- it's

          21    going to be a page or something, so you can

          22    read it, you can see whether you agree and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            290


           1    then the other thing is keeping those

           2    together.  That the other thing I've had

           3    difficulty over the years, it's getting

           4    better, oh ok so the percent standards(?) we

           5    met on this two years ago.

           6              MS. KIRKENDALL:  I don't remember

           7    what that was.

           8              DR. SITTER:  I don't remember that.

           9    Was I here at that -- did I attend that

          10    meeting, was that my break out session?  This

          11    also happens right is that you don't

          12    necessarily get assigned to the continuing

          13    breakout sessions right, the same group.  So

          14    a historic -- a history certainly --

          15              DR. HENGARTNER:  This is an idea of

          16    how to more formally be able to package this,

          17    because I mean the fact that we've been

          18    talking about all these things for years now

          19    and I think I mean you're (inaudible) that is

          20    true -- I mean, we talk, we listen, there's a

          21    good synergy.  We're very happy that you

          22    bring up this stuff early on because that's




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            291


           1    where we have the most impact.

           2              MS. KIRKENDALL:  That's right.

           3              DR. SITTER:  And even -- over the

           4    years, maybe one of the strongest

           5    recommendations, like how this committee

           6    should work.  I mean Carol put a lot of - did

           7    a lot of complaining on our behalf, about how

           8    we need the material earlier, we want to be

           9    in the process earlier, if you want to -- so

          10    I mean it's almost sort of self evaluation of

          11    your quality process or is a part of the

          12    quality process.

          13              MR. BLAIR:  When you were giving

          14    the overview and the introduction, Randy

          15    asked -- I think it might also be useful to

          16    find out how they presented it, I mean this

          17    motion of packaging that a lot of the things

          18    are already there and conveying that you

          19    know, that we do have a process in place and

          20    that we do have something that's not just

          21    when a bunch of people get together every now

          22    and then -- you know you have some systematic




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            292


           1    way of eliciting information of using it and

           2    giving feedback and so forth.  So you know --

           3    finding out how some of the other agencies

           4    convey what they're doing may also be

           5    something that could be done.

           6              MS. KIRKENDALL:  And also as I said

           7    I think that the ones that we've been able to

           8    see were the reviews that were done last year

           9    and the year before and they are changing,

          10    there were things that were agreed as

          11    providing outside expert advice last year. So

          12    they're getting tougher.

          13              DR. HENGARTNER:  Yes.

          14              MS. KIRKENDALL:  But I do think

          15    that the packaging is like a lot of us, and

          16    so we really need to be.  Well, we did this

          17    in a fairly short period of time and we

          18    probably didn't engage everybody in the

          19    agency that could have provided some good

          20    input because you know, when you do something

          21    in a short period of time.

          22              MS. WAUGH:  And also since OMB is




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            293


           1    interested in standards EIA has and standards

           2    that relate to the survey process and on

           3    survey results that if they could put some

           4    kind of matrix, this is what a packaging is

           5    for EIA, that would demonstrate that we are

           6    meeting our standards whether IT programs

           7    that do rounding procedures or our response

           8    rates that meets the OMB response rate

           9    requirement.  I mean there all the surveys

          10    meet that requirement but if we could at

          11    least demonstrate that we're monitoring that

          12    information and that we can identify which

          13    surveys meet those standards.  That may also

          14    be a way of communicating to OMB that we are

          15    aware -- both aware we need to do the

          16    improvements, but also aware we're either

          17    meeting or exceeding their expectations.

          18              MS. KIRKENDALL:  Well actually that

          19    is being done by a different part of OMB than

          20    the parts that does the standards, they don't

          21    talk to each other.

          22              MS. WAUGH:  Oh okay.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            294


           1              DR. HENGARTNER:  Is anyone auditing

           2    the OMB?

           3              MS. KIRKENDALL:  And what they were

           4    particularly -- it wasn't so much that they

           5    didn't think that we had high quality

           6    service.  I think they do believe that we do

           7    have high quality service, I think they

           8    actually think we think pretty highly of OMB.

           9    In fact I think our budget examiner tried

          10    hard to us good grades on some of those and

          11    she'd come and she talked to us and she'd

          12    given us information about what she thought

          13    and then she'd go back and talk to some

          14    mysterious people at OMB and our grade would

          15    go lowered.  So you know they're working on

          16    the standards, they're trying to raise the

          17    level across the Federal government and that

          18    means they were playing, it was a whisper,

          19    where you whisper to me, I whisper in this

          20    guys ear and he whispers in the next ear and

          21    who knows what 10,000 Johnny finally tells us

          22    what we've said.  So you know, we put our




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            295


           1    ideas together and we talk to Laurie and

           2    Laurie goes and talks to whoever and they

           3    came out with a grade -- so we need to be

           4    able to come up with, sometimes documentation

           5    or evidence that is clear to everybody.  So

           6    even when you whispered around the table you

           7    say, oh yeah.  Clearly a program evaluation

           8    for --

           9              MS. WAUGH:  The other part is that

          10    you presented earlier on and then Brenda

          11    talked about her 3 tiers of program

          12    assessment and maybe -- and this is only a

          13    disconnect for me, maybe other people saw the

          14    connection but you talked -- that OMB wanted

          15    to know about relevant effectiveness or

          16    efficiency, I can't remember, it might have

          17    been both.

          18              MR. BLAIR:  Effectiveness.

          19              MS. WAUGH:  Okay, anyway there were

          20    five or six things listed, and they've been

          21    provided some guidance on these things that

          22    we don't think are feasible but I didn't see




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            296


           1    the top tier of Brenda's addressing

           2    relevance, and to me there needs to be a 4th

           3    tier or something that really packages what

           4    Brenda's proposing that actually is

           5    communicating it in a way that OMB is asking

           6    for it.  This is the packaging issue.

           7              MS. KIRKENDALL:  You've got to have

           8    relevance and the effectiveness somewhere in

           9    there.

          10              MS. COX:  I'm looking at it as a

          11    Statistician and these are -- as a survey

          12    statistician -- and these are surveys.  So as

          13    I'm writing it with more phenomenon,

          14    statistical perspective, there's the subject

          15    matter perspective in which you say gee we're

          16    collecting this, we should be collecting

          17    that.

          18              MS. KIRKENDALL:  And maybe that's

          19    another thing that we should be looking at in

          20    terms of a program evaluation, instead of

          21    having a statistician say are we doing things

          22    right, have an outside expert look at what




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            297


           1    we're doing and say are we doing the right

           2    thing.

           3              DR. HENGARTNER:  Well but that goes

           4    back to the customer.

           5              MS. KIRKENDALL:  Yes, it's going

           6    back to the customer.

           7              DR. HENGARTNER:  What's the

           8    relevance, I mean if you have the customer up

           9    front then -- your template, then you have to

          10    know them.

          11              MS. COX:  I think that it's a part

          12    of the process you say what are we measuring,

          13    how are we measuring it, which is the best

          14    way to measure it?  Because if you decide you

          15    need these changes in what you're measuring

          16    -- well it's still evaluated how you've been

          17    measuring it and you can make changes there

          18    so I don't see the two as being -- I have to

          19    have this done before I can do this.

          20              DR. SITTER:  I think that one

          21    aspect of meeting that might want to

          22    emphasize the relation with customers, is




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            298


           1    that it's open.  I mean I'm sure a lot of

           2    external evaluations are not open.  These are

           3    public meetings, any of your customers can

           4    come and see what you're doing they can even

           5    evaluate -- they can criticize they process

           6    by which you're trying to make your process

           7    better.  You're wide open and out in the open

           8    and I think that, that's what Nick was saying

           9    about your data, I mean it's out there, the

          10    public has got it.  What better quality check

          11    than their response.

          12              MS. NORMAN:  In fact there was a

          13    small point that I was going to bring up.  It

          14    was actually when you mentioned the

          15    documentation and you referenced the examples

          16    of natural gas problems that we were talking

          17    about earlier and there actually is a mild

          18    kind of internal documentation we had set for

          19    these things you know, this is why we chose

          20    the 90% cut off versus this and we have all

          21    this kind of stuff.  I think one of the

          22    issues that might need to be raised, is maybe




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            299


           1    we should be more candid with that

           2    information.  Maybe that information should

           3    be a little bit more readily available, needs

           4    to be moved over and maybe formalized a

           5    little bit more but maybe we should be

           6    sharing that and then only -- I have one

           7    other point for your template, but I don't

           8    know if it was already suggested or not, I

           9    might have missed it.

          10              But within the template that you're

          11    handing out the independent reviewers maybe

          12    one of the things that should be part of the

          13    template is asking the independent reviewer,

          14    what did we miss.  Could you have provided us

          15    with any other topics that we should be

          16    reviewing that we missed in this template?

          17    It's just a quick simple question but they

          18    might come back with something that we just

          19    haven't thought of.

          20              MS. KIRKENDALL:  But how we've been

          21    doing with the 914, we have done a lot of

          22    stuff with the data and one of the things




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            300


           1    we're contemplating or I think we would like

           2    to do is to put together a topic contributed

           3    paper sessions for ASA.  Have Preston talk

           4    about simulation studies and we'll talk about

           5    samples in general and maybe have John talk

           6    about the sampling an estimation method and

           7    how it's working in practice so.  It's

           8    because there are a lot of interesting bits

           9    and pieces for this question?

          10              DR. SITTER:  Is there any chance of

          11    getting that data?

          12              MS. KIRKENDALL:  Probably not, I

          13    think that's putting together from purchased

          14    versions.  I wish I could share it because I

          15    think we could get a lot of good, really good

          16    part with this one.  Yeah that's some

          17    interesting data there.

          18              DR. HENGARTNER:  Well there's still

          19    a summer internship right Hardy?

          20              DR. SITTER:  You've got to have a

          21    student.

          22              MS. KIRKENDALL:  Yes there is a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            301


           1    summer internship, that's a possibility.

           2    There's another proposal that's out there and

           3    I haven't heard yet.  Joe I don't know if

           4    you've talked to John about how he feels

           5    about the data that you'd like to use in your

           6    --

           7              MR. SEDRANSK:  John suggested that

           8    it was perhaps easier to get the data

           9    statistics he was talking about.

          10              MS. KIRKENDALL:  Okay that's

          11    interesting, so he thinks that you could have

          12    access to the data that he's using now?

          13              MR. SEDRANSK:  Yes that's what he

          14    said at lunch.

          15              MS. KIRKENDALL:  Oh well that's

          16    interesting, maybe if you can get it, it can

          17    be more broadly shared.  We'll have to find

          18    out, he's used data sources.

          19              DR. SITTER:  If you want you can

          20    blind him at whatever level.

          21              MS. KIRKENDALL:  Yes, so there may

          22    be something that could be done -- yes.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            302


           1              DR. SITTER:  But I must admit my

           2    interest might be more academic than advisor.

           3              MS. KIRKENDALL:  It probably was

           4    but on the other hand you came up with

           5    interesting observations you might very well

           6    share them with so.

           7              DR. SITTER:  I'd have to see if I

           8    was allowed to release that information.

           9              MS. KIRKENDALL:  But anyhow that

          10    would be -- that would help us to put

          11    together documentation to -- if we wrote a

          12    bunch of EIA Topic contributed paper sessions

          13    so that it would be easy to take from the

          14    package and EIAs documents that describe the

          15    work from where we got lurking around out

          16    there --

          17              SPEAKER:  Okay.  Great, so we're a

          18    little behind but not too bad, so I guess

          19    it's time to go ahead and summarize the

          20    breakout session.

          21              DR. HENGARTNER:  We had a

          22    presentation about the idea of designing a




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            303


           1    template that would essentially guide

           2    external reviewers in surveying or evaluating

           3    in divisional surveys and in families of

           4    surveys and eventually the whole EIA package

           5    -- well the whole EIA.  We went over

           6    essentially what I think we've been

           7    discussing over the last two years about data

           8    quality, confidentiality and all those other

           9    goodies that are in this.  There was quite a

          10    discussion afterwards beyond just the

          11    specifics, an outcome to the specifics that

          12    we thought we could supplement to the

          13    presentation.

          14              There was also some discussion

          15    about why we the ASA committee didn't qualify

          16    as an external committee and how we could

          17    package our work here more efficiently, in

          18    particular it all came down to documentation.

          19    I think that was one of the late motif of our

          20    discussion with documentation, if we could

          21    document what we're doing here, if we can

          22    document the process by which decisions are




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            304


           1    made and if that documentation would be

           2    transparent then that would provide a good

           3    spark in showing how independent we are.

           4              About the specifics there's a

           5    number of suggestions for example, that this

           6    template should include notions of interview

           7    falsification, they should look at time to

           8    complete the questionnaire that outliers.

           9    There was also very good points made about

          10    continuous monitoring and I'm -- the ideas is

          11    that not only should we look at survey in

          12    separate, we should monitor them and try to

          13    see that we can anticipate "boo-hoo's" before

          14    they happen and Meshe gave us a few good

          15    examples on how to do that.

          16              There was also the notion that when

          17    we talk about making this template, the

          18    template was done fairly independently; I

          19    hope that's correct.  But I there was a

          20    synergy to be gained between EIA, which is

          21    thinking a lot about data quality and so

          22    forth and the template and somehow I'm sure




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            305


           1    that will happen as the template will go to

           2    EIA and then we are going to look into the,

           3    or maybe we should add this, we're already

           4    doing it anyway, right.  And then update it

           5    in that sense.  And there was however, some

           6    question still about how the template will be

           7    used.

           8              We didn't know too much about that,

           9    other things again, I think everybody talks

          10    on touched on this documentation.  I mean as

          11    long as there's good documentation for the

          12    surveys, that's going to go a long way in

          13    helping us evaluate those surveys.  So, it's

          14    not only the template but really gets the

          15    documentation in place.  The last thing that

          16    I'd like to mention is cost, that's also

          17    something extremely important when you think

          18    about evaluating the survey is cost and cost

          19    benefits for example, improving aspects of

          20    the survey and that allows us then to say,

          21    well this is a thing that we can give in our

          22    budget, you give us twice the money we'll do




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            306


           1    better.  And that might be a good thing to

           2    actually have, and so this is a short summary

           3    and I'm sure the other members here can chime

           4    in and see if I have left out anything else.

           5    But that's really it.

           6              CHAIRMAN BREIDT:  Thank you Nick,

           7    any questions or comments?  Okay, then I'll

           8    go ahead and launch into the other summary.

           9    Susan Sereika was going to be here and

          10    summarize this session but she has a detached

          11    retina and Bill said, well she's got two eyes

          12    so what's the problem.  He was joking, he's a

          13    nice guy.  So I'll go ahead and summarize

          14    this.

          15              Actually Susan was all set up to

          16    call in and conduct this by conference call

          17    but her doctors' appointment ran late.  So

          18    there's dedication, a shining example for all

          19    committee members here.  Okay so let me

          20    summarize.  This is Doug Hales's break out

          21    session, independent evaluation of forecast

          22    models and analysis and Joe Sedransk provided




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            307


           1    a lot of input as well, in terms of panel

           2    discussion and Tom Broene gave us a lot of

           3    background on the OMB reporting process.  So

           4    Doug started with a little history on the

           5    independent external reviewer process and he

           6    gave us a little bit of background on the

           7    PART review that was received from OMB and

           8    actually kind of broke down this "No" that we

           9    saw on Nancy's slide.

          10              So basically there is no recurring

          11    independent evaluation of sufficient scope

          12    and no independent evaluations of the entire

          13    program, these were kind of quotations lifted

          14    out of the report.  So it's really the scope

          15    that seemed to be the issue and it did not

          16    seem to be the case that you could just

          17    repackage the existing materials and have

          18    that fly for OMB.  What was really missing

          19    was the macro level questions like are these

          20    analysis topics appropriate, what's the

          21    effectiveness and impact of these analysis,

          22    is the program attaining it's objectives and




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            308


           1    are program resources being allocated

           2    effectively.

           3              So these are not question that are

           4    addressed by the independent expert reviewers

           5    who are looking at individual components and

           6    they are not addressed by putting your code

           7    into the public domain and so on.  So these

           8    are the kind of macro level questions that

           9    were of interest and the committee generally

          10    agreed that what was needed here was a

          11    high-powered review team.  So this is going

          12    to be the HPRT from here on.

          13              And this could be modeled after a

          14    scientific advisory committee or an academic

          15    external review team.  So these people would

          16    need to have some familiarity with EIA

          17    products and methods, but no real interest in

          18    the outcome, so the standard sort of external

          19    review team that you want to put together.

          20    The thinking is that EIA would need to do a

          21    lot in terms of their current micro level

          22    evaluation and so putting together citation




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            309


           1    and plan, results from IERs and documenting

           2    the source and motivation of all the

           3    different analytical products they put

           4    together, did this come from legislators, did

           5    it come from industry, did it come from

           6    Secretary of Energy or whatever.

           7              And this is the painful part this

           8    is like an academic self study that takes a

           9    lot of effort to put together and you piece

          10    this thing together.  That's what you hand to

          11    me high- powered review team, the HPRT and

          12    then they come with answers to the macro

          13    level questions.  One of the things that

          14    would also need to be in this micro level

          15    review would be a detailed review of some

          16    limited number of analytic products, so this

          17    might be done on kind of a rotating scheme

          18    where you missed out all the analytic

          19    products that are put together by EIA.

          20              Some of those change relatively

          21    slowly in time some are clearly changing more

          22    rapidly in response to current needs so you'd




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            310


           1    rotate through the list with longer rotation

           2    times for the stuff that changes less

           3    frequently.  But at some point in the review

           4    cycle you would need to have a detailed

           5    review of these products that again goes into

           6    the summary for HPRT to look at.  Another

           7    comment here was that this entire -- all this

           8    stuff put together needs to be absorbable by

           9    the HPRT in a relatively short period of

          10    time, because they don't have an infinite

          11    amount of time to review this stuff.

          12              And in putting this together if you

          13    look at the model of a departmental external

          14    review team, typically the dean is the one

          15    who contacts the members of this review team,

          16    after the suggestion of names by the

          17    department but the -- there needs to be an

          18    analogue of the dean in this review process

          19    and maybe that would be somebody in OMB or

          20    maybe that would be somebody from National

          21    Academy of Science and Statistics.

          22    Somewhere, someone is involved early in the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            311


           1    selection appointment of this review team to

           2    immediately establish the independence of

           3    this review panel, so they have an

           4    opportunity and feedback if necessary, that's

           5    not inappropriate to me.

           6              So, this team then gets together,

           7    evaluates all the micro level stuff that's

           8    been put together by EIA and that's what

           9    gives you the independent evaluation of

          10    sufficient scope.  So -- and that should

          11    answer OMBs objection, because -- I think all

          12    panelists agree that there's a lot of very

          13    compelling independent reviews that already

          14    being done.  There is some additional -- not

          15    exactly packaging but a decisional step in

          16    the hierarchy that needs to take place in

          17    order to get to the level of sufficient

          18    scope.

          19              So this team could really answer

          20    questions like what would happen if this

          21    whole program the EIA went away and that

          22    seems to be what OMB might be looking for.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            312


           1    So I think that's roughly our discussion

           2    today, do you guys want to add anything to

           3    that or --

           4              DR. NEERCHAL:  I think we can

           5    probably find out that the OMB review clearly

           6    indicates that they have already bought here

           7    micro level products.  The -- in page 3 of

           8    that presentation the bullet clearly says

           9    that they value or they have seen the value

          10    of your data and analysis.  So you have sold

          11    the micro level products.

          12              MS. KIRKENDALL:  Our budget

          13    examiner likes us.  She really -- she's been

          14    on our side, she's been very supportive so,

          15    remember when we have an advantage that way,

          16    not everybody is so fortunate.  So was that

          17    modeled after your -- the model that you put

          18    together was modeled after the kind of review

          19    that were done by the academic department?

          20              CHAIRMAN BREIDT:  That's right and

          21    a typical external review team for an

          22    academic department would come in and review




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            313


           1    the whole program.  So undergraduate and

           2    graduate teaching, research and everything.

           3    So they would have in front of them the self

           4    study report that the department put together

           5    and then they would come on campus for a few

           6    days and talk to everybody, faculty, staff,

           7    students and then probably deans and then

           8    they would just complete their report.

           9              MS. KIRKENDALL:  I wouldn't be

          10    surprised if that was the model that OMB was

          11    thinking of, naturally now that you describe

          12    it and in many ways that's sort of what's the

          13    kind of thing that CN Status would do too,

          14    except they're a lot more expensive.  So

          15    expensive you can hardly afford to have them

          16    do it.

          17              DR. NEERCHAL:  I think this was

          18    also pointed out that in academic reviews, I

          19    usually view it as an opportunity for the

          20    department to make a pitch for more

          21    resources, more lines, more faculty, more,

          22    more, more, more.  I think that since you




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            314


           1    have not gotten any specific guidelines from

           2    OMB, there is nothing wrong in turning this

           3    into an opportunity like that.

           4              MS. KIRKENDALL:  Uh-huh.

           5              MS. KHANNA:  And I think just to

           6    make the process a little more fine tuned, it

           7    would be useful -- we talked about how it

           8    will be useful to get friends at the AIA.

           9    There are people who know the EIA well, it's

          10    just not somebody who maybe just knows a

          11    particular product that the EIA producers but

          12    really doesn't know how the entire agency

          13    works, as if would -- that way it's not

          14    overwhelming for the review members of the

          15    HPRT.

          16              MS. KIRKENDALL:  Right.

          17              DR. BURTON:  You don't like that.

          18              SPEAKER:  Yeah for once we can use

          19    an acronym that nobody else does.

          20              DR. NEERCHAL:  And may I quote Joe

          21    on this one, "former members of the committee

          22    are okay," like retiring chairmen.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            315


           1              CHAIRMAN BREIDT:  So are there

           2    questions or comments?  We're just going into

           3    further discussions --

           4              MS. KIRKENDALL:  Yeah actually I

           5    mean, the rest of it was more discussion

           6    about the same and what could the ASA

           7    committee do, how can we make use of your

           8    results and we talk on it about some of that.

           9    We were more down on the weeds I think, I

          10    think you're discussion is really -- that

          11    would be something we can actually do, we

          12    just have to put together the package.

          13              DR. SITTER:  But it meshes well and

          14    we talk mostly about the package.

          15              MS. KIRKENDALL:  It meshes well.

          16    We were talking about what EIA and what's are

          17    the kinds of documentation we should put

          18    together that might be more convincing.

          19              DR. SITTER:  But we were thinking

          20    of what would an external reviewer want to

          21    look at and that's really what Jay was saying

          22    -- its really -- Because I know when we do




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            316


           1    our -- the last external review that our

           2    department had was to become its own

           3    department.  So it was exactly one of these

           4    situations where we have a mandate upon, and

           5    we did a tremendous amount of effort into --

           6    you know, putting -- presenting ourselves in

           7    a way -- in a certain way and that's easy to

           8    do if you're good and I think you're good.

           9    So it's not a hard thing to do, it should be

          10    viewed as an opportunity and not something to

          11    be afraid of.

          12              If you're not good then that's a

          13    different ball game but if you actually are

          14    doing a good game and trying to do a good job

          15    which I think is true then, it would be just

          16    an opportunity and I think the two

          17    discussions seem to have measured really well

          18    and that way.  We were talking about the same

          19    thing, which is good.

          20              MS. KIRKENDALL:  Yes.

          21              DR. FEDER:  I'd just like to make a

          22    distinction between being good and being good




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            317


           1    and improving which is also important, and

           2    also I think the goal is not just to make the

           3    OMB appreciative but also just to be good.  I

           4    think -- as a committee member I'd like

           5    mostly to think that we are helping the EIA,

           6    not only maintain it's very high standard and

           7    appreciation in the community but also to

           8    continue to improve.

           9              MS. KIRKENDALL:  Well we've always

          10    -- we always do this, the whole point is to

          11    improve but - - how we try to use you, that's

          12    a different aspect.

          13              DR. FEDER:  Well certainly.  I

          14    think since I am sort of half way through my

          15    term here I am very impressed with the

          16    attention and kind of attitude that I am

          17    seeing at the EIA to us not as a nuisance but

          18    rather as a tool.  It's very -- to me it

          19    makes me feel good about investing my time

          20    and my employers' time in coming here and

          21    making contributions if we can.

          22              DR. NEERCHAL:  I would like to




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            318


           1    really second that because you know, I have

           2    worked with a number of agencies in DC and I

           3    think that I'm really impressed with the kind

           4    of products that I've seen and also the --

           5    what Colleen said this morning it was amazing

           6    to hear that, you know, from a person who is

           7    actually doing this thing.  That it's not,

           8    you know, when she has a very high score or

           9    something, I don't have room to improve

          10    anymore.  I think that's a fantastic thing to

          11    hear and I think that's really, really

          12    impressive.

          13              So one of the things I'm going to

          14    do is to check out that website and see the

          15    agencies that's scoring higher than you.  I'm

          16    going to check it out tonight.

          17              DR. FEDER:  Are you going to defect

          18    to another committee?

          19              DR. SITTER:  I think you also have

          20    to realize that this is tough.  Defending

          21    yourself publicly and going up early in the

          22    process and saying this is what we're trying




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            319


           1    to accomplish.  Boy, I mean, most people

           2    would be pretty tough or organizations would

           3    be in a pretty tough place in that situation,

           4    that's not easy.  You don't have your final

           5    product where you thought of everything and

           6    you -- and you know, you already made the

           7    mistakes and corrected them.  You're holding

           8    yourself up publicly in front of the worst

           9    group here that has no particular --

          10    allegiance to you at all.  So I think that's

          11    --

          12              MS. KIRKENDALL:  Well actually you

          13    do, I mean it's not formal but especially

          14    after you've been here for a while, I've seen

          15    little allegiances.

          16              DR. SITTER:  Oh I sure hate that I

          17    gave you that impression.

          18              MS. KIRKENDALL:  That doesn't mean

          19    you'd always be nice to us.

          20              MR. CARUSO:  Jay.

          21              CHAIRMAN BREIDT:  Yes.

          22              MR. CARUSO:  One thing about the --




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            320


           1    that I completely understand is opportunity

           2    business where you -- you know, you use this

           3    as an opportunity for your own agenda, in our

           4    case more money.

           5              CHAIRMAN BREIDT:  Uh-huh.

           6              MR. CARUSO:  But in the PART, I

           7    think success is getting what you ask for.

           8    Getting more of what -- really I mean -- I

           9    don't know.  Anybody who's -- this is

          10    relatively new, this is the third year of

          11    PART but I don't know anybody who's gotten

          12    more than their request by -- so, it's a --

          13    the risk is almost all on the downside.  I

          14    think that it's not to see with the PART.

          15    Most likely, most people get their budget

          16    cut, from your request to - - but I also --

          17              CHAIRMAN BREIDT:  In an academic

          18    setting it usually doesn't work either.

          19              DR. NEERCHAL:  I've never seen it.

          20              MS. KHANNA:  Well when you're down

          21    in the dumps and you telling you do this or

          22    we'll shut the department down, they'll do




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            321


           1    it.

           2              MS. KIRKENDALL:  Well, it sounds

           3    like one of the things that maybe we should

           4    do is to think about putting together

           5    documentation that might be convincing, or

           6    one of these outside groups to review and

           7    then maybe get you guys to provide us advice

           8    on the documentation -- well what would be

           9    better, what would make the documentation

          10    stronger.

          11              DR. NEERCHAL:  What were the

          12    specific steps involved in the process that

          13    you went through last time.  Did you put

          14    together a document and send it over to OMB?

          15    How did it work?

          16              MS. KIRKENDALL:  Yes.  Thomas can

          17    tell you about that, he did a lot of it --

          18              MR. BROENE:  Well for example we

          19    did list all the service reports we had done

          20    and we had some examples of where the market

          21    had moved.  But looking back on it, maybe it

          22    wasn't organized well enough maybe it wasn't




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            322


           1    packaged quite right.  What else can --

           2              MS. KIRKENDALL:  Well I mean let's

           3    see - - we put together, based on the

           4    questionnaire that's on the web.  So you

           5    could see what they start from.  Tom put

           6    together answers -- there was a committee,

           7    Tom was a key player -- he wasn't the only

           8    player, there were a few others and then they

           9    went through and talked about assembling the

          10    information in response to the question --

          11    what did we think would be convincing or

          12    demonstrate that we were good in a certain

          13    area.  They would screw around and then we'd

          14    come down to some of the hard questions and

          15    we'd spend more time looking at those.

          16              MR. BROENE:  I mean we did mention

          17    that all the models are in the public domain

          18    and we mentioned that data is somewhat

          19    internally reviewed because it's used by

          20    other people within the agency and you know,

          21    maybe I wasn't systematic enough -- I don't

          22    know exactly, but it did imply.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            323


           1              DR. NEERCHAL:  No, I think my

           2    impression is that it flew because I think

           3    you were actually selling your products, your

           4    specific products and I think that clearly we

           5    have seen the value of it.  I mean, but they

           6    were looking for slightly something

           7    different, you know, I don't think that they

           8    -- and who -- and another question I have is,

           9    who read it at OMB who specifically -- you

          10    know, which office, which person --

          11              MS. KIRKENDALL:  Our budget

          12    examiner was the key person who would

          13    interface with us and she went through it a

          14    couple of times, she came and that was our

          15    support.

          16              MR. BROENE:  At least.

          17              MS. KIRKENDALL:  At least.  And she

          18    provided input and then she'd go back and

          19    chat with her people and I don't know who in

          20    OMB -- probably her.

          21              MR. BROENE:  I had the impression

          22    it was a committee of some type that reviewed




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            324


           1    it.

           2              MS. KIRKENDALL:  Yes, that was

           3    probably more than just her office too -- It

           4    probably was a broader office, because I

           5    think all of the budget examiners are doing

           6    this for their agencies and the budget

           7    examiners are spread apart -- OMB offices.

           8    So it's -- yes it's probably a team at OMB,

           9    but it's Tom's office led by a fairly high

          10    level of people I think.  I don't know the

          11    names.

          12              DR. NEERCHAL:  And that is why I

          13    think that that the comment might have come

          14    from other people.  You're budget examiner is

          15    obviously familiar with your products to some

          16    whereas the other committee members are not

          17    familiar with this, so they were looking for

          18    the high level information.

          19              CHAIRMAN BREIDT:  So if we took

          20    something very similar to the OMB report and

          21    had be the core of what goes to the HPRT --

          22              MS. KIRKENDALL:  Right, organize




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            325


           1    our report for the HPRT around what they

           2    wanted naturally.

           3              DR. NEERCHAL:  And probably cover

           4    it with the very moderate things -- you know,

           5    non-technical you know, write up of al these

           6    things that is palatable to a wide --

           7              MR. CARUSO:  And the other -- in

           8    the Heads of Statistical Agencies meeting

           9    that Nancy and I go to on a monthly basis.

          10    This topic has been discussed and that was

          11    pretty clear from that discussion that there

          12    is inconsistencies -- like somebody would

          13    say, "oh well we got a good PART score" and

          14    another Head would say, "well how did you

          15    handle this" and they'd say, "well we did it

          16    this way" and they said, "well we did it that

          17    way and we got a bad score.  So the -- I

          18    think part of it is that this is a new

          19    process and certainly for our first year so

          20    -- But I do think that we clearly need to do

          21    this macro on how to deal with this macro

          22    issues.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            326


           1              MS. KIRKENDALL:  And actually I

           2    think starting from the package we put

           3    together for OMB isn't bad and the questions

           4    and then just have -- take a fresh look at

           5    some of them and then get an input from a few

           6    other folks around and even if we don't share

           7    the whole package with you at least you'd

           8    share the features that we have.  Some

           9    questions as to whether they're convincing or

          10    not and you may have some clear ideas about

          11    it, either how to say it or what are the

          12    things to document or --

          13              CHAIRMAN BREIDT:  The timeline on

          14    the next round -- next summer.

          15              MS. KIRKENDALL:  It's probably next

          16    summer we think.  At least we suspect that we

          17    will be asking about the things that we

          18    didn't do well.  They may not ask to see all

          19    the stuff they gave us a hundred percent on.

          20    I mean, why would they.  So that would be my

          21    guess that they would be looking at the ones

          22    that we might be working on actually.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            327


           1              DR. NEERCHAL:  I think Randy's

           2    right, ours is a very young campus so we have

           3    a few departments and we also were required

           4    to do this program review for undergraduate

           5    only programs only very recently.  So we had

           6    one program to go through this review for the

           7    first time ever -- You know, this is the kind

           8    of situation that -- you know, the chair of

           9    that program should've called another chair

          10    who had gone through it and she did not do

          11    that and so she tried to do it alone in some

          12    sense and it was not a very good situation

          13    because of this.

          14              And it was not because they were

          15    doing anything wrong, but it was the

          16    inexperience of -- so -- I used to see many

          17    of them because I am in the APB.  The program

          18    -- you know -- something -- a local committee

          19    that kind of go through some of these

          20    documents.  So one of the things we

          21    recommended is to have a mentor thing you

          22    know.  As soon as someone is goes through




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            328


           1    this, if they have never gone through this

           2    they should, you know, hook up with another

           3    chair who has gone through this.  And that

           4    other chair should kind of play the mentoring

           5    role on this one because this is one of those

           6    things where you have no example and you

           7    don't know exactly what your package should

           8    look like.

           9              DR. FEDER:  What Dr. Nagaraj was

          10    talking about reminds me that in Britain when

          11    my department was -- we were in social

          12    statistics department and we were teaching

          13    some of the mathematics courses but when they

          14    had their research placement exercise and

          15    information of the teaching is so on -- first

          16    of all they had to hand over the department

          17    in another university that got the highest

          18    marks, come and tell us about their

          19    experience and then we also had a retreat and

          20    went to that retreat and got free lunch and

          21    some kind of discussions and there was a lot

          22    of practice.  And what we found out and I




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            329


           1    think it's a bit similar to the experience

           2    here, those institutions that got -- at the

           3    first time when that exercise was done got

           4    higher marks and partially it could be -- and

           5    by the way we also knew what was the

           6    composition of the committee that came to

           7    review us and what you could tell about them,

           8    it was a retired engineer actually, that

           9    headed that committee.

          10              And we think that it's also an

          11    issue that after the first round we'd done

          12    that they would go back to say, you know guys

          13    you are too liberal.  So there was a time we

          14    met -- then of course there's also

          15    independent personalities and views that

          16    affected this world that were very different.

          17    But what we did is, a lot preparation we had

          18    the retreat and we had someone come in to

          19    tell us like which guy was talking about,

          20    talking to people -- how come you did better

          21    than us?

          22              DR. SITTER:  Also what I think was




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            330


           1    a major thing -- is for you to hear -- from

           2    people to know what is happening in your

           3    organization.  One of the things I think in

           4    our review is that was mentioned -- there's a

           5    strength was just sort of the uniformity of

           6    what people wanted the department to be,

           7    where they wanted it to go, their dedication

           8    to certain aspects of it.  And its not that

           9    you get together as a team and decide that,

          10    "oh this is the front we're going to put

          11    out."  It's not like that, its just -- just,

          12    from something like a retreat you actually

          13    have discussed it together and you understand

          14    that different people in your organization

          15    view things differently but they don't --

          16    knowing what the process is about, it's quite

          17    different than sort of somebody walking in on

          18    the hallway, "Oh yeah, you're there reviewer

          19    oh yeah".  You know, they don't really

          20    understand that there's this big process

          21    going on and what it's going to mean and

          22    stuff like that.




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            331


           1              So they may not have thought about

           2    it ahead of time and they get very, very much

           3    caught off guard and then -- come across is

           4    quite different than they really are.  So

           5    preparation there's just no substitute for

           6    preparation and what's part of the

           7    preparation is that the people that are going

           8    to be interacting with somebody -- external

           9    reviewer coming in to talk to them, somebody

          10    from OMB, even people preparing stuff, that

          11    they understand what's going on, a bigger

          12    picture or something like that.

          13              CHAIRMAN BREIDT:  Are there

          14    comments or questions?

          15              MS. KIRKENDALL:  Those are good

          16    ideas, at least we have something we can go

          17    on -- a few things, sorry -- more than we

          18    have time to address unfortunately.

          19              CHAIRMAN BREIDT:  Okay, so I think

          20    I'll invite public comments.  Even then while

          21    we adjourn the meeting, see you tomorrow.

          22                   (Whereupon, at 5:02 p.m., the




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382





                                                            332


           1                   PROCEEDINGS were continued.)

           2                       *  *  *  *  *

           3

           4

           5

           6

           7

           8

           9

          10

          11

          12

          13

          14

          15

          16

          17

          18

          19

          20

          21

          22




                        BETA REPORTING & VIDEOGRAPHY SERVICES
                                www.betareporting.com
                 (202) 638-2400     1-800-522-2382   (703) 684-2382