AMERICAN STATISTICAL ASSOCIATION

                         COMMITTEE ON ENERGY STATISTICS

                                     WITH THE

                        ENERGY INFORMATION ADMINISTRATION





























                             Thursday, April 6, 2006

                                 Washington, D.C.









                                                             2


          1    AMERICAN STATISTICAL ASSOCIATION MEMBERS:

          2       NICOLAS HENGARTNER, Ph.D., Chair

          3       NAGARAJ K. NEERCHAL, Ph.D., Vice Chair

          4       DEREK BINGHAM

          5       JOHNNY BLAIR

          6       MARK BURTON, Ph.D.

          7       CUTLER CLEVELAND

          8       JAE EDMONDS, Ph.D.

          9       MOSHE FEDER, D.Sc.

         10       BARBARA FORSYTH

         11       WALTER HILL

         12       NEHA KHANNA

         13       THOMAS RUTHERFORD

         14       SUSAN M. SEREIKA

         15       DARIUS SINGPURWALLA

         16    PARTICIPANTS:

         17       BOB ADLER, Energy Information Administration

         18       MARGOT ANDERSON, Energy Information
                  Administration
         19
                  JUSTINE BARDEN, Energy Information Administration
         20
                  BETTY BARLOW, EI-13
         21
                  JAKE BOURNAZIAN, Energy Information
         22       Administration



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                             3


          1    PARTICIPANTS (Cont'd):

          2       SUSAN BUCCI, Energy Information Administration

          3       GUY CARUSO, Energy Information Administration

          4       JOHN CONTI, Office of Integrated Analysis and
                  Forecasting
          5
                  AUDREY CORLEY, EI-45
          6
                  JOHN PAUL DELEY, Energy Information
          7       Administration

          8       STAN FREEDMAN, Energy Information Administration

          9       DWIGHT FRENCH, Energy Information Administration

         10       WILL GIFFORD, Energy Information Administration

         11       HOWARD GRUENSPECHT, Energy Information
                  Administration
         12
                  VICKI HAITOT, Energy Information Administration
         13
                  BILL HELKIE, Energy Information Administration
         14
                  SHANNON HOWELL, Energy Information Administration
         15
                  ALETHIA JENNINGS, Energy Information
         16       Administration

         17       STAN KAPLAN

         18       NANCY KIRKENDALL, Energy Information
                  Administration
         19
                  TOM LORENZ, Energy Information Administration
         20
                  PAUL McARDLE, EI-01
         21
                  TINA PEARSON, Energy Information Administration
         22



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                             4


          1    PARTICIPANTS (Cont'd):

          2       MARK SCHIPPER, Energy Information Administration

          3       SCOTT SITZER, Energy Information Administration

          4       BOB SCHNAPP

          5       JOHN STAUB, Office of Integrated Analysis and
                  Forecasting
          6
                  PHILLIP TSENG, Ph.D., Energy Information
          7       Administration

          8       BILL WATSON, CNEAF

          9       SHAWNA WAUGH, Energy Information Administration

         10       PAULA WEIR, Energy Information Administration

         11       WILLIAM WIENIG

         12       JOHN WOOD, Energy Information Administration

         13

         14                       *  *  *  *  *

         15

         16

         17

         18

         19

         20

         21

         22



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                             5


          1                      C O N T E N T S

          2    Open Meeting, Nicolas Hengartner Chair,       7
               ASA Committee on Energy Statistics
          3
               Greeting and Remarks, Guy Caruso,            14
          4    Administrator, Energy Information
               Administration, EIA
          5
               Updates for the Committee Since the Fall     29
          6    2005 Meeting, Nancy Kirkendall, Director,
               Statistics and Methods Group (SMG), EIA
          7
               How Can Modeling Suggest Data Needs?  Open   40
          8    Discussion.  Nancy Kirkendall, SMG Chair,
               Margot Anderson, and other invited EIA
          9    senior management

         10    2006 MECS:  Looking at Past Performance      89
               Statistics to motivate New Methods of
         11    Collection.  Bob Adler and Tom Lorenz,
               EMEU, EIA.
         12
               ASA discussant on 2006 MECS past            155
         13    performance:  M. Feder

         14    ASA discussant on New Oil Production SAGE   162
               Model:  N. Khanna
         15
               Additional Committee Discussion             168
         16
               Invitation for Public Comments, Nicolas     184
         17    Hengartner, Chair, ASA

         18    Measuring Perceptions of Applying           185
               Alternative Disclosure Limitation
         19    Methods, Jake Bournazian, SMG, EIA.

         20    Functional Requirements for EIA's Internet  235
               Data Collection System, Stan Freedman,
         21    SMG, EIA.

         22



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                             6


          1    CONTENTS (Cont'd):

          2    ASA Discussant on Functional Requirements   304
               for EIA's Internet Data Collection System,
          3    D. Singpurwalla

          4    ASA Discussant on Improving the SAGE        314
               Petroleum Refinery Model:  T. Rutherford
          5
               Additional Committee Discussion             323
          6
               Making Adjustments to Survey Data           330
          7    When the Collected Data Do Not Meet
               Expectations, Stan Kaplan, CNEAF, EIA
          8
               Invitation for Public Comments              388
          9
               Adjournment of Thursday meeting             388
         10

         11

         12                       *  *  *  *  *

         13

         14

         15

         16

         17

         18

         19

         20

         21

         22



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                             7


          1                   P R O C E E D I N G S

          2                                             (8:39 a.m.)

          3              DR. HENGARTNER:  Oyez, oyez, oyez.

          4    Ladies and gentlemen, thank you very much for

          5    your attention.  Since we're already 10

          6    minutes late, let's start.

          7              I'd like to welcome you all to

          8    another meeting of the American Statistical

          9    Association Advisory Committee on Energy

         10    Statistics, and again, I want to emphasize

         11    this is an ASA meeting and not an EIA

         12    committee, and we meet periodically for

         13    providing input and advice to the EIA.

         14              This is an open meeting, open to

         15    the public, and public comments, of course,

         16    are very welcome, either in form of oral

         17    comments -- and time we'll set aside for this

         18    at the end of today and at the end of

         19    Friday's sessions -- as well as written

         20    comments that can be sent either to the EIA

         21    or to the ASA.

         22              I'd like to remind that all



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                             8


          1    attendees should hopefully sign in at the

          2    entrance -- and if you're here you've

          3    probably done that -- and also provide e-mail

          4    address for whatever reason.

          5              Restrooms:  Restrooms are most

          6    likely on that side, way over there, and

          7    probably on the other side, way over there.

          8              There is an open line for phone.

          9    If you want to have someone call in and leave

         10    messages, let me give you the phone number.

         11    The area code is (202) 586-0832; the other

         12    phone number where they may call here and

         13    leave messages during our stay at the meeting

         14    is, again, (202) 586-0946.

         15              So for the committee member, you

         16    know Kathleen Wert and Donna Arrington, they

         17    are from the ASA, they will help you with

         18    travel reimbursement and other

         19    technicalities, so please see them if you

         20    have any questions.

         21              I'd like to remind all the speakers

         22    and all the people commenting to please speak



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                             9


          1    clearly and loudly into the microphone.  This

          2    will be highly appreciated by the gentleman

          3    here who is making transcripts of our

          4    proceedings.  And finally, before we start,

          5    I'd like to go around the table, and actually

          6    the entire room, so that everybody can

          7    present themselves, and we'll take it from

          8    there afterwards.  So let me start.  My name

          9    is Nick Hengartner; I'm from Los Alamos, and

         10    I'm chair of the ASA Committee on Energy

         11    Statistics.

         12              MS. KIRKENDALL:  I'm Nancy

         13    Kirkendall, the director of the Statistics

         14    and Methods Group in Energy Information

         15    Administration.

         16              MR. CARUSO:  Guy Caruso,

         17    administrator, Energy Information

         18    Administration.

         19              MR. RUTHERFORD:  Tom Rutherford,

         20    I'm a professional economist.

         21              DR. BURTON:  Mark Burton, economist

         22    from the University of Tennessee.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            10


          1              MR. SINGPURWALLA:  Darius

          2    Singpurwalla, Law, Economic, and Consulting

          3    Group.

          4              MS. MILLER:  Rene Miller, EIA.

          5              MR. BOURNAZIAN:  Jake Bournazian,

          6    SMG and EIA.

          7              MR. RASMUSSEN:  Erik Rasmussen for

          8    the administrator, EIA.

          9              MS. LEE:  Audrey Lee at DOE.

         10              DR. TSENG:  Phillip Tseng, SMG,

         11    EIA.

         12              MS. JENNINGS:  Alethea Jennings,

         13    SMG, EIA.

         14              MR. STAUB:  John Staub, EIA.

         15              MR. SWEETNAM:  Glen Sweetnam, in

         16    OIF in EIA.

         17              MR. CONTI:  John Conti, OIF.

         18              MS. KHANNA:  Neha Khanna, Economics

         19    and Environmental Studies, Binghamton

         20    University.

         21              MR. CLEVELAND:  Cutler Cleveland,

         22    Center for Energy Environmental Studies,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            11


          1    Boston University.

          2              MR. BINGHAM:  Derek Bingham,

          3    Department of Statistics, Simon Fraser

          4    University.

          5              DR. NEERCHAL:  Nagaraj Neerchal,

          6    Department of Math and Statistics at UMBC.

          7    Also the vice chair of it.

          8              MR. HELKIE:  Bill Helkie, EIA.

          9              MR. WATSON:  Bill Watson, EIA.

         10              MR. LORENZ:  Tom Lorenz, EMEU, EIA.

         11              MS. HOWELL:  Shannon Howell, EIA.

         12              MR. FENNELL:  Dean Fennell, CNEAF,

         13    EIA.

         14              MS. FRENCH:  Carol French, Office

         15    of Oil and Gas, EIA.

         16              MR. DELEY:  John Paul Deley, NEIC,

         17    EIA.

         18              MR. LU:  Ruey-Pyng Lu, EIA.

         19              MR. DISBROW:  Jim Disbrow, OIT,

         20    EIA.

         21              MS. HOJJATI:  Behjat Hojjati, EIA.

         22              MR. FAWZI:  Aloulou Fawzi, OIF,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            12


          1    EIA.

          2              MS. HOLT:  Susan Holt, Office of

          3    the Administrator, EIA.

          4              MR. BRADSHER-FREDERICK:  Howard

          5    Bradsher-Frederick, EIA and SMG.

          6              MR. STROUD:  Ron Stroud with SMG,

          7    EIA.

          8              MR. WOOD:  John Wood, NRPD, EIA.

          9              MS. WAUGH:  Shawna Waugh,

         10    Statistics and Methods Group.

         11              DR. HENGARTNER:  Thank you very

         12    much.  Let me continue.  First of all, I'd

         13    like to remind you this is a DOE meeting, and

         14    it's under the DOE terms; that means that

         15    Nancy Kirkendall here is the designated

         16    official, and she has the right to interrupt,

         17    adjourn, or take over the meeting as she sees

         18    fit.

         19              MS. KIRKENDALL:  Boy, talk about

         20    power.

         21              DR. HENGARTNER:  The other thing

         22    I'd like to tell you is I think this is going



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            13


          1    to be a very interesting meeting.  Before we

          2    start the meeting, I'd like to welcome Derek

          3    Bingham; he's from Simon Fraser University,

          4    he's essentially their subs for Randy Sitter,

          5    so that's big shoes to fill, but I know Derek

          6    will do a great job at this.

          7              The first session, we're going to

          8    have the pleasure of hearing the EIA

          9    director, Mr. Guy Caruso, followed by Nancy

         10    Kirkendall's overview of the Statistics and

         11    Methods Group, and there are going to be two

         12    sessions in the morning, I believe, and then

         13    lunch -- very important, never forget lunch.

         14    And lunch is going to be in room -- the

         15    first-floor room which we've been all -- if

         16    you don't know where it is, follow the crowd,

         17    that's where it is.  And tonight, we're going

         18    to have dinner at the Marrakesh, which is a

         19    Moroccan restaurant; it's a seven-course

         20    meal, I believe, and so I'd like a show of

         21    hands to see who might be wanting to join us

         22    there for tonight so that we can make



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            14


          1    appropriate reservations.  So --

          2              MS. KIRKENDALL:  I have two.

          3              DR. HENGARTNER:  Two, so -- seven.

          4    Are you coming for dinner, Barbara?

          5              MS. FORSYTH:  Not tonight.  I can

          6    barely make it for breakfast.

          7                   (Laughter)

          8              DR. HENGARTNER:  Do you want to

          9    introduce yourself?

         10              MS. FORSYTH:  Sorry to be late.

         11    I'm Barb Forsyth.

         12              DR. HENGARTNER:  So I think that's

         13    all there is.  So tomorrow, reminder, we're

         14    going to resume at 8:30, but I remind you

         15    about those technicalities at the end of the

         16    day.  So without further due process, it's a

         17    pleasure to recognize Mr. Guy Caruso, the

         18    administrator of the Energy Information

         19    Agency.  Thank you very much.

         20              MR. CARUSO:  Thank you, Nick, and

         21    good morning everyone.  I want to make,

         22    again, a special thanks to all of the ASA



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            15


          1    committee members, and we appreciate your

          2    taking the time and having this meeting to

          3    advise us twice a year and as I've said many

          4    times in previous meetings how much we really

          5    appreciate your advice and how useful it is.

          6    And I know you see the results of it -- I

          7    hope -- from meeting to meeting and on a very

          8    practical way, whether it's frames of our

          9    surveys, to the website improvements, to

         10    methodological changes in modeling, all of

         11    these have benefited by the counsel and

         12    advice of this committee.  So thank you very

         13    much, and thank you, Nick, for chairing the

         14    session, and I'd look forward to, again,

         15    having a good two days of discussions.

         16              As always, in that six months'

         17    interval between meetings, the EIA has been

         18    very busy with a large number of activities,

         19    whether they be our outlooks or new surveys,

         20    improvements in old surveys, et cetera, all

         21    of which we'll talk about today.

         22              The usual Annual Energy Outlook,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            16


          1    going out to 2030 this year, was released in

          2    its entirety in February, and it's available,

          3    of course, on the website, and I think when

          4    one looks at what's the main highlights of

          5    the AEO '06, this year, clearly, the

          6    reassessment of the oil price outlook was by

          7    far the biggest change and has implications,

          8    of course, for interfuel competition as well

          9    as the oil market outlook, and --

         10              One of the things when we make

         11    reassessments -- in this case, oil prices --

         12    and indeed, even natural gas prices are up

         13    considerably in this outlook -- it stretches

         14    the NEMS, the National Energy Modeling System

         15    that many of you are familiar with.  And

         16    clearly, this is a case where, this year,

         17    looking at much higher prices, both for oil

         18    and natural gas, and the implications for not

         19    only oil but the other sectors besides

         20    transportation, such as electric power and

         21    the implications for coal -- all brought to

         22    bear some significant methodological



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            17


          1    challenges, and I think some of them will be

          2    discussed today, but I think, more

          3    importantly, we'll be looking hard at those

          4    of how the model responds to these new

          5    assumptions, and, as I'll mention when I talk

          6    about the budget, we're hoping to launch a

          7    serious upgrade of our National Energy

          8    Modeling System, beginning in the second half

          9    of '06, but continuing, we think, if money is

         10    appropriated, for the next several years.

         11              So, again, that was the big change,

         12    and you could see it here, about a

         13    $20-per-barrel increase over the long-range

         14    assumptions here through 2030, going from

         15    roughly $30 a barrel to roughly $50 a barrel

         16    on average.  And part of it is the investment

         17    side of things, particularly from OPEC

         18    countries, but also the recognition that the

         19    cost of doing business in the upstream in the

         20    oil sector is increasing with the commodities

         21    boom, and the cost of doing just about

         22    everything from drilling through the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            18


          1    downstream and refining and distribution.

          2    All have contributed to that.  We are still

          3    not among the camp that says that world oil

          4    is peaking and declining in the near term and

          5    therefore that's what's driving prices.

          6    That's not what we're saying; we're saying

          7    that the issues driving prices are mainly

          8    above the ground, not below the ground, and I

          9    know John Wood is here, and he'll talk about

         10    mainly the new survey for crude, but

         11    certainly John and his people have been very

         12    firm in their belief that there's still lots

         13    more oil to be discovered and more productive

         14    capacity to be added over the next 20 to 30

         15    years.

         16              Of course, short term tends to get

         17    a lot more attention, and our latest

         18    short-term issue, as we head to the driving

         19    season this year, is the transition in

         20    reformulated gasoline composition by the

         21    decision being made by almost all refiners to

         22    remove MTBE -- methyl tertiary butyl ether --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            19


          1    from reformulated gasoline as the main

          2    oxygenate and octane booster and replacing it

          3    with ethanol for the most part.  That

          4    transition is underway, and our view is that

          5    it could be a bit of a rocky road to make

          6    this transition, and that could lead to some

          7    supply dislocations in certain regions of the

          8    U.S., and it's gotten some attention both in

          9    Congress and in the administration.  And of

         10    course, we've all witnessed some run-up in

         11    gasoline prices already this spring; only

         12    part of that, we believe, is part of this

         13    transition, but nevertheless, it's an issue

         14    that EIA is deeply involved in and raised the

         15    flag early on, in January, by a paper that

         16    was done by Joanne Shore in our oil and

         17    natural gas office.

         18              Again, as I mentioned, we continue

         19    to look for ways to improve the collection of

         20    data, in this case, on the upstream crude oil

         21    collection.  As you know, we've started

         22    collecting natural gas production data,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            20


          1    heavily relying on the advice of this

          2    committee in developing that form, and John

          3    Wood's here to talk about the 914, the

          4    expansion of the data there, but we are

          5    considering expanding that in 914 form to

          6    include the collection of crude oil

          7    production and very much appreciate hearing

          8    your advice on that.

          9              One of the things that's really

         10    improved the productivity of collecting data

         11    -- and the quality, we believe -- is the

         12    Internet data collection systems that you've

         13    heard about previously from the various --

         14    particular from our coal and natural -- in

         15    nuclear and electricity -- CNEAF office.

         16    That's going along very well.  Several IDCs

         17    were launched in the last year or so, and

         18    they're operating pretty well, we think, and

         19    definitely have improved the speed, the

         20    timeliness, and accuracy of the responses

         21    because so much of the editing is done at the

         22    respondent's side of things and therefore



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            21


          1    there's less follow-up and it allows us to do

          2    other things, to work on the quality and the

          3    distribution of the data.  So that's going

          4    very well and appreciate very much all of

          5    your help in giving us advice on that.

          6              We've -- after a long process, the

          7    voluntary reporting of greenhouse gas

          8    programs -- guidelines have been revised, and

          9    we're about to begin the redesign of that

         10    program, and we expect that we'd start around

         11    the first of June and would be fully

         12    implemented during 2007.  This has gotten a

         13    lot of attention with respect to the issues

         14    of climate change, and EIA's playing a very

         15    important role in -- and will continue to

         16    play an important role in the office that is

         17    headed by John Conti, OIF.

         18              As many of you knew from being

         19    participants in the improvement of our

         20    website, we did launch the new site in

         21    November.  We've gotten substantial positive

         22    feedback on this redesign, and continue to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            22


          1    work on improvements and certainly, again,

          2    welcome any advice you would have on the new

          3    website.  It's a work continually in

          4    progress, as you're aware, and we're

          5    currently doing a customer survey on the

          6    kids' page, which is one of the most popular

          7    components of our website.  And on the

          8    personnel side, just this week, Monday, a new

          9    Web champion, Gina Pearson, has come on

         10    board.  She'll be the director of our

         11    National Energy Information Center, NEIC,

         12    succeeding John Weiner, who many of you know

         13    from his many years as heading up NEIC.

         14    Among Gina's many talents is her experience

         15    and success in redesigning websites.  She's

         16    most recently come from the Energy Research

         17    Service, where she completely redesigned --

         18              MS. KIRKENDALL:  Economic.

         19              MR. CARUSO:  Economic Service?

         20              MR. KIRKENDALL:  Economic Research

         21    Service.

         22              MR. CARUSO:  Economic Research



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            23


          1    Service.  It's part of the Department of

          2    Agriculture, right?  Yeah.  And she's done a

          3    outstanding job there and at Department of

          4    Education, and we're looking forward to even

          5    greater things as our Web evolves.

          6              I mentioned one component of our

          7    budget request for 2007, which is currently

          8    being discussed on Capitol Hill.  That is the

          9    request for an increase of several million

         10    dollars, which partly will be used for this

         11    upgrading of the National Energy Modeling

         12    System, and we look forward to hopefully

         13    getting that money appropriated and working

         14    with you all when this gets more to where we

         15    move from a planning to implementation phase

         16    and getting your advice on that.

         17              This year, we have a budget of a

         18    bit more than $85 million; it's a small

         19    increase from '05, and enables us to pay for

         20    about 370 federal employees and a number of

         21    contractors.  We have, as I mentioned, asked

         22    for about $4.5 million increase for '07, with



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            24


          1    a small increase in FTEs to 375, but more

          2    importantly, the increase would go to, as I

          3    said, upgrading the NEMS as well as some

          4    other areas.

          5              DOE is now moving toward a

          6    five-year budget cycle, and we're certainly

          7    participating in that, and, as I mentioned,

          8    we think the NEMS upgrade will be a multiyear

          9    task.  It will require at least three years

         10    to go from the planning through actual

         11    utilization of that new revised NEMS.  And

         12    when the current system was designed and put

         13    into use, it took about three years.  I think

         14    it started in '91 and it was '93 before the

         15    Annual Energy Outlook was fully utilizing the

         16    current NEMS.  So clearly we look forward to

         17    this five-year cycle that the department is

         18    now engaged in, and our targets for

         19    timeliness and quality continue to be quite

         20    stringent.

         21              And the other part of the

         22    longer-term -- five years, if you will -- is



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            25


          1    our continued need for the succession

          2    planning and workforce planning that we've

          3    talked about before this group previously,

          4    and that is with an organization that's now

          5    approaching 30 years.  Many of our most

          6    experienced people are reaching the

          7    retirement age, and we're having a fairly

          8    substantial turnover.  Last year, we lost 16

          9    senior EIA employees, and we've been at least

         10    successful so far in the recruitment and

         11    succession to those 16, but we're looking at

         12    probably 20 to 25 senior people leaving in

         13    '06 and again in '07.  So, again, as I've

         14    said in the past, any assistance you can be

         15    in terms of recruitment of your students or

         16    your associates that you come into contact

         17    with, we very much appreciate that because

         18    you know the kind of work we do here and the

         19    type of people that we need to keep up the

         20    tradition of high quality.

         21              About one year ago, in response to

         22    OMB evaluation of our budget, one of the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            26


          1    things they said, and we talked to you about

          2    and asked your advice, was no one's ever

          3    really done an overview evaluation of EIA.

          4    We've seen evaluations of specific surveys,

          5    specific components of the program, but what

          6    we'd like to see is an independent external

          7    review of EIA in its entirety, and I was

          8    fortunate to be able to convince Denny

          9    Ellerman of MIT to chair a team to do exactly

         10    that, look at what it is that EIA does and

         11    are we doing the right thing, is the main

         12    mission we asked of Denny and his team -- and

         13    he went around and recruited four other

         14    outstanding individuals who many of you know:

         15    Paul Joskow of MIT; Kathy Cooper's a former

         16    undersecretary of Commerce and former chief

         17    economist of Exxon Mobil; Jay Hakes, my

         18    immediate predecessor; and Phil Sharp, who

         19    was then at the Kennedy School and now is the

         20    director of Resources for the Future and

         21    former congressman.  They looked very hard at

         22    what EIA's original legislation authorization



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            27


          1    asked of us 30 years ago and how we've done

          2    in meeting that mandate, and, even more

          3    importantly, looking out in the future.  Are

          4    we preparing for what the future's going to

          5    bring and the pressures and the requirements

          6    on us?

          7              They met twice last year, and we

          8    expect that we'll receive their full report

          9    -- we've gotten a preliminary briefing from

         10    Denny last month -- and we expect to have the

         11    full report, which we would make available,

         12    of course, to you, and we'd hope to have a

         13    full discussion of what they have found in

         14    their review, and, more importantly, what

         15    they are recommending for the future for EIA,

         16    including areas where we may want to place

         17    more emphasis and areas where we may want to

         18    place less emphasis and indeed, even to maybe

         19    even stop doing some things.  So we found it

         20    to be an outstanding opportunity to actually

         21    almost do a self-evaluation in order to

         22    prepare for the external evaluation.  As you



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            28


          1    had so wisely recommended, we approached this

          2    kind of like a visiting scholar's approach

          3    that many of your universities use, and I

          4    think that was a very wise advice on your

          5    part because we found it to be very useful.

          6              As I mentioned, we are approaching

          7    our 30th year, having come into being as the

          8    Energy Information Administration in 1977,

          9    and so we're looking forward to having a

         10    celebration some time.  Probably it'll be a

         11    rolling celebration throughout '07 but we'll

         12    focus it mainly on the educational aspects

         13    that we think we can play an important role

         14    in the energy education of not only the

         15    decision makers, but the public mainly

         16    through our website.

         17              So we'll be asking for your ideas

         18    and input as well as we plan on this, so,

         19    again, let me close by once again thanking

         20    you, Nick, and all the members of the

         21    committee for your past advice and especially

         22    for the advice we'll be getting today,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            29


          1    tomorrow, and in the future.  So if there are

          2    any questions, or anybody like to go into any

          3    more detail on any things I've mentioned, I'd

          4    be happy to do that now or any time during

          5    the next couple days.

          6              Okay, thanks.

          7              DR. HENGARTNER:  Thank you very

          8    much.

          9                   (Applause)

         10              DR. HENGARTNER:  I'd like now to

         11    welcome Nancy Kirkendall to give us a review

         12    of what the Statistics and Methods Group has

         13    been up to in the last six months.

         14              MS. KIRKENDALL:  What I'm actually

         15    going to review is what we did in response to

         16    some of the committee's suggestions from the

         17    sessions we had last time.  Sometimes that's

         18    the same thing and sometimes it's not.

         19              I have picked out a number of the

         20    sessions that we had last time, and I thought

         21    I'd give you a little update on what we've

         22    done or what we haven't done in some cases.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            30


          1    So let me start out -- this is the Short-Term

          2    Energy Outlook.  Last time, Margot Anderson

          3    talked to you about her plans to produce a

          4    report that would analyze the forecast errors

          5    produced from the STEO, the Short-Term Energy

          6    Outlook.  You made a number of

          7    recommendations.  One of the things a lot of

          8    people said -- and this was more than one of

          9    the people in the room -- said that one of

         10    the reasons for developing a model is to

         11    identify new data needs.  And when I went

         12    through the suggestions, I thought, okay,

         13    well, now what do we do.  So we've put that

         14    on the agenda to talk about in more detail,

         15    and we'd like to get more of your ideas on

         16    what we can do with that idea.

         17              The other suggestion that I thought

         18    was interesting was to separate the model

         19    into two parts, or the modeling errors into

         20    two parts:  The error in the modeling and the

         21    error in the input variables.  And that's

         22    something that I still think is really



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            31


          1    interesting and useful.  We haven't had a

          2    chance to do that yet.  However, Margot's

          3    group has produced a draft of a forecast

          4    error report.  They've done analysis of

          5    forecast errors for one-, six-, and

          6    twelve-month forecast horizons, and that is

          7    currently out for review, so hopefully we'll

          8    have something to share with you next time to

          9    get more input on it.

         10              Vehicle energy use.  Mark Schipper,

         11    also from the Office of Energy Markets and

         12    End Use, gave a talk about the work they've

         13    been doing with the Department of

         14    Transportation.  This is basically to fill a

         15    data gap caused by losing the residential

         16    transportation energy consumption survey.

         17    And so the committee suggested that they

         18    liked the modeled data and you also

         19    recommended that we continue collaborating

         20    with the Department of Transportation, and of

         21    course we're planning on doing that.  And

         22    since the last meeting we have offered a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            32


          1    public-use version of a report from the

          2    modeled data, so there's been further work on

          3    that.

          4              There were two sessions by Rene

          5    Miller of the Statistics and Methods Group

          6    last time.  One of them talked about some

          7    data quality initiatives, how to come up with

          8    a nice little product that would illustrate,

          9    perhaps, a different data series to try to

         10    measure the same thing and use it as an

         11    educational tool.  The other one talked about

         12    discrepant estimates.  Any time you have a

         13    survey to measure something, if you have two

         14    surveys, you get two different numbers.

         15    Sometimes this is useful and sometimes it's

         16    mystifying.  So we talked to you about that.

         17    So this is a summary of your advice.  One of

         18    them -- and Mark Bernstein told us this all

         19    regularly through his time on the committee

         20    -- and that is that in all of our

         21    presentations, we're supposed to state our

         22    objectives clearly.  Hopefully, this time,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            33


          1    you'll see the objectives more clearly in the

          2    presentations, but we'll see.  But in terms

          3    of advice on this particular issue, you

          4    suggested that we need to focus on the

          5    biggest data quality problems first, which

          6    makes a lot of sense -- sometimes it's not

          7    easy to figure out which those are -- to

          8    measure data quality, do an internal report,

          9    and then develop a Web product after that.

         10    So the purpose is to evaluate data quality,

         11    so the advice was to have a more systematic

         12    approach that would lead to possibly a Web

         13    publication, but maybe not in some cases.

         14              The other thing that you have

         15    strongly recommended that we do is to inform

         16    users about data quality issues, and on that

         17    one, we know we need to do that, but we're

         18    not clear quite how to yet.  The issue has

         19    been raised with the interoffice issue group

         20    that Rene Miller chairs and Howard

         21    Gruenspecht is a member of that.  So we'll

         22    still be debating how to do that.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            34


          1              We talked about -- we've looked a

          2    little bit at looking at performance measures

          3    to try to find some measures that would be an

          4    indicator of data quality problems we should

          5    look at.  This is just ongoing work.  Rene's

          6    group is going to take on, in -- sometime in

          7    the '80s or '90s there was an evaluation -- a

          8    comparison of data from the Residential

          9    Energy Consumption Survey and data that we

         10    get on our supply surveys.  And naturally

         11    enough, they didn't match, and we tried to

         12    explain why.  So that report was done -- it

         13    was a good report, but it was done a long

         14    time ago, and they're going to update it.  So

         15    it's a good model to use.  We need to look at

         16    different data -- at the data sources and see

         17    what we can learn from that.  So that --

         18    we'll work on that beginning sometime this

         19    spring or summer.

         20              And then the last one I already

         21    mentioned.  The survey self-assessment

         22    process -- this is Tom Broene's work -- and



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            35


          1    it was a big undertaking.  Tom interviewed

          2    all of the survey managers in EIA to try to

          3    help fill out a little evaluation form, and

          4    unfortunately it turns out to be a huge

          5    burden to do that.  And you obviously

          6    recognized that and have suggested that we

          7    need to be more flexible in how we develop it

          8    and how we implement it because we need to

          9    have a process we can actually do on a

         10    regular basis.  So that's sort of a summary

         11    of what your advice was.

         12              We are reconsidering the process,

         13    and the only thing that we think we'll keep

         14    from the old process is the -- we ask for

         15    survey-specific targets for improvement, and

         16    I think we still would like to do that.  At

         17    least it tries to focus each survey on what

         18    they need to do -- or it could be survey

         19    system -- flexibility idea.

         20              This was an analysis between -- I

         21    think they were trying -- it was Bill Trapman

         22    and Leila from the Office of Oil and Gas --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            36


          1    it was a modeling exercise trying to look for

          2    predictive relationships between futures

          3    prices and Henry Hub prices.  And I think the

          4    committee was supportive of the analysis and

          5    didn't have particular recommendations for

          6    what to do different, but they thought with

          7    that particular analysis -- you did advise

          8    trying to incorporate NYMEX prices into STEO

          9    for predictive purposes.  And that's sort of

         10    interesting because I think that's one of the

         11    things the external review team has also

         12    talked about, is that EIA should make better

         13    use of some of the public data, like the

         14    NYMEX prices.  So that's kind of interesting

         15    when advice kind of comes together.  Probably

         16    means that we need to be serious about taking

         17    it.

         18              And then finally, last time we

         19    reported to you on the frames evaluation that

         20    was done between -- it was a cooperative

         21    relationship between the Census Bureau and

         22    EIA.  It was a good effort.  You commented



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            37


          1    that we had a pretty high coverage on these

          2    surveys -- 75 to 100 percent by volume -- and

          3    that was especially true, given there were a

          4    lot of problems in matching.  The name and

          5    address -- all we had to match was name and

          6    address lists, and they were not exactly the

          7    same.  They weren't formatted the same and so

          8    there was a level of detail that was missing.

          9    You suggested collecting EIN numbers to help

         10    in matching; I think we are considering that,

         11    and we'll see how that plays out.  And you

         12    suggested that we should use a cost-benefit

         13    approach in deciding whether to try to fix

         14    some of these frames.  I think that means

         15    work on the ones that are important and don't

         16    worry so much about the others.  And so of

         17    course we don't have as many resources as we

         18    might want for this sort of thing, so we're

         19    definitely doing that.  Right now, CNEAF has

         20    identified a different data gap related to

         21    their frame, and that's what they're working

         22    on.  They've discovered that we're missing



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            38


          1    some production of waste coal, and they're

          2    trying to add that on the production side,

          3    which is not the surveys that we're doing

          4    here.

          5              But another comment on this

          6    evaluation, tomorrow after the break we're

          7    going to be giving a little certificate of

          8    appreciation to the people from the Census

          9    Bureau who helped us on this project and who

         10    gave presentations to you.  So that was

         11    really good.

         12              Well, that's my summary from last

         13    fall.  If you have any questions, please ask.

         14              Okay, well, I think we have an

         15    interesting set of topics to talk to you

         16    about today, and we look forward to your

         17    advice.

         18              DR. HENGARTNER:  Thank you very

         19    much, Nancy.

         20                   (Applause)

         21              DR. HENGARTNER:  Before we

         22    continue, I'd like to invite the people who



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            39


          1    trickled in after the first set of

          2    introductions to introduce themselves to the

          3    microphone so that you can be on the record

          4    as being present.  Moshe, do you want to

          5    start?

          6              DR. FEDER:  My name is Moshe Feder,

          7    I'm a statistician from the Research Triangle

          8    Institute.

          9              MR. HILL:  Walter Hill, St. Mary's

         10    College of Maryland, Department of Political

         11    Science.

         12              MR. FREEDMAN:  Stan Freedman, EIA.

         13              MR. ZHANG:  Bin Zhang, EIA.

         14              MS. BLUMBERG:  Carol Blumberg of

         15    Winona State University, and I'm actually

         16    interviewing for a job here today.

         17                   (Laughter)

         18              MR. BROENE:  Tom Broene, EIA.

         19              MS. ANDERSON:  Margot Anderson,

         20    EIA.

         21              MS. BARDEN:  Justine Barden, EIA.

         22              MS. WEIR:  Paula Weir, EIA.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            40


          1              MR. WEINIG:  Would you step forward

          2    and use one of the microphones, please?

          3              MR. BENNECHE:  Joe Benneche, EIA.

          4              DR. HENGARTNER:  Okay, thank you

          5    very much.  I'd like to continue -- we're on

          6    schedule, fortunately, with a -- I think it's

          7    an open discussion, is that correct, by --

          8    how can we model suggested data needs, and

          9    it's going to be essentially led by Margot

         10    and Nancy and other invited EIA senior

         11    management.

         12              MS. KIRKENDALL:  There is a summary

         13    of the comments you made.  I think it's under

         14    item one in your notebooks, and all I did was

         15    pull out some of the quotes that I thought

         16    were appropriate for this discussion.  EIA is

         17    really fortunate that we are both users --

         18    producers of data and we use our own data,

         19    and you would hope that we could do a good

         20    job with a feedback relationship between the

         21    users and the producers of the data.  That

         22    would help improve both sides of the issue.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            41


          1    However, we haven't ever managed to figure

          2    out how to do that, and so we would be

          3    interested in your advice on how the modeling

          4    and data and have a good feedback -- how do

          5    you learn from your modeling what your data

          6    needs are?  Any ideas on what kind of a

          7    feedback mechanism might work, if you've seen

          8    anything like that in the other places.

          9              MS. KHANNA:  I have a very

         10    preliminary question -- this could be because

         11    I wasn't here the last time -- which without

         12    language and models, how do you figure out

         13    what your -----?  How does the EIA even start

         14    that process?

         15              MS. ANDERSON:  Well, we already --

         16    in most cases, we're not starting up with any

         17    new models.  We basically have two or three

         18    models that we use for -- do I need --

         19              MR. WEINIG:  I couldn't hear.  I'm

         20    not sure anyone here heard the opening

         21    remarks.

         22              MS. KHANNA:  Sorry.  Okay, I hope



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            42


          1    you can hear this now.  My basic question

          2    was, without knowing a model, how do you even

          3    begin to assess your data needs?  And so for

          4    me the starting point always is here is what

          5    I'm trying to model, and that almost always

          6    drives my data needs.  So I'm a little

          7    surprised by the question in the first place.

          8              MS. ANDERSON:  I think we were

          9    looking at it from the aspect of having

         10    existing models and how do you enhance

         11    existing models and identify the data gap

         12    that you might fill by collecting new data or

         13    doing new surveys or collecting already

         14    commercially available data.  We don't have a

         15    lot of models in EIA; we tend to have a

         16    couple of forecasting models, a short-term

         17    model, some long-term forecasting models, but

         18    for the most cases, we're not building new

         19    modeling frameworks.  So sort of the question

         20    that first comes out of is, we have these

         21    data programs first and the modeling came

         22    somewhat after having the data programs



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            43


          1    intact.  So the data programs were -- what

          2    kind of data do we have that can support the

          3    modeling that we would like to do, the

          4    short-term and the longer-term forecasting.

          5    And it's opposed to the other way around of

          6    starting a model from fresh and then

          7    determining what data.  John might have some

          8    ideas on if we go through this process, when

          9    we go through this process, of redoing the

         10    national energy model, and it certainly might

         11    engender some new questions that we might

         12    want to either collect some data or rejigger

         13    some data that we're already collecting or

         14    use a portion of data more frequently.  But

         15    right now we're not building any new models

         16    that would require us to reevaluate the data

         17    programs, we're trying to get a sense of how

         18    do we use the models that we have to ask

         19    ourselves what data might improve the

         20    predictive capability of those models.  I

         21    think.  That's the question I'm asking -- or

         22    at least --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            44


          1              MR. CONTI:  Let me maybe expand on

          2    that a little bit in getting back to the

          3    question.  First of all, I think once upon a

          4    time, when people were trying to understand

          5    the energy industry, it wasn't what data do

          6    we need for the models, it was what analytic

          7    information did we need to have in order to

          8    understand the energy questions that were

          9    being asked of us.  And so EIA created data

         10    programs to provide those answers.  As the

         11    questions became more complex, they developed

         12    models to help them do the analytics to

         13    answer the questions.

         14              Now we ask the question, well, what

         15    does the model tell you in terms of data

         16    requirements.  Well, the model often tells us

         17    that we are missing key data that we'd like

         18    to have.  And we've identified areas in the

         19    past where this data is missing.  For

         20    example, in the ethanol area, we sat down

         21    with people, both in DOE and other parts of

         22    EIA, and identified -- we'd like to have this



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            45


          1    data, and they said, "Well, thank you very

          2    much, we'd like to have it, too.  We can't

          3    afford to give you that data."  As a matter

          4    of fact, just a couple of weeks ago, EIA

          5    announced it was eliminating two data forms,

          6    one of which helps us understand crude

          7    quality and crude prices, something that's

          8    probably of analytic interest in the future.

          9    It's not the fact that you identify a need

         10    with the model and then, okay, we just run

         11    out and get the data, you still have to have

         12    a budget to be able to get that data.  So --

         13    that's where I'll stop.

         14              I'm John Conti, director of the

         15    Office of Integrated Analysis and

         16    Forecasting.

         17              MS. KIRKENDALL:  The other comment

         18    was a lot of our data series -- EIA was

         19    organized and established in '78, and there

         20    was a predecessor organization, too.  So our

         21    data collections have been around for a long

         22    time, and we tinker with them.  You add



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            46


          1    products, you subtract products, you find a

          2    few new surveys you need to do.  But a lot of

          3    the data programs have been around for a long

          4    time.

          5              MS. ANDERSON:  And never intended

          6    to be necessarily used in a modeling context.

          7    They were almost ends of themselves because

          8    people either gave a series or gave

          9    information on series that weren't really

         10    asked to do -- look at these relations --

         11              MS. KIRKENDALL:  Well, except that

         12    you were providing the data for people who

         13    want to do that kind of analysis.  That's

         14    what our data program's for; it's supposed to

         15    be -- doesn't have to be for us.  And so it's

         16    supposed to support analysis that's outside

         17    the EIA, too.

         18              MS. ANDERSON:  Correct.

         19              MS. KIRKENDALL:  It's just that we

         20    do have an opportunity to learn more about

         21    what data are needed from our own internal

         22    uses and modeling.  So that's sort of -- how



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            47


          1    do we do that a little better.  And John

          2    talked about the ethanol usage -- is there

          3    another way of -- and that's going to be

          4    coming around and we'll probably end up doing

          5    something about that.

          6              MR. CLEVELAND:  Well, I'm glad to

          7    hear somebody say that, but as far as I

          8    understand it, I think key staff people have

          9    identified data needs in the past then it

         10    really doesn't get past that, identifying the

         11    need.  I don't think the problem is

         12    identifying the need, I think the problem is

         13    figuring out how could we meet that need

         14    within existing budgets.

         15              MR. KIRKENDALL:  Or get that need

         16    moved up to a higher-level discussion when

         17    we're talking about allocation of scarce

         18    resources.  At least at that discussion point

         19    we know what's on the table for data needs.

         20    You might not be able to do anything, but at

         21    least it gets into the discussion.

         22              MR. CLEVELAND:  I guess one of my



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            48


          1    concerns is to what extent the questions that

          2    you're even asking have been constrained by

          3    the historical data collection and modeling

          4    process, and I'm wondering -- and I'm sure

          5    that your external review panel is asking if

          6    we had to blow it up and start all over --

          7    which I'm not proposing to do -- what

          8    questions would we be asking and what models

          9    would we be trying to build to answer those

         10    new important questions and, in turn, what

         11    data -- what do we need to do with that,

         12    rather than approaching it from the other

         13    way.

         14              And a secondary question is, to

         15    what extent is the demand for data by people

         16    outside of the EIA, like me, in terms of

         17    determining what you do and don't collect,

         18    and how do you balance that against your own

         19    internal needs?

         20              MS. KIRKENDALL:  So a question is

         21    what's a good way for getting input from you,

         22    for example, on what you think the data needs



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            49


          1    are -- and other external users.

          2              MR. CLEVELAND:  Yeah, except when I

          3    hear the budget situation, it's hard to me to

          4    even conceive that our needs would be that

          5    high on the list.  But it seems like

          6    something you should be doing.

          7              MS. ANDERSON:  It kind of goes back

          8    to this issue of defining your objectives.

          9    If you don't know why you're doing something,

         10    whether it's to create a structural model or

         11    a forecasting model, you're not clear what

         12    the questions are that you're trying to

         13    answer, you're going to fail on trying to

         14    determine what data must you collect and who

         15    do you need to go ask about your data.

         16              So it goes back to the advice that

         17    we've been given and we try to take, but

         18    you've got to define your objectives because

         19    we can't do everything.  And it's not such an

         20    easy exercise to define your objectives in an

         21    organization that has such a diverse client

         22    base and feels that it is required to answer



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            50


          1    or cover all aspects of the energy sector.

          2    And we really do try to stay balanced; we

          3    really do try to have universal coverage, and

          4    regret always when something has to go away

          5    and there are multiple objectives that

          6    sometimes work at cross-purposes.

          7              But I think it is really

          8    identifying the key objectives, and it is

          9    asking ourselves, if we could start all over

         10    again, would we do what we have today -- even

         11    to ask ourselves that in little microcosms.

         12    If we didn't have a short term for our

         13    forecasting program, would we create one or

         14    would we do something else?  And those are --

         15    we really created that form -- I think the

         16    external study team has caused all of us to

         17    take a long breath and kind of ask some of

         18    those fundamental questions.  But even if we

         19    go through that exercise internally, you run

         20    up with the same kind of constraints that

         21    John's identifying.

         22              How can you communicate that to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            51


          1    folks that tend not to live and breathe this

          2    and are much more -- adhere to the status

          3    quo?  We've been getting it all these years,

          4    we've been getting this product all these

          5    years, change is very jarring.

          6              So I don't want to say it's

          7    impossible, but you kind of have to tackle

          8    these things on the margins a bit.  But it

          9    does go back to define your objectives:  What

         10    are we trying to do here at EIA?

         11              MR. CONTI:  Maybe getting back to

         12    the first part of that question, Guy

         13    mentioned earlier how we will be developing a

         14    National Energy Modeling System, and as we

         15    start down this path, one of the first things

         16    we'll do is we'll do a requirements analysis.

         17    And within this requirements analysis, we'll

         18    be speaking to a bunch of our stakeholders to

         19    try and understand the questions that they

         20    want us to be able to answer in the mid to

         21    long term.  And then, assessing all of those

         22    needs, we'll try and determine which ones are



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            52


          1    feasible to address -- and there'll probably

          2    be some things we won't be able to address --

          3    and then the ones that we think that are

          4    relevant to address and possible to address,

          5    we'll do an assessment of, does the data

          6    exist, and if the data doesn't exist, we'll

          7    do an assessment of, well, how expensive is

          8    it to get that data.  So in the end, we might

          9    not be doing it, but we will have gone

         10    through a couple of steps to, does it need to

         11    get done, how expensive is it to get done,

         12    where can we get the data, and how do you get

         13    that into a model.

         14              MR. RUTHERFORD:  One other comment

         15    I'd just add to that is the -- Tom Rutherford

         16    -- I think the discussion has been quite --

         17    my comment at the last thing was perhaps a

         18    bit of a nave question because clearly it's

         19    kind of obvious, well, duh, that's what we

         20    do, right, is we collect data, we do models,

         21    we answer questions.  So obviously this is

         22    part of the process.  The question is, to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            53


          1    what extent can you formalize that process,

          2    right.  So it's obvious that you guys are

          3    involved in -- I wasn't suggesting by my

          4    comment that you weren't going about things

          5    this way.

          6              MR. CONTI:  I think that's

          7    important.  How do we formalize the need for

          8    specific data elements, since I'm the modeler

          9    and not the data collector, and where does

         10    that fit into the priority list on the data

         11    collector's side?

         12              MR. RUTHERFORD:  Yeah.

         13              MR. CONTI:  Is it their number one

         14    priority?  They have a bunch of priority

         15    things they want to do, and being able to

         16    answer some question from someone they don't

         17    know is not necessarily very high on their

         18    priority list.

         19              MR. RUTHERFORD:  The second comment

         20    I'd make is that I think the process -- if we

         21    think about the cost-benefit -- I think your

         22    portraying of the issue as a cost-benefit



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            54


          1    tradeoff is right on the money -- but I'd say

          2    it's more difficult than solving a linear

          3    program to determine this cost-benefit

          4    process because clearly the collection of

          5    data involves discrete decisions about

          6    setting up the apparatus for collecting data,

          7    and then the process of just turning the

          8    crank to get the data out is probably much

          9    less costly.  So it's really an integer

         10    program you have to solve to determine what

         11    to do.  In other words, once you have a

         12    series going, then the cost of keeping it

         13    going is probably much less than the process

         14    of deciding what to do.  So I'd say it's sort

         15    of a difficult issue, but I think that --

         16    just my question really harks back to this

         17    idea:  We have policy issues which can be

         18    solved on the basis of back-of-the-envelope

         19    calculations that are informed by data, and

         20    there are other more subtle questions that

         21    models help tease out which may identify data

         22    that aren't obvious at the outset.  And so



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            55


          1    just being aware of that as part of the

          2    process I think is helpful.

          3              MS. KHANNA:  There's another way of

          4    framing the discussion, which is to say are

          5    specific elements of the energy market

          6    changing in any way fundamentally, and how

          7    does that drive our modeling and therefore

          8    data -- or data and therefore modeling --

          9    needs.  Because I look at the EU and the huge

         10    shift in oil prices, which makes a lot of

         11    sense to me, but that's a whole different

         12    band than we've seen historically.  I know

         13    you're talking about 30 year band and the $50

         14    to $60 price range; what's driving that?

         15    That's a fundamental change in the oil market

         16    as far as I can tell.

         17              MS. ANDERSON:  The external study

         18    team focused a lot on this, particularly in

         19    the area of electricity and electric sector

         20    restructuring and whether we really were

         21    collecting the data that were consistent with

         22    these huge changes in that part of the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            56


          1    market, and I think we'll have some

          2    recommendations on what we might do to be

          3    more up to date in that area.  So they

          4    certainly were cognizant of these issues, and

          5    they also talked a lot about the

          6    international factors driving markets, which

          7    would certainly come to your point at looking

          8    at oil markets in a different way.

          9              MS. KHANNA:  But it also goes back

         10    to John Conti's point about taking the

         11    discussion up to a different level.  If you

         12    can frame it in the sense, look, here is how

         13    the market is changing, and we have a whole

         14    different set of questions we want answered,

         15    it might help you overcome the hurdle which

         16    you are perceiving.

         17              MS. ANDERSON:  I keep harking back

         18    to if there are different clients for data

         19    series, they may be people like you who are

         20    using the raw data into a model you're

         21    creating that looks a whole lot different

         22    than any models we're doing, and then the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            57


          1    client base that is for model output, which

          2    is a different sets of clients.  And so our

          3    data are not only for input for our models,

          4    but serve their own purpose, their own end

          5    result, and trying to find areas where what

          6    we're collecting for that audience is the

          7    same as what we're using internally, and

          8    that's where you have a really great

          9    situation.  But there's an awful lot of data

         10    we collect that doesn't go directly into a

         11    short-term or a long-term model, but is used

         12    purely as output for answering perhaps more

         13    basic questions but not integrated questions

         14    are maybe not used in a modern framework.

         15    And is that client any less valuable to us or

         16    valuable to EIA than the clients that use the

         17    model base that -----

         18              MS. KHANNA:  I'd be willing to

         19    hazard that the same users that use your raw

         20    data for their modeling would be upset if you

         21    actually stopped providing some of the series

         22    that you are because you're both users of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            58


          1    your raw data as well as your model output

          2    for the most part.

          3              MS. ANDERSON:  That's right.

          4              DR. BURTON:  I'm going to suggest

          5    something that probably is close to heresy.

          6    The discussion and the decision processes

          7    that you're talking about take place in the

          8    private sector all the time, but they're

          9    guided by pricing.  If you're constrained in

         10    what you're able to do, you provide the

         11    products that people are willing to pay the

         12    most for and you observe that.  You don't

         13    have -- at least given the current structure

         14    -- the ability to do that.  You don't have

         15    the price signals from the users, both

         16    internal and external, that a private firm

         17    would have, but that's what you're trying to

         18    do, is essentially develop a decision process

         19    to substitute for what a market would do in a

         20    private setting.

         21              DR. NAGARA:  Could I ask a specific

         22    question?  Are there some examples where you



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            59


          1    felt that if you can run this survey instead

          2    of once a year, twice a year, that would make

          3    a tremendous impact on your model accuracy,

          4    for example?  I'm wondering whether there's,

          5    based on all the models more or less here

          6    today, I just wish I had more data on this

          7    particular one?  I think that in order to

          8    answer Nancy's question, to really bring it

          9    up to a higher level of discussion, if you

         10    have a example of something ---- say, hey,

         11    listen to me, you are doing pretty good here,

         12    but you can to four times as well by spending

         13    just a little bit more.

         14              MS. ANDERSON:  Well, maybe the best

         15    example is on the consumption survey side,

         16    that if the Short-Term Energy Outlook had

         17    annual consumption data from the Residential

         18    Energy Consumption Survey, we would be more

         19    accurate in our ability to predict heating

         20    expenditures or fuel usage across the year by

         21    region.  Yes, that's probably an area where

         22    we would enhance the accuracy of our



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            60


          1    estimates.  Again, we get back to -- it's

          2    unlikely that we're -- I mean, we're going

          3    the opposite direction.  We're collecting it

          4    in less-frequent intervals ----- some years.

          5    Most of the other data series that drive the

          6    Short-Term Energy Outlook are collected on a

          7    monthly or weekly basis, and we're

          8    constrained by the monthly basis for the

          9    Short-Term Energy Outlook, and I don't think

         10    we want a weekly Short-Term Energy Outlook.

         11    Maybe we do, but I keep hoping we don't.

         12                   (Laughter)

         13              DR. NAGARA:  Please don't say that.

         14              MS. ANDERSON:  But certainly there

         15    are some data that we've cut back on,

         16    definitely on the consumption side, that I

         17    think would help both of our modeling

         18    efforts, the long-term and the short-term.

         19              MS. KIRKENDALL:  The only problem

         20    with that is that the consumption surveys are

         21    among the most expensive ones we run, and

         22    that's why the frequency has been reduced.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            61


          1    Not that they weren't useful.

          2              MR. CLEVELAND:  Responding to

          3    Mark's comment, which I found interesting,

          4    thank God our decisions aren't driven by

          5    market considerations, and if they ever start

          6    becoming that, you're in deep trouble.  Look

          7    at the kids' page, the hits on the kids'

          8    page, how important -- you're in a really

          9    difficult place because energy now, more so

         10    than ever, is a huge business, and energy

         11    data, billion-dollar decisions, daily, hinge

         12    on your information.  But energy also is

         13    becoming more important in a broader social

         14    context.  The role of energy in education --

         15    look at the number of hits on the kids' page

         16    -- is amazing.  Information about climate

         17    change and CO2, the increasing role of energy

         18    in international markets, which aren't these

         19    short-term financial issues, are also

         20    increasingly important.  And so you really --

         21    I would argue those issues need serious

         22    attention as well as the stuff that people



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            62


          1    are banging on your door for every day, and

          2    that puts you in a -- in one sense, a good

          3    position because energy is more important

          4    throughout all aspects of society, but in a

          5    budget-constrained environment it's a

          6    difficult row to hoe.

          7              MR. WOOD:  I've certainly said in

          8    front of this group -- and I was here before

          9    the beginning, I think, so that's more than

         10    30 years -- that the analytical needs of EIA,

         11    as mandated -- those areas where you have a

         12    responsibility for -- and anything we can

         13    justify a reason for including -- ought to

         14    drive the data collection.  In practice, over

         15    time, sometimes that doesn't -- since a

         16    data-collection enterprise may be quite

         17    expensive, to change them -- and they always

         18    have constituencies, if only internal -- is

         19    sometimes difficult.  But nevertheless, the

         20    analytical needs of the organization and the

         21    public for energy data and analysis are what

         22    ought to drive the data collection process,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            63


          1    and that means periodically something new

          2    might have to be added and something old

          3    might have to be dropped, although I would

          4    caution against dropping a lot of things just

          5    because they went down in Web hits during one

          6    quarter -- the proverbial drive my company so

          7    that the market numbers are hit for this

          8    quarter because long-term data series that

          9    extend back 50 and 100 and 150 years are

         10    really important if you think you're going to

         11    be giving insight into questions which may

         12    unfold over 25 or 50 or 100 years.

         13              Secondly, I'm going to ask a

         14    question tomorrow, so I might as well start

         15    asking it today, on how in detail you

         16    actually make a decision whether to add

         17    something or cut something.  And in

         18    particular, in the 914 survey last year, in

         19    large part, potentially to get through the

         20    OMB process, we decided that we needed to get

         21    natural gas production data for a couple of

         22    major producing areas, and so did we want the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            64


          1    number one area or did we want the top five

          2    or the top ten, and top ten sounded like a

          3    big number, so we ended up with six -- and

          4    everybody else.  Six is a nice number; it's

          5    divisible by two and one and three, but the

          6    compelling logic is slightly not totally

          7    obvious.  On the other hand, there are also

          8    budget questions:  The more states we collect

          9    in detail, the larger perceived burden it is

         10    on the respondents, and two, the larger

         11    budget requirements for EIA.  And so what we

         12    did, and what drove it was, these are the top

         13    six producing areas, so we're going to do

         14    them individually because we ought to pay

         15    attention to them.  I think I'm going to

         16    suggest we do more like the top ten when we

         17    modify the 914, and there's something that

         18    goes with some of the opening remarks.  We

         19    think there are going to be some major

         20    increases in production from unconventional

         21    oil resources.  Those states right now do not

         22    produce very high amounts of oil, but we



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            65


          1    assume one or two of them will, and we assume

          2    -- or I assume -- that we ought to add them

          3    to the list of the states we are formally

          4    going to collect data from.  And so that is

          5    how to -- where do we cut it off?  There are

          6    37 states that produce oil and gas, it seems

          7    like the top ten would be a useful group to

          8    have detailed state-level information on, and

          9    we're going to do that in the time period of

         10    a month.  The next question is, if we assume

         11    that the future of oil production is going to

         12    ride on unconventional resources, should we

         13    add a couple of states right now that we

         14    assume are going to change their production

         15    levels in the immediate future or in the next

         16    decade.

         17              MR. BINGHAM:  Can I ask a quick

         18    question?  I'm Derek Bingham, by the way.  So

         19    I'm just looking at the remarks from last

         20    time so I -- this is my first time here, so

         21    bear with me -- but I'm just kind of curious

         22    -- so two people commented about prediction



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            66


          1    errors, I guess, and I'm curious how much

          2    time is spent kind of explaining the

          3    prediction errors in some of these models.

          4    And the reason I would say that, for instance

          5    -- this may correlate with what Lee's saying

          6    -- is that if you didn't include these areas

          7    that may be quite important, then at least

          8    you could go back after the fact say, well,

          9    it turns out that these areas are now

         10    becoming an important part of the oil supply

         11    or whatever it is that they're producing.

         12    They're producing whatever it is.  And I

         13    guess my question is for looking at and

         14    explaining why you missed in a model and then

         15    either building components into the model for

         16    that reason because they're now important

         17    where before they weren't.  Are there large

         18    efforts in that respect?

         19              MS. ANDERSON:  Well, I can talk

         20    about what we're doing in the short-term

         21    context, and John can talk about for the

         22    longer term.  This is the program that Nancy



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            67


          1    alluded to in the beginning in her bullets

          2    that we are -- her office is now and her team

          3    is reviewing as a proposal we have on the

          4    table for looking at forecast areas for about

          5    40 key variables that come out of the

          6    Short-Term Energy Outlook, precisely to

          7    answer the questions you're asking, try to

          8    understand why we're off, and if that's a

          9    missing data problem, could we fix it by more

         10    data, could we fix it by changing the model,

         11    as well as just informing the public of how

         12    accurate are we and explaining that in some

         13    cases you can't get any more accurate than

         14    what we're doing because these are often due

         15    to weather variations and things that we're

         16    not going to really enhance our predictive

         17    capability of.

         18              So, in fact, we're trying to do a

         19    better job of explaining to our readership

         20    what our errors are, as well as to ease that

         21    information back to our modelers to try and

         22    either fix the models or improve data, data



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            68


          1    accuracy -- whether that's a problem with the

          2    accuracy of the actual data, the timeliness

          3    of the data, or, in fact, missing data, we

          4    don't know yet, but we would certainly want

          5    to move the program in that direction.  John

          6    already does some of that in the long-term

          7    model.

          8              MR. CONTI:  We've had a forecast

          9    evaluation report, I believe, that EIA's

         10    published for about a decade and a half now,

         11    and it doesn't go into as many variables as

         12    Margot does or will; we have more aggregate

         13    variables.  But we've done it for every AEO

         14    since 1982, and so some of those earlier

         15    AEOs, we actually have, where our 15- or

         16    20-year predictions are actually data values

         17    now, and we go back and we assess why we

         18    didn't predict -- well, if we notice big

         19    discrepancies, we try and go back and assess

         20    why we didn't predict it correctly, and we

         21    see if we still have that same faulty logic

         22    and models we use today.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            69


          1              We do tend to look at the most

          2    recent years to see if we're trending

          3    differently from the actual numbers and to

          4    try and understand that.  But we don't sort

          5    of have -- it's just a matter of resources

          6    and how much resources you allocate to each

          7    of these efforts, but we do do that.

          8    Actually, one question that I was going to

          9    ask the committee somewhere in the next

         10    couple of days was in this forecast

         11    evaluation report we look at the sum of the

         12    absolute errors to give an indication of how

         13    far we're off.  Well, that's a good idea of

         14    how far you're off, but I'm always interested

         15    also in bias.  I'm not so worried if we're

         16    equally dispersed high and low, but I'm more

         17    worried if we have a bias to our forecast.

         18    So it was a question I was going to -- what's

         19    a good measure of bias?  I was thinking of

         20    maybe putting out graphs where people could

         21    visualize it.  To me, that's the most

         22    instructive rather than some other



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            70


          1    statistical single number measurement.

          2              MS. ANDERSON:  And we'll be using a

          3    bias measure in our estimates.  It's one that

          4    we're proposing.

          5              DR. HENGARTNER:  One of the nice

          6    things about the model is that it's not too

          7    picky of what kind of data it needs.  I find

          8    that the many applications that -- Rolling

          9    Stones are right, you don't always get what

         10    you want, but you get what you need.

         11              In a way the question I'm asking

         12    is, we are very focused on continuing

         13    historical series, which is a good thing, but

         14    in terms of improving our models and

         15    predictions, there are other sources of data

         16    that may be available out there essentially

         17    for free -- maybe not exactly free, but close

         18    to free -- which are not exactly what we

         19    want, but they're close enough that they

         20    could be incorporated into the modeling

         21    effort.  That's what models are for, right?

         22    To borrow strength of things that kind of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            71


          1    inform us -- and yes, it's not a perfect

          2    match, but it has value.

          3              And so while I don't have any

          4    specific examples -- I see John nodding his

          5    head -- I'd like you to tell me, yes,

          6    actually, that's something I've been thinking

          7    about.

          8              MR. CONTI:  I'm sure there are a

          9    lot of analysts in the audience here, but I

         10    know there are variables or data we use that

         11    EIA does not collect, that we purchase from

         12    outside sources, and only because it's just a

         13    lot cheaper to do.  It's used commercially,

         14    so somebody picks it up and sells it to a lot

         15    of people, and we just buy it and procure it.

         16    So I don't know if that was the direction

         17    you're going in, but for example, in

         18    commercial floor space, we tend to buy those

         19    projections.  We buy a lot of our

         20    macroeconomic information.  And we use other

         21    sources from other agencies.  So we purchase

         22    data where it's cost effective, but if it's



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            72


          1    just raw energy data that no one else

          2    collects, and we sort of have responsibility

          3    for more or less -- if it were to be

          4    collected, we would be the ones to collect it

          5    -- and it gets back to prioritizing -- how do

          6    you prioritize which data you collect?

          7              For example, John mentioned it

          8    became very relevant for us to collect this

          9    weekly natural gas storage information.  I

         10    don't need that in my long-term models,

         11    Margot doesn't probably need weekly in her

         12    monthly models, but I --- most of us would be

         13    lynched if we said we shouldn't collect it

         14    weekly because, no, we don't use it.  No,

         15    there are a lot of people outside of EIA that

         16    would kill us if we stopped collecting it on

         17    a weekly basis.

         18              DR. HENGARTNER:  But I'm trying to

         19    answer more the question, what data do we

         20    need or what does our model inform us about

         21    data needs.  And as I said, there is this

         22    dichotomy of two things, and you again go



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            73


          1    back, prioritize, what are the questions you

          2    want to answer --

          3              MR. CONTI:  Right.

          4              DR. HENGARTNER:  -- what are our

          5    clients' need in terms of raw series, but if

          6    you're going to go and really answer the

          7    question, what does the model tell us about

          8    our data needs --

          9              MR. CONTI:  Right.

         10              DR. HENGARTNER:  -- then you are

         11    entitled, and I would highly encourage you,

         12    to think broadly and in new ways because

         13    that's what models are for.  And raw data is

         14    something else, and I don't want to discount

         15    your need of raw datas -- and I'm not

         16    advocating that either, so --

         17              MR. CONTI:  Right.

         18              DR. HENGARTNER:  -- don't get me

         19    wrong here -- it's just -- I'm trying to

         20    answer very pointedly the question that I

         21    think was set out in the beginning.

         22              MR. CONTI:  Okay.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            74


          1              DR. HENGARTNER:  Yes?

          2              MR. WATSON:  Bill Watson with

          3    CNEAF.  I'd like to come back to the issue of

          4    use of more data by external parties,

          5    especially people that are trying to do their

          6    own modeling.  We do have at EIA -- at least

          7    from my perspective -- a fairly systematic

          8    way to understand what the data needs are of

          9    people that like our AEO modeling effort,

         10    come to our AEO outlook meeting, and we get a

         11    fair amount of feedback from them.  John has

         12    already talked about an assessment that would

         13    look at the data and the data needs for any

         14    enhancements or changes in the modeling

         15    efforts in that area.  I'm not aware -- and

         16    this is partly a question, partly a statement

         17    -- that we have anything like that on the

         18    side of the users of our survey data.  We

         19    don't have an annual meeting, or we don't

         20    have a systematic way to communicate with

         21    people that just use our data series.  Many

         22    of them would be modelers, and they would



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            75


          1    have some good opinions probably and good

          2    recommendations about whether we're missing

          3    some data gaps there from their perspective.

          4    So I guess maybe the question I would ask to

          5    the committee is, do we need to have some

          6    sort of balance here between the internal use

          7    of the data for modeling and the use of the

          8    data by external modelers and some way to

          9    assess on the external side what the data

         10    gaps may or may not be.

         11              Just to give an example, in the

         12    coal area, especially in the last 12 to 24

         13    months, we get a continuing series of calls

         14    from people asking about coal prices:  What

         15    data do you have on coal prices; what data do

         16    you have on coal transportation costs; what

         17    data do you have, other than your annual data

         18    that's with a two-year lag, on coal stocks.

         19    And so we're hearing this from the outside

         20    community, but it's not good, I think, to

         21    base recommendations on ad hoc information

         22    such as telephone calls; you need to do it



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            76


          1    systematically.

          2              MR. CLEVELAND:  Well, that raises

          3    the question of how well do you know who your

          4    client base is.  I mean, there's Web hits and

          5    downloads, but that's pretty limited kind of

          6    information.  And so I agree the Annual

          7    Energy Outlook meeting, which I have

          8    participated in, is a good way to get

          9    feedback from at least some slice of your

         10    users for that, but it seems to me that some

         11    other type of form -- we don't necessarily

         12    have to do it every year -- whereby you try

         13    to identify who these other users are of the

         14    various types of data you collect, bringing

         15    them together and getting some of that kind

         16    of feedback.  But the question is how do you

         17    think you know who your users are?  We know

         18    the squeaky wheels are the ones that -- we

         19    know who the big squeaky wheels are, but

         20    that's clearly not who we want to be making

         21    all our decisions around.

         22              DR. BURTON:  The question that you



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            77


          1    asked, I think is where I thought this

          2    discussion was probably going to go to begin

          3    with.  And one thing that I find is fortunate

          4    -- and may be a way to sort of sell modeling

          5    efforts -- is by doing modeling internally,

          6    you're likely to run into a lot of the same

          7    problems that we as external users encounter,

          8    and the fact that you do do modeling

          9    internally is a way to inform the process

         10    about what data is most useful.

         11              MR. CLEVELAND:  So you do do

         12    customer surveys?  What kind -- what are

         13    those like?

         14              MS. KIRKENDALL:  We've done --

         15    let's see, when are we done?  Where's

         16    Colleen?  I want to -- we've participated in

         17    the American Customer Satisfaction Index, and

         18    that's a survey that's associated with the

         19    Web so it's a random selection and people are

         20    invited to participate in the American

         21    Customer Satisfaction Index.  That's one way

         22    we've done surveys of people who are on our



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            78


          1    Listservs, thinking that people who sign up

          2    to be on a Listserv are keenly interested in

          3    those products they've signed up for.  And of

          4    course, it's not complete, but --

          5              MR. CARUSO:  And we do the ones

          6    where -- the pop-up surveys for users.

          7              MS. KIRKENDALL:  We do pop-up

          8    surveys for users.

          9              MR. FREEDMAN:  We've done surveys

         10    of the NEMS conference -----.

         11              MS. KIRKENDALL:  Yeah, we do

         12    surveys of the NEMS conference.  They do that

         13    regularly to monitor how well the conference

         14    runs.  We've also done surveys -- it's a

         15    sample from the Listserv for the AEO and for

         16    the STEO -- to get some input from them on --

         17              DR. BURTON:  Do you get a good

         18    response rate from them?

         19              MS. KIRKENDALL:  Well, no.

         20              MS. ANDERSON:  No.

         21              MS. KIRKENDALL:  About 25 percent.

         22              MR. ANDERSON:  Nor is there often



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            79


          1    space to get really at some of the detailed

          2    questions you want to get at that maybe if

          3    you brought people together in a focus-type

          4    setting, you might get -- and you --

          5              MS. KIRKENDALL:  Yeah.

          6              MR. CONTI:  I was going to say, we

          7    do that with -- we have a follow-up on the

          8    NEMS conference where we get sort of really

          9    heavy users of our models and our forecasts,

         10    and we do in-depth surveys with them and try

         11    and understand their needs and faults they

         12    have or problems they've identified.  And so

         13    we do what we do at the conference for

         14    everybody, and we get a fair response rate,

         15    and then we go into detail with some people

         16    from a broad array of areas that do energy

         17    modeling.

         18              MR. SINGPURWALLA:  Do the people

         19    outside the Energy Department, are their

         20    issues and needs different than the ones

         21    inside?

         22              MR. CONTI:  Sometimes they're more



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            80


          1    specific.  Yeah, they have a more specific

          2    interest and they want to go into greater

          3    detail than we do. But a lot of times, we

          4    tend to focus on people that are trying to do

          5    similar things.  Sort of broad national-level

          6    forecasting.

          7              MR. SINGPURWALLA:  Right.

          8              MS. ANDERSON:  But have we ever

          9    done anything that might get the users of the

         10    consumption surveys together in a room or in

         11    a conference and ask specifically about the

         12    coverage and the use of the data that we are

         13    putting out, whether it's -- how we put it

         14    out or how frequently we put it out or the

         15    depth of coverage, and I don't think we've

         16    done that on all of our surveys, we've

         17    probably done it on some and asked more

         18    Web-based questions.  But we probably haven't

         19    gone into as full detail on some of our

         20    surveys; it might be beneficial to do that.

         21              MR. CLEVELAND:  Yeah.  I don't

         22    think the issue is NEMS so much --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            81


          1              MS. ANDERSON:  Yeah.

          2              MR. CLEVELAND:  -- because I think

          3    you could do a pretty good job of getting

          4    feedback on that, it's the other broader

          5    array of data products whose constituencies

          6    are much flatter and broader and less

          7    well-identified than I think is the issue

          8    here.

          9              MR. CONTI:  Some of those issues do

         10    come up.  We typically have workshops -- and

         11    we meet a couple times a year -- in the

         12    modeling area, and some of those go back to

         13    -- like the industrial workshop tends to go

         14    back to the Manufacturing Energy Consumption

         15    Survey, and we get people from EPA and

         16    sometimes outside the Alliance to Save

         17    Energy, and there'll be discussions about the

         18    MECS at that point in time -- and Tom's been

         19    there and people from your office have been

         20    there to talk about the MECS.  So it's not

         21    focused on the MECS, but it's focused on

         22    industrial modeling and some of the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            82


          1    discussions deal with the MECS.

          2              MS. KIRKENDALL:  We do -- when

          3    we're considering redesigning surveys -- it

          4    used to be at least -- but when electric

          5    power was going to redesign their surveys,

          6    and I think they did this in natural gas, too

          7    -- they had a series of focus groups with

          8    users, or different kinds of users.  So you

          9    might have our own users and then other

         10    federal users and other groups of users that

         11    you could identify to come in and talk about

         12    data needs.  So those kinds of meetings would

         13    be held at the time you want to redesign the

         14    forms, so it's not an ongoing effort.

         15              MS. FORSYTH:  Those same people

         16    might also be used, not just to talk about

         17    data needs, but to talk about other data

         18    users that might be beyond the scope of the

         19    current data collection so that they'd be a

         20    source of perhaps exploratory data in that

         21    kind of question that you're asking.  There

         22    are some that will listen.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            83


          1              DR. HENGARTNER:  We're 10 minutes

          2    early.  Any more comments?

          3              MS. KIRKENDALL:  Well, if you think

          4    of anything, let us know.

          5              MR. CLEVELAND:  I do have one other

          6    one, actually.  And I'm wondering if the

          7    external review panel -- this seemed to be

          8    one of the fundamental questions that they

          9    must have talked about or -- did they collect

         10    any other information in their review about

         11    whether or not -- how we go about collecting

         12    data and identifying who the users are and so

         13    on and so forth?

         14              MS. KIRKENDALL:  They looked at all

         15    of our -- we put together a lot of

         16    information from our customer feedback, all

         17    the vehicles that we have, so they got

         18    summaries of all of that.

         19              MS. ANDERSON:  They certainly made

         20    some suggestions on things we ought to take a

         21    second look at and really ask ourselves why

         22    are you collecting this, is it still relevant



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            84


          1    to collect this information, and for us to

          2    really ask ourselves that question we're

          3    going to have to do precisely what you asked:

          4    Make sure we fully understand who's using it

          5    and what the costs are of scaling back or

          6    eliminating entirely.  They did feel that due

          7    to change in mission or change in the sector,

          8    there are some things that we're collecting

          9    that are really not relevant to the evolution

         10    of the energy sector, or could be augmented

         11    by private sector data, and it may, in fact,

         12    be a net savings to EIA.  So I think that's

         13    an interesting discussion.

         14              MS. KIRKENDALL:  Yeah, but they

         15    really didn't go into detail on our core

         16    program; they sort of assumed that the core

         17    program and the data that we need to inform

         18    the energy users was, if not wonderful, at

         19    least it was necessary, and they didn't get

         20    into the details because it wasn't a long

         21    enough --

         22              MR. CLEVELAND:  Sure.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            85


          1              MS. KIRKENDALL:  -- time period to

          2    do that, really.  So that there were a few

          3    things they identified as low-hanging fruit

          4    that they weren't sure were relevant.

          5              MS. KHANNA:  I just had one

          6    cautionary plea as you consider using data

          7    collected by other sources, especially

          8    private sector, is to focus a lot of data on

          9    the accuracy and robustness of the data

         10    collection process as well as the final data

         11    that you get.  And a great advantage of EIA

         12    collecting its own data is you know exactly

         13    how it is collected and what the pitfalls and

         14    strengths of it are.  If you buy -- as a data

         15    user, I mean, I'm always worried about that.

         16              MR. RUTHERFORD:  We have five

         17    minutes to go until the planned time, so I

         18    thought I'd just add a comment.  I thought

         19    Bill's comment about thinking creatively

         20    about ways of soliciting information about

         21    how data is used is an intriguing one.  I'm

         22    thinking there are two ways that you could



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            86


          1    leverage this:  One would be to sort of

          2    organize a special issue in the Energy

          3    Journal or something about a specific topic

          4    that uses, say, natural gas data from DOE

          5    would be the theme to try to leverage input

          6    from academics who have worked in the area.

          7    Also, if you're thinking about sort of model

          8    revisions at DOE, having a chance to sort of

          9    get a survey up close of what's been done

         10    with your data would be useful for thinking

         11    about how to move forward.  And the other

         12    thing is perhaps prizes.  I don't know, you

         13    could have some sort of -- anyway, there's --

         14    I just -- I think it was just -- this is sort

         15    of off the top of my head; it just seemed

         16    like an interesting idea.

         17              MR. CLEVELAND:  Yeah, I like the

         18    idea of this special issue.  An analogous

         19    thing on the modeling front might be to get

         20    Hill Huntington and the Stanford Energy

         21    Modeling Forum folks together and maybe have

         22    one year -- one cycle of the modeling forum



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            87


          1    to essentially give you guys feedback on what

          2    you guys are doing.  I think that would be a

          3    very interesting use of some pretty

          4    high-powered talent.

          5              MR. CARUSO:  Yeah, we've worked

          6    very closely with Hill and the EMF, so that

          7    would be good.  I like both of those

          8    suggestions.

          9              DR. HENGARTNER:  Okay, seems this

         10    discussion has ran its course.  We're

         11    fortunate to be ahead of schedule, and this

         12    is a good thing.  I'd like to take a break

         13    and remind you that we're going to continue

         14    afterwards in break-out sessions in Room 7-E

         15    -- that's here; Derek Bingham, Johnny Blair,

         16    Moshe Feder, Barbara Forsyth, myself, Walter

         17    Hill, and Darius Singpurwalla should remain

         18    here.  And Mark Burton, Mr. Cleveland, Jae

         19    Edmonds, Neha Khanna, Nagaraj Neerchal, and

         20    Tom Rutherford will go downstairs to Room

         21    5-E.  And we will reconvene in this room

         22    afterwards for discussion from these



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            88


          1    break-out sessions at 11:35.  So we have

          2    about 15 minutes break, and before we all

          3    leave this room, those who trickled in, one

          4    last time, please identify yourselves so we

          5    can have you on the record.  Thank you very

          6    much.

          7              MR. GRUENSPECHT:  I'm Howard

          8    Gruenspecht with EIA.

          9                   (Recess)

         10              DR. HENGARTNER:  This is the next

         11    break-out session.  It's going to be the 2006

         12    MECS.  By the way, I was told that those

         13    microphones are picking up just fine if you

         14    project towards them.  I'm just relaying the

         15    message I heard from the master.  So this is

         16    the 2006 MECS looking at past performance

         17    statistics to motivate new method of

         18    collection, and it's Bob Adler and Tom Lorenz

         19    who are going to be leading our discussion --

         20    at least giving us an introduction -- and

         21    hopefully we're going to all chime in and

         22    argue and discuss some more after this



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            89


          1    presentation.  So, welcome Tom.  Thank you.

          2              MR. ADLER:  Okay, I'm Bob Adler,

          3    and that's Tom Lorenz.  First of all, can

          4    everybody hear me?  We will be talking about

          5    how the 2002 MECS performance statistics kind

          6    of justify and motivate the 2006 MECS and our

          7    big redesign.  We want to thank Rick Hough,

          8    Vicki Haitot, and Stacey Cole, and since

          9    Susan's here, I'll thank her too, Susan

         10    Bucci --

         11              MS. BUCCI:  Thanks.

         12                   (Laughter)

         13              MR. ADLER:  I know which side of

         14    the bread the butter's on.  They helped us

         15    prepare some of the data and also read some

         16    of the drafts we had of the paper.

         17              I'm having trouble with this.

         18              Okay, as I said, we'll be looking

         19    at two kinds of performance statistics.  The

         20    first is data status flags, and the second

         21    will be what we've been calling differential

         22    response rates.  Hopefully that will lead to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            90


          1    a discussion on these topics, how we boost

          2    the Internet participation, performance

          3    statistics that would be useful to track.

          4    When we talk about that, the risks in

          5    targeting non-response follow-up to larger

          6    establishments, and there'll be a few others,

          7    and I hope it'll sort of generate from there

          8    -- or degenerate.

          9              The organization of this talk,

         10    we'll first -- hopefully briefly -- we'll

         11    talk about the background on the MECS.  Then

         12    I'll talk about the examination of the data

         13    status flags that are resident on the StEPS

         14    database, which keeps the MECS data.  Then

         15    we'll go into a discussion.  Tom will talk on

         16    the Excel version of the electronic forum

         17    that we introduced in 2002 to a select group

         18    of respondents, and we'll look at the

         19    performance statistics between those two

         20    groups.  And then I'll come back and talk

         21    about the differential response rates.

         22              Okay, the MECS.  I think most of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            91


          1    you know what it is.  It is the government's

          2    look at manufacturing establishments for

          3    their energy consumption.  We hope it's

          4    unique, otherwise OMB wouldn't let us do it.

          5    It's based on the NAICS codes.  We'll try not

          6    to talk a lot of NAICS during this

          7    discussion.

          8              What is important for this is that

          9    we have three survey forms that go out; we

         10    have the "A," "B," and "C."  The "A" is

         11    mailed to most of the establishments; it's

         12    sort of the default form.  We have the "B"

         13    form, which goes to only petroleum

         14    refineries.  It's actually a shorter form

         15    because it would otherwise duplicate some of

         16    the other EIA data systems that are out

         17    there.  And finally, the "C" form, which is

         18    sent to our energy-intensive and industries

         19    that have complicated flows, so we ask a

         20    little bit more; they're a little bit more

         21    customized.

         22              It has been a self-administered



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            92


          1    written questionnaire.  As I said, we are

          2    moving away from that, hopefully, and we did

          3    have this 2002 Excel version that was our

          4    first step in doing that.  It's designed and

          5    sponsored by EIA, although Census will be

          6    helping with the design this year, too.  And

          7    it is totally confidential, according to

          8    Title 13.  It's mandatory by federal law.

          9    People have a tendency to not get too excited

         10    by that.  And it's a statistical sample from

         11    a list frame of 15,000 establishments.

         12              Okay, these are the data status

         13    flags.  It makes a nice acronym of RAISE, and

         14    the "R" flags and the "A" flags are the two

         15    that really are the most common, and I'll

         16    really be contrasting those.

         17              The reported data is an "R."  And

         18    this means that one way or another, a

         19    respondent has come back and said this is

         20    what the data is.  Now, it could have been

         21    through the report form itself, or it could

         22    have been a follow-up using telephone or an



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            93


          1    e-mail.

          2              The "A" flag is an analyst

          3    correction of a reporting error.  That means

          4    just that.  There are a number -- if they can

          5    see that a wrong unit was used or a wrong

          6    price was used, then they're going ahead and

          7    based on their knowledge and what's on the

          8    form, they have enough to bring together

          9    something to that and then change the data

         10    appropriately -- we hope.  But it is an

         11    analyst correction, and I'll get back to the

         12    reported data -- also could have analyst

         13    corrections.

         14              "I" is imputed data.  It's really

         15    -- and you guys can help me at a later time

         16    -- I don't think -- it's very rare, it just

         17    means that they -- it's usually imputing an

         18    expenditure or price -- expenditure or a

         19    quantity from one or the other based on -- if

         20    that's all we have available.

         21              "S" is very important; it's the

         22    source data.  If we don't have -- this



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            94


          1    particular year we ran it the same time as

          2    the Census of Manufacturers -- or now it's

          3    called the Economic Census/Manufacturing --

          4    and that's actually drawing data from another

          5    source.

          6              And "E" is a system edit -- it's

          7    also rare -- and it means that we've seen

          8    enough of it so that we can put a program

          9    edit.

         10              Let me get back to the "R" flag for

         11    just a second.  The "R" means that we can

         12    look at -- that generally, you're talking

         13    about an actual response data.  However, we

         14    need to somehow differentiate, in some cases,

         15    whether an analyst was involved or not.  Now,

         16    StEPS allows us to do that by comparing two

         17    versions of the data:  The edited version of

         18    the data and the "reported" version of the

         19    data, and I know they sound the same.  By

         20    doing that, you can then determine whether an

         21    analyst was involved in actually making that

         22    correction.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            95


          1              Now, some of these data, we're

          2    going to talk about "R" as if it's good and

          3    "A" as if it's bad; obviously, there are some

          4    problems in that.  Now, the "S" data means

          5    it's a good piece of data, but there's some

          6    trouble with our questionnaire because we had

          7    to grab that piece from another data source.

          8    So one -- I guess that's -- let's just get

          9    into the data then -- and I've handed this

         10    out, and I guess it's dark so you can't read

         11    it there either -- but what this is, it's a

         12    nice graph, although it probably is against

         13    all of our SMG-EIA standards, but it's a good

         14    way of looking at something very quickly and

         15    glancing down and seeing that, okay, where

         16    are the concentration of "A"s and "S"s and

         17    "R"s that point out something amiss with the

         18    data.  Now, the blue flags -- blue is the

         19    reported data -- and I've labeled one, two,

         20    and three.  The LPG nonfuel is number one,

         21    and you can see there are relatively few that

         22    are actually reported data.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            96


          1              I talked a little bit before about

          2    the "C" form; usually, if you're not on the

          3    "C" form, you should not have LPG as a

          4    feedstock or a nonfuel, so what that is is

          5    essentially analysts coming in and saying,

          6    I'm sorry, this is not right, they shouldn't

          7    have that.  And again, you guys will tell us

          8    if that's true later.  And also, within that

          9    blue, we've had the most difference between

         10    the reported analyst type and the reported

         11    actual data, so that's something that we need

         12    to look at, and I think it's something we can

         13    clear up a bit more with the electronic form.

         14              The second, number two, is actually

         15    much more important.  The natural gas as a

         16    fuel, as you might guess, is very common;

         17    9,000 cases reported natural gas as a fuel,

         18    and that's sample cases.  So what you're

         19    looking at, the blue, that looks pretty good,

         20    and there was less than 10 percent of the

         21    blue that actually represent analyst

         22    corrections.  However, that green is about 20



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            97


          1    percent -- a little less -- and if we add the

          2    others -- and that really means that analysts

          3    have to spend their time correcting that

          4    data.

          5              Now, what is that data?  A lot of

          6    times, it's a missed unit, a missed price, a

          7    lot of things that we can collect better if

          8    we had an electronic version of the form.  So

          9    a lot of that data -- now, if we were to take

         10    20 percent of 9,000 cases, you can sort of

         11    get at what exactly the analysts have to do

         12    and spend their time.  Imagine shifting that

         13    to the respondent and have them do it; we can

         14    probably get the MECS out a lot faster -- at

         15    least that's out goal.

         16              Finally, the third one is the other

         17    LPGs, it's abbreviated 'enes as nonfuel

         18    propylene, ethylene, butylene.  What that

         19    means -- it's sort of a rare type of LPG, and

         20    I really just want to point out the blue --

         21    purple -- whatever that color is at the other

         22    end that's not blue, not green, and I -- it



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            98


          1    looks black to me.  But what that is is the

          2    source data.  We had to load up from the

          3    Census of Manufacturers their nonfuel use,

          4    and as it turned out, what they do is they

          5    call it material use -- is that right -- and

          6    that term -- uses a material -- and that term

          7    seemed to have keyed them off to respond

          8    correctly more than our standard MECS terms

          9    of nonfuel and feedstock.  And yes, we did

         10    explain what nonfuel and feedstock, and yes,

         11    they were supposed to respond.  Nevertheless,

         12    somehow they got it in their heads that they

         13    weren't supposed to.

         14              Okay, I've already talked about all

         15    of this, so I'm going to pass it to Tom, who

         16    will talk about the differences between Excel

         17    versions.

         18              MR. LORENZ:  Hi, everyone.  You

         19    guys have any questions in between, just

         20    holler out, just let me know.  As Bob

         21    mentioned, in 2002 we created an Excel

         22    version of one of our forms, and it happened



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                            99


          1    to be the "C" form.  The reason why we chose

          2    the "C" form was, one, because it was our

          3    longest form, and number two, because, as Bob

          4    mentioned, the respondents of that type of

          5    form are the higher-intensity energy users,

          6    such as the chemical, paper, and primary

          7    metal industries, so they're more used to

          8    filling out energy-type data, and we felt

          9    that they would probably best be able to fill

         10    out the Excel version.  We made the Excel

         11    version look like the written questionnaire

         12    so that the ease of -- and anxiety that

         13    people would have of filling out an

         14    electronic questionnaire would be somewhat

         15    reduced.  The added capabilities that we had

         16    to the Excel version, we had price and value

         17    range checks of most of our major energy

         18    sources.  We also had calculations --

         19    additions and things like that -- for our

         20    total consumption of energy sources.  And we

         21    copied those totals to where we needed them

         22    at other points in time in the form -- with



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           100


          1    the paper form, we have the respondent just

          2    copy the number themselves into the other

          3    sections such as end use and fuel switching.

          4    So we did that automatically for them.

          5              There were some limitations that we

          6    had with this Excel form, mainly time was a

          7    factor.  Because we had to have the

          8    respondents save the information to a CD

          9    and/or diskette, we couldn't include all the

         10    online or edits that we wanted -- screen

         11    edits that we really truly wanted for our

         12    Excel version.  We also -- because it was an

         13    Excel version, and we were sending it in --

         14    we were having the respondents send in either

         15    a CD or diskette.  It wasn't a true Internet

         16    data collection system in the sense that they

         17    were not reporting it via the Internet.  So

         18    we didn't have the capability of having the

         19    Census Bureau incorporate some of the

         20    programmatic and integrated systems for their

         21    data capture.  And one of those faults was

         22    that we really had to -- because they sent us



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           101


          1    the information, they had to key it in the

          2    same manner as they would a paper form, which

          3    was one of the downfalls, and that's why

          4    we're actually really targeting going more

          5    towards an IDC which would eliminate a good

          6    portion of that keying aspect, which

          7    increases our error.

          8              For the Excel version we had 323

          9    responses, which was about 8.9 percent of the

         10    total number of "C"-form responders.

         11    However, the Excel responders accounted for

         12    about 12.4 percent of the total consumption

         13    that was reported using the "C" form.  Now,

         14    we had about 23 respondents who sent us a

         15    diskette or a CD -- mainly a diskette -- that

         16    failed for some reason or another.  About 80

         17    percent of those 23 respondents, it was

         18    incorrectly formatted was the message that

         19    was received.  The other was a bad diskette.

         20    So whether it was -- I don't know what

         21    exactly happened in the mailing process or

         22    they didn't format it or what the -- but



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           102


          1    that's just another reason for us to really

          2    target the Internet data collection process.

          3              DR. HENGARTNER:  Have you tried to

          4    run those diskettes on a Mac?

          5              MR. LORENZ:  Yeah.  And it just

          6    wasn't -- there was either nothing on it or

          7    it wasn't reading or just something was -- I

          8    know now a lot of times that the diskettes

          9    are already formatted, so I don't know really

         10    what happened in the process.

         11              I'm going to just talk a little bit

         12    about a couple of the major energy sources,

         13    particularly first and foremost, electricity,

         14    the electronic versus the paper, and I'm

         15    going to target, as Bob mentioned before, the

         16    reported data, which is the "R"s and then the

         17    "A" flag, which is the analyst corrections.

         18    With the first -- actually the third and

         19    fourth columns, the one edit that we had was

         20    for the price and range checks.  As you can

         21    see, there's not much of a difference between

         22    the electronic and paper forms.  However, the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           103


          1    last column, the 96.3 percent and the 85.7

          2    percent, there's about a 10 percent

          3    difference there, and we attribute that to

          4    the fact that there was an edit there where

          5    we actually calculated the information for

          6    them instead of having them do it.  So that

          7    increased our electronic response rate to a

          8    much higher percentage.

          9              Now, with natural gas, there are a

         10    few things actually with natural gas.  I'll

         11    go through the third column, where the total

         12    natural gas purchased.  As you can see,

         13    there's quite a variance in the "A"s between

         14    the electronic and paper, about 11 percent.

         15    And if we go back here -- I'm going to go

         16    back real quick -- you notice that there's

         17    quite a bit higher than what's on the

         18    electricity.  We attribute that to the fact

         19    that a lot of times we ask for the respondent

         20    to give us their information in thousands of

         21    cubic feet, which is Mcf, and with natural

         22    gas, they have about five or six different



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           104


          1    various type of units that they could report

          2    it in.  So one of the things we're looking at

          3    for 2006 is to incorporate some type of a

          4    drop-down box to have them choose their units

          5    so that we can actually, I guess, correct it

          6    for them and process it for them in the

          7    processing system.

          8              Also, in the total natural gas fuel

          9    consumption column, which is the last column,

         10    again, you can see the variance where it

         11    worked in the same manner as in electricity,

         12    where there was about a 12 percent difference

         13    in their total consumption between electronic

         14    and paper.  And again, we attribute that to

         15    the fact that they didn't have to calculate

         16    and add up their numbers; we did it for them.

         17    And the one last thing that I want to mention

         18    is down in the bottom right-hand corner, the

         19    8.3 percent is attributed to an "E" flag, and

         20    that was systematic error that we found.  We

         21    basically just made a systematic correction,

         22    let the system do the correction for us.  So



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           105


          1    we're going to incorporate some of these

          2    things into the system that we're building

          3    where the Census Bureau is held, and we're

          4    using their system, which is Census Taker, so

          5    we can get better responses much faster and

          6    improved.

          7              DR. FEDER:  I may have missed it --

          8    you said in '02 it was up to the respondent

          9    to choose which mode or which instrument to

         10    use, right?

         11              MR. LORENZ:  Right.

         12              DR. FEDER:  For '06, it's going to

         13    be all electronic, or --

         14              MR. LORENZ:  That's our goal, yes.

         15    We want to -- yeah, but we can't really

         16    expect it, though, because that's our goal is

         17    to hopefully have as many people as possible

         18    because it'll save on mailing the forms out

         19    and all that, but we also do understand that

         20    there are some manufacturers we have --

         21    because it is a survey sample -- we have some

         22    that are large establishments and some that



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           106


          1    are small establishments, and sometimes they

          2    may not have the capabilities or their

          3    establishment or company may not allow them

          4    access to the Internet.  So we do have a

          5    paper form.  But we're going to really market

          6    the electronic form or the Excel or Internet

          7    data collection systems very heavily.  And

          8    that's our goal is to really reduce the

          9    mailing that we have to do with the survey

         10    and instead have them complete the survey

         11    electronically through the Internet.  Thus,

         12    we can do a lot of the programmatic things

         13    behind the scenes in the data processing

         14    system and collect the data through the

         15    Internet.

         16              So, with that, I'm going to hand

         17    this back over to Bob to talk about response

         18    rates.

         19              SPEAKER:  Well, if it gets to be a

         20    higher response rate if they respond and

         21    decide which instrument to use, -------------

         22    I'm trying to imagine what you're thinking.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           107


          1    You think -----.  If the respondent chose the

          2    instrument, you'd get a higher response rate.

          3    If the respondent can't use the electronic

          4    form or doesn't have Excel or has trouble

          5    with the floppy disks or ---- being used, the

          6    response rate would be higher.

          7              MR. LORENZ:  Well, the economics of

          8    running the MECS, it's going to be more

          9    economical if we don't have to mail out all

         10    the forms.  So we're hoping to market that as

         11    much as we can, but we can't limit that to --

         12    because, depending upon whether they have the

         13    Internet access or not, we can't say, you

         14    can't fill out this survey because you don't

         15    have Internet access, so we do have to have

         16    some type of a backup.  And that paper form

         17    is our backup.  Typically, you will --

         18    hopefully we will see -- what we saw is that

         19    there was a little bit better response rate;

         20    there were fewer corrections that needed to

         21    be made with the Excel form in 2002, so we're

         22    hoping that that holds true for the 2006 when



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           108


          1    we go true Internet.

          2              DR. FEDER:  If you send them a free

          3    laptop, they'll be happy to --

          4                   (Laughter)

          5              MS. FORSYTH:  I had a question.

          6    When you're talking about response rates

          7    there, that you had less edits, are you

          8    talking about item non-response or unit

          9    non-response or both?

         10              MR. LORENZ:  I think it's item.

         11              MS. FORSYTH:  Item level.  And you

         12    were talking about unit non-response, right?

         13    Okay.

         14              MR. SINGPURWALLA:  Is it cheaper

         15    overall to do paper or Excel?

         16              MR. LORENZ:  Well, we had to pay

         17    for the printing costs -- it's -- hopefully,

         18    it's going to be -- it'll probably be about

         19    the same, but what we're hoping is that in

         20    the back end -- yeah, it'll reduce our

         21    editing time.

         22              MR. SINGPURWALLA:  Yeah.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           109


          1              MR. LORENZ:  And so in that sense

          2    it'll save us some times.

          3              MS. FORSYTH:  Getting rid of those

          4    diskettes will help too.

          5              SPEAKER:  The Internet thing.

          6              MR. LORENZ:  That was really our

          7    only method of collecting it at that point in

          8    time, but -- yeah.

          9              DR. FEDER:  I wonder if the '02

         10    mode that was user selected did not affect

         11    even some of the responses.  Clearly there

         12    could be some association between the type of

         13    industry, the type of respondent, and their

         14    choice of modes.  So there could be potential

         15    bias in letting them --

         16              MR. ADLER:  That's one of the

         17    questions we had.

         18              DR. FEDER:  That's the answer.

         19    That's it.  I'll go with Bob.

         20              MR. ADLER:  Yeah.  The problem with

         21    using that is that it is a self-selecting

         22    group.  On the other hand, we're not doing an



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           110


          1    experiment there, we were actually collecting

          2    data, so -- we thought that it would work out

          3    better that they did, in fact -- I hope those

          4    statistics show that they did do better.

          5              Now, did they do better because

          6    it's just a better group of responders or

          7    that our edits on site actually helped?  I

          8    hope it's the latter.  I don't really have a

          9    reason to think that this was a better group

         10    of responders, but I don't know.  I don't

         11    think we have enough to really discern that.

         12              MS. BUCCI:  Tom, on the paper

         13    version, when you were making those

         14    comparisons, were you looking at all three

         15    form types, the "A," "B," and "C," or just

         16    the "C"?

         17              MR. ADLER:  Just the "C."

         18              MR. LORENZ:  Just the "C," yeah.

         19              MR. ADLER:  So we did try to at

         20    least have that much -- it was the same

         21    industries overall.  They were

         22    energy-intensive industries.  We did not



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           111


          1    count the "A" and the "B," which would

          2    probably show a lot more errors because they

          3    -- the "A" especially has a tendency not to

          4    know what we're talking about.

          5              I'll go ahead and talk about this

          6    real quickly because I know we're running

          7    long.  Another performance statistic that we

          8    looked at was really response rates, and

          9    we've been unfortunately suffering from

         10    declining unit response for, I think, a

         11    couple of survey cycles.  I think maybe our

         12    highest was in '94.  In '98 it's gone down,

         13    and 2002 also has gone down.  We think that

         14    partly that is downsizing, competing with the

         15    overseas market more than they have been,

         16    could be an attitude toward government

         17    surveys -- I'm not allowed to say that, but I

         18    did.

         19                   (Laughter)

         20              MR. ADLER:  Yeah, right there.  But

         21    there's a general sense that's probably

         22    happened within the last couple of decades



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           112


          1    that surveys are not the first priority of a

          2    manufacturing company and establishment.  On

          3    the other hand, our budgets are being

          4    squeezed, as well as the manufacturers, so we

          5    have to do things to shift our money and

          6    shift our resources to best get at a better

          7    quality of the data, even if we can't keep up

          8    the non-response rate or the response rates

          9    that we had a decade ago.

         10              MR. SINGPURWALLA:  Is there a

         11    reason that the budgets are being squeezed

         12    for the surveys?  Do you have any hypothesis

         13    on that?

         14              MR. ADLER:  I think it's -- that's

         15    a tough question for me to answer.  I think

         16    it's just that the budgets were --

         17              SPEAKER:  -------

         18              SPEAKER:  Really?  It's everything

         19    across the board.  It's not like money's

         20    being shifted.

         21              MR. ADLER:  I don't think money is

         22    being shifted away from our type of survey



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           113


          1    into others, but I think, yeah, I think we

          2    are just seeing a decline --

          3              MR. SINGPUERWALLA:  Well, even,

          4    like, surveys in general across -- even

          5    outside DOE --

          6              MS. KIRKENDALL:  Actually, EIA's

          7    probably suffered more than other agencies in

          8    terms of declining budgets.  We have a little

          9    graph that we put in our budget material that

         10    shows other agencies --------- a little bit

         11    and ours were --

         12              MR. ADLER:  Do you guys do more

         13    surveys than most other --

         14              MS. KIRKENDALL:  No, but that's a

         15    comparison to other statistical agencies, so

         16    I don't know whether I've ever compared

         17    counts of surveys, but we all do surveys.

         18              MR. FRENCH:  I don't know if there

         19    is time for comment from the audience, but to

         20    be fair, effective with fiscal year '06, the

         21    consumption area of EIA did get a several

         22    hundred thousand dollar a year boost in our



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           114


          1    funding, which essentially brings us back to

          2    be able to conduct our consumption surveys as

          3    we have them right now, and as we've had them

          4    the last couple of cycles -- once again, at

          5    those same levels, with higher cost levels,

          6    of course.  So right now we're okay at the

          7    levels that we're doing things.

          8              MR. ADLER:  Our survey is selected

          9    on the basis of an energy measure of size,

         10    which is the cost of fuels in electric energy

         11    from various sources, the annual survey of

         12    manufacturers.  It's on the frame and we then

         13    use that as a size measure.  So the achieved

         14    coverage rate is in terms of this measure of

         15    size.  It's a quantity over a quantity.  The

         16    response rate that you see there is mailed

         17    forms and acceptable returns.  We got more

         18    mail than we're showing there, but some of

         19    these returns brought our actual response

         20    rate down.

         21              What we did is we targeted the

         22    cells that have the most influence on MECS



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           115


          1    data, and so that's -- on the left-hand side

          2    you see a cell size that's generally, in

          3    terms of that cell's measure of size, we

          4    called -- we rated the influence of that cell

          5    on the total MECS data that we come out.

          6    Really, I guess it would be the total fuels

          7    correspond most to that.  These are the

          8    target coverage rates and, just to get over

          9    that, I think if you see that -- we also

         10    targeted certainty and non-certainty cases

         11    differently -- we felt that not only are they

         12    quantitatively different, they are

         13    qualitatively different -- so that we have a

         14    different non-response adjustment factor

         15    based on certainty and non-certainty.

         16              As you can see, we did achieve

         17    actually a better coverage rate than the

         18    target in all cases, regardless of our

         19    response rate, and so we're going to just

         20    publish that.  No, we do have to show both,

         21    but I think that the achieved coverage rate

         22    gives us another view of the quality of the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           116


          1    reported data, and I think something to take

          2    into account when we look at the MECS data --

          3    when we achieve the coverage rate that we

          4    were after, we stopped following up on those

          5    cells, and that's where the money -- that's

          6    where we can target the budgets a little bit

          7    better by doing that.

          8              Okay, I think we've already talked

          9    a lot about this.  We'll shift to an Internet

         10    collection.  We are going to do something to

         11    get back in contact with the actual people

         12    who fill out the forms earlier than we have

         13    to try to boost our response rate and also

         14    sell the Internet version of the form.

         15              The status flags will have a little

         16    bit more information because they'll be two

         17    character, so I think we'll be able to

         18    discern a little bit more from the status

         19    flags than we have before.  And then the

         20    questions for the committee -- I think you've

         21    already started talking about them.  We

         22    talked about the possibility of bias, and



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           117


          1    we'd like to know, boosting participation in

          2    the electronic form and other performance

          3    statistics that may be useful for us to

          4    track, both now and in the future, and this

          5    last part, what are the risks in targeting

          6    non-response, really, to larger

          7    establishments within a cell.  That's it.

          8    Thank you.

          9                   (Applause)

         10              DR. FEDER:  Can you put back the

         11    questions to the committee?  The slide with

         12    the questions?

         13              MR. ADLER:  Oh, sorry.

         14              DR. FEDER:  I couldn't --------

         15              MS. FORSYTH:  On the last page.

         16              DR. FEDER:  Sorry.

         17              MS. FORSYTH:  You want 1 and 2 or 3

         18    and 4?

         19              DR. FEDER:  Barbara commented that

         20    it's in the handout, so --

         21              MS. FORSYTH:  Oh, it's in the

         22    handout, okay.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           118


          1              DR. HENGARTNER:  Yes, Moshe?

          2              DR. FEDER:  My question actually is

          3    why do you do it every four years?  Why not

          4    have sort of a rolling survey?  Looks to me

          5    like this activity is very intense when you

          6    do it, doesn't provide you any information in

          7    the three years when you don't have the

          8    survey.  Why not have survey part of the

          9    population every year and make it easier?  I

         10    mean, is there a reason why it's done this

         11    way?

         12              MR. LORENZ:  Cost.

         13              MR. ADLER:  Well, there is that

         14    aspect of cost.  I think that the starting up

         15    each time has been a problem, and I think

         16    even though we would go to a smaller

         17    population, the actual getting them geared up

         18    is -- there's a base-level cost that is

         19    associated with it.  I really should defer

         20    that question to Dwight.  That's what you get

         21    paid the big bucks for.

         22              DR. FEDER:  But would it be cheaper



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           119


          1    to not do it every four years but do it on a

          2    continuing basis and --

          3              MR. LORENZ:  Well, one of the

          4    things -- can I add something to that -- one

          5    of things I think -- maybe I'm hoping -- and

          6    I think Paul agrees with me -- is that once

          7    we do get this Internet data collection

          8    system up and running, that we can then

          9    incorporate it maybe once every other year or

         10    something of that nature where it's more

         11    frequent because we can -- more timely and

         12    edit the data and get it out.

         13              DR. FEDER:  In surveys of human

         14    populations, usually what one is trying to do

         15    is try to balance good level estimates with

         16    trend estimates, and to do that, you cannot

         17    do it like that.  I know presidential

         18    elections cannot be held every year, but this

         19    is not the case.  I would try for a smarter

         20    "design" that can help measure annual changes

         21    with better estimates of the level of the

         22    absolute number.  I know from my experience



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           120


          1    in Canada, where the census is held every

          2    five years, that it's a huge thing to every

          3    five years gear up for that.  So I know some

          4    things are easier that you've been

          5    continuously learned from the experience than

          6    to keep modifying.  Here, if you you're your

          7    recommendation, yeah, you should do this and

          8    that, and you actually implement it, and it

          9    doesn't work, you have to wait another four

         10    years to change it.

         11              MR. LORENZ:  That's right.

         12              MR. BINGHAM:  Yeah.

         13              DR. FEDER:  With an Internet

         14    instrument, it's easier to make changes on a

         15    flow basis and good experience, so just a

         16    question.

         17              MR. FRENCH:  Dwight French again.

         18    You're not the first person to have mentioned

         19    continuous measurement; it was mentioned in

         20    terms, I believe in many past years, of both

         21    our residential and our commercial building

         22    surveys as well.  There are some logistics



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           121


          1    problems that would prevent continuous

          2    measurement in those surveys -- maybe a

          3    little easier to do in the MECS environment

          4    than in the personal interview survey

          5    environment.  However, this issue of, well,

          6    you can change the form and then go on -- if

          7    we were to do one quarter of the sample every

          8    year and try and do more continuous

          9    measurement, and you kept changing the

         10    questionnaire, then when you tried to do

         11    rolling four-year average statistics, you

         12    would essentially be rolling methodology as

         13    well.  And so one of the temptations to do

         14    this in batch mode is you do a survey, you

         15    see what doesn't work, you make your changes,

         16    you do it again four years later.

         17              DR. FEDER:  But, Dwight, I'm sure

         18    you will agree that comparing the '06 with

         19    the '02 will be very questionable in this

         20    case because of the weak changes that you are

         21    doing now.

         22              MR. FRENCH:  There are changes all



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           122


          1    the time that cause this issue.  For example,

          2    when we went from SIC to NAICS, we had a big

          3    dislocation, and we did do some comparability

          4    statistics, but we still had issues to deal

          5    with in regard to that.  Yes, we come up with

          6    those issues all the time.

          7              MR. SINGPURWALLA:  Is the most

          8    expensive part of performing the survey the

          9    follow-ups to it?

         10              MR. ADLER:  I'd say the editing

         11    portion, which is, yeah, receiving the data,

         12    processing the data, and cleaning it up,

         13    whether that's actually calling back or not,

         14    is probably where you get most of your money.

         15    I think -----, yeah.  The mail-out costs are

         16    substantial, but I think they're not the

         17    lion's share.

         18              MR. BINGHAM:  Were there

         19    substantially fewer errors with the

         20    electronic forms, and the errors that did

         21    occur, were they different than what had

         22    occurred before?  It's two questions.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           123


          1              MR. LORENZ:  I did a good portion

          2    of the editing, and Vicki also did some of

          3    the editing as well.  And, yeah, I did notice

          4    quite a few less errors.  Most of it was good

          5    reported data.  Sometimes when there was an

          6    error, or there was something that was

          7    questionable, I would sometimes call the

          8    respondent myself, and a lot of times, they

          9    were hammering about this Internet -- this is

         10    a good first step, let's go Internet, but

         11    they really liked the Excel form as it was;

         12    they thought it was a good first step, and a

         13    lot of the errors I saw were mainly in -- a

         14    good portion of them were keying, but that's

         15    going to happen.  They're not going to --

         16              DR. HENGARTNER:  So just in terms

         17    of cost, looking at this nice little graph we

         18    have here, in fact, you actually look at all

         19    the data you've collected, an analyst looks

         20    at it and then makes a determination if it's

         21    good or bad, right?  And so getting better

         22    data collection may not reduce the cost of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           124


          1    having to go through the whole data set

          2    anyway.

          3              MR. ADLER:  Well, I think it will

          4    reduce the cost.  One, redesigning the

          5    questions as we look at the one thing about

          6    the 'enes that needed a better word.  It's

          7    like having a cognitive interview as we're

          8    going live.  So I think that should help us

          9    with the errors.  And I think the big thing

         10    is shifting the burden of data cleanup from

         11    the analysts to the respondents, and I

         12    wouldn't put that to them like that, but they

         13    would be saving us money and actually helping

         14    us clean up the quality of the data.

         15              We would still have to run the

         16    edits through, but the feeling is that the

         17    data itself would be cleaner coming out from

         18    the electronic versions.

         19              MS. FORSYTH:  Can I just

         20    cosmetically make a small change to say that

         21    you're not really putting a burden on the

         22    respondent, you're putting it on the form.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           125


          1              MR. ADLER:  Okay.

          2              MS. FORSYTH:  I mean, it's the form

          3    -- so that something that's not the

          4    respondent is doing the work -- I mean, the

          5    form is the technical consistency in --

          6              MR. ADLER:  All right, but the

          7    respondent --

          8              MS. FORSTYH:  -- gives you better

          9    data.

         10              MR. ADLER:  Yeah, and it may take

         11    them five seconds longer than they did

         12    before, but yes.

         13              MS. KIRKENDALL:  Well, but that may

         14    lead to getting a phone call from EIA --

         15              MR. ADLER:  Absolutely, yeah, yeah.

         16              MS. KIRKENDALL:  -------

         17              DR. FEDER:  It's the drop-down unit

         18    selection.  That by itself is a huge --

         19              MR. ADLER:  Right.

         20              MR. SINGPURWALLA:  Is the goal to

         21    run it through some automatic edit checks?

         22              MR. ADLER:  We would be doing that.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           126


          1              MR. SINGPURWALLA:  And then you'd

          2    spit back something like an e-mail to the

          3    user.  It'd be, like, hey, this looks funky

          4    or --

          5              SPEAKER:  No, that's not --

          6              MR. FREEDMAN:  We're going to talk

          7    about this a little bit this afternoon in a

          8    broader context, but the general objective is

          9    to provide that feedback to the user in real

         10    time so they can actually correct it on the

         11    screen and so the incremental amount of time

         12    is not noticeable to the respondent.

         13              DR. FEDER:  What are you suggested

         14    could be added in addition to this.  It would

         15    be nice because if I were a respondent, I

         16    would like to write down what I actually

         17    responded to -- I would pull my records -- so

         18    it would be nice to get --

         19              MR. LORENZ:  One of the things that

         20    I think we're going to -- and I learned this

         21    from the Fed CASIC -- is that we're going to

         22    use the paper form as kind of a worksheet for



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           127


          1    them so that they can kind of work -- almost

          2    like doing your taxes, they can use the paper

          3    form as a worksheet and keep it themselves or

          4    even save a copy on their computer.

          5              MS. FORSYTH:  Well, traditionally,

          6    it's pretty easy to hit a "print" button as

          7    well, so that you just print what they --

          8              MR. LORENZ:  Yeah, create --

          9              MR. SINGPURWALLA:  Or you could --

         10              MS. FORSYTH:  It depends on --

         11              MR. SINGPURWALLA:  ---- file of

         12    some sort like a significant report that

         13    could look like your old paper form even --

         14    it just has the values right in there.

         15              MS. HAITOT:  Can I go back to

         16    Nicolas's -- my name's Vicki Haitot, by the

         17    way -- we didn't look at every single case.

         18    We ran searches and it would pull up things

         19    that were awfully wrong.  And sometimes we'd

         20    have to call back.  The electronic form is

         21    going to keep us from having to call those

         22    people because it will not have as many at



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           128


          1    the other end, and if we do the drop-down box

          2    for natural gas, we'll have them pick their

          3    unit because a lot of the natural gas issues

          4    was we had to figure out what units they were

          5    reported in because it obviously wasn't cubic

          6    feet.  And we have to call them, and then

          7    they would not necessarily be there, and

          8    they'd take weeks to get back to you.

          9              MR. LORENZ:  Or the person has

         10    moved on and changed jobs or whatever, so --

         11              MS. HAITOT:  It will save a little

         12    bit of time because we're not going to have

         13    as many that pop out of the edits or the

         14    searches we do that are wrong.

         15              DR. FEDER:  The issues of response

         16    rates -- have you considered an incentive,

         17    like fridge magnets or something?

         18              SPEAKER:  It doesn't work.

         19                   (Laughter)

         20              DR. FEDER:  Well, we had a survey

         21    of industries where we gave some things for a

         22    desktop memo holder, very nice one -- I wish



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           129


          1    I could get one -- I joined that project too

          2    late to get it -- things like that.  So when

          3    I -- maybe not a magnet.  But incentives do

          4    work, it's strange how -- one survey, by the

          5    way, sent respondents a $2 bill, which is a

          6    collector's item.  That was for survey of

          7    physicians, I believe.  Because if you send

          8    them 10 bucks, they're not impressed by it,

          9    but a $2 bill --

         10              MR. ADLER:  I don't think we've

         11    really looked at incentives, and I guess we

         12    were looking at just how much it would cost

         13    to really get them to do it.  And you're

         14    right, eventually you get to a real person,

         15    but these real people have to respond to

         16    whatever pressures they're under to do other

         17    work, and yeah, we just thought that what

         18    could we give them that would go against that

         19    that would compete with that?  And we haven't

         20    come up with any -- I think they're -- a

         21    vacation day, that was maybe one.  I don't

         22    think we had the ability to do that.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           130


          1              MS. FORSYTH:  I think Moshe does

          2    have a good point that it doesn't have to be

          3    a good thing in order to create goodwill, and

          4    really, especially with the mandatory survey,

          5    that's what something arriving with your

          6    invitation to participate does, it

          7    establishes goodwill and kind of a positive

          8    feeling in the respondent.

          9              MS. KIRKENDALL:  Do you know who

         10    your respondent is beforehand?

         11              MR. ADLER:  Well, that's -- then

         12    again, I think we have an address, a person

         13    we are addressing it to based on the frame

         14    information, but that's not normally -- or

         15    often, I should say -- the person who will

         16    eventually be completing the MECS.  So,

         17    again, we will be trying to identify that

         18    person earlier than we have before, and that

         19    means that maybe we'll be drawing the sample

         20    earlier and have a pre-mail-out.  So, yeah, I

         21    mean, I think if we could nail down that,

         22    then the idea of doing some sort of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           131


          1    incentives would work better.

          2              MS. FORSYTH:  We're using an

          3    untested model.  Like, in a year, I can give

          4    you results, but we have a facility study

          5    that we're conducting where there's a point

          6    of contact at the facility, and then, of

          7    course, the person who's going to key in the

          8    data.  And we don't know the person who's

          9    going to key in the data, but we have to

         10    recruit through the point of contact, and

         11    we're giving incentives at both levels:  A

         12    somewhat personalized incentive for the point

         13    of contact, and then a food incentive that

         14    can be distributed across multiple staff who

         15    may be involved in actually keying the data.

         16              And again, I don't know if it works

         17    or not, but that was one way of getting

         18    around the, we don't know who it is, and it's

         19    probably someone who's not going to receive

         20    an incentive package if we send just one.

         21    And I'd add that the incentive that goes to

         22    the point of contact is relatively



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           132


          1    inexpensive.  I'm kind of surprised at how

          2    inexpensive it is.

          3              MS. KIRKENDALL:  What is it?

          4              MS. FORSYTH:  For us, it's a

          5    portfolio -- it's a letter-sized -- it's not

          6    leather-bound, but it has a pad and a pencil,

          7    and it has the study logo on the cover, so

          8    it's kind of nice looking, and it's something

          9    people can carry to meetings and make notes

         10    in.  And a further advantage is on the

         11    inside, we have -- I shouldn't be giving away

         12    all our secrets, but --

         13                   (Laughter)

         14              MS. FORSYTH:  But on the inside,

         15    you have point of contact information on kind

         16    of a yellow sticky-type thing that they can

         17    pull off, so it actually serves multiple

         18    purposes, the incentive.

         19              DR. FEDER:  Can respondents, if

         20    they are too busy -- you mentioned that time

         21    was -- can they start filling the form and

         22    then save it and go back to it --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           133


          1              MR. ADLER:  Yeah, definitely.

          2              MR. LORENZ:  Yes.

          3              DR. FEDER:  Okay, good.

          4              MR. LORENZ:  Part of the system --

          5              DR. FEDER:  Good, good.

          6              MS. FORSYTH:  Interesting to

          7    measure when and how often that happens.

          8    Just --

          9              DR. FEDER:  I once quit responding

         10    to a survey because I had to go, and the

         11    system didn't let me save it, and I just -- I

         12    don't want to use any bad word, but I just

         13    said, I'm not going to do it because --

         14              MR. LORENZ:  Yeah, I mean, we --

         15              DR. FEDER:  I hope you are not

         16    writing that.

         17                   (Laughter)

         18              MR. LORENZ:  Yeah, the system we

         19    have to use because we are working through

         20    the Census Bureau, because we do collect the

         21    data under Title 13, we could do one of two

         22    things.  We could either create it ourselves



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           134


          1    and program it ourselves, which could take

          2    years and months, or we could use their

          3    system, which is Census Taker, and that seems

          4    like a more viable option to me.  Is it

          5    perfect as far as collecting the data

          6    perfectly all the time?  No.  But then again,

          7    no system is.  So it is what it is, and

          8    that's the best we have available.

          9              MR. SCHIPPER:  With Census Taker,

         10    is it a programming language that's applied?

         11    Because I know that other surveys that -- us

         12    and Dwight's group -- the system that's used

         13    is VLAZ, and we have a VLAZ programmer.  Do

         14    you have a Census Taker application that you

         15    can then program your own --

         16              MR. LORENZ:  No, there's no

         17    programming language.  It's going to be used

         18    throughout the whole Internet and all that.

         19    And Stan will get further into that this

         20    afternoon because we looked with the IDC on

         21    part of that team as well, and we looked at

         22    Census Taker as part of the IDC team, and



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           135


          1    that was one of the systems that we looked

          2    at.

          3              DR. HENGARTNER:  In terms of

          4    statistics, is everybody satisfied?  I mean,

          5    I'm looking at the achieved average coverage

          6    rate of maybe 85 to 90 percent.  Should we be

          7    concerned with non-response?  That's a pretty

          8    nice number, just on face value.

          9              MR. ADLER:  I guess, again, that's

         10    the coverage rate.  Our non-response is, in

         11    terms of actual forms, going down.  I'd be

         12    happy not to, but I think there is a concern,

         13    and I think there's an OMB standard that we

         14    are in danger of going below the threshold,

         15    and that has me worried.  And I would like to

         16    convince OMB that even if that happens by one

         17    percent, we are trying to maintain this

         18    coverage rate that is still maintaining the

         19    quality of the data.

         20              MS. KIRKENDALL:  I think we're ----

         21    coverage rate is probably more important

         22    indicator than actual response rate.  You



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           136


          1    might have to make a case, but I think you

          2    can make a case.

          3              MS. HAITOT:  We called a lot of the

          4    large guys that didn't respond to it to get

          5    the coverage rate up there.

          6              DR. FEDER:  That partially depends

          7    on the level of the estimate that you're

          8    interested in.  For national estimates and so

          9    on, the coverage rate is what you should

         10    worry about.  If your data contain some

         11    specifics on units that are not contributable

         12    to the overall coverage rate important for

         13    some ----- of the market, then you should be

         14    interested in those particular response

         15    rates.  Because if I wanted to use the data

         16    to do some design on small companies, their

         17    contribution to the response rate might be --

         18              MS. KIRKENDALL:  Well, if you want

         19    to do a study on small companies, then a

         20    response rate may not be quite what you --

         21              DR. FEDER:  Small companies or some

         22    more esoteric types of energy use.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           137


          1              DR. HENGARTNER:  If I have the

          2    choice of improving coverage rate between the

          3    certainty and the non-certainty samples,

          4    which one would you focus on, Moshe?

          5              DR. FEDER:  I don't think I have an

          6    easy answer to this question, frankly.  It

          7    again depends on the use of the data.

          8    Certain units tend to be bigger and easier to

          9    approach; they had a person who's easy to be

         10    tasked with responding and so on.  But I

         11    think often uses data on smaller companies on

         12    the non-certainty units.  I would say you

         13    need to have good response rates from both.

         14    I wouldn't target one in particular.

         15              MS. KIRKENDALL:  See, but that's

         16    always a tradeoff, of course, because if you

         17    take the same resources and you don't focus

         18    on the big ones, then your overall coverage

         19    rate is not going to be very --

         20              DR. FEDER:  That's what I think --

         21    well, Mark, maybe when he gets back, we

         22    should ask him this question because that's



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           138


          1    one question for a specialist and a data user

          2    rather than a statistician.

          3              MS. FORSYTH:  I want to go back to

          4    your point about the small niches that you

          5    might be missing and whether you know about

          6    those.  That's not really a question of bias,

          7    right, because if they're small, they

          8    wouldn't necessarily contribute much to --

          9              DR. FEDER:  That's not --

         10              MS. FORSTYH:  -- so it's --

         11              SPEAKER:  It's a different --

         12              DR. FEDER:  The bias is not on a

         13    national level, the bias is at the small area

         14    level.

         15              MS. FORSYTH:  Right, right,

         16    within --

         17              DR. FEDER:  Yeah.

         18              MS. FORSYTH:  -- so you wouldn't

         19    see it.  Okay.

         20              MR. HILL:  But are you going to

         21    mention you're doing a stratified sample and

         22    you really want -- there are two questions:



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           139


          1    One is the overall number --

          2              DR. FEDER:  Right.

          3              MR. HILL:  -- and one --

          4              DR. FEDER:  Exactly, exactly.

          5              MR. HILL:  --------.

          6              DR. FEDER:  Exactly.

          7              MS. KIRKENDALL:  Well, that's

          8    another difficulty.  --- is a very large

          9    survey that tries to be everything to

         10    everybody, so targeting that one

         11    particular --

         12              DR. FEDER:  One size fits all.

         13              MS. KIRKENDALL:  -- user's needs is

         14    impossible.  So I think that's why the

         15    targeted response rate was going to try to

         16    get at least the national level of estimates

         17    or a pretty good quality, and yeah, you get

         18    down to regional or however you cut it down

         19    -- it might not be quite so -------.

         20              MR. ADLER:  Yeah, I worry when

         21    people do microdata work, and they start

         22    using ---- maybe by industry, by region, and



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           140


          1    really getting down into trying to identify

          2    differences there.  We really don't guarantee

          3    that the sample works at that level.

          4    Certainly, if you're really cutting it down

          5    and if people have the option, those that

          6    have census agent status to do those things,

          7    and it's also possible that the data aren't

          8    as clean for those establishments, for those

          9    items at that point because they don't impact

         10    on the tables that we present, but they may,

         11    in fact, impact on the researchers' data at

         12    that point.

         13              DR. FEDER:  But my point is -- and

         14    I remember when I joined this committee three

         15    years ago that I was surprised that certain

         16    establishments were not included in some data

         17    collection efforts just because they were

         18    small.  And that is not the current

         19    statistical practice; the current practice is

         20    to downweigh them, but not to exclude them

         21    altogether.  And then when I look at these

         22    coverage rates, which are good, and we say



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           141


          1    that's okay, even though the response rate

          2    may be going down, I just want to say, you're

          3    right for the national estimates, and that

          4    might be okay.

          5              I think we need to ask a data user

          6    for their view on that, because they might

          7    tell you that some of the interest might be

          8    in new technologies that might not be covered

          9    because their contribution to the coverage

         10    rates might be low, but the interest might be

         11    going high because of the trend, because we

         12    know that in the future, maybe their niche is

         13    going to be more important, and we might miss

         14    a new development in the market that analysts

         15    might want -- also my concern was the

         16    four-year frequency because did we know four

         17    years ago what's going to be with the energy

         18    market, what we are seeing now?  No.  So this

         19    is why I'm asking.

         20              But again, my only knowledge about

         21    energy is how much I pay for the gas I buy,

         22    that's why I think sometimes small-area



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           142


          1    estimation -- I'm sorry, I didn't hear the

          2    technique, but the interest in small-area

          3    statistics might be worth it.  That's all.

          4              MR. SCHIPPER:  To answer your

          5    question, the Census Bureau conducts the

          6    census every five years, but also conducts an

          7    annual survey of manufacturers, which does

          8    not embody the scope or level of detail of

          9    energy-related information that the MECS

         10    would cover.  But there are trend data

         11    available related to energy costs and

         12    electricity usage on an annual basis.  So

         13    essentially MECS comes around every four

         14    years to kind of benchmark the entire energy

         15    portfolio for manufacturing.

         16              DR. FEDER:  And I presume that some

         17    of the modeling is taking all that data into

         18    account, right, so we can --

         19              MR. SCHIPPER:  In other words, the

         20    energy modelers at EIA enjoy it when MECS

         21    comes around, but they have alternative data

         22    for those intervals --- MECS ----.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           143


          1              MR. GIFFORD:  Just another comment

          2    on the small establishments.  The ones that

          3    are excluded from the MECS frame -- and Bob,

          4    you may remember this better than I -- I

          5    think are only account -- in terms of

          6    coverage, about two and a half percent.  So

          7    if -- and you really need to think about the

          8    motivation for the survey.  For -- want to

          9    concentrate on publishing total consumption

         10    numbers, then those guys may need to go just

         11    because it's not --

         12              MR. ADLER:  They represent about

         13    150,000 -- these are rough numbers -- 150,000

         14    out of about 350 --

         15              MR. GIFFORD:  Right.

         16              MR. ADLER:  So by cutting them out,

         17    we've lowered the population number by

         18    150,000, we're down to 200,000, but we also

         19    still maintain 98 percent of the payroll in

         20    manufacturing as a measure of the frame

         21    coverage.  So I think it's a good tradeoff,

         22    but obviously when you talk about



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           144


          1    establishment counts -- and some of our data

          2    do show that -- we have to be clear that

          3    that's what we're talking about, that this

          4    particular population --

          5              MS. KIRKENDALL:  This survey is

          6    more like other agencies' probability

          7    surveys, it's not a cutoff survey.

          8              MR. ADLER:  No.

          9              MS. KIRKENDALL:  Every survey has

         10    some cutoff.  A lot of the Census Bureau's

         11    cut off at five establishments or something

         12    because you just can't get any information

         13    for the itty-bitty tiny guys.  And that's

         14    where our frame comes from.  We do have other

         15    cutoff samples, but that's not what this one

         16    is.  And we do have a probability sample of

         17    the smaller guys, and they, too, try to get

         18    the response in and just can't concentrate on

         19    it.  They concentrate instead on getting the

         20    response rates up on the big ones, the most

         21    energy-intensive ones.

         22              DR. HENGARTNER:  The one thing I'd



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           145


          1    like to see, which you haven't shown us, what

          2    is the non-response rate on the follow-up?

          3    Because I'm sure not every follow-up

          4    survey --

          5              MS. KIRKENDALL:  Interesting

          6    question.

          7              MS. FORSYTH:  Are you talking about

          8    unit follow-up or item follow-up?

          9              DR. HENGARTNER:  I would say unit

         10    for sake of argument.  Yeah, not -- item,

         11    it's too much work; unit -- I'm going back

         12    then ask the company, please give us the

         13    data.  Now, I'm not convinced that they're

         14    going to respond 100 percent to your

         15    follow-up.

         16              MR. ADLER:  Oh, they don't.

         17              DR. HENGARTNER:  They don't.  So, I

         18    mean, one of the questions to ask, tell me

         19    what statistics would be interesting to know

         20    -- well, follow-up non-response is in a way

         21    indicative of actually diminishing returns --

         22              MR. BINGHAM:  You might want to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           146


          1    measure whatever your -- these are the large

          2    establishments here and the small ones as

          3    well -- to see what information on --

          4              DR. HENGARTNER:  And send it to the

          5    IRS.

          6                   (Laughter)

          7              MS. KIRKENDALL:  You may have an

          8    indication of that.  Did you follow up with

          9    all of the non-respondents in the certainty

         10    groups?  No.

         11              SPEAKER:  Is that a question?  I

         12    mean --

         13              MS. KIRKENDALL:  Well, yeah, I was

         14    asking the Census Bureau who knows these

         15    things.

         16              MS. BUCCI:  No.  We start with the

         17    larger ones.

         18              MS. KIRKENDALL:  Okay, so even

         19    within a certainty group, you may not do

         20    follow-up with everyone of the --

         21              MS. FORSYTH:  But once you get your

         22    coverage rate, you stop, right?



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           147


          1              SPEAKER:  Right.

          2              MS. KIRKENDALL:  Okay, right.

          3              DR. FEDER:  But you need not only

          4    the total numbers of non-response, you need

          5    the characteristics so one could even build a

          6    propensity model for making future

          7    adjustments.  Because if you know more about

          8    the type of establishments that do not

          9    respond, you can do something with it, such

         10    as use non-response adjustment that's taking

         11    into account the industry, the size, the

         12    location, and whatever.

         13              MR. BINGHAM:  This plot might

         14    actually help you in some sense and tell you

         15    ----.  So I guess my question would be in

         16    terms of coverage on certain mixtures of

         17    butylene, propylene fuels, you might have an

         18    idea of who the users -------.

         19              MR. LORENZ:  That's what we do,

         20    yeah, yeah.

         21              MR. BINGHAM:  And if you somehow --

         22    you know you got really ----- some of these



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           148


          1    particular ----.

          2              MR. ADLER:  I should say that when

          3    I -- this was a special final non-response

          4    follow-up, which is the calling.  We do send

          5    out letters to everybody who doesn't respond.

          6    And it lays out the mandatory requirement,

          7    and they get as many as two of that --

          8              MS. BUCCI:  Yeah, it's two mail

          9    follow-ups.  They actually get the form and a

         10    letter saying that they did not respond, and

         11    then the third follow-up would be phone

         12    calls.

         13              MR. ADLER:  Right.  And so I didn't

         14    make that clear, but we're not letting people

         15    off the hook right away.  We do have a

         16    general follow-up -- a couple of ones -- and

         17    it doesn't succeed in scaring them enough, so

         18    we then go to this other one.  I'm sorry, but

         19    that's what a mandatory requirement is

         20    supposed to do, and it's not working.  That

         21    piece of it doesn't work as well.

         22              MS. BUCCI:  One of the things you



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           149


          1    may want to consider for Census -- I mean,

          2    obviously, Census is also a mandatory survey

          3    -- when we are having problems with response

          4    rates, we actually have real high-up letters

          5    coming from counsel or whatever going

          6    directly to the presidents of the companies

          7    saying that this is mandatory, here are the

          8    penalties for not filing --

          9              MS. KIRKENDALL:  Right, send it to

         10    somebody other than the contact --

         11              MS. BUCCI:  -- we send it to

         12    somebody pretty quickly, those type of

         13    things, and we don't send it to everyone, but

         14    we do target certain establishments or

         15    companies to get those letters.  So that

         16    might be a possibility.

         17              MR. ADLER:  Tom called --

         18              MS. BUCCI:  Yes, Tom --

         19                   (Laughter)

         20              MR. LORENZ:  I'm not ashamed.

         21              SPEAKER:  Is that fair?

         22              MS. BUCCI:  I'm sorry --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           150


          1              MR. FREEDMAN:  Do you have any

          2    measure of the effectiveness of that?

          3              MS. BUCCI:  I'm not sure.

          4              MR. LORENZ:  And I talked to the

          5    CFO.

          6              MS. KIRKENDALL:  He called the

          7    president and talked to the CFO, he says it

          8    works.  And so other ways of contacting them

          9    may be good, too.  Yeah, the other thing --

         10    just ------ talk may be redesigning our

         11    second and third contact letters to try to

         12    make it sound like -- I don't know if

         13    threatening would help, but to try to make it

         14    sound like they really ought to respond for

         15    some reason, and then -- it's an area of

         16    research, I'm sure.  I mean, everybody's

         17    worried about response rates, and I was

         18    thinking about how to encourage people to

         19    respond to your surveys.

         20              MS. FORSYTH:  Right.  Not many

         21    people that I work with get the advantage of

         22    it being mandatory --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           151


          1              MS. KIRKENDALL:  Right.

          2              MS. FORSYTH:  -- and so --

          3    actually, an experiment in a mandatory

          4    context -- I mean, the Census has probably

          5    done them, experiments of the tone you take

          6    in these follow-ups and the impact of the

          7    tone, but I don't think it would have be a

          8    big experiment to --

          9              MS. KIRKENDALL:  Yeah, to get --

         10    right, to -- actually, that wouldn't be so

         11    bad if they could build into the -- just --

         12              DR. FEDER:  Yeah, but non-response

         13    could be for failure to comply could go to

         14    the wrong contact person.

         15              MS. FORSYTH:  That's right.

         16              DR. FEDER:  Point of contact.

         17              MS. FORSYTH:  Point of contact.

         18    Yeah, point of contact is a huge issue.

         19              DR. FEDER:  -- address or sometimes

         20    company going out of business or changing

         21    ownership, but I presume, then, those letters

         22    hopefully come back to you.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           152


          1              MR. LORENZ:  Not always.

          2              MS. FORSYTH:  Not if it's the wrong

          3    person.

          4              MS. BUCCI:  Part of what happens is

          5    our point of contact comes from the annual

          6    survey of manufacturers, and that's updated

          7    in our systems.  When we get a contact for

          8    the MECS survey, we can keep it in StEPS, but

          9    we don't pull our frame from StEPS, so we

         10    don't have a place to keep that contact, so

         11    analysts are going back and looking at it --

         12    may not be printed on the form.  So they are

         13    typically going to an accountant, and we want

         14    to hit the plan engineer for the survey.

         15              MS. HAITOT:  But a lot of times,

         16    the MECS respondent isn't there after four

         17    years either.

         18              MS. BUCCI:  That's true.

         19              SPEAKER:  Right.

         20              SPEAKER:  That's a long time.

         21              MS. BUCCI:  One of the things that

         22    I think we've tossed around, too, for the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           153


          1    electronic reporting is possibly doing

          2    company visits up front, offering the service

          3    of the Census Bureau or EIA staff to go and

          4    help them -- yes, Tom --

          5              MS. HAITOT:  And Vicki.

          6              MS. BUCCI:  -- and Vicki to go out

          7    and help them fill out their forms and --

          8    through the census-taker and try to encourage

          9    a response in that way.  But that costs, too.

         10              MR. GIFFORD:  One more comment on

         11    the MECS is -- when we met with the

         12    census-taker folks, this is quite a different

         13    survey than anything they've used it for

         14    before in terms of size.  And it's every four

         15    years -- and EIA -- the Internet data

         16    collection surveys, for the most part, are in

         17    smaller surveys that have responses every,

         18    say, month or week or year, where, with this,

         19    you kind of lose the learning by the

         20    respondent, you have contact problems, so we

         21    don't really know how it's going to go down,

         22    but hopefully it should be fairy successful,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           154


          1    but you don't know at this point because we

          2    asked for some statistics on a comparable

          3    survey, and they just didn't have it at this

          4    point.

          5              MR. ADLER:  Well, I think, though,

          6    that this does set us up better for the

          7    future than sticking with the electronic

          8    questionnaire.  I think, as Tom was saying,

          9    by doing that we may have the ability to go

         10    out more frequently, and then that learning

         11    curve would be improved.  But yeah, the real

         12    reason, I think, is that even if no one has

         13    seen it, if it's brand new to every

         14    respondent that comes out, we really think

         15    that we will just have a cleaner data set

         16    faster than we would if continuing with the

         17    written form.

         18              MR. SINGPURWALLA:  Well, it's

         19    automatically consolidated, too, which is

         20    nice.  So analysis will be a lot cleaner

         21    and --

         22              MR. ADLER:  Yeah.  So eventually we



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           155


          1    cut out keying entirely, and that's another

          2    five -- certainly a block of money that could

          3    be shifted to other areas.

          4              DR. HENGARTNER:  Nagaraj, don't go

          5    out.

          6                   (Laughter)

          7              DR. HENGARTNER:  I think we're

          8    wrapping up.  We have the other break-out

          9    session coming up from downstairs, so I'd

         10    like to thank everybody for their comments,

         11    and we'll be summarizing this break-out

         12    session shortly.  Thank you very much.

         13                   (Applause)

         14                   (Recess)

         15              DR. HENGARTNER:  I'd like to have a

         16    summary of both break-out sessions.  Our

         17    first session was about the 2006 MECS survey

         18    about past performance and predictions of

         19    what's going to happen, and I'd like to

         20    invite Mr. Moshe Feder to summarize our

         21    discussion we had earlier today.

         22              DR. FEDER:  Thanks, Nick.  First of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           156


          1    all, MECS stands for Manufacturing Energy

          2    Consumption Survey, and what's interesting

          3    about this survey, it's done every four

          4    years.  And that's for cost consideration and

          5    I may say a little bit about this.  There's

          6    some information that was provided to me

          7    after the break-out session.

          8              The last time MECS was conducted

          9    was in, I guess, '03 on the '02 data, and

         10    four years later, in '07, MECS is going to be

         11    conducted on the '06 consumption, so that's

         12    why it's called 2006 MECS.  In 2002,

         13    respondents had the option of responding

         14    either using a paper instrument or filling

         15    out an Excel equivalent of that that was made

         16    very much similar to the "C" form.

         17              There are three forms, "A," "B,"

         18    and "C," "C" being the most complex, and

         19    that's the instrument that energy-intensive

         20    establishments receive or ones that have

         21    complicated energy flows.  The experience had

         22    been generally that with the Excel



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           157


          1    instrument, there was less need for

          2    correction of information.

          3              In addition, for the '06 survey,

          4    the intent is to have some further

          5    enhancements; for example, one that I really

          6    liked is having a pull-down menu for the

          7    choice of units because it turns out that

          8    different establishments use different units,

          9    and that caused some problem in the past.

         10    Obviously, if you use a unit that's 10,000

         11    times another unit, your data will be quite

         12    different.  So that's another enhancement.

         13    They do have attached to every measurement

         14    flags that indicate whether it's the original

         15    raw data as provided by the respondent, an

         16    analyst correction, imputed data, information

         17    coming from another survey, or a system edit.

         18              I won't go into the specifics --

         19    that would take too much time -- but there

         20    was an interesting -- as much as possible,

         21    having the original data; obviously, edited

         22    data and system-edited data and so on are



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           158


          1    less desirable.  There was less of that with

          2    the electronic format, and hopefully in the

          3    past, all respondents, with a very few

          4    exceptions, will use the electronic form.

          5              We discussed also the issues of

          6    response rates and coverage rates and which

          7    measure to use and what should be done to

          8    improve response rates.

          9              Now, what is a coverage rate?

         10    Turns out that a few establishments dominate

         11    the estimate for any particular kind of

         12    activity, and those contribute more,

         13    obviously, to the coverage rates.  Small

         14    establishments don't have big impact.  And

         15    the target, if I got this correctly, for the

         16    coverage rate is about 90 percent.  There has

         17    been a slight decline in response rates, but

         18    for most purposes the big interest is in the

         19    coverage rate because as long as you cover --

         20    it's 90 percent of the target measure, then

         21    you're okay.  However, there are some OMB

         22    requirements for response rates as well.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           159


          1              One of the issues that I asked is

          2    why have this every four years.  Obviously,

          3    there's a cost involved here, but why not

          4    divide it over the four years, and why --

          5    explain to me something which I think is a

          6    very good point.  There are two reasons,

          7    actually, to not have it every year.  One is

          8    if you divide it over four years, then you

          9    get lots of fluctuations in your estimates

         10    because of the skewed nature of the

         11    distribution.  If in one year you survey the

         12    bigger establishments, you get different

         13    estimates than in other years.  Another is

         14    the disclosure analysis, which is required

         15    and actually mandated by law.  The smaller

         16    the sample you have, the more likely you are

         17    to disclose information, and also it will

         18    require more resources.

         19              So it is a cost-effective thing to

         20    do it every four years, and someone else

         21    provided another point of information:  There

         22    is annual collection by the Census of some



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           160


          1    relevant information, so it's not like every

          2    four years we collect data, and then we have

          3    three years of total --

          4              MS. KIRKENDALL:  Preparing.

          5              DR. FEDER:  Yeah.  And so we do

          6    have the ability to use for modeling some

          7    measurement of change across years, not just

          8    every four years.

          9              Now, I come into the issues of

         10    non-response and how to improve response

         11    rates and what should the measure be.  And I

         12    think people generally accepted that coverage

         13    rates are more important than response rates,

         14    although response rates might be of interest

         15    as well, especially since you can have

         16    biasing in the smaller domains.  But overall,

         17    the coverage rates are a good measure of the

         18    effectiveness of the survey.

         19              How do you improve response rates?

         20    Well, follow-up is one tool, and we know that

         21    even with follow-up, not everybody responds.

         22    We also need to have not just information on



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           161


          1    response rates, but information on the

          2    non-respondents.  Who are the

          3    non-respondents, how to they differ from the

          4    ones that responded, especially after the

          5    follow-up, so you can do some non-response

          6    adjustments and know who you're missing.

          7              We also talked about, briefly, the

          8    issue of incentives.  You can improve

          9    responses by handing out some incentives.

         10    Those should be given both to the

         11    point-of-contact person and also to people

         12    who actually do the entry of the data, and we

         13    give some examples from other surveys.  In

         14    addition, we know that some non-response is

         15    caused by bad contact information; especially

         16    since this is done every four years.  Maybe

         17    in the intervening time some companies

         18    changed officers or structure, addresses, and

         19    so on, so attention must be given to that as

         20    well.

         21              We did not adhere to the list of

         22    questions very specifically, so I'm not going



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           162


          1    to go through each one individually, but I

          2    think that pretty much covers what we talked

          3    about.

          4              DR. HENGARTNER:  Thank you very

          5    much, Moshe.  Anybody else who was in the

          6    break-out session who wants to add something

          7    to Moshe's rsum?  Okay, thank you very

          8    much.  I'd like now to invite Neha Khanna to

          9    summarize the discussion on a new oil

         10    production representation for the SAGE model.

         11              MS. KHANNA:  Thank you.  This was a

         12    very lively discussion -----.  Just to give

         13    the rest of the committee a brief overview of

         14    what the SAGE model is, it's, as we

         15    understood eventually, a 16-region,

         16    42-end-use sector, linear programming,

         17    cost-minimization model that takes demand as

         18    given, and also it's a model for non-OPEC

         19    production, it takes OPEC production or

         20    price, depending on which area you want to

         21    look at it, as given.  And it took us about a

         22    few minutes to actually reach that level of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           163


          1    understanding because we had so many

          2    questions, I think we didn't let the

          3    presenters even finish what they had to say.

          4              The questions were, I think, like

          5    the other group; we didn't exactly answer the

          6    questions that we were asked.  We didn't even

          7    allow the presenter to reach their questions

          8    because we kept butting in.  But I think part

          9    of the reason for that is that the model is

         10    still under development, and so we wanted

         11    more details than were, in fact, brought to

         12    the tables by the presenters partly because

         13    they don't as yet have those details

         14    themselves.

         15              So I think the one punch line --

         16    I'll come to the detailed questions -- but

         17    our one punch line was, we want more

         18    information, we want details, details, and

         19    more details, and we want to see more

         20    presentations on the SAGE model and how it's

         21    being developed.

         22              So that's the summary of the entire



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           164


          1    session.  The specific questions that we were

          2    particularly concerned about were, for

          3    example -- I'll take some examples.  One is,

          4    how do we -- how will the model -- there's a

          5    cost model and oil production model which

          6    sort of gives us resource codes for different

          7    non-OPEC regions, and that goes into a SAGE

          8    model that I understand was originally

          9    developed for actually greenhouse gas

         10    emissions and is now being expanded for oil

         11    production.

         12              And so one of the questions that we

         13    spent a lot of time on was, how is reserve

         14    expansion and technology incorporated into

         15    the oil production model.  Because the main

         16    question -- the main source of concern there

         17    was the USGS uses -- its assumption that the

         18    technologies that are used and are available

         19    in the U.S. and North Sea are what are going

         20    to be applicable and available in the rest of

         21    the world, and that's a very powerful

         22    assumption, and if that's an assumption that



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           165


          1    EIA is happy to buy into, that must be

          2    clearly and explicitly stated, because you

          3    may have users of the model who may not buy

          4    into that assumption.

          5              So this is sort of the question of

          6    what are you assuming, make it explicit, and

          7    how sensitive are your modeling results to

          8    these assumptions; it's something that we

          9    would definitely like to have a lot of detail

         10    on.

         11              A question that came up again was

         12    to sort of -- may be more difficult for the

         13    EIA to explicitly answer, but something to at

         14    least keep in mind is that the USGS

         15    estimates, if we look at the historic world

         16    assessment model, the probability curves for

         17    undiscovered resources have shifted

         18    rightwards, especially in the last

         19    assessment, not so much in the previous

         20    assessment.  And the question is, if they're

         21    going to use the SAGE model to project 30

         22    years onwards, as it's being done today -- or



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           166


          1    maybe up to 2100, another figure that was

          2    thrown up -- what are we going to assume

          3    about those curves?  Are we going to assume

          4    they're static, are we going to try and

          5    project what the USGS is going to do?  That's

          6    a question -- the longer the time frame gets,

          7    the more important that question becomes.

          8              Another question that we wanted to

          9    really get more detail on was, based on

         10    non-oil prices -- because that's what we

         11    understood that the model is taking as an

         12    input -- how does the model convert -- or how

         13    do we model the conversion of resources,

         14    undiscovered resources, into reserves, proven

         15    reserves?  How does that happen?  That detail

         16    was not available as yet, and it's, I think,

         17    one that is going to require a lot of

         18    discussion and the committee will have a lot

         19    of interest in that detail.

         20              Likewise, we talk about developing

         21    Hubbard curves, which are just sort of oil

         22    production curves for each reason, rise or



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           167


          1    each peak or decline, and the location of

          2    those curves is sensitive to price ------

          3    options.

          4              And so the question was, in the

          5    future, as prices and technologies change, we

          6    may have expected prices and expected

          7    technology curves.  How are these curves

          8    going to shift, what is the mechanism built

          9    into the model that shifts these curves,

         10    because that's an important detail, an

         11    important assumption that people will want to

         12    know about.  And then in the end, we went

         13    into another discussion which maybe should

         14    have actually come in the beginning, which

         15    is, how -- the basic question was how complex

         16    does the model need to be.  Because as we

         17    understand this, there is a huge number of

         18    equations and a large number of variables,

         19    but is that driven more because this is a

         20    linear programming model?

         21              We could actually collapse the

         22    whole thing into a much smaller model by



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           168


          1    removing nonlinear curves and maybe growth

          2    rates in those curves, which is just one

          3    parameter that can carry the entire curve.

          4              A more fundamental question is what

          5    is driving this model.  Is it because we had

          6    it and we want to use it?  Is it because

          7    we're trying to answer a certain kind of

          8    question?  That's something which we really

          9    got into in the end, but I think it's an

         10    important question to ask.  I think I'm going

         11    to leave it there.  As I said, it was a very

         12    lively discussion.  I've just given you a

         13    flavor of the questions that we asked.  I

         14    hope I've picked up the most important ones,

         15    but if other people who were in the room

         16    found something else more important or

         17    equally important, please add to my --

         18              DR. FEDER:  I have a question.

         19    What do actually the consumers of this model

         20    get?  Do they get the actual model or outputs

         21    from this model?

         22              MS. KHANNA:  We didn't ask that



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           169


          1    question, so we don't know the answer.  I'm

          2    assuming the outputs from the model, but --

          3              MR. STAUB:  Most of the output of

          4    the model is used for producing the

          5    International Energy Outlook publication.

          6    And in theory -- we provide, like, for

          7    example, capacity data on electricity

          8    generation for the world to various consumers

          9    and things like that.

         10              DR. FEDER:  What I meant is, if a

         11    user wanted to do sensitivity analysis, do

         12    parameter sweeps, or anything to use the

         13    actual model, could they do it, or --

         14              MR. STAUB:  If they have two years,

         15    they could probably learn it and manipulate

         16    it and things like that.  It's --

         17              DR. FEDER:  Okay.

         18              MS. BARDEN:  Well, and also, a lot

         19    of what we were talking about in the meeting

         20    was a new supply representation for just

         21    world oil, which would --

         22              MS. KHANNA:  That's right --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           170


          1              MS. BARDEN:  -- still be separate

          2    from SAGE.  So in a sense, we within EIA

          3    would actually be consumers for those two

          4    models, and we would be able to change

          5    assumptions and get different results out

          6    that we could then feed into SAGE, and the

          7    general public would be the consumers of the

          8    SAGE results of those different cases.

          9              MS. KHANNA:  Yeah, that's a really

         10    good point.  Thank you for mentioning that.

         11    We didn't talk about SAGE, really, we talked

         12    about the oil production model because we got

         13    all so excited about that.

         14              DR. HENGARTNER:  Any more questions

         15    and comments since it was such a lively

         16    discussion?  Thank you very much.  We're

         17    getting along.

         18              At this stage, I'd like to invite

         19    additional comments from the committee if

         20    anyone else wants to add to those

         21    discussions.  And if not, I'd like to invite

         22    the public for general comments.  Okay --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           171


          1              MR. HILL:  Will you take minor

          2    comments on the -- oil, you're looking at

          3    country-level data or do you actually have

          4    company level?  What are the units of

          5    analysis of the larger oil model?

          6              MS. BARDEN:  It's based on USGS

          7    assessment to a large extent, so their

          8    granularity would be the lowest we could go

          9    down.  The results of the model of this oil

         10    supply model would be at the country level;

         11    SAGE is at a 16-region world level.

         12              MR. HILL:  I think some of the

         13    political discussions might be at the level

         14    of the company.  I guess you won't pick that

         15    up.  You won't pick that up.

         16              MS. BARDEN:  No.  I mean,

         17    especially once we roll things up into the 16

         18    regions within SAGE --

         19              MR. HILL:  It's all aggregated.

         20              MS. BARDEN:  Yeah.

         21              MR. CLEVELAND:  It's built from the

         22    ground up with the USGS data, which is



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           172


          1    geologic province-level data, so you probably

          2    put political boundaries over that and

          3    ascribe their numbers to particular nations,

          4    I'm assuming.  So we're stuck with the

          5    geologists' data.

          6              DR. HENGARTNER:  I just want to

          7    remind the committee -- I will inform you

          8    that many years ago the statisticians said,

          9    bring us the problems early so we can have an

         10    input.  Right?  And we keep repeating that

         11    mantra, we want to see the data early on.

         12              Well, this new oil production model

         13    is exactly such an attempt, and our input to

         14    them is probably going to have a big impact

         15    on how things are going to evolve in the

         16    future.  At least that's the thought.  So,

         17    again, want to leave it open that if you have

         18    ideas and things that come to mind about this

         19    particular model and endeavor, do not

         20    hesitate -- now is the time to speak up

         21    because now is the time that we actually can

         22    bring input and expertise that, as a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           173


          1    committee, EIA will value.

          2              Yes, Neha?

          3                   (Laughter)

          4              DR. HENGARTNER:  Oh, oh, you opened

          5    Pandora's box.

          6              MS. KHANNA:  Yeah, you put me -- I

          7    want to take a yard, of course.  Well, I

          8    think my personal reaction to that is that's

          9    fantastic, but we need a lot more detail,

         10    which I understood the constraints today, it

         11    was not possible to provide us those details.

         12    But we would love to have more details so we

         13    can chew on them and then spit them out in

         14    different directions and reach some kind of

         15    consensus about where the model should go.

         16              DR. HENGARTNER:  So in that case,

         17    how do you want those details?  Do you want

         18    them to e-mail them to you?

         19              MS. KHANNA:  I'm happy to look at

         20    them.  I will not speak for other people.

         21              DR. HENGARTNER:  Okay, you've

         22    written that down, right?



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           174


          1                   (Laughter)

          2              MR. SINGPURWALLA:  Like more

          3    collaborative, instead of just seeing a quick

          4    pop twice a year.  Would people be interested

          5    -- and I don't even know if this is possible

          6    -- in getting back and forth with people who

          7    would be focusing on the model throughout the

          8    year?  Or is that putting more on your plate

          9    than you want?

         10              MS. KHANNA:  Well, yes and no.  Of

         11    course, it's putting more on my plate, or on

         12    anyone's plate on the committee.  I think

         13    what was great about the break-out session

         14    was that different people in the room had

         15    different strengths and different expertise,

         16    and, as a group, it worked really well, I

         17    think.  And I think that was exciting about

         18    the session.

         19              So I think it may be that if we

         20    know, for example, that we're going to

         21    develop this model, first of all, we should

         22    probably have it more frequently on the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           175


          1    agenda.  And secondly, provide a lot more

          2    detail well in advance, not just a week

          3    before -- and I know we've talked about this

          4    and there are reasons why this happens and

          5    good reasons.  But I think something like

          6    this, which is going to be so important,

          7    maybe if we got three weeks before or several

          8    days before so we could really have a time to

          9    go into it and think about it -- maybe even

         10    e-mail each other about it or something --

         11    that would be great.

         12              DR. NEERCHAL:  I think today, the

         13    break-out session, it seems to me the details

         14    needed were on the SAGE model, to some

         15    extent.  I think we needed to know what was

         16    in the SAGE model, and so some of it was put

         17    into the rhetoric that is kind of a

         18    continuing work based on SAGE model, and it's

         19    kind of a -- they need to know who the

         20    audience is going to be, whether they need

         21    the discussion on SAGE model or not.  I don't

         22    know how you're going to coordinate that back



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           176


          1    and forth, just to figure out -- I think it

          2    is fair to say we need the information more

          3    than one week ahead of time, but at the same

          4    time, you send it to me two months before the

          5    meeting starts, I'm not even going to open

          6    that attachment sometimes.  So I think we are

          7    equally guilty of that kind of --

          8              DR. HENGARTNER:  Yes?

          9              MR. CLEVELAND:  I agree

         10    wholeheartedly, and I think this is a little

         11    bit different issue.  You're talking about

         12    modeling world oil production, which is the

         13    hottest energy issue on the planet right now,

         14    and so clearly you're going to have to be

         15    extra careful about how you document it and

         16    so on.  And it would be useful -- the devil

         17    is in the details, and it also -- our

         18    interaction with you is -- you have a

         19    contractor here who's doing a lot of the

         20    work, the EEA, and so if you really want to

         21    get into some of the important technical

         22    details, you may want to bring in someone



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           177


          1    from that shop to be involved in these

          2    discussions because -- I mean, Phillip and

          3    Justine did a good job with what they had,

          4    but some of the shapes of these functions --

          5    a lot of the technical details are only known

          6    by the people who are really working on them.

          7              So it does speak to the issue of --

          8    on this particular model, I think you want to

          9    bend over backwards to make sure that you're

         10    getting all the advice that you possibly can

         11    be brought to bear on it because there are

         12    going to be a lot of people looking at what

         13    the EIA has to say about the peak in world

         14    oil production.

         15              DR. HENGARTNER:  Moshe?

         16              DR. FEDER:  Just on the advance

         17    receiving of information, Nagaraj, I think if

         18    I -- if Nick or -- when you become our

         19    chairperson, send me the information.  I'm

         20    going to have to discuss it; I can assure you

         21    I will open it.  It's like when you get the

         22    reminder from the general, you actually -- so



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           178


          1    I think -- I actually -- in the past, I

          2    really requested it.  It's good to have the

          3    information in advance, especially if I'm

          4    supposed to be discussing it, or even be in

          5    the break-out session for that.  So a

          6    selective provision of this material will be

          7    useful to those who actually have to say

          8    something meaningful about it.

          9              But if this is maybe feasible,

         10    maybe it could be posted on the Web page and

         11    all you need to do is let her know when it's

         12    available because I know how people are

         13    working right up to the last minute, revising

         14    the material.  If you put it on the Web page,

         15    I would look at it, not if I'm not in that

         16    break-out session, frankly, but if I am, I

         17    will, just because maybe I'm slow and I need

         18    the time to think about it.

         19              DR. HENGARTNER:  Barbara?

         20              MS. FORSYTH:  I wanted to ask a

         21    question about the process for the world

         22    model development.  Is there a technical



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           179


          1    working group that's working with you so

          2    that, in essence, we'd be working somewhat in

          3    a parallel technical advisory capacity?

          4    Because sometimes a technical working group

          5    can serve a more focused function and then we

          6    meet probably less frequently.

          7              SPEAKER:  That's a good idea.

          8              DR. FEDER:  That's a good idea.

          9              MS. KHANNA:  You have a smaller

         10    group of people.

         11              MS. FORSYTH:  Right.

         12              DR. HENGARTNER:  Mark?

         13              DR. BURTON:  That actually goes to

         14    the point that I wanted to make in that we

         15    certainly encourage being involved early in

         16    the process.  This is one where if we've not

         17    been, it probably would have served well to

         18    have been involved even earlier.  I think

         19    that Tom made some good points in the

         20    break-out regarding sort of the conceptual

         21    framework of the model and what some of the

         22    alternatives might have been at an earlier



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           180


          1    point.  So, again, I'm not familiar with the

          2    history in particular, but this may have been

          3    a place where the committee could have added

          4    some value from sort of day one.

          5              MR. RUTHERFORD:  Being a model

          6    builder myself, though, I just -- I'm always

          7    concerned about model-building by

          8    committee --

          9                   (Laughter)

         10              MR. RUTHERFORD:  -- so I tend to be

         11    a little bit skeptical about -- I think today

         12    was good.  We got to complain about this and

         13    that.  I would also add, in terms of getting

         14    your two bits in early, most of what I said

         15    today, I said in my review of NEMS in 1993,

         16    so I think my views are known.  And the

         17    general, I just think it's kind of good.  But

         18    I agree wholeheartedly with the idea of

         19    bringing in the people who are doing the

         20    modeling so we can basically have an up-front

         21    interchange.

         22              So I guess I sort of disagree with



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           181


          1    the rest of the committee.  I think that

          2    either I'm involved in making a model, or I'm

          3    just giving my best reaction.  I think that

          4    studying in detail what the modeler is doing

          5    and then writing out -- I mean, really, to be

          6    honest, to be completely frank, taking the

          7    time to write out the English words that

          8    describe what the problem are in the model is

          9    much harder for me than just writing out the

         10    GAMS code for how the model should be

         11    structured.  That's my own take on it; it's

         12    just generally -- I'd rather just write the

         13    math down.  But if I'm not going to be

         14    involved in doing that, I think that the

         15    format was fine, it's just I think that the

         16    idea of bringing the people actually doing

         17    the work so we could ask specific questions

         18    would be helpful.  So I tend to be a little

         19    bit less -- it's not that I don't want to be

         20    involved in helping things, but I'm very kind

         21    of -- having been on the receiving end of

         22    advice from committees about models, I tend



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           182


          1    to be reluctant to be on the --

          2              DR. FEDER:  Giving end.

          3              MR. RUTHERFORD:  -- delivering end,

          4    right.

          5              MR. HELKIE:  I think the operable

          6    thing what Tom said was 1993, okay, and this

          7    is 2006.  People change.  I think we're about

          8    ready to get to a more comprehensive re-look.

          9    And I think the English language reaction,

         10    especially of people who have experience

         11    building models, would be very useful early.

         12    And I think if we got to pay you to do it,

         13    that's fine, but I think it's still

         14    worthwhile to get a broad view of people who

         15    have experience building models who care

         16    about the output of the models and can

         17    provide their perspective the policy

         18    community.

         19              I just think that's useful, and as

         20    you say, it's more useful than going into the

         21    guts and saying, well, gee, that primer ought

         22    to be point five instead of point six or



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           183


          1    something like that.  Because that's not as

          2    useful as getting in in the beginning and

          3    saying, this is where it ought to go, right,

          4    and these are the reasons why.  Okay, so I

          5    guess I disagree with your conclusion.

          6              DR. HENGARTNER:  To wrap it up, I

          7    think it's fair -- oh, oh, oh, no.

          8              MR. CLEVELAND:  No, I said my

          9    piece, thanks.  Didn't I?

         10              DR. HENGARTNER:  Yes, you did.

         11              MR. CLEVELAND:  I did.

         12              DR. HENGARTNER:  You did.  I would

         13    suggest that part of the compromise is to

         14    definitely talk about this modeling some more

         15    in six months.  If people are interested in

         16    being involved in giving feedback, they

         17    should be.  I mean, Neha offered some of her

         18    time, and may -- I'm putting her on the spot,

         19    which is unfair --

         20              DR. FEDER:  But you'll do it

         21    anyway.

         22                   (Laughter)



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           184


          1              DR. HENGARTNER:  But I think

          2    there's room for polite debate.  The

          3    formation of a committee is something that is

          4    outside, I think, the charter of the ASA

          5    Committee of Energy Statistics, although it

          6    is an idea that Nancy definitely can

          7    consider.

          8              So I'd like to invite, again, the

          9    general public to make comments.  And before

         10    we all leave for lunch, I actually realize

         11    that we are privileged to have among us Bill

         12    Helkie, who is now technical advisor to the

         13    EIA and actually comes from the office of

         14    Alan Greenspan.  So welcome.

         15              MR. HELKIE:  Thank you.

         16                   (Applause)

         17              DR. HENGARTNER:  Okay, so lunch is

         18    downstairs, and we will reconvene here after

         19    lunch at 1:15.  Thank you very much.

         20                   (Whereupon, at 12:15 p.m., a luncheon

         21                   recess was taken.)

         22                       *  *  *  *  *



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           185


          1             A F T E R N O O N   S E S S I O N

          2                                             (1:15 p.m.)

          3              DR. HENGARTNER:  Okay, welcome back

          4    to the afternoon session.  It's my great

          5    pleasure to introduce Jake Bournazian, who's

          6    going to talk about measuring perception of

          7    applying alternative disclosure limitation

          8    methods.

          9              MR. BOURNAZIAN:  Well, good

         10    afternoon.  Today's presentation is about a

         11    new research proposal that's going to involve

         12    studying the perceptions of our users and

         13    survey respondents on some unique issues

         14    relating to data confidentiality.

         15              There have been prior studies done

         16    on user perceptions in the past, but what's

         17    special about this research proposal is

         18    there's never been a study on the perceptions

         19    of applying alternative disclosure

         20    limitations -- methods for protecting

         21    confidential data.

         22              When we're talking about studying



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           186


          1    perceptions on this topic, we really need to

          2    study two separate groups.  Not only are we

          3    going to be studying the perceptions of two

          4    distinct groups, but we also want to get

          5    different information from these two groups.

          6    Specifically, here, we're data users.  We

          7    want to know what they're thinking about and

          8    how they feel on the data utility of viewing

          9    tables that are filled with either all

         10    adjusted values or some cell values that are

         11    adjusted in a table.  What's the data utility

         12    of that kind of product to them?

         13              In the second group, we want to be

         14    looking at our survey respondents, and we

         15    want to know what they're thinking about when

         16    they're informed that an agency is

         17    considering applying a different disclosure

         18    limitation method, such as data swapping.

         19    Will they become less diligent and accurate

         20    in filling out their survey forms if they

         21    know or are aware of these alternative

         22    methods that are going to be applied?



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           187


          1              In this morning's session, I heard

          2    a comment made that it's not good to base

          3    decisions on ad hoc information.  And other

          4    speakers mentioned that we should be knowing

          5    a little bit on how our data users are

          6    perceiving and using this data.  Well,

          7    everyone in this room might have some

          8    subjective opinion about these issues here up

          9    on the screen, but for the agency to act in

         10    its best interest in fulfilling its mission,

         11    we need objective, accurate information which

         12    relate to these perceptions.  Next slide.

         13              So let's put the study in the

         14    proper context of what we're talking about,

         15    because this is very technical area.  New

         16    disclosure methods are available.  I grouped

         17    them into two categories for today's

         18    discussion:  The first category is going to

         19    adjust the data prior to doing any kind of

         20    tabulation.  You may have two branches of

         21    these -- one is data swapping and the second

         22    noise addiction.  Data swapping's been around



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           188


          1    for decades, very common, prevalent use to

          2    protect household data, frequency data, but

          3    recently it's being used to protect tabular

          4    data.  In the Census 2000 information

          5    products, they used data swapping to protect

          6    that data for some of their tables.  Also the

          7    Census Bureau pioneered noise addition in

          8    1998 on a research and development survey.

          9    National Agricultural Statistics Service

         10    implemented noise addition on the

         11    Agricultural Resource Management Survey last

         12    year.  They unveiled an online query system

         13    where users go to their website and customize

         14    and design their own queries, get their

         15    tables, and the confidentiality of the data

         16    is protected through noise addition.

         17              There's another category here,

         18    solves the problem from a different approach.

         19    We're going to adjust the data after the

         20    aggregate estimates are calculated.  So we're

         21    going to adjust after tabulation.  We're

         22    referencing here controlled tabular



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           189


          1    adjustment; this is a broad category which

          2    references a family of methodologies,

          3    relating all to the same kind of approach.

          4    Next slide.

          5              So let me explain very briefly what

          6    these methodologies do by operating on a

          7    single example.  We're going to take the same

          8    problem, we're going to solve it three

          9    different ways using these methodologies, and

         10    you can see how they work.  All right, so the

         11    problem here is we have a table with a

         12    sensitive cell needing protection.  That's

         13    what triggers some kind of action from the

         14    agency.  Next slide.

         15              Now, the most common method across

         16    all federal agencies is cell suppression, and

         17    it's certainly the method used here at EIA.

         18    When you're talking about applying cell

         19    suppression for protecting confidential data,

         20    cell suppression, in my opinion, is a one-way

         21    street.  Cell suppression says, from the

         22    agency's standpoint, we're going to meet our



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           190


          1    needs -- which is to protect the

          2    confidentiality of those company-level

          3    responses -- but how about the user that's

          4    interested in one of those suppressed cells?

          5    Well, you're out of luck because we have to

          6    take care of our need as an agency.  And

          7    there's a total loss to the user.  Well,

          8    that's the way life has been for the last two

          9    decades.  What happens, though, is that

         10    through time, industry changes, these tables

         11    can accumulate pockmarks.  Some tables might

         12    have 20 to 40 percent suppression, but that

         13    suppression could grow, sometimes frustrating

         14    the utility of the table altogether or almost

         15    destroying a lot of the value.

         16              So you may be thinking, wow, if

         17    you've got that much suppression, redesign

         18    the table.  Well, it's not that simple when

         19    you have limited resources.  Limited

         20    resources for a program office is going to

         21    put this on the lowest of low priorities

         22    because if you have other production



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           191


          1    activities going on, redesigning a table is

          2    not right up on there to spend your

          3    resources.  So what happens, like a house

          4    that has paint flaking off of it and the need

          5    of repair, these EIA publications can carry

          6    four of these tables over the years, which

          7    carry a lot of pockmarks from their cell

          8    suppression.  But we've got some solutions.

          9    Mathematics and IT science to the rescue!

         10    Next slide.

         11              Okay, let's look at math and

         12    science, part one.  I told you we can try and

         13    protect that sensitive cell by using a method

         14    called a controlled tabular adjustment.  The

         15    approach here is we're going to go after that

         16    sensitive cell and adjust its value a

         17    sufficient distance away from its actual

         18    value.  And you can specify a percentage

         19    protection range you want to give that cell.

         20    Now, once you adjust that sensitive cell, you

         21    have to go back and adjust some of the

         22    non-sensitive cells in that table.  And in



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           192


          1    this example, it's three rows, three columns,

          2    I have marginals -- totals down at the bottom

          3    of each column and end of each row.  So it's

          4    a very simple table.  Now, I adjusted actual

          5    values here in my example.  I could have

          6    taken the reciprocal value of those cell

          7    values or I could have taken the logged

          8    value, and I could have said, well, I don't

          9    want any of my totals to change, just adjust

         10    the interior cells.  Or in this case, in my

         11    example, I allowed the marginals to be

         12    adjusted, thereby saving some interior cells

         13    from needing to be adjusted at all.  Once

         14    again, there's a lot of variations, but it's

         15    the same approach when you're using one of

         16    these variations on controlled tabular

         17    adjustment.  Next slide.

         18              All right, so math and science,

         19    part two.  We're going to solve this problem

         20    from a completely different approach.  We

         21    want to now modify the company-level

         22    responses prior to doing any tabulation.  And



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           193


          1    we're going to try and protect that sensitive

          2    cell this way:  With noise addition, you can

          3    specify, once again, the percentage

          4    protection you want to give those sensitive

          5    cells.  You can also target the noise and add

          6    more noise to the sensitive cells and avoid

          7    adding any noise to the non-sensitive cells.

          8    You have to take a real specific approach,

          9    though, in using this methodology.  You're

         10    going to first want to randomly assign the

         11    direction of the noise, then come back in and

         12    randomly assign the noise factors.  You can

         13    pick a number of types of distribution, but

         14    whatever distribution you start off with in

         15    adjusting upward, you must take the exact

         16    opposite distribution in adjusting downward.

         17              When you assign that noise, what

         18    you are searching and going to obtain is a

         19    distribution of the noise factors that's

         20    symmetrical around number one because

         21    ultimately you do not want to impact your

         22    aggregate estimates to the lowest bar, you



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           194


          1    want to save any adjustment on that.  So in

          2    this example, we've drilled down to the

          3    company-level values, and now we're applying

          4    our noise factors.  Now, another point here

          5    for this methodology is once a noise factor's

          6    assigned to a company, that company gets that

          7    noise factor no matter where its data appears

          8    in a table, in a sensitive cell or

          9    non-sensitive cell.  So here we adjusted the

         10    values and came up with a new cell total to

         11    protect the actual true value, which is

         12    deemed sensitive from whatever primary rule

         13    we're using.  Next slide.

         14              Okay, and then also, like I pointed

         15    out, every company in that table's getting a

         16    noise factor, and so that means wherever that

         17    company's data appears in the table, it's

         18    getting adjusted the same way.  Here,

         19    non-sensitive cells are also adjusted, but to

         20    a lesser extent; less noise is added to those

         21    cell values.  Next slide.

         22              Okay, let's go back and solve the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           195


          1    same exact problem a third way.  We're going

          2    to use data swapping.  Now, the way you apply

          3    data swapping for protecting tabular data is

          4    pretty much the same way you did it when you

          5    were protecting frequency tables in

          6    public-use files; that is, you go after your

          7    sensitive cells that your primary rule has

          8    identified, try to identify what's going on

          9    in those cells, what's unique, and you're

         10    going to design a data key structure to

         11    target data swapping on your sensitive cells

         12    to the best of your ability.  Now, I used a

         13    very simple key structure, three variables.

         14    You could go on for a lot longer because I

         15    see data keys five variables or more.  But

         16    the point here is we're going to go after a

         17    sensitive cell and do some swapping.  You

         18    only need to find a proper swapping

         19    candidate, and for me, in this case, sales

         20    type, product type, seller type.  You could

         21    have variations on seller type -- could be a

         22    certain type of power producer or it could be



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           196


          1    a certain size of supplier.  You might have a

          2    lot of medium-sized suppliers; we do the swap

          3    and recalculate and I do adjusted value.  You

          4    want to keep your swap rates real low when

          5    you apply this methodology and don't want to

          6    go over five percent because this methodology

          7    impacts your cell values a lot more than the

          8    other two methodologies do.  And of course,

          9    you need some kind of swapping variable when

         10    you apply this methodology.

         11              So let me just summarize this area

         12    that we're going to want to study perceptions

         13    on.  Yes, cell suppression is the most common

         14    method; these new methodologies have been

         15    developed specifically to meet a growing user

         16    demand for more data.  One point to note is

         17    noise addition and controlled tabular

         18    adjustment can adjust your non-sensitive

         19    cells, and it may not cause a statistically

         20    significant change from the actual estimates.

         21    But in more simplified terms, data swapping

         22    is more like using a rubber mallet to impact



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           197


          1    a cell, and controlled tabular adjustment and

          2    noise addition is like using a tack hammer to

          3    impact a sensitive cell.  But the benefits

          4    are clear.  You're going to get a fully

          5    populated table, more usefulness.  And I have

          6    no problem footnoting a table; if you

          7    overshot your mark in cell value changes by

          8    more than 10 percent, you could put a flag on

          9    those cells in the table and tell the user

         10    these cells are flagged because they're not

         11    as accurate as other cells in a table.  An

         12    intruder can't take that information and

         13    unlock your table, and a data user would

         14    really find that kind of performance measure

         15    useful in working with the data.  Of course,

         16    the cost -- got to watch it.  These

         17    methodologies need to be applied in an art

         18    form as much as a science.  And of course,

         19    it's subject to criticism that you can really

         20    screw up a table in applying these

         21    methodologies.  To that criticism, I always

         22    reply, well, the delete key on your personal



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           198


          1    computer can really devastate your hard drive

          2    if used inappropriately, but when that key is

          3    used appropriately, it's a very important

          4    tool on your keyboard.  Same with these

          5    methodologies; you have to be very careful in

          6    applying them.

          7              Okay, so let's look at the survey

          8    literature in this area because the Census

          9    Bureau has been a leader in doing studies on

         10    user perceptions, and they did a study six

         11    years ago; it was the survey of business

         12    perceptions of confidentiality.  Here they

         13    were studying what kinds of information do

         14    businesses deem confidential.  They also

         15    wanted to explore how they felt about data

         16    sharing -- this was the closest study I could

         17    find that parallels what we're talking about

         18    this afternoon.  Well, they built it as a

         19    nice frame file from purchasing Dun &

         20    Bradstreet, 11.3 million businesses, drew a

         21    sample of 5,000 companies, set up four

         22    strata, based on employment -- 0 to 50



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           199


          1    employees, 50 to 250, 250 to 500 and

          2    companies with over 500 employees -- but each

          3    stratum had 1,750 companies in it.  They also

          4    looked at companies that were multi-location

          5    and single location, split there.  But

          6    anyway, they ran the survey, did it by mail,

          7    came up with a pretty low response rate,

          8    under 11 percent.  Three reasons they cited

          9    in their report:  One, it was a voluntary

         10    survey; two, it was done by a contractor in a

         11    contractor's name, not in the name of the

         12    Census Bureau; three, they conducted the

         13    survey over the holidays.  So here, probably

         14    a good sample design, probably a good survey

         15    questionnaire they used, too.  But timing

         16    really just wiped out a lot of hard effort

         17    there.  Timing can be everything sometimes in

         18    life, and this showed it.

         19              But anyway, there was some useful

         20    information coming from this survey.  This

         21    was present in each one of the strata, so we

         22    can rely on that something is going on in the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           200


          1    minds of the businesses.  They were split on

          2    whether the government could protect

          3    confidential data; the greater the trust and

          4    the belief in competency of the statistical

          5    agency, the more willingness to provide data.

          6    And I pulled these out because I felt it was

          7    noteworthy.

          8              I think EIA has good credibility

          9    across all its energy markets, and we could

         10    probably work from this if we were to conduct

         11    such a study.  Also relating to data sharing,

         12    they repeated this portion of the study and

         13    were finding that the respondents were

         14    filtering.  Businesses, when they're asked

         15    about how they felt about data sharing were

         16    thinking, well, does this reduce my reporting

         17    burden?  Everything was filtered in that

         18    process and in those terms.  And we may want

         19    to consider that also, because we're talking

         20    about alternative methods.  So if a company

         21    hears, well, EIA's considering data swapping,

         22    does that mean if I don't report in April



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           201


          1    2006, just swap my data from January, and

          2    we'll call it even.  Then data swapping may

          3    sound pretty good to that respondent.  So

          4    we'll have to address maybe reporting burden.

          5    Next.

          6              Okay, data user studies:  A lot

          7    done here by the Census Bureau.  I didn't

          8    spend a lot of time on this, because data

          9    users -- or households -- are a lot different

         10    than our EIA data users, but we might be able

         11    to learn something here.  One thing that came

         12    through was identical to the business

         13    perceptions:  Households equally had a

         14    distrust, and they were split on the ability

         15    of the agency to protect their data.  They

         16    were questioning that confidentiality pledge.

         17    Now, what also was interesting, the Census

         18    Bureau relied on two modes of data collection

         19    throughout 20 years:  Focus groups and mail.

         20    They did some cognitive research, but that

         21    was mainly to design a good questionnaire,

         22    although they tried to draw some conclusions



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           202


          1    from that cognitive research, but focus

          2    groups and mail are their primary modes.

          3    Next slide.

          4              Okay, so specifically, to the study

          5    we're talking about here at EIA, we have some

          6    additional considerations.  One, we're

          7    talking about disclosure limitation

          8    methodologies, and these different methods

          9    impact the data differently -- I tried to

         10    show you that with a generic example -- and

         11    also these methods are more appropriate,

         12    depending on the type of data.  You don't

         13    want to pick one method and say, well,

         14    that'll work on all kinds of data, you want

         15    to taper it -- what's appropriate.

         16              Very important point here:  EIA has

         17    a very diverse user community.  We've got

         18    government agencies, a wide range of federal

         19    agencies, state agencies, local, municipal

         20    governments, we've got analysts -- and I can

         21    carve that category up between consulting

         22    firms, financial analysts, policy analysts,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           203


          1    academia, researchers -- media -- I can break

          2    media up into TV, radio, newspapers,

          3    magazines.  So a lot of variety across our

          4    user community.  Level of statistical

          5    knowledge is limited to the folks we want to

          6    reach and talk to, and I'm presenting here to

          7    a roomful of sophisticated statisticians on

          8    some methods, but that's not going to be the

          9    same when you're talking to companies and

         10    data users.  Also, some negative attitudes

         11    exist.  I think for that reason, the Census

         12    Bureau on a longitudinal basis is always

         13    surveying user perceptions.  The pendulum

         14    between privacy and willingness to accept

         15    some intrusion from the government and

         16    reporting information is always swinging back

         17    and forth since they passed the Privacy Act

         18    in '74.  It goes back and forth -- I mean,

         19    it's where you want to do these studies -- I

         20    heard somebody say there's no systematic way

         21    of communicating with our users and then

         22    measuring.  And this is one way that the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           204


          1    study, it gets conducted, we want to repeat

          2    it or at least carry it on longitudinally to

          3    pick up that swing in the pendulums.

          4              We saw it swinging the other

          5    direction, away from privacy, after 9/11,

          6    then people hear about the President wanting

          7    to allow wiretaps without a court order, and

          8    you see that pendulum moving back.  So we've

          9    got to recognize doing this study, what's

         10    your atmosphere and climate in which you're

         11    conducting it.  Next slide.

         12              All right, I'd like to wrap up with

         13    this slide.  I put down on this slide some of

         14    the big issues that were coming to my mind

         15    when I was writing this paper.  This first

         16    issue, though, when I was thinking about it,

         17    reminded me of what my mother used to tell me

         18    when I was growing up.  She'd say, Jake,

         19    sometimes you can't have your cake and eat it

         20    too.  That used to really annoy me, but I

         21    thought of the same thing when, on this first

         22    issue, who should be the target respondent.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           205


          1    The reason why is because that contact person

          2    opens their mail.  When they get mail from

          3    EIA, they open it -- oh, survey -- they fill

          4    it out -- I mean, that's why they get paid,

          5    that's their job.  They even know the data --

          6    wouldn't that be nice.  Well, the Census

          7    Bureau always picked the authority figure

          8    with the company, and that's a very good

          9    point, too, because why should I have the

         10    comptroller of a company filling out a survey

         11    form when I really want the president's

         12    perception on this issue?  Well, can EIA have

         13    their cake and eat it too?  Well, I thought,

         14    if it's a mail survey, maybe I'll mail it to

         15    the contact person and tell him to give it to

         16    his boss.  Or mail to the authority figure

         17    and say ------ next year, so-and-so's your

         18    contact person, best contact person in 30

         19    years, why don't you get some feedback from

         20    them in filling this out?  Somehow maybe we

         21    can share ideas, or we may have to pick one.

         22    I don't know.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           206


          1              Next issue, should all survey

          2    respondents be included in the sample?  If

          3    Southern Minnesota Power and Electric is in

          4    that survey on perceptions, do I need

          5    Northern Minnesota Power and Electric?

          6    Maybe.  Maybe people in Minnesota think

          7    differently in the north than they do from

          8    the south, but we have to address that in

          9    picking off our survey respondents to prove

         10    we're going to participate.

         11              How about selecting those data

         12    users?  I heard our administrator throw out a

         13    statistic this morning; he said in February

         14    2006, we had 300,000 users on our website --

         15    and as I mentioned before, quite a diverse

         16    user community.  Do we want just Internet

         17    users?  Well, we have a lot of other users,

         18    although the Internet's our primary mode of

         19    disseminating data.  How about that mode of

         20    data collection?  Okay, Census Bureau relies

         21    on focus groups and mail, but we even have a

         22    session on Internet data collection and we



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           207


          1    have other modes available today.  Maybe they

          2    could be used, or a combination of these

          3    modes, but maybe we can talk about that.

          4    Also, like I said, this is a technical area.

          5    I used a generic example to explain these

          6    methods pretty quickly for you -- I used text

          7    in my paper with brief examples, but

          8    customized examples always work better for

          9    me.  When I'm trying to understand something,

         10    what works better?  But if you get too far

         11    down the line on a customized example, aren't

         12    you giving a road map to a data intruder?  I

         13    believe most of our survey respondents, at

         14    one point in their life or another, have

         15    picked at the table they report data on and

         16    tried to look at what their competitors were

         17    doing.  But you might have to mask it if it's

         18    customized.  Actually, on this issue, since

         19    you're the first audience I've tried these

         20    generic examples on, this will be our own

         21    focus group on this point.  As long as more

         22    than nine people don't talk on that.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           208


          1              Okay, other issues.  I ran out of

          2    font space here on this slide, so -- but

          3    there's a lot of other issues.  Anything

          4    you're thinking about that we need to be

          5    paying attention to or looking at in

          6    designing this study, we'd love to hear.

          7              Now, we just summarize that this is

          8    a pretty daunting project.  I've talked to

          9    some other colleagues and other federal

         10    agencies about doing this kind of a study,

         11    and they've shied away from it because it's

         12    very challenging, and it's also a topic that

         13    agencies really don't like to go too public

         14    on when they talk about their data

         15    confidentiality practices.  But behold the

         16    turtle:  It only makes progress when it

         17    sticks its neck out, and I think we, working

         18    together, sharing these ideas today, not only

         19    can we design an effective study that will

         20    beat a response rate of 11 percent, I think

         21    we can design a study on perceptions which is

         22    going to give us real useful information so



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           209


          1    that we can understand, gain some insights,

          2    on what our users and survey respondents are

          3    thinking about when EIA considers applying

          4    alternative disclosure limitation methods.

          5              DR. HENGARTNER:  Thank you very

          6    much, Jake.

          7                   (Applause)

          8              DR. HENGARTNER:  I'd like to invite

          9    Nagaraj to start the comment session if you

         10    have some prepared comments for us.

         11              DR. NEEERCHAL:  First of all, I

         12    would like to really thank Jake for a really,

         13    really clear presentation, and the write-up

         14    is also very clear, and I really enjoyed it.

         15    It's particularly important to me because I'm

         16    brand new to this subject, so I think that

         17    helped me a lot.  And I think that's why I

         18    think that my comments probably are going to

         19    be more from the user point of view.  I think

         20    -- it's -- I'm not an expert in this area.

         21    This is a situation where there is a real

         22    cop-and-robber type of thing going on.  I



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           210


          1    think as Jake kind of alluded to at the end,

          2    for Bill Winkler, who is an expert in this

          3    record linkage stuff from Census Bureau, he

          4    always says that if there is an algorithm

          5    that masks the data, I have an algorithm that

          6    will unearth it.  I think I heard him say in

          7    one of his presentations.  So I think that is

          8    definitely the case.  Having said that, I

          9    don't think I need to go over the methodology

         10    here.  I think Jake has done a very good job

         11    of presenting it.

         12              I do want to say couple of points.

         13    If I'm a user, I really do not -- perhaps I'm

         14    not even able to understand the different

         15    methods.  So all I really want to know is how

         16    does the data look before you apply the

         17    method and after you apply the method?  I

         18    want you to just give me a few tables to look

         19    at.  And that is one of the things I'll keep

         20    in mind when I'm designing a survey like

         21    that.  Instead of hammering the user with a

         22    lot of algorithms and asking their opinion



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           211


          1    about which is going to be better, I would

          2    probably focus on giving them a few examples

          3    -- hey, this is what it's going to look like

          4    under method one, this is how it's going to

          5    look under method two, and before and after.

          6    So I think that's one approach.  Now, for

          7    example, I think clearly in all the examples

          8    Jake presented, the lowest data point in the

          9    table always stayed as the lowest data point

         10    in the table.  Lowest and highest.  If you

         11    have something like electric power, I think

         12    you always know who is the "highest" --

         13    everybody knows that.  Income data, everybody

         14    knows Bill Gates is on the top of that list.

         15    So you can mask it, you can add noise to it,

         16    if it is the highest -- if you know it is the

         17    population, who that point is.

         18              So I think that the user can

         19    determine, based on the before and after that

         20    there are such features, the ones that matter

         21    to that user the most, whether those features

         22    are still recognizable or not.  This is one



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           212


          1    way to really get the feedback from the user

          2    in terms of what are these important features

          3    that they really want to be masked or not

          4    masked, depending on their point of view.

          5    Having said that, I think that is why, if

          6    you're looking at people who are responding,

          7    I think that it's always going to look a

          8    little different.  They probably don't want

          9    their data to be that organized.  On the

         10    other hand, users of the data, they want full

         11    accuracy.  And the more you mask, the

         12    correlation between the presented data and

         13    the real data is going to go down, and if it

         14    is very low, then what good is that data.  So

         15    I think those are the issues to be concerned,

         16    and I'm very pleased at the way it is set

         17    out.  I think you have done a lot of review

         18    of the literature; I think you're working

         19    with the people from that area, other parts

         20    of the government, and I think that it seems

         21    to me that some of these issues will be --

         22    I'm not the person to advise you on that ----



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           213


          1    you will get good advice from those areas.

          2              One other thing I wanted to mention

          3    -- I think that it probably is not associated

          4    with this area -- is Larry Cox from NCHS.

          5    Recently he has been working on this same

          6    topic, changing -- changing is a layman's way

          7    of saying it -- changing data, but respecting

          8    some column totals and row totals, and

          9    applying some linear programming type of

         10    algorithms to tables and presenting that.

         11              And the only other comment I have

         12    is that when you present this data, you have

         13    to say if it is masked or not.  So already

         14    you're giving away some of your information.

         15    The moment I know that the table is masked,

         16    and I'm trying to use it, I think -- I'm one

         17    of those people -- I just say, let me see how

         18    they mask it.  I think that's the first thing

         19    that's going to come to my mind.  Maybe

         20    everybody is like that, I don't know.  I say,

         21    let me see what might be the algorithm.  If I

         22    know you're going to use one of the three,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           214


          1    then I'll say, let me see, let me guess which

          2    one -- and then of course I cannot discover

          3    the real data because the noise level is

          4    different and you had to give me a whole lot

          5    of parameters, but I still would play with

          6    it.  So other than that, I think it's a very

          7    nice presentation, laid out nicely, and ready

          8    for the committee to jump in.  I think these

          9    are very clear questions.  I think the

         10    committee should jump in and give some

         11    solutions.

         12              DR. HENGARTNER:  Thank you very

         13    much.  I'd like now to move over to Moshe,

         14    who has kindly agreed to step into Johnny

         15    Blair's shoes.

         16              DR. FEDER:  Disclosure limitation

         17    is very important because I find -- I work in

         18    this area, too, although my interest is in

         19    microdata files and not as much in tabulated

         20    data.  And first of all, I want to say that

         21    was an excellent presentation.  I really

         22    enjoyed hearing the issues that you guys



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           215


          1    have.  I find every day new ways of potential

          2    breach of confidentiality.  Recently, I've

          3    seen some ideas that people came up with to

          4    breach confidentiality using even the server

          5    weights in macrodata files, and things like

          6    that, so I think it's very important.  And

          7    one thing that I think is very important to

          8    keep in mind, all it takes is one breach of

          9    confidentiality to do serious damage not just

         10    to the EIA's data collection efforts, but to

         11    any agency's data collection, because once it

         12    appears in the press and people will know

         13    that there's potential for confidential

         14    information to leak out, then that would

         15    cause a big damage to response rates,

         16    accuracy of responses, and so on.  So I think

         17    it's very important.  And perception, like

         18    you said, is very important here.  There's

         19    always a tradeoff between data quality and

         20    protection of confidentiality.  The more you

         21    protect, the less information is there, so

         22    striking a good balance is very important.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           216


          1              I have a couple of questions here

          2    for Jake and to others who work in this.  You

          3    presented two ways of approaching it.  One is

          4    pre-tabulation, by adding either noise or

          5    doing data swapping to the data before the

          6    tables are being extracted -- and actually,

          7    modeling is a similar issue.  If you ------

          8    the aggression model, you can sometimes will

          9    information by looking at the regression

         10    coefficients, especially if some of them are

         11    categorical, and you can use that to infer on

         12    some individuals.  And so in particular, when

         13    people provide multiple products using the

         14    same data, how do you make sure that by using

         15    two tables or two regression models or in

         16    some other way combine information from these

         17    and make -- I'll give you an example that I

         18    know the Australians are working on.  They

         19    have an online system when you submit a

         20    request for a table, and you submit one table

         21    and say, give me this information for

         22    individuals age, let's say, 12 to 17, then



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           217


          1    you do another one, let's say, 16 to 18, and

          2    you cross the two tables and get an age range

          3    which is too narrow by the limitations of any

          4    single table and combining it.  So there are

          5    many challenging issues here, and the first

          6    approach is to treat the data prior to

          7    tabulation avoids that because if the

          8    microdata file are protected, then the tables

          9    in theory should be protected.

         10              The three-way table is another

         11    challenge.  You mentioned data-swapping,

         12    which I think is a common technique, but

         13    wouldn't the two swapping partners know

         14    enough -- if I can recognize my data,

         15    wouldn't I then know something about the

         16    other person or entity?  So I think there are

         17    two, and that's almost unlike any other

         18    statistical activity that we do.  Usually, a

         19    statistician's job is to provide as much

         20    information as you can, period.  Here, not

         21    only are we supposed to suppress information,

         22    but do it in a way that someone will not be



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           218


          1    able to -- I think Nick or Nagaraj mentioned

          2    that they quoted Bill Winkler saying that he

          3    can always come up with a way to break the

          4    code, so I think if you can maybe shed some

          5    light on that.  How do you constantly avoid

          6    new ways to break the code, and what do you

          7    do with data swapping if a person who's

          8    responsible for the data that's part of the

          9    swapping will be able to undo the process by

         10    identifying his establishment's response?

         11              I'll stop here.

         12              DR. HENGARTNER:  Thank you very

         13    much.  I'd like to open the discussion to the

         14    general committee.  We have about 10 minutes

         15    to do so.

         16              I have a question.  What I find

         17    with those data confidentiality issues,

         18    especially from the perception point of view,

         19    is that I'm not convinced that we all

         20    understand how data swapping or how adding a

         21    little bit of noise will ensure the

         22    confidentiality.  How do I know that adding



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           219


          1    noise will make me safe, right?  And maybe it

          2    doesn't.  And if you are going to make a

          3    survey and explain which method you like

          4    best, you ought to do yourself the favor --

          5    and the reader as well, or the person you're

          6    serving -- to explain why it works.  I'm

          7    always tempted to say that, well, you mention

          8    that some of the readers are not that

          9    statistically literate; in fact, you don't

         10    need to be statistically literate or not -- I

         11    mean, I'm not even convinced, in light of

         12    Nagaraj and Moshe's discussion that adding

         13    noise sometimes truly does ensure

         14    confidentiality.  If you have an outlier, he

         15    might remain an outlier.  And so being able

         16    to make that discussion would be interesting

         17    -- or that distinction.  Yeah?

         18              DR. FEDER:  I can tell you that at

         19    least one agency -- and we won't go into --

         20    not one that I'm actually very closely

         21    related to -- all they do is do treatment of

         22    some records and say, since you never know



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           220


          1    which one was treated -- or you don't know

          2    which data were contaminated by noise, you

          3    never know enough information.  That would

          4    not apply the case of the EIA simply because

          5    you have big establishments whose domination

          6    of some cell estimates is big and so on.

          7              But that's a good question.  What

          8    are we trying to do here?  We want to create

          9    enough distortion to the data that you never

         10    know for sure enough about the protected

         11    information, but what is enough?  I know that

         12    Chris Skinner -- and actually, it goes back

         13    to the Bethlehem -- and I forget who the

         14    other author of that paper is -- they came up

         15    with some disclosure risk measures which

         16    involve uncertainty measures and what you

         17    need to decide is what -- and that's tied to

         18    perception -- what is the perception of the

         19    public about enough uncertainty that can be

         20    viewed as protection of confidentiality.  But

         21    the least you can do is say, I can tell you

         22    that the probability of identification and



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           221


          1    subsequent -- putting some confidence limits

          2    on protected information is such-and-such.

          3    And if that uncertainty is enough, according

          4    to perception of the public, then you're

          5    protected.

          6              DR. NEERCHAL:  Could I ask a

          7    question?  What are the specific types of

          8    information that your respondents would be

          9    most sensitive towards in terms of being

         10    published?  I think you're already publishing

         11    some data online or large ---- reports, and

         12    obviously -- have you had anybody complain

         13    about why did you put that number up there?

         14    That is one of the things that would be

         15    important to identify:  The kind of

         16    information that they're most sensitive

         17    about.  And I think on that side of it, it's

         18    important to get your stakeholders at the

         19    table.  I think here -- I think whether you

         20    are doing an adequate job or not will depend

         21    on who has something to "lose" -- and I'm not

         22    -- may not be financially, but something to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           222


          1    worry about, they don't like to be published.

          2    I think that would be something I would like

          3    to know.

          4              MS. KIRKENDALL:  Let me respond.

          5    In the electric power area the Office of Coal

          6    has done a lot of work with respondents

          7    because they go back and forth.  They'd like

          8    to be able to make data public.  And so every

          9    time, they'll go make a proposal on the

         10    Federal Register notice, saying that we're

         11    going to make all of these data items public.

         12    And so they get a ---- price and the

         13    company's saying, well, it's all right for

         14    these variables, but we need to protect

         15    those.  So I think they're unique in the

         16    federal government that some of their surveys

         17    -- they have a lot of surveys -- some data

         18    items are confidential and some are not.  The

         19    ones that are not confidential are actually

         20    put up on the Web, they're

         21    company-identified, and the ones that are

         22    confidential have to be treated in some way



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           223


          1    in the tables to make sure they're protected.

          2    So you're right, and you do have to work with

          3    your respondents.  And it may be that that's

          4    a way to be able to make more available is

          5    just to find out which data items on the form

          6    can be made public.

          7              DR. NEERCHAL:  And you may want to

          8    let them know that, hey, this particular

          9    column is highly correlated with this one,

         10    so --

         11              MS. KIRKENDALL:  They're typically

         12    worried about cost information, about price

         13    information.  I guess in one survey, they're

         14    worried about stock information, which I was

         15    surprised about, but the money --

         16              SPEAKER:  Anything that has a

         17    dollar sign.

         18              DR. HENGARTNER:  Just to make sure,

         19    Neha, Mark, and then Stan.  So that's the

         20    order.

         21              MS. KHANNA:  As a user of data, I

         22    have a question.  It's fine to swap data or



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           224


          1    to contaminate the data in some way for

          2    confidentiality.  But at the same time, we

          3    don't want to destroy inherent relationships

          4    within the data, which, of course, you're not

          5    going to even know what relationship a user

          6    is looking for.  But that raises an issue of

          7    how much noise are we going to add.  On the

          8    one hand, we may want to change the order and

          9    ranking of the data, but that may be

         10    destroying some relationship.  I don't know

         11    if there's anything in the statistical

         12    literature out there that addresses this

         13    question.

         14              DR. HENGARTNER:  Mark?

         15              DR. BURTON:  The one thing I love

         16    about these meetings is they give me repeated

         17    opportunities to show how little I know.  I

         18    work with some transportation data, where the

         19    firms reporting the data have the option of

         20    having values masked.  It turns out that most

         21    of them elect not to.  They just, for

         22    whatever reason, are not sensitive.  Some,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           225


          1    however, will go through the masking process.

          2    If you have individual respondents that do

          3    want confidentiality and other respondents

          4    that don't, what does that do to the

          5    statistical characteristic of the data?

          6              DR. HENGARTNER:  Stan?

          7              MR. FREEDMAN:  Nancy said something

          8    that reminded me of a discussion that we had

          9    with a respondent one time when we were out

         10    doing a site visit -- and Jake, I'm sorry I

         11    didn't think of this sooner because I would

         12    have mentioned it to you.  The issue of

         13    sensitivity also relates to timeliness.  Not

         14    disclosing stock data or cost-of-fuel data,

         15    for example, might be important to the

         16    respondent three months or six months after

         17    it's reported, but a year or two years

         18    afterwards, it's no longer important.  It

         19    loses its significance because it's no longer

         20    timely.  So you may need to figure that into

         21    the study as well.  And like I said, I'm

         22    sorry I didn't think about -- but we talked



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           226


          1    about it earlier, but I think it depends on

          2    the data and it loses its sensitivity over

          3    time.

          4              DR. HENGARTNER:  Moshe?

          5              DR. FEDER:  Yeah, the issue that

          6    Neha talked about -- destroying multivaried

          7    relationships between cells -- is actually --

          8    Jake did allude to that, and I was going to

          9    ask about it, too, because in my activity and

         10    disclosure limitation, we do a

         11    post-processing assessment of the "damage"

         12    that we have done to the data, and

         13    traditionally, what we have done is we looked

         14    at univariate statistics, overall estimates

         15    of whatever we are estimating or think people

         16    might want to estimate, and we are talking

         17    about here tons of data, so what you do is

         18    you do some distribution on that and say how

         19    many estimates stayed within, say, two

         20    percent of the original ones, how many went

         21    up, how many went down.  And then we look at

         22    some multivariate relationships, which



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           227


          1    originally had been just limited to estimates

          2    by some domains.  So if I were interested,

          3    and I survey human populations, how many

          4    estimates for females have change, how many

          5    for males, how many by some region, and so

          6    on.  But then I started being worried more

          7    about what Neha is worried about because

          8    analysts want to know how do my regression

          9    models remain after we've done the analysis.

         10    And this is precisely what I'm going to do

         11    when I get back to my office, hopefully on

         12    Monday, and propose some modeling.  And

         13    someone actually was on team, but did it for

         14    another project -- but I was pleased to see

         15    that in his case -- and we are talking that

         16    case on a logistic regression model -- the

         17    regression coefficients have changed, but not

         18    the direction.  So, you say, this contributes

         19    to the outcome, but not exactly the same way.

         20    And I recommend to do the same thing.  I

         21    don't think you can worry about in the design

         22    stage exactly how much you can afford to do,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           228


          1    but post-processing analysis, which is also,

          2    within limitation, disclosed to the public,

          3    is a good idea.  So you can say we have done

          4    some multivaried modeling and we know that,

          5    yes, there are some changes to regression

          6    coefficients or whatever, but they generally

          7    keep the same direction, so inference is not

          8    too bad.  But, yeah, that's something that

          9    I'm worried about, too.

         10              DR. HENGARTNER:  Yes?  Can you come

         11    in closer and state your name, please?

         12              MS. HOWELL:  I'm Shannon Howell,

         13    and I have a question.  I was just thinking

         14    about a case that I saw where suppression of

         15    confidential information came up, and with

         16    any of these alternative methods, if anybody

         17    could see a problem with very thin cells,

         18    wherein you might have just one or two

         19    respondents, if you have one respondent, you

         20    publish something that's not the number they

         21    gave you, they might become slightly upset or

         22    worry about how you're handling the data.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           229


          1    And two, if you have just a couple of

          2    respondents, they might very easily be able

          3    to figure out what you're doing.

          4                   (Laughter)

          5              DR. HENGARTNER:  First Moshe, and

          6    then the gentleman in the back.

          7              DR. FEDER:  I actually happen to

          8    know a little bit about what's done on this

          9    cell, which is not necessarily one where the

         10    sample size is small.  Could be a cell

         11    dominated by one major market player and also

         12    by some kind of cooperation between

         13    individuals dominating the cell to share the

         14    information and thereby inferring about the

         15    remainder.  And I know that in Canada, I

         16    believe the number was fifteen or just one of

         17    two, and in our case, we have a more complex

         18    cell suppression formula, but it's got to be

         19    sufficiently large so you won't be able to

         20    find out what's in it by using some

         21    information that you know from private

         22    sources.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           230


          1              MS. HOWELL:  Obviously in these

          2    cases, you could still suppress the

          3    information entirely, withhold it.  But other

          4    than going to greater extent of alternative

          5    data, could you get around that?

          6              DR. FEDER:  Mathematically, these

          7    questions are very challenging.  That's why

          8    the first approach that Jack (sic) was

          9    talking about -- treating the microdata

         10    before production of the table -- you avoid

         11    that.  And in fact, there's an issue with the

         12    three-way table, there's an issue with

         13    someone combining one table with one

         14    regression model and so on.  Where does it

         15    stop, how do you know?  Or, as I said, using

         16    information on yourself or on your

         17    competitors that are sharing it with you --

         18              DR. NEERCHAL:  I think one of the

         19    other options I think you should keep in mind

         20    is, can you get away by publishing only some

         21    of the ----- information and if there is some

         22    particular way to be sensitive.  And then ask



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           231


          1    yourself, do some regional summaries only,

          2    and is that enough for the kind of users you

          3    have in mind.  I think that's -- or if it is

          4    a regression ---- population of correlation

          5    -- I think you have to ask that question.

          6    I'm sure the person who worked in this

          7    problem is the one who invented the phrase

          8    "damned if you do and damned if you don't."

          9                   (Laughter)

         10              DR. HENGARTNER:  There's one

         11    question there, and then one more after that,

         12    and I am unfortunately going to have to break

         13    up the session, but I invite people to

         14    continue discussion; maybe tomorrow we'll

         15    have some time left at the end of the

         16    meeting.  So two more comments.

         17              MR. WATSON:  Bill Watson, CNEAF.

         18    Here I'll talk only about cull, and it may

         19    not necessarily generalize to other data sets

         20    that we have at EIA.  Almost all of our

         21    microdata is confidential, so nobody except

         22    staff that are at EIA or at other federal



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           232


          1    agencies that will sign a data disclosure

          2    agreement will actually be able to get at the

          3    microdata.  And we do not ever mask the data,

          4    the microdata, but when we go to tables with

          5    it, we have some very well thought out

          6    suppression rules, and so we'll -- you see a

          7    lot of "W"s in our tables -- withheld data.

          8    An objective of the withholding rules is to

          9    make sure that nobody could back out or use

         10    mathematics to calculate somebody else's

         11    data.  So this would be in a situation where

         12    there were only one or two people reporting

         13    in a state, and you had a region total or you

         14    had other states in a national total, and you

         15    could back out some of the data that was

         16    associated at the firm level with somebody.

         17    And that works to maintain the

         18    confidentiality.  It does have the obvious

         19    problem, and every once in a while --

         20    especially in the areas that have already

         21    been mentioned -- anything associated with

         22    dollars, such as price or cost, and also with



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           233


          1    stocks -- people are frequently calling us

          2    and saying, can't you release that data,

          3    can't we see that microdata there?  Because

          4    that's really what they're after, but I was

          5    kind of surprised to see this topic here

          6    because I thought we had sort of cottoned on

          7    to the withholding thing and that was it.  So

          8    I'm very happy, actually, to see that the

          9    conversation is going this way now.

         10              And I also support the idea that

         11    Stan brought up, that we ought to have a

         12    sunshine rule here to perhaps see if we could

         13    convince our respondents and after one year

         14    or two years or whatever it is, if the

         15    microdata no longer would be withheld or not

         16    be confidential after a certain period of

         17    time.

         18              DR. HENGARTNER:  Last person.

         19              MS. WAUGH:  I'm Shawna Waugh from

         20    Statistics and Methods.  One of the

         21    non-statistical solutions not discussed, and

         22    I think ---- included in the survey is



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           234


          1    whether or not companies are willing to sign

          2    waivers.

          3              SPEAKER:  We can't hear you,

          4    Shawna.

          5              MS. WAUGH:  Another non-statistical

          6    solution is asking companies to sign waivers.

          7    And EIA has done that in the past, for

          8    example, with your case where one company is

          9    reporting a large quantity and therefore

         10    might be able to discover the amount.  I

         11    really do think that working with our

         12    respondents is another way by requesting

         13    waivers, and that that might be an important

         14    aspect to research when you're looking at

         15    disclosure.

         16              DR. HENGARTNER:  Thank you very

         17    much.  This seems to be a very interesting

         18    topic, and I do not want to cut down the

         19    discussion, so I made a note to myself to

         20    reopen this discussion if anybody else wants

         21    to add to it at a later time during our

         22    meeting.  But I'd like to proceed ahead to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           235


          1    two break-out sessions and I'd like Mark

          2    Burton, Cutler Cleveland, Jae Edmonds, Neha

          3    Khanna, Nagaraj Neerchal, and Tom Rutherford

          4    to go downstairs to Room 5E-069 for a

          5    break-out session on improving the SAGE

          6    petroleum refinery model.  Yes, that seems to

          7    be a follow-up.

          8                   (Recess)

          9              DR. HENGARTNER:  Okay, ladies and

         10    gentlemen, let's continue.  I'd like to

         11    welcome Stan Freedman to discuss with us the

         12    functional requirements for EIA Internet data

         13    collection systems for our break-out session.

         14    So welcome, Stan.

         15              MR. FREEDMAN:  Thank you very much.

         16    Can we turn the lights out up front, please?

         17    Okay, what I would like to do is put this

         18    project in some historical and organizational

         19    context and then talk about the actual

         20    functional requirements we have come up with

         21    for the IDC, the Internet system, and then

         22    get comments from the committee, primarily



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           236


          1    related to did we leave any functionality out

          2    of the system, were there other systems we

          3    should have looked at and considered, and,

          4    from a respondent's point of view, do you

          5    think that this will be a successful Internet

          6    system to get information into EIA.

          7              About a year ago, maybe a little

          8    bit more, a subgroup of the Strategic

          9    Planning Group at EIA charged a number of us

         10    with the responsibility of coming up with a

         11    functional requirements document for an

         12    EIA-wide system to collect data over the

         13    Internet.  To be very clear about the

         14    boundaries of the project, this was

         15    functional requirements, it was not

         16    specification for code, it was not to

         17    actually paint the screens or decide the size

         18    of the buttons or whether the "submit" button

         19    should be in the upper left-hand corner or

         20    the lower right-hand corner, or the color of

         21    the font or the background.  Those things

         22    need to be decided and determined through



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           237


          1    usability testing and literature review, and

          2    one size, we felt, did not necessarily fit

          3    all, so it was not appropriate to dictate a

          4    specific set of screens and that kind of

          5    thing.  What we wanted to do was talk about

          6    functionality, what did the system need to be

          7    able to do, how would it satisfy respondents'

          8    needs, how would it satisfy the needs of the

          9    people at EIA who would be using the system

         10    to collect the information.  So that's what

         11    we were charged with.

         12              Our research methods:  We first

         13    formed an interoffice team.  The reason for

         14    that is, as you might expect, if you want an

         15    agency-wide system, you need agency-wide

         16    buy-in.  So there was a representative from

         17    each of the offices at EIA, except for the

         18    Office of Resource Management that handles

         19    our personnel budget.  All of the major

         20    surveys at EIA were represented, as was the

         21    Office of Information Technology.  Those of

         22    us from the Statistics and Methods Group who



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           238


          1    served on the team were involved in usability

          2    and interface with users and that kind of

          3    thing.  So it really was an interagency

          4    group.

          5              I would like to quickly introduce

          6    the members of the group; they are the ones

          7    who really did all the heavy lifting and the

          8    ones who worked very hard for a year in

          9    coming up with these requirements.  It wasn't

         10    always easy for us to get on the same page or

         11    to agree what some of these functional

         12    requirements ought to be, but I think what

         13    you'll see is that we have a wide variety of

         14    things.  We have Betty Barlow from OIT.

         15    Betty, you could just raise your hand?

         16    Audrey Corley from the Office of Oil and Gas;

         17    John Paul Deley, who's back there from NEIC,

         18    who was our records management and retention

         19    expert on the project --

         20                   (Laughter)

         21              MR. DELEY:  That's for sure.

         22              MR. FREEDMAN:  -- and never let us



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           239


          1    forget about the importance of documentation

          2    and maintaining our information.  My arm was

          3    twisted to be the chair of the group, or the

          4    cat-herder, as the case may be.  Alethea

          5    Jennings from SMG more than ably assisted me

          6    with that, and bailed me out whenever there

          7    was a problem, and told me that I was really

          8    off track and wasn't paying any attention.

          9    We had Tom Lorenz, who you heard from this

         10    morning, from the Energy Markets and End Use,

         11    who was representing the consumption surveys.

         12    Paul McArdle, whom I saw in the back here,

         13    who has an Internet survey that is

         14    operational now.  So the people we had on the

         15    committee were not simply theoretical, they

         16    are people who have implemented IDCs at EIA.

         17    Rick Salkov -- I didn't see Rick, is he here?

         18    He's from the Office of Coal, Nuclear,

         19    Electric, and Alternative Fuels.  Just about

         20    all of their surveys are being collected over

         21    the Internet now, so his experience with that

         22    was invaluable.  And Paula Weir, also from



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           240


          1    the Oil and Gas, who has a survey in her area

          2    that is collected over the Internet and also

          3    has done quite a bit of research in the area

          4    of IDCs and has, I believe, even written some

          5    papers in that area.

          6              So we had a good mix of

          7    individuals, both in terms of application and

          8    theoretical knowledge.  Where we are right

          9    now:  In February, we briefed the Goal Four

         10    Subcommittee of the Strategic Planning Group.

         11    Nancy, who I think is at another session, is

         12    actually the champion of that group.  We

         13    basically told them where we are.  We had

         14    come up with the handout that you got a week

         15    or two ago of the specific functional

         16    requirements of the system.  We're now in the

         17    process of actually drafting the report, and

         18    that doesn't mean that we are past the point

         19    of input from the ASA committee; in fact,

         20    quite the opposite.  Had we come to you a

         21    year ago with a bunch of very unformed ideas,

         22    I'm not sure that your input could have been



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           241


          1    as thoroughly understood and digested as I

          2    hope it will be today.

          3              The first thing we did in trying

          4    not to reinvent the wheel is we visited other

          5    statistical agencies and organizations that

          6    collected information over the Internet.  We

          7    visited Census -- you heard Tom this morning

          8    talk about a software system there, Census

          9    Taker, that is used -- I guess, Tom --

         10    primarily on their economic surveys, and we

         11    talked to them about that, what

         12    functionalities they had; we had a demo of

         13    that.  We also had a demo of the system that

         14    is going to be used to collect the 2010

         15    Census, the household survey.  We met with

         16    people from Bureau of Labor Statistics,

         17    Agriculture, we met with Westat, and we

         18    looked at the Blaise system, and how that

         19    worked.  We also took advantage of the

         20    systems that were already in place at EAI; we

         21    had extensive input and briefings from

         22    electricity and coal in terms of what worked



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           242


          1    for them.  Paul spent a lot of time talking

          2    about his survey, Greenhouse Gases, and the

          3    shop survey.  We also did fairly extensive

          4    literature search, looking at papers,

          5    conference proceedings, and seminars that had

          6    been attended by the committee members.

          7              Now, you may be asking yourself the

          8    question which we asked ourselves when we

          9    started out:  If there already are a few

         10    offices at EIA, maybe between a third and a

         11    half of our surveys that are collecting some

         12    or all of their data over the Internet, why

         13    do we want an EIA-wide IDC?  What's the

         14    point?  There are some things out there that

         15    already exist.  Obviously, it's going to cost

         16    money to develop an agency-wide system.

         17    There was a belief by the strategic planning

         18    committee -- and I think that we subscribe to

         19    this, or our group does -- that in the long

         20    run, it is more efficient and cost-effective

         21    to have a system and to have a modular system

         22    to collect information than to have three or



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           243


          1    four different systems that have to be

          2    supported with different sets of programmers,

          3    that have to be supported with different

          4    computational platforms, and things like

          5    that.

          6              Part of the political environment

          7    that we were operating in was to design a

          8    system that would allow existing systems to

          9    be migrated into it when there were

         10    significant changes to the forms, the data

         11    that were collected, or the software systems

         12    simply needed to be updated.  It is not and

         13    will not be our intention to make a proposal

         14    and then impose it upon everybody at once;

         15    these are for new systems or systems that

         16    will be revised.  So that was one of our

         17    operating criteria.

         18              We basically had four guiding

         19    principles:  The first was that it be easy to

         20    implement, modify or -- as I said before --

         21    migrate to the IDC.  The second is that it

         22    provide tools that enable the survey group --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           244


          1    and by that, we mean the people who are

          2    actually running the survey -- to have

          3    ownership of their survey application.  One

          4    of the problems when you try to develop an

          5    agency-wide tool is to get people to buy into

          6    it and accept it and to take ownership of it.

          7    Is it theirs, do they have a stake in its

          8    outcome, and will it satisfy their needs?

          9    The third point, which is probably the most

         10    important from outside the agency looking in

         11    -- and Barbara and others mentioned that this

         12    morning when we were talking about the MECS

         13    -- is the IDC, the Internet collection

         14    system, has to provide value-added from the

         15    respondent's point of view.  Why do they want

         16    to bother to use it?  If they're used to

         17    faxing in a form or filling in an electronic

         18    spreadsheet, what's the value, what's the

         19    benefit to them in filling in the data over

         20    the Internet?  And the third is that anything

         21    we did needed to promote high quality

         22    information coming into EIA, and it had to do



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           245


          1    that through editing that was available to

          2    the respondent.  One of the goals was to

          3    reduce the errors in the data before they

          4    came into EIA -- that users be clearly

          5    notified about when their data was submitted

          6    to EIA -- in other words, they could

          7    understand where they stood in the system.

          8              Moshe mentioned this morning about

          9    filling in a survey form, getting halfway

         10    through, the session isn't saved, not wanting

         11    to be bothered, so that needed to be handled

         12    -- and then sort of the commonsensical things

         13    about the navigation being clear and

         14    straightforward and the system be usable.  We

         15    wanted to make sure that everything that was

         16    ever developed there would be a partnership

         17    between the coders, the developers, our

         18    Office of Information Technology, and the

         19    program offices.  The responsibilities would

         20    include OIT developing and maintaining the

         21    overall infrastructure and providing tools

         22    for the program offices to implement their



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           246


          1    particular surveys.  And one of the things

          2    that we found when we looked at previous

          3    experiences at EIA is that there needed to be

          4    assistance provided to our respondents

          5    through help centers and things like that

          6    when they tried to use the IDC.  The idea

          7    here is that we would have a modular system

          8    that survey managers and subject matter

          9    experts could pick and choose the things that

         10    applied to their survey.  One size, we

         11    realized, did not fit all.  If you have a

         12    survey with four questions on it, the look or

         13    the screen that would be painted for that

         14    survey would be very different, for example,

         15    than for the MECS survey that has a lot of

         16    complicated questions and might have skip

         17    patterns and things like that.  Our intention

         18    was to have functionality that would include

         19    all of those things and then have the survey

         20    managers pick and choose those pieces that

         21    best fit their system, and then that be

         22    integrated into their particular online



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           247


          1    collection.

          2              We also are recommending to the

          3    Strategic Planning Committee that we initiate

          4    a pilot to put one survey into this IDC.  And

          5    although we are still in the process of

          6    talking about how that implementation ought

          7    to happen -- and I would be very interested

          8    to hear the views of the committee in terms

          9    of how we make that successful -- the idea

         10    would be to start out simply and build the

         11    system modularly and not try to put

         12    everything into the system at one time.

         13              We divided functional requirements

         14    into several areas, and these are discussed

         15    in detail in the handout that you got, and I

         16    can pull up any of the slides to talk about

         17    what the specifics are on that.

         18              The first area of functionality is

         19    actually the collection of the data.  It is

         20    what the respondent sees when they log into

         21    the system.  What functionalities are

         22    required, what should that common look and



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           248


          1    feel be, and that kind of thing.  That is the

          2    heart of the system and took us the longest

          3    to come up with specific functional

          4    requirements.  The system needed to have

          5    interoperability; the pieces needed to link

          6    together.  And the IDC needed to link as

          7    seamlessly as possible with the production

          8    systems that actually aggregate the data and

          9    do editing after the information is in and

         10    produce the publications.  Again, the IDC is

         11    simply the pipeline that gets the data past

         12    EIA's firewall.  Then the data would be

         13    passed to the existing processing systems.

         14    This was not a system to be developed to go

         15    from the respondent's submission to the

         16    Internet table on the Web.  So it had to link

         17    seamlessly with that.

         18              The other thing we felt was

         19    important was to have some sort of

         20    performance measurement in the system, which

         21    would include response rates.  There is

         22    functionality recommended to talk about



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           249


          1    errors that are made on specific cells in

          2    submissions, which would give you some idea

          3    of what questions respondents were having

          4    particular problems with and things like

          5    that.  And again, we have someone here today

          6    who can talk more about what those

          7    performance measures would be.

          8              Retention and disposition were

          9    important.  How long is the data maintained

         10    in the IDC?  What might the respondent have

         11    access to, in terms of what they had

         12    submitted previously?  How long should we

         13    keep that information in our system, both to

         14    have an audit trail during the production

         15    cycle and an audit trail later on?  If it's

         16    an annual survey, does it need to be for the

         17    whole year, does it need to be for two years,

         18    does it need to be for three years?  That

         19    functionality needs to be built into the

         20    system; the needs for different offices

         21    obviously will be different, but there needs

         22    to be a retention and disposition schedule of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           250


          1    that information.

          2              Probably the most important lessons

          3    learned from our site visits, and certainly

          4    from the input that we got from the

          5    Electricity Division, was that of essential

          6    support services.  The one thing that you

          7    don't want to happen is to roll out an

          8    Internet system and nobody's able to log into

          9    it because the survey will only handle six

         10    people at a time instead of 600 people at a

         11    time, when they're all trying to submit the

         12    data at the same time.  Can people access

         13    their passwords?  Do their passwords work?

         14    Whom do they call when they don't work?  They

         15    want somebody on the other end of the phone

         16    who will be able to answer their questions.

         17    And it's not always the same person;

         18    sometimes it's an IT person, sometimes it's a

         19    subject-matter person.  Users tend not to

         20    discriminate -- if they don't understand the

         21    question on a form, they're going to call the

         22    first phone number that they see in front of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           251


          1    them or click the link to the first e-mail,

          2    and they don't want to be referred to six

          3    places for an answer to the question.  That

          4    was something that EIA may not have planned

          5    as well for in the beginning with some of

          6    their systems as they could have; we didn't

          7    want to make that mistake again.  And that's

          8    a pretty resource-intensive activity.

          9              And the other area of importance

         10    was security.  Right now, EIA uses a single

         11    sign-on system; it seems to work well, and

         12    our recommendation is that we continue to use

         13    that, the idea being there is one secure

         14    place where a respondent would go to be able

         15    to access any of their Internet forms.  They

         16    wouldn't have to go to four or five different

         17    URLs -- they may need different passwords or

         18    authentications for different surveys that

         19    they're responding to, but there would be one

         20    place where they go.  So those were the areas

         21    that we were concerned about.  Again, from

         22    the respondent's point of view, it was the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           252


          1    common look and feel.

          2              The other lesson that we learned

          3    and heard over and over again -- in fact, I

          4    can remember Bob Schnapp, who I think is in

          5    the back of the room, saying to us at a

          6    meeting, the value of the IDC to us is to

          7    force the respondents to provide us with the

          8    correct data the first time.  And we talked

          9    about some of those things this morning.

         10              MECS is interested in eliminating

         11    keying errors and unit errors -- people

         12    reporting in volume instead of heat units and

         13    vice versa -- there's a lot that you can do

         14    on the front end of an Internet system to

         15    ensure that those mistakes are not made.  We

         16    classified our editing into two categories,

         17    and you'll see if you go through the specific

         18    requirements, one of them -- Paul, help me

         19    out here if I'm not using the term we're

         20    currently using in the report -- hard edits

         21    versus soft -- okay, thank you.  I was out

         22    for two weeks sick, and the words might have



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           253


          1    changed -- but the idea was still the same,

          2    and that is that there were some edits that,

          3    when they happened, the respondent would be

          4    notified and notified in a place on the form

          5    that made some sense that they could see, and

          6    they would not be able to proceed until they

          7    corrected those edits -- things like

          8    incorrect state codes, incorrect units, and

          9    things like that, those were the hard edits,

         10    you don't get to submit your data until

         11    you've satisfied those.  Then there were the

         12    soft edits, things that might be statistical

         13    edits, things that would have range checks,

         14    and our decision was to leave whether they'd

         15    be hard or soft to the program people, but

         16    those edits could be overridden with

         17    explanations or not overridden, depending on

         18    what the edit would be, but the respondent

         19    would still be notified.  Again, over and

         20    over again, we heard that many of the errors

         21    are clerical or keying errors that are made

         22    when there's a submission, and the computer



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           254


          1    is real good at picking those things up, and

          2    every effort should be made to capture as

          3    many of those as possible and eliminate them

          4    before the submission actually came into EIA.

          5              So with that, I would like to first

          6    open it up to any other member of the

          7    committee who would like to add anything

          8    specifically about their areas, common look

          9    and feel, interoperability, performance

         10    measurement, to help inform the committee and

         11    then to get comments back from the committee

         12    in terms of, did we leave anything out, are

         13    there any systems out there that we should

         14    have looked at that I haven't mentioned.

         15              If you're interested, we passed

         16    around a flow chart of how the data would

         17    come into the system.  If you're interested

         18    in that level of detail, we could certainly

         19    go over that to explain how that would work.

         20    So if we could have the lights -- and then is

         21    there anybody from the committee who'd like

         22    to add anything either about common look and



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           255


          1    feel, performance measurement --

          2              DR. HENGARTNER:  Thank you very

          3    much, Stan.  So as Stan suggested, this is

          4    now an open-floor discussion, so I invite

          5    everybody sitting on the sideline to move in

          6    closer, especially those who are expert about

          7    Internet and Web design.  Please feel free to

          8    join us, the committee members as well that

          9    Stan was referring to.  There are plenty of

         10    spaces here.  It makes our discussion easier

         11    and more lively.  Thank you.  You, too, Stan.

         12              MR. SINGPURWALLA:  Where is all the

         13    data stored when you push it through the

         14    Internet thing?  Is it on, like, Oracle

         15    tables or --

         16              MR. FREEDMAN:  I'm going to ask

         17    Betty to address that question.

         18              MS. BARLOW:  Yeah, it would be

         19    Oracle.

         20              MR. SINGPURWALLA:  Okay.

         21              MR. FREEDMAN:  Do you want to

         22    explain a little bit about how the data would



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           256


          1    come in and the storage of passing the files,

          2    if that would be helpful -- at least in terms

          3    of how things work now and how we potentially

          4    envision it.

          5              MS. BARLOW:  Well, they'll be

          6    stored -- if I say the word "screen subnet,"

          7    do you know what that would refer to?  It's

          8    an area that's walled off from our internal

          9    LAN and also walled off from the Internet.

         10    So that's where we could have the databases

         11    and the applications.  So the data come into

         12    that Oracle database in the screen subnet.

         13    And there are actually two possibilities that

         14    Stan sort of alluded to.  We understand that

         15    some of the survey managers will want to have

         16    that data transferred to another database

         17    inside because they already have a processing

         18    system which has its database.  So they'll

         19    want to use a different database for their

         20    internal processing.  We also envision that

         21    some of the survey managers might want to use

         22    that database in the screen subnet directly,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           257


          1    use that very database, to do whatever they

          2    do inside their internal processing.  So both

          3    options would be available.

          4              MR. FREEDMAN:  One of the things

          5    that happens now on some of the IDCs in the

          6    electricity are -- and I think, Scott, maybe

          7    in the coal area too -- is that if a

          8    respondent does not submit their data over

          9    the Internet -- if they fax it in, for

         10    example -- EIA people can use the system to

         11    key the data into the database.  The other

         12    thing is there will be, I'm sure,

         13    requirements from some of the survey systems

         14    to allow respondents to look at their

         15    historical information when correcting an

         16    edit flag.  So there will be functionality

         17    built into the system for those surveys that

         18    require it that would allow respondents to go

         19    back and look at their historical

         20    information, and that gets into the issue of

         21    how long and what kind of data get retained

         22    in the IDC.  The IDC is not intended to be a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           258


          1    permanent repository for data for 20 years --

          2    it doesn't make sense and it's not practical

          3    -- but there will be a period of time when

          4    there are data that have to be stored in

          5    order to satisfy edit criteria and that kind

          6    of thing.  I don't know if that's more of an

          7    answer to your question than you wanted --

          8              MR. SINGPURWALLA:  No, no, no,

          9    that's a great.  Wouldn't how much data you

         10    can save depend more on the size of the work

         11    or database behind the IDC?

         12              MR. FREEDMAN:  I would have to

         13    refer that to Betty.

         14              MS. BARLOW:  Which is variable.  We

         15    can always increase it.

         16              MR. SINGPURWALLA:  Yeah, yeah.

         17              MS. BARLOW:  And there's another

         18    point, too, that I would make.  Some people

         19    are concerned about having data in the screen

         20    subnet.  We have the intention -- I don't

         21    know when it's going to happen -- of putting

         22    a proxy server in the screen subnet, and that



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           259


          1    proxy server is simply there to route

          2    incoming information.  And so in that case,

          3    you could have both the application and the

          4    app server -- containing the applications --

          5    and the database server inside, on the LAN,

          6    and you have less exposure in that case.  So

          7    that's the plan down the road somewhere.

          8              DR. HENGARTNER:  I have one concern

          9    with the ability of recalling data from past

         10    survey.  That concern has to do with

         11    confidentiality.  Would you be able to spit

         12    back data that you are deeming confidential

         13    to a user which may be different from the one

         14    who had actually entered the data?

         15              MR. FREEDMAN:  The only one who

         16    would have access to the system is the person

         17    who had the password in the company.  Now, if

         18    a company chose to give their password out to

         19    all sorts of people in different departments,

         20    I'm not sure what we could do about that.  I

         21    think I understand what your point is --

         22              DR. HENGARTNER:  It's not even a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           260


          1    point, it's a question.  It's a red flag in

          2    my mind that if you are providing data back,

          3    then that is not quite publishing it, but it

          4    comes very close to it, and it's a danger.

          5              MS. BARLOW:  But it's only

          6    accessible to the company people, or people,

          7    as Stan said, to whom they choose to give the

          8    password, or somebody else has identity --

          9              MR. SINGPURWALLA:  So only I could

         10    see the data that I put in.

         11              MS. BARLOW:  Well, not you

         12    literally, but your company.

         13              MR. SINGPURWALLA:  Well, your

         14    company.

         15              MS. BARLOW:  Yeah, your company.

         16              MR. SINGPURWALLA:  And whoever ----

         17    to the committee --

         18              DR. FEDER:  Someone who's trusted

         19    by the company.

         20              MR. SINGPURWALLA:  Mm-hmm.

         21              DR. HENGARTNER:  Yeah, but this is

         22    one more password you need to crack to get



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           261


          1    all this information.  Let's face it, if I'm

          2    going to make industrial espionage, that

          3    would be a weak link, and I would be

          4    concerned about that.

          5              MS. BARLOW:  Well, you have to have

          6    a user ID and password to get access to that

          7    particular data.

          8              DR. FEDER:  It's like I have my

          9    password for my banking online --

         10              MS. BARLOW:  Right, same thing.

         11              DR. FEDER:  -- if I give it to

         12    someone, it's my --

         13              SPEAKER:  It's your problem, not

         14    the bank's.

         15              DR. FEDER:  Can I just -- I

         16    actually liked the flow chart that was --

         17              MR. FREEDMAN:  Would you like us to

         18    put up on the screen --

         19              DR. FEDER:  No, no, no, no.  I ---

         20    the help files that I liked.  I would just

         21    add one more thing, a user feedback form

         22    that's separate from the questionnaire, where



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           262


          1    you go by e-mail to the developers, not part

          2    of the questionnaire because you know and I

          3    know how long it takes until survey data

          4    processed.  I just would like you to go and

          5    say I used your system, and I really wanted

          6    to get to this, but the skip pattern prevents

          7    me from the -- would you please enable me --

          8              MR. FREEDMAN:  I understand, yes,

          9    yes.

         10              DR. FEDER:  -- or something like

         11    that, so a user feedback.

         12              MR. FREEDMAN:  Okay.  Bob, I think,

         13    was going to mention something about --

         14              MR. SCHNAPP:  Yeah, well, I think

         15    about going back and looking back at previous

         16    years' data, an issue that for -- raised the

         17    other day --

         18              MR. FREEDMAN:  Bob, they want you

         19    to use the mike.

         20              MR. SCHNAPP:  I thought I talked

         21    loud enough.  I might not talk good, but I

         22    can talk loud, they say.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           263


          1              There's an issue about going back

          2    in time to look at previous years' data, and

          3    in a different respect, it's important

          4    because if the company is sold, is the new

          5    company allowed to see the previous several

          6    years' worth of data?  Do they buy the data

          7    with the company or not?  And I'm sure that

          8    there are contracts that cover it -- they

          9    don't come to us for us to see -- and there

         10    are a lot of companies that sell off part of

         11    their assets, keep assets, and they don't

         12    want the new company to know some of the

         13    things that they were doing before or costs

         14    that they had.  So it's something that was

         15    raised, at least in our group recently, that

         16    we were going to begin to address, and so

         17    maybe if people here have an idea, people

         18    have to listen to it.

         19              MR. FREEDMAN:  That's something we

         20    actually hadn't thought about in our group.

         21    I don't think it's -- Paula, you wanted to

         22    say something.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           264


          1              MS. WEIR:  Well, I was going to

          2    say, I think it varies by survey, and that's

          3    one of the areas that we thought each survey

          4    would decide what their edit criteria going

          5    in -- there's a part of this called the

          6    administrative module that we would be using

          7    to put in our parameters and our rules and

          8    all that, and within that, as well as what's

          9    hard, what's soft.  It would be within that

         10    framework that the survey manager would

         11    decide.  They could just choose not to do

         12    historical edits, period, or if there's a

         13    sale, merger, that you don't go back in time.

         14              But, Stan, the part I was going to

         15    comment on is that -- in case people didn't

         16    get a chance to see this -- Stan had talked

         17    about two points, one of which was that we

         18    really view this as a way to save money

         19    because of data entry.  There is no money to

         20    be saved in data entry; I mean, that's it.

         21    There is some money in systems design, but

         22    the other point is in the data editing.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           265


          1    Forty percent of our survey costs are spent

          2    on data editing.  The more we can get the

          3    respondent to do, that's money back in our

          4    pocket that we could use on systems as well

          5    as overall quality.  So it's the overall

          6    quality that we also expect to go up.

          7              And the second point was about the

          8    buy-in; from the respondent's point of view,

          9    why should I do this?  I'm doing just fine,

         10    thank you very much -- is the two things that

         11    we're considering -- or what we've put in the

         12    requirements is the electronic file transfer

         13    that Betty referred to a little bit.  Because

         14    a large company doesn't want to be entering

         15    data in through an IDC, so we will have a

         16    format for electronic file transfers.  Once

         17    the data's in to us, they can have the same

         18    screens, it's treated the same, but they

         19    don't have to go through that data entry

         20    process.

         21              And the second thing is unlike

         22    household surveys, where the incentive for



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           266


          1    response -- you're always trying to balance

          2    the quality of the edits with the high

          3    response rates -- in household surveys, there

          4    are incentives to be gotten; you give them 25

          5    bucks for filling out the survey or something

          6    like that.  In business surveys, such

          7    incentive haven't existed until Internet data

          8    collection, and the thoughts that are being

          9    considered is they're information that you

         10    can give back to the company that they

         11    wouldn't have otherwise.  But you've got a

         12    perspective, different, because you've got a

         13    lot of data.  What can you give back to that

         14    company to encourage them to use this tool?

         15    And that's the other part we're encouraging,

         16    that the use of that, the development of that

         17    type of approach, you can still do it within

         18    a confidentiality framework, but we see

         19    things that nobody else sees, and we have all

         20    the data.  And that's how you try to get

         21    their buy-in.

         22              MR. FREEDMAN:  And if the committee



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           267


          1    has any idea of what some of those things

          2    might be, that would be helpful for us.

          3              DR. HENGARTNER:  First Derek and

          4    then --

          5              MR. BINGHAM:  I think this morning,

          6    it was mentioned -- so you were talking about

          7    keying errors and things like that and that

          8    there's potential for lack of -- maybe not a

          9    lot of potential for reduction in that, but

         10    the analyst goes in and changes it.  That was

         11    the one thing I learned from Tom this

         12    morning.  He can go in and say, that's not

         13    right, and he'll change things.  Now, here's

         14    a question:  If you have historic data up

         15    there, do Tom's changes get flushed back into

         16    the original database where the historic data

         17    is?

         18              MS. WEIR:  Yes.

         19              MR. BINGHAM:  Are you going to

         20    notify a company beforehand that --

         21              MS. BARLOW:  That's a survey

         22    decision.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           268


          1              MR. BINGHAM:  Because if you're

          2    going in, right, I could imagine -- so if

          3    there's the original database, which the Web

          4    application sends data to, it gets

          5    transferred over -- there's probably

          6    functionality that transfers this data to

          7    some other database, which the survey folks

          8    use -- and an analyst like Tom goes in and

          9    says, that's wrong, I'm changing that.  There

         10    has to be a procedure to send it back to the

         11    original database, and the next year the guy

         12    logs in and says, how -- a piece of historic

         13    information comes up, and it's not what he or

         14    she entered -- they may be puzzled as to why

         15    this wasn't the same, and they would think of

         16    it as an error.  So I'm just curious if there

         17    are any reporting mechanism to say, oh, by

         18    the way, these are the things that we

         19    changed.

         20              MR. McARDLE:  I can address that,

         21    at least on a survey-specific -- for the

         22    greenhouse gas survey.  If we find an error,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           269


          1    and we change it, we will contact the company

          2    then and there, via telephone, and get the

          3    contact representative from the entity

          4    submitting the data to discuss that and make

          5    sure they're on board with the change.  And

          6    that's how we'll close it out.  We'll note

          7    that in what we call a -- we keep a

          8    transcript of these -- not necessarily a

          9    transcript, but a chronicle of these

         10    communications, and we'll note it in that

         11    document that we spoke to so-and-so on this

         12    date, and we agreed to change this number and

         13    we'll put that into our auxiliary database of

         14    contacts.

         15              MR. SINGPURWALLA:  You're

         16    suggesting, though, that the functionality

         17    exists for notifying the respondent if those

         18    changes are made.

         19              MR. BINGHAM:  I don't know, so

         20    something like that, like a log that would --

         21    if they look at the -- accessing the historic

         22    data, then it would either be flagged and



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           270


          1    say, this was changed and --

          2              MR. McARDLE:  In some cases -- say,

          3    if there was extensive changes -- we may ask

          4    them to submit another certification letter.

          5              MR. BINGHAM:  Sure.

          6              MR. McARDLE:  But if it's something

          7    small, we'd probably just note it in what we

          8    call our RTD, which is a record tracking

          9    document, which tracks those transactions, so

         10    to speak.  But if it's extensive, we'll say,

         11    okay, we got a lot of changes, go look at

         12    this again, and then send us another

         13    certification letter saying that you agree

         14    that these are the data that are represented

         15    in the company.

         16              MR. WATSON:  Bill Watson, CNEAF.

         17    I'd like to come back to the point that was

         18    raised earlier about the committee's desire

         19    to design a common or one-fits-all-size IDC

         20    system for EIA.  We're pretty much down the

         21    road at this point with IDC, and I think the

         22    model that we've been using up to this point,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           271


          1    I would characterize it as being a

          2    distributed computing model.  And I think it

          3    has a lot of strengths compared to having

          4    something that is an attempt, actually, to

          5    define enough stuff so that everybody could

          6    use a common model.  And the reason I say

          7    that is because in almost every other area

          8    where we do IT, it has already been

          9    demonstrated many, many times -- for the most

         10    part, there are some exceptions -- that

         11    distributed computing tends to be a better

         12    model than centralized computing.

         13              Now, in the distributed computing

         14    system that we now have with our IDC, I

         15    really see four essential things that make

         16    the system work and make it work well.  One

         17    is the use of user's guides.  It's absolutely

         18    essential that our respondents understand

         19    what they're expected to do and have

         20    something that they can see electronically

         21    that's very simple, very visual, that helps

         22    them get through the single sign-on system,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           272


          1    that helps them get to their survey form.  We

          2    had very good luck with providing those to

          3    our respondents in the case of certain call

          4    surveys.

          5              A second thing is a notification

          6    system.  People need to be reminded that

          7    there's a certain due date, and they might

          8    have to be reminded several times.  And the

          9    IDC systems really facilitate that.  It's

         10    very easy to send mass e-mail notices out to

         11    people and remind them.  So you can get very

         12    high response rates by due dates.  We

         13    typically get 80 to 85 percent of people

         14    responding on coal surveys by the due date.

         15    So that would be another feature.

         16              And then, increasingly, we have

         17    developed tools that allow us to look at

         18    response rates, to look at how many forms are

         19    clean, how many forms come in by IDC versus

         20    fax or e-mail, and all of these things are

         21    now electronic.  All of the work that we did

         22    previously, a lot of it would be a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           273


          1    combination of manual methods and electronic

          2    methods.  That has fallen to the side; our

          3    methods are just so much better in that area.

          4              And then I think the most important

          5    thing here is best practices.  I think we

          6    have, for the most part, not explored that

          7    very well because what's happened, at least

          8    from my perspective, is that coal does its

          9    IDC, electric does its IDC, renewables, they

         10    do their IDC, oil does their IDC, natural gas

         11    does their IDC -- we're not talking with each

         12    other, we're not sharing our experiences.  I

         13    have no idea about what other IDC systems

         14    look like.  I'm just kind of curious about

         15    why you didn't think about sort of the way we

         16    were going and decided maybe that's not so

         17    good and we need to go in a different

         18    direction here.  Because I think the

         19    direction we're going -- if we really exploit

         20    best practices and have some sense of what

         21    works and works well -- because people have

         22    demonstrated it works well -- it's going to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           274


          1    be more powerful than almost anything else we

          2    can do here.

          3              MR. FREEDMAN:  Bill, maybe I wasn't

          4    as clear as I could be in the presentation.

          5    I think that we did exactly what you're

          6    suggesting that we do, that we did look at

          7    the best practices of each of the groups

          8    within EIA and organizations outside of EIA

          9    to draw upon that to come up with the

         10    functional requirements document.

         11              In terms of whether the computing

         12    ought to be centralized or distributed in

         13    terms of costs and efficiency, I'm not

         14    necessarily the best one to give you

         15    statistics on that, but from what I have seen

         16    and from what we have seen looking at other

         17    systems, there are some efficiencies to be

         18    gained not to have to support multiple

         19    platforms and multiple systems, so that when

         20    there is a change that needs to be made or

         21    software modifications, it doesn't need to be

         22    made once in coal and once in electricity.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           275


          1              But in terms of looking at best

          2    practices and integrating those things and

          3    taking the work that's already been done and

          4    integrating that into the functional

          5    requirements of the IDC, I think that's

          6    exactly what the team did, as opposed to

          7    saying we're throwing everything out and

          8    we're starting over again.  In fact, we

          9    purposely did not do that.  We built on

         10    what's been done in electricity, what's been

         11    done in coal, greenhouse gases, and the other

         12    statistical agencies.  So I'm sorry if that

         13    wasn't as clear as it could be.

         14              Please, Audrey.

         15              MS. CORLEY:  I'm Audrey Corley.

         16    Almost everything that you named are

         17    requirements that we have in the functional

         18    requirements document:  The easily accessible

         19    user's guide, electronic reports for

         20    performance statistics, non-respondent early

         21    reminders through mass e-mail notification,

         22    looking at best practices from BLS, from



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           276


          1    internally -- we went to Agriculture, we went

          2    to the Census Bureau, saw two different types

          3    of Internet data collection tools at the

          4    Census Bureau, and we came up and saw

          5    CNEAF's.  Everything that you name, with the

          6    exception of the computer system, we have,

          7    and it's going to be in the document as a

          8    requirement.  So we've been doing this for

          9    about a year and a half.  And we come from

         10    all different areas in EAI, so we definitely

         11    took into account everything that you're

         12    saying.

         13              MR. FREEDMAN:  And in fact, when

         14    you see -- if you go through the detailed

         15    slide, and then once the requirements

         16    document is actually written, I suspect that

         17    you will recognize many of the things in

         18    those requirements that exist in the coal

         19    system right now, because the coal and

         20    electric systems are pretty well developed

         21    and have a pretty long track record now --

         22    what, two or three years -- with



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           277


          1    electricity --

          2              MR. McARDLE:  Four.

          3              MR. FREEDMAN:  -- four years for

          4    what worked and what didn't work.  And we're

          5    not trying to redo those.  It's pretty clear

          6    that some of the things that are working

          7    there work real well.  And some of the best

          8    practices work well, and some of the things

          9    where CNEAF got off to a little bit of a

         10    rough start served to inform the discussion

         11    about how the requirements ought to be

         12    different.  Betty --

         13              MS. BARLOW:  Stan didn't give you

         14    any details about the common look and feel,

         15    but we really think that this is an important

         16    point for future surveys in an IDC in EIA for

         17    this reason.  We have a fair number of

         18    respondents who report on more than one

         19    survey in more than one area, too, more than

         20    one energy area.  And we feel it's important

         21    that the respondent, first of all, always

         22    knows he's at EIA and that he doesn't have to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           278


          1    learn over again when he goes to his second

          2    survey what he learned on his first survey.

          3    So we want some commonality.  We're not

          4    trying to say they're all going to be exactly

          5    alike -- and Stan did make that point --

          6    that's very important for us.  There's a lot

          7    of room for the survey manager to make

          8    choices.

          9              However, we do think that there

         10    should be common structure and that we have,

         11    for example, a common look to the buttons, a

         12    common set of buttons so the respondent

         13    always knows what it means from one survey to

         14    another about the word "submit," say, or the

         15    word "cancel."  He knows what that means, he

         16    doesn't have to relearn it for the second

         17    survey.  We felt that these ideas were

         18    extremely important, and that's one of the

         19    good reasons why we think that this IDC is a

         20    good idea, in addition to saving development

         21    costs, which is what Paula mentioned.

         22              MR. SINGPURWALLA:  How many times



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           279


          1    could a responder be hit by a survey within a

          2    year?  Like, how many times am I going to

          3    have to log in, typically?

          4              MS. BARLOW:  Well, weekly is 52.

          5              MR. SINGPURWALLA:  Weekly.

          6              MS. BARLOW:  We have some weekly

          7    surveys.

          8              SPEAKER:  Some surveys are weekly,

          9    some are monthly, some are annually.  The one

         10    I work on is quadrennially.

         11              MR. SINGPURWALLA:  Yeah, that's the

         12    one.

         13              SPEAKER:  So it's kind of a

         14    significant job if you're being hit with a

         15    weekly survey.

         16              MR. FREEDMAN:  Also you need to

         17    appreciate the fact that an electric power

         18    generator might get two or three electricity

         19    surveys, and some of those might be monthly,

         20    and some of them might be annual, so there

         21    could be 15 or 20 times a company -- not

         22    necessarily the same respondent filling in



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           280


          1    each form -- but the company have to submit

          2    some survey instrument to EIA.

          3              MR. SINGPURWALLA:  Can the IDC,

          4    based on my common sign-in name and password

          5    -- and I don't even know if this is the right

          6    term for it -- push data the other way so I

          7    don't have to type in my address all the

          8    time?

          9              MS. BARLOW:  Yes.

         10              MR. McARDLE:  Yes, we can do that.

         11              MR. FREEDMAN:  That functionality

         12    is a requirement if the survey wants to do

         13    that.  Many of EIA's surveys populate the

         14    form -- back in the old days, when they were

         15    printed and sent out -- with historical

         16    information that the respondent would simply

         17    update or change, especially when it comes to

         18    contact information, street address, phone

         19    numbers, and that kind of thing.  So yes, the

         20    functionality will be there to populate the

         21    form when the respondent pulls it up.

         22              MR. WATSON:  A part of that is when



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           281


          1    the respondent gets to that page and they see

          2    that information, if that's a new person

          3    who's been hired by the firm, and they don't

          4    see their name there, and their name needs to

          5    be there in the existing systems and in this

          6    system that's being designed, that would be a

          7    functionality for somebody to enter a screen

          8    and provide a change to that information,

          9    which would then be put into the system, so

         10    that the next time they logged on, that

         11    information would be updated to them.

         12              MR. SINGPURWALLA:  I don't know the

         13    surveys all that well, but I'm thinking that

         14    they probably have different sections in

         15    them, in one survey.

         16              MR. WATSON:  They do.

         17              MR. SINGPURWALLA:  Have you guys

         18    gotten to the point where you figure out when

         19    you log in to something, is there, like, a

         20    home screen for the user?

         21              MR. WATSON:  Yes.

         22              MS. BARLOW:  The first screen that



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           282


          1    we envision that they would see -- after the

          2    login, of course -- is a screen that gives

          3    them all the access and allows them to update

          4    their account profile, like contact

          5    information, and then gives them access to

          6    whatever surveys they should have access to.

          7              MR. SINGPURWALLA:  Oh, and, like,

          8    the name and address form, whatever, if they

          9    want to make changes, like I said,

         10    basically --

         11              MS. BARLOW:  Right, yeah.

         12              MR. WATSON:  But then once you're

         13    into the form -- and I guess this would be

         14    carried over to whatever the committee has or

         15    team has designed here -- many of the IDC

         16    systems have tabs at the top, like you see in

         17    a lot of Windows applications, and so you're

         18    tabbing from one part of the form to the

         19    other.  And people have talked about the

         20    payoff here in terms of self-editing, there

         21    is a -- call it the process tab, which you

         22    sort of hit at the end, and it will give you



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           283


          1    a tally on how many things have thrown up

          2    edit flags.

          3              So there would be a place there

          4    that you would click to see all the edits,

          5    and you could either say, oh, I made a

          6    mistake and go back and correct the data; or

          7    the data are correct, and you enter an edit

          8    override comment.  Now, that edit override

          9    comment is captured electronically and then

         10    somebody on the federal or contract staff on

         11    the receiving side sees that in an electronic

         12    report and most of the time, these days,

         13    people put in reasonable answers, and so you

         14    just say, okay, that's the end of it.  The

         15    previous paper system, you had to call that

         16    person and get them on the phone and clear

         17    that edit with them.

         18              And just to give you an idea about

         19    what's happening, on one of the surveys that

         20    I run, on the old paper system, in an annual

         21    cycle, we would have roughly 800 telephone

         22    calls of the type I'm talking about to clear



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           284


          1    an edit because it came up on a paper form.

          2    We got the paper form, we couldn't talk to

          3    the person except by calling them on the

          4    phone.  Now, they clear that -- and there are

          5    a few that we have to follow up on; we follow

          6    up on roughly 80 of them.  So it's reduced 90

          7    percent of the effort, and that's a

          8    tremendous payoff to the assistants.

          9              MR. SINGPURWALLA:  Have you guys

         10    quantify it like how much this is going to

         11    save at all?

         12              MR. WATSON:  I have attempted to,

         13    but I don't know ---- share with anybody.

         14              MR. SINGPURWALLA:  Then it's

         15    probably pretty good.

         16              MR. WATSON:  It should be

         17    substantial at some point.  I think the

         18    important thing to keep in mind, here, of

         19    course, is these systems are sort of building

         20    to the point where they're fully implemented,

         21    and we have a good database of e-mail

         22    addresses and all of that stuff that you need



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           285


          1    for good electronic systems.  So it takes a

          2    little while to get up the curve.  But I

          3    think when you get up the curve -- my

          4    expectation is that we ought to be able to do

          5    what we were doing previously with a quarter

          6    of the money and staff.  That's sort of what

          7    I see happening here.

          8              MR. SINGPURWALLA:  My last question

          9    would be, did you speak to end users as well

         10    when getting it ------?

         11              MR. WATSON:  You mean respondents?

         12              MR. SINGPURWALLA:  Yeah,

         13    respondents.

         14              MR. WATSON:  No, we haven't.  There

         15    is a requirement during the development

         16    effort, however, to go out and do user

         17    testing with respondents, and EIA has

         18    actually done that with the shop program with

         19    Paula and greenhouse gases, where we've

         20    actually gone out with a laptop and gone

         21    through -- well, and we did it with the

         22    electricity markets as well, where we've



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           286


          1    actually gone out and done user testing and

          2    gotten feedback in terms of navigation, look

          3    and feel, understanding what the buttons are.

          4    And as we begin to develop these pieces,

          5    we'll want to test that out to make a

          6    determination at that point where the layout

          7    ought to be.

          8              The other thing that we have found

          9    from previous testing that -- again, I didn't

         10    go over any of the details of the

         11    requirements because it would take a couple

         12    days to go over all of the specifics, but one

         13    of the big things that we always found when

         14    we talked to respondents from the user

         15    testing point of view was the issue of

         16    notification, submission, and then being able

         17    to save the information in some way that they

         18    could share it with others in the company,

         19    either for verification before the submission

         20    takes place, or as a historical record.  So

         21    those functional requirements are also in the

         22    system.  People are very unsatisfied with



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           287


          1    closing a browser and not getting a

          2    notification back that the data actually

          3    arrived at EIA.  Even though it may have

          4    arrived 20 minutes before when they moved

          5    from one screen to another, unless they see

          6    something in front of them -- I mean, I'm the

          7    same way.  Unless something comes back and

          8    says, thank you, I have it --

          9              MS. FORSYTH:  With a confirmation

         10    number.

         11              MR. FREEDMAN:  Yeah, exactly.  So

         12    that kind of functionality -- we already know

         13    from testing and talking to users, but I

         14    think there are many things that we will find

         15    out about the forms.  I also think that that

         16    testing informs design process.  There is not

         17    a requirement -- and I have a personal

         18    opinion about this that I'm not sure everyone

         19    on the committee shares -- but I don't think

         20    that there ought to be a requirement that the

         21    Internet form replicate the paper form.

         22              Now, there are people who do not



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           288


          1    agree with that, and it's a legitimate

          2    debate.  But I think the testing will help

          3    inform that process, and certainly the survey

          4    manager will be able to choose -- some of the

          5    CNEAF forms work very well with tabs because

          6    there are different schedules, but there are

          7    other forms that may work better in a more

          8    linear top-to-bottom, depending on whether

          9    there were skip patterns and things like

         10    that.  We didn't want to mandate everything

         11    had to be in tabs or schedules, and there are

         12    things you can do on the Internet or

         13    electronically that you can't do on a piece

         14    of paper, and again, that may change the

         15    design of the form.

         16              MS. PEARSON:  Tina Pearson, NEIC.

         17    To follow up on that, I have a question --

         18    this is my first week here, so bear with me

         19    if it's somewhat nave.  The electronic forms

         20    do not replace the paper forms -- do they

         21    coexist or can you go completely electronic?

         22              MR. FREEDMAN:  This is kind of an



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           289


          1    interesting paradox.  John and Paul, help me

          2    out here if I get the laws wrong.

          3                   (Laughter)

          4              MR. FREEDMAN:  There is an

          5    e-government act that encourages federal

          6    agencies to allow people --

          7              MR. LORENZ:  Paperwork Reduction

          8    Act.

          9              MR. FREEDMAN:  Paperwork Reduction

         10    Act, thank you -- to respond to them

         11    electronically, and that's a good thing, and

         12    that's the direction we want to move in.  On

         13    the other hand, when you clear a data

         14    collection through the Office of Management

         15    and Budget, you have to allow a company to

         16    fill it out on a piece of paper, put it in an

         17    envelope, and mail it back to you.  You

         18    cannot force them to do it electronically.

         19    So Audrey's raising her -- is there an

         20    exception, Audrey?  Correct me if I'm wrong.

         21              MS. CORLEY:  No, I don't have an

         22    exception, I have a comment.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           290


          1              MS. PEARSON:  So you do both.  You

          2    do both.

          3              MR. FREEDMAN:  Well, we have to

          4    have the capability to do both, to have

          5    something faxed, to e-mail it in.  There have

          6    been offices here at EIA that have strongly

          7    encouraged and put procedures in place to

          8    make sure that their respondents will fill

          9    data in over the Internet, and it has met

         10    with a pretty high degree of success.  But if

         11    somebody were to write to their Congressman

         12    and say, I want to mail this form back to

         13    EIA, four months later when we get it through

         14    the X-ray machine with the anthrax, we'll get

         15    the form.  Audrey, go ahead, please.

         16              MS. CORLEY:  I just wanted to give

         17    an example.  We have one survey we conduct

         18    annually -- I'm in Oil and Gas, we conduct 32

         19    surveys.  We have respondents that will write

         20    to you who -- especially on a crude oil and

         21    gas production survey -- and they say, we

         22    don't have a fax machine, we don't have a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           291


          1    computer, that's the way we like it, send us

          2    the paper.  And you don't want to tell them,

          3    no, you -- we don't want to end up not giving

          4    them any way to report, so we send them the

          5    paper.  But what we first try to do is e-mail

          6    them -- but, of course, they don't have

          7    e-mail.  So there's always at least -- in the

          8    short run, there are going to be a number of

          9    companies that you still have to mail the

         10    forms out to.

         11              MS. BARLOW:  What is it,

         12    percentage-wise?  Do you have a notion?

         13              MS. CORLEY:  Okay, Paula, I -----.

         14                   (Laughter)

         15              MS. WEIR:  In oil and gas, I

         16    believe we've been using some other

         17    electronic methods that -- the Excel

         18    spreadsheets have been --

         19              MS. BARLOW:  Yeah, it's all

         20    electronic.

         21              MS. WEIR:  -- the most successful,

         22    yeah, and our predecessor, -----.  But it



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           292


          1    does vary.  CNEAF has certainly the highest

          2    Internet of all --

          3              MS. BARLOW:  Well, I didn't mean

          4    Internet, I just meant electronic versus

          5    paper.

          6              MS. WEIR:  Any electronic.  I think

          7    they're still ahead --

          8              MS. CORLEY:  I think the surveys --

          9    it's 50-50.

         10              MS. BARLOW:  Really?

         11              MS. CORLEY:  A couple, like the 23,

         12    which is an annual survey, many, many more

         13    people still mail in.  But for the monthly

         14    surveys and for the weekly -- shoot, for the

         15    weekly, it might be 70, 80 percent.

         16              MS. WEIR:  It varies by the

         17    frequency that the survey's conducted, as

         18    well as the population.  We have the gasoline

         19    and diesel -- there's no electronic other

         20    than telephone.  I mean, it's a CATI system,

         21    but it's never going to be an Internet system

         22    because when you're sitting at the gas pump,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           293


          1    you can't send your data through Internet

          2    right now.  Maybe in the future, they'll all

          3    have something like that, but right now, it's

          4    not so -- that's why it varies so much by

          5    survey.

          6              But we believe -- with all this

          7    multimode, is what we usually call it --

          8    there's multimodes of going on -- we see the

          9    Internet as probably the most important in

         10    development, and we also see it as the point

         11    at which it can all be integrated no matter

         12    how it comes from to start getting all this

         13    stuff together in one place for better

         14    performance measures to design our survey

         15    instruments, in whatever form it's in, better

         16    because we will have that information out in

         17    one place to do it regardless of multimode.

         18    But we're trying to think way out there in

         19    developing this.

         20              MS. PEARSON:  Could you talk a

         21    little bit about what to extent, if at all,

         22    the feeling is in the short term or the long



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           294


          1    term how much the IDC would feed the public

          2    information, dissemination mechanism that

          3    you've got, be it Web, print, e-mail, those

          4    kinds of things?

          5              MS. BARLOW:  This is not

          6    dissemination at all.  It's just a

          7    collection --

          8              MS. PEARSON:  Right, I understand.

          9    This is just collection --

         10              MR. McARDLE:  Actually, I can

         11    address that.  There's actually another EIA

         12    committee called the Database Management

         13    System Commonality Team that's also reporting

         14    to the EIA's Strategic Plan Goal 14, so we're

         15    addressing that issue, and I think Nancy

         16    Kirkendall is trying to set up a briefing for

         17    you on -- I'm a member of that team too, as

         18    is ------, but he's not here.  But we're

         19    addressing that issue as well, but this is

         20    just designed for the data collection aspect

         21    of it.  The back end and the dissemination is

         22    where the database management systems come



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           295


          1    in, and that's a separate step along the way.

          2              MS. WEIR:  Stan?

          3              MR. FREEDMAN:  Go ahead, Paula, I

          4    just --

          5              MS. WEIR:  Separate, but related.

          6    We also recognize that as the data come in

          7    the door, there are opportunities that only

          8    exist as the data come in the door.  The

          9    whole idea of metadata as it passes through

         10    to dissemination, we think is very important,

         11    so we're trying to hold that concept at the

         12    same time, even though it's not a

         13    dissemination system, it is the beginning of

         14    the throughput of information.

         15              MS. FORSYTH:  Stan?

         16              MR. FREEDMAN:  Please.

         17              MS. FORSYTH:  You asked about

         18    potential incentives to respondents, and so

         19    while you were talking, I was making a list.

         20    And I'm a nave observer, so you guys

         21    probably already know these -- well, you can

         22    handle this --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           296


          1              MR. FREEDMAN:  Oh, okay, I was

          2    going to put my -- okay.

          3              MS. FORSYTH:  So I'm nave, and you

          4    probably already know this because some of

          5    the advantages or some of the incentives are

          6    already implicit to some of your paper users,

          7    but you're suggesting offering technical

          8    assistance, and so the ability to get

          9    tailored technical assistance like, I'm

         10    having a problem with Question X.  Either

         11    through electronic communication or through a

         12    telephone conversation -- that's a benefit

         13    that's not easy to get when you're doing a

         14    paper form, it's just not as automatic or not

         15    as interactive.  You were talking about

         16    feedback on status, and I think that's not

         17    something that you get right away from a

         18    paper form, certainly not from a mailed paper

         19    form.  So we've received your submission,

         20    well, that's a benefit to the respondent,

         21    compared to a mail-back.

         22              Maybe with a fax, "fax received" is



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           297


          1    sufficient, but I think that feedback on

          2    their status is an incentive.  We talked this

          3    morning in the talk -- I think it was Bob

          4    Adler's talk -- about the ability that an

          5    Internet data collection system gives you to

          6    be flexible with reporting units.

          7              I understand that some of your

          8    respondents now have their forms ready to go,

          9    but you will always have new respondents who

         10    will have to convert themselves to your fixed

         11    units or they could use your IDC and select

         12    their own units for reporting and have the

         13    form do the computations required, so I think

         14    that's a potential benefit, the flexibility

         15    it gives you in reporting.  Queries to their

         16    own data, we've already talked about.  I

         17    think that's useful not only for editing

         18    purposes, but maybe they want a historical

         19    record or maybe they want to go back and get

         20    trends over time, and so --

         21              MR. FREEDMAN:  One of the things --

         22    just to interrupt you quickly, because it is



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           298


          1    my favorite line form the cognitive

          2    interviews that we do, respondents will

          3    always say to us, I take great store in what

          4    I reported the last time.  So very often,

          5    they fill out the forms by looking at what

          6    they reported last reporting period --

          7              MS. FORSYTH:  Well, this makes

          8    their submitting burden so easy.  They just

          9    pull up the last form and --

         10              MR. FREEDMAN:  Right.  There are

         11    some people who look at that as a

         12    disadvantage -- yeah, I understand that

         13    --------- it is a two-edged sword, without a

         14    doubt.

         15              MS. FORSYTH:  A related benefit is

         16    reports of their own data.  You could

         17    formulate reports that they can't necessarily

         18    query.

         19              MR. FREEDMAN:  That's what Paul was

         20    talking about.

         21              MS. FORSYTH:  Right, right.  The

         22    potential for reduced burden -- you already



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           299


          1    mentioned this -- from follow-up calls

          2    because of errors or inconsistencies, the

          3    ability to link to a dissemination system

          4    kind of seamlessly -- and then someone

          5    already mentioned the -- again, our

          6    relatively seamless opportunity for

          7    electronic data to transfer, which I think is

          8    probably really appealing to some of your

          9    larger suppliers.  I mean, they're probably

         10    there and ready.

         11              MR. FREEDMAN:  Many large

         12    respondents have said to us, what we'd really

         13    like to do is push a button on the 5th of the

         14    month and transmit the data.

         15              MS. FORSYTH:  Right, right.

         16              MR. FREEDMAN:  That functionality

         17    will be built into the system.  We'll have to

         18    specify the format, of course, so they won't

         19    have to use the IDC to key in numbers --

         20              MS. FORSYTH:  Right, right.

         21              MR. FREEDMAN:  -- they'll just

         22    simply submit the data from their database.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           300


          1              MS. FORSYTH:  Right, right.  So

          2    those are the main things I thought of, and

          3    maybe other people have ideas.

          4              MR. FREEDMAN:  You actually raised

          5    something about assisting respondents.  Stan

          6    Kaplan is going to talk in a few minutes

          7    about the EIA 920, which is a combined heat

          8    and power survey, and some respondents on

          9    that survey are very problematic because of

         10    the complexities of figuring out the number

         11    they have to report to EIA; it's not

         12    something that's generally recorded.  Stan

         13    said to me in a offline conversation one day,

         14    wouldn't it be nice if we had an IDC where

         15    people would put in some input parameters and

         16    the IDC would compute the number for them

         17    that they want to submit to EIA, and then

         18    they could look at it and verify it and --

         19    again, there are dangers there as well, but

         20    that's a potential use of an IDC technology

         21    to improve response, so that's a point well

         22    taken, although we don't have that in the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           301


          1    functional requirements right now, it's --

          2              MS. FORSYTH:  Right.  And that's

          3    survey-specific.

          4              MR. FREEDMAN:  Yeah.  Oh,

          5    absolutely.

          6              MS. JENNINGS:  I just have one

          7    quick thing I'd like to say.  Alethia

          8    Jennings.  I know we didn't want to get into

          9    details, and we were trying to keep this

         10    brief, but the one thing I do want to

         11    mention, because of the fact that Tom Lorenz

         12    and Bob Adler presented the MECS to you

         13    today, the MECS is a very long survey.  How

         14    many pages is it physically?

         15              MR. LORENZ:  On average, between

         16    the three forms, it's 60 pages.

         17              MS. JENNINGS:  It's 60 pages.

         18    That's a very long survey, and a survey that

         19    long -- it's a value to the respondent to

         20    have one of the feature that we have built in

         21    as a requirement here, which is a progress of

         22    measure.  And we've talked about the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           302


          1    respondent being able to not only save the

          2    data, but to see how far along they are in

          3    the respondent process, the submission

          4    process, before they save the data, and

          5    that's very useful, more useful than on a

          6    short form.  But something like the MECS,

          7    that's very valuable, and that saves the

          8    respondent some of the burden as well.  So I

          9    think that's an important point we need to

         10    bring out.

         11              DR. HENGARTNER:  Last comment.

         12              MR. WATSON:  A question.  Stan,

         13    could you give us some idea about sort of

         14    what the process is now with the team going

         15    forward?  Will there be a rollout, are you

         16    going to come around and sort of present

         17    results to different groups?

         18              MR. FREEDMAN:  What I hope will

         19    happen is that we will finish up the report

         20    in a week or two, we will give it to our

         21    champion, Nancy Kirkendall, she will think

         22    it's wonderful, give it to Howard



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           303


          1    Gruenspecht, who will give us a half a

          2    million dollars to begin to build the system.

          3              But in reality, what will happen is

          4    we'll present the report to the Goal Four

          5    Subcommittee, functional requirements, then I

          6    think they will vet that through the

          7    strategic planning process.  They will

          8    probably review it, and then if they want us

          9    to, we will roll it out into the rest of the

         10    organization, give them the opportunity to

         11    comment on -- I think Nancy maybe can talk

         12    about that a little bit more once we give the

         13    report to the Goal Four Subcommittee for the

         14    IDC.

         15              MS. KIRKENDALL:  Well, we have to

         16    decide what to do with it.

         17              MR. FREEDMAN:  Okay.  I think,

         18    Bill, the intention is to get the buy-in from

         19    the Strategic Planning Committee and then

         20    look towards an application for

         21    implementation to see if, in fact, we can

         22    develop a modular system that will work in



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           304


          1    concert with what we have now.  But I think

          2    the key issue there is funding.

          3              DR. HENGARTNER:  Thank you very

          4    much.  We have a 10, 15-minute coffee break.

          5                   (Recess)

          6              DR. HENGARTNER:  Okay, ladies and

          7    gentlemen, let's resume the last leg of

          8    today's meeting.  I'd like to recognize

          9    Darius Singpurwalla, who will give us the --

         10    is he here?  Yes, he is.  Okay, just checking

         11    -- who is going to give us a rundown of what

         12    we've learned during the functional

         13    requirement for EIA Internet data collection

         14    systems.  Darius.

         15              MR. SINGPURWALLA:  Sure.  Thanks,

         16    Nick.  I want to give a big round of applause

         17    to Stan, who came up and gave a great

         18    presentation --

         19                   (Applause)

         20              MR. FREEDMAN:  Thank you, but that

         21    round of applause goes to the committee who

         22    did all the work.  I guess I want to give you



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           305


          1    a couple PowerPoint clients.

          2              MR. SINGPURWALLA:  There was the

          3    rumor that he's sick for two weeks, but I

          4    just heard he was actually at the beach,

          5    so --

          6                   (Laughter)

          7              MR. FREEDMAN:  I was sick at the

          8    beach.

          9              MR. SINGPURWALLA:  Yeah.  So that's

         10    good.  One thing that we've been talking

         11    about the past day -- and you know we got a

         12    lot more -- another presentation on it this

         13    afternoon, which was just as good as the

         14    first one -- is the whole Internet data

         15    collection system that we're moving to.  The

         16    EIA's putting out a bunch of surveys,

         17    obviously, and for lots of really good

         18    reasons they want to move away from either

         19    the paper-pen method of responders filling

         20    out the surveys and then mailing them back,

         21    or, as we heard about today, a technological

         22    step up from that where they do it in Excel



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           306


          1    and then send it back on either, like, a CD

          2    or a diskette, and then the third evolution

          3    that what they're going for here is they want

          4    to do something on the Internet where the

          5    responders will turn on their computers and

          6    then log into the Internet data collection

          7    system, a screen's going to pop up, and they

          8    can fill out various information about

          9    themselves or various surveys that they apply

         10    to that they want to fill out.  Then that

         11    information is pushed back there to the EIA

         12    or wherever they're going to be storing their

         13    Oracle database, and that will be stored in

         14    the database, and the database back here can

         15    be used for reporting or whatever.  So that's

         16    kind of the direction they're taking it right

         17    now, which is good.

         18              It's a good subject for me to hear,

         19    because this is kind of something that I do a

         20    lot in my own job, and I think one of the

         21    things that's really impressing me about the

         22    way that this committee's doing it is that



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           307


          1    they're getting the requirements and

          2    everything that they want to do and the

          3    functionalities that they want to do, they're

          4    laying those out up front.

          5              The focus of this committee, as

          6    Stan said in the beginning, is not to make

          7    the screens look pretty or -- it didn't seem

          8    to be like working with the actual database.

          9    They want the functionalities in there as to

         10    how the IDC is going to work, and that's a

         11    really big thing that you need to do, and

         12    it's often a step that I know from experience

         13    gets skipped, and then at the end of the day,

         14    there are going to be a lot of people

         15    scrambling around, being, like, why didn't we

         16    have this functionality, why didn't we have

         17    that it's the project that I'm working on

         18    currently, and I've already got about 10

         19    voice mails of people demanding why something

         20    hasn't been put into the data input part of

         21    the database that I'm working on.  And it's

         22    not my fault --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           308


          1                   (Laughter)

          2              MR. SINGPURWALLA:  -- but it's more

          3    related to the planning, that people didn't

          4    see it up front, and so I think it's great,

          5    and I think it's a super-important first step

          6    that a lot of people ignore.  I thought one

          7    of the most useful handouts that we got that

          8    I don't think came on the website with the

          9    agenda on it is this data collection flow.

         10    So, I mean, if we could get copies

         11    distributed to the rest or put this up on the

         12    thing, I think it's great because it really

         13    gives -- and skimming through it really

         14    quick, just the first page, you can really

         15    see what the IDC is planning on doing and

         16    what's its functionality is actually going to

         17    be.  We talked about it a little bit more in

         18    depth, and I kind of mentioned some -- it's

         19    in the beginning of the recap -- of a user

         20    being able to log on and that one specific

         21    user's going to have access to all the

         22    surveys that he's going to do.  He'll pop up



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           309


          1    right up to a home page for his own self, and

          2    then he can just start clicking and going

          3    nuts and doing whatever he or she needs to do

          4    to get their job done.  And in a situation

          5    where you're filling out a weekly survey --

          6    which I didn't know existed -- like, that's

          7    going to be really helpful to do, like, you

          8    don't have to type on your name and address

          9    the whole time and even when we had some

         10    paper-and-pencil surveys going around, they

         11    kind of rectified that problem.  It's good to

         12    know that this IDC is going to have some of

         13    the functionality to do that.

         14              There were two presentations given

         15    to us.  One is the one that Stan went over,

         16    the functional requirements for EIA's data

         17    collection system, which is kind of a good

         18    overall view of how it works.  And then the

         19    second one, which is kind of the one that I

         20    burned through last night, goes in a lot more

         21    detail about some of the functionality that

         22    the IDC is going to have, and as someone who



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           310


          1    works in this industry a lot, everything that

          2    I would expect and want to be there is there:

          3    The whole graphical user interface, the

          4    single sign-on that we talked about.  There's

          5    going to be a common look and feel between

          6    each of the different surveys across the

          7    board, so if you're filling out one survey,

          8    you kind of get used to that feel, and then

          9    maybe your job responsibilities grow, and

         10    they give you the other survey to fill, you

         11    click on that, it's not going to look like

         12    apples and oranges, I think -- two really

         13    different things.  That's what I'm going for

         14    here.  So I'd also recommend popping through

         15    the second presentation as well to get a

         16    better idea on what the specific

         17    functionalities are.  I think that was

         18    useful, too.

         19              Some of the other things that the

         20    IDC is going to be able to do is, it's going

         21    to be fully centralized, so I believe that

         22    that means at the end of the day all those



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           311


          1    surveys are going to be in one Oracle table.

          2    Centralized core structure -- these are from

          3    my notes yesterday, the single sign-on, and

          4    all the other functionalities.

          5              The last thing that I wanted to say

          6    that I thought was really good is that

          7    Barbara came up with a really nice list of

          8    incentives that businesses can actually use

          9    to be going ahead in responding to this.

         10    It's, like, we -- this is my third meeting --

         11    we talk about surveys a lot, and incentives

         12    always seem to come up.

         13              And this is the first time that I

         14    really thought about it on a business level,

         15    instead of something like a household level,

         16    because you're not going to send $25 to a

         17    business; that's like a meal reimbursement

         18    for them when they worked late on Saturday,

         19    that's a cakewalk.  But Barbara gave -- I

         20    managed to write down five different

         21    incentives, the whole technical assistance

         22    thing.  It's information.  Businesses use



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           312


          1    this information, and they're going to use

          2    this information to make decisions, and those

          3    decisions are either going to make them or

          4    save them a lot of money, which is better

          5    than the $25 gift certificate to Pizza Hut.

          6              Flexibility on reporting is another

          7    incentive, and then queries to their own

          8    datasets if they've been able to input this

          9    information, and now they can slice it and

         10    dice it in any way they want.  And, again,

         11    reporting.  So those are some of the

         12    incentives that Barbara touched up on.

         13    Again, I think it's so important to mention

         14    that they're doing all the stuff up front as

         15    opposed to doing it on the fly.  And sending

         16    some programmer in there, and he builds one

         17    interface, and then you keep with the

         18    back-and-forth, the back-and-forth.  I think

         19    when you build tools in general, any type of

         20    software tool, it's the up-front planning

         21    that's really going to save you time and

         22    money down the road, and that's the biggest



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           313


          1    thing that I got from this, is that they're

          2    really on the ball with that.  So it's

          3    impressive.  I'm done unless --

          4              DR. HENGARTNER:  Thank you, thank

          5    you.  Does anyone want to add anything to

          6    Darius' very extensive summary of our

          7    discussions?  Yes?

          8              MR. HILL:  An offhand comment.

          9    Some of the surveys are weekly, some of them

         10    are every four years, so that's a huge

         11    difference --

         12              DR. HENGARTNER:  Walter, Walter --

         13              MR. HILL:  During the morning

         14    session, there was a comment that those

         15    surveys are done every four years, and there

         16    was a comment -- it was almost an offhand

         17    comment this afternoon that some of the

         18    surveys are weekly, so it seems to me that's

         19    a huge burden that's placed on the group to

         20    try to come up with surveys over that length

         21    of -- I mean, ------- for about 200 -- from

         22    one week to 200 weeks that they're working



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           314


          1    with.

          2              MR. SINGPURWALLA:  Yeah.

          3              MR. HILL:  It's a depressing task

          4    that they have.

          5              MR. SINGPURWALLA:  Oh, yeah.

          6              DR. HENGARTNER:  Anybody else wants

          7    to add to this discussion?  Okay, thank you.

          8    I'd like to move on to the feedback from the

          9    other break-out session on improving the SAGE

         10    petroleum refinery model, and I invite Tom

         11    Rutherford to give us that summary.

         12              MR. RUTHERFORD:  Yes.  John Staub

         13    gave us a very nice introduction and overview

         14    of the factors and the considerations that

         15    are sort of informing their attempts to

         16    improve upon the SAGE representation of

         17    refinery sector in the SAGE model.  This was

         18    discussed this morning.  SAGE is this

         19    recursive model that characterizes a

         20    16-region global model of demand and supply

         21    factors influencing a wide range of energy

         22    sectors, 42 energy products in total.  So



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           315


          1    it's a very large model for which the

          2    existing refinery sector is extremely

          3    simplistic and unacceptable for purposes of

          4    -- it violates simple physical laws, they're

          5    basically -- it just doesn't pass the laugh

          6    test is what Alan Mann would say.

          7              So what we're currently engaged in

          8    is a process of improving upon this model,

          9    which gives rise to considerations of

         10    different ways in which this could be proved.

         11    And the outline -- the paper talk this

         12    morning was very interesting in terms of its

         13    discussion of what the sort of the background

         14    policy dimensions were and what the range of

         15    factors are that influences the market for

         16    refinery capacity.  And particularly we

         17    looked at what's the utilization of funneling

         18    capacity has moved from 60 or 70 percent in

         19    the mid '80s up to over 90 percent globally

         20    in the last couple years.  So there's been

         21    big change in terms of structure the

         22    industry, and there's this awareness that



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           316


          1    expansions have to take place.

          2              Now, one of the things that's a

          3    challenge for working in this area is that we

          4    have this long history of chemical engineers

          5    using precisely the tools we're using for

          6    calculating economic equilibria have been

          7    used for years to do refinery engineering.

          8    So this process of -- the problem is that

          9    different crude types hold different

         10    products, and the way you combine them and

         11    recombine them and what types of processes

         12    you invest in will determine what comes out.

         13    What comes out includes things like heavy

         14    distillates, jet fuel, light gasoline,

         15    heating oil -- there's a whole range of

         16    products -- and how you combine the crude

         17    oils will determine what comes out and the

         18    process of designing a refinery involves

         19    choosing what products you want to make from

         20    what the crude grades are that come in.

         21              In terms of thinking about the

         22    model, it's a forward-looking -- or it's a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           317


          1    model that's thinking about the future, and

          2    we have a backdrop where -- and this was very

          3    nicely laid out in the talk, how the

          4    differences in what the range of products are

          5    that are consumed differ dramatically across

          6    countries.  In particular, the U.S. tends to

          7    be much more reliant on much lighter products

          8    -- diesel fuel and gasoline -- whereas China,

          9    other developing countries, have

         10    traditionally been much heavier users of

         11    heavy products.  But those are converging,

         12    and so we're going to see expanding demands

         13    for light products, and as a consequence,

         14    there'll be a need for large-scale

         15    investments globally in refinery capacity.

         16    And so those are the types of processes that

         17    they're hoping to be able to extend SAGE to

         18    deal with.  Their approach is relying on a

         19    long history of refining models that have

         20    been built at DOE, and it was going to focus

         21    on a fairly simplistic -- I would call it --

         22    refer to it as a reductionist model which



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           318


          1    basically comes up with a series of simple

          2    representations of the processes involved in

          3    doing refining, and then having the model

          4    choose how those processes are combined to

          5    produce given products.  So the investment

          6    decision in each period will be based on

          7    minimizing the cost of producing of these

          8    various sets of outputs.

          9              But what makes the whole process

         10    complicated is you have a set of outputs, you

         11    have a set of inputs, and the refining model

         12    provides an intermediary between those

         13    decisions, and so there's quite a bit of

         14    complexity there.  The challenge is to

         15    introduce sufficient complexity to

         16    characterize the decision process in how the

         17    market operates, but not too much because you

         18    don't want to make this unmanageable, and

         19    that's where there's a bit of a -- I think

         20    there was quite a bit of discussion about

         21    what the appropriate level of complexity is

         22    in the model.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           319


          1              In economics, there's always this

          2    tension between top-down versus bottom-up.

          3    If you build a bottom-up model, then you

          4    basically assert that you characterize all

          5    these various -- this menu of choices that

          6    the model has, and the model will then choose

          7    among this large menu and make decisions, and

          8    then at the end of the day, if somebody says,

          9    well, why does your model produce this number

         10    you can write down, you say, well, we

         11    installed the catalytic cracker in China in

         12    this year, and that's led to this result,

         13    right, so you can point to very specific

         14    details.

         15              The problem with that approach is

         16    if you try to keep it simple then you may not

         17    be happy with what the model spits out.  And

         18    so then you have a challenge having to deal

         19    with, how do we fix problems that arise in

         20    terms of the model not being plausible.  In

         21    other words, in the questions for the

         22    committee, one of the questions was, how do



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           320


          1    we -- several preliminary runs of the model

          2    demonstrated a problem in which it used too

          3    much butane to produce gasoline.  One

          4    solution is to include octane constraints,

          5    but the drawback is the model becomes more

          6    complicated.  So the problem with the

          7    bottom-up model is invariably, you keep

          8    returning to the model having to add

          9    constraints to make it pass the current set

         10    of requirements for plausibility, and so that

         11    leads at the end of the day to a model which

         12    is quite complicated.  You have a lot of

         13    things to explain about what you're doing.

         14    And so there's a bit of a -- we discussed --

         15    I thought the -- again, the discussion, as

         16    was this morning's discussion, was quite

         17    lively, and I think quite informative, and we

         18    didn't, I think, reach any agreement -- we

         19    agreed that -- together, I think the one

         20    thing we agreed on is that this is a darn

         21    tough market to model, and there are lots of

         22    challenges.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           321


          1              There were several suggestions for

          2    alternative approaches that might be simpler,

          3    sort of more top-down type approaches, but

          4    there was also discussions about what the

          5    virtues were of the approach that was being

          6    used.  I thought Phillip laid out quite good

          7    defense of the fact that they have a long

          8    history of doing this, and they're quite

          9    confident that they can make sense of this.

         10    But I think the jury is still out; you can't

         11    really judge the cake until you eat it.

         12              So I think that this is an

         13    interesting line of research, and I think

         14    there's quite a bit of interest in the

         15    project and how the model will be applied and

         16    what we can learn from it.  So I think that

         17    that -- I don't know if I've touched on

         18    everything.  I'll leave it to the other

         19    people who were there.

         20              Oh, the final thing I forgot to

         21    mention was that one of the interesting sort

         22    of challenges in the current round of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           322


          1    thinking, it depends on how you reflect on

          2    factors that have led to the lack of

          3    investment and refining capacity in the U.S.

          4    in the last decade, and if -- are there are

          5    different schools of thought about this, but

          6    one of the schools of thought we'd like to

          7    entertain is the idea that there might be

          8    continued restrictions or reluctance to

          9    invest in refining capacity in the U.S.,

         10    which would lead us to a very

         11    different-looking world and the world market

         12    maybe trade in refined products rather than

         13    trade in crude oil.  And then the question

         14    is, where do the refined products get

         15    produced, how do we characterize the factors

         16    that give rise to where refineries tend to

         17    get installed, and there was discussion about

         18    transport costs, and there's a way of penny

         19    switching.

         20              One of the challenges is basically

         21    how we model this whole process, and I'd say,

         22    again, I think the committee's feeling was,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           323


          1    this is a tough one to characterize, to

          2    explain concretely.  Clearly if the model

          3    produces forecasts that all the refinery

          4    capacity will be installed in Venezuela in

          5    the next 20 years, that this will be a

          6    difficult one to sell politically, but the

          7    model may choose that because that may be the

          8    cheapest way to do things.

          9              So there are some challenges, I

         10    think, in terms of producing a model which

         11    will provide forecasts which can be published

         12    in the AEO and will be referred to by large

         13    groups of people.  There are political

         14    ramifications, which are not part of the

         15    model, but I think that that's one of the

         16    challenges with this type of formulation.

         17              DR. HENGARTNER:  Thank you very

         18    much, Tom.  Does anyone want to add to this

         19    discussion?

         20              DR. FEDER:  I have a just a

         21    question or a comment.  Does the model

         22    produce just one outcome or does -----



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           324


          1    probability with the outcome that the

          2    refineries are going to be in Venezuela or

          3    something like that?

          4              MR. RUTHERFORD:  It's

          5    deterministic, but I guess we'd let Phillip

          6    answer it.

          7              DR. TSENG:  I think when we have

          8    refinery representation in a SAGE model, the

          9    choices will be -- we have 16 regions.  And

         10    in the next 25, 30 years, there will be

         11    additional refining capacity equivalent to

         12    about 45 million barrels a day processing

         13    capability.  And it has to be located

         14    somewhere.  And it could be Venezuela, or it

         15    could be Middle East, it could be in Asia or

         16    South America.  So the issue here is if we

         17    start the model with basic economic

         18    principle, then it could be, just like Tom

         19    mentioned, Venezuela.  And politically, it's

         20    impossible.  And then the issue will be, if

         21    we don't model it, how do we address the

         22    tradeoff crude oil imports and petroleum



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           325


          1    product imports.  So we like to get really

          2    comments from the committee as exactly what's

          3    the best way to handle those kind of issues.

          4              DR. FEDER:  If I can draw on my

          5    experience, it's a project I've been involved

          6    with for about a year on modeling of

          7    epidemics.  What one is trying to do is get

          8    both some scenario or scenarios, but also

          9    some sensitivity of the results to

         10    stochasticity in the model and assumptions

         11    that might affect the outcome, and like any

         12    statistical analysis involved, not just point

         13    estimates, but also standard errors and some

         14    kind of forecasting error.  So it would be

         15    good to allow the model to run multiple times

         16    with likely different parameters and see how

         17    sensitive the outcome is to that.  So if, for

         18    instance, a relative advantage of Venezuela

         19    over, let's say, the Middle East in terms of

         20    certain -- I'm sure I'm not saying it right

         21    -- but some ways, how would the outcome

         22    change, if it matters?  If it does not



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           326


          1    matter, then fine, but as a statistician, I'm

          2    a little bit uncomfortable with having a

          3    point estimate without a standard error

          4    attached to it.  So -- I mean, how much can

          5    we trust the output if you don't know how

          6    uncertain --

          7              DR. TSENG:  I think we can run

          8    scenarios, but another important issue is

          9    still -- the bottom line is if one region --

         10    like in North America -- there's investment

         11    in refining capacity, then at the end of the

         12    day, more refined products will have to come

         13    in to meet the end-use demand.  And that's

         14    kind of question that we believe will come to

         15    EIA maybe a few years down the road.  Because

         16    the product import will become more

         17    significant, and in general, the value of

         18    refined product is also higher than crude

         19    oil.

         20              MR. RUTHERFORD:  I should say that

         21    the discussion about Venezuela as the

         22    supplier, that was my interjection, just to



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           327


          1    make sharper what the issues were.  This was

          2    not something that was discussed in any

          3    document that I've seen at DOE.

          4                   (Laughter)

          5              DR. FEDER:  But it's a good hook to

          6    ask my question.  My point is only, suppose

          7    that we had some forecast on the future gas

          8    consumption of the average family sedan

          9    and/or SUV, and suppose that I put 39 in my

         10    model, and that gave me a certain prediction.

         11    I would like to run the model again with 37

         12    instead of 39 and see does that really change

         13    qualitatively, not quantitatively, the

         14    output, things like that.  So, sensitivity to

         15    input parameters is always an issue.  And

         16    that gives you a way of assessing variability

         17    in the outcome.

         18              DR. TSENG:  That's one way to run

         19    the model once it's set up.  It can be done

         20    easily.  We can do parametric analysis.

         21              DR. FEDER:  I mean, it's like

         22    weather forecasting models.  And in fact, if



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           328


          1    you remember, was it Ophelia or one of those

          2    hurricanes, they had four models -- and I

          3    think we talked of it last time -- that three

          4    models gave a projected course that went

          5    eastward and missing even Cuba altogether,

          6    and one projected that it would go to Maine,

          7    which was all the way north.  So that's the

          8    same thing as doing multiple runs of the

          9    model, you run slightly different models, and

         10    you look at the output, and then you say,

         11    yeah, I think this might be the future course

         12    of pricing and needs and so on, but I need to

         13    attach some probability to that outcome and

         14    say that will happen with us 70 percent

         15    probability, but 30 percent probability will

         16    have another outcome.  I may be asking for

         17    too much, I don't know, just a statistician's

         18    thoughts about model outcome.

         19              MR. CLEVELAND:  I think you're

         20    right, and think the types of information

         21    that they're trying to incorporate in the

         22    model would allow you to -- well, not



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           329


          1    necessarily assign error bars to data points,

          2    run simulations that could answer those types

          3    of questions, but I think one of the tangents

          4    we discussed was that a lot of the decisions

          5    that actually get made about where refineries

          6    are plunked down are not going to be based on

          7    the type of economic costs you're going to be

          8    able to model.

          9              There are different types of risk

         10    that oil companies, national or private, are

         11    going to face in different regions of the

         12    world, which are very difficult to quantify

         13    economically, in economic terms.  So the

         14    issue for EIA is going to be assume we can

         15    get a handle on the uncertainty of economic

         16    data -- how much do we want to get into the

         17    business of talking about risk associated

         18    with doing business at different parts of the

         19    world, which is really going to be one of the

         20    driving forces behind investments in these

         21    big capital-intensive projects.

         22              DR. HENGARTNER:  Any more



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           330


          1    questions, comments?  Okay, I'd like to move

          2    on to the next item on our agenda.  I'd like

          3    to welcome Stan Kaplan, who is going to talk

          4    about making adjustments to survey data when

          5    the collected data do not meet expectations.

          6              MR. KAPLAN:  This presentation is

          7    entitled "Making Adjustments to Survey Data

          8    When the Collected Data Do Not Meet

          9    Expectations," but an alternative title could

         10    have been "Good Intentions Gone Awry" because

         11    what you're going to learn is that we had a

         12    data problem, took steps to try and fix it --

         13    actually fairly sophisticated steps -- and

         14    ended up with a new problem probably as big

         15    as the original problem.  And in fact, I'm

         16    not going to speak a whole lot about the

         17    adjustment process itself, which you'll see

         18    is really fairly mechanical and I don't think

         19    all that interesting.  The interesting thing,

         20    I think -- you all can judge -- is what led

         21    to the problem and what steps we're

         22    considering to definitively fix it.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           331


          1              Now, just as a way of getting

          2    started, break in with questions any time.

          3    And also, there's a lot of detail in here,

          4    and I'm actually going to skip over a lot of

          5    these slides.  A lot of the detail here is

          6    for you to read in your leisure hours.  I'm

          7    happy to go through anything, but I'm not

          8    going to try and bog us down in all the

          9    minutiae which is discussed in the

         10    presentation.

         11              Where am I aiming at?  Doesn't

         12    matter.  Okay.  This is about electric power

         13    data, and one of the most key and fundamental

         14    pieces of data we collect is fuel

         15    consumption, fuel consumed to make

         16    electricity.  That's a very basic piece of

         17    information, how much gas, coal, oil, and so

         18    on is being consumed each month, each year,

         19    to produce electricity in the United States.

         20    And we collect that data from basically two

         21    types of respondents, two types of power

         22    plants:  One is traditional power plants,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           332


          1    that is, it's a plant which was built for one

          2    purpose, which is to make electricity, and it

          3    consumes fuel and makes electricity and

          4    that's what it does.  But there's a second

          5    kind of respondent, sometimes referred to as

          6    co-generators, more properly I guess as

          7    industrial-commercial combined heat and power

          8    plants, CHP plants.  And that's going to be

          9    the focus of the discussion today.  CHP

         10    plants are most typically industrial

         11    facilities, and their primary business is not

         12    making electricity, their primary business is

         13    making paper, making chemicals, making

         14    refined products.  There are commercial CHP

         15    plants -- these are most typically hospitals

         16    or other institutions which are producing

         17    electricity as well as space heat.  But

         18    they're primarily industrials, and one of the

         19    key things to keep in mind is that they

         20    produce electricity as a secondary business;

         21    their primary interest is consuming fuel to

         22    make what's referred to sometimes as process



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           333


          1    steam, other times as useful thermal output,

          2    or UTO.  And they use that steam for space

          3    heating or as part of manufacturing process.

          4    They use some of the steam to make

          5    electricity, they consume a lot of the power

          6    themselves typically, they may sell some of

          7    it on the grid.  But again, their major

          8    business is making paper, making chemicals,

          9    making widgets, whatever.  The electricity is

         10    a side light business for them.

         11              Now, these combined heat and power

         12    plants, one reason they exist is that they

         13    can be much more efficient and therefore much

         14    more economical than building a stand-alone

         15    power plant and a stand alone boiler to make

         16    steam for a process.  So there are any number

         17    of configurations for these plants, but

         18    here's a typical one:  Now, for a moment,

         19    just ignore this piece, okay?  So if you have

         20    a ordinary stand-alone power plant, the fuel

         21    goes in -- in this case, into a combustion

         22    turbine -- electricity comes out, but there's



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           334


          1    also waste heat, these hot exhaust gases,

          2    which, in a typical power plant, are wasted,

          3    they're vented to the atmosphere or otherwise

          4    just lost.

          5              If you could put back into your

          6    mental picture this piece up here, this is

          7    what makes it a combined heat and power

          8    plant.  Rather than these hot exhaust gases

          9    being wasted, the energy is captured -- in

         10    this case in what's called a heat recovery

         11    boiler -- so the otherwise lost heat is now

         12    being used to make steam, and the steam goes

         13    off to the process, which again, you're

         14    making paper, making chemicals, whatever.  So

         15    more of the fuel coming in at this end is

         16    ultimately recovered in something useful at

         17    the back end, either electricity or steam to

         18    accomplish the manufacturing process.  So

         19    these plants can be far more efficient than

         20    building two separate plants.

         21              However, it leads to a question,

         22    and this is an important question for EIA,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           335


          1    which is, how do we allocate the fuel?

          2    You've got fuel coming in, but you have two

          3    things coming out:  Electricity and process

          4    steam.  So the respondents to our surveys

          5    have to allocate some of the fuel as being

          6    used to generate steam and heat and some of

          7    the fuel being used to generate electricity.

          8    And this is the issue which has proven to be

          9    very difficult, one we've struggled with now,

         10    really, for years.

         11              Why is this difficult?  Okay, there

         12    are two reasons I would focus on.  The first

         13    is that when we ask people how do you

         14    allocate fuel to electricity versus fuel to

         15    process steam, we're asking them for a data

         16    element which normally they don't collect for

         17    their own purposes.

         18              When we ask someone how much total

         19    fuel do you consume or how much electricity

         20    do you produce, we're asking for a piece of

         21    information which is of interest to them

         22    because it's vital to their own cost



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           336


          1    accounting, to knowing how much revenue

          2    they're going to bring in.

          3              When we ask them how are you

          4    allocating fuel to electricity versus fuel to

          5    UTO, it's a question which, for the most

          6    part, the respondents simply aren't

          7    interested in.  They don't compute it for

          8    their own purposes.  So we're asking them to

          9    do something which is outside their normal

         10    course of business.

         11              Then to further compound the

         12    problem, we're asking them to compute

         13    something for which there is no standard

         14    methodology.  There's no theoretically

         15    correct single way to make the computation,

         16    and beyond that, there's no single way which

         17    it is typically done in the business.

         18              All right, so what has EIA done?

         19    For a number of years, we took the allocate

         20    fuel -- compute the fuel allocation by first

         21    determining useful thermal output.  Without

         22    going through all the calculation



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           337


          1    permutations, basically, we would figure out

          2    this piece -- fuel use to make steam -- we do

          3    collect this piece, the total fuel use --

          4    they certainly are interested in that and

          5    they give us that information -- then you do

          6    a subtraction; what's left over is fuel use

          7    generated electricity.  And we went to that

          8    approach only after careful consultations

          9    with industry where we were assured that this

         10    information was available.  Only it wasn't.

         11    And so we ended up getting lots of numbers

         12    coming in which just didn't make sense.  They

         13    produced plan efficiencies or planned heat

         14    rates which were just way out of bounds.

         15              So that just was not working; it

         16    was taking a tremendous amount of effort on

         17    the part of our staff to try to clean up the

         18    data through innumerable phone calls and

         19    e-mails.  By way of context, combined heat

         20    and power plants account for -- I think it's

         21    about five percent of total U.S. generation,

         22    but they have been consuming, and still are



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           338


          1    consuming, about a third of all the resources

          2    we put in to this area of data collection.

          3    All right.  So that didn't work so well.

          4              Now, the other alternative --

          5    right, eight other alternatives -- again, we

          6    collect this -- this is no problem, total

          7    fuel -- get from the respondents fuel use to

          8    make electricity.  In other words, get it

          9    directly, and then by subtraction, we can get

         10    fuel used to generate heat and steam, which

         11    is of interest to people.  So we went through

         12    an even more elaborate process in which we

         13    had a lot of internal discussions, had an

         14    outside consultant assisting us, went through

         15    about 40 altogether interviews with actual

         16    power plants in which we discussed concepts,

         17    went through test forms, had them fill out

         18    test forms.  I think there's also discussions

         19    even with this committee about this at least

         20    twice.

         21              And so we launched our new form,

         22    it's called the EIA-920.  And here I am today



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           339


          1    because the change in methodology produced

          2    both radically different and radically

          3    unexpected results.  Basically, it was the

          4    wrong answer.  And for reasons I'll discuss

          5    -- not just because it doesn't make us feel

          6    good, but its numbers which in many cases

          7    just don't make sense.

          8              Now, before I go further into this

          9    as to why it's a problem, we need to at least

         10    briefly just go over just a few measures of

         11    combined heat and power plant efficiency.

         12    There are three things to talk about here.

         13    One is the heat rate of a plant, which is the

         14    BTUs of fuel input required to produce one

         15    kilowatt hour of electricity.  The more

         16    efficient a plant is, the fewer BTUs of input

         17    required to product one output unit of kWh

         18    output, all right?

         19              The second one is the

         20    power-to-steam ratio, which in essence is the

         21    ratio of the heat value of the electricity a

         22    plant is producing to the heat value of the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           340


          1    steam it is producing for manufacturing

          2    process, all right?  So the more fuel being

          3    devoted to power production, the higher the

          4    ratio will be, okay?

          5              And then finally, we have thermal

          6    efficiency, which is the overall efficiency

          7    of the plant.  And what you have here is the

          8    ratio of the sum of the heat content of the

          9    power and the steam produced to total fuel

         10    input.  So, for example, if the thermal

         11    efficiency of a plant is 50 percent, that

         12    means that half of all the fuel input is

         13    being put to a productive use, whether it's

         14    making electricity or making steam, and half

         15    of it is wasted.  By contrast, if you look at

         16    a typical stand-alone power plant, the

         17    thermal efficiency would be, at best, about

         18    33 to 35, 36 percent.  So a combined heat and

         19    power plant should be way above that.

         20              All right, now, the approach we

         21    took -- go back for a second -- if you

         22    recall, was we went to this approach:  Ask



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           341


          1    them how much fuel's being used to generate

          2    electricity, subtract it from total fuel, you

          3    get fuel used to generate steam.  Well, what

          4    we found was -- in retrospect, this maybe is

          5    not so surprising -- is that when we asked

          6    people to focus on how much fuel they are

          7    using to make electricity, that completely

          8    changed the orientation of their thinking,

          9    and they began to focus -- in some cases

         10    exclusively -- on what do I need to compute

         11    fuel for electricity -- I'm not even going to

         12    worry about really what it takes to compute

         13    fuel for useful thermal output.  As a result

         14    of this change in focus and the way they were

         15    looking at the problem, the responses we got

         16    in 2004, the first year we did this, showed

         17    vastly more fuel being used to make

         18    electricity than in the past.

         19              Basically, we asked people how much

         20    fuel are you using to make electricity, and

         21    they answered us.  But they didn't answer us

         22    in the expected way, they didn't go through



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           342


          1    some calculation to figure out, well, what

          2    should it really be as versus fuel for steam,

          3    they just took kind of a stand-alone approach

          4    and came back to us with how much fuel is

          5    being used to make electricity.  And you can

          6    see this in the measures of efficiency.  In

          7    the 2003 data collection, the average heat

          8    rate for a combined heat and power plant was

          9    about 9,800, 9,900 BTUs per kWh.  In 2004, it

         10    ballooned to almost 14,000, and that was

         11    because the respondents, when we asked them

         12    how much fuel they used to make electricity,

         13    well, they answered us by focusing on that

         14    question.  Here is the power-to-steam ratio,

         15    which goes from .33 to .41.  What this,

         16    again, reflects is more fuel being used to

         17    make electricity than being used to make

         18    steam.  And then finally, this is the average

         19    thermal efficiency of the plant, and it

         20    plummets from 60 percent to 49 percent.  And

         21    again, actually, it reflects two things:  One

         22    is plants allocating much more of their fuel



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           343


          1    to electricity production, which is

          2    relatively inefficient, than to making steam,

          3    which is more efficient, so you see the

          4    decline there.  What we also had -- that's

          5    why I'm saying these numbers in some cases

          6    just don't make sense.  We had quite a few

          7    respondents who allocated zero fuel to making

          8    steam, to making useful thermal output.

          9              MR. CLEVELAND:  Stan?  Can I just

         10    interrupt?

         11              MR. KAPLAN:  Yes.  Sure.

         12              MR. CLEVELAND:  I'm sorry to

         13    interrupt.  How were they making that

         14    assessment?  They know the fuel is going in a

         15    combustion turbine, and what comes out the

         16    two ways is a function of the efficiency of

         17    the combustion turbine.  So when you ask them

         18    how much fuel is being used to make

         19    electricity, what are they basing that on?

         20              MR. KAPLAN:  Well, you see, there

         21    are different ways you could calculate it.

         22    You could calculate a stand-alone



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           344


          1    hypothetical heat rate for the combustion

          2    turbine okay?

          3              MR. CLEVELAND:  Right.

          4              MR. KAPLAN:  You can measure the

          5    temperature of the steam at different points

          6    in this process.  You can measure the

          7    temperature of the gases coming out.

          8    Actually, I've got to admit, some of this

          9    goes beyond certainly my technical expertise,

         10    but we did enough work to verify that there

         11    are doable ways of measuring how this fuel's

         12    being allocated.  There's also a more

         13    hypothetical way, which I skipped over here,

         14    but it's in the presentation, which is called

         15    "Thermal Displacement," in which basically,

         16    you calculate what your fuel consumption

         17    would look like for a hypothetical

         18    stand-alone point and use that as the basis

         19    for making your calculations.  So there's a

         20    variety of ways to do it.

         21              What we found, though, was that

         22    people -- again, after all these interviews



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           345


          1    and such -- people, although they told us

          2    they could do it and would do it, just

          3    weren't doing it, and what I'm about to get

          4    to, in fact, is a most extreme example, which

          5    is the plants which allocated zero fuel --

          6    did I answer your question, by the way --

          7              MR. CLEVELAND:  Yeah, mm-hmm.

          8              MR. KAPLAN:  -- to making steam.

          9              You come back to our case here, we

         10    had 177 respondents out of 610 tell us that

         11    although they were producing steam, they

         12    didn't use any fuel to make it.  And the

         13    reason they said that -- and this is a

         14    typical answer -- is that, well, the fuel

         15    goes in to my combustion turbine, and I'm

         16    going to run this thing no matter what

         17    because I need electricity to make my plant

         18    go.  So all the hot exhaust gases that come

         19    out of it, it's a freebie.  It's going to be

         20    there anyway.  So therefore, I consume zero

         21    fuel to make the hot gases which then go here

         22    to the heat recovery boiler to make the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           346


          1    steam.  So therefore I've consumed no fuel to

          2    make the steam which I do produce, all right?

          3              So this didn't work out real well.

          4    And in many cases, we ended up with reported

          5    efficiencies for these plants which were

          6    comparable to or even worse than if they had

          7    built two stand-alone power plants with none

          8    of the efficiencies of capturing the waste

          9    heat.  Those numbers just don't make sense.

         10    I mean, again, this is just not a question of

         11    this not making us feel good or not meeting

         12    expectations from a reasonable -- call it

         13    theoretical -- standpoint, it just doesn't

         14    add up.  And we have the plants saying they

         15    produce -- yeah, sure.

         16              DR. NEERCHAL:  How long when they

         17    designed the plant should they have had some

         18    calculation of the ratio of the fuel what

         19    this part was and that part should --

         20              MR. KAPLAN:  You would think that,

         21    or you might think that.  I certainly thought

         22    of going into this, for example, that at some



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           347


          1    of these plants, they'd be very carefully

          2    measuring their steam flows.  Some of these

          3    plants, like paper mills, have very

          4    complicated flows of steam; it goes all over

          5    the plant to different processes, the

          6    temperatures change, the pressures change.  I

          7    know that I had thought they were all

          8    measuring this carefully for their cost

          9    accounting.  Their answer is, they don't.

         10    They don't.

         11              They're interested in the total

         12    amount of fuel going in, what it's costing

         13    them, they're interested in how much

         14    electricity they produce, what the total cost

         15    of their final products is, but as far as

         16    doing detailed measuring and detailed

         17    metering, surely there are exceptions, but

         18    for the most part, they just don't do it.

         19              Yeah?

         20              MR. RUTHERFORD:  It's basically

         21    impossible to evaluate the state of that of

         22    that efficiency measure.  If you don't know



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           348


          1    what the heat content of the steam that's

          2    being used in the process is, then --

          3              MR. KAPLAN:  It's very hard.  It's

          4    very hard.

          5              MR. RUTHERFORD:  You basically

          6    can't do it.

          7              MR. KAPLAN:  And that's why one

          8    recommended method is this thermal

          9    displacement method which you do a series of

         10    hypothetical calculations to determine what

         11    the fuel allocation should be.  And I'll get

         12    back to that.  All right.

         13              So because of these problems, we

         14    had to correct the data in the aggregate,

         15    which was just wasn't acceptable, but we

         16    didn't have the time or the resources to go

         17    back and call hundreds of respondents and

         18    help them figure out the right answer.  So we

         19    needed to go through a process to adjust the

         20    data.

         21              I'm going to go through this fairly

         22    quickly because -- you all can judge, I don't



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           349


          1    think it's all that interesting; it was a

          2    fairly mechanical process.  Our object was to

          3    bring the aggregate measures of efficiency

          4    for this sector, the combined heat and power

          5    sector, into a reasonable range compared to

          6    historical trends.  We wanted to change as

          7    little reported data as possible.

          8              The total fuel reported was never

          9    adjusted -- we felt that was a good number --

         10    just the allocation of fuel.  Out of 610

         11    reporting plants, we ended up adjusting 184.

         12    And the basic process was that each CHP plant

         13    was evaluated against a maximum acceptable

         14    value for the power-to-steam ratio based on

         15    their 2003 data.  If a plant exceeded the

         16    maximum, the allocation of fuel for the plant

         17    was adjusted either to match the 2003

         18    power-to-steam ratio or, if that number, also

         19    in retrospect, didn't look so good, to match

         20    a 2003 industry-wide average for the plant

         21    type.  And then the process had various

         22    checks to make sure we didn't end up



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           350


          1    ultimately with numbers which, in some other

          2    respect, didn't make sense.  Yes?

          3              MS. KHANNA:  But both of those

          4    numbers ---- 2003 number, for example, is

          5    based on the earlier survey which asked them

          6    to look at the other side of the equation --

          7              MR. KAPLAN:  That's right.

          8              MS. KHANNA:  -- so that's also a

          9    suspect number to start off with.

         10              MR. KAPLAN:  Yes, it is.  It is.

         11    We think it's a much better number, it makes

         12    a whole lot more sense, but there was no

         13    question that there were problems with that

         14    earlier data.  Now, for that earlier data,

         15    though, I will say this:  We did go through

         16    the process of making literally hundreds of

         17    phone calls.  We had I don't know how many

         18    people over a period of months making

         19    hundreds of phone calls to speak with the

         20    respondents directly, try to understand why

         21    the numbers looked, in many cases, odd, and

         22    get them more in line.  So while that data is



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           351


          1    certainly not perfect, we went through a very

          2    long and arduous process to make it at least

          3    reasonable.  And so in trying to bring the

          4    2004 data in line with the 2003 data, I think

          5    we were taking a reasonable approach.

          6              There's a lot more detail here on

          7    the adjustment process ratio I'd be happy to

          8    discuss later.  It goes on at some length.

          9    We ended up adjusting, I guess, about 180

         10    plants -- 184 plants, the ------ again being

         11    basically to bring the plants which seem most

         12    out of line into line with their own

         13    historical patterns, and so we ended up with

         14    aggregate data which not surprisingly matches

         15    the historical average.

         16                   (Laughter)

         17              SPEAKER:  Duh.

         18              MR. KAPLAN:  Yeah.  Pretty amazing,

         19    huh?  So the heat rate for 2003 in the

         20    aggregate had been about 9,900, it had

         21    ballooned in the unadjusted 2004 data to

         22    almost 14,000.  In the adjusted data, it's



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           352


          1    back down towards around 10,000.

          2              DR. NEERCHAL:  Could I ask a

          3    question?

          4              MR. KAPLAN:  Yeah.

          5              DR. NEERCHAL:  --- mention it --

          6    maybe I missed it -- but how did you choose

          7    the 184 plants to adjust?

          8              MR. KAPLAN:  Well, what we did was,

          9    first we wanted to adjust the reported data

         10    as little as we could, all right?  Secondly,

         11    there are plants -- some of these plants are

         12    very complicated configurations, and from our

         13    information, we could identify frequently

         14    which of those plants had the most

         15    complicated configurations.  We want to stay

         16    away from those because the more complicated

         17    the configuration, the harder to tell if the

         18    data was really bad or if there was just

         19    something going on, given that it was a

         20    complicated plant.  So that was an important

         21    criteria for narrowing it down.

         22              We ultimately ended up focusing on



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           353


          1    plants which are called steam-electric

          2    plants, very straightforward boiler plants,

          3    and also certain kind of combined cycle

          4    plants, which have what are called heat

          5    recovery steam generators.  So that narrowed

          6    it down to a fairly small subset of plants

          7    which were potential candidates for

          8    adjustment.

          9              Then of these potential candidates,

         10    we checked to see which ones actually had

         11    out-of-range indicators and that narrowed it

         12    down to the 184, which I mentioned, which

         13    were ultimately adjusted.  Does that help?

         14              DR. NEERCHAL:  Yes.  Are there

         15    plants that you'd say are doing a pretty good

         16    job of doing the allocation?  Do you know?

         17              MR. KAPLAN:  Yeah, some plants

         18    certainly do --

         19              DR. NEERCHAL:  How many do a good

         20    job out of the 610?

         21              MR. KAPLAN:  That's a good

         22    question.  Fewer than half.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           354


          1              MR. CLEVELAND:  Well, the point is

          2    you don't know what good is because it's a

          3    totally arbitrary method.

          4              DR. NEERCHAL:  No, based on their

          5    explaining their method clearly in a

          6    particular ------.

          7              MR. CLEVELAND:  This is exactly

          8    what we went through in the 1960s with

          9    regulating gas prices.  The precursor of FERC

         10    had to regulate gas prices based on cost of

         11    production; the problem is you have oil that

         12    you drill a well -- which costs so much -- up

         13    out of the ground comes oil and gas, so how

         14    much of that cost do you assign to oil and

         15    gas, and they had hearings for a gazillion

         16    hours and it's totally arbitrary.

         17              MR. KAPLAN:  Yes, it's definitely

         18    analogous.  But one thing we do know, though,

         19    is that while it may be largely arbitrary,

         20    what we do know is that it's got to be, when

         21    you look at it, more efficient than if you

         22    had built two separate plants --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           355


          1              MR. CLEVELAND:  Well, but that's

          2    not the question.  Everyone knows it's more

          3    efficient, the question you started with was

          4    the assumption that you have a perceived need

          5    to allocate a certain amount of energy to a

          6    certain amount of electricity so it fits in

          7    with the rest of your stuff.  So --

          8              MR. KAPLAN:  All right, so here are

          9    the adjusted heat rates, here's the adjusted

         10    power-to-steam ratios that came back in line

         11    from '03 to -- reported '04 to adjusted '04,

         12    and then finally the thermal efficiency of

         13    the overall plants.  All right, now this is

         14    all well and good.  We are currently

         15    collecting 2005 data, which is going to have

         16    the same problems as the 2004 data and 2006

         17    data.  And we're going to need to adjust that

         18    data.  We clearly need some definitive

         19    solution for this, or else we're going to be

         20    in the data adjustment business for CHP

         21    plants for the indefinite future, which all

         22    good things to the side, is from a resource



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           356


          1    standpoint really not acceptable.  It just

          2    occupies too much time, too much in the way

          3    of staff resources, to be constantly trying

          4    to solve these problems.

          5              Now, one thing we're currently

          6    doing in conjunction with our SMG team is

          7    we're testing with respondents additional

          8    instructions intended to help them complete

          9    the survey.  However we'd be happy to be

         10    surprised, but it's highly unlikely that

         11    simply sending out new instructions is going

         12    to solve the problem.  Yeah?

         13              MR. FREEDMAN:  The fact of the

         14    matter is that the companies we called, it is

         15    just as problematic with these additional

         16    instructions.  Some of them don't understand

         17    the concept.  We went to respondents who were

         18    reporting free steam, and they'd go through

         19    and they'd say, oh, yeah, this is a nice

         20    instruction booklet, it makes a lot of sense,

         21    and we'd ask why they were reporting free

         22    steam, and they said, oh, well, you didn't



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           357


          1    want us to allocate it, so I -- it was a good

          2    idea, it was well intentioned, in some

          3    instances a light bulb went off where the

          4    respondent said, oh, okay, I see what you

          5    want to do, but it was not the silver bullet

          6    that we had hoped it would be.

          7              MR. KAPLAN:  Right.

          8              MR. FREEDMAN:  You haven't seen

          9    those results yet, but we have.

         10              MR. KAPLAN:  Okay, okay.  All

         11    right.

         12              MR. FREEDMAN:  That means we don't

         13    have to write a report?

         14              MR. KAPLAN:  Yeah, sure.  The basic

         15    problem -- and again, it's worth just briefly

         16    recapping -- why we don't expect or have now

         17    seen that a simple fix like better

         18    instructions is not going to work is because

         19    we have two fundamental issues.  The first is

         20    that we're trying to collect something which

         21    is basically not of interest to many or most

         22    of these respondents and not a piece of



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           358


          1    information that they compute or otherwise

          2    gather as part of their normal course of

          3    business, which certainly makes it unusual,

          4    if not maybe even unique -- I haven't thought

          5    through all the data we collect -- but it's

          6    certainly unusual compared to the vast

          7    majority of the data we collect on the

          8    electric side, that we're asking for

          9    something which is normally not of interest

         10    to them, okay?

         11              And the second problem is that,

         12    again, there is no hypothetically correct or

         13    otherwise standardized way to do the

         14    allocation.  So what do we do -- and we're

         15    just in the earliest stages of discussing how

         16    to go about, again, finding our definitive

         17    solution.  But the basic direction we're

         18    heading is that we're going to have to be --

         19    meaning EIA -- much more -- you can call it

         20    directive or have to narrow the options for

         21    respondents far more than in the past.  This

         22    can mean specifying the estimation method



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           359


          1    that has to be used -- just telling them you

          2    have to use this approach to calculate it,

          3    and also specifying typical or acceptable

          4    values for key input into the method, like

          5    the efficiency of a boiler if the respondent

          6    does not have reasonable data.

          7              Now, as far as implementing this,

          8    there are sort of different levels of

          9    control.  One is, we can build into our --

         10    most of our data is collected online using

         11    what's called the Internet data collection

         12    system.  One thing we can do is build into

         13    this what's called the IDC system a combined

         14    heat and power calculator that the respondent

         15    would have right their on his or her screen

         16    that would say enter these numbers here, push

         17    the button, out pops the allocation.

         18              Another approach is for EIA to make

         19    all the estimates itself.  That is, given the

         20    information we do know from other surveys

         21    about plant configurations and historical

         22    data we've gathered, we'll just collect the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           360


          1    data we know that they have -- like total

          2    fuel -- take typical values and just do the

          3    estimates ourselves.  That's another possible

          4    approach.  Again, we're in -- it's the

          5    earliest stages of talking about this, but

          6    the basic thrust of our internal discussions

          7    is that we have to be more directive, we have

          8    to narrow the options for how this can be

          9    done, and we have to be ready to really just

         10    provide key inputs rather than letting

         11    respondents go off and put in numbers which

         12    just produce results which are just not

         13    defendable.

         14              So considerations as far as what

         15    approach to implement -- is more research

         16    needed into the typical values or typical

         17    ranges for key parameters such as what the

         18    power-to-steam ratio should be for a certain

         19    kind of plants or what the boiler efficiency

         20    should be?  Will we need to collect

         21    additional information on plant

         22    configurations?  That would be a difficult



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           361


          1    thing to do, but certainly worth considering.

          2    Very important is what is the development

          3    time and cost for implementing different

          4    approaches?  Right now, we're heading towards

          5    what we call our Electricity 2008 process,

          6    which is based on our three-year cycle for

          7    reauthorizing our forms with OMB.  And our

          8    intent is that if we're going to have a new

          9    process, to have it ready to go by certainly

         10    no later than January 2008, and hopefully

         11    this time have the definitive process.

         12              After I had finished the

         13    presentation and e-mailed it off, I realized

         14    I should have added one more slide.  There's

         15    a famous cartoon ran in The New Yorker years

         16    ago by a man named Arno, and it shows an

         17    airplane that's just crashed, and there's a

         18    guy parachuting out and a bunch of generals

         19    going, oh, my God.  In the foreground is this

         20    professor rubbing his hands together with a

         21    big smile on his face saying back to the

         22    drawing board --



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           362


          1                   (Laughter)

          2              MR. KAPLAN:  And we are somewhat in

          3    that mode.  Any suggestions you all have

          4    would be very welcome.  We really do need to

          5    kind of put this one to bed; it's been a

          6    persistent problem for a long time now.  Any

          7    questions, comments, suggestions?  Any

          8    solutions we can implement tomorrow would all

          9    be welcome.  Yes?

         10              MR. CLEVELAND:  One, I think you're

         11    right, I think you need to impose your own

         12    system of doing this because there are way

         13    too many ways in which engineers and the

         14    financial folks at different institutions

         15    would do this allocation.

         16              What about -- this could be

         17    completely off the wall -- you're focused on

         18    the engineering side of things to do this

         19    allocation, but these are a lot of economic

         20    decisions that these people are making.  And

         21    so why not allocate, think about, or

         22    investigate allocating the fuel import based



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           363


          1    on the relative values of the outputs of the

          2    steam and the electricity --

          3              MR. KAPLAN:  Okay.

          4              MR. CLEVELAND:  -- because one of

          5    the problems with the engineering approach is

          6    that you're taking a heat unit of electricity

          7    and a heat unit of steam and assuming they're

          8    equal, but from an economic perspective,

          9    they're certainly not.  The marginal product

         10    of a heat unit of electricity is much higher;

         11    that's why you're willing to pay a lot more

         12    money for a heat unit of electricity.  So

         13    basing things on heat units when you're

         14    looking at steam and electricity is

         15    economically bogus.

         16              So you might think about looking at

         17    -- and you can -- even though they may not be

         18    this -- electricity is easier value, the

         19    steam is harder, but clearly there's an

         20    avoided cost here -- there's an opportunity

         21    cost to this waste heat, and if they didn't

         22    capture the waste heat to generate the



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           364


          1    process steam, they would have to generate

          2    the process steam in a different way, and

          3    that would have a cost that you could assess.

          4    So you might look at the allocation based on

          5    relative value of the product rather than

          6    some kind of thermodynamic efficiency.

          7              DR. NEERCHAL:  Except that in one

          8    -- and I think this is a very nice start, but

          9    I don't know that they are consciously making

         10    that allocation or letting their system do

         11    that allocation for them.  That part is not

         12    clear to me, that sometimes they have

         13    inherited a system that happens to be doing

         14    the allocation for them or are trying to

         15    allocate.

         16              MR. CLEVELAND:  I think they think

         17    about the economics very closely and are

         18    making those decisions --

         19              MR. KAPLAN:  Well, they're

         20    certainly interested in the total value of

         21    what's coming out, the electricity and the

         22    manufacturing product which the steam is used



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           365


          1    to produce.  But what continues to be

          2    surprising is that they don't meter

          3    internally more carefully given the

          4    importance of the inputs to what you get at

          5    the other end.  They just don't do it.

          6              DR. HENGARTNER:  Well, isn't this a

          7    case of beauty is in the eye of the beholder,

          8    that if you ask the question why are you

          9    running that turbine, one will say it's

         10    electricity and they'll be holding it free

         11    steam, something free, that's wonderful.

         12    Someone else may say he needs the steam to

         13    run something else, and by the way, he also

         14    gets electricity.  And so, depending on what

         15    they value, they will give different answers,

         16    at least -- this is very nave ------.

         17              MR. KAPLAN:  It's not nave from

         18    the standpoint of the problem we have at all

         19    because -- I gave as an example the case

         20    where they're allocating zero fuel to useful

         21    thermal output.  We do have other people

         22    whose view is I'm producing steam to make my



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           366


          1    widgets, and I'm going to make my widgets no

          2    matter what.  And consequently, they end up

          3    allocating very, very -- never zero, but --

          4    well, rarely it's zero -- but very, very

          5    little fuel to the waste steam which is

          6    captured in this case to make the

          7    electricity.  So you end up with heat rates

          8    which, in some -- I don't know if it's

          9    thermodynamics, but the heat value of a unit

         10    of electricity kilowatt hour is 3,412 BTUs

         11    per kilowatt hour, and we've had people come

         12    in with heat rates below that, which is

         13    physically impossible.

         14              MR. CLEVELAND:  But why would you

         15    even entertain a system that would produce

         16    that result?  I don't understand what --

         17              MR. KAPLAN:  Well, you see, from

         18    their standpoint -- it's the reverse of the

         19    example I gave, actually.  Their standpoint

         20    is, I'm a paper mill.  I'm going to make

         21    paper, and from my standpoint, all this fuel

         22    is being used to make paper.  I make



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           367


          1    electricity, too, so I allocate a little bit

          2    to making the electricity, but the number's

          3    so small because they're focused on the paper

          4    part of it that you end up with the results

          5    which just don't add up.

          6              MR. CLEVELAND:  You can just stop

          7    listening to them and do something on your

          8    own that makes more sense.

          9              DR. HENGARTNER:  But that does

         10    exactly what you were saying, is that it's

         11    the economics, and their view of the

         12    economics is that I need to produce steel, or

         13    I need to produce electricity.  And

         14    therefore, the question you should ask first,

         15    what is your primary line of business, what's

         16    your primary focus.  And depending what it

         17    is, it's going to be one calculation or the

         18    other one that's going to be relevant in

         19    their eyes, and maybe in yours as well,

         20    because the one who gets free steam or free

         21    electricity, what there is, that's how it's

         22    perceived, and maybe that's how we should



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           368


          1    calculate it.  I don't know.

          2              DR. BURTON:  Nick, I don't think so

          3    because I think the goal in this is to figure

          4    out how much of a very real fuel you're using

          5    to produce a very real amount of electricity.

          6    So to make it arbitrary I don't think is --

          7    you're not going to be adding together what

          8    you want to be adding together.

          9              MR. KAPLAN:  Yes.

         10              MR. RUTHERFORD:  I'll just say

         11    there are all these deep issues about how

         12    these guys operate their business, so I'm not

         13    going to dwell on them.  I'm just focusing on

         14    how you fix the data.  So, yeah -- the only

         15    comment I'd suggest -- and there undoubtedly

         16    are statisticians here that'd be better able

         17    to give you specific suggestions but -- it

         18    sounds similar to problems we've run into

         19    with household surveys where we don't have

         20    information about where people earn their

         21    money, and we have to try to figure out

         22    things, and we -- there's this technique



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           369


          1    called small sample estimation, which,

          2    basically, when you have numbers that are

          3    missing for some set of the data, you can use

          4    a statistical model to impute rather than

          5    simply projecting them all onto the extreme

          6    value.  I'm just concerned -- and again, this

          7    is a -- in other words, if you had

          8    information about what their line of business

          9    was, what the split -- how much electricity

         10    made -- you could basically run a regression

         11    and produce an estimation from your full

         12    sample on what the factors were that gave

         13    rise to what you were trying to estimate.

         14              Then for those outliers that you

         15    don't trust their response, you could fill in

         16    using the statistical model rather than

         17    simply just saying we'll just give them the

         18    minimum value.  It's a very mechanical

         19    suggestion, but I'm just --

         20              DR. NEERCHAL:  That is what I had

         21    in mind when I asked him.  Are there at least

         22    some reliable data.  --------



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           370


          1              MR. RUTHERFORD:  The thing is, to

          2    do that you have to have more information,

          3    you need to know something about what field

          4    -- what they're producing and what -- you're

          5    going to be able to construct a model, a

          6    statistical model.

          7              DR. HENGARTNER:  Neha?

          8              MS. KHANNA:  So I have a nave

          9    question, which is, it seems like there's a

         10    lot of institutional effort being spent on

         11    collecting one piece of information.  So the

         12    question is, why do you do this?  I mean, how

         13    important is this piece of information to

         14    your overall exercise?  Would it matter if

         15    you didn't have this information at all?

         16              MR. KAPLAN:  That's a very good

         17    question.  Compared to the total resources

         18    which go into the surveys I collect,

         19    generation of fuel consumption, we're putting

         20    enormously disproportionate amount of those

         21    resources into collecting from those combined

         22    heat and power plants, trying to get the data



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           371


          1    straight.  We do have to collect the data,

          2    I'll say for at least two reasons.  One is

          3    that people do want to have complete data.

          4    Well, it may just be five percent of looking

          5    at -- in terms of generation and also small

          6    percentages of fuel, it's still part of the

          7    puzzle.  And we know that customers do want

          8    to have that piece.

          9              The other part, though, which is

         10    kind of a subset of what I just mentioned,

         11    has to do with natural gas.  There is a fair

         12    amount of natural gas which is consumed in

         13    various kinds of combined heat and power

         14    plants.  And we know from customer comments

         15    -- again, speaking from personal experience

         16    -- that in the past, when the EIA data did

         17    not make it easy to figure out how much gas

         18    was being used by industrial consumers to

         19    make steam versus making electricity, for

         20    people interested in studying the natural gas

         21    business, that was a real problem.  And --

         22    sorry.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           372


          1              MR. SITZER:  I was going to ask --

          2    had a comment.

          3              MR. KAPLAN:  All right, and so that

          4    is a particularly important piece which we

          5    need to fill in.

          6              MR. SITZER:  I'm Scott Sitzer, I'm

          7    at the Office of Standards --

          8              SPEAKER:  Scott, would you step up

          9    to a microphone, please, or have a seat at

         10    the microphone?  Thank you.

         11              MR. SITZER:  I want to answer your

         12    question also.  I think your question maybe

         13    is not so much why do we collect the data,

         14    but why do we spend so much time trying to

         15    allocate it between electricity and UTO, and

         16    I think there are two groups of users that

         17    are interested in it.

         18              One is our internal modelers, and

         19    even though none of them is here right now,

         20    they're really the ones who noticed when we

         21    first collected the information for 2005 or

         22    2004 how much the heat rate had increased for



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           373


          1    CHP, so they brought it to our attention and

          2    they thought that we should possibly do

          3    something about it.

          4              And the other is people who are

          5    interested in promoting such things as CHP

          6    because they believe that it's a more

          7    efficient way of producing electricity and

          8    also producing the steam that goes into other

          9    industrial processes.  And they have made an

         10    argument about how much more efficient it is

         11    and how much fewer BTUs of fuel you need to

         12    produce electricity.  So we have attempted to

         13    fill that data gap.

         14              But I think your question's very

         15    good:  Why should we do this.  If we collect

         16    the information on consumption and we collect

         17    the information on electricity, maybe we need

         18    to leave it to others to figure out how to do

         19    that allocation.

         20              MR. CLEVELAND:  I agree.  This is

         21    not the place to make the argument for

         22    combined heat and power, I don't think.  You



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           374


          1    know what, another easy way to do it is to

          2    say if that steam turbine that they're

          3    generating electricity with is -- let's say

          4    it has a standard efficiency and you're

          5    burning 10,000 BTUs of fuel to get one

          6    kilowatt hour of electricity, and the

          7    question is, how much fuel was used to

          8    generate the kilowatt hour of electricity,

          9    the literal answer is 10,000 BTUs of fuel.

         10              MR. KAPLAN:  And that is a

         11    perfectly legitimate approach, and that is

         12    the kind of direction which we probably need

         13    to move, something which is straightforward

         14    -- I was mentioning before, and you'll see it

         15    in here -- this thermal displacement approach

         16    which is, in so many words, is the kind of

         17    thing you just said.  That is, make a

         18    hypothetical calculation of, say, what the

         19    stand-alone efficiency should be and then use

         20    that as your basis.

         21              DR. HENGARTNER:  Walter?

         22              MR. HILL:  Yes, I've got a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           375


          1    variation on what a couple of comments have

          2    probably been said.  Maybe the question is

          3    along the lines of can you impute the number

          4    from things that are unknown by the other

          5    companies, given a boiler size of X with this

          6    configuration and maybe something else, can

          7    you estimate how much is --

          8              MR. KAPLAN:  Well, in a pretty

          9    simple way, that's really what we did for

         10    many plants for the 2004 data; that is, if

         11    the measures of efficiency for a plant for

         12    2004 were out of line, and then if we looked

         13    at their own measures for 2003, and those

         14    were also out of line, then we used typical

         15    industry averages.

         16              MR. HILL:  I've seen the

         17    electricity data on there, the Web page, and

         18    it is noticeable, so it would be nice -- it's

         19    nice to have the number, even though you're

         20    talking five percent or so of the use.

         21              MR. KAPLAN:  Well, you see, you've

         22    just invalidated one of my key assumptions.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           376


          1    You're the second person today to say they

          2    actually look at those numbers.

          3                   (Laughter)

          4              MR. KAPLAN:  And prior to today,

          5    I've had nobody actually give me a phone call

          6    or say anything to indicate they actually

          7    look at those allocations, but evidently,

          8    someone's doing it.

          9              DR. BURTON:  Let me be the third,

         10    but I also need to add that when I looked at

         11    them, I thought, how'd they do that.

         12                   (Laughter)

         13              SPEAKER:  Now you know.

         14              DR. FEDER:  You mentioned for the

         15    2004, you used the adjustment whenever the

         16    steam-to-power ratio passed a certain

         17    threshold, and then Scott mentioned that the

         18    modelers were interested in the data.  And I

         19    was just thinking, now, if you're free to

         20    model to that, you'll get -- and indeed, you

         21    showed in that graph that we did see an

         22    increase in that ratio compared to '02.  So



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           377


          1    clearly using this cutoff does introduce some

          2    bias into this simple model of power-to-steam

          3    ratio.

          4              MR. KAPLAN:  Well, it's certainly

          5    not a particularly sophisticated technique

          6    for adjusting the data, and --

          7              DR. FEDER:  I was thinking along

          8    what Neha was asking:  Why collect it and not

          9    leave it there and let the modelers worry

         10    about it, even internally here, use whatever

         11    methods they use, use historical data and not

         12    -- sorry to say this among statisticians --

         13    massage the data.

         14              MR. KAPLAN:  Well, that's one

         15    approach.  Another approach is to find a way

         16    to do the allocation which is both simple,

         17    economical, and reasonable.  If we don't do

         18    the allocation at all -- I'm willing to think

         19    about this -- but particularly for people

         20    interested in natural gas data, that may pose

         21    a problem.

         22              MR. CLEVELAND:  Why?



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           378


          1              MR. KAPLAN:  Because when you're

          2    trying to figure out what's going on with

          3    natural gas consumption, a significant chunk

          4    of it -- I don't want to call it a

          5    percentage, but it's material, it's not tiny

          6    -- is in the industrial sector.  And when

          7    people have done analyses of what is driving

          8    gas demand, when you look in the industrial

          9    sector and you can't tell how much of it is

         10    going to make electricity and how much of it

         11    is going to make steam, it makes the analysis

         12    difficult.  Now, one could say, well, fine,

         13    do your own allocation -- and maybe that's

         14    the thing we should do -- but that is a

         15    particular issue which we're going to have to

         16    think through and address.

         17              MR. CLEVELAND:  I think that's the

         18    approach, though, because everyone -- there

         19    are many different allocation schemes and the

         20    type of analysis that you're going to be

         21    using the allocation scheme in is going to

         22    drive your choice of allocation scheme.  And



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           379


          1    so I don't think you picking one is going to

          2    be, by definition, suboptimal for a large

          3    percentage of the people wanting that data.

          4    So why not just let the people who are

          5    interested in that choose their own

          6    allocation scheme?  And they have to defend

          7    it.

          8              MR. KAPLAN:  Another argument is

          9    that if everyone rolls their own, then when

         10    people start trying to compare data they're

         11    all going to have different numbers, and if

         12    EIA has a methodology and people think it's

         13    reasonable, then it establishes a common

         14    starting point for studies of the data.

         15              DR. FEDER:  Whenever respondents

         16    have to make adjustments, you get some

         17    strange things.  Again, from the Canadian

         18    experience in this area, census in Quebec,

         19    they found out that, for some reason, many

         20    more women tend to describe their

         21    relationship as common law marriage where the

         22    men described it as just a



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           380


          1    boyfriend-girlfriend relationship.  So if you

          2    did the estimates based -- no, that's a true

          3    fact.  If you use --

          4              MR. KAPLAN:  I can believe it.

          5              DR. FEDER:  -- oh, and Nick is from

          6    Quebec, maybe he can -- so if you did the

          7    estimates using females or males, you got

          8    different responses.  So there is a

          9    perception issue here, or a user definition.

         10    That's why I thought what Neha said, why

         11    collect it, why not let users decide that

         12    it's more -- no, maybe it's --

         13              MR. KAPLAN:  You actually just

         14    raised an interesting and a really good

         15    point, which is that we're talking about the

         16    respondents.  Well, who literally is the

         17    respondent?  Sometimes it's the plan

         18    engineer, sometimes it's the data clerk in

         19    some central office.  So, again, men versus

         20    women, plan engineers versus data clerks.  I

         21    think it's another argument for imposing more

         22    uniformity on how this is being done.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           381


          1              DR. HENGARTNER:  Barbara, ---- have

          2    stolen your thunder.

          3              MS. FORSYTH:  Hey, that was my

          4    thunder.

          5                   (Laughter)

          6              MS. FORSYTH:  I think -- I was

          7    supposed to be the discussant, and I think --

          8    my main point was that when you collect data

          9    two different ways and they should add up and

         10    they don't, and you spend way too much money

         11    figuring out why they don't, you probably

         12    have at least as much a cognitive problem as

         13    you have a statistical problem, and I think

         14    that's basically what we're saying here, that

         15    the models that respondents are using to

         16    report their data don't match the model that

         17    EIA has of the process, and if you have a

         18    different model, you should estimate it and I

         19    think your new approaches, computing them

         20    yourself or developing a Web tool that

         21    computes it are both far better than

         22    collecting data from two perspectives,



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           382


          1    neither of which is right, or both of which

          2    are right.

          3              MR. FREEDMAN:  And in fact,

          4    Barbara, I think given our discussions with

          5    respondents -- and I take some responsibility

          6    for problems that Stan was describing --

          7              MR. KAPLAN:  We'll be talking about

          8    that later.

          9              MR. FREEDMAN:  He's very generous

         10    not to mention our partnership in this.  It's

         11    too generous to even say that there are

         12    perceptual or cognitive models that are going

         13    on here.  A lot of it is, this doesn't make

         14    any sense to us.  I don't collect this, I

         15    don't want to really be bothered --

         16              MS. FORSYTH:  Exactly.

         17              MR. FREEDMAN:  -- so I'm going to

         18    give EIA a number that will result in not

         19    being called back. Well, in 2003, it didn't

         20    work out --

         21              MS. FORSYTH:  It didn't work.

         22              MR. FREEDMAN:  -- because they got



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           383


          1    called back, and in 2004, I guess maybe it

          2    did because --

          3              DR. BURTON:  They didn't get called

          4    back.

          5              MR. FREEDMAN:  They didn't get

          6    called back, but it is not an easy conceptual

          7    problem or a measurement problem to ask

          8    somebody to report something that's not in

          9    their perceptual framework.

         10              MS. FORSYTH:  Right, exactly.  So

         11    it is a cognitive model.

         12              MR. FREEDMAN:  Yeah.

         13              DR. NEERCHAL:  Wait -- I would be

         14    curious ----- out of the 610 data points,

         15    they have chosen to adjust 184 of them and

         16    not adjust the others, so give me those 184

         17    at the ------ and use one of the imputation

         18    type of idea that Tom was talking about and

         19    see how that compares to the adjustment that

         20    you have actually implemented.  That would be

         21    a curious exercise, I think.

         22              DR. BURTON:  Truthfully, though, I



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           384


          1    would have some concern that even the ones

          2    that didn't trigger the adjustment in the

          3    past, that they simply managed to make up

          4    better numbers.

          5              DR. NEERCHAL:  I'm getting to the

          6    actual exercise.

          7              MR. KAPLAN:  We know one

          8    respondent, for example, who we had to

          9    contact for something, but their numbers look

         10    good every single month, and it's because --

         11    we know this from talking with the

         12    respondent -- because the respondent has

         13    picked out what he thinks is the right

         14    power-to-steam ratio, he uses it every month,

         15    and he makes sure the numbers add up right.

         16                   (Laughter)

         17              DR. NEERCHAL:  They're inliers.

         18              MS. KHANNA:  I just -- I had a

         19    comment aside to your comment, which is

         20    environmental economists do this all the

         21    time.  We elicit willingness to pay numbers

         22    for scenic views, things that people don't



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           385


          1    even think about.  So there are techniques,

          2    flawed no doubt, but they are there to try

          3    and elicit data values that are not always

          4    correct.

          5              MR. FREEDMAN:  I'd be interested in

          6    hearing more about that from you because this

          7    is a problem in other areas, too, not just

          8    with ----.

          9              DR. HENGARTNER:  But there's one

         10    big caveat:  To be able to make that -----,

         11    you have all the data collected beforehand,

         12    because it's a regression-type technique.

         13              MS. KHANNA:  No, no, you actually

         14    can ------ to pay estimates.  I mean, ---

         15    collect.  You got out and ask Nick, what

         16    would you be willing to pay for the view

         17    outside here.

         18              MS. FORSYTH:  They don't behave

         19    coherently either, right?

         20              MS. KHANNA:  No, that's what I

         21    said, it's not a method without its problems,

         22    but it's a well-established field.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           386


          1              DR. BURTON:  But there's a material

          2    difference between doing some sort of

          3    willingness to pay and saying how much of

          4    this fuel was used to generate electricity.

          5    There's the willingness to pay is their own

          6    perception, and what you do is you try and

          7    help them refine that perception.  Here their

          8    perception has nothing to do with what is or

          9    is not the correct answer.

         10              MS. KHANNA:  I don't think so.  I

         11    actually think there is a perception because

         12    if you've -- you got to --

         13              DR. BURTON:  Their perception

         14    doesn't matter, though, in generating the

         15    right answer.  There is a certain amount of

         16    that fuel that's going into generating

         17    electricity, not more, not less, that much.

         18              MS. KHANNA:  It depends on how you

         19    think of it.

         20              DR. BURTON:  You just think of

         21    another answer --

         22              MS. KHANNA:  If you think of it



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           387


          1    like an economist.  I've got --

          2              DR. BURTON:  Gee, I hope you do.

          3              MS. KHANNA:  I've got -- apropos,

          4    I'm producing electricity, everything else is

          5    my byproduct, which I didn't really care

          6    about five years ago, but today, because we

          7    have this technology that I can capture this

          8    heat and use process heat, I'll do it, and

          9    that's why the person says zero fuel to

         10    process heat because it's not why he did it.

         11              So it is a question of perceptions,

         12    and I agree that the willingness-to-pay

         13    technique is looking for something different,

         14    but your point was perception and cognition,

         15    and that's what the technique is aimed at

         16    trying to get at and how do they understand

         17    cognitive processes.  And there may be

         18    something there.

         19              MS. FORSYTH:  You could

         20    certainly --

         21              MR. CLEVELAND:  I'd be interested

         22    in hearing more about it.



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382





                                                           388


          1              MS. FORSYTH:  And you certainly

          2    could use it to model the differences between

          3    the estimates you get from these two

          4    different approaches, that willingness to pay

          5    could shed light on the discrepancies.

          6              MR. KAPLAN:  Is that it?

          7              DR. HENGARTNER:  I think so.  Thank

          8    you very much, Stan.

          9              MR. KAPLAN:  Okay, thank you.

         10    Sure.

         11                   (Applause)

         12              DR. HENGARTNER:  This brings us to

         13    almost the end of today's session.  There is

         14    an invitation to the public to comment.

         15              Okay, then it's my pleasure to

         16    adjourn this meeting until tomorrow.  I

         17    remind you we are starting promptly at 8:30.

         18                   (Whereupon, at 5:01 p.m., the

         19                   PROCEEDINGS were adjourned.)

         20                       *  *  *  *  *

         21

         22



                               BETA COURT REPORTING
                               www.betareporting.com
                   202-464-2400                     800-522-2382