AMERICAN STATISTICAL ASSOCIATION (ASA)

                MEETING OF THE ASA COMMITTEE ON ENERGY STATISTICS

                WITH THE ENERGY INFORMATION ADMINISTRATION (EIA)

                    OF THE UNITED STATES DEPARTMENT OF ENERGY

                                Washington, D.C.
                           Thursday, October 20, 2005



                                                             2

          1    COMMITTEE MEMBERS:

          2       NICOLAS HENGARTNER, Chair
                  Los Alamos National Laboratory
          3
                  MARK BERNSTEIN
          4       RAND Corporation

          5       JOHNNY BLAIR
                  Abt Associates
          6
                  MARK BURTON
          7       University of Tennessee

          8       JAE EDMONDS
                  University of Maryland
          9
                  MOSHE FEDER
         10       Research Triangle Institute

         11       BARBARA FORSYTH
                  Westat
         12
                  WALTER HILL
         13       St. Mary's College of Maryland

         14       NAGRAJ NEERCHAL
                  University of Maryland
         15
                  DARIUS SINGPURWALLA
         16       LECG

         17       RANDY SITTER
                  Simon Fraser University
         18
               EIA PERSONNEL:
         19
                  BOB ADLER
         20
                  MARGOT ANDERSON
         21
                  COLLEEN BLESSING
         22
                  TOM BROENE

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                             3

          1    EIA PERSONNEL (CONT'D):

          2       GUY CARUSO

          3       JOHN PAUL DELEY

          4       ALOULOU FAWZI

          5       HOWARD BRADSHER-FREDRICK

          6       STAN FREEDMAN

          7       CAROL FRENCH

          8       DWIGHT FRENCH

          9       BILL GIFFORD

         10       BEHJAT HOJJATI

         11       SUSAN HOLTE

         12       ALETHEA JENNINGS

         13       RAY KASS

         14       ROBERT KING

         15       NANCY KIRKENDALL

         16       TOM LORENZ

         17       RUEY-PYNG LU

         18       PRESTON McDOWNEY

         19       RENEE MILLER

         20       MICHAEL MORRIS

         21       ERIK RASMUSSEN

         22       MARK RODEKOHR

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                             4

          1    EIA PERSONNEL (CONT'D):

          2       TOM RUTHERFORD

          3       BHIMA SASTRI

          4       MARK SCHIPPER

          5       JOHN STAUB

          6       LAWRENCE STROUD

          7       AMY SWEENEY

          8       PHILLIP TSENG

          9       KEN VAGTS

         10       ANGELA VEITCH

         11       SHAWNA WAUGH

         12       PAULA WEIR

         13    ALSO PRESENT:

         14       SUSAN LISS
                  Federal Highway Administration
         15
                  NANCY McGUCKIN
         16       United States Department of Transportation

         17       WILLIAM WEINIG
                  ASA
         18
                  KATHLEEN WERT
         19       ASA

         20

         21

         22                       *  *  *  *  *

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                             5

          1                      C O N T E N T S

          2    AGENDA SESSION:                            PAGE

          3    Administrator's Remarks                      13

          4    Committee Updates:

          5       Since Spring 2005 Meeting                 31

          6       Results of Simulation Study               39
                  for the EIA-914
          7
                  EIA's Regional Short-Term                 46
          8       Energy Outlook

          9    Short-Term Forecasting Performance           71
               Measures and Accuracy Evaluation
         10
               Preserving EIA Trustworthy                  117
         11    Datasets, Model Documentation
               and Contextual History
         12
               Learning from the Past:                     187
         13    Updating Data Quality Efforts

         14    Can Discrepant Estimates                    232
               Be a Good Thing?
         15
               Survey Self-Assessments                     277
         16

         17

         18                       *  *  *  *  *

         19

         20

         21

         22

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                             6

          1                   P R O C E E D I N G S

          2                                             (8:32 a.m.)

          3              DR. HENGARTNER:  Ladies and

          4    gentlemen, oyez, oyez, oyez.  Good morning,

          5    everybody, welcome to this meeting.  This is

          6    an ASA, not an EIA, meeting.  This committee

          7    periodically provides advice to the EIA.

          8    This meeting is open to the public and public

          9    comments are welcome.  Time will be set aside

         10    for comments at the end of each morning and

         11    afternoon sessions.  Written comments are

         12    welcome as well and may be sent to either the

         13    ASA or the EIA.

         14              All attendees including guests and

         15    EIA employees should sign in at the register

         16    in the hall and include their e-mail

         17    addresses.  So there's a little bit more of

         18    the formalities.  The rest rooms are located

         19    either on one end of the hall at this end and

         20    the other end of the hall.

         21              The telephone in this room shares a

         22    line number.  Let me give you that.  It's

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                             7

          1    202-586-3071 and you can also get messages

          2    through the following number if you need,

          3    586-6273 and 586-6202 and probably it might

          4    be a nice gesture to turn off cell phones.

          5              Kathleen Wert with the ASA Meeting

          6    Department is here and available for the

          7    committee members' questions on expense

          8    reimbursements, and I'm sure she will be

          9    happy to help us, she has in the past been

         10    very helpful.  In commenting each committee

         11    member is asked to speak toward a microphone.

         12    The microphones are here if you don't speak

         13    clearly and loudly we will be reminded.  The

         14    transcriber will appreciate that.

         15              The speakers from the audience are

         16    asked to use the microphone that's over

         17    there.  We did have a portable one but

         18    because of the echo I think that might be

         19    better.  Also, if you need help Bill Weinig

         20    will help you with this.

         21              Okay, the announcements.  First I'd

         22    like everybody to introduce themselves in the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                             8

          1    room.  I will start with myself.  My name is

          2    Nick Hengartner and I'm the chair of the

          3    committee.

          4              MS. KIRKENDALL:  Nancy Kirkendall,

          5    Director of the Statistics and Methods Group,

          6    EIA.

          7              MR. CARUSO:  Guy Caruso,

          8    Administrator at EIA.

          9              DR. FEDER:  Moshe Feder, RTI,

         10    committee member.

         11              MR. BERNSTEIN:  Mark Bernstein,

         12    member of the committee.

         13              DR. BURTON:  Mark Burton,

         14    University of Tennessee.

         15              DR. EDMONDS:  I'm Jae Edmonds at

         16    the Pacific Northwest National Laboratory.

         17              MS. FORSYTH:  I'm Barb Forsyth.

         18    I'm a member of the committee.

         19              DR. SITTER:  Randy Sitter, Simon

         20    Fraser University.

         21              MR. McDOWNEY:  Preston McDowney,

         22    Statistics and Methods Group, EIA.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                             9

          1              MR. TSENG:  Phillip Tseng, SMG,

          2    EIA.

          3              MS. BLESSING:  Colleen Blessing,

          4    National Energy Information Center, EIA.

          5              MS. MILLER:  Renee Miller,

          6    Statistics and Methods Group, EIA.

          7              MS. FRENCH:  Carol French, Office

          8    of Oil and Gas, EIA.

          9              MS. VEITCH:  Angela Veitch,

         10    Petroleum Division, EIA.

         11              MR. SASTRI:  I'm Bhima Sastri.  I'm

         12    with the Office of Integrated Analysis and

         13    Forecasting, EIA.

         14              MR. STAUB:  John Staub, EIA.

         15              MS. HOJJATI:  Behjat Hojjati, EIA.

         16              MS. JENNINGS:  Alethea Jennings,

         17    Statistics and Methods Group, EIA.

         18              MR. FAWZI:  Aloulou, Office of

         19    Integrated Analysis and Forecasting, EIA.

         20              MS. WEIR:  Paula Weir, EIA.

         21              MR. LORENZ:  Tom Lorenz, EIA.

         22              MR. BRADSHER-FREDRICK:  Howard

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            10

          1    Bradsher-Fredrick, EIA.

          2              MR. VAGTS:  Ken Vagts, EIA.

          3              MS. HOLTE:  Susan Holte, EIA.

          4              MS. SWEENEY:  Amy Sweeney, EIA.

          5              DR. HENGARTNER:  Thank you very

          6    much.  For your information, under the terms

          7    of the DoE grant to the ASA Nancy Kirkendall

          8    here may chair but must attend each meeting.

          9    And she's authorized to adjourn the meeting

         10    if she determines this to be in the public

         11    interest.  She must approve all meetings of

         12    the advisory committee and every agenda.

         13    Also she may designate a substitute in her

         14    absence.

         15              I think that we have a good agenda.

         16    We've all seen a little bit of what's going

         17    to be on the program.  I'm looking forward to

         18    it.  There are going to be four breakout

         19    sessions and a few presentations here.

         20              We have also one new committee

         21    member I hope to be able to introduce to you

         22    shortly and with both sadness and pleasure

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            11

          1    I'd like to recognize the participation of

          2    two of our former members.  It's going to be

          3    their last session.  One is Mark Bernstein

          4    here and the other Randy Sitter.  Both of

          5    them have been with us for six years and

          6    their contributions have been valuable to the

          7    committee and we'd like to recognize this.

          8              MR. CARUSO:  Thank you, Nick.  Let

          9    me start with Mark since he's the closest

         10    one.  There's couple of small tokens of

         11    appreciation for you, Mark.  There's one from

         12    Nancy and me and this one's from Secretary

         13    Bodman.  So, Mark, thank you.

         14              MR. BERNSTEIN:  Thanks so much.

         15              MR. CARUSO:  We'll miss you here.

         16              MR. WEINIG:  Could you and Mark

         17    come up towards the front so we can get a

         18    couple of additional pictures?  Thank you.

         19              DR. HENGARTNER:  We're thinking of

         20    putting Mark's picture on the wall here.

         21              The second member who's going to

         22    depart us and whose last meeting it is as

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            12

          1    well is Randy Sitter and I'd like also to ask

          2    us to acknowledge him.

          3              DR. SITTER:  Thank you very much.

          4              DR. HENGARTNER:  So just a note on

          5    the program.  The first address this morning

          6    will be done by Guy Caruso on the EIA general

          7    state of affairs.  Then Nancy Kirkendall as

          8    the past director of the EIA Statistics and

          9    Methods Group will tell us about what's going

         10    on.  And other summary presentation will be

         11    from Preston McDowney, Margot Anderson, and

         12    that's it for the locals.  Good.  Now, the

         13    important thing, lunch.

         14              DR. SITTER:  Oh, already?

         15              DR. HENGARTNER:  Lunch will be as

         16    usual served on the first floor at 12:25.

         17    It's in the same corridor.  Just follow me.

         18    I'll find it.  And also there's going to be

         19    dinner tonight in honor of Mark.  We are

         20    going to go to a pub and I know that Bill

         21    Weinig has sampled them and has chosen us the

         22    best pub in DC.  To make life easy for our

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            13

          1    organization can I have a show of hands of

          2    who will be attending?

          3              Thank you very much.  Tomorrow

          4    morning there's going to be breakfast at 7:45

          5    and we will resume the meetings here at 8:30.

          6    Finally, just a reminder, I am the Chair and

          7    my responsibility is to keep us on time and

          8    therefore we're 20 minutes late.  Don't

          9    worry, we'll catch up, so thank you very much

         10    and I'd like to ask Guy to give us the state

         11    of affairs.

         12              MR. CARUSO:  Thank you, Nick, and

         13    good morning to the members of the committee

         14    and everyone else joining us this morning.

         15    It's a pleasure always to welcome the ASA

         16    Energy Statistics Committee and, as I have

         17    said many times, we find the work and advice

         18    of your committee extremely useful and that

         19    will show up in my remarks this morning as

         20    well as presentations throughout the next two

         21    days.  It's been put to some very practical

         22    use just in recent days in our analysis of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            14

          1    the hurricanes and in our short- term outlook

          2    and winter fuels outlook particularly.  It's

          3    always sad to see members of the committee

          4    leave but Mark and Randy have been with us

          5    for six years and they had extremely useful

          6    comments and then some of the other

          7    activities with Randy, some of Randy's

          8    students and other activities, so we wish

          9    them well and we'll be seeing you in other

         10    venues.

         11              Let's very quickly go over what's

         12    been a very busy six months since we've last

         13    met.  Most recently, of course, our services,

         14    our analysis, our data have been under close

         15    scrutiny as the analysis in the aftermath of

         16    both hurricanes Katrina and Rita have had

         17    such a devastating effect on our energy

         18    infrastructure in addition to the more

         19    important personal tragedies of the

         20    individuals involved but we've had a large

         21    number of request for our services, for our

         22    analysis, for our testimony, and just last

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            15

          1    week released our winter fuels outlook, which

          2    we'll talk a little bit more about later, and

          3    one of the things we've put to very good use

          4    is your advice on regional short-term energy

          5    outlook.  I'll present a couple of examples

          6    to that but Margot Anderson, whose office is

          7    responsible for producing that analysis, will

          8    go into more detail later.

          9              And I'll also say a little bit

         10    about the redo of our website which we're

         11    very excited about and again your comments on

         12    that last meeting were taken into account and

         13    you'll see that hopefully by the end of

         14    November when the new website is up and

         15    running.

         16              And couple of comments about the

         17    budget, the last time we mentioned that we

         18    now have an external review team and they are

         19    about to meet for the second time and we'll

         20    talk a little bit about that and I think

         21    that's the highlights.

         22              Increased activity due to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            16

          1    hurricanes, of course, has been the request

          2    we've had on a regular basis for our data and

          3    for our analysis and clearly, as always, it

          4    moves financial markets, it does shape

          5    policy.  It was very instrumental in the

          6    decisions made by Secretary Bodman and

          7    recommendations by Secretary Bodman to the

          8    President about utilization of the SPR and

          9    the activities of the International Energy

         10    Agency.

         11              We've had nine testimonies since

         12    Labor Day before Congress and numerous

         13    briefings and there's been enormous activity

         14    on the website, particularly on the situation

         15    report that Margot Anderson's office

         16    publishes every day, so just when we thought

         17    things couldn't get any busier I think again

         18    we're being asked to provide the latest

         19    information and we're tracking as shown

         20    something like this which we released last

         21    week what's been happening to US crude oil

         22    production in the Gulf of Mexico since the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            17

          1    onset of Katrina and Rita.  And we, of

          2    course, project that out through the winter

          3    and it's instrumental in our analysis of

          4    where we think crude oil prices are going but

          5    perhaps even more critical this winter will

          6    be natural gas, which has suffered, I think,

          7    even a larger blow to the infrastructure than

          8    oil.

          9              It's still about 10 percent of our

         10    gas productions off line as we stand here

         11    today and the restoration is coming perhaps

         12    slower than I think many of us thought it

         13    would be and it's partly because the on-

         14    shore infrastructure for processing the gas

         15    and distributing that gas has had such

         16    extensive damage.

         17              And then, of course, the refinery

         18    situation probably is improving a bit quicker

         19    than the production of oil or natural gas.

         20    Nevertheless, as we've all witnessed as we

         21    pull up to the pump, the high price of

         22    gasoline but probably more important for this

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            18

          1    time of the year, heating oil prices are even

          2    higher than gasoline, which is relatively

          3    unusual for heating oil to be selling at a

          4    premium to gasoline but as you can see by

          5    this chart that the upper, the darker, line

          6    there is heating oil and diesel fuel.  Prices

          7    will stay above gasoline for the foreseeable

          8    future certainly through the winter and

          9    that's largely because of the refineries.

         10    Ten percent of our refineries still remain

         11    offline.

         12              So we're expecting very tight

         13    markets for crude products of particularly

         14    heating oil and jet fuel but gasoline as

         15    well.  And natural gas the average consumer

         16    is going to be paying about $16/1,000 cubic

         17    feet in the US and that's at least a 50

         18    percent increase over what you as a consumer

         19    would have been paying a year ago.

         20              So it's a huge political issue and

         21    I think there's a lot of frustration in

         22    Congress and elsewhere about what to do about

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            19

          1    it because there are very few levers in the

          2    short term especially for natural gas because

          3    there's no world market for natural gas

          4    compared with crude oil products.  So prices

          5    have attracted an enormous increase in the

          6    imports of gasoline but in the case of

          7    natural gas there are no available spare

          8    supplies on the world market.

          9              I mentioned that the work that we

         10    did with you and the advice you gave us on

         11    the regional short-term energy outlook was

         12    invaluable and our August STEO included the

         13    regional projections and had been very useful

         14    in our ability to make specific briefings to

         15    individual congressmen or policy makers about

         16    the different regions.  And so once again the

         17    work of this committee has translated itself

         18    into a better product for EIA and I think a

         19    more useful product in policy analysis and

         20    we've had independent expert reviews looking

         21    at these models and Margot will go into that

         22    in much more detail in a few moments.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            20

          1              This is an example of how we've

          2    been able to utilize that and we now break

          3    down the heating bills, in this case natural

          4    gas, by four regions.  The model itself, we

          5    used nine census regions, as you may recall,

          6    but for purposes of presentation we are using

          7    four regions but the STEO itself goes into

          8    much more detail.  So the richness of the

          9    data and the analysis has really been useful

         10    and, of course, as you all know, every time

         11    you give the customer a little more detail

         12    they want more and it raises more questions.

         13    Yesterday when I presented this to the House

         14    Energy Committee those who lived in different

         15    regions wanted to know why are we paying 55

         16    percent more this year when other regions are

         17    only 29 percent so it generates more

         18    questions and more work for Margot and her

         19    team and she'll talk about that.

         20              In a continuing effort to improve

         21    the longer term outlook we've been working to

         22    improve the National Energy Modeling System,

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            21

          1    NEMS.  We're extending in this year's outlook

          2    our projection period to 2030 for the first

          3    time.  That will be out, if all goes well,

          4    about the end of November with a reference

          5    case and I think it's going to be pretty

          6    interesting because our price assumptions

          7    obviously are going to be much higher this

          8    year than they were last year and there are

          9    some pretty interesting results with respect

         10    to the long-term impact of higher prices of

         11    oil, of all energy but oil and natural gas in

         12    particular, what it means for the mix of

         13    generation for electricity, what it means for

         14    imports and the demand for products and the

         15    economic effect of those prices on what we'd

         16    call nonconventional sources of both oil and

         17    natural gas such as the oil sands of Canada.

         18    Even coal to liquids is breaking some new

         19    modeling ground and we'll be adding also the

         20    latest Manufacturing and Energy Consumption

         21    Survey, MECS.

         22              I didn't want to put anything on

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            22

          1    this chart because it's still very tentative

          2    but we've asked the Secretary and now OMB but

          3    it's still in the very preliminary stages.

          4    We've asked for an substantial increment of

          5    money over the next three years to completely

          6    redo NEMS.  As many of you know, and

          7    certainly Jay was part of that, this is a

          8    modeling system now about 15 years old and

          9    we've been trying to patch it up and you've

         10    seen some of those patches in some of the

         11    sessions we've had with you.  Clearly it

         12    needs a real top to bottom look and if we get

         13    that funding I think we can make a lot of

         14    improvements and incorporate some of the

         15    structural changes that have occurred in the

         16    industry and a number of other things that

         17    we're gleaning from the data and so hopefully

         18    I'll be able to report to you in the spring

         19    meeting that we now have that approval from

         20    OMB and that's the budget for '07 that would

         21    be presented by the President in January.

         22              I just wanted to give you a little

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            23

          1    heads up about that because we'll certainly

          2    be asking for your advice and assistance as

          3    we move forward on that.  It's not just a

          4    question of if but when and if we don't get

          5    as much money as we've asked for we're still

          6    going to try to move, do something, because

          7    it's clear that a number of the components of

          8    NEMS need updating.

          9              The other topic which you've been

         10    directly involved in is the National Natural

         11    Gas Production Survey.  I think it was one of

         12    the first items that we discussed when I got

         13    here three and a half years ago and it's

         14    finally come to fruition.  We now have the

         15    914 data on the website and we'll be running

         16    that in parallel with the existing system for

         17    a while to do our data testing and quality

         18    control and Ken Botts and Nancy's teams have

         19    both worked very hard on this and Preston's

         20    going to talk a bit more about natural gas in

         21    his presentation.  And we're also changing a

         22    little bit on the data for the Texas Natural

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            24

          1    Gas and we'll talk about that in more detail.

          2              I mentioned about the '07 budget.

          3    Well, '06 budget here we are what, three

          4    weeks into the fiscal year and this budget

          5    hasn't been approved yet.  As many of you

          6    know, it's being debated and we're on a

          7    continuing resolution.  What was reported out

          8    of our appropriations committee was about $86

          9    million for '06.  It's up $2 million more

         10    than '05 and keeping our number of FTEs at

         11    369.

         12              The '06 budget obviously is going

         13    to be subject to pretty intense debate

         14    because of the need to pay for the federal

         15    costs in restoration and recovery in the Gulf

         16    Coast so most people are expecting there will

         17    be some impact on everyone's budget so we'll

         18    be working closely with our congressional

         19    appropriation staffers to encourage them to

         20    think of EIA as part of the solution and not

         21    part of the problem.  So maybe we won't

         22    suffer too badly but stay tuned on that one.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            25

          1              I mentioned the website being

          2    redone.  The website hits continue to exceed

          3    our expectations and clearly during September

          4    and thus far in October has been as much

          5    activity as I think we've ever seen and

          6    Margot's office's products of the daily

          7    situation report and this winter fuels

          8    Outlook, STEO, getting a lot of attention.

          9              And I mentioned the external review

         10    Team, I think, last meeting.  We may have

         11    mentioned it but I'm not sure.  Did we,

         12    Nancy?

         13              MS. KIRKENDALL:  We mentioned it.

         14    I think we gave them Denny Ellerman's name as

         15    Chair but I'm not sure we told them about the

         16    other members yet because I don't think that

         17    was fixed yet.

         18              MR. CARUSO:  This external review

         19    Team we're getting, we're anticipating an

         20    extremely useful report coming from Denny

         21    Ellerman's team and we have outstanding

         22    members, Kathy Cooper, who is a former

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            26

          1    Undersecretary of Commerce now back in Dallas

          2    teaching, Jay Hakes, my predecessor as

          3    administrator, Paul Joskow, whom you all know

          4    is the head of the MIT Energy and Environment

          5    Policy Center, and Phil Sharp, now the

          6    Director of Resources for the Future.

          7              We met in July and had a very good

          8    discussion about their views and where we

          9    think that they could be useful in mapping

         10    out where EIA might be going over the next

         11    five years, what our mandate to them was, and

         12    Denny, as he always does, has been very

         13    thorough and even while he's spending a year

         14    abroad on sabbatical getting lots of e-mails

         15    and requests and they're going to meet in

         16    about a month and that will be the critical

         17    bringing together of the team and then after

         18    that will prepare a report for us and it will

         19    be very useful as we look out in our

         20    strategic plan and particularly as we respond

         21    to OMB's request that we look at the whole

         22    program.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            27

          1              They criticized our part submission

          2    because we had submitted individual reviews

          3    of individual components of what EIA does but

          4    we had never done a complete review of what

          5    EIA's mission is, what it should be, are we

          6    doing the right things, so that's what this

          7    is all about and I think it's going to be

          8    extremely useful and with that team that you

          9    see there I can't help but think that it will

         10    be very important to EIA's future planning.

         11              As I mentioned, we're going to have

         12    a new home page in our website and we're

         13    hoping it will be available by the end of

         14    November.  We had invaluable comments from

         15    this group at the last meeting.  It's now

         16    gone through internal testing and review and

         17    I've been looking at it myself over the last

         18    few days and I think it's going to be

         19    extremely popular, even make EIA's website

         20    even more popular.

         21              With that, Nick and members of the

         22    committee, I once again want to thank you for

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            28

          1    everything you've done for EIA and I know

          2    you'll see from the further sessions how

          3    important your advice has been and how we put

          4    it to use and why we know that we've still

          5    got plenty of room to improve and we look

          6    forward to working with you, Nick, and the

          7    rest of the committee.  Thanks.

          8              DR. HENGARTNER:  Thank you very

          9    much.

         10              MR. CARUSO:  I'll be happy to

         11    answer any questions.

         12              DR. HENGARTNER:  The strategy is if

         13    someone in the committee has a question we

         14    have a tent and we lift it.  Questions,

         15    anyone?  I had actually a few questions

         16    concerned with the response to natural

         17    disasters and, more generally, disruption to

         18    the energy infrastructure.  And the first

         19    question is are we actually well equipped to

         20    report in a timely manner such disruptions?

         21    Secondly, are we gathering the right data to

         22    be able to do that?

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            29

          1              MR. CARUSO:  I think EIA itself is

          2    pretty well equipped because we do our weekly

          3    surveys particularly for petroleum and

          4    natural gas.  We have a ready-made network

          5    that we can reach out to in an emergency.  As

          6    you could imagine, in an emergency like this

          7    a number of our respondents were either

          8    evacuated or offline so it was more difficult

          9    but I think we've played a pretty important

         10    role.  The actual emergency response and

         11    operational side of that are the

         12    responsibility of another office within DoE

         13    called Office of Energy Reliability but we've

         14    been feeding our information through them in

         15    some instances and then we have our own

         16    product in the case of the daily situation

         17    report on our website.

         18              But clearly you find out in

         19    situations like this there are gaps and

         20    particularly on the natural gas side much

         21    more difficult because there are 30,000

         22    operators as opposed to the oil side where

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            30

          1    you have a more limited universe.

          2              DR. HENGARTNER:  Yes, Jae.

          3              DR. EDMONDS:  As we go through

          4    further sessions will we be hearing more

          5    about plans for NEMS and the approach that

          6    you're going to be taking as you rethink that

          7    architecture?

          8              MR. CARUSO:  Yes, I would hope,

          9    Jae, by April that the next meeting of this

         10    group that that would be a fairly significant

         11    part of the agenda although you may recall

         12    the last time we did it it took three years.

         13    And that's what we've requested, a three-year

         14    budgetary plan.  The first year we would

         15    actually start using some of our own money in

         16    '06 and try to develop the architecture and

         17    the last time we did it by taking I think it

         18    was between a handful and 10 people out of

         19    their day to day jobs, creating a task force

         20    and designing the NEMS with the assistance of

         21    contractors as you remember.

         22              And we even had a national academy

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            31

          1    review of that and I think it was '91 or '92

          2    that Jim Sweeney, I think, chaired.  So the

          3    answer is yes, but not this meeting, partly

          4    because other than the in-house agreement

          5    that this needs to be done we now need to get

          6    OMB on board.  I'm hopeful.

          7              DR. EDMONDS:  Right, well, I think

          8    it's a very exciting prospect.

          9              DR. HENGARTNER:  Thank you very

         10    much.

         11              MR. CARUSO:  Thank you, Nick.

         12              DR. HENGARTNER:  Next we will hear

         13    from Nancy Kirkendall, who is going to talk

         14    to us about what's going on in the Statistics

         15    and Methods Group at EIA.

         16              MS. KIRKENDALL:  Well, actually

         17    what I'll talk about is some of the

         18    suggestions that you gave us before and what

         19    we've done as a result of them.  A lot of

         20    that is what we're doing in SMG2, though,

         21    since that seems to be how I entertain my

         22    troops.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            32

          1              We've talked to you a number of

          2    times about our frames.  We had an overall

          3    team evaluation of all of our frames last

          4    year and we talked to you about it in the

          5    spring.  As you know we've also had the

          6    Census Bureau that's been evaluating five of

          7    our frames and on Friday you'll be hearing

          8    the last of their reports.  They've completed

          9    their study or it's almost complete at least.

         10              So one of the recommendations you

         11    made after the last time was that we might

         12    not want to evaluate every frame on a

         13    periodic basis every three to five years.

         14    Even you thought that perhaps we should use a

         15    more targeted approach, pick the frames that

         16    are in most need of help and look into them

         17    in more detail and come up with actual

         18    recommendations for improvement.

         19              So this is an ongoing effort.  We

         20    are going to think about your recommendations

         21    and I've asked the frames team, which Howard

         22    Bradsher-Fredrick is chairing, a group of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            33

          1    people from throughout EIA, to think about

          2    what we should be doing in the long term in

          3    terms of frames evaluation.  So we will

          4    contemplate that and come back and talk to

          5    you about it in the spring.

          6              For one thing the Census Bureau's

          7    report is new and we haven't had a chance to

          8    look at it and try to figure out what we

          9    should do about that too so you will have a

         10    chance to give us some input on that

         11    evaluation and what we should do going

         12    forward.

         13              As Guy mentioned, the web usability

         14    testing was very useful.  We've taken most of

         15    your suggestions into account.  These are

         16    some of the specifics.  They've done the

         17    prototype and, as Guy mentioned, we are

         18    actually testing it within EIA.  This is the

         19    redesigned system and what we're supposed to

         20    be doing, we being EIA staff, is finding bugs

         21    in it.  So we're trying to debug it before it

         22    goes live and hopefully the new system will

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            34

          1    go live in November or December so we're

          2    aiming to have something soon.

          3              The external review team, in fact

          4    we came to you I guess was it in the fall

          5    last year when we first asked you what we

          6    should do in terms of getting an external

          7    review?  So this whole effort came from some

          8    of your recommendations.  In the spring you

          9    particularly suggested that we should do a

         10    self-study before we have the external review

         11    team come in.

         12              And we did think about it, we came

         13    up with a plan, we thought perhaps we'd have

         14    senior managers in the EIA answer a subset of

         15    the questions we were asking the committee to

         16    provide that as answers to begin with.  We

         17    debated about how we could possibly do it in

         18    the time frame we had because they were

         19    having their first meeting in July.  And then

         20    when we asked Denny Ellerman if he thought we

         21    should go ahead with this, he's the chair of

         22    the external review team, he said he didn't

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            35

          1    think that we needed to do it.  So we

          2    breathed a great sigh of relief and didn't do

          3    it so we didn't take your suggestion but we

          4    did think about it a lot.  We can do

          5    something more like the preliminary review in

          6    future iterations of the external review

          7    team.

          8              Some of your suggestions, however,

          9    did get rolled into a new document we

         10    prepared particularly for the external review

         11    team.  That's the overview document that

         12    describes EIA and its products and we will

         13    send it to you for a review.  I think it's a

         14    very nice document.  We haven't really had

         15    something that looked at EIA as a whole to

         16    describe what we do and why we do things.

         17    Future thoughts are that we should take that

         18    document and edit it and make it available on

         19    our website so any comments you have on the

         20    document would be most welcome.

         21              This is just, again, the list of

         22    people from the external review team and

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            36

          1    their affiliations.  It's really a great

          2    group.  Denny is fantastic.  I'm chairing the

          3    support team and just this week he sent out

          4    assignments to all of his teammates about

          5    what they're supposed to talk about at their

          6    meeting in November so I think we're going to

          7    have some really interesting results and

          8    we'll talk to you about what their

          9    recommendations are and what we plan to do

         10    about it in the spring.

         11              This is everything else.  You've

         12    given us some recommendations on the short-

         13    term energy outlook and we've accepted and

         14    implemented some of your suggestions and

         15    you're going to hearing a lot more about STEO

         16    from Margot.  We asked you about incentives

         17    in the residential energy consumption survey.

         18    I believe that the incentive experiment has

         19    been approved by OMB and we're going forward

         20    with that.

         21              The way these things go it will

         22    probably be a long time before you hear about

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            37

          1    the outcome of that experiment.  This is a

          2    very large survey, it takes a long time to

          3    run, but we'll tell you the outcome of the

          4    experiment once we get there.

          5              On survey evaluations this was

          6    Brenda Cox talking about coming up with a

          7    template for evaluation of survey programs.

          8    We're going to have her continue to finish

          9    off the petroleum marketing ones and then

         10    we've to digest what we can do with the

         11    ideas.  One of the things that we've already

         12    done with template is to use it as an outline

         13    for documentation for new surveys.  So even

         14    if we could have our documentation in a more

         15    consistent form that's not a bad outcome of

         16    that exercise.

         17              Last time on the EIA 826, the

         18    electric power sales and revenue report, we

         19    talked about an experiment predicting annual

         20    data from annual data so we could actually

         21    find out how various estimation methods

         22    worked.  And your recommendation was that we

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            38

          1    should do a similar experiment to the best we

          2    could with monthly data, which is a grand

          3    idea, we do plan to do that, we haven't done

          4    it yet.

          5              You're going to be hearing a talk

          6    tomorrow morning that is looking at the 826

          7    and the 861 but they're trying to look at how

          8    to tie together time series methods with more

          9    stationary regression methods in that survey.

         10    And that work resulted from an analyst who

         11    looked at our historical data and noticed

         12    some funny bumps in it.

         13              The EIA 914, Guy has talked about

         14    that.  One of the interesting things, I think

         15    it was on a slide but you probably didn't

         16    catch it is in the State of Texas, as you

         17    know, Randy's student did a lot of work.

         18    That state had wonderful data with patterns

         19    that could be used in predicting for the

         20    future and she helped us come up with a nice

         21    way of doing that.

         22              Well, in January 2005 Texas changed

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            39

          1    the way they're reporting data so there is

          2    now a huge change in Texas data and it makes

          3    us even more thankful that we now have the

          4    914 to help us come up with our new

          5    estimation methods.  And with that I'll

          6    introduce Preston, who's going to give you an

          7    update on the 914.  As you recall, he gave

          8    you some simulation results a couple of

          9    meetings ago and you made some

         10    recommendations so this is really just

         11    finishing up the report on what worked and

         12    what didn't work in trying to estimate

         13    natural gas production from a sample survey.

         14    So Preston.

         15              MR. McDOWNEY:  Good morning.  I'm

         16    going to update for the simulation results

         17    for the EIA 914.  In the spring 2004 meeting

         18    Inderjit Kundra introduced the development of

         19    the 914 survey to collect monthly data

         20    considering a probability proportion to size

         21    sample at the time and it goes with the

         22    estimated US production with a coefficient of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            40

          1    variation of 1 percent at the US level and 5

          2    percent at the regional level.

          3              We have seven regions, Texas to

          4    Federal Gulf, Louisiana, New Mexico,

          5    Oklahoma, Wyoming, and the rest of United

          6    States excluding Alaska.  We're going to use

          7    the presumed optimal allocation formula to

          8    determine the sample size based on the

          9    desired CV.  We calculated that the formula

         10    of a sample of 176 companies will give us the

         11    1 percent CV at the US level but we needed

         12    358 companies to get the desired CVs in each

         13    region.

         14              With that sample in 2004 at the

         15    fall meeting I presented the simulation

         16    studies.  We used simulated four estimation

         17    procedures.  The first was the probability

         18    proportion to size estimator, the weighted

         19    least squares estimator, which is the ratio

         20    estimator, ordinary least squares estimator,

         21    and a difference estimator, which uses the

         22    prior, in this case years, response as a

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            41

          1    predictor for the future.

          2              For the weighted least squares and

          3    the ordinary least squares procedures we had

          4    two methods, the first using the noncertainty

          5    group only in estimation procedure and the

          6    second using both the certainty and the

          7    noncertainty groups in estimation procedure.

          8    For both methods we had three variations, one

          9    using all the operators in the method, the

         10    second removing the outliers, and the third

         11    removing outliers and influential variables

         12    observations.

         13              The results of that simulation,

         14    these are the percent errors for the total

         15    natural gas production at the US level.  All

         16    of the procedures, methods, and variations

         17    had a negative bias.  The probability

         18    proportion to size estimator had the smallest

         19    bias but a very large variation and range of

         20    errors.  The weighted least squares, ordinary

         21    least squares and the difference estimator

         22    all had smaller ranges but a higher negative

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            42

          1    bias.  So we concluded in the fall that the

          2    PPS sampler wouldn't give us the desired

          3    accuracy.  The ASA committee recommended

          4    using the Hajek estimator as a nonbiased

          5    estimator.  We decided to use a 90 percent

          6    cut-off sample at the national level and John

          7    Wood made a presentation on the need for a

          8    bias adjustment.

          9              In the spring 2005 meeting John

         10    Wood presented the preliminary January and

         11    February 2005 data.  He demonstrated the

         12    estimation procedure, which is a variation of

         13    the ratio estimator, and he presented the

         14    adjustment for the negative bias.  He

         15    reported a 99.4 percent response rate that's

         16    volume rated.

         17              So after the fall meeting we

         18    started Stage 2.  We had a new frame and we

         19    wanted to see if the frame made a difference

         20    in our simulation procedures.  We also

         21    implemented the Hajek estimator and we

         22    implemented a stratified sample.  And this

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            43

          1    time we only did two regions, Texas and the

          2    Federal Gulf, and they're the two largest

          3    regions.

          4              In the first stage we had annual

          5    data in the frame and the sources were the

          6    EIA-23 survey data.  We had state data and

          7    multiple third party data.  And Stage 2 was

          8    monthly data and this time we had a single

          9    third party as the data source.  The first

         10    time we had the seven regions in a way

         11    integrated so if we picked a company in Texas

         12    if they were put in any other region we took

         13    their data also.  In the second stage we

         14    simulated each region individually so there

         15    was no integration here.

         16              With the new frame we used the same

         17    formula as in the first one but we got

         18    different sample sizes.  In Texas we had 96

         19    companies and 20 in the Federal Gulf.  In

         20    Stage 2 there were 61 companies and 19 in the

         21    Federal Gulf, and that's due just to the

         22    frame.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            44

          1              We had the same four estimation

          2    procedures and we added the Hajek and the

          3    stratified.  This is the Hajek estimator, the

          4    stratified sample versus random stratified

          5    sample with replacement.  We had six non-

          6    certainty strata in Texas, two non-certainty

          7    strata in the Federal Gulf.

          8              We're going to summarize the whole

          9    project by showing the percent errors in the

         10    total estimated regional production and I did

         11    that by just summing all the monthly

         12    estimates within a region.  The stratified

         13    sample had the strongest negative bias and a

         14    fairly large range of errors.  The PPS sample

         15    had a much reduced negative bias but a very

         16    large range of errors.

         17              The Hajek was a little less biased

         18    and smaller range of errors.  The difference

         19    in the model-based methods, the difference

         20    estimator had a very small bias and very

         21    small range of errors.

         22              I'm going to skip to the last two.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            45

          1    These are the model-based methods.  When you

          2    add the certainty and the non-certainty

          3    together you really reduce your range of

          4    errors and moderately improve on the bias as

          5    opposed to when you don't add the certainty

          6    companies.

          7              We get a similar story in the

          8    Federal Gulf.  The stratified sample, that's

          9    the best prospect but there are only two

         10    strata and seven companies in the non-

         11    certainty category.  The Hajek does give you

         12    a much smaller  than the PPS sample

         13    in the Federal Gulf.  The difference

         14    estimator performance is pretty much the same

         15    and the model-based does reduce the range of

         16    errors but, unlike in Texas, actually

         17    increases your bias.

         18              So in general the probability

         19    methods had the largest variation in many

         20    cases especially you need to go back and look

         21    at the months themselves with the most

         22    negative bias.  The difference estimator and

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            46

          1    the model-based estimators with the certainty

          2    and the non-certainty companies had the

          3    smallest variation and least bias.  And we

          4    didn't show them but outlier and influential

          5    variables made little difference in the

          6    results.  So that's my update.

          7              DR. HENGARTNER:  Thank you very

          8    much.  We will now also hear the EIA regional

          9    short-term energy outlook by Margot Anderson.

         10              MS. ANDERSON:  Good morning,

         11    everyone.  I want to thank Nancy and Bill for

         12    inviting me to come and speak with you today

         13    updating you on what we've done to the

         14    regional Short-Term Energy Model.  I was here

         15    in April to talk to you about work that was

         16    currently underway.  As Guy mentioned in his

         17    presentation, it's become a workhorse for us

         18    over the last couple of months.

         19              Back in the halcyon days of August

         20    before the hurricanes when we were just

         21    starting to launch the regional model little

         22    did we know how useful it might be during the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            47

          1    hurricanes and after to look at the aftermath

          2    and to be able to disaggregate some of the

          3    results at a finer level of detail than we

          4    were able to before.

          5              So I want to give you an update.  I

          6    want to recap from 2005.  I want to look at

          7    the schedule because there is still work for

          8    us to do.  I want to give you an overview of

          9    the key components and some of the challenges

         10    that we're facing as we go forward.  I remind

         11    you that the purpose was to expand the

         12    national model into more regional detail than

         13    we had had before.  Clearly we are

         14    constrained by the amount of subregional data

         15    that we have available to put in to the

         16    model.

         17              In many cases we can go to the nine

         18    census regions, as Guy mentioned.  In some

         19    cases we can't break it down into that much

         20    detail but we do try to regionalize to the

         21    extent that we can regionalize for most

         22    energy sectors.  And we did this in the hopes

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            48

          1    of providing additional relevance to the

          2    model, to enhancing our ability to look at

          3    regional supply, demand, and prices, clearly

          4    to answer the kinds of request we get at the

          5    regional level.

          6              Guy is absolutely right.  I think

          7    people would like a household projection.

          8    We're unable to do that.  We get a lot of

          9    requests still, of course, at the state level

         10    and the regions are not fine enough for a lot

         11    of our customers who would like us to go into

         12    more detail.  We just don't have the state

         13    level data to do that in most cases.

         14              We want to add depth, greater

         15    understanding of what's going on.  It's not

         16    just to feed the customers but it's for us to

         17    understand how these markets are interacting

         18    particularly at a regional level.  And there

         19    are lots of disparities in supply, demand,

         20    and price across the regions and we thought

         21    it would be helpful and enrich our analysis

         22    of the short-term energy situation and

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            49

          1    outlook to have a more regionalized model.

          2              So that was our motivation.  We

          3    talked about wanting to launch in the summer

          4    of '05.  I'm happy to report we were able to

          5    launch in August, as I said, the halcyon

          6    days.  Before the hurricanes we thought it

          7    might be smooth sailing, we would launch 70

          8    percent of it in August, we would continue to

          9    do the updating in September and October, in

         10    November we'd have the whole thing ready.

         11              Well, we got hit by Katrina and by

         12    Rita and it meant that we weren't able to do

         13    some of the development that we wanted to and

         14    continue working on the electricity dispatch

         15    model, which is still not operational but

         16    will be shortly, and we wanted to focus more

         17    of our efforts on using the regional model to

         18    put out the September and the October STEO.

         19              In September we did something

         20    unusual because it had been just a few days

         21    since the hurricane.  For one of the first

         22    times ever we did a non-single estimate.  We

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            50

          1    did a forecast that looked at a slow, medium,

          2    and fast recovery, which isn't something we

          3    normally do in the short-term energy outlook.

          4    We tend to have a baseline outlook.  But we

          5    were able to look at three different recovery

          6    patterns because in those first days we

          7    didn't really know where it was going, we

          8    didn't have a good assessment of the damages,

          9    and we didn't want to commit ourselves to a

         10    single baseline forecast so we had three

         11    forecasts.

         12              By the time Rita hit we had about a

         13    good three weeks to figure out what was going

         14    on in the Gulf and the October forecast went

         15    back to a single baseline, a medium recovery

         16    forecast, but again was also coupled with the

         17    winter fuels outlook.  You saw many of the

         18    slides that were used in the winter fuels

         19    outlook in Guy's presentation and we were

         20    able to look at heating expenditures for the

         21    winter in most cases for the nine census

         22    regions, something we had not been able to do

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            51

          1    before.

          2              So, as I said, we're going to get

          3    the electricity component working in the next

          4    couple of months.  The delay was really due

          5    to the fact that the hurricanes just

          6    sidestepped us from the development angle to

          7    the use angle.  We are ongoing in our expert

          8    review.  We have all of that documentation

          9    ready to be reviewed and we are working with

         10    SMG to put together a panel to review the

         11    documentation.  We made the decision to go

         12    live without having the documentation

         13    reviewed.

         14              We just didn't feel that we could

         15    do them in the sequence that we would

         16    normally do, review and then go live.  We

         17    wanted to go live and we will look at this as

         18    a living model.  If we need to make the

         19    adjustments based on comments we get back

         20    from the expert reviewers we're more than

         21    happy to do so but we really wanted to

         22    implement the regional model so the expert

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            52

          1    review will occur in '06, Bill, '06.

          2              The recap, as I said earlier, we've

          3    broken it into these sectors and we have the

          4    regional components associated with each

          5    sector, far more information than we had

          6    before.  We also have an expanded query

          7    system associated with the STEO models so it

          8    allows you to go in and get the data that we

          9    use for the subregions that we're looking at.

         10              We did get rid of the interactive

         11    model that people could use to generate their

         12    own estimates in STEO.  It's too unwieldy to

         13    make the regional model available in that

         14    kind of a PC format.  If we ever get there we

         15    might be able to make it available sometime

         16    in the future but for now we have not made

         17    that feature available so that is one service

         18    we're not providing to customers but we're

         19    giving you all the detail in terms of the

         20    data that we can and will be as transparent

         21    as possible in the website with the

         22    documentation.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            53

          1              Regional detail varies.  I

          2    mentioned this earlier.  It just depends on

          3    what region and what field you're looking at

          4    to determine how much data we can get to a

          5    more disaggregated level.  We talked last

          6    time about there were strong regional

          7    differences, availability of data, more

          8    limited data for propane and for heating

          9    fuels.

         10              For heating oil it doesn't matter

         11    as much because a lot of those fuels are used

         12    in a very concentrated area.  So sometimes it

         13    doesn't matter as much that you don't have

         14    the regional detail that you'd like to have.

         15    These are the regions.  I'm sorry it doesn't

         16    show up better.  It didn't color on the STEO

         17    page.  But this gives you an idea.  We have

         18    the census regions and census divisions and

         19    this is how we've broken out in the winter

         20    fuels outlook in the October STEO so you can

         21    look at prices and expenditures for those

         22    regions which has been very helpful this last

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            54

          1    time out because we were able, as I said, to

          2    go into more detail which matters more when

          3    prices were higher and supplies were tighter.

          4    In other years where you might have had

          5    expected a more normal year there probably

          6    would have been less emphasis on the regions

          7    because there wasn't as dramatic a change

          8    between this year and last year in terms of

          9    expenditures.

         10              And I think that showed up nicely

         11    on Guy's presentation where he showed the

         12    damage associated with the hurricanes and the

         13    expectations on prices that are attributed in

         14    part to the hurricanes, in part to the

         15    general situation we find ourselves in tight

         16    global markets and tight domestic markets.

         17              But the winter fuels outlook

         18    mattered more this year because of these

         19    dramatic changes in the energy sector so it

         20    was nice to be able to have additional

         21    information for people to make decisions on

         22    and for policy makers to consider where there

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            55

          1    might be regional disparities associated with

          2    heating expenditures.

          3              Observations, I've been at EIA a

          4    year, about 14 months, and so I'm looking at

          5    it as somebody that jumped into the

          6    development of the regional model while it

          7    was ongoing.  I was not there at the

          8    beginning when a lot of the work was really

          9    developed so I certainly can't take any

         10    credit for that nor could I take credit for a

         11    lot of the day to day work that goes on in

         12    it.  That is attributable to Mark Rodekohr

         13    and Dave Costello, who are part of the STEO

         14    team and lead the STEO team.

         15              It is resource-intensive to do this

         16    kind of work.  That sounds like an obvious

         17    observation, but I think probably more

         18    resource-intensive than we might have

         19    anticipated when we walked down this path;

         20    however, the value that you get out of a

         21    regional model is quite large.

         22              You asked earlier about are we

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            56

          1    capable of looking at some of these energy

          2    emergencies.  The regional models can allow

          3    us to do that better.  As we develop it even

          4    more and more it's even going to add to that

          5    capability.

          6              So we use something called the

          7    rules of thumb when we look at international

          8    supply disruptions and it's a fairly simple

          9    way to look at supply shortages and how they

         10    might affect GDP in the United States and how

         11    they might affect oil prices.  This model

         12    would be able to help us look at disruptions

         13    whether they are happening in the United

         14    States due to a weather event or somewhere

         15    else; it's almost irrelevant where it

         16    happens.

         17              The regional model will allow us to

         18    look at some of these contingencies.  So I

         19    think this is a powerful tool particularly,

         20    because we didn't have this capability on the

         21    natural gas side or on the electricity side

         22    which we will have in the not too distant

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            57

          1    future. So, again, I said earlier, looking at

          2    regional disparities and conducting

          3    contingency analysis, it's going to be a very

          4    powerful tool.

          5              And it can also be a fulcrum for

          6    data collection and data sharing within EIA.

          7    My office is very dependent upon all the

          8    other offices in EIA to get the data to put

          9    into the model and so it means that there

         10    often has to be more cooperation and

         11    coordination across the offices to make sure

         12    that we're getting the data that we need in a

         13    timely fashion to put into the STEO.

         14              So I think it enhances the

         15    collectiveness of the data and how the data

         16    are used in a forecasting and modeling

         17    framework within EIA and has raised a lot of

         18    questions about data that are collected and

         19    the turn-around of data to get them prepared

         20    and model ready for the STEO model.  So it's

         21    raised some issues and some challenges that

         22    we have here in EIA on dealing with some of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            58

          1    the data issues of making sure that STEO is

          2    ready to go every month without fail.  We

          3    can't delay STEO.  It's on the same kind of

          4    schedule that our weekly data series are on.

          5    It's not optional.  It has to go out every

          6    month.

          7              I urge you to look at the regional

          8    STEO.  You can look at it in the new web page

          9    in the not too distant feature.  The STEO is

         10    now completely redesigned and we think it's

         11    much easier to navigate through the tables,

         12    through the text, through the links.  The

         13    query system is there.  We've added some

         14    interesting little features that I think make

         15    it a much more user-friendly product and a

         16    much more attractive product.  It's always

         17    been a high quality product and remains so

         18    today, I think, and the regional has only

         19    enhanced it.

         20              So I can answer any questions.  Do

         21    you want me to answer questions on the

         22    regional model at this point before I go into

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            59

          1    forecasting?

          2              DR. HENGARTNER:  Yes, Moshe.

          3              DR. FEDER:  Margot, do the regional

          4    STEOs add up to the national STEO and how is

          5    it accomplished if so?

          6              MS. ANDERSON:  Yes and no.  We

          7    still have a national forecast and sometimes

          8    it's an adding up and sometimes it's a

          9    national forecast that is decomposed down.

         10    So it isn't such a seamless system that every

         11    little regional area adds up to a national

         12    estimate.  Mark, stop me when I go into an

         13    area I don't need to go into.

         14              DR. FEDER:  Whether you aggregate

         15    up or disaggregate down it always adds up,

         16    right?  There's no --

         17              MS. ANDERSON:  Well, it has to.  I

         18    mean, there is no separate national forecast

         19    and then we match it to a separate system

         20    that's a regional forecast.  They're an

         21    integrated system and so the real question is

         22    are we getting a different national forecast

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            60

          1    than we would've gotten without the regional

          2    model.

          3              DR. FEDER:  That's a good question.

          4              MS. ANDERSON:  That's an

          5    interesting question.  We haven't answered it

          6    yet.  Mark --

          7              MR. RODEKOHR:  Yes, you are.

          8              MS. ANDERSON:  Thank you, well, we

          9    are.

         10              MR. RODEKOHR:  Yes, we do add

         11    everything up but the regional area is

         12    differed by the fuel.  For example, petroleum

         13    is based on a pad structure and natural gas

         14    is based upon a census division structure.

         15    But we do add it up and it's not a top-down

         16    notion where you get a national forecast and

         17    you share it out.  It's the other way around.

         18    We get the regional forecast and we build

         19    them up.

         20              MS. ANDERSON:  But the question

         21    about whether it's the same as we got with

         22    just the national model.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            61

          1              MR. RODEKOHR:  When you add up

          2    things over certain time spectrum they're not

          3    a lot different in many respects.  A good

          4    example is what Guy did here with the fuel

          5    outlook.  When you look at the regional

          6    differences on weather patterns and you add

          7    them all up it really doesn't make a lot of

          8    difference.  But it's important to the

          9    regions themselves as to those difference.  I

         10    don't know if that's helpful.

         11              MS. ANDERSON:  If we could really

         12    disaggregate, looking at forecast errors

         13    before the regional model and forecast errors

         14    after the regional model, that might help you

         15    determine levels of accuracy when we were

         16    using just the national model or whether we

         17    were using just the regional model.  What

         18    matters now is the model that we're using is

         19    as accurate as we can possibly get it.

         20              MR. RODEKOHR:  Can I say one more

         21    statement?  It may be easier to get a much

         22    better accurate forecast at a national level

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            62

          1    because all sorts of things are balancing out

          2    but from a regional perspective it really is

          3    important to get those things better and

          4    that's what we could not do before.

          5              DR. HENGARTNER:  Mark?

          6              MR. BERNSTEIN:  I might say I

          7    really liked when you did the different

          8    scenarios after Katrina stuff and I would

          9    encourage you to think about continuing

         10    something like that rather than giving a

         11    single estimate.  Even in the short term as

         12    you go along there is always something that

         13    you look out the next couple of months and

         14    say well, there's something we're not

         15    particularly sure of but there is some

         16    uncertainty critical out there in the next

         17    few months and it would be nice to recognize

         18    that and perhaps let that one float.  And it

         19    may be different ones in different months

         20    where is this winter going to be particularly

         21    cold or something else.

         22              And so I would encourage you to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            63

          1    think about having a range even in the short-

          2    term outlook.  So that's just one comment.

          3              The question is in the short- term

          4    energy outlook are the regions the same for

          5    fuels and electricity or are they like NEMS,

          6    the regions are different?

          7              MS. ANDERSON:  No, the regions are

          8    different but, getting back to the first

          9    point, it's a source of some debate within

         10    EIA or at least friendly conversation about

         11    how much our forecasting models ought to try

         12    and hit a bull's-eye.  Now, the parenthetical

         13    here is that, of course, in the winter fuels

         14    outlook we do a colder and a warmer scenario

         15    so that we give estimates depending on

         16    whether we expect it to be 10 percent warmer

         17    or colder from the NOAA forecast.

         18              But many other forecasters will put

         19    out reports that might look at cleverly

         20    titled worst case scenario or intensive

         21    hurricane season forecast and so they will

         22    give this range of estimates depending on

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            64

          1    what they think might be something that would

          2    disrupt the market over the coming months.

          3    Certainly we know that these markets are not

          4    smooth sailing and if you look back one year,

          5    you look back five years, you look back 10

          6    years there is always some unanticipated

          7    event.  And to be able, maybe, to anticipate

          8    those better or to give a range on estimates

          9    is certainly something that the NEMS model

         10    does on a regular basis.

         11              They run 30 cases or so every year.

         12    They may highlight the reference case and

         13    talk mostly about the reference case but

         14    they're always doing these other scenarios.

         15    We have tended not to do that in the STEO

         16    framework mostly because of time constraints

         17    because it has to get out every month and

         18    then the text constraint of being able to

         19    talk about those other scenarios but I don't

         20    think it's an issue that we've put to rest

         21    yet.

         22              I'm always concerned about making

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            65

          1    it appear as if we're trying to hit a

          2    bull's-eye every month when in fact what

          3    we're really trying to do is generate a

          4    reasonable story and a discussion about what

          5    we think is impacting the markets and to give

          6    a fairly accurate assessment of what we think

          7    the interactions are and what the prices are

          8    going to be and demands, et cetera.

          9              So I think there might be some

         10    value in that because often people will take

         11    the estimate and really misuse the point

         12    estimate and they won't be nuanced by what we

         13    all know as forecasters and researchers that

         14    there are some wide bands here.  And it's our

         15    obligation to educate people about the

         16    uncertainty that might be associated with

         17    these estimates.  So I think it's a point

         18    well taken.

         19              DR. HENGARTNER:  Yes.

         20              MR. RODEKOHR:  One last point, if

         21    you do look in the last 100 STEOs you will

         22    notice on world oil prices there is a red

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            66

          1    band and it has always been there.  And so

          2    you can't say we've ignored that uncertainty.

          3    People just don't focus on it.

          4              MS. ANDERSON:  The confidence

          5    interval that we put on some of the prices.

          6              MR. RODEKOHR:  Yes, on some of the

          7    prices, right.

          8              MS. ANDERSON:  But we don't focus

          9    on it either.  I mean, we have the red band

         10    but we don't talk about what that really

         11    means and what's motivating us to do that and

         12    how people should interpret that information

         13    so point well taken.

         14              DR. SITTER:  How are you generating

         15    that red band?  Is that scenario variability?

         16              MS. ANDERSON:  Mark.

         17              MR. RODEKOHR:  The red band is

         18    generated upon the standard deviations from

         19    the model we use to do prices.  Now, again,

         20    that's the model we use but the real point

         21    estimate on the prices is done by more of a

         22    Delphi approach where we sit around and we

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            67

          1    talk to each other and we get that but we do

          2    have a model and it does give you two

          3    standard deviations away from it and that's

          4    where the red band is generated from.

          5              And I would like to say if you went

          6    back historically and looked at how well we

          7    did versus that red band the model's probably

          8    pretty close.  In other words 95 percent of

          9    the time you're going to be within that band

         10    understanding the limitations, there are

         11    monthly prices, there are averages, et

         12    cetera.  So that's how we do it.

         13              MS. ANDERSON:  But we could test

         14    that.

         15              MR. RODEKOHR:  Well, it's very

         16    time-consuming, yes.

         17              DR. HENGARTNER:  What I liked about

         18    Mark's initial comment that started this

         19    discussion is that in fact the model is

         20    changing.  You're changing the assumption in

         21    the model and so this red band although tied

         22    to standard deviation is useful.  The

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            68

          1    question is how sensitive is it to model

          2    departures that can come from external shocks

          3    like the hurricanes.

          4              MS. ANDERSON:  Well, the red band

          5    is only on a couple of price variables that

          6    are exogenous to the model.

          7              MR. RODEKOHR:  That's right,

          8    exactly.  It's not all-encompassing.  That's

          9    correct.  It's on the most price variable

         10    that a lot of people come to the STEO for a

         11    WTI.

         12              DR. SITTER:  Right, that was the

         13    point I was trying to make is that's not

         14    really addressing this issue.  This is a

         15    completely different measure of variability

         16    than the issue of different scenarios.

         17    Different scenarios could actually be handled

         18    in a more Bayesian framework if you wanted

         19    to.

         20              MS. ANDERSON:  If we use the STEO

         21    mostly to generate the short-term energy

         22    forecast I'm hopeful and my vision is that we

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            69

          1    would be able to use the STEO to do analysis

          2    outside of just generating the forecast so

          3    you would see that you might put in a couple

          4    of different scenarios of what-ifs associated

          5    with either some kind of an event or a change

          6    in policy and be able to look at a short-term

          7    impact by using the short-term model.

          8              We've tended to use it for the STEO

          9    product itself as opposed to a workhorse of

         10    energy analysis and then in that case you

         11    would certainly want to explore alternatives

         12    but it's a point well taken.

         13              DR. HENGARTNER:  Well, thank you

         14    very much.

         15              MS. ANDERSON:  You're welcome.

         16              DR. HENGARTNER:  Before continuing

         17    I'd like to introduce to you Tom Rutherford,

         18    who just joined us.  He is a new committee

         19    member from Colorado, correct?

         20              MR. RUTHERFORD:  Michigan.

         21              DR. HENGARTNER:  So welcome very

         22    much to our committee.  Those members of the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            70

          1    audience who have joined us since we started,

          2    if you could please introduce you into the

          3    microphone for the records, please.

          4              MR. SINGPURWALLA:  Darius

          5    Singpurwalla, LECG.

          6              MR. BLAIR:  Johnny Blair, Abt

          7    Associates.

          8              MR. SCHIPPER:  Mark Schipper, EIA.

          9              MS. McGUCKIN:  Nancy McGuckin,

         10    Department of Transportation.

         11              MS. WAUGH:  Shawna Waugh,

         12    Statistics and Methods Group.

         13              MS. TAYLOR:  Yvonne Taylor, EIA.

         14              MR. RODEKOHR:  Mark Rodekohr, EIA.

         15              DR. HENGARTNER:  Thank you very

         16    much.  Before continuing I again want to ask

         17    if any member of the committee has some

         18    questions for Nancy's or Preston's

         19    presentations.  I know it's a little bit back

         20    but I'd like to wrap up the update for the

         21    committee section.

         22              DR. SITTER:  I have one question

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            71

          1    for Preston.  Is it possible to get a

          2    write-up together with the previous write-up

          3    because this was too terse for me to

          4    understand what has been done since the

          5    previous write-up.

          6              MR. McDOWNEY:  Sure.

          7              DR. SITTER:  Thanks.

          8              DR. HENGARTNER:  Anything else?

          9    Okay, thank you very much.  So let's

         10    continue.  The next presentation will be on

         11    short-term forecasting performance measure

         12    and accuracy evaluation by Margot, yes.

         13              MS. ANDERSON:  I'm going to talk

         14    about the STEO performance indicators

         15    project.  I have my partners in crime here as

         16    usual, Mark Rodekohr and Mike Morris, who

         17    have done all the yeoman's work on this.  I'm

         18    just here to try and provide context and put

         19    into an approach that not only talks about

         20    some of the technical matters but why we're

         21    doing it and what we hope to do with it in

         22    the future.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            72

          1              This is a growing program.  It came

          2    out of a conversation that we had last time

          3    we here, in April, about what to do, about

          4    really understanding how well we're doing

          5    with our forecast.  And it isn't so much to

          6    be a report card and to say you got an A or a

          7    B or you were absolutely right or absolutely

          8    wrong.  But we really needed a way to do some

          9    performance indicators to help us improve

         10    what is now a much more complicated model as

         11    well as to take a look at how well we're

         12    doing.  So it's really to serve a couple of

         13    purposes to develop a systematic way to look

         14    at performance indicators and forecast

         15    errors.

         16              So when I was here last in April we

         17    discussed the necessity for developing some

         18    diagnostic tools for gauging forecast

         19    accuracy and we've got 1,500 variables in

         20    this thing now that we could look at and that

         21    we're forecasting and do we look at all of

         22    them, do we look at every conceivable way to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            73

          1    measure forecast accuracy or do diagnostics,

          2    and what might be the minimum set to give us

          3    the maximum information?  So we had some

          4    conversations about that.

          5              We discussed how to use the

          6    information not only to troubleshoot data

          7    problems but to improve the model, maintain

          8    confidence in EIA's forecasting capability.

          9    We tend not to report forecast errors on a

         10    regular basis for STEO to the general public.

         11    We do keep some track internally to meet some

         12    OMB requirements but we have not tended of

         13    late to have a program that is transparent to

         14    the users of STEO about how are you doing.

         15              We also discussed how we might

         16    share these results.  It can be burdensome to

         17    do this every month and to have a long report

         18    about what we've got and what it means but

         19    there may be a low cost way or a relatively

         20    straightforward way to tell our readership

         21    what we are showing in terms of forecast

         22    errors and diagnostics.  We think that's a

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            74

          1    good goal.

          2              So what did we do?  Well, since

          3    April while we were working on the regional

          4    model we also talked about doing some

          5    preliminary work on looking at some forecast

          6    errors and choosing some metrics and taking a

          7    look at what they were telling us.  These are

          8    not final results.  These are not looking at

          9    every variable.  I think we looked at about

         10    40 variables.  I'm going to show you about 15

         11    to show you the kinds of metrics we're

         12    looking at and what some initial results are.

         13              So it's an ex-post analysis.  We

         14    examine how are the projections compare to

         15    the data but we don't go back and revise the

         16    data in this particular presentation.

         17    Whether we need to do that in the future is

         18    determined by how much did the data change

         19    from revision to revision to revision.

         20    That's time-intensive.

         21              This analysis starts with monthly

         22    forecast in '03.  We can go back longer on

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            75

          1    many series.  We just did '03 for this

          2    exercise.  And what we're going to be looking

          3    at is our one-month, our six-month, and our

          4    12-month ahead forecast errors so I'm not

          5    going to be talking about any diagnostic

          6    tools of looking at outliers and

          7    distributional concerns about the data.  This

          8    is primarily a discussion about forecast

          9    errors.

         10              We looked at a couple of standard

         11    measures, we looked at the mean absolute

         12    error, we looked at the mean absolute percent

         13    error, we looked at the bias to tell us the

         14    percent of times that we were under or over-

         15    predicting.  In some cases we're under-

         16    predicting 100 percent of the time but only

         17    by a little bit so you want to look at a

         18    couple of things to determine how well you're

         19    doing.  We looked at an F-statistic to look

         20    at the statistical significance between the

         21    actual data and the predicted data to tell us

         22    something about how off we were, given the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            76

          1    incredible variation in some of the data

          2    series alone.

          3              So some initial observations, no

          4    surprise to anybody in this room.  The price

          5    data, particularly the international price

          6    data like WTI, are really hard to predict and

          7    we're usually wrong.  Sometimes we're wrong

          8    by a little bit, sometimes we're wrong by a

          9    quite a bit more, but we're pretty good and

         10    pretty consistent when we're looking at

         11    variables that are within our control in the

         12    sense that we have modeled the markets

         13    carefully.

         14              The demand, the consumption

         15    variables, for most sectors are relatively

         16    stable.  The price variables are the ones

         17    that give us fits.  So those are some initial

         18    observations.  Obviously those that are in a

         19    self-contained US system we tend to be able

         20    to model that and where it tends to be less

         21    variability there tends to be more knowledge.

         22    The data also tend to be better.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            77

          1              I hope you can read this.  Mark,

          2    you're the one that's most far but can you

          3    read this at all?

          4              MR. RODEKOHR:  Barely, yes.

          5              MS. ANDERSON:  Well, squint.  I'll

          6    try and walk you through here.  What we're

          7    looking at, the month ahead is the 1, the 6,

          8    the 12.  The first three columns are the

          9    absolute percent errors that tell you on

         10    average how much you're off.  Then there's

         11    the bias that looks at the percent of time

         12    that you're under or over and then the F-

         13    statistic, which is testing the difference

         14    between the data series and the forecasted

         15    series to determine whether you are

         16    statistically different.  If so you better go

         17    look and find out what's going on.

         18              These are the prices, the WTI and

         19    motor gasoline and heating oil and the

         20    natural gas prices, and you can see that the

         21    bias in some cases is 100 percent.  So for

         22    WTI since 2003 100 percent of the time we've

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            78

          1    been under-predicting.  When we're one month

          2    ahead we're under-predicting 23 percent.  The

          3    error widens the further out we go.  So not

          4    so bad one month ahead, not so great six

          5    months ahead, even worse 12 months ahead.

          6    And that's the case on the bias as well.  Our

          7    predictions are progressively off as we go

          8    out for the one or the six or the 12 month

          9    ahead.

         10              Again, that's not really surprising

         11    and I would venture to say that if you were

         12    looking at other forecasters that are playing

         13    this game they've got a similar track record.

         14    That doesn't mean we can't do better and in

         15    fact we should be able to use this

         16    information to circle back into the modeling

         17    framework and say if we're under-predicting

         18    all the time maybe we ought to examine what

         19    our priors are about why we're under-

         20    predicting.

         21              If we're under-predicting just a

         22    little bit, 1 or percent, I tend to be less

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            79

          1    worried about it.  If we're under-predicting

          2    by these larger percentages that's something

          3    that we need to examine in how we're coming

          4    up with some of these price variables.

          5    Again, the WTI is determined exogenously,

          6    outside of the model.  The others are related

          7    to WTI so they feed off wherever we start

          8    with WTI.  While natural gas has its own

          9    markets and prices are much more determined

         10    within US borders it is still often pulled up

         11    or triggered by what's going on in WTI.

         12              So this is giving us some insight

         13    into how we're doing.  And one of the

         14    questions that I'm hoping you guys can help

         15    us with is to say well, why aren't you

         16    looking at these four other statistics.  Why

         17    didn't you do this kind of analysis?  We

         18    would like some feedback on what might be

         19    other approaches that we could take to

         20    looking at these variables that would lend

         21    themselves to fairly easy interpretation so

         22    that if we put them out on a website

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            80

          1    customers would get an easy understanding of

          2    what this meant.  We don't really want to

          3    make this purely an academic type of

          4    exercise.  Jae.

          5              DR. EDMONDS:  Margot, would you

          6    just remind me what's WTI again?

          7              MS. ANDERSON:  It's the West Texas

          8    intermediate crude price.  It's a benchmark

          9    for crude oil prices.

         10              DR. EDMONDS:  Thank you.

         11              MS. ANDERSON:  Motor gasoline is

         12    what you get at the pump.  The blue line is

         13    the one-month, the next dotted line is the

         14    six-month, and, of course, the 12-month is

         15    the shorter line, less at the series. You all

         16    have these in your handouts.  I'm giving you

         17    some more statistics about what we're looking

         18    at.  The solid line is zero so it's showing

         19    you how much above and beyond we are

         20    predicting depending on how far out we go.

         21              Again, it's not clear to me whether

         22    the best estimate is the one-month ahead or

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            81

          1    the six-month ahead.  Is it the 12 month

          2    ahead?  What is the best measure of a

          3    forecast error if you don't want to flood

          4    your audience with every conceivable

          5    permutation of every kind of statistic or

          6    variable to look at?  So we have some

          7    deciding to do about that minimal set of

          8    variables and statistics to give maximum

          9    information.

         10              There are gasoline prices again.

         11    This is just really a picture of what that

         12    summary table showed you.  Mike, correct me

         13    if I'm wrong.  The scales on these are all

         14    different so your eyeball can't compare one

         15    to another but the price variables, again,

         16    have the higher F-statistics than some of the

         17    demand variables, which I will get to.

         18              Wellhead natural gas prices, this

         19    is indicative of what a volatile market this

         20    has been over the last couple of years and

         21    how difficult it has been to anticipate these

         22    changes in the natural gas market and I think

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            82

          1    that this picture really demonstrates that

          2    story.

          3              A couple of demand variables, we

          4    looked at natural gas and looked at

          5    electricity sales data.  Again, it looks a

          6    whole lot better in terms of the F-statistic.

          7    The bias is telling us that as we move out we

          8    were over-predicting on the natural gas

          9    demand.  I suppose this is intuitive.  The

         10    further out you go the more difficult it is.

         11    I guess that makes some sense.  If the one-

         12    month aheads are really based on last month

         13    you may tend to always do better one month

         14    ahead than you would six months ahead or one

         15    year ahead.  I'm thankful I'm not in John

         16    Conti's shoes because he has to predict 20

         17    years out and that's a difficult thing to do.

         18              We've got pictures of these as well

         19    of the petroleum demand.  Again, we're much

         20    closer to where we need to be, which is zero,

         21    which is, as I said earlier, on these demand

         22    and consumption variables we do tend to do

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            83

          1    better, on natural gas demand a bit more off

          2    the mark related to what I said earlier about

          3    the vast changes in this market.

          4              Total electricity demand, our

          5    regional model will help us with this.  None

          6    of these are regional, by the way.  These are

          7    all national level estimates.  It goes back

          8    to the question that we were talking about

          9    earlier and what Mark was saying about the

         10    ability of the regional models to hit their

         11    mark.  We haven't yet incorporated into this

         12    regional forecast errors because we don't

         13    have the time series to do so but eventually

         14    we would want to look at how the regional

         15    models are doing as well.  Right now it's

         16    just on the national side.

         17              So we are in the early stages of

         18    this and we are looking for help and looking

         19    for feedback on what we can do.  We want to

         20    write this up in a paper.  As we move on we

         21    want to write it up in a more academic paper

         22    and justify why we're picking what we're

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            84

          1    picking and being able to show a bit more on

          2    what these data look like in terms of their

          3    distributions, in terms of outliers, and in

          4    terms of just some basic diagnostic tools of

          5    looking at the data series.

          6              I think being able to automate a

          7    lot of this will really help us now that we

          8    have this much larger model and to pick out

          9    some anomalies that are inevitably going to

         10    creep in when we have over 12,000 variables

         11    that are in the model, maybe predicting

         12    1,400, so it's a very complicated model.

         13              So what statistic should we look

         14    at?  What variables should we look at?  The

         15    obvious ones are on the front page every day

         16    but there are a lot of other interesting

         17    variables that can provide tremendous insight

         18    as to how well the model is doing that ought

         19    to be added to our menu of variables that we

         20    want to check out.  It's, again, not just for

         21    public consumption.  It's to feed back into

         22    making the model a better model.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            85

          1              We certainly want to focus on the

          2    areas where there are significant error and

          3    where we can do something about it.  In some

          4    cases there is not much we can do about it.

          5    There may not be that many variables or

          6    structural equations in the model that we can

          7    change depending on where the variable comes

          8    from if it's determined exogenously.  But

          9    there are certainly some ways that we can

         10    consider options to reduce the uncertainty or

         11    the forecast error by adding variables.

         12              There is seasonality in the models

         13    and we haven't adjusted for that in these

         14    forecast errors.  And it might be wise if we

         15    talked a little bit about the value of doing

         16    some turning point analysis.  There are quite

         17    a few turning points in these kinds of data

         18    and some of these metrics are not so great

         19    when you get close to a turning point.  And

         20    so maybe doing turning point analysis might

         21    be something that would be useful on some of

         22    these series so that's something to consider

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            86

          1    as well.

          2              And we've got to figure out a way

          3    to publish this without creating a huge

          4    burden on the staff to do this but creating

          5    as much transparency as we can.  I think

          6    we're obligated to our customer to let them

          7    know how well we do and we hope that that

          8    would engender others in this game to play

          9    along.  There are lots of people forecasting

         10    and there are lots of people that don't have

         11    the credibility that we have and don't have

         12    the obligation to be transparent and

         13    responsive.

         14              And so I think this is a part of

         15    EIA's overall agenda, to reveal what we can

         16    reveal and to keep the discussion open and to

         17    generate feedback, solicit feedback, and make

         18    the adjustments where we can.  These are

         19    official government statistics and we need to

         20    feel confident of them.  I've said all along

         21    when we started this process that it wasn't

         22    so we could have an uh-huh and point a finger

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            87

          1    at somebody and say see, I knew you were

          2    wrong.

          3              I have great confidence in this

          4    model and I think we're doing probably pretty

          5    well compared to what others do and I think

          6    we're pretty accurate to the extent that we

          7    can be.  There is always room for improvement

          8    but I would like to have a tool to be able to

          9    say not only am I relatively confident but

         10    that Guy Caruso gets up somewhere and talks

         11    about the results from this model that he has

         12    the confidence that he needs because he has

         13    the statistics behind him to show that what

         14    we're doing is as good as we can do given the

         15    volatility of these markets and the sometimes

         16    unpredictability that goes on in these

         17    markets.  So we're looking to you guys to

         18    help us out on making this a better program.

         19              Mike or Mark, did you want to add

         20    anything to this presentation?

         21              MR. RODEKOHR:  Just one little

         22    thing, I think exactly what you said is

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            88

          1    right.  We used to publish the forecast

          2    errors and I agree we ought to started it up

          3    again.  Budget reasons stopped it but

          4    nevertheless I do think that that adds

          5    something to what other people don't do.  You

          6    will find that the other private consultants

          7    will always tell you when they were right.

          8    They will never tell you when they were

          9    wrong.  And to the extent we can add an

         10    historical record and be transparent about it

         11    and say here is what we did right and here is

         12    what we did wrong I think that's a value

         13    added to what we're doing.  That's all.

         14              Michael.

         15              MR. MORRIS:  Again, my name is

         16    Michael Morris.  I work with the STEO team

         17    and I look at petroleum demand issues

         18    primarily.  I don't want to duplicate or

         19    replicate what Margot or Mark has said.  I

         20    just say that I have output for the 37 series

         21    that we currently cover which are all

         22    national in nature.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            89

          1              We'll do the regional, perhaps,

          2    starting in a year or two from now when we

          3    have more of a track record behind us.  And

          4    if anybody wants to look at my wonderful F-

          5    statistics and biases and all the rest of it

          6    I'd be happy to show you so you get a flavor

          7    of what this is about and I'd be happy to

          8    stick around and show it to any interested

          9    parties.  Thank you, that's all I have to

         10    say.

         11              MS. ANDERSON:  Thank you, Michael.

         12              MR. MORRIS:  Any other questions

         13    for me or anybody else on this?

         14              DR. HENGARTNER:  So actually, we

         15    have two discussants right now and you're one

         16    of them, Tom.

         17              MR. RUTHERFORD:  Sure.

         18              DR. HENGARTNER:  So you may as well

         19    start with your comments and you can weave in

         20    your questions and then Moshe also will have

         21    some comments and I probably will open up the

         22    discussion to the floor after that.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            90

          1              MR. RUTHERFORD:  I'm a newcomer

          2    here so I wasn't completely certain of what

          3    my role was going to be but I'm very happy to

          4    be here and I really found the presentation,

          5    the discussion, and the broader thing quite

          6    stimulating.  I should also say at the outset

          7    as a caveat that I'm really not a time series

          8    forecaster in terms of the details of this.

          9    I'm more of a user of this type of work for

         10    the purpose of constructing policy-oriented

         11    models.

         12              Let me first begin by being a bit

         13    of the devil's advocate and just reflecting a

         14    bit on what the mission is because there is a

         15    lot of discussion about customers as though

         16    were a profit-making forecasting enterprise.

         17    And I would submit that my view of it, and

         18    this is just my personal view, is that the

         19    purpose, the main reason, for having a model

         20    and paying attention to what the model does

         21    and how it forecasts is to have a better

         22    handle on what data are important to be

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            91

          1    collecting.

          2              If we think about private goods and

          3    public goods that are being produced within

          4    the Department of Energy the public component

          5    is the information, what data do we need to

          6    generate in order that someone could do a

          7    good forecast.  And by doing a forecasting

          8    model you get a good handle on what data is

          9    important and could be useful.  There is this

         10    public component to that in that once you

         11    produce the data then it's useful to all

         12    sorts of people.

         13              And clearly the types of things

         14    you're forecasting there's a strong incentive

         15    for the private sector to want to do this

         16    right as well.  So if you can produce the

         17    data that permit the private sector to do

         18    these forecasts well you've accomplished a

         19    large fraction of your objective.  I don't

         20    think you have to worry about you're the one

         21    who is forecasting for mom and pop's delivery

         22    service and that if they don't know what the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            92

          1    prices are they're going to go out of

          2    business.

          3              The fact of the matter is that mom

          4    and pop's store has their own incentive to

          5    look at what information they need and what

          6    forecasts they need to buy and there should

          7    be a market that they have for going out to

          8    buy the forecast; however, the private

          9    sector, no individual firm that's doing the

         10    forecasting has the incentive to go out and

         11    do the surveys to collect the data they need

         12    to do a good forecast so the purpose of the

         13    government agency being involved is to

         14    identify what data is important and to help

         15    do it.

         16              Having said that, I think we also

         17    have to be a little bit humble.  I don't know

         18    how many of you enjoy watching hurricane

         19    forecasts.  This has been a good year for

         20    doing that.  It's one of the things that

         21    whenever I watch hurricane forecasting done

         22    by modelers I'm intrigued by the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            93

          1    interrogation they do with the modelers about

          2    these things.

          3              I'm completely humbled by the

          4    precision with which they can forecast these

          5    things.  I mean, there are four different

          6    models and all of them predict that four

          7    hours from now this storm is going to turn 90

          8    degrees to the right.  How did they do that?

          9    Well, it's because the physical sciences,

         10    storms and so on, they don't care what other

         11    agents anticipate the storm is going to do,

         12    right.  There is much less simultaneity.

         13              Thunder clouds don't care about

         14    what the forecaster's telling them is going

         15    to happen whereas in our game, the social

         16    sciences, you forecast something, you affect

         17    the market, it goes on.  It's a very

         18    difficult game.  So I think we have to

         19    humbled about what out objective is and we

         20    can't hold the accomplishments of NOAA and

         21    the various weather forecasting units up as a

         22    metric for evaluating the precision of our

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            94

          1    work because we're in a much harder

          2    environment here.

          3              So those are my two basic points,

          4    that what you're doing is hard but you also

          5    think about the mission and that the idea of

          6    using the model as a means of identifying

          7    what data are crucial to the enterprise is

          8    really the most useful thing you can do with

          9    this.  And so that'd be my basic reaction

         10    about what to put in the model.

         11              I think that this analysis if you

         12    can address it from the standpoint of what

         13    additional data, had we had it, would have

         14    improved the forecast and using it as a means

         15    of doing a cost-benefit calculation for what

         16    additional surveys you need to do or what

         17    additional data items you need would be a

         18    very useful way to approach the problem in

         19    thinking about what the ultimate mission is.

         20              I didn't have enough time to read

         21    any of the background materials about the

         22    model and the details.  It looks like more of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            95

          1    a statistical exercise.  As a structural

          2    modeler I'm a little bit nervous therefore

          3    about the uses of the model for doing policy-

          4    oriented work because I think that you have

          5    to think about what instruments, what levers,

          6    are in there and who knows what and how do

          7    decisions get made so I have some questions

          8    about that.  I didn't completely understand

          9    but hopefully in the next year or something

         10    I'll learn more about it and be able to give

         11    you more feedback.

         12              So, just to conclude, I apologize

         13    for not being able to tell you more about

         14    specifically what variables to put in and so

         15    on but I can just share with you my general

         16    perspective on what the objective is.

         17              DR. HENGARTNER:  Thank you very

         18    much.  Moshe?

         19              DR. FEDER:  Actually I've been

         20    preempted by what you said about Wilma

         21    because when I was watching last night I was

         22    wondering how come two models say it will go

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            96

          1    up to Maine and another model said that it'll

          2    go to the Yucatan Peninsula in Mexico and

          3    actually I had Wilma written here so I'm

          4    going to cross this out.

          5              And then I wanted to talk about

          6    regional forecasting errors and Margot said

          7    we are doing it yet but I'm going to talk

          8    about it anyway because it relates to one of

          9    my suggestions.  But I just want to focus on

         10    one question, which is the forecasting errors

         11    and how to measure them, what to do about

         12    them, and I'm going to explain it just a

         13    little bit.

         14              And one technical comment, I don't

         15    know if other statisticians in the audience

         16    were confused by the term "bias" that you

         17    heard there.  It's an indicator of bias but

         18    it's not what we call bias so I suggest

         19    another name.  It's the frequency you are

         20    under- or overestimating, and when I looked

         21    at the slides because I didn't have the

         22    benefit of the actual presentation I said how

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            97

          1    come it's minus 100 percent, I don't

          2    understand and so I can cross out this too

          3    and a lot of it.  But I wanted to mention a

          4    few things here.  Actually what Mark said

          5    about the way they do the estimation Mark was

          6    talking about the way you construct a model

          7    and I share what you said, Tom, about how you

          8    construct your models.

          9              Demographers actually do the same

         10    thing when they provide us, statisticians,

         11    with population projections.  They use a lot

         12    of discussions, brainstorming.  They have

         13    inputs from statisticians which we do have

         14    standards ready which they put into the model

         15    and then they use their judgment to come with

         16    actually, according to John Row, very good

         17    estimates so I'm sure your method works well

         18    and what we can contribute is a little bit on

         19    how to estimate the accuracy.  And what I

         20    think you should do, and I looked at some of

         21    the plots, looks to me like there could be

         22    some serial correlation which you could

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            98

          1    exploit and the other part, which you said

          2    you are not doing yet, original patterns.

          3              So I would look at the prediction

          4    errors both temporally and specially and see

          5    is there anything that we are missing, and

          6    that could contribute to finding out which

          7    data you need to augment your model.  Say,

          8    why is it that always in the northeast we

          9    underestimate and why do we overestimate in

         10    California or why is it that we have a long

         11    period of underestimation and than we have

         12    overestimation.  Could we use auto-regressive

         13    models or something else?  And you mentioned

         14    seasonality, which was another thing that I

         15    crossed out because I was going to talk to

         16    you about it.

         17              So in the interest of catching up

         18    with time just one more comment, mean

         19    absolute error is a good measure but I think

         20    mean squared error is another tool.  Actually

         21    the F-statistic may be answering that, Mark.

         22    Is it?

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                            99

          1              MR. RODEKOHR:  It is.

          2              DR. HENGARTNER:  Thank you very

          3    much, Moshe.  Yes, Jae.

          4              DR. EDMONDS:  I thought that was a

          5    fascinating presentation.  I think it's

          6    actually interesting that everybody brings up

          7    this hurricane analog and I think it is

          8    actually an interesting analog.  I think the

          9    hurricane modeling is very similar to your

         10    problem and that is it's a specialty, it's

         11    really a short-term forecast.  It's a much

         12    more of a statistically based model.  And

         13    then after they do that they get in a room

         14    and they say do we believe any of this

         15    whereas the weather models are structural

         16    models.

         17              Weather models are based on physics

         18    and hurricane models are based on statistics

         19    and judgment.  And in fact I remember sitting

         20    at the beach in South Carolina and there was

         21    a hurricane coming and we're looking at the

         22    weather forecasts and the weather forecasts

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           100

          1    were clear skies, low wind, and then the next

          2    part of the weather forecast was we're

          3    evacuating.  And that's because the hurricane

          4    is a subgrid phenomenon and so it does not

          5    show up in the weather model.  It is just not

          6    there so the weather model is totally

          7    incapable of dealing with it.

          8              But I have actually a real question

          9    which I think may be helpful.  When you were

         10    looking at the forecasts and comparing to the

         11    observed ex-post I presume that you were

         12    looking at the forecast as given and the ex-

         13    post?

         14              MS. ANDERSON:  Yes.

         15              DR. EDMONDS:  My hypothesis is that

         16    the WTI, which I now know what it means, and

         17    the other variables are strongly correlated

         18    to an exogenous input, which would be the

         19    world oil price.  Is the world oil price an

         20    input?

         21              MS. ANDERSON:  No, the world oil

         22    price is determined outside of the model.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           101

          1              DR. EDMONDS:  That's what I mean.

          2    It's an input to the system.

          3              MS. ANDERSON:  It's an input to the

          4    system, yes.

          5              DR. EDMONDS:  It's an input to the

          6    system and so you are taking that as given.

          7    My question is what would this have --

          8              MR. RODEKOHR:  I don't want to

          9    interrupt you but the world oil price is a

         10    basket of about roughly 300 crude oil types.

         11    What we're forecasting in our short-term

         12    outlook is a representative price for crude

         13    oil that Americans pay.  The world oil price

         14    is not an input into the WTI.  What we are

         15    forecasting is the WTI price.

         16              MS. ANDERSON:  Right, but a lot of

         17    it is coming at the output from the model.

         18              DR. EDMONDS:  Let me just say the

         19    question I was going to ask.  If you went

         20    back and put in all the real input variables

         21    and ask the question how would my forecast

         22    have changed if I had known exactly what the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           102

          1    world market was doing and you'd gotten all

          2    that right my hypothesis is that you would

          3    look really good.

          4              MS. ANDERSON:  Absolutely.

          5              DR. EDMONDS:  Anyway is that

          6    tendency to have under-forecast the WTI and

          7    the other variables simply because you were

          8    in a period at which the input values were

          9    actually under and in fact if you went back

         10    and you put in the real values whether you'd

         11    look better.  That's the question.

         12              MS. ANDERSON:  And I think, Mark,

         13    if we'd gone back before '03 I bet the

         14    forecast would have been more stable.

         15              MR. RODEKOHR:  I think there is a

         16    little bit of confusion here, though, and I'd

         17    like to try to clear it up.  When we do the

         18    world oil price or WTI or IRAP projections we

         19    don't use a model.  We use inputs from

         20    different models and we use judgment.  That's

         21    what determines the forecast.  So there is no

         22    precise model that we use.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           103

          1              And I know what you're getting at.

          2    You're saying if you had a model and you went

          3    and put it back, actual events, would you get

          4    it closer?  And the answer is generally yes

          5    but because it's a lot of judgment there is

          6    no model per se in this instance.

          7              Let's think petroleum demand.

          8    There is a model and if you went back and put

          9    into that model actual weather, actual GDP

         10    growth, as opposed to what we assumed will we

         11    get it more accurate?  Answer is yes.

         12              MS. ANDERSON:  But even the world

         13    oil price that we're using for the Delphic

         14    method and we're still under-predicting it,

         15    other if the model is just our brains somehow

         16    or other we're not taking something into

         17    consideration and it goes back to this data

         18    issue of what's missing from our Delphic

         19    method that we're not taking into

         20    consideration that might dampen the error on

         21    that much less adding variables to all of

         22    these other models.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           104

          1              DR. EDMONDS:  Right, so it has to

          2    do with is your problem in the STEO.

          3              MS. ANDERSON:  Some of it is, some

          4    of it isn't.

          5              DR. EDMONDS:  Or is it in fact that

          6    you needed a better a better way of feeding

          7    the STEO?

          8              MS. ANDERSON:  Well, back to your

          9    question about correlation on these petroleum

         10    product prices are often all driven by this

         11    Delphic method of WTI.  Even though there are

         12    models for the motor gasoline price there are

         13    more components within the model that help us

         14    determine what that is but we know a large

         15    component of that is driven off of whatever

         16    we're deciding in the Delphic process so they

         17    are correlated.  We can explain more what's

         18    going on in the motor gasoline situation than

         19    we really can with the equations for a world

         20    oil price.

         21              MR. RODEKOHR:  For example, if I

         22    know when the hurricanes were coming, what

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           105

          1    damage it was doing, if I had known who was

          2    going to blow up a pipeline in Iraq or

          3    somewhere else, or when the Argentineans went

          4    out on strike or when the Nigerians went out

          5    on strike, then yes, I could do a better job.

          6    But I don't know that and so you're stuck.

          7              DR. EDMONDS:  Right, but the

          8    question I had had to do with partitioning.

          9    If you think about drawing the boundary

         10    around your short-term forecasting tool is

         11    the problem inside that box or is the problem

         12    that you've really got a world out there

         13    where you're taking inputs and you've got to

         14    live with some fundamental uncertainties

         15    around it.  And so this apparent bias may in

         16    fact just be an artifact of the fact that

         17    during that period you were having this ramp-

         18    up for a lot of reasons that were going on

         19    and a part of it --

         20              MR. RODEKOHR:  You can't just do

         21    it.

         22              DR. EDMONDS:  Exactly, and so

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           106

          1    actually inside the box is working great.

          2              MS. ANDERSON:  It's working much

          3    better.  That's for sure, yeah.

          4              DR. EDMONDS:  And so I think that

          5    little diagnostic will help you to know

          6    whether you've got a serial correlation

          7    problem in the model or whether it's outside

          8    the model.

          9              MS. ANDERSON:  And that's why you

         10    break them up because you knew there is a

         11    whole set of variables that we're probably

         12    doing the best we possibly can.  You'd want

         13    to investigate that but all these

         14    international price variables are not really

         15    within the system so you don't want to

         16    penalize yourself for something that may not

         17    get any better than what you're doing right

         18    now.  And we focus mostly on the variables

         19    that may be more under your modeler's

         20    control.

         21              MR. RODEKOHR:  I agree but, as I

         22    was trying to say, there are so many random

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           107

          1    events in this world that it --

          2              MR. CARUSO:  Could I?

          3              DR. HENGARTNER:  Yes.

          4              MR. CARUSO:  The random events,

          5    obviously we can't predict them.  But

          6    probably a better example for what Jae's

          7    point is in responding to Tom is two years

          8    ago most of the world thought that in 2004

          9    the world oil market would grow by 1-1 and a

         10    half million barrels a day in '04 which would

         11    have been a 1 and a half percent growth.

         12    What happened was China grew on its own by 1

         13    million barrels a day in 2004.  No one

         14    predicted that because the data was bad on

         15    China.

         16              If we had known China was going to

         17    grow a million barrels a day in 2004 we

         18    probably would have still under-forecasted

         19    the price of WTI but we probably would have

         20    gotten closer.  So that's a perfect example

         21    of why if we had better data in that

         22    particular one variable, which is critical.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           108

          1    It's, of course, external to the model.

          2              DR. HENGARTNER:  Thank you very

          3    much.  Mark.

          4              MR. BERNSTEIN:  I think you're

          5    going to find that there is somewhat more

          6    than that external data that's going to go on

          7    for the reason for under-forecasting prices.

          8    I actually do have a weather forecasting

          9    story which I'm not going to tell anyone

         10    because we're running behind but maybe later.

         11    But, Margot, you mentioned that you think

         12    you're doing as good or better than private

         13    forecasters and one of the things that you

         14    might ask yourself is are they also

         15    underestimating prices on a regular basis.

         16              And I know you guys get some of

         17    those forecasts and it wouldn't be

         18    necessarily too difficult to ask yourself

         19    that quick question and this is my guess

         20    because we're actually doing a report on

         21    gasoline prices and why gasoline prices don't

         22    seem to match the equilibrium estimates that

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           109

          1    economists try to make on them is that there

          2    is fundamentally something missing in the

          3    models.  I'm not sure what they are yet.

          4              One of the things is futures and

          5    options and trader behavior and things like

          6    that which affect the markets in very

          7    unpredictable ways but may be getting more

          8    predictable over time at least in the bias in

          9    the direction.  Our observations have been

         10    that in most cases gasoline prices have been

         11    under-forecasted across a variety of

         12    different types of forecasts and so there is

         13    something missing from that and I would

         14    encourage you to look to see if everybody is

         15    doing it but then fundamentally thinking

         16    about what goes beyond the model that you

         17    can't really model and how much of that is

         18    driving prices.

         19              What we've seen over the last

         20    couple of years is impact of futures and

         21    options and the trader behavior having more

         22    and more of an impact on gasoline prices than

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           110

          1    they had, say, three or four years ago and as

          2    that becomes more prevalent it's going to be

          3    harder to predict that type of behavior.

          4              And so it's something just to keep

          5    in mind as you're thinking about evaluating

          6    these things.  What we're actually doing is

          7    we're building a production/supply curve for

          8    California for gasoline and looking at

          9    demands and then comparing the actual prices

         10    to what an equilibrium would tell you should

         11    have happened under those conditions.  After

         12    we get that done we'll show it to you.  It

         13    may give you some insights as to what's

         14    beyond the model.

         15              DR. HENGARTNER:  Randy has been

         16    very patient.

         17              DR. SITTER:  It's just a very short

         18    comment, I hope.  Tom has talked about your

         19    focus should be more on what data you should

         20    collect.  And Jae commented on the fact that

         21    there are certain things completely out of

         22    your control and it's hopeless to try and

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           111

          1    model.  Conditional on those you're not going

          2    to have much chance.

          3              But some of the comments from your

          4    colleagues are defending that in a sense so I

          5    think Thomas is correct.  Your focus may not

          6    be correct.  We're doing the best we can to

          7    predict and trying to somehow defend that and

          8    really that's not necessarily the point.

          9              I think that what you're doing is

         10    using one summary statistic and what can be

         11    done here and what Jae was getting is that if

         12    you break down the sources of where your

         13    errors are coming from you'll better

         14    understand what data you're missing or where

         15    your model structure may be wrong or needs

         16    some more components.  And how well you are

         17    or aren't doing right now is more about what

         18    you should be doing and not how you have

         19    done.  I just get a sense that that's not

         20    really the focus.

         21              MS. ANDERSON:  No, I agree and I

         22    hope that I said that I don't think that it's

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           112

          1    about hitting us hard.  I think that's the

          2    wrong way to market, looking at accuracy or

          3    diagnostics, because I think it's about using

          4    feedback to improve the model.  I hadn't

          5    really thought about it in terms of the data

          6    that you're missing.  I'm thinking of it as a

          7    specification of the model.

          8              We ultimately can't get away from

          9    trying to make the model better at predicting

         10    what it's doing but the idea, and this gets

         11    back to doing scenarios, I don't think that

         12    we ought to have our primary goal as saying

         13    we hit a target and then to flagellate

         14    ourselves when we're off the target.  It's

         15    really to have really good robust structural

         16    models.

         17              But how do you get at that unless

         18    you're looking at a forecast accuracy

         19    measure?  It can be a diagnostic tool to open

         20    up the model to tell you what you might do

         21    better.  And I don't know how else to do that

         22    unless I look at these kinds of metrics.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           113

          1              DR. SITTER:  No, I think the

          2    metrics are fine.  I just think that the

          3    process of the exercise of investigating why

          4    they're giving you a particular pattern is to

          5    break it down.  Like you talked about, for

          6    example, the WTI price percolates down into

          7    the others so if you had the WTI price

          8    correct would you have gotten the others

          9    correct?  Would they have been satisfactorily

         10    correct?

         11              If that's true then you can focus

         12    on why your WTI forecast is not correct.  And

         13    I think that Jae's comment was that basically

         14    he can even go one step further.  I mean, if

         15    you'd had the correct inputs from things you

         16    don't really control or events like that

         17    would you in fact have got it correct?  If

         18    that's true then stop beating yourself about

         19    the model and start focusing on the things

         20    that you are in control of.

         21              And Mark said that that's probably

         22    not going to be entirely the case but still

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           114

          1    these kinds of exercises will give you a much

          2    better idea of what proportion of your

          3    problem is coming from what source and then

          4    you use a Pareto principle if you want to get

          5    better; that is, the things you can fix that

          6    are causing the most problems you fix first.

          7    That's all.

          8              MS. ANDERSON:  Helpful, thank you.

          9              DR. HENGARTNER:  Thank you.  Yes.

         10              MR. FAWZI:  My name is Aloulou.

         11    I'm from OIAF.  Whether you're tracking the

         12    natural gas production report, the price, and

         13    the deviation between different indexes by

         14    region and why they deviate compared to NYMEX

         15    or Henry Hub so --

         16              MS. ANDERSON:  I can't answer that

         17    but I think there is somebody in our

         18    organization that's been working on natural

         19    gas prices and how they relate to NYMEX.

         20    There are some folks that are actually

         21    looking at that issue.  Bill  can

         22    probably help you much more on that question

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           115

          1    than I think we could but people are trying

          2    to answer this question, is NYMEX a good

          3    predictor of natural gas prices.

          4              MS. KIRKENDALL:  Don't we have a

          5    session coming up tomorrow?  I think we have

          6    a session on that tomorrow, a breakout

          7    session.

          8              MS. ANDERSON:  It may be less of a

          9    STEO question than it is a broader question

         10    about natural gas prices.

         11              DR. HENGARTNER:  Thank you very

         12    much.  I like to go to the break, one of

         13    those important parts of the meeting.  I want

         14    to remind you that after the break we are

         15    going to go to breakout sessions.  Mark,

         16    Johnny, Mark, Moshe, Tom, Susan, and Randy

         17    are going to go to Room 8E and those who I

         18    haven't named are going to come downstairs to

         19    Room 5E, that's the fifth floor, 069, and

         20    I'll try to gather you before we leave.  So

         21    let's take a 15-minute break.  Thank you.

         22                   (Recess)

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           116

          1              MR. WEINIG:  Ladies and gentlemen,

          2    I'd like to introduce Mark Schipper with the

          3    Energy Information Administration, Energy

          4    Markets and End Use, and Mark will be talking

          5    about vehicle energy use what he did and what

          6    it tells.  Mark?

          7              MR. SCHIPPER:  Good morning.

          8              MS. KIRKENDALL:  Mark, before we

          9    start this is supposed to be a discussion

         10    session so I would like to get some of you

         11    folks from the audience to come here and sit

         12    around the table.  One would be Dwight French

         13    and anybody else from your staff that you

         14    think ought to come down.

         15              MR. SCHIPPER:  How about some DoT

         16    folks?

         17              MS. KIRKENDALL:  DoT folks would be

         18    good too.

         19              MR. FRENCH:  Dwight French, Energy

         20    Information Administration.

         21              MS. LISS:  Susan Liss, Federal

         22    Highway Administration.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           117

          1              MR. GIFFORD:  Will Gifford, EIA.

          2              MR. BROENE:  Tom Broene, EIA.

          3              MR. ADLER:  Bob Adler, EIA.

          4              MR. FREEDMAN:  Stan Freedman, EIA.

          5              MR. SCHIPPER:  Is that everyone?

          6    Good morning.  Thank you for the opportunity

          7    to talk to you today about a new product that

          8    EIA has produced which is out of the

          9    mainstream of what the consumption surveys

         10    have produced in the past because we

         11    hopefully did some work on vehicle energy

         12    use; however, it was a study that did not

         13    originate here at EIA.  For this reason I

         14    have some acknowledgment for the Federal

         15    Highway Administration, which conducted the

         16    initial survey that EIA then used to augment

         17    with energy-related data.

         18              As part of the Federal Highway

         19    Administration I think they coupled with the

         20    Bureau of Transportation Statistics and the

         21    National Highway Traffic Safety

         22    Administration to conduct the National

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           118

          1    Household Travel Survey and DoE's Oak Ridge

          2    National Laboratory was very helpful in

          3    estimating vehicle miles traveled which EIA

          4    subsequently used and modified as needed so I

          5    just want to acknowledge their help in all

          6    this.

          7              The reason we undertook this work,

          8    I gave a quick slide in terms of what

          9    transportation energy use has done in this

         10    country since 1949 to the latest number was

         11    2003; however, I think the 2004 Annual Energy

         12    Review is out.  Transportation has increased

         13    its share of the energy in this country with

         14    a couple of pauses in it looks like in the

         15    '74, and the '79 to '82 period and '90-'91.

         16    And the latest was 2001, which was the year

         17    the National Household Travel Survey was

         18    conducted, but I just want to give an

         19    overview as to one reason why we're looking

         20    at transportation especially as it relates to

         21    oil because petroleum in terms of motor

         22    gasoline is about two-thirds of the petroleum

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           119

          1    use of the transportation sector and what it

          2    boils down to is light duty vehicles and how

          3    they're used and the energy consumed by them

          4    is important for the nation.

          5              And basically why we're interested,

          6    my office as part of EMEU conducts some

          7    demand side energy surveys.  Basically we

          8    work on the demand side of the energy

          9    equation.  We survey residential buildings,

         10    manufacturing plants, and commercial

         11    buildings and up until 1994 we conducted a

         12    residential transportation study.  That

         13    started in '83 and looked like it was

         14    conducted every three years until 1994 when

         15    it was discontinued.  After that there was a

         16    dark period where there were no

         17    transportation statistics in terms of light

         18    duty vehicles at EIA.

         19              However, my office decided to

         20    investigate using the 2001 National Household

         21    Travel Survey as a replacement for that data

         22    series and through the next release is the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           120

          1    National Personal Travel Survey which DoT

          2    plans to implement in 2008.  I put a question

          3    mark by that because funding is always an

          4    unknown when it comes to these large surveys.

          5              We are working with the folks at

          6    Federal Highway.  I think that's the reason

          7    for the name change is that it's now a

          8    Federal Highway Administration product only

          9    and so it's been renamed the National

         10    Personal Travel Survey.

         11              What we did, basically we augmented

         12    the National Household Travel Survey with

         13    energy-related data.  What that means is

         14    we've put in fuel economy estimates for the

         15    vehicles, we imputed a fuel type, we also

         16    imputed the energy consumption based on fuel

         17    economy and vehicle miles traveled and then

         18    through data series available here at EIA we

         19    embedded energy prices to calculate energy

         20    costs for the year.

         21              A little example here, our energy

         22    consumption estimate was essentially an

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           121

          1    on-road, in-use fuel economy measure as the

          2    inverse of that multiplied by VMT at an

          3    annual level.  There was a little bit more

          4    involved and that's in the paper in how that

          5    number is calculated but in a general

          6    nutshell that's how it was calculated.

          7              Our data sources, as I mentioned,

          8    the fuel economy estimates came from the

          9    National Transportation Safety

         10    Administration, their corporate average fuel

         11    economy program, and because they were a

         12    participant in the NHTSA survey they were

         13    more than willing to provide us with that

         14    data series unlike the Environmental

         15    Protection Agency, which had the similar

         16    data, however because of restrictions we're

         17    unable to release that to EIA for our use.

         18              What that meant is that EIA had to

         19    do additional work because there are

         20    different calculations used by the corporate

         21    average fuel economy program to give credit

         22    to flex fuel vehicles, to alternative fuel

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           122

          1    vehicles, which goes through more of a

          2    political process in obtaining credits and

          3    how those credits are added over time.  So

          4    EIA had to go through a process of removing

          5    all that information and unfortunately it

          6    took a very long time to do; however, we were

          7    able to do so with the help of the corporate

          8    average fuel economy folks.

          9              And EIA's data experts came into

         10    play because the corporate average fuel

         11    economy program didn't come into existence

         12    until 1978 so anything that was pre-1978

         13    model years had to rely on EIA's previous

         14    residential transportation studies and some

         15    data experts who just had a good feel for

         16    fuel economy measures and that would be

         17    Dwight French, who sits over here on my left.

         18              The fuel price data by month by

         19    state came from EIA supplier surveys.  I've

         20    listed the two here.  I believe they're still

         21    going in terms of their coverage.  I think

         22    they're even expanding their coverage to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           123

          1    select cities within states and so whether we

          2    can subsequent to that 2008 study use that

          3    information we can evaluate whether that is a

          4    possibility.  And the federal, state, and

          5    local excise taxes came from FHWA.

          6              I provide here a little schematic

          7    for how it was all put together.  Basically

          8    we have the three databases from the fuel

          9    economy estimates from the EPA, NHTSA, the

         10    safety administration, and the NHTS which

         11    provides the vehicle miles traveled which was

         12    the reason I threw EIA on here is because we

         13    modified those VMT estimates to represent a

         14    possession of the year.  In other words DoT

         15    provides an annual estimate for VMT; however,

         16    EIA was only interested in how long that

         17    vehicle was possessed by the household and

         18    the VMT associated with that possession.

         19              The EIA provided the retail price

         20    information and essentially, to put these

         21    boxes together, the work that EIA did is

         22    embedded in this red box where we essentially

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           124

          1    select a vehicle from the corporate average

          2    fuel economy database, adjust its fuel

          3    economy to on-road, in-use values, and then

          4    merge that with VMT estimates for the actual

          5    possession to calculate energy consumption,

          6    which is then subsequently used to calculate

          7    vehicle fuel expenditures from the retail

          8    fuel price database.

          9              Essentially that whole process

         10    comes under linking process of one to many.

         11    Take an example of our Volkswagen Jetta 2001

         12    automobile.  That's all the information that

         13    we're provided from the NHTS sample

         14    respondent so when we link that to the CAFE

         15    program, which has a host of information, a

         16    rich data source in terms of its number of

         17    cylinders, engine size, fuel type, we come up

         18    with I think there are around eight different

         19    configurations of a Volkswagen Jetta 2001

         20    model year ranging from a fuel economy

         21    standard of 52.6 miles per gallon, which is a

         22    turbo diesel, not many of them sold in this

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           125

          1    country, to a Volkswagen Jetta around 25.1

          2    miles per gallon.

          3              So it's linking one to many and our

          4    job in that relationship is to select one.

          5    This one to many relationship, I want to

          6    quantify it so that you could come up with a

          7    reasonable approximation of how good this

          8    linking process was for EIA.  This is a chart

          9    of the number of eligible donor vehicles and

         10    a number of sample vehicles that were

         11    matched.

         12              Unfortunately you all don't have

         13    the slides but the table provides the number

         14    of eligible donor vehicles.  In other words

         15    that Volkswagen Jetta would be a vehicle on a

         16    one to one match.  We had around 5200

         17    vehicles where there was a one to one match.

         18              Where there were two to five

         19    vehicles we had almost 20,000 vehicles that

         20    were matched with only donor vehicles of two

         21    to five.  And the quality of it goes from top

         22    to bottom where it was presumed that the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           126

          1    imputation quality is very high at the top

          2    and very low at the bottom of this table, and

          3    we have a little over 47,600 cases of sample

          4    vehicles from the NHTS.

          5              How we adjusted fuel economy, we

          6    start with EPA's unadjusted fuel economy

          7    values, which come directly from the CAFE

          8    program.  We then adjust that for on-road

          9    shortfall.  In other words the vehicle as it

         10    hits the road, what does it get on-road.

         11    Now, that's an estimate or adjustment that

         12    has been in agreement since 1985.  In fact

         13    EPA uses those adjustments when you view the

         14    fuel economic sticker on a new vehicle.  By

         15    law there is a sticker placed on every new

         16    vehicle that identifies the highway and city

         17    and combined fuel economy estimate so that

         18    users or purchasers have clear information

         19    when they make their purchase choice and fuel

         20    economy is one of their justifications for

         21    the purchase.

         22              The second adjustment is something

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           127

          1    unique to the transportation surveys that EIA

          2    used to do where we adjust for in-use.  In

          3    other words adjust for the location of the

          4    vehicle, the driving patterns, the usage of

          5    that vehicle, which is something that the EIA

          6    has applied from a paper produced by Crawford

          7    in 1983.

          8              The reason I put the site down here

          9    is that that model hasn't been adjusted or

         10    recalibrated since 1983, I think, using fuel

         11    economy information obtained through travel

         12    and fuel purchase diaries so they actually

         13    went out and talked to the users of these

         14    vehicles and tracked their travel as well as

         15    their consumption of energy.

         16              The details are very rich because

         17    this is a new product that EIA has produced

         18    and to maintain a high level of transparency

         19    of what we did and all the downfalls, all the

         20    pitfalls, and all the imputations that we did

         21    because we had essentially added additional

         22    information to a survey based on reported

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           128

          1    information.  So it's 100 percent imputed

          2    information.

          3              So we made all this information

          4    available on this website and we verified the

          5    procedure for adjustment in 1991 and 1994

          6    RTECS; however, because we don't have fuel

          7    purchase diaries or travel diaries linked

          8    there is no way that we could verify the

          9    procedures that we used for 2001 NHTS study.

         10              The adjustments, I basically just

         11    threw the equations out here.  The first two

         12    equations deal with the shortfall adjustment

         13    produced by Hellman that people are in

         14    agreement since '85 on.  The last two

         15    equations deal with an additive adjustment

         16    where we have the gallons per mile ratio.

         17    That's a GPMR equation which we add delta to

         18    which is Crawford in his '83 publication

         19    developed to give a total fuel economy

         20    in-use, on-road estimate of the GPMR

         21    statistic and essentially here it is in terms

         22    of the clarity of it all.  It's well

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           129

          1    documented in the appendices.

          2              So I wanted to give an example

          3    going back to that Volkswagen Jetta where the

          4    composite MPG is 25.1 miles per gallon.

          5    Let's just presume that we selected that

          6    vehicle.  By month I circled the annual to

          7    remind me that I should tell you that we make

          8    our adjustment by month; however, we publish

          9    annual estimates, not monthly estimates;

         10    however, we do build out from the months.

         11              Using a straight fuel economy

         12    estimate directly from the CAFE program, we

         13    would have given the vehicle a 25.1 mile per

         14    gallon estimate.  Using the shortfall, which

         15    would drop it by the Hellman-designated

         16    shortfall adjustment, we would have dropped

         17    that estimate to 21.3 miles per gallon;

         18    however, based off of that delta regression

         19    estimate however it was approximated, I don't

         20    know if it was regression, we can track what

         21    the fuel economy annualized estimate would

         22    have been had the vehicle traveled the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           130

          1    typical amount of 15,000 miles.  You can see

          2    the fuel economy varies by month and that in

          3    my opinion is dealing more with the use of

          4    the vehicle whether it was used for highway

          5    or city driving where as you drive more or

          6    VMT increases you're driving more on a

          7    highway level than a city.

          8              So you see that the summer months

          9    you peak and you go more towards the estimate

         10    provided directly as a composite estimate.

         11    Now, if we went to a household with only

         12    4,000 miles of travel that is the

         13    distribution that we would see by month and

         14    the annual estimate would be around 15 miles

         15    per gallon so you can see there is a wide

         16    range.  We really adjust the fuel economy

         17    estimate based on the annualized VMT estimate

         18    and if we go between 4,000 miles all the way

         19    up to 15,000 miles you can see the trend

         20    basically converges to that 21.6 miles per

         21    gallon estimate, which would be a 15,000

         22    miles annual VMT.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           131

          1              Knowing that we're using 100

          2    percent imputed estimates, EIA conducted some

          3    sensitivity analysis of our national energy

          4    consumption estimates from this product and

          5    also provides a lot of guidance to users of

          6    the day.  There's a very large report making

          7    its way through a Category 1 review.  I

          8    believe the committee obtained a copy of that

          9    but not for citation or distribution.  Let me

         10    know.  Did you all receive a report?  Okay.

         11              In that sensitivity analysis

         12    instead of selecting one vehicle we made room

         13    for three selections of fuel economy estimate

         14    for a given vehicle.  One would be the sales-

         15    weighted value of fuel economy, in other

         16    words of those eight Volkswagen Jettas we

         17    would pick the sales-weighted fuel economy

         18    estimate for that Volkswagen Jetta in the

         19    sample database.  At the national level that

         20    type of a procedure or protocol lowered the

         21    consumption estimate by 2 percent.

         22              When we selected the maximum fuel

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           132

          1    efficiency designated by the 95th percentile

          2    of those sales we lowered the consumption by

          3    9 percent, which you would expect.  We're

          4    picking the most fuel-efficient vehicles

          5    automatically for each of the sampled

          6    vehicles and to cover our lower balance we

          7    picked the minimum fuel economy estimate,

          8    which was designated the fifth percentile by

          9    sales, which raised our consumption by 7

         10    percent.

         11              So with those numbers along with

         12    our sampling or nonsampling errors we in our

         13    report and in our documentation provide notes

         14    to the readers through an author's note that

         15    the use and the usefulness of these data

         16    should be evaluated against the researcher's

         17    project requirements.  If you are requiring a

         18    very exact estimate for your research

         19    requirements there is no need to use these

         20    numbers; however, if this type of a range of

         21    uncertainty is okay for your research then

         22    please feel free to use it or misuse it

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           133

          1    because the tendency is to misuse the numbers

          2    rather than to use them correctly.

          3              In addition to that, and I believe

          4    this is the first time that EIA put out

          5    tables in this way, we added some

          6    interactivity within the tables in an Excel

          7    spreadsheet where the user can specify based

          8    on the sample count cut-offs and the measures

          9    of precision.  In other words each estimate

         10    is supported by an underlying sample.  If a

         11    researcher requires 20 vehicles or 20

         12    households to support their estimate or their

         13    quality of the estimate that they're looking

         14    for we allow them to change those parameters

         15    or the measure of the precision in terms of

         16    the CV or relative standard error, allow them

         17    to change the viewing of the tables.  In

         18    other words they show up as red italicized

         19    numbers rather than the typical black bolded

         20    numbers.

         21              We haven't had any feedback as to

         22    whether this is helpful but we put it out

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           134

          1    there as one more level of guidance for our

          2    users.  And we are working with NPTS for

          3    their 2008 and this is probably going to take

          4    place in four areas of interest.  One is the

          5    verification of our EIA adjustments, in other

          6    words verification of that Crawford 1983

          7    model.

          8              What we would need to verify that

          9    is fuel purchase and travel diaries and I put

         10    a little qualifier on here that this is very

         11    unlikely to happen.  It's very expensive to

         12    do and from current discussions unless there

         13    is a lot of funding for it it is very

         14    unlikely to take place.  Now, to minimizing

         15    the imputation errors there are some aspects

         16    that are likely to occur and those would be

         17    collecting additional vehicle

         18    characteristics.

         19              As I mentioned before, we only have

         20    the make model, model year, and the type of

         21    vehicle.  If we had additional information

         22    related to, for example, the type of fuel or

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           135

          1    the price paid for that fuel or the number of

          2    the cylinders of the fuel or the gold

          3    standard would be the vehicle identification

          4    number, which has embedded in I think a 10 to

          5    16-digit code on each vehicle, a way to

          6    decode all the engine characteristics of that

          7    vehicle that again would be the gold standard

          8    but again putting a probability on collecting

          9    vehicle identification numbers I put fairly

         10    unlikely.

         11              Capturing the vehicle disposition

         12    data, as I mentioned before, EIA was

         13    interested in the actual time of possession,

         14    not an annualized VMT estimate, and it's very

         15    likely that DOT will go ahead and capture

         16    that information in its survey instrument

         17    which would be very helpful.  Rather than EIA

         18    discerning that information from odometer

         19    readings and auxiliary information provided

         20    by the respondent we would have that as a

         21    reported data item.

         22              Also the last item is the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           136

          1    Statistical Efficiency Act.  I don't know if

          2    DOT is going to use that for the NPTS but

          3    there was a signed into law by Bush II about

          4    a year ago?

          5              MR. FRENCH:  December '02, yes.

          6              MR. SCHIPPER:  December '02 which

          7    allows statistical agencies to share data for

          8    statistical purposes.  We went through a

          9    legal process of sharing data last time.  The

         10    ability to easily share information makes

         11    things a lot easier especially when we're

         12    working alongside of DOT but trying not to

         13    hamper their survey processing because they

         14    have their own daily users to satisfy and

         15    their own scheduling to work with.

         16              The question that we had for the

         17    committee is this is a new product by EIA.

         18    It's 100 percent imputed product.  It comes

         19    from a solid source, the National Household

         20    Travel Survey, but the question that we had

         21    is do the extensive assumptions that we make

         22    to create this data cause any concerns when

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           137

          1    we label this as an official EIA data

          2    release.  Even though we put all caveats in

          3    the documentation, in the report, on the

          4    spreadsheets that are sent out as a data

          5    file, is that enough or should we have

          6    additional concerns or should we continue to

          7    expend the resources to do this again in

          8    2008?

          9              And if we do move ahead are there

         10    any ground rules that we should establish in

         11    methods, data requirements, quality of

         12    information with DoT before we undertake such

         13    work because the more information that we can

         14    provide DoT at the front end of a project and

         15    and how clear we are the better the product

         16    will be at the end in where to focus their

         17    time with their contractor vetting the

         18    information because, again, EIA is working

         19    alongside DoT.  We're not the primary focus

         20    of their study.

         21              Also just general improvements,

         22    data, methods, and so on, and thank you for

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           138

          1    your time and attention and you don't have

          2    the slides but there was some mention how EPA

          3    calculates their fuel economy which is then

          4    handed off to the safety administration,

          5    NHTSA.

          6              I threw in the testing protocols

          7    that EPA goes through for the city and the

          8    highway.  You can tell which is city and

          9    which highway by the miles per hour.  And

         10    that the testing protocols are done in some

         11    general summaries and that they are done with

         12    the vehicles air-conditioning or other

         13    accessories turned off.  So I just threw that

         14    in to give you an idea of how EPA comes up

         15    with their numbers to begin with and I'll

         16    look forward to answering your questions and

         17    getting your input.

         18              DR. FEDER:  What if I delegate you

         19    to lead the discussion?

         20              MR. BERNSTEIN:  Well, but why don't

         21    you start off if you have some comments?

         22              DR. FEDER:  No, actually before you

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           139

          1    raised some questions that we would like

          2    probably to answer after people have made

          3    their comments.  So why don't we start?

          4    Mark, go ahead.

          5              DR. BURTON:  Well, since this is a

          6    relatively new data product who's using it

          7    and how and what motivated the development of

          8    this initially?

          9              MR. SCHIPPER:  I'd say the first

         10    motivation was that EIA had a data gap in its

         11    transportation statistics.  Since 1994 we did

         12    not have any information related to light

         13    duty vehicles and household use of energy.

         14    EIA through its national energy modeling

         15    system at one time had demographic

         16    information embedded in it in terms of light

         17    duty vehicle use.  I believe in the last go

         18    around that information or that module of the

         19    NEMS model was ripped out because it didn't

         20    have that information.  Now the question is

         21    is this information good enough to go back

         22    and look at re-evaluating whether that type

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           140

          1    of a module could be replaced or improved in

          2    the modeling system.  Those would be the two

          3    items.  Dwight, do you have any additional

          4    items as to who was looking for this

          5    information?

          6              MR. FRENCH:  Well, everybody looks

          7    for it.  We have the same set of customers as

          8    we would have for any of our consumption

          9    information from Congress and congressional

         10    staff, the education, the energy research

         11    community, and so forth.

         12              DR. BURTON:  I just wondered if

         13    there were specific research questions that

         14    had motivated the development of the data.

         15    So you're saying that from an internal

         16    standpoint potentially you send the revised

         17    NEMS and then anybody else that wants it --

         18              MS. KIRKENDALL:  I think one reason

         19    they did the analysis was that they had lost

         20    the residential transportation energy

         21    consumption survey in '94 so there were data

         22    available that had been used within EIA.  It

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           141

          1    was lost because of the resources.

          2              MR. RUTHERFORD:  I will tell you

          3    I've gotten comments from the academic and

          4    energy research community between 1994 and

          5    now that say how can you call yourself an

          6    energy statistics community when you don't

          7    have information about energy use for two-

          8    thirds of the oil consumption.  I mean, in

          9    essence we don't know what the fuel economy

         10    of the US household vehicle fleet is.  This

         11    is our best approach to try to get at it but

         12    to ask somebody what the fuel economy of the

         13    vehicle fleet is nobody can give you an

         14    absolute certain answer to that.

         15              MS. LISS:  If I may add, the

         16    household vehicle fleet has expanded

         17    considerably since 1994.  I mean, what we

         18    looked at in 2001-2002, it's a huge growth in

         19    the number of vehicles.

         20              DR. BURTON:  And, I would assume,

         21    significant change in the composition in

         22    terms of fuel economy.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           142

          1              MS. McGUCKIN:  SUVs were introduced

          2    after 1994 as a major market segment.

          3              MR. BERNSTEIN:  I agree that

          4    absolutely this needed to be there.  You

          5    asked a question if she could do it in 2008.

          6    I mean, for a variety of reasons yes, but the

          7    most critical reason internally for EIA which

          8    gets back to Mark's comment is can this be

          9    used by NEMS and STEO to do a better

         10    forecast.  If it can that should guide what

         11    you choose to do because that ought to be the

         12    primary use.  Internally you try to get a

         13    hand on vehicles and vehicle use.

         14              The other question is so that's one

         15    in terms of whether you should do it in 2008.

         16    You really have to answer that question

         17    because if they can't use it you might think

         18    about whether this is the best approach to

         19    take.

         20              DR. FEDER:  Since there were three

         21    questions raised I would like to maybe go one

         22    question at a time and actually that's just

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           143

          1    as good question as any so any more comments

          2    on should this be redone in '08 and what

          3    should the emphasis be?

          4              MR. SCHIPPER:  Within the EIA you

          5    have the informal relationships with the NEMS

          6    modelers and I'm aware that the NEMS in terms

          7    of the transportation side will look at this

          8    information of VMT estimates of energy use as

          9    a benchmark in their own mind in looking at

         10    the information whether their modeling makes

         11    sense or agrees or disagrees and how it

         12    disagrees.  The use of this information in

         13    NEMS on a formal or an informal basis because

         14    you are going to have modelers interested in

         15    going hey, I'll take any information out

         16    there.  Knowing the caveats, knowing how it

         17    was put together, this is beneficial to me as

         18    a modeler to provide additional information

         19    related to my model; however, formally am I

         20    taking this number directly and embedding it

         21    in a process that is codified in the NEMS

         22    systems.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           144

          1              Which level are we talking about,

          2    the formal or the informal, or is the

          3    informal less of a justification for doing

          4    this work than the formal process?  I would

          5    assume so but in terms of resources this was

          6    a lot of time by energy analysts, not survey

          7    logistics, so it was a whole different skill

          8    set that went into doing this work in knowing

          9    the transportation sector and household and

         10    vehicle use rather than knowing survey

         11    logistics of how to get information from a

         12    household so there are differences.

         13              DR. FEDER:  Can we go back?  I'm

         14    sorry.

         15              MR. RUTHERFORD:  I just had a

         16    question of information.  I think about the

         17    use of this type of information or this type

         18    of survey.  A policy question is if we're

         19    faced with the need to try to abate energy

         20    use at the household level how effective is

         21    the CAFE program versus other types of

         22    instruments for achieving the same objective.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           145

          1              I was just wondering in the survey

          2    does it give you very much insight into this

          3    question about rebound effect, because, I

          4    mean, CAFE is a very blunt instrument because

          5    there's this relationship between miles per

          6    gallon and number of miles driven that's

          7    quite central to the analysis and I just

          8    wonder.  It's that elasticity I'm wondering

          9    about rather than actually what the shares

         10    are in the current fleet.  I just wonder if

         11    that's something that --

         12              MR. SCHIPPER:  Probably the fuel

         13    purchase diaries, that type of information.

         14    EIA conducts weekly gasoline surveys of how

         15    the price changes with how much energy has

         16    been supplied so you get that elasticity

         17    tracked over time to get that effect of the

         18    price.  But are you talking about changing

         19    fuel economy in terms of I'll get a vehicle

         20    that is 10 percent more fuel efficient, I'm

         21    going to then drive 20 percent more, that

         22    type of elasticity?

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           146

          1              MR. RUTHERFORD:  Yes, I mean,

          2    basically there is this effect that if you

          3    try to target fuel economy then that doesn't

          4    necessarily control how much fuel use you

          5    manage.

          6              MR. SCHIPPER:  Right, and I think

          7    CAFE does use the rebound effect in their

          8    calculation or setting of the CAFE targets

          9    and I just don't have enough information as

         10    to whether this study could provide

         11    elasticities in the way that you are looking

         12    for.  By the time you get the elasticities

         13    it's too far back in the past and things have

         14    changed.  You want to have the elasticities

         15    at least on a quarterly basis rather than

         16    three years, four years, back.

         17              MR. FRENCH:  I also wonder if at

         18    this point if cost elasticities don't swamp

         19    any MPG elasticities because the price is

         20    changing so rapidly.  Certainly that's true

         21    on a short term.  I think we are on the cusp

         22    of being on some elasticity changes in the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           147

          1    short to mid-term and probably more likely in

          2    the longer term.  Before we didn't do

          3    anything on CAFE over 20 years and VMT just

          4    keeps going up inexorably so it isn't as if

          5    by not doing anything we kept VMT under

          6    control.  That's not true.  Maybe VMT would

          7    have gone up faster or something if we had

          8    instituted it but those are hypothetical-s.

          9              MS. LISS:  One thing we also look

         10    at is time elasticities, travel time, travel

         11    time budgets, because from our viewpoint

         12    we're seeing some saturation in travel.

         13    Because of the congestion and the time it

         14    takes people are only willing to spend X

         15    amount of time behind the wheel each day.

         16              MS. McGUCKIN:  I was just going to

         17    say that you should possibly look up the

         18    Victoria Transport Policy Institute.  They

         19    have the travel elasticities report on their

         20    website right now.  It's very interesting how

         21    inelastic miles is.  People just travel.

         22    They don't care how much it costs, they don't

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           148

          1    care what their MPG is, they just do what

          2    they have to do every day.

          3              MR. BERNSTEIN:  I must say that

          4    over the time period that those estimates

          5    were made you basically had declining real

          6    prices of gasoline with increasing miles per

          7    gallon for the most part.  Anecdotal evidence

          8    in the last six months might suggest

          9    otherwise as public transit systems have seen

         10    increasing ridership and we have had when we

         11    have been trying to do some of these things

         12    with other bits of data and the problem is

         13    the time series of data sees very little

         14    variation in price over a long enough period

         15    of time to get anything useful out of it and

         16    it's mostly declining real prices.  And so

         17    it's hard to say what that really tells you

         18    about consumer behavior over the long run if

         19    you actually begin to see.  Now, these last

         20    couple of years of volatile prices, hopefully

         21    there'll be enough information and data

         22    coming out for us to be able to get some

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           149

          1    sense as to how consumers actually do react.

          2              DR. BURTON:  Is that the silver

          3    lining that we'll have enough variation in

          4    the data to make some good estimates?

          5              MR. BERNSTEIN:  I hope so.

          6              DR. BURTON:  I was going to add to

          7    what Mark said in that you're right.  Miles

          8    tend to be very insensitive; however, the

          9    fuel prices will affect modal choice and all

         10    you've to do is look at transit numbers of

         11    the last six months to see that that's very,

         12    very true.

         13              DR. FEDER:  Can we just answer the

         14    first question that Mark had concerning the

         15    assumptions that went into the study that are

         16    extensive, as you pointed out, and I was

         17    wondering.  I don't know most of the terms

         18    VMT and what NEMS exactly is and so on.  My

         19    concern was actually about that.  Can you

         20    just explain first why you asked this

         21    question?  Maybe then they can answer it.

         22              MR. SCHIPPER:  Since it's a new

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           150

          1    product that EIA has produced, in other words

          2    we did not go out and draw a sample and

          3    collect this information directly from the

          4    respondents.  We imputed it based on survey

          5    responses.

          6              And in terms of should we be doing

          7    this because EIA when it puts out information

          8    it's taken literally.  In other words it's

          9    used and misused because it's the information

         10    that EIA has produced.  Even though we have

         11    placed caveats on the quality of the

         12    information we are still held to a different

         13    standard of government statistics because it

         14    was obtained from EIA's website or the report

         15    was obtained from our National Energy

         16    Information Center and has the EIA label on

         17    it.  So there was concern to say whether this

         18    type of a product is something that EIA

         19    should release.

         20              DR. BURTON:  Can I address that?  I

         21    do more transportation work than I do energy

         22    work and I probably use BTS products more

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           151

          1    than I use EIA products so I'm familiar with

          2    data products in both worlds and the

          3    limitations on the reliability of the BTS

          4    products are just as real and they're just as

          5    good about warning users about those

          6    limitations and people still misuse them.

          7    The trade analysis framework is a wonderful

          8    example of how people have taken something

          9    and used it in ways it was never supposed to

         10    be used.

         11              But at the end of the day what you

         12    have done, I think, is at least as good as

         13    any of the other products that I see on the

         14    transportation side.  I have no problem with

         15    it at all nor would I have any problem using

         16    it and defending it and I'd probably misuse

         17    it too.  When you need an answer you do just

         18    about anything.

         19              DR. SITTER:  I disagree with your

         20    characterization of what you did.  I mean, to

         21    me you have data and you basically modeled

         22    with it.  You can view it that way.  Your

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           152

          1    data was categorical, maybe, and you used

          2    auxiliary information and so forth but

          3    ultimately you had data and you built models

          4    around that data.  It wasn't data you

          5    personally collected, other agencies

          6    collected it, but I wouldn't really view it

          7    as imputation.  I don't think that's exactly

          8    what you did.

          9              Imputation suggests that you

         10    release data with artificial data points and

         11    that's not what really you did.  I mean, I

         12    really wouldn't characterize it that way at

         13    all.

         14              MR. BERNSTEIN:  I'm not sure what

         15    you called it but if you call it an estimate

         16    of vehicle energy use like you call forecast

         17    estimates then you get away from this issue

         18    of is it actually data because I think

         19    Randy's right.  It's not primary.  You're

         20    using primary data but --

         21              DR. FEDER:  Do the interactive

         22    tables you mentioned offer any way of getting

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           153

          1    some handle on the uncertainties?

          2              MR. SCHIPPER:  The only

          3    uncertainties would be related to the measure

          4    of precision associated with the estimate and

          5    that the sample supporting that estimate in

          6    terms of the actual responding household or

          7    the number of vehicles.  We didn't carry

          8    along two or three different measures for

          9    each statistic.  We calculate it using the 95

         10    percent sales weighted value.  If we carried

         11    all those along, basically did the study

         12    three different ways, that would be a measure

         13    of uncertainty associated with each estimate.

         14    It's a possibility to do something like that;

         15    however, I don't know if we would confuse the

         16    readers to say well, which one should I use.

         17              Why not the 60th percentile?  Or

         18    once we start giving flavors of the data

         19    you're left with different paths to take.  If

         20    there is an agreement out there in

         21    statistical community, say, you should be

         22    looking at these types of values, 75

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           154

          1    percentile and the 50 percentile or somewhere

          2    along those lines, and that's something we

          3    could evaluate.  Otherwise the user who'd say

          4    well, which one should I use.  You gave me

          5    three estimates.  I like this one.  I'm going

          6    to use this one because it fits my research.

          7              MR. BERNSTEIN:  Well, but that's

          8    okay too.

          9              MR. SCHIPPER:  And I really like

         10    the idea that it's a model-based estimate

         11    instead of an imputed estimate.

         12              DR. BURTON:  I think if we're going

         13    to look for a way to describe that I would

         14    say that it's a bit more deterministic, it's

         15    not a purely stochastic model, and I think

         16    there are probably some engineering aspects

         17    that go into determining some of the

         18    parameters so it may be a bit more

         19    deterministic but it's still a model.  I

         20    agree with Randy.

         21              MR. BERNSTEIN:  In terms of the

         22    assumptions I would say a couple of things.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           155

          1    One is I believe EPA is changing how they are

          2    doing 

          3              MR. SCHIPPER:  They are looking

          4    into by the end of the year trying to better

          5    approximate the fuel economy that a person

          6    would actually obtain driving a vehicle on

          7    American roadways.

          8              MR. BERNSTEIN:  So what would be

          9    interesting, of course, is well, in 2008 you

         10    should be expecting to do something else but

         11    also have a close idea as to your previous

         12    ones and also be thinking about how to update

         13    that 1983 estimate somehow, I mean, without

         14    doing the detailed stuff that they did and

         15    see what can be done to generate some of

         16    those.

         17              MR. FRENCH:  Just a short comment

         18    about that, when we first used the RTECS we

         19    did a diary data collection and we did it

         20    through the 1985 cycle of RTECS and so when

         21    the '83 study came out and we decided after

         22    '85 for budgetary reasons to go from a diary

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           156

          1    collection to one where we were just

          2    collecting VMT and modeling the miles per

          3    gallon we looked at the '85 data versus the

          4    '83 modeling procedure and, lo and behold,

          5    for major population subgroups it was within

          6    a couple of percent uniformly and so we went

          7    ahead and we used it '88, '91, and '94.

          8              At the time in '94 I said this

          9    thing has been around for a decade now.

         10    Should we test it?  So in '95 we did a

         11    research study.  We did a small research

         12    study, it cost us a couple of $100,000, but

         13    we went out and did diaries on 500 vehicles

         14    nationwide, subsample of the '91 RTECS, I

         15    guess it was, and once again we found the

         16    model for large population subgroups to be

         17    within a couple of percent.  I was expecting

         18    that model approach to have deteriorated for

         19    determining on-road but it hadn't, amazing to

         20    me.

         21              And so I was planning on continuing

         22    to use that; however, the budget realities of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           157

          1    the mid-'90s intervened and by '96-'97 time

          2    frame we had decided to change the

          3    consumption surveys and one of the things we

          4    did was drop RTECS.

          5              So we haven't really looked at that

          6    since then but you're right.  To the extent

          7    that EPA changes their approach toward

          8    determining what they're actually going to be

          9    putting on the stickers and so forth that

         10    whether in itself is good or bad as an

         11    adjustment may not relate in the same way to

         12    what EPA is going to produce in its new

         13    adjusted miles per gallon estimates and so we

         14    might have to think of changing that

         15    supplemental adjustment or doing something

         16    different for a couple of different reasons.

         17              MR. SCHIPPER:  We may have to have

         18    two paths because they're not going to go

         19    back and retest.  They're only going to test

         20    forward.  So you're still going to have the

         21    vehicle stock out there that's pre-2005,

         22    2006, whenever EPA would institute a new

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           158

          1    dynamometer test.

          2              MS. LISS:  And the average vehicle

          3    age of the household vehicle fleet is 8.9

          4    years so you're talking about a large number

          5    of vehicles.

          6              MR. SCHIPPER:  Especially light

          7    trucks, which stay around forever.

          8              DR. SITTER:  I was interested in

          9    your answer to the question concerning

         10    components of variance.  From what you're

         11    saying you do have estimates of error for

         12    things like weighted sales estimate so if you

         13    start just stepping back one thing your final

         14    formula is a ratio-type estimate based on

         15    three estimates, VMT, miles per gallon, and

         16    cost per gallon.

         17              MR. SCHIPPER:  Right.

         18              DR. SITTER:  Do you have standard

         19    errors on those three estimates?

         20              MR. SCHIPPER:  Standard errors in

         21    the possible selections for each vehicle or

         22    just the tallied estimate for each?

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           159

          1              DR. SITTER:  Well, I mean, a

          2    standard error based on the data that you

          3    originally built these on.  I mean, some of

          4    them you've got to have a standard error in

          5    the weighted estimated sales and then you use

          6    that to weight the average miles per gallon.

          7    You can treat those as fixed.  I mean, these

          8    are all sampling errors.

          9              So ultimately you must be able to

         10    percolate your sampling errors up in some

         11    way.  I think that was more of the question.

         12    Your answer of giving three estimates, I

         13    don't think that's really what was being

         14    asked.

         15              MR. SCHIPPER:  So we do calculate

         16    the precision for the fuel economy estimate

         17    and the VMT estimate in a population sense.

         18    Would that be basically the variance of the

         19    three components, VMT multiplied by the

         20    inverse of fuel economy times the price to

         21    get the expenditure?  Each one of those has

         22    its own variability.  I think in the sampling

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           160

          1    sense we treat everything as a fixed value

          2    and it's a composite of those three

          3    components put together.  Now, if you don't

          4    add up the variability of each one of them

          5    and the correlations among them is that where

          6    you're trying to get?  In other words it's

          7    not a linear because you get a ratio.  You're

          8    right.

          9              DR. SITTER:  You could linearize it

         10    so I think that was the question.  Was it you

         11    that asked, Moshe, about that?

         12              MR. SCHIPPER:  Well, it also could

         13    be more interesting that the fuel economy

         14    itself would be a less variable item because

         15    in terms of what's out there, in terms of

         16    population, the variability of fuel economy

         17    is going to be much less than VMT.  VMT is

         18    going to drive the variability of the

         19    estimate.  Price, fixed value, we have a

         20    month and a state item.  I'd have to dig into

         21    the results.  I think that's a census of

         22    the --

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           161

          1              MS. KIRKENDALL:  No, but these are

          2    samples and they do provide sampling errors

          3    and that's probably going to be small,

          4    though.

          5              DR. SITTER:  So I think, that was

          6    the confidence band issue about your

          7    interact.

          8              MR. SCHIPPER:  Another way to look

          9    at it, instead of just one, take all three.

         10              DR. SITTER:  And then you linearize

         11    the ratio and you can at least have some

         12    confidence interval-type information when you

         13    estimate.

         14              DR. FEDER:  I have a question that

         15    maybe shows lack of understanding about what

         16    this is about but the name of the study is

         17    about energy use and there was some issue

         18    about the cost.  What are people mostly

         19    interested in?  Is it the expenditure or the

         20    energy consumption or total volume?

         21              MR. SCHIPPER:  Again it depends on

         22    the researcher.  We would be interested in

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           162

          1    the energy use, the amount of energy consumed

          2    in terms of what's the impact or the profile

          3    of the light duty vehicle fleet in energy

          4    required by the country; however, there will

          5    be interest in knowing the expenditures for

          6    certain types of households, people on the

          7    lower income levels, elderly householders,

          8    the amount of money that's spent for

          9    transportation use or their personal

         10    transportation use.  That will be of interest

         11    to other researchers but I think the main

         12    statistic that EIA is looking for and,

         13    Dwight, you correct me if I'm wrong, is the

         14    consumption, the amount of energy consumed.

         15              MR. BERNSTEIN:  Let me come back to

         16    Thomas' point related to this which is if I

         17    were Guy Caruso sitting here today saying

         18    what do I need to know to answer

         19    congressional questions it is can we do

         20    anything to reduce demand for gasoline to

         21    reduce pressure on prices in the short term

         22    and perhaps in the long term.  And one of the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           163

          1    questions you guys need to ask yourselves is

          2    is there anything in here that you can give

          3    to him or to researchers who are studying

          4    that question.  Is there anything in here

          5    that can get at that question?  I mean,

          6    that's the number one issue on the minds of

          7    most politicians these days when it comes to

          8    energy is gasoline prices.

          9              What can this tell us about what we

         10    can do in the near term or the long term, if

         11    anything, to reduce pressure on prices which

         12    comes about as demand issues as well as then

         13    linking it up to the production and supply

         14    stuff that you get from other parts of the

         15    EIA?  That's one of the real critical

         16    elements you got to be thinking about.

         17              MR. SCHIPPER:  This has been a

         18    lengthy process.  The report that you all

         19    were forwarded is up to over 200 pages long.

         20    The initial report that came from the study

         21    was appendices and methodology tables and a

         22    very short write-up in the front, just a

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           164

          1    basic summary of here are some numbers, they

          2    went up, they went down.

          3              The current report embodies more

          4    analysis, more how you can use this

          5    information.  One of the key components is a

          6    composition of energy use based on household

          7    types, fuel economy, vehicle types, vehicle

          8    vintages, which really didn't pan out to be

          9    anything, but there are items that we provide

         10    analysis for over an above what we had

         11    intended to do because we intended to come

         12    out with a data product that here it is, here

         13    are the caveats, here's something useful, but

         14    in addition to that we put in a lot of

         15    analysis work into trying to get at those

         16    questions.

         17              Now, we get at those questions

         18    based on the year of the information that we

         19    have so we're always based on the

         20    characteristics and the advance that occurred

         21    in the sample years.  In 2001 we had 9/11,

         22    which has a potential impact on those

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           165

          1    numbers.  So those things that we can't

          2    control for may have been of interest to

          3    other researchers to compare but that's one

          4    of the caveats.  We answer those questions

          5    but it's only within the realm of the survey

          6     collected so you go from

          7    '94 to 2001.  A lot of stuff has happened

          8    between those two time frames and we lump it

          9    all into one year or the other.

         10              MS. McGUCKIN:  Just to have fun, I

         11    was going to say we look at those questions

         12    at DoT.  For instance, one of the things

         13    people ask us is what kinds of trips would

         14    people stop doing because of the rise of the

         15    gas prices.  Obviously it's very easy to do

         16    for work trips but not so easy for shopping

         17    and they may drop shopping trips or social

         18    rec trips because of that.

         19              So don't take the cost off.  We

         20    really do love this partnership because we

         21    gain as much or maybe more.  I don't mean to

         22    say that while we're negotiating the next

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           166

          1    ground but we get a lot from this

          2    partnership.

          3              MR. BERNSTEIN:  That just

          4    emphasizes the need to work together to

          5    answer some of these questions that are going

          6    to be critical for when your bosses have to

          7    go up to give testimony in Congress on what

          8    could we do in the next four months to reduce

          9    pressure on gasoline prices.  I mean, it

         10    comes back around to that and there's clearly

         11    a lot of information in DoT of things you do

         12    with the data that these guys can actually

         13    make use of and vice versa and I know it's

         14    hard.

         15              The other question I wanted to ask

         16    just on the EPA data, you said they wouldn't

         17    give it to you.  So (a) do you know why,

         18    which you don't have to answer now, but (b),

         19    I mean, if you do a FOIA request or if

         20    somebody outside EIA does a FOIA request is

         21    there a reason why they wouldn't give it?

         22              MR. SCHIPPER:  Well, there was no

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           167

          1    reason provided.  EPA through DoE's fuel

          2    economy guide does put out a rich data source

          3    of fuel economy estimates.  The second item

          4    is the sales figures for each configuration

          5    of the vehicle.  That information EPA wasn't

          6    willing to send to us.  NHTSA was because

          7    they had both the sales information and the

          8    fuel economy estimate just so we had to go

          9    through the second step of removing all the

         10    CAFE program calculations because EPA will

         11    conduct calculations for CAFE program and

         12    then we just had to back everything out.

         13              MR. BERNSTEIN:  Well, I'll be happy

         14    to make a FOIA request if you want 

         15              MR. SCHIPPER:  And one more thing,

         16    we find it beneficial to have all the travel

         17    and trip information from the NHTS study or

         18    now the NPTS.  And there are users, I think,

         19    Professor Golub at USC.  Is it USC or --

         20              MS. McGUCKIN:  I think it's UC

         21    Berkeley, isn't it?

         22              MR. SCHIPPER:  Basically tying

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           168

          1    these two data sets together and we put out a

          2    public use file for that purpose so

          3    researchers can do a lot of unique and

          4    interesting things with energy data as well

          5    as all the travel and trip information.

          6              DR. FEDER:  Just one question on

          7    the NPTS, is there going to be any

          8    demographic component and breakdown of travel

          9    by work, shopping, or recreational travel so

         10    we can answer some of the questions that Mark

         11    was raising about how people respond to

         12    rising costs and things like that?

         13              MS. McGUCKIN:  Yes, we get detailed

         14    demographic and trip information from travel

         15    diaries every household member keeps in a

         16    single day so they report together so we also

         17    get their household interaction and vehicle

         18    use.

         19              DR. BURTON:  How available is that

         20    data to the public?

         21              MS. McGUCKIN:  It's on the website.

         22    You can download it.  We also have a table in

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           169

          1    it to --

          2              DR. BURTON:  You mean in that level

          3    of desegregation?

          4              MS. LISS:  Oh, yes, the absolute

          5    desegregate data set with identifying

          6    information removed is on our website and in

          7    addition on our website is an online analysis

          8    table which allows you to go in and run the

          9    data without having to go through the six

         10    months of learning curve on this data set.

         11              DR. FEDER:  Actually it's a very

         12    tangential thing but I should mention that

         13    Nick and I are both interested in academics

         14    and how they juggle and this data can find

         15    its use for us.  Apparently some people

         16    travel long distance and carry the viruses

         17    and diseases with them and it's good to know.

         18              MS. McGUCKIN:  How about tiger

         19    mosquitoes, tiger mosquitoes got carried on

         20    the freeway system from just the rush of air.

         21              DR. HENGARTNER:  It's actually more

         22    important that similar kind of data is used

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           170

          1    to build transportation models at the urban

          2    level like TRANSIMS and they are used to do

          3    academic modeling but that's really far out.

          4    Actually I would rather wish to stop short

          5    this discussion instead of starting one.  If

          6    we can talk offline I'd like to know more

          7    what happened here.

          8              If I can invite the rest of the

          9    people out there.

         10              Welcome back from the breakout

         11    session.  I would like to have them

         12    summarized, the two breakout sessions, for

         13    the rest of the committee.  Walter Hill will

         14    provide us the first summary about the

         15    session on preserving EIA trustworthy data

         16    sets, model documentation, and contextual

         17    history.

         18              MR. HILL:  And I got much more than

         19    I could easily summarize 

         20    couple of times that we need more money, more

         21    staff.  That was a key point 

         22    there are many documents available 

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           171

          1     three main points was the

          2    mission  one of them might be

          3    what you ideally would have and then what  we

          4    actually have  have open

          5    documentation, have an honest system

          6     very transparent.  Unfortunately

          7    there's no overall inventory.  There's no way

          8    of scheduling.  There is no standard

          9    available which makes it obviously difficult

         10    to track records if you're looking for

         11    something quite far back.

         12              Much material now is in boxes.

         13    They gave a number on the order of 10,000 so

         14    there are tens of thousands of boxes of

         15    records available so obviously searching for

         16    something specific, say, pick a date, 1983,

         17    1984, is a challenge.

         18              Some of that you may find with an

         19    online search but there's no guarantee that

         20    you'll be able to search for records or

         21    retrieve the records or ensure the

         22    authenticity of a record from what's

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           172

          1    available now.

          2              So that  issues dealing

          3    with the mission, problems with the mission.

          4    Ideally the management ought to be in a way

          5    that theres quality control, there's

          6    credibility.  Records are used to forecast so

          7    you want information available for

          8    forecasting.  Accessibility should ensure

          9    common language.  You should be able to find

         10    the responsible person or entity giving

         11    records so if something is inconsistent you

         12    could maybe contract on the source to correct

         13    the inconsistency.  There's actually a little

         14    story given at the end, a problem in the way

         15    records were kept where there was a record

         16    that was lost.  It was hard to track down

         17    what had happened to that record.

         18              From the academic side a number of

         19    these things that are ideal are types of

         20    things we would typically want, reliability,

         21    validity, authenticity.  A government agency,

         22    they mentioned things that wouldn't be seen

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           173

          1     by academics  want the records

          2    secured.  You want records certifiable and

          3    you want the records stored and compliant

          4    with government regulations.  And those are

          5    the types of things that it sounds like most

          6    of those are available in one way or another

          7    although there might be difficulty for

          8    specific cases.  Schedules might vary from

          9    case to case.  Reliability might vary from

         10    case to case, for example.

         11              The depository ought to have

         12    procedural accountability, financial

         13    sustainability.  There should be standard

         14    storage procedures in the ideal system.  And

         15    finally what actually exists and in what

         16    actually exists there are records kept in

         17    various formats, various time periods, from

         18    various different agencies.

         19              Some I noted may be not in the

         20    PowerPoint presentation but notes 

         21     for example, in microfilm.  I

         22    remember that.  Those are  in one

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           174

          1    section and then online materials in another

          2    section and then boxes are in another

          3    section.  So I can easily imagine that if

          4    you're looking for material in 1981 oil

          5    prices it may be difficult to find exactly

          6    what you're looking for.

          7              In those different schedules

          8    available we've had some discussion about

          9    when material should be 

         10    destroyed  deleted.  That's an

         11    issue.  In some sense there's a trade-up

         12    between having everything available with

         13    minute by minute documentation would be too

         14    much but insufficient information is also a

         15    problem if you don't have enough so that a

         16    researcher could track down annual

         17    information then it's not enough so there is

         18    some happy medium between those two extremes.

         19    We were not actually sure where that should

         20    be.  Apparently there's not a standard about

         21    what should be done.

         22              It was recommended that procedures

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           175

          1    ought to be standardized, there ought to be a

          2    way of tracking documents that are stored,

          3    and they're looking for ways to maintain the

          4    information about models.  Some models change

          5    quite regularly.  In my research I haven't

          6    noticed that but looks like that's a severe

          7    problem, something we would be concerned

          8    about, models being changed from year to

          9    year.

         10              A couple of recommendations 

         11     maybe one or two 

         12    it would be good if there was a central

         13    content management place or procedure of some

         14    sort so that if you're looking for record you

         15    can go to one central place and there will be

         16    documentation on where to go from that

         17    central space.

         18              And it would be good to have

         19    information on what is needed over time.  It

         20    was noted that they're coming up on the 30th

         21    year anniversary.  Information going back to

         22    early 1980s is there but how much do you want

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           176

          1    to keep and how much is really needed to be

          2    kept?

          3              Also there's anecdotal talk about

          4    somebody doing a dissertation in the early

          5    1980s.  The material isn't on record but it

          6    is still possible to get information because

          7    there's an oral history available but that

          8    will disappear at some point 

          9     might disappear 

         10              DR. HENGARTNER:  Thank you very

         11    much.  Does anybody else who attended that

         12    breakout session want to add to Walter's

         13    summary.  I just want to say again I think

         14    that archiving is extremely important and

         15    unfortunately under-appreciated by most.  It

         16    always seems like a drag on everybody but

         17    having good records and history so that one

         18    could go back and especially for researchers

         19    it's a good thing.

         20              And I really want to applaud the

         21    EIA actually already thinking of trying to do

         22    that in a systematic way.  I think doing that

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           177

          1    in itself, the whole process, puts the EIA

          2    ahead of the curve.  So I just want to finish

          3    this on a positive note that it's important

          4    and it's a good thing that we are thinking

          5    about this although I don't have a quick

          6    answer to all the questions that were raised

          7    during the breakout session.

          8              The next breakout session was on

          9    vehicle energy use, what we did and what it

         10    can tell us, and I understand that Moshe is

         11    going to summarize this session.

         12              DR. FEDER:  It's very hard to

         13    summarize everything that's been said.  It's

         14    quite an impressive study actually that

         15    combines a lot of data and many assumptions

         16    that are made in its composition but let me

         17    just briefly try to explain what this is all

         18    about, the questions that the committee was

         19    asked, and some of the answers that we gave.

         20              The vehicle energy use study

         21    actually is about a few things such as what

         22    is the energy that's been used, what is the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           178

          1    cost of that energy, and then I thought those

          2    were the two questions but it turns out that

          3    there's a lot of interest in how people would

          4    respond to changes in cost and how policy

          5    makers could reduce the consumption of oil

          6    which is one of the questions that are

          7    repeatedly asked.

          8              So the process of this study was to

          9    combine data from different sources.  Just in

         10    a nutshell how do you measure energy

         11    consumption and its cost?  You take a vehicle

         12    and you know its theoretical energy

         13    consumption in miles per gallon which you

         14    multiply by the cost and there you are but in

         15    fact the data come from different sources.

         16    There's an adjustment.  The EPA has a

         17    theoretical number which is obtained by lab

         18    tests.  It's more efficient to look in the

         19    lab than on the highway so there's an in-use,

         20    on-road adjustment for the fact that highway

         21    and city driving are different.  They consume

         22    more energy than in the lab.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           179

          1              In addition we know that seasonal

          2    variation affects our driving differently.

          3    People who drive more in the summer because

          4    of recreational needs consume less energy

          5    because it's highway travel and so on and all

          6    these adjustments are based on assumptions

          7    that will be made.

          8              In addition it turns out

          9    information from the surveys and we had the

         10    mention of the RTECS, which was discontinued

         11    in 1994.  There's the NHTS and so what I want

         12    to explain is in 2008 there's going to be

         13    NPTS, another survey.  All those provide

         14    typically a car manufacturer, a model name,

         15    and a year.  And it turns out that to almost

         16    every such model there are many actual types

         17    of vehicles that range quite variably in the

         18    MPGs, the consumption.

         19              So one example, a 2001 Jetta that

         20    can range between 25.1 to way over so you

         21    need to make an assumption which you can't

         22    and they used a one to many linkage algorithm

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           180

          1    and, of course, there's a lot of uncertainty

          2    coming from that.  Again there are the

          3    adjustments for the on-road, in-use that

          4    people make and other things as well.

          5              The data, again there was a mention

          6    here in the discussion that there was some

          7    collection of data through diaries and then

          8    there was some modeling done and comparisons

          9    actually show that the modeling performed

         10    quite well but we don't know how future use

         11    is going to be in response to the rise in

         12    energy cost and shortages which are described

         13    as elasticities.  I'm not sure I'm saying

         14    this right but I'm not an econometrician.

         15    And all these things combined lead to

         16    estimates of energy use and energy cost.

         17              The questions to the committee were

         18    concerning the extensive assumptions that

         19    went into the model, are those appropriate,

         20    what can you do about it, any concerns about

         21    it, and so on.  Then we referred to

         22    imputation and the committee said actually we

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           181

          1    should refer to this as modeling and it's

          2    okay to make assumptions in modeling and be

          3    understood about the uncertainties.  Also one

          4    of the products of this study is interactive

          5    tables which allow for accounting of some of

          6    the uncertainties especially about the

          7    assumptions that are being made.

          8              And another comment was made about

          9    other studies that this is at least as good

         10    as any other study that has been seen by the

         11    committee members.  So I think the people

         12    here, and I'm not an expert in this, but the

         13    others thought it was a good product.

         14              The other question was should we do

         15    this again in 2008 after the collection from

         16    the NPTS and again what ground rules should

         17    be set with regard to the DoT collaboration

         18    because a lot of data is coming from DoT and

         19    again definitely it should be redone in 2008.

         20              And then the third question was

         21    very encompassing.  It was about what

         22    improvements can be done to the data and to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           182

          1    the methods and some of you may have heard

          2    the responses because you already came into

          3    the room where we were talking about it.  And

          4    I think one comment was made that the EPA is

          5    changing the way it's doing some of the

          6    statistics that are being used here

          7    especially with regard to the fuel

          8    consumption so definitely use that.

          9              And another comment was made about

         10    the methodology or the methods.  One of the

         11    committee members wondered about the

         12    components of variance and how that affects

         13    the standards, how that affects the output,

         14    and it would be interesting to be able to use

         15    all these sources of uncertainties and the

         16    components of variance to know what range the

         17    predictions from the model can be, not just a

         18    single number.

         19              Mark, maybe you want to add because

         20    you probably understood much more than I?

         21              MR. BERNSTEIN:  No, I think you

         22    covered things mostly.  One of the things

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           183

          1    that we did emphasize thinking about 2008 is

          2    where else in EIA this information is going

          3    to get used, particularly NEMS and STEO, and

          4    while it is important for those of us outside

          5    EIA to have this data because it is a big gap

          6    but it's got to be used and if you're really

          7    going to do it well in 2008 it's got to have

          8    a market internal to EIA as well and so it

          9    should be designed so that NEMS could take

         10    the benefit from the effort.

         11              DR. HENGARTNER:  Anyone else from

         12    that committee who wants to add to this?

         13              MR. RUTHERFORD:  I'll just add an

         14    observation.  I thought it looked like it was

         15    quite a productive interaction with the

         16    Department of Transportation.  The EIA

         17    presenter was apologizing about the fact that

         18    they haven't done their own survey but my

         19    impression was that this looked like a very

         20    good way to work because they can double up

         21    on the survey and share information back and

         22    forth and so I was quite impressed.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           184

          1              MR. BERNSTEIN:  Yes, I agree you

          2    guys ought to publicize the fact that you're

          3    working well with DoT.

          4              DR. HENGARTNER:  I actually had a

          5    question.  Can you backtrack and find out how

          6    consumers are going to react by buying

          7    smaller cars because of energy consumption or

          8    is that completely unrelated because there

          9    seems to be an interaction here, at least

         10    anecdotally?

         11              MR. RUTHERFORD:  We had a

         12    discussion about rebounds and DoT and this

         13    was one of the points we discussed.

         14              DR. HENGARTNER:  Well, thank you

         15    very much.  I don't want to be between the

         16    committee and its lunch.  Before we leave I

         17    want to invite the public for any comments if

         18    they have any.  That's fine.

         19               So the committee will adjourn for

         20    lunch and reconvene at 1:25 p.m. up here to

         21    continue with Learning from the Past, a

         22    presentation by Renee Miller.  Thank you very

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           185

          1    much.

          2                   (Whereupon, at 12:23 p.m., a

          3                   luncheon recess was taken.)

          4

          5

          6

          7

          8

          9

         10

         11

         12

         13

         14

         15

         16

         17

         18

         19

         20

         21

         22

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           186

          1             A F T E R N O O N  S E S S I O N

          2              DR. HENGARTNER:  Good afternoon,

          3    everybody.  Welcome back to the committee

          4    meeting.  Before I start I would like to

          5    remind you about dinner plans if Bill wants

          6    to fill us in?

          7              MR. WEINIG:  Dinner tonight,

          8    reservations are at 6:00 o'clock.  It's at

          9    Gordon Biersch.  It's a beer pub and

         10    restaurant.  Nancy and I were there two weeks

         11    ago.

         12              MS. KIRKENDALL:  For breakfast.

         13              MR. WEINIG:  The iced tea is

         14    terrific.  It's at the edge of Chinatown.  So

         15    it will be 9th and F Street NW.  You can use

         16    the red line, for example, to Chinatown,

         17    Gallery Place and then exit to the south,

         18    towards the river, and go to 9th and F.

         19              DR. SITTER:  If you want to walk a

         20    little bit, I mean, if you're coming from

         21    here you can take the blue or orange and then

         22    get off at Metro Center and it's only three

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           187

          1    blocks.  Get off the 12th and F at Metro

          2    Center and just walk three blocks rather than

          3    changing trains.

          4              DR. HENGARTNER:  Excellent, good.

          5    So let us start with the program hoping to

          6    stay on time as much as possible.  The next

          7    talk will be by Renee Miller about learning

          8    from the past, updating data quality efforts.

          9              MS. MILLER:  Good afternoon.  I'm

         10    pleased to be here to talk about our past

         11    data quality efforts.  These are projects

         12    that I worked on for many years in the '80s

         13    and early '90s and we're now thinking about

         14    bringing back some aspects and we would like

         15    your help to decide what we should do.

         16              We'll start by talking about the

         17    past because what's past is prologue, those

         18    who cannot remember the past are condemned to

         19    repeat it, and my favorite, this is like dj

         20    vu all over again.  The past is prologue

         21    doesn't sound so bad but we don't want to be

         22    condemned and we don't want dj vu all over

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           188

          1    again and that's why we want your help.  I'm

          2    going to tell you about our past efforts,

          3    what worked well and what didn't, and then

          4    Alethea Jennings is going to tell you about

          5    what we're thinking about for the future in

          6    terms of a web product and then she will have

          7    some specific questions for you.

          8              Our past data quality efforts took

          9    several different forms.  Most of these were

         10    done in the predecessor office to the

         11    Statistics and Methods Group, which was

         12    called the Office of Statistical Standards,

         13    although some were done by the program

         14    offices or working very closely with the

         15    program offices.  We had state of the data

         16    reports which were detailed assessment

         17    reports that were publicly available and we

         18    presented information about what we knew

         19    about the strains and limitations of the

         20    data.  We also prepared in-house reports that

         21    had recommendations on how to improve data

         22    quality and how to improve the process and we

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           189

          1    also at one point experimented with a quality

          2    profile.  This was also a publicly available

          3    assessment report.

          4              Some of you may be familiar with

          5    this product.  The Census Bureau or The

          6    National Center for Education Statistics put

          7    these out as well.  During our business

          8    reengineering efforts we also tried to

          9    develop a summary measure of data quality and

         10    we also present feature articles on data

         11    comparisons.  And I'll tell you a little bit

         12    more about each of these efforts.

         13              For the reports that became known

         14    as the state of the data reports we looked at

         15    results of validation studies.  These were

         16    very extensive studies done in the early days

         17    of EIA.  We were a new agency then trying to

         18    establish our credibility and there was

         19    concern that our respondents, particularly

         20    the oil companies, weren't accurately

         21    reporting to us all the time.

         22              And so we conducted audits of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           190

          1    company records and we carefully looked at

          2    our frames and other aspects of the data

          3    collection.  We gained a lot of information

          4    from these validation studies but they

          5    weren't very popular with the respondents, as

          6    you might imagine, or with the survey

          7    managers and they were very expensive and so

          8    when our budget was reduced in the early '80s

          9    they were discontinued.

         10              One of the reasons that they

         11    weren't popular with the survey managers was

         12    that the survey managers felt that the

         13    results were misleading.  When you conduct an

         14    audit you often times find things that don't

         15    quite look right and they thought that the

         16    validation studies were making things look

         17    worse than they look actually were because,

         18    after all, the EIA data tracked fairly well

         19    with data from other sources.  So in order to

         20    have a balanced approach in the state of the

         21    data reports we also included comparisons

         22    with other sources so we conducted the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           191

          1    comparisons at the company level and at the

          2    aggregate level.

          3              Now, comparisons can raise more

          4    issues than they resolve because you don't

          5    know how good the comparative series is, you

          6    don't know whether it's validated, sometimes

          7    you don't have documentation, but never-

          8    theless we found them useful in raising

          9    questions about specific features of each of

         10    the data collections.  And I found them

         11    particularly useful when we could do them at

         12    the company level and we can match company

         13    level data.

         14              And then when we found

         15    discrepancies we could call the respondents

         16    and say how come you reported this on one

         17    form and then on the other and back then, we

         18    had very knowledgeable respondents and they

         19    were able to answer these questions.  I'm

         20    told that this is more of a challenge now,

         21    the respondents aren't as knowledgeable, and

         22    so if we were to conduct an effort such as

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           192

          1    this we might not get to the bottom of the

          2    matter like we did in the earlier days.

          3              Over time there were fewer

          4    comparative series because there was a

          5    consolidation of data collection efforts and

          6    we no longer had the validation studies and

          7    so we started relying on outlier detection

          8    techniques to look for symptoms of problems

          9    in the data.  These were also conducted at

         10    both the company and aggregate level.  We

         11    used graphical techniques.  Sometimes we used

         12    regression analysis to detect any anomalies

         13    and, as with the comparisons, we followed up

         14    because anomalies don't necessarily mean data

         15    errors.  There could have been a change in

         16    the market or in the company structure.

         17              This is an example of a data

         18    comparison we did many years back.  The data

         19    are from 1977 to 1981.  This is a comparison

         20    of EIA motor gasoline supply data and federal

         21    highway data and the dark line is the EIA

         22    data and it was consistently below the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           193

          1    federal highway data.  So we initially

          2    thought well, the federal highway data may

          3    have included some double counting but then

          4    we looked a little more and we realized that

          5    we were missing the blenders.  And so this

          6    was beginning of an effort to get the

          7    blenders into our survey  And you will hear

          8    more about this type of comparison a little

          9    later on.

         10              After we completed a state of the

         11    data report for each of the major energy

         12    areas we consulted with independent expert

         13    reviewers to get ideas for what we should do

         14    next and they recommended that for the next

         15    round we prepare less detailed assessment

         16    reports but that in addition we prepare

         17    in-house reports that contain recommendations

         18    on how we might fix some of the problems.

         19              So we decided to look at the issues

         20    arising from the data evaluations and try to

         21    determine do these stem from the way the form

         22    was designed, does the form need to be

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           194

          1    changed.  We also looked to see whether there

          2    were going to be changes in the industry

          3    structure that should be reflected in the

          4    form design.  These in-house reports evolved

          5    over time, I don't think any two were exactly

          6    the same, but they all contained those

          7    features of looking at the form's design.

          8              We also focused on response,

          9    nonresponse, and processing error and we

         10    looked to see whether the quality control

         11    activities that we had in place were adequate

         12    to control for those types of errors and we

         13    also looked at standards pertaining to those

         14    topics.  And we developed recommendations,

         15    which we then gave to the program office.

         16              Another activity that we performed

         17    a little later on was the development of a

         18    quality profile.  This report also presented

         19    comparisons with other data series but it

         20    wasn't its focus as it was for the state of

         21    the data reports.  The focus here was mainly

         22    on sources of nonsampling error and I listed

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           195

          1    them, coverage, nonresponse, measurement

          2    error, data processing, imputation, and

          3    estimation.

          4              Actually Tom Jobine, who some of

          5    you may know, did the quality profile for us

          6    for the residential energy consumption

          7    survey.  And he looked at all the residential

          8    energy consumption surveys that had been

          9    conducted up until that point to see how they

         10    changed so that if somebody was doing

         11    longitudinal analysis they would be able to

         12    see the changes over time.  This report was

         13    very well received in the statistical

         14    community but unfortunately we didn't really

         15    publicize it and so we didn't really receive

         16    requests for it.

         17              I understand the Census Bureau has

         18    gone forward and is continuing to prepare

         19    quality profiles.  They call them accuracy

         20    reports and they have them for the American

         21    community survey and they cover these topics.

         22    They are not quite as detailed as the one

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           196

          1    that we did on the residential energy

          2    consumption survey though.

          3              Another effort at around the same

          4    time was during our business reengineering

          5    days when we decided to look for a summary

          6    measure of data quality.  And this came up

          7    several times and we decided it couldn't be

          8    done but nevertheless it kept coming again

          9    and so finally we decided to try to do

         10    something but ultimately decided that this

         11    wasn't really possible and I'm bringing it up

         12    not so much because I think we should revisit

         13    trying to come up with the summary measure of

         14    data quality but because there may be

         15    something here that might be useful for a new

         16    product.

         17              What we did was we decided to look

         18    at the dimensions of data quality and we

         19    identified five dimensions, sampling error,

         20    measurement error, coverage, nonresponse, and

         21    what we called methodological consistency,

         22    which was whether a series was consistent

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           197

          1    over time.  Several of these items are

          2    similar to what was in the quality profile.

          3              Ideally if we could quantify each

          4    of these dimensions then the idea would be we

          5    would somehow combine it and we would have

          6    our summary measure but we didn't really have

          7    quantitative information for each of these

          8    measures.  Well, we had sampling error and

          9    you're thinking well, you have response rates

         10    but we didn't have nonresponse bias.  We

         11    didn't have an estimate of measurement error.

         12    That was the type of information that we got

         13    from the validation studies that were

         14    discontinued.

         15              We really didn't have information

         16    on coverage and we didn't know how to

         17    quantify methodological consistency.  So

         18    since we couldn't come up with a quantitative

         19    measure what we decided to do was to try a

         20    qualitative measure so we decided to gather

         21    all the information we had together for each

         22    of the series and we figured well, we would

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           198

          1    rate each series on each of these dimensions

          2    on a 1 to 5 scale.

          3              And we tried to develop guidelines

          4    as to what was a 5 versus a 4 if we were

          5    going to use a 1 to 5 scale like that.  This

          6    got very complicated very quickly.  There

          7    were concerns about the time it would take to

          8    do this, whether we could ensure consistency,

          9    and there was a comment made that well, even

         10    if we could ensure consistency would we

         11    really be providing the user with any more

         12    information than they could get from reading

         13    our explanatory notes.

         14              So then a suggestion was made.

         15    Perhaps we should focus on standardizing the

         16    material in our explanatory notes.  And this

         17    was never really pursued but there may be a

         18    possibility here if we could concisely

         19    summarize the information we have on each of

         20    these error types and present it.  But since

         21    we don't have information necessarily, we

         22    don't have definite information for each of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           199

          1    these error types, it's a real challenge

          2    coming up with a concise statement and once

          3    you start putting in all the caveats you get

          4    back to the level of detail that we have in

          5    our explanatory notes but I think this is

          6    still something to think about.

          7              Moving on to an activity that has

          8    continued into the present, feature articles,

          9    we have feature articles on data comparison

         10    for petroleum supply and petroleum marketing

         11    data that the Petroleum Division performs

         12    annually and they're available on the web.

         13    And we have a one-time product on data

         14    comparisons for alternative transportation

         15    vehicle fuels that was done by the Statistics

         16    and Methods Group that's also a web article.

         17              To summarize our data quality

         18    efforts we could put them into three

         19    categories.  We had efforts to inform the

         20    public, that was the state of the data

         21    reports, the quality profile, the feature

         22    articles.  We had efforts to improve the data

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           200

          1    or the process.  That would be our in-house

          2    reports to improve the data.  And we also had

          3    an attempt to measure data quality.  We were

          4    more successful with the efforts to inform

          5    the public and to improve the data than we

          6    were in coming up with the summary measure.

          7              Overall what worked well was that

          8    we had information on how different data

          9    series tracked over time so that if questions

         10    came up we had some answers.  In working on

         11    this we also gained a lot of knowledge.

         12    There's stuff that I learned from looking at

         13    the data series 20 years back that still

         14    comes in handy.

         15              Now, you may wonder well, do the

         16    survey managers get much out of this.  I'm

         17    not sure we identified problems that they

         18    didn't already know about but we were able to

         19    help bound the problems so that when we had

         20    discussions about them we all had the same

         21    points of references.

         22              In terms of format what worked well

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           201

          1    was the question and answer format that we

          2    tried for one of our reports.  What didn't

          3    work well was that we had a lengthy review

          4    process.  In general the process was lengthy.

          5    It required a lot of resources, both EIA and

          6    contractor support.  We had a lot of input,

          7    we produced a lot of output, but we weren't

          8    necessarily sure of the outcome because at

          9    least for the reports that were publicly

         10    available we didn't have a lot of users.

         11              So we're now looking to the future.

         12    We're looking for something that's, number 1,

         13    affordable and that would be useful and we're

         14    just at the beginning stages of thinking

         15    about this and I'll turn it over to Alethea

         16    now, and she'll show you what we're thinking

         17    about and get your comments.  Thanks.

         18              MS. JENNINGS:  I'm going to take

         19    this down for just a moment so you don't

         20    confused.  She'll be back.  Good afternoon.

         21    Rene has shared with you some of the quality

         22    efforts that EIA has undertaken in the past.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           202

          1    Some things worked well for us and some

          2    things didn't work well for us.  Now we'll

          3    talk about the future.  In order to take

          4    advantage of what we think are the positive

          5    aspects of our previous work we're

          6    considering presenting data comparisons in a

          7    question and answer format on the web.

          8              Now, given our experience in

          9    preparing various reports, feature articles,

         10    and other types of data comparisons we think

         11    a product of this type would actually provide

         12    a useful and friendly product for our users.

         13    We also discovered that there are several

         14    advantages to providing this as a web

         15    product.

         16              The first advantage is that we

         17    could present information as we obtain it

         18    rather than waiting for a detailed product to

         19    be completed.  Initially we could always

         20    provide general information and then provide

         21    links for more details such as to methodology

         22    papers or to the actual survey forms and

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           203

          1    instructions themselves.

          2              Another advantage, as always, is

          3    that it is cost effective.  We always have to

          4    be mindful of our budget and it's always a

          5    good idea to keep that in mind.  But the

          6    third advantage is that since EIA staff would

          7    develop the product for the web themselves

          8    the staff members would benefit from the

          9    learning process since there is an element of

         10    training involved.  So let's take a look at

         11    our first example.

         12              This is a demand comparisons graph.

         13    If we look at the graph, which, by the way,

         14    is in thousand barrels per day, this is based

         15    on information currently available on motor

         16    gasoline comparisons so we're using this as

         17    our example.  EIA presents information on

         18    motor gasoline product supplied and motor

         19    gasoline sales.  Motor gasoline sales data

         20    are also available from the Federal Highway

         21    Administration.

         22              This graph illustrates these three

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           204

          1    data series, which are considered to be proxy

          2    measures of motor gasoline demand, and you

          3    can see quite clearly from looking at the

          4    graph that the data points are relatively

          5    close although there is some variation.  With

          6    regard to this graph we have some examples of

          7    questions that we might pose as a means of

          8    discussing data comparisons.

          9              This is our potential format.  This

         10    is the way that we actually envision our web

         11    product and it's just one of the formats that

         12    we're thinking about where the graph will be

         13    placed above the questions.  This is another.

         14    Now, question number 1, both EIA and the

         15    Federal Highway Administration present data

         16    on motor gasoline sales.  The actual question

         17    is why do this figures differ.

         18              We could start in answering this

         19    question by specifying the points of

         20    collection, we could talk about who the

         21    respondents are, and then we would provide

         22    links to the surveys where those links are

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           205

          1    available.  We could also explain that EIA

          2    respondents are suppliers who sell to local

          3    distributors, to local retailers, and/or

          4    their end-users.  These are respondents who

          5    are asked to report sales volume by grade and

          6    category.

          7              Looking at the Federal Highway

          8    Administration data, they collect data from

          9    the states.  Their respondents are wholesale

         10    distributors who are asked to report gross

         11    volume of gasoline and their data are taken

         12    from the state taxation reports.

         13              Let's take a look at our next

         14    sample question.  EIA also presents data on

         15    motor gasoline product supplied.  Is this the

         16    same as demand and sales?  What is the

         17    relationship to consumption?  Now, the

         18    previous slide dealt with the differences

         19    between the data series that come from

         20    different agencies.  This slide is actually

         21    more about the concept that's represented.

         22              We compute product supplied as net

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           206

          1    production plus imports minus exports and

          2    then we take into consideration the change in

          3    stock.  Now, product supplied is a proxy for

          4    demand and consumption but it is not exactly

          5    the same.  This is a challenge for EIA

          6    because of the fact that we do use the terms

          7    interchangeably.  Right now we're currently

          8    in the process of revisiting the way that we

          9    describe product supplied in our glossary.

         10    Nevertheless we do think that it's worthwhile

         11    to have a question included in the web

         12    product that's based on concept.

         13              Question number 3, do the sales and

         14    product supply series show the same pattern

         15    over time?  While it seems that the series do

         16    generally show the same pattern EIA faces a

         17    couple of challenges.  The first challenge is

         18    in deciding how much detail to go into and

         19    the second is in explaining why the series

         20    don't always move in quite the same way over

         21    time.  And this is a challenge for us at EIA

         22    because we may never be able to actually

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           207

          1    determine why that is.

          2              Our experience with comparison data

          3    tells us that it's likely that we won't have

          4    answers to all of the questions that the

          5    comparisons pose so we have several questions

          6    for the committee.  First we'd like to ask

          7    you what your general or your specific

          8    thoughts are about what we are proposing.  Do

          9    you see any merit in going forward with a web

         10    product to show the users what we currently

         11    know?  And finally we'd like to know if you

         12    have any other ideas based on your experience

         13    or from our past work that we should consider

         14    for a web product on data quality.  We'd

         15    appreciate having your input.

         16              DR. HENGARTNER:  Thank you very

         17    much.  We'll start out first with two ASA

         18    discussants, who I hope will then stimulate

         19    the rest of us to have a broader discussion,

         20    and I would like to start with Mark

         21    Bernstein.

         22              MR. BERNSTEIN:  Thanks for the good

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           208

          1    presentations.  I notice you put Randy and me

          2    on the same discussants and so both our last

          3    meeting and couldn't get to hear who are

          4    probably polar opposite in what we are

          5    probably going to talk about but either way

          6    and Nancy is going to be glad this is my last

          7    one because she has heard me say this every

          8    single time.  And maybe one of you guys is

          9    going to pick up this mantra.

         10              This seems to be the same problem

         11    I've had and maybe it's the presentation as

         12    opposed to what you actually did because I

         13    can never tell what you actually did was the

         14    presentation but fundamentally what's your

         15    objective, and believe the objective is how

         16    to make sure the data quality is the best you

         17    can make it and how to acknowledge when there

         18    are data issues, it seems to me, the two

         19    basic objectives.

         20              I appreciate the concept of not

         21    committing the same mistakes as in the past;

         22    however, you've learned a lot over the years

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           209

          1    and statistics have changed and data have

          2    changed and sometimes spending a lot of time

          3    going back in the past is irrelevant for the

          4    future but I really think you've got to be

          5    very clear what your objective is and you

          6    start with the objective and you move from

          7    there and I just didn't see the objective

          8    come out in these presentations.

          9              A little side note, the first thing

         10    that occurred to me when I was reading this

         11    and I saw those three data series, the first

         12    question I would ask is why do we need all

         13    three data series.  Are two of the three

         14    enough and could you save some money by

         15    getting rid of one?  I can understand where

         16    the differences are and I see where they are

         17    used but the fact of the matter is they don't

         18    differ that much and if you could save money

         19    by getting rid of one of them and so when you

         20    are going to do these data comparisons if

         21    you're doing these data comparisons for a

         22    bunch of different stuff and you're finding

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           210

          1    two or three data series answering similar

          2    questions it's the first step in trying to

          3    weed out things you don't need.

          4              The other thing is what you need to

          5    do to acknowledge when you know there are

          6    problems in the data and how that has to get

          7    reflected on the website.  For example, I

          8    think it's still there, and I haven't checked

          9    recently, this issue I brought up maybe it

         10    was a year ago or the last meeting.  There

         11    was a problem in the state level data for

         12    2003 for four or five different states which

         13    we uncovered while we were using the data,

         14    and my researcher said there is this problem.

         15    These things are all going up and down like

         16    this.

         17              So I asked Nancy.  I was here at a

         18    meeting.  So, Nancy, can you find out what it

         19    was?  And the answer came back as yeah, we

         20    knew there was a problem, but the fact of the

         21    matter is on the web there was nothing that

         22    said we know there is a problem and we're

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           211

          1    working on fixing it.  And so those of us

          2    doing the research end up finding this

          3    problem and not knowing what to do about it

          4    or what we end up doing is extrapolating

          5    between the two years because we couldn't

          6    have  and it just wasn't

          7    feasible.

          8              But when you know or suspect there

          9    is a data problem what you absolutely need to

         10    have is either a pop-up, though a lot of

         11    people have pop-up stoppers so that may not

         12    work, but something very specific that says

         13    either we know there is a problem in this

         14    data and we were fixing it or you don't put

         15    the data out there or we suspect there is a

         16    problem, we're not sure what it is yet, but

         17    be careful.  You really need to have that out

         18    there.

         19              And given the fact that now

         20    everything has basically been taken off the

         21    web, all the data stuff, it's not that hard

         22    to put something up there that says what your

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           212

          1    latest feeling about the data is for those

          2    people who use it.  So I think you've got to

          3    acknowledge when you know there are issues

          4    and that that acknowledgement has to be up

          5    there if the data is out there.  That really

          6    needs to happen, and that's my short comment

          7    for 

          8              DR. HENGARTNER:  Thank you very

          9    much, Mark.  Randy, do you want to respond to

         10    Mark?

         11              DR. SITTER:  Well, I disagree with

         12    everything that Mark said.

         13              MS. FORSYTH:  For six years?

         14              DR. SITTER:  Except what he said

         15    today.  It was a very nice presentation and a

         16    nice write-up.  I think you've done a lot of

         17    work and so I'll start with I know absolutely

         18    nothing about this basically.  I don't use

         19    data.  I tried.  Data just makes it harder.

         20              So in any case I do have a few

         21    comments but now that you know I know nothing

         22    about this then you can take them as you

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           213

          1    like.  Actually I think that this notion of

          2    what is an objective is really important

          3    because when I was reading it I was also

          4    saying here I'm not sure what this was about

          5    because in one sense you want to have as good

          6    a quality as possible.

          7              That, I think, was the objective

          8    but a lot of what was presented was how can

          9    we present the quality problems we have and I

         10    think those are two different things and I

         11    would hope the second one is not as important

         12    except where it really does impact and that

         13    to me is part of being good quality.

         14              So there are different kinds of

         15    presenting when we have a quality problem.

         16    One is the measure of it and how can we

         17    measure it and the other is we know there is

         18    a problem, we should let you know there is a

         19    problem, and letting you know there is a

         20    problem is probably more important than

         21    necessarily knowing exactly why.

         22              And your example is a good one.  I

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           214

          1    mean, I have an outlier, okay?  Any good

          2    statistician if he's got one that looks

          3    different from everybody else first thing he

          4    should do is go and try and find out why

          5    because it may not be a real outlier.  I

          6    mean, it may be a real outlier and not

          7    something --

          8              DR. FEDER:  Legitimate.

          9              DR. SITTER:  So if I know that you

         10    think this is suspect I'm going to treat it

         11    quite differently than if I don't know if you

         12    really thinks it's real.  So that you need to

         13    know and I think that comes across in some of

         14    the things that came to me.

         15              So that really comes down to this

         16    web-friendly thing.  It was difficult for me

         17    to tell who your users were.  The questions

         18    are very general and who they link to may be

         19    more specific but I think that thought really

         20    has to go into it if you're going to do this

         21    kind of thing about who your audience is and

         22    at what level you expect them to go into your

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           215

          1    links.

          2              The questions and answers you put

          3    at the top looked to me to be to perhaps

          4    statistically unsophisticated or not too

          5    sophisticated a user, not the people who are

          6    actually going to get into the data.  And

          7    what Mark was talking about are people that

          8    are going to actually get into the data.

          9              And for the people on the front end

         10    of that if those are really your target then

         11    you should at least let them know, for

         12    example, in your example that even if they

         13    collected exactly the same data they would

         14    get different series and that's the part of

         15    the issue because they would.

         16              The way that everything was worded

         17    on this thing was like these are real

         18    numbers, they're just measuring three

         19    slightly different things, no recognition of

         20    sampling variability.  They could in fact be

         21    measuring exactly the same thing, which is

         22    with noise, I mean it's hard to tell the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           216

          1    difference.  So that was one thing.

          2              The other thing was you've got your

          3    cost things if you're going to do something

          4    like this and there are pros and cons to a

          5    web-based structure.  I don't know if it's a

          6    pro or a con but there are inconsistencies

          7    across products so different surveys you're

          8    going to have different information up there.

          9    That can be a strength if it's done

         10    correctly.  It can be confusing if it's done

         11    poorly.

         12              And then also within products

         13    linking to reports that were created for a

         14    completely different purpose can be more

         15    confusing than it's worth sometimes with

         16    these things.  It's the easiest and cheapest

         17    way to do it.  You spend all your effort on

         18    that very front-end part and then you just

         19    slap links into it but the links are semi-

         20    related or they were done for a different

         21    purpose so they don't really tie in together

         22    and if you want a good web product that means

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           217

          1    that the things you're linking to you have to

          2    know what they're being linked from and so

          3    that certainly requires at the very least

          4    some thought into how you write those things.

          5              So I would suspect that where

          6    you're going to start with is your

          7    measurement of quality, then your internal

          8    report, and then what you're going to put on

          9    the web and there should be a pattern to what

         10    that means and a strategy for how that's done

         11    that doesn't increase workload too much but

         12    recognizes that these things are going to be

         13    linked from above in a web-based system.

         14              So a simple example is to

         15    completely change notation.  I mean, it's the

         16    nightmare if you link from three different

         17    papers and they're all using different words

         18    for different things like you use words

         19    interchangeably, maybe they use them

         20    differently, really a bad idea, and that can

         21    happen so easily when you're doing this kind

         22    of stuff.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           218

          1              The other thing is if you're going

          2    to do it when you're designing and

          3    implementing a survey you're measuring and

          4    judging these quality issues constantly so a

          5    process that captures it so that it can be

          6    put on what you're envisioning easily.

          7              So, summarizing your explanatory

          8    notes, you need to standardize what that

          9    means not just on the web but as you're

         10    developing them so that you don't have to do

         11    it three times.  I mean, you don't want you

         12    going back and looking at what other people

         13    have written, figure out how you're going to

         14    put it on the web.  That's just building work

         15    for yourself and there is no real need to do

         16    that.

         17              And again that relates back to

         18    these feature articles.  There needs to be

         19    some structure to them to standardize them

         20    across and within products so that you know

         21    these three levels and how you are going to

         22    put them up there.  Then it shouldn't

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           219

          1    increase your workload too much because

          2    you're doing all those stuff now.  It's just

          3    that different segments of your organization

          4    are doing different things and no doubt they

          5    have different terminologies for it and so

          6    forth and a different way of doing it.  So

          7    you don't want to make it too homogenized

          8    because you need variety as well.  The

          9    different surveys are different but some key

         10    things would be important.

         11              So those are the thoughts that came

         12    to mind and I don't know how they answer.  I

         13    guess those are the thoughts.  Do you see

         14    merit in a web-based product?  Yes.

         15    Advantage and disadvantage and other ideas

         16    for a web product?  No, I have no other ideas

         17    for a web product.  I like the question and

         18    answer as a front end.  I think that's okay.

         19              DR. HENGARTNER:  Johnny?

         20              MR. BLAIR:  Yes, just a couple of

         21    comments.  I think that there did seem to be

         22    this mix of talking about data quality

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           220

          1    improvement and reporting about data quality

          2    and I think it's hard to mix those in the

          3    discussion or in terms of really developing

          4    what you're going to do with them and how do

          5    you communicate about data quality.  Well,

          6    those are quite different things in a program

          7    for improving data quality and then this

          8    reporting aspect.

          9              I certainly agree with the notion

         10    that it's important to acknowledge when you

         11    think that there is a problem in the data but

         12    it's also, I think, a little bit of a tricky

         13    issue in that what's a problem for one person

         14    isn't a problem for someone else; that is,

         15    how do you decide what's a problem?  There

         16    are going to be some maybe that almost

         17    everyone would agree on the extreme outlier

         18    or whatever but there are always going to be

         19    some gray areas too where depending on your

         20    use of the data whether it constitutes a

         21    problem.

         22              If you see a spike in a standard

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           221

          1    error, well, for some people that's going to

          2    be a problem, for others not, so I think one

          3    of the things that leads to is the difficulty

          4    of the question-answer approach is selecting

          5    the questions and then stating the question

          6    in a way that seems, for lack of a better

          7    term, a fair statement because a problem can

          8    be described in in different ways, the same

          9    problem.  Two different statisticians may put

         10    a slightly different spin on it.

         11              So I think that while the question-

         12    answer approach might be a useful one that

         13    it's probably only a starting point in that

         14    you have to have some kind of mechanism for

         15    responsiveness to your users who then ask

         16    follow-up questions or who raise questions

         17    that you didn't initially think of and you

         18    probably can't respond to everything but

         19    there has to be some sort of selection.

         20              But I think that if the beginning

         21    and end point is the questions and answers

         22    that you generate that you're probably going

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           222

          1    to have a fair amount of dissatisfaction from

          2    some segment of your users and it's pretty

          3    hard to satisfy everyone.

          4              So I think aside from pointing out

          5    the obvious things, and hopefully there

          6    aren't a lot of those that you have each time

          7    that you've got to have a laundry list of

          8    here's our data set now here are the 50

          9    problems with it so hopefully this isn't

         10    something that happens a lot and that there

         11    are more not so much things described as

         12    problems but limitations in terms of the use

         13    of the data, what can these data support and

         14    what kinds of things can they support, so let

         15    me just stop with that.

         16              DR. HENGARTNER:  Thank you very

         17    much.  I'd like also to respond to the web

         18    questionnaire form.  When I was listening to

         19    the talk, I mean, I was looking at it.

         20    That's a great teaching tool.  Now, a lot of

         21    the users like Mark here probably already

         22    know all that stuff but for someone who is

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           223

          1    new starting to see the subtleties on what

          2    you are measuring and reporting becomes

          3    important because it's not just that it

          4    sells, no, because sales are at what level,

          5    what this supports, and it goes with Johnny

          6    is saying, and in a way you do address data

          7    quality in a sense and in the following way.

          8    Then you say exactly what it is that you're

          9    measuring.

         10              Instead of saying here are sales

         11    you say these are sales and they're sales at

         12    this level and this is how we collected it.

         13    And if you don't like the results, well, then

         14    go to the Department of Transportation.  But

         15    in a way it's really just defining the

         16    context.  And again I think that's more of a

         17    teaching tool.  There's a need for that but

         18    I'm not sure every one of your customers will

         19    find that important but I found it very

         20    interesting to listen to so I think there is

         21    some value to it.  Darius?

         22              MR. SINGPURWALLA:  Yes, I mean, to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           224

          1    put it on the web, I think, is a great idea,

          2    actually, to put it out there and let

          3    everyone see it and be able to use it in the

          4    way that they wanted to.  And I think Johnny

          5    made a lot of the points that I wanted to

          6    make.

          7              One of the questions I had was who

          8    is the public.  Like, is it going to be

          9    someone like me who's just going to be

         10    sitting around and my PC looking for data for

         11    an analysis to use because sometimes what

         12    works in industry, like Johnny said, what's

         13    someone's problem might be the way that I'm

         14    about to make a company a million dollars so

         15    looking at those differences.

         16              And in response to the question and

         17    answer session thing one of the things that I

         18    was wondering if you have the capability of

         19    doing was actually putting, like, data or

         20    some sort of rolled up version of data out on

         21    the web for users to manipulate their own way

         22    because it's impossible for you to probably

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           225

          1    think of every question that I'm going to be

          2    looking to answer but, I mean, in previous

          3    projects that I've worked on I know that

          4    we've been able to make web interfaces for

          5    users to actually go like an Excel pivot

          6    table like drag and drop and look at

          7    different cross tabs and do their own plots

          8    over things.  If you have the budget to do

          9    that and the technology know-how that might

         10    be a good way to answer some of the

         11    limitations of putting your own questions out

         12    there will have as well.

         13              DR. HENGARTNER:  Nagaraj?

         14              DR. NEERCHAL:  I think I agree with

         15    a couple of comments and I want to reiterate

         16    that it seems to be that the two series in a

         17    very general sense that the observation that

         18    supports the data quality is the fact that

         19    they track each other.  The fact that they're

         20    differing is something everybody expects them

         21    to differ at a small scale.

         22              It seems to me that those two

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           226

          1    things come up together in that discussion

          2    and I feel unsatisfied listening to that

          3    answer.  I think there's a grave question,

          4    why do these two series differ, but when I

          5    listen to the answer that's like, of course

          6    everything is different in a way.  And so I

          7    think that you really want to phrase the

          8    question directly in terms of quality, that

          9    directly addresses quality rather that just

         10    the why numbers differ.

         11              Of course, two numbers are going to

         12    differ so I feel that maybe you want to focus

         13    your questions more towards quality.  That's

         14    what I felt.  It's a great question but when

         15    I listened to the answer I felt unsatisfied

         16    that then it's something leaving unsaid.  So

         17    you're listing a number of things that

         18    contribute to why they're different.  That's

         19    not really going to tell me a whole lot about

         20    on the other hand the fact that they're

         21    tracking each other.  Should they track each

         22    other?  Are they measuring the same thing in

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           227

          1    a very general sense?

          2              So once again I agree.  I think

          3    this is really not meant for experts.  The

          4    experts don't need this kind of feedback.

          5    They probably have very focused questions in

          6    their mind and they probably would want to

          7    have a chance to e-mail to you those specific

          8    questions.  I think it's a great teaching

          9    tool, definitely, very nicely done, and I

         10    think there's a place for those things but I

         11    think you want to focus your customer.

         12              DR. HENGARTNER:  Thank you.  Moshe?

         13              DR. FEDER:  I just wanted to

         14    respond to one minor point that Darius made

         15    about having a web interface to the data.  I

         16    learned it in another very different context

         17    that users with the intent of retrieving some

         18    confidential information that they are not

         19    entitled to because it provides information

         20    with disclosure potential about the

         21    respondent can do that by querying the

         22    systems in different ways until they have

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           228

          1    enough cross-sections to identify a person.

          2    So I just want to caution if there's anything

          3    in the data at the micro level that you put

          4    behind such a system it's very interesting

          5    mathematically and otherwise to build some

          6    protection into the systems so that won't be

          7    possible.  That's my only comment.

          8              DR. HENGARTNER:  Tom?

          9              MR. RUTHERFORD:  I just was going

         10    to put in my two cents again as someone who

         11    tends to be oriented toward questions and

         12    objectives.  You don't develop the data just

         13    for the sake of maintaining it but we

         14    actually try to answer questions with the

         15    data.  This often happens with graduate

         16    students they want to build a trade model so

         17    they have to build a global data set and they

         18    spend all their time working on peripheral

         19    aspects of the data when really they won't

         20    identify what they need to know about the

         21    data set till they have their model working

         22    and they know what they're focused on.  Then

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           229

          1    it becomes clear that, well, this elasticity

          2    is important or this share or these are the

          3    tax rates that really matter.

          4              So my concern is just that I think

          5    that having a group within DoE that

          6    identifies data problems and tries to help

          7    users identify what the cause of this or that

          8    inconsistency is is really useful because if

          9    problems are identified as part of someone's

         10    actually trying to use the data to do an

         11    analysis that focuses you on what matters.

         12    But in the process of just going through the

         13    whole data set and looking for potential

         14    problems you could spend a lot of time and

         15    perhaps not be as contributing as useful.

         16              In other words the role of having

         17    that sort of facility if someone who helps

         18    identify things reduces the need for

         19    extensive documentation because  can

         20    get feedback without necessarily having to

         21    look up every single detail.  But my general

         22    concern is just this broad brush approach to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           230

          1    all the whole data set is not perhaps as

          2    productive  respond to specific

          3    questions.

          4              DR. HENGARTNER:  Thank you very

          5    much.  Johnny, toss a coin.

          6              MR. BLAIR:  I was just going to

          7    make one quick point and that is that the

          8    language that you use is important when

          9    you're talking about these issues.  If you

         10    label something as here's a data quality

         11    problem versus here's something that has

         12    somewhat a higher variance than we expected

         13    it can come across quite differently.  And

         14    how people look at what it is that you're

         15    putting out can leave a quite different

         16    impression depending on what labels you put

         17    on it.

         18              So I just caution that when you

         19    start putting these out, particularly keeping

         20    in mind the wide audience that you have from

         21    the students and novice users to the very

         22    high level experts, that the language that

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           231

          1    you use is important and you want to give a

          2    lot of thought to that, I think, as you're

          3    developing this approach.

          4              DR. HENGARTNER:  Randy?

          5              DR. SITTER:  Just one comment on

          6    improving quality generally, you can probably

          7    just ask people what's your biggest quality

          8    problem.  They'll probably tell you.  Focus

          9    on that.  The biggest problems should always

         10    be focused on first and sometimes it's so

         11    easy to lose track of that.

         12              So ask yourself what's your biggest

         13    problem.  If you know what it is, which you

         14    probably know your top three right now, just

         15    focus on those things first.  And I think

         16    that if you use that strategy in quality

         17    improvement generally you'll always have good

         18    quality.  And the other one in quality that's

         19    always true is a watched pot never boils.

         20    Keep that in mind.

         21              DR. HENGARTNER:  Thank you very

         22    much.  Well, that was a good discussion.  I'd

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           232

          1    like to now to move on to our next

          2    presentation.  Again I'd like to welcome

          3    Rene Miller, who's going to discuss can

          4    discrepant estimates be good.

          5              MS. MILLER:  This is actually a

          6    follow-on to what we were just talking about

          7    and I'd like to thank Lawrence Stroud for

          8    helping me get the material together for

          9    this.  I originally called the talk Measuring

         10    the Economy May not Be As Simple as 1, 2, 3

         11    after a Washington Post headline that Paula

         12    Weir pointed out to us.

         13              In this article they noted that the

         14    Census Bureau puts out two different sets of

         15    poverty estimates.  The lower line is based

         16    on the official estimate and the other line

         17    was based on a poverty measure that the

         18    National Academy of Sciences recommended.

         19              MS. KIRKENDALL:  Also put out by

         20    the Census Bureau.

         21              MS. MILLER:  Also put out by the

         22    Census Bureau.  So I thought well, this is

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           233

          1    adding knowledge and I think in general this

          2    was well received by the statistical

          3    community that the Census Bureau was

          4    providing some additional information but yet

          5    they got a headline, "Conflicting Data," and

          6    this made us start to wonder about some of

          7    our data series.  Do we have conflicting

          8    data?  Are we causing confusion or are we

          9    helping users?  So I just want to show you

         10    some examples and get your thoughts on this.

         11              If you look at our petroleum data

         12    and go through our petroleum navigator you

         13    have the option of getting data on

         14    consumption and sales and we show data on

         15    both products supplied and sales volumes.

         16    Now, these are the data that Alethea showed

         17    you and so you saw that they weren't exactly

         18    the same but we do provide some description

         19    as to what these concepts are before you

         20    actually get the data and in our minds these

         21    are two separate series.

         22              This goes back to what Mark was

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           234

          1    saying.  Well, do we really need both of

          2    them?  But through the years we've had both

          3    of them and we consider these two distinct

          4    series and I guess something to consider is,

          5    first of all, is this confusing for users and

          6    then there are other implications about what

          7    we show.

          8              To make things more interesting for

          9    our consumption data I don't know how clearly

         10    this comes out but we also have data from

         11    end-users.  So, for example, we have data

         12    from manufacturing establishments, this is

         13    our industrial page, and then we also have

         14    sales and delivery data collected from energy

         15    suppliers and we label it as such but yet

         16    they're both used as surrogates for

         17    consumption.

         18              Now, the data from the

         19    manufacturing energy consumption survey

         20    aren't supposed to be exactly the same as the

         21    data from the industrial sector because MECS

         22    is a subset of the industrial sector.  So

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           235

          1    basically here we just hope that it's lower,

          2    which it is in most cases, but over the years

          3    there have been challenges in reconciling our

          4    data from end-users with the data that we get

          5    from suppliers.

          6              Another example of series that may

          7    be considered conflicting data, we have sales

          8    of fuel oil and kerosene and then we also

          9    have what we call adjusted sales of fuel oil

         10    and kerosene.  The adjusted series is bench-

         11    marked to product supplied and we did this

         12    because in the early days of our sales survey

         13    I don't think we were entirely comfortable

         14    with the frame.

         15              We weren't sure it was complete and

         16    so we benchmarked to product supplied and

         17    then over time we have developed users who

         18    were interested in this series so we show

         19    both sets of numbers.  If you click on

         20    distillated fuel oil and kerosene, for

         21    example, you'll get links to definitions and

         22    to the explanatory notes and the explanatory

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           236

          1    notes explain the difference between the two

          2    series.

          3              Another case where we may be

          4    showing conflicting data, this is a new

          5    situation, natural gas production.  We've

          6    been showing data obtained from the state

          7    agencies and estimates and this is what we

          8    show on our natural gas web page.  And then

          9    recently we have data from the EIA-914.  You

         10    take a completely different path to get there

         11    and we say this is the first release of the

         12    data and these data are not yet official.

         13    And we thought we were doing a service by

         14    providing these data.  We could have held

         15    back until we finished our evaluation but we

         16    thought it was worthwhile getting comments.

         17    But this is another situation where we have

         18    two sets of numbers that represents the same

         19    concept.

         20              The situation isn't just limited to

         21    data as far.  As our forecasts, the short-

         22    term energy outlook is produced monthly and

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           237

          1    the annual energy outlook, as its name

          2    implies, comes out annually and if you look

          3    at the summary tables at first it looks like

          4    we're not showing data for the same years

          5    because the AEO shows data for 2010, for

          6    instance, and the short term energy outlook

          7    shows 2005 and 2006.  But we do have an

          8    annual table for the annual energy outlook

          9    and so there are some overlapping years and

         10    since the forecasts were not made at the same

         11    time they're not the same but we make it hard

         12    for users to make the comparison.

         13              The data are in different units.

         14    The STEO, I believe, is in physical units and

         15    the AEO is in Btu.  And the STEO is in

         16    current dollars and the AEO is in 2003

         17    dollars so users can make the comparison, but

         18    they can't do it that easily.

         19              I started out by showing you census

         20    data and in May there was a seminar that the

         21    Committee on National Statistics put on where

         22    not only people from Census but National

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           238

          1    Center for Health Statistics and Bureau of

          2    Labor Statistics talked about their

          3    challenges and it was called "Discrepant

          4    Estimates, Love Them or Leave Them."

          5              One of the comments at that seminar

          6    was well, maybe discrepant estimates aren't

          7    such a bad thing.  After all, you're showing

          8    the variability in the data.  So the question

          9    for you is what do you think about discrepant

         10    estimates.  How can we help the public better

         11    understand them?  Do you think we should just

         12    restrict what we show?  Do we concentrate on

         13    developing better caveats?

         14              We welcome your thoughts on these

         15    and other issues because there are

         16    implications in terms of what we collect and

         17    what we present.  It's your turn now.  Thank

         18    you.

         19              DR. HENGARTNER:  Thank you very

         20    much.

         21              DR. SITTER:  One last chance, Mark.

         22              DR. HENGARTNER:  I'd like to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           239

          1    welcome Jae Edmonds to present his comments.

          2              DR. EDMONDS:  This would seem to

          3    be, as Rene said, a follow-on to the

          4    preceding presentations as a more general

          5    phrasing of the problem but I think it really

          6    is essentially the same problem which we

          7    looked at in particular in the last session

          8    and therefore much of what I would say has

          9    already been said, not necessarily by

         10    everyone but it has been said.

         11              The place I would start is that if

         12    you're worried about a journalist writing a

         13    piece in which they observe that at some

         14    point EIA said something and then they said

         15    something else and those two could be

         16    confused and you want to protect yourself

         17    against naivet or mischief then I think

         18    there's really only one strategy and that is

         19    disbanding EIA because there's just no other

         20    way to do that.  I mean, it's impossible.

         21              The problem that you have raised

         22    occurs all the time and in fact one of the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           240

          1    things that I did was one of my colleagues,

          2    Jerry Stokes, started something called the

          3    Atmospheric Radiation Measurement Program,

          4    another DoE enterprise.  Radiation in this

          5    sense is solar radiation so you don't need to

          6    be too worried about that.

          7              But they will frequently put

          8    together four instrument packages and get an

          9    observation of the same event in the same

         10    place at the same time, one from the ground,

         11    one from an airplane, one from a balloon, and

         12    one from a satellite, and the odds of all

         13    four of these giving the same value is

         14    vanishingly small.  What he said is what do

         15    you do about this.  Well, he said, what we do

         16    about it is we flag it.  When there are

         17    differences we just put a flag there.

         18              When you need to have an official

         19    number he said that what they do is the armed

         20    scientists get together and they decide

         21    whether one of them is the best quality for

         22    that particular observation and choose it.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           241

          1    Sometimes they choose none of the above.

          2    They'll take an average sum of them.  Some of

          3    the times it will just be their best expert

          4    judgment but they note it and they in fact

          5    note that these other values exist.  As they

          6    say, it happens all the time, a good example

          7    of where you just choose one and you let the

          8    others go.

          9              If you're looking for fossil fuel

         10    CO2 emissions you can get them in two

         11    flavors, one demand side and one based on

         12    supply estimates, and the supply side is

         13    invariably chosen as the estimate of

         14    preference and that's what you get reported

         15    out as your global fossil fuel CO2 emissions

         16    for the world.  But other instances you'll

         17    actually have averaging going on.

         18              I think you need to distinguish

         19    between the situation in which you actually

         20    have an observation for the same thing at the

         21    same time using two different instruments and

         22    you have different observations.  And much of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           242

          1    what you were discussing was in fact you were

          2    measuring things which were really different

          3    but you were worried about the potential for

          4    confusion and so this notion of flagging, I

          5    think, is a useful one.

          6              The more difficult problem occurs

          7    when there are multiple data from different

          8    observational environments where they in

          9    principle should have some kind of

         10    relationship to one another.  For example, if

         11    you were measuring something for industry it

         12    ought to invariably be a bigger number than

         13    manufacturing alone because industry is a

         14    superset of manufacturing.

         15              If the source of the estimate, of

         16    course, is different then in fact you can

         17    have these situations in which you have this

         18    flip-flopping of the relationship which you

         19    would expect a priori and when that arises

         20    that's an occasion for asking why and

         21    probably a place where you again want to put

         22    a flag with some notion that you're looking

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           243

          1    into this or at least whatever the state of

          2    play really is.

          3              I think that those instances where

          4    you're looking at data there you in principle

          5    have an event which occurred and therefore in

          6    principle you should be able to have a true

          7    measurement.  I think that's different than

          8    the case in which you're looking at forecasts

          9    where you're looking forward out of a

         10    foundation of history and the fact that you

         11    have two or three different models that were

         12    designed for different purposes which give

         13    forecasts for the same future date that may

         14    not be the same I think the basic result that

         15    you're trying to get out of models is some

         16    kind of insight and in fact I don't worry

         17    about trying to make sure that those can

         18    always be reconciled.

         19              Those in my mind can be very useful

         20    in that they can give you a reflection that

         21    the future in fact is not etched in stone and

         22    there is not one future that that must

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           244

          1    evolve.  So those are the things that came to

          2    my mind in response to your presentation.

          3              DR. HENGARTNER:  Thank you very

          4    much.  We have some time so I'd like to open

          5    discussion some more about data quality and

          6    the topic we've heard so far this afternoon.

          7    Yes?

          8              DR. BURTON:  Just to follow up on

          9    what you said, in a world without any

         10    resource constraints more data are always

         11    better.  The idea that something would be

         12    suppressed or laid aside or not be made

         13    available because of the potential to

         14    introduce confusion troubles me because there

         15    are two possibilities.

         16              One, you have the same things being

         17    measured in two different ways or

         18    alternatively you have two series that have

         19    similar but not quite identical definitions.

         20    In either case, looking at the differences

         21    between the series, whether it's from two

         22    different measurement directions or whether

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           245

          1    it's from just similarly defined series,

          2    looking at the differences between those can

          3    sometimes be really instructive.  We can find

          4    things in there that are sometimes very

          5    useful to us.

          6              The only thing I'd ask is that you

          7    be very careful in how things are labeled so

          8    that I as a user when I look at them realize

          9    that I have two different series and why.  If

         10    you've got petroleum supply as reported by

         11    state agencies then it needs to say that if

         12    it's petroleum supply as reported by

         13    producers then as long as I can see those

         14    differences I'm better off having both

         15    series.

         16              DR. HENGARTNER:  Nagaraj?

         17              DR. NEERCHAL:  How did the Census

         18    Bureau respond to that story?

         19              MS. KIRKENDALL:  Actually the

         20    answer is that in response to the CNSTAT, the

         21    Committee on National Statistics, study on

         22    poverty measures the Bureau of the Census

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           246

          1    came up with not just one extra poverty

          2    measure but, I don't know, 14.  And they have

          3    been publishing all of these experimental

          4    measures for a long time now.  I think they

          5    started when I was at OMB so I think it's,

          6    like, five years.

          7              And I think the thought was that

          8    OMB at some point would pick one to make it

          9    the new poverty measure but they haven't done

         10    that and so we're still going along with all

         11    these 14 different poverty measures so there

         12    are some reasons.  I think the article picked

         13    on OMB for not picking one.  They weren't

         14    ready to make a decision.

         15              MS. MILLER:  Yes, I mean, the

         16    Census Bureau points it out that it wasn't up

         17    to them.

         18              MS. KIRKENDALL:  The Census Bureau

         19    is not going to pick how to define poverty

         20    for the United States.  That's a

         21    responsibility with the Office of Management

         22    and Budget, and they wanted to do that, yes.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           247

          1              DR. NEERCHAL:  So that analogy

          2    applied here also then?  If there are two --

          3              MS. KIRKENDALL:  We don't have any

          4    series that I know of that are quite at that

          5    same level of importance as the official

          6    poverty measure for the United States.  And

          7    so we don't have things that are dictated

          8    quite so much as to how we should do them.

          9              DR. HENGARTNER:  Mike?

         10              MR. HILL:  Actually I thought the

         11    department example is a nice one.  I actually

         12    get that question from students, that they

         13    have several different measures, so it's well

         14    known enough so a lot of undergraduate

         15    students know they  two books they

         16    have different numbers it's supposed to be

         17    the same data for the same year for the same

         18    phenomenon.  You  different

         19    numbers so I thought it was a good choice of

         20    an example.

         21              I just  the initial

         22    part of that maybe as a response to one of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           248

          1    the questions from the last what, last half

          2    hour, last session.  I mean, who is the

          3    end-user?  So in some sense one of the end-

          4    users must be the readers of the Washington

          5    Post or somebody on that level.  If that's so

          6     the public so you're not dealing

          7    with experts.  You want to be fairly general

          8    at that level.

          9              You could imagine something at

         10    another level you could have questions that

         11    come up  in the Middle East in

         12    January 1991.  One should go check to see

         13    whether there is a change in oil consumption.

         14    At that level you want one set of data.  And

         15    then at another level you're actually at the

         16    level  the last hour where you

         17    want several different time series and you

         18    want to check the differences between those

         19    different time series.  Those are several

         20    different levels and different types of

         21    users.  Maybe that's your graduate student.

         22              They're completely different types

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           249

          1    of users and I just would think that you'd

          2    want different presentations for each one

          3     but the problem is challenging

          4    from that end.  There are many different

          5    users out there who'll want different data,

          6    different time series.

          7              DR. HENGARTNER:  Thank you.

          8    Johnny?

          9              MR. BLAIR:  Just another comment on

         10    the same issue of when you've got these

         11    comparisons that differ, that jumping right

         12    to the comparison seems to me that you want

         13    to be careful not to overlook that first you

         14    need a clear description of each of the

         15    individual pieces of it before you get to the

         16    comparison both in terms of the description

         17    of it and how it's labeled.  You might just

         18    bypass the issue if that part is done

         19    correctly and completely first.

         20              And that's in a sense a problem

         21    with the web, though I suppose it can happen

         22    with text too, is that someone goes right to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           250

          1    the graphic and starts to look at things

          2    before having read the background and part of

          3    the problem is getting them to go back and

          4    read the background and see what they're

          5    looking at for each of the individual

          6    measures.  So I think that needs to be on

          7    your list as well.

          8              DR. HENGARTNER:  Yes, Mark?

          9              MR. BERNSTEIN:  I think this came

         10    up also in the previous session and I just

         11    want to make it clear again in this session

         12    which is the first thing you need to ask

         13    yourself is should there be a discrepancy.

         14    And the answer to that question leads you to

         15    two different  to do about it.

         16    If there shouldn't be a discrepancy because

         17    they're answering different things you need

         18    to explain yes, they should be different and

         19    here is why they should  bring

         20    up and then if there shouldn't be a

         21    discrepancy then you've got to start dealing

         22    with, well, why is there and what do we need

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           251

          1    to do to fix it.

          2              And so when you do see those things

          3    the should question should come first and

          4    then that sets you into two different paths.

          5              DR. HENGARTNER:  Johnny?

          6              MR. BLAIR:  Part of the

          7    interpretation is what your expectation is,

          8    that is, what it is that you expected to see

          9    quite often in looking at data.  On another

         10    committee that I'm involved with they look at

         11    test scores of 4th and 8th grade kids and

         12    they look at differences in math scores and

         13    reading scores and so forth and every now and

         14    then I'm looking at trends over time.  You

         15    see a mean score go up radically or you see a

         16    score go down radically and I discovered

         17    early on as a member of that committee that

         18    it's only when the score goes down radically

         19    that it's considered an anomaly.  So one has

         20    to be careful and keep in mind that there are

         21    expectations as to how you interpret these

         22    things.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           252

          1              MR. BERNSTEIN:  That's interesting

          2    to say because I've just been dealing with

          3    that in my son's school and last year all of

          4    a sudden the 4th graders are just like way up

          5    here and everybody was touting it as a great

          6    success and I said wait a minute.  Next year

          7    can be very different and you ought to be

          8    worried about whether next year it's going to

          9    go down like this and you're going to start

         10    wondering oh, my God, we're not doing as well

         11    though I don't think energy quite works that

         12    way.

         13              DR. HENGARTNER:  Nagaraj?

         14              DR. NEERCHAL:  I really liked the

         15    way Mark put it.  I think that those two are

         16    the really key issues that will make those

         17    questions focus a little bit better.  First

         18    thing it's going to give you a definition of

         19    what each series is and the clear definition

         20    will go a long way into what the user is

         21    going to expect to see.  And another thing is

         22    that I felt that way when I saw the graphs

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           253

          1    and I know pictures are great but once you

          2    pull those pictures, once I see that graph, I

          3    want them to be the same regardless.  It's

          4    like a psychological thing.  I say I want

          5    these two series to be the same because they

          6    are going so closely together.

          7              So you can give me a long

          8    explanation as to what they are doing.  I

          9    think are you sure, and that is particularly

         10    the case when you're talking about this

         11    annual model-based forecast.  I don't think

         12    it's possible for you.  One or two web

         13    screens may not be enough to explain the

         14    differences between the forecasting methods.

         15    My experience with this committee shows that

         16    forecasting methods are quite complex and

         17    they are quite different so you may be

         18    setting yourselves up for a impossible task

         19    of explaining two complicated models and they

         20    should be different anyway mathematically

         21    even if they are starting with the same or

         22    identical data set.  So you may want to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           254

          1    temper the expectation of the user to some

          2    extent in your write-up.

          3              DR. HENGARTNER:  Anybody else?

          4    Yes, Renee?

          5              MS. MILLER:  I just wanted to

          6    follow up on the comment that Jae made about

          7    flagging.  This also has to do with what Mark

          8    was saying about identify whether there

          9    should be a discrepancy.  The situation we

         10    have, like, with the manufacturing energy

         11    consumption data and our data on industrial

         12    sector consumption, so we don't expect them

         13    to be exactly the same but we do expect the

         14    data from the manufacturing energy

         15    consumption survey to be lower than the

         16    industrial sector figure.

         17              So for residual fuel oil we've

         18    noticed that it's not lower.  We've been

         19    looking into this for quite a while and we

         20    haven't quite found out what's causing this.

         21    So would that be something that would be

         22    useful to put a flag on either or both?

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           255

          1              MR. BERNSTEIN:  Absolutely.

          2              MS. MILLER:  But what would we say?

          3    Wouldn't we be raising more questions?

          4              MR. BERNSTEIN:  Well, but you've

          5    got to raise those questions.  Absolutely, if

          6    you know that the manufacturing fuel oil

          7    should not be higher than the industrial and

          8    that's for sure and it is you need to make

          9    sure and that if somebody is going to use it

         10    they're going to use it at their own peril.

         11    But they need to know that they're using one

         12    of those numbers at their own peril.

         13              DR. SITTER:  Well, this one is a

         14    striking example of it doesn't matter who the

         15    user is.  If this is an unsophisticated user

         16    that wouldn't know that that's supposed to be

         17    true they're going to screw it up.  If it's a

         18    sophisticated user they're going to identify

         19    it for you and they're just going to be very

         20    puzzled by your data.  They're going to be

         21    asking you what's going on.  So it's got to

         22    be flagged for every level of user.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           256

          1              MS. MILLER:  Actually we haven't

          2    gotten questions from the outside users.

          3    We've gotten questions from internal users.

          4              DR. SITTER:  That's because the

          5    unsophisticated users won't notice.  They're

          6    just using it.

          7              MR. BERNSTEIN:  But you might be

          8    getting people using one or the other and the

          9    fact is you know that one or the other isn't

         10    exactly right.  Maybe they're both not right

         11    but they can't both be right so it's one or

         12    the other.  And you may get a lot of users

         13    using one and a lot of users using the other

         14    and you really should let them know that

         15    there may be a problem here.  I know that's

         16    hard because then you get blasted for having

         17    poor data.  But on the other hand you've got

         18    to be honest about it and up front.

         19              MS. MILLER:  I mean, the challenge

         20    is we don't really know what to say.  We've

         21    looked into a number of things and so far --

         22              MR. BERNSTEIN:  You have a little

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           257

          1    thing that says the manufacturing fuel oil

          2    data is higher than our total estimate for

          3    industrial.  We don't know why or we're

          4    working on the problem but the fact is you

          5    don't have to go into a lot of detail other

          6    than the fact that you let people using each

          7    one know that there is this issue hanging out

          8    there and it's possible that a user out there

          9    would actually have a solution that you

         10    haven't thought of and somebody is going to

         11    come up and say, well, maybe it's because of

         12    this, this, and this.

         13              DR. HENGARTNER:  So you're

         14    suggesting to have like a prize to explain?

         15              MS. MILLER:  Right, a prize.

         16              DR. BURTON:  This is the sort of

         17    thing that as a data user absolutely scares

         18    me to death because when I look at that I

         19    don't know if this is one case where things

         20    are messed up or whether they're all messed

         21    up but this just happened to be the one where

         22    the manufacturing exceeded the industrial.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           258

          1    So, I mean, if I see even one record like

          2    that it makes you disinclined to use the data

          3    at all.

          4              MS. MILLER:  But if we flagged it

          5    wouldn't that --

          6              DR. HENGARTNER:  No, because then

          7    that shows you care about the quality of your

          8    data.  The others that are not flagged people

          9    will have confidence in them and they'll say

         10    well, at least they made due diligence.  If

         11    the end-user finds that error without you

         12    having flagged that then he's going to say

         13    the same problem occurs in all the series.

         14              DR. BURTON:  That helps, flagging

         15    helps, but that's the sort of problem that

         16    you've just got to fix.

         17              MS. MILLER:  But that's the

         18    problem.

         19              DR. BURTON:  Mark said fix it.

         20              MR. HILL:  May be three or four

         21    different types of users.  The sophisticated

         22    user wants to see the flag there and wants to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           259

          1    know that I'm not crazy, right.  There is

          2    something wrong and they know that it's wrong

          3    and either they're fixing it or they're not

          4    fixing it.  Then with the nonsophisticated

          5    user there's another problem 

          6     a hidden item here.  Years ago

          7    there would have been a hard copy some place.

          8    I don't know if that's what you're worried

          9    about, is a hard copy that says there's an

         10    error here, and you don't want that hard copy

         11    around for 100 years  now it's

         12    all web-based so  an error

         13    you could put it out in whatever you've

         14    designed for that.

         15              DR. SITTER:  And one of the things,

         16    answering Mark's comment there, is just that

         17    if you have a history, if you do this all the

         18    time and they see that you fix it and you are

         19    able to identify them as small problems, he's

         20    not going to be as shaky in the future.  You

         21    know what I mean?  He's seen this in the past

         22    and, generally speaking, your data is of good

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           260

          1    quality and you're keeping track of it, so of

          2    a general reputation of dealing with these

          3    things well and not these big global problems

          4    that sometimes occur.  That will give them

          5    more confidence and they'll still be able to

          6    use your data.

          7              MS. MILLER:  I think the tendency

          8    has been once we have resolved the problem

          9    then we describe and document and footnote

         10    it.  But when we haven't we just --

         11              MS. KIRKENDALL:  We can't figure it

         12    out.

         13              MS. MILLER:  We just keep looking.

         14              MR. HILL:   actually

         15    I've personally used a data set in one of the

         16    class examples.  It's an energy consumption

         17    and there are two points that are off-scale

         18    and I have no idea  it would be

         19    nice to have a note that we?ve got these two

         20    points are  something happened 

         21     whatever the story is for those

         22    two points it would be nice to know.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           261

          1              MS. MILLER:  You're saying even if

          2    we don't have the story --

          3              MR. HILL:  You don't 

          4              MR. RUTHERFORD:  It's really a

          5    matter of psychology, right?  It's a matter

          6    of educating your users to the fact that this

          7    is a process.  This isn't something that's

          8    given on high.  It reminds me a bit of in

          9    Russia under the Soviet system Gaz Komstat

         10    (?) collected all the data and they did all

         11    these surveys and everything was knock,

         12    knock, bang, bang, everything was very tight.

         13    And then under fiscal federalism there is

         14    this formula they've adopted for how they

         15    share funds from the center and suddenly

         16    there is a lot more debate about what these

         17    actual statistics are.

         18              And in Russia it's as though

         19    suddenly the world has changed.  We knew

         20    exactly how things were before and now

         21    there's debate that goes back and forth and

         22    people feel very unsettled by it whereas as

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           262

          1    an American company you say Okay, well, this

          2    is the fact of the matter.  This is a survey.

          3    How do you expect this to be so precise?

          4              So I really agree completely with

          5    Jae that you flag it and you say we're

          6    working on this as what we know.  And the

          7    fact that the Washington Post has some writer

          8    who has this misconception about how the

          9    world works who takes off and writes some

         10    crazy article, I wouldn't worry about that.

         11              DR. SITTER:  I think it's just a

         12    matter of setting up a best practice and

         13    following it.

         14              MR. BERNSTEIN:  How are most of the

         15    issues uncovered or maybe, like, what are

         16    they?  Or is that too hard a question?

         17              MS. MILLER:  Well, I mean, this

         18    particular issue with residual fuel oil came

         19    up actually when the people who were doing

         20    the long term forecasting were trying to

         21    decide what to use as their base.  Do they

         22    use the MECS number or do they use the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           263

          1    industrial number?  So they were the ones who

          2    noticed it so it was internal users.

          3              MS. KIRKENDALL:  We're fortunate

          4    that we have a lot of internal users.  We

          5    have several groups of modelers who are very

          6    knowledgeable heavy-duty users.

          7              DR. BURTON:  If I'm using data from

          8    a new source that I haven't used in the past

          9    when I start to use the data just in the code

         10    I'll make sure that I've checked the things

         11    I've got just to see how good the data are

         12    before I take off.

         13              DR. HENGARTNER:  Just as an

         14    example, Mark, if you'd find an error would

         15    you go back and flag it to the EIA or what

         16    would you do?

         17              DR. BURTON:  It depends on what

         18    impact it could potentially have on the

         19    analysis.  There are times when I would and

         20    there are times where if I've got 500,000

         21    records I'd just throw it out.  If it happens

         22    100,000 times then I got to go do something

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           264

          1    about it.

          2              MR. BERNSTEIN:  Well, in my case if

          3    it was one state in that one year that had a

          4    problem I probably would have ignored it but

          5    there was a group of problems.  Before I was

          6    part of this group I probably have contacted

          7    the EIA because we needed to use the data and

          8    I would have said what's going on here.  And

          9    the problem I had in this case was that the

         10    analyst knew there was a problem.  So oh, you

         11    didn't find this other state too?

         12              MR. RUTHERFORD:  That cost us a

         13    week's worth of work, actually, and to me

         14    it's a business setting type of thing.  And

         15    that was a week's worth of wasted money that

         16    we could have avoided if there had been just

         17    some flag there that said here are the states

         18    that we know there is a problem.

         19              DR. NEERCHAL:  Could I ask a

         20    related question?  Do you have an e-mail or

         21    something that if you find problems with this

         22    data please let us know?  If an unknown user

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           265

          1    somewhere in the corner of the country is

          2    using the data and finds out a data issue

          3    can we motivate that person to come back to

          4    you with a question?

          5              MS. KIRKENDALL:  Our e-mail, you

          6    can send that to the webmaster on our website

          7    and those e-mails do get back to people to

          8    solve problems.

          9              MR. BERNSTEIN:  And all the reports

         10    and things have a contact.

         11              MS. KIRKENDALL:  And they all have

         12    contacts, yes.

         13              DR. NEERCHAL:  Do you get e-mails

         14    like that in the past?

         15              MS. KIRKENDALL:  Yes, we get

         16    e-mails with questions about quality and is

         17    this wrong.  We do get some comments like

         18    that.

         19              MS. MILLER:  They don't all go to

         20    one place.  They go to a particular subject

         21    area.

         22              MS. KIRKENDALL:  Some of them come

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           266

          1    into the NEIC or the webmaster and he splits

          2    them out and sends them out to the

          3    appropriate person with the questions about a

          4    certain product.  He'll send it back to the

          5    owner of the product to resolve.

          6              DR. NEERCHAL:  Keeping track of

          7    that also could be a quality indicator.  If

          8    you have fewer and fewer questions about a

          9    particular database it's a good indication

         10    that they're weeding out the errors.

         11              DR. SITTER:  Sometimes if you have

         12    one person or somebody that actually sees

         13    them all in a more global sense you can

         14    discover bigger problems that aren't specific

         15    because they're happening in different places

         16    for different surveys.

         17              DR. HENGARTNER:  Randy, do you

         18    think one could detect a frame drifting by

         19    just looking at the data or errors like that?

         20    That's one of the issues always.  I mean,

         21    where does the source come from and the

         22    sampling frame is --

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           267

          1              DR. SITTER:  That's a good example

          2    of something that you might see if you've got

          3    somebody that's looking a little more

          4    globally and not so hot spot specific is that

          5    the complaints can be in relation to

          6    something, some other source or something,

          7    and you start to realize that maybe it's

          8    something bigger than just your methodology.

          9              MS. KIRKENDALL:  Actually I think

         10    that first graph that Renee showed that was a

         11    little hard to see from that very old

         12    document did illustrate that because EIA

         13    compared its results to Federal Highway for

         14    some years and there was, like, a consistent

         15    difference and we couldn't explain the bias

         16    completely but there are some anomalies and

         17    then they started to get apart.

         18              Then people realized, well, there's

         19    something changing and we weren't capturing

         20    it and they discovered the blenders.  And

         21    that's one of the reasons why the data

         22    comparisons can be useful.  I'm just looking

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           268

          1    at them periodically and you can explain some

          2    things and you can try to improve the

          3    relationship between the series.  But looking

          4    for changes between the series can help you

          5    when your frame's deteriorating, something's

          6    changed in the industry, a number of things.

          7              MR. SINGPURWALLA:  Well, and for

          8    repeatable surveys too back in my old job we

          9    used to look at distributions of the variable

         10    itself against the previous month or year or

         11    whatever it was repeated over time.  And

         12    that's always shown that the boat had sprung

         13    a few leaks, lots of, like, really simple

         14    internal quality control measures that it

         15    would sound like a SAS program setup or

         16    something and you feed in your data set and

         17    it just spits out one report.  It's, like,

         18    really quick to do and you can program it in

         19    Excel to, like, automatically highlight

         20    variations that are big.

         21              DR. HENGARTNER:  Any more questions

         22    and comments about data quality?  Yes.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           269

          1              MS. WAUGH:  Hi, I'm Shawna Waugh

          2    and I work with Renee.  I actually worked on

          3    the residual fuel problem.  Renee asked a

          4    question earlier about how to label this.

          5    And what occurred to me was that FDA, I think

          6    it's FDA anyway, provides labeling, like, if

          7    you buy things from the grocery store, if you

          8    buy cigarettes.  It's got warnings on it.

          9    What kind of warning label can we put on a

         10    problem like this?

         11              MR. BERNSTEIN:  Maybe it's more

         12    like the side effects, little in the

         13    commercials the little tiny thing at the

         14    bottom, the side effects of this.

         15              DR. EDMONDS:  Don't take the

         16    industrial data together with the

         17    manufacturing.

         18              MS. KIRKENDALL:  In answer to your

         19    question this was a debate a long time ago

         20    and then the petroleum data, the imports

         21    data, are of much less accuracy, especially

         22    preliminary ones, than the other data.  It's

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           270

          1    just really variable and it's hard to get

          2    them right.

          3              So preliminary estimates came from

          4    the weekly and everybody knew that wasn't

          5    very good.  So the way they talked about

          6    showing the difference was to label it with a

          7    little skull and crossbones as a footnote

          8    symbol.

          9              MR. BERNSTEIN:  But in all

         10    seriousness, I mean, you do have to think

         11    about how you say it and I think somebody

         12    here, I think Johnny, said that how you say

         13    it, the interpretation is either that there's

         14    something bad or wrong or that on a positive

         15    note there are issues in data all the time

         16    and you're working on fixing it.  So the

         17    warning flag or the flag could be simply we

         18    recognize there may be a discrepancy in this

         19    data or there may be a problem in this data

         20    that is being worked on.  And if you have

         21    specific questions here is how to get some

         22    answers on it.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           271

          1              DR. EDMONDS:  I think it's just

          2    being straightforward in the instance where

          3    you have an occasion where manufacturing is

          4    greater than industrial you just say we would

          5    expect that the relationship between this

          6    data set and the values in the other data set

          7    would be as follows and they aren't and

          8    therefore we are looking into this and watch

          9    this site because when we uncover this we

         10    will post the results of our investigation.

         11              MR. RUTHERFORD:  Reflecting, I was

         12    thinking I had seen something like before.  I

         13    was trying to rack my brain as to what it is.

         14    It's in compilers.

         15              If you use the best Fortran

         16    compiler around I think it's Leahy.  It's

         17    been around for a long time.  Tom Leahy goes

         18    back to UNIVAC, a long time back, but on his

         19    website they post not just what are the

         20    outstanding bugs they're trying to fix but

         21    they give you the history of all the bug

         22    fixes they've done back for 15 years.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           272

          1              So really high quality software

          2    manufacturers actually not only do they tell

          3    you what the problem of the week is.  They

          4    tell you historically what problem was

          5    generated when.  And I think it's simply just

          6    this idea that they want to reinforce this

          7    notion that hey, we're in there with you,

          8    we're doing this, we're working on this.  By

          9    not hiding it they're being up front about

         10    the fact that they're actually actively

         11    working on it.

         12              DR. HENGARTNER:  The entire open

         13    software is like that.  The Linux kernels,

         14    they have the same thing.  LaTeX is also the

         15    same thing.  Kanouth has the whole list of

         16    errors, when he has found them, what he has

         17    fixed.

         18              DR. SITTER:  And then, of course,

         19    when they update the upgrade they put across

         20    and these were all fixed in the previous --

         21              MR. RUTHERFORD:  And now we're

         22    going to introduce the following bugs.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           273

          1              DR. SITTER:  Actually I require

          2    flags of my students.  Whenever they give me

          3    an estimated probability that's outside the

          4    range 0 to 1 they must flag it or they will

          5    get 0 no matter what they've done.  And I

          6    state that up front and I still give out a

          7    few zeroes.

          8              DR. HENGARTNER:  Also the other

          9    thing is just preliminary estimates, mark

         10    them as such.

         11              MS. KIRKENDALL:  That we do.

         12              DR. HENGARTNER:  And people will

         13    know, right?  And they won't expect them to

         14    be final results.

         15              DR. SITTER:  I think the point

         16    being made here and I think underlying all of

         17    this is that there's a notion of

         18    embarrassment that we want our data to be

         19    perfect.  I think you have a bigger mission

         20    and that is to educate the public that 

         21    data is not perfect.  It's not the way it

         22    works.  So implicitly that you're trying to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           274

          1    give the best possible data and that you're

          2    working on it is not something you should be

          3    embarrassed about at all.  You should be

          4    proud of it.

          5              DR. NEERCHAL:  To add to that, I

          6    think simply because you're putting a warning

          7    on one data point and no warning on the other

          8    the no-warning one is not perfect, obviously.

          9    As far as you know you have not discovered

         10    any errors on it.  In front of every sentence

         11    the understood phrase is, "I think."  It

         12    would soften your language quite a bit by

         13    starting every sentence saying, "I think."

         14              MR. BERNSTEIN:  I do realize,

         15    though, it differs and you do have this

         16    political dimension, I mean, in which

         17    somebody in the House and somebody on their

         18    staff says hey, these guys got bad data over

         19    here, let's cut their budget.  I realize that

         20    issue permeates some of it but if you

         21    carefully craft it and if it's not showing up

         22    on every database then you should be all

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           275

          1    right.

          2              DR. HENGARTNER:  And that's also

          3    something that the review committee can bring

          4    up and say well, this is a desirable

          5    practice.  I mean, if the review committee

          6    comes and tells us well, yes, this is a way

          7    of practicing if the Congress starts to push

          8    back and say we don't like this, well, yes,

          9    but we were told by the experts, not ASA but

         10    the experts, that this was a desirable

         11    practice and that the users would understand

         12    what it means.

         13              DR. EDMONDS:  And in fact you'd

         14    much rather have your flag up there than have

         15    some committee staffer discover it.

         16              MR. BLAIR:  And I think you also

         17    want to have some way to convey the process

         18    to the user, not just that you have the flag

         19    but that there is something that says this is

         20    in general how we deal with things that we

         21    don't expect to see in the data, that we flag

         22    it and then someone's working on it in the

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           276

          1    future.  You might see it removed or you

          2    might see an additional explanation or

          3    whatever but some description of what the

          4    process is.  I think that goes along with

          5    this idea of that you're on the user's side,

          6    that you're not trying to conceal anything,

          7    that you're flagging things, but you're also

          8    working on them.

          9              But I think that needs to be said

         10    more generally whether there are flags or not

         11    hanging on a particular thing so that's

         12    constantly reinforced, what the process is.

         13              DR. BURTON:  And truthfully it

         14    could be worse than having something like

         15    this reported in the press.  I actually had a

         16    reporter call me one time.  I had transposed

         17    two numbers and instead of reporting it he

         18    gave me a chance to correct it.  Subsequent

         19    to that I did thousands of dollars of free

         20    work for him.

         21              DR. HENGARTNER:  Let's take a

         22    15-minute break.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           277

          1                   (Recess)

          2              DR. HENGARTNER:  Welcome back.

          3    This is the last session for this afternoon.

          4    Tom is going to talk about survey

          5    self-assessment and he told me in confidence

          6    that it's going to be short.

          7              MR. BROENE:  That's right.  Thank

          8    you, Nicolas.

          9              MS. FORSYTH:  Well, my commentary

         10    is going to be long.

         11              MR. BROENE:  We will see.  The

         12    other thing I didn't tell him is I really

         13    have two talks here.  When I did the practice

         14    run everyone said you've got two talks and

         15    you didn't integrate them.  So I'll tell you

         16    that up front.

         17              I'll tell you first about what's

         18    going on in Europe.  What we know now, some

         19    of it's a little old, some of it's third

         20    hand, some of it's new.  And then I'll tell

         21    you about our efforts at EIA.  That's really

         22    the best way I could think of to do it.  So

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           278

          1    we'll find out first what's going on in

          2    Europe.

          3              So the basic idea is we heard about

          4    self-assessments being done and we heard

          5    about it in that Stat Sweden first.  And

          6    people said that it was well received and

          7    that they were making good progress so we

          8    decided to try something similar.  And we did

          9    discuss this with the committee about 18

         10    months ago when we were just starting the

         11    interviews.

         12              So let's go to what I know about

         13    what's going on in Europe.  Most of this from

         14    Sweden and from Eurostat is from a conference

         15    in May of 2004 in Germany.  We didn't attend

         16    but David Marker of Westat attended.  He gave

         17    his notes to Nancy.  Nancy gave one talk and

         18    I'm summarizing what Nancy said so I hope

         19    it's right.

         20              So Stat Sweden started this in 2002

         21    and Westat worked with them on the

         22    development of their quality program.  And

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           279

          1    I'm not sure what the product is, if that's a

          2    survey or a group of surveys, but they're

          3    doing 200 products.  They're about a third of

          4    the way through them.  There was someone

          5    giving a talk at the joint statistical

          6    meeting this summer and so they've done 69 of

          7    them so far.

          8              I'm not sure if everything is done

          9    only once every five years or if some are

         10    done more frequently.  It could very well be

         11    some are done more frequently than every five

         12    years but they have a 20-page self-

         13    assessment.  The survey manager, either

         14    individually or in a group, I'm not sure,

         15    fills it out.  And then that's reviewed by a

         16    team of others from Stat Sweden, people in a

         17    different office but within the agency.

         18              Those auditors have had one day of

         19    training on how to audit and they do two or

         20    four audits during the course of the year.

         21    The nice thing about this is that it's very

         22    intensive.  It's one week.  The auditors come

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           280

          1    in, review the assessment, they talk with the

          2    survey staff, my guess is probably more than

          3    once, and then they write up their report and

          4    they are through in one week.

          5              And then they have follow-up.  They

          6    have about the equivalent of one FTE.  They

          7    have people working part-time.  I think they

          8    said three people working one-third time.

          9    And I'm not sure if that rotates but somebody

         10    is following up keeping track of all the

         11    recommendations.  They're averaging 10

         12    recommendations and two good suggestions,

         13    examples to share, and then they keep track,

         14    what's been resolved, what's being worked on,

         15    what are they thinking about, and no progress

         16    on yet.

         17              And we're seeing this too.

         18    Computer systems were the biggest category of

         19    things needing improvement but that's not a

         20    quick thing to fix usually.  For Stat Sweden

         21    the documentation was second and the easiest

         22    to correct.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           281

          1              DR. EDMONDS:  Tom, will you remind

          2    me what are we assessing?  What is the self-

          3    assessment focused on assessing?

          4              MR. BROENE:  For us it's on our

          5    surveys and I think it is for Stat Sweden

          6    also.  It's how is your survey going, what's

          7    working well, what isn't, what is your

          8    response rate, what's your estimation

          9    procedures.  The only thing I don't know for

         10    sure is if they are grouping their surveys at

         11    Stat Sweden or possibly it could be like

         12    output also.  I'm not sure about that.  But

         13    for us we're looking at on a per survey

         14    basis.

         15              That's a quick summary of Stat

         16    Sweden.  I hope this is from Eurostat.  It's

         17    possible it's from the UN.  But this is the

         18    DSAP.  They have a 55-page questionnaire that

         19    the survey major fills out.  Some of the

         20    questions are real short.  I mean, a big long

         21    list of select your imputation procedure.

         22    Others are rank order your unmet data needs

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           282

          1    and priority and describe your efforts to

          2    address those deficiencies and so that's a

          3    55-page questionnaire.  And then at the end

          4    of each section you get a question rate

          5    yourself.  So, like, for coverage what's your

          6    under coverage?  We have a little, we have

          7    some, virtually none.  The further away you

          8    are from the center the better the score.

          9              And so this gives you a way of

         10    displaying a dozen measures in one two-

         11    dimensional surface.  And then behind this is

         12    a 55-page questionnaire that the survey

         13    majors fill out.

         14              I gave an earlier version of this

         15    talk this summer and I was in the same

         16    session with Maria, the head of the quality

         17    management at Portugal's INE.  She agreed to

         18    let me use her slides provided I put in an ad

         19    for the ISI meetings.  I've heard good

         20    reports of the past ISI meetings and there

         21    are some posters in the hallway there.  No

         22    tickets but I've heard good reports and

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           283

          1    Portugal agreed to host this.  Switzerland

          2    had to back out at the last minute.

          3              So they have 800 employees.

          4    They've been in business about 70 years.

          5    There are four parts to their quality

          6    management system.  The framework has some

          7    ISO norms.  They are following a European

          8    foundations quality model.  Documentation,

          9    they have handbooks, they have checklists,

         10    they have minimum documentation requirements.

         11              I'm just going to cover the

         12    involvement and the audits.  Maria spent

         13    about a third of her talk on this.  So there

         14    are 800 employees.  How to get the 800

         15    employees involved?  She had eight slides on

         16    this.  This is my summary of those eight.

         17    They have internal meetings once a month.

         18    Everyone is invited.  She thought it was

         19    working well.

         20              They have a suggestion box.  You

         21    can do it by paper or electronically.

         22    Supposedly top management answers every

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           284

          1    suggestion and then there are awards both for

          2    the best suggestion during the year and for

          3    the best improvement in quality.  So those

          4    are some of the things they are trying to get

          5    the 800 employees involved.

          6              In terms of the audits it's mainly

          7    a matter of are you following the checklists

          8    in the handbooks.  I mean, if you've got the

          9    good procedures laid out, the minimum

         10    documentation requirements, then it's a

         11    question of are you following them and do you

         12    need any changes in those procedures.

         13              Again, they have two or three

         14    auditors on each team.  I'm not sure if they

         15    rotate.  It looks like they were more on a

         16    permanent basis in Portugal but I'm not sure

         17    about that.  But they were trying to really

         18    involve the auditors.  It wasn't just that

         19    they come in and critique things.  It's

         20    really trying to work as a consultant and try

         21    to address solutions.  And so that was one of

         22    the things they were trying to do there.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           285

          1              That's what I know about what's

          2    going on in Europe and that's a summary.  But

          3    we didn't know any of that when we started on

          4    this.  We started this in 2003 and we formed

          5    a team.  Bob King and Paula Weir are here.

          6    They were on it.  We had at least one

          7    representative from each of our offices that

          8    run a survey and we worked with David Marker

          9    and David Morganstein at Westat.  They came

         10    in here and ran a couple of sessions for us

         11    and helped us draft a list of questions.

         12              And we tried to address all ongoing

         13    aspects.  We didn't look at the startup

         14    issues.  I don't know if there was a separate

         15    handout of the questions.  I'm not sure that

         16    got distributed but if you're interested I

         17    have that set of questions.  It's half-dozen

         18    pages including a summary of quality

         19    measures.

         20              The basic goal, again, this is just

         21    one approach that we have to look at the

         22    quality of our surveys and the basic goal was

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           286

          1    to get their opinion.  We didn't have what

          2    Randy just suggested, the one question,

          3    what's your biggest problem.  Maybe we should

          4    have asked that also.  We didn't have that

          5    but to get their assessment on the topics and

          6    then get a list of what they know about their

          7    data quality measures and their goals we

          8    asked for a one year ahead goal.

          9              So the format we are following is

         10    to explain the purpose, have a group meeting

         11    with myself and Nancy and Shawna Waugh of

         12    SMG, and then Shawna and I met individually

         13    with each survey major or at most two

         14    surveys.  We tried one meeting where we had

         15    eight surveys and Nancy and I were pretty

         16    confused.  At least I was confused.  That was

         17    too much to absorb especially for someone

         18    like me who's in a different office.

         19              So we're not demanding proof, we do

         20    ask follow-up questions, but, I mean, if

         21    someone says they have 100 percent response

         22    rate we ask them how they do it but we don't

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           287

          1    ask them for proof.  So far we are posting

          2    these internally.  I heard some of the past

          3    discussions just few minutes ago but the

          4    decision so far was that we would hopefully

          5    get a more honest appraisal if we did not

          6    post these on the web.

          7              And so the procedure so far is that

          8    Shawna and I write the reports to spare the

          9    survey managers the extra work of writing it.

         10    And then they get to review it and they have

         11    final say on what's included but then we're

         12    just posting it internally.

         13              We have not discussed most of this

         14    with EIA management yet.  The interviews are

         15    completed but I'm still working on the

         16    summary reports.  We've had good cooperation.

         17    One recommendation that was given 18 months

         18    ago here that we didn't follow was to

         19    interview everyone.  We didn't get to the

         20    contractors.  We ran out of time.  We did do

         21    some of the IT staff.

         22              There was a learning curve for

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           288

          1    myself and others on this.  It takes a while

          2    to do these especially since we are in a

          3    separate office.  And, of course, some

          4    managers are easier to interview than others.

          5    Most were pretty good.

          6              We got a lot of variety and who

          7    responds to our surveys.  There's a lot of

          8    staff turnover in the organizations that we

          9    contact.  They get a lot of new contacts in a

         10    given year.  Some of the contacts are

         11    contract employees so they're not even

         12    employees of the firm.  The changes of the

         13    data collected, that's a real understatement

         14    there to say they're time consuming for us.

         15    It can be very chaotic for some cases.

         16              One survey manager suggested that

         17    we should do a cost-benefit study before we

         18    make changes to our forms, try to figure out

         19    really how many of our respondents are going

         20    to be able to provide the data, how easily,

         21    how good is it, and then what's the magnitude

         22    going to be when we get it.  Is it going to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           289

          1    be 1 percent of the total?  Is it going to be

          2    something bigger than that?

          3              We got a wide variety of procedures

          4    in terms of automated data collection.  We're

          5    all over the map.  The most advanced is for

          6    the EIA-23.  It's an annual survey of oil and

          7    gas reserves and several software firms are

          8    marketing software to populate that

          9    automatically.  And they've also filed all

         10    the forms required by the SEC.  I ran over

         11    this a while back, several months back, with

         12    Howard Gruenspecht, our deputy.  He already

         13    knew about everything we told him through one

         14    of our surveys and then Nancy asked some

         15    details about a specific estimate for another

         16    survey and he said well, that wasn't covered.

         17    I don't know.  So we got a lot.  We're

         18    running about five to six pages per survey

         19    but still if you get into great detail unless

         20    it was brought up it's not going to be

         21    covered here.

         22              I gave this talk in December of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           290

          1    last year at FCSM and somebody said if you're

          2    doing a self-assessment you've got to have

          3    tough questions.  There is some evidence that

          4    two of our questions at least were too soft

          5    because people contradicted each other when

          6    they shouldn't have.  Two survey majors in

          7    the same group are contradicting each other.

          8              So we asked about training.  Do you

          9    have good training for new staff?  Or cost,

         10    can you estimate what it costs to run this

         11    survey?  Well, one person said yes, the other

         12    said no.  So those questions, I think they're

         13    still useful but they may not have been tough

         14    enough.  On the goals being selected they're

         15    pretty reasonable.  Most are projects already

         16    underway.  We did try to avoid making new

         17    work.

         18              That's a summary of our current

         19    status and any comment is appreciated,

         20    especially how often.  Can we group these by

         21    program?  Do we need a longer questionnaire

         22    and then a more rigorous review?  Do we need

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           291

          1    an expanded list of questions?  And who

          2    should be on the team?  Should it include IT

          3    people, for example?  And that's it.

          4              DR. HENGARTNER:  Thank you very

          5    much, Tom.  I'd like to invite Johnny Blair

          6    to provide his comments or responses to this

          7    presentation.

          8              MR. BLAIR:  First my general

          9    comment is that it looks like you're

         10    progressing pretty well, that you've got a

         11    reasonable structure that you are working

         12    within.  The kinds of things that you are

         13    doing seem sensible and you're obviously now

         14    focusing on obtaining this in a sense

         15    baseline information about your surveys.

         16              But what I want to do is to talk

         17    about and try to respond to some of your

         18    questions, at least, probably not all of them

         19    but in the context of looking forward as well

         20    as what you are doing right now to get this

         21    off the ground because I think it's important

         22    particularly with a self-assessment that you

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           292

          1    start now thinking about the direction that

          2    it's going to go in and how you keep it

          3    effective and keep people engaged and

          4    effective is not necessarily uniform.

          5              I think it has to be a dynamic

          6    system that's going to respond to changes in

          7    the survey changes and the economic climate

          8    that you operate in and so forth.  So I think

          9    that keeping the program healthy is something

         10    that you want to begin thinking about now

         11    even though your primary focus is on getting

         12    it off the ground.  As far as the frequency,

         13    for example, in the future I think that this

         14    is a prime example of a flexibility that

         15    depends to some extent on the results that

         16    you get, that if something is looking really

         17    great you may not need to follow up on that

         18    one as soon as another survey appears to have

         19    a serious problems.

         20              Another issue in looking at the

         21    frequency is part of the next your follow-up

         22    is going to be to look at were the changes or

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           293

          1    the goals implemented that had been set and

          2    so you have some expectation as to what's a

          3    reasonable period of time to expect to have

          4    goals implemented.  If the goals are

          5    primarily IT kinds of things there may be a

          6    different expectation in terms of time to

          7    implement than if the focus is, say, on

          8    response rate or on doing something to

          9    upgrade a sampling frame or whatever.

         10              Also as far the frequency to keep

         11    in mind, and I think this was something in

         12    maybe the Swedish plan, that it's not

         13    necessarily an all or nothing follow-up, that

         14    it can be focused and that maybe a way to

         15    make it more efficient.

         16              Second issue, coming back to this

         17    notion of keeping in mind that when I look at

         18    this list that you have under the EIA

         19    approach that begins with frames and sampling

         20    error and response and imputation and this

         21    long list is to keep in mind that you're

         22    addressing different categories of survey

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           294

          1    performance and survey quality; that is,

          2    you've got a set here that relates directly

          3    to total survey error as far as question

          4    content or response rates and so forth.

          5              And you've got other things that

          6    relate more to the user as far as

          7    documentation and communication,

          8    confidentiality, and so forth and then

          9    submissions related to cost.  I think over

         10    time as you're setting goals you want to do

         11    two things.  One, in each area I think you

         12    want to, going back to Randy's advice, what's

         13    the big issue there that needs to be

         14    addressed and focusing on that because you

         15    can't do everything but that also you want to

         16    balance that with over time being sure that

         17    you cover some of these other areas, that is,

         18    the things that are related to the user as

         19    well as the things that directly impact total

         20    survey error.

         21              You want to keep the staff engaged

         22    in this.  Any kind of self-assessment, that's

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           295

          1    important.  You want them buying into it and

          2    one of the best ways of getting people to buy

          3    into something is letting them have input

          4    into it.  And so when you're thinking about

          5    some of the questions that you're posing to

          6    us also whatever mechanism, informal or

          7    whatever, but to give your staff the

          8    opportunity to comment, to have input on the

          9    overall process as well as specific parts of

         10    it that affect them.

         11              As far as the team composition

         12    there's the question about whether IT should

         13    be on it.  I think that some rotation is

         14    possible.  Who should be on it?  It would

         15    depend to some extent on your expectations.

         16    You don't go into this completely blind.  You

         17    know something about the particular survey

         18    and where there might be issues presumably

         19    before you begin the assessment.

         20              So certainly if there were 65

         21    percent of the issues or whatever related to

         22    IT you probably had some inkling of that

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           296

          1    before you actually collected the data and

          2    get the 65 percent number and so that would

          3    suggest in that particular example that you

          4    want to have some representation from IT to

          5    do up a proper assessment and to properly set

          6    goals.

          7              And also consider, though, that it

          8    may deviate somewhat from the notion of

          9    self-assessment but to look at, at least

         10    periodically, certainly after you've gotten

         11    beyond the baseline of bringing in someone

         12    from the outside to participate and I'm

         13    thinking mainly someone from their user

         14    community who will have likely a different

         15    perspective than a totally internal view.

         16              I mean, this should be primarily

         17    internal.  That's the point of it.  But it

         18    doesn't have to be exclusively internal.  You

         19    can bring in someone user, consultant,

         20    whatever, from outside even if they're only

         21    involved in part of the process.  You may get

         22    some fresh or different perspectives.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           297

          1              As far as this question about

          2    review all the surveys in one program area in

          3    the same year, assuming that's feasible just

          4    operationally, there may be some advantages

          5    to that particularly if similar goals come

          6    out of the individual assessments and there

          7    can be some cross-survey collaboration to

          8    help address those goals.

          9              So if you've got a program of

         10    similar surveys and one of the goals relates

         11    to some data collection issue that's to some

         12    extent common across them then they can work

         13    together.  You might get a more efficient

         14    process than if those assessments come out in

         15    different years and then people are working

         16    on essentially the same problem at different

         17    times.  So there are, I think, some

         18    advantages to doing a program area.

         19              Then just a couple of other points,

         20    as far as whether you need to ask tough

         21    questions I don't know that I would put it

         22    that way.  I'm not sure exactly what a tough

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           298

          1    question is.  I think you want to leave a

          2    little room for interpretation, that you want

          3    to ask specific questions and not leave a lot

          4    of wiggle room because there obviously are

          5    some potential even if not fully conscious

          6    issues that I'm being assessed and if I say

          7    that we're doing good training that that

          8    reflects on our group and me individually.

          9    So you don't want to leave a lot of room for

         10    interpretation in a self-assessment set of

         11    questions.

         12              Anything else here?  On the

         13    question of whether you should add this

         14    greatly expanded list of questions that's

         15    hard to answer without looking at the

         16    questions but I certainly would tend towards

         17    fewer questions, identifying areas, and then

         18    drilling down in those specific areas rather

         19    than trying to hammer everything to death,

         20    again with the idea that you can't address

         21    everything.  You have some idea where issues

         22    are so why ask a huge number of questions.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           299

          1    Ask enough to see if there is an unknown

          2    problem that surfaces but you don't have to

          3    hammer it into the ground.

          4              I'll just wrap up by saying I think

          5    that it's important as much as you want to

          6    have a identifiable program with an identity

          7    to it at the same time it's got to have some

          8    degree of flexibility and it's got to have

          9    elements in it that keep the staff engaged so

         10    that after the first time when the novelty

         11    wears off that it's not oh, no, here comes

         12    this awful drudgery that I've got to do every

         13    couple of years or whatever.  I'll stop with

         14    that point.

         15              DR. HENGARTNER:  Thank you very

         16    much, Johnny, and the second discussant is

         17    Barbara.

         18              MS. FORSYTH:  Right, so I want to

         19    say first of all that I agree with Johnny

         20    that it's a nice structure that you have set

         21    up.  You've got some processes that you

         22    follow and the different surveys are exposed

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           300

          1    to the same process so it's a fair process

          2    and it's a transparent process so those are

          3    all strengths, I think, and I think Johnny's

          4    points about planning for the future and

          5    planning for ways of keeping individuals

          6    engaged are really good points since you will

          7    be implementing this across time.

          8              I looked at the statistics you had

          9    up for Statistics Sweden about their self-

         10    assessment program and I'm thinking that out

         11    of 200 that they planed to implement they've

         12    conducted 36 in two years and it's like this

         13    is a very intensive process and so no, I

         14    wouldn't increase the questionnaire but it's

         15    clear that it's a process that involves not

         16    only a lot of your investment but also a lot

         17    of investment from the survey staff and so

         18    anything to facilitate it and get feedback

         19    quickly is obviously wise.

         20              I wasn't really clear on the

         21    purpose for the evaluation.  So I understand

         22    that it's at the survey level rather than at

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           301

          1    a program level and I understand that what

          2    you're looking at are some measures of survey

          3    quality, and this is something that came up

          4    earlier as well, you could be documenting

          5    survey quality, you could be evaluating

          6    survey quality, or you could be looking for

          7    ways to improve survey quality or, of course,

          8    you could be doing all three but what you do

          9    the most of depends on which of those goals

         10    is most important to you.

         11              It sounds like the latter two

         12    goals, evaluating survey quality and

         13    identifying ways to improve survey quality,

         14    are probably the more important goals but in

         15    the results you present a lot of them are

         16    documenting differences across surveys and

         17    that's more documenting than evaluating or

         18    improving.  So the questions you ask people

         19    to think about and the questions you take

         20    into the assessments probably will vary

         21    depending which of those goals are most

         22    important.  It's not clear to me yet which of

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           302

          1    those are or should be most important to you.

          2              I liked a lot of things about your

          3    approach.  Things I really liked are first of

          4    all that you were drawing on European

          5    experiences.  They've done a lot of quality

          6    work and so it's nice that you're drawing on

          7    it and making some analogies and identifying

          8    some things that you're doing that are

          9    different, perhaps.

         10              Just as a sidebar, I don't know if

         11    anyone else would be interested in this but I

         12    think it would be really interesting to hear

         13    a methods paper on what they're doing in

         14    Europe compared with what you are doing here,

         15    perhaps how the different approaches are

         16    tailored to the different goals.  At least my

         17    understanding is in Europe they have pretty

         18    structured quality improvement activities and

         19    it sounds like this is less structured and so

         20    it would be interesting to see how your

         21    approach has been adapted for perhaps the

         22    less structured, more informal, quality

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           303

          1    improvement activities that you maybe

          2    interested in.  So, just as a sidebar, it

          3    might be interesting to get a methodological

          4    comparison and identify some dimensions for

          5    tailoring.

          6              I think it was smart to hire a

          7    consultant to help you with the assessment

          8    content.  And I also think it was smart to

          9    select a relatively narrow area to focus on

         10    survey operations as opposed to survey

         11    planning or survey design.  Survey operations

         12    is still pretty broad but it could be worse.

         13    So I think it's pretty smart to select a

         14    focused area like that.

         15              I think Johnny said this.  I really

         16    like that your surveys pick annual goals.  I

         17    think they are annual.  They pick goals.  I

         18    really like that that's an output of the

         19    process.  And I'm not sure that you mentioned

         20    this in your talk but it was in the paper

         21    that you post the successes.  You let them

         22    select successes that they'd like to be

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           304

          1    posted so that the entire organization can

          2    learn from hard lessons that have been

          3    learned by specific surveys.

          4              And I have in my notes I imagine

          5    it's important to give them some control over

          6    what you publicize.  But I can see from what

          7    Johnny said getting staff involved, giving

          8    them some input into the process, that it's

          9    probably very important motivationally to

         10    give them some control over what gets

         11    publicized.  It seemed funny to me that you'd

         12    let them edit what you wrote up in the

         13    summary but I can see now why that would be

         14    important, to make them part of the process.

         15              I wondered a couple of things and

         16    these are like methodological questions.  So

         17    I'm wondering how the survey has reacted to

         18    you guys as interviewers.  It seemed like you

         19    had interviewers from a separate group come

         20    in and talk to them about their survey and

         21    that could work really well or it could pose

         22    weird interview dynamics.  Maybe that's what

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           305

          1    you meant when you said some people are

          2    easier to interview than others.  But I'd be

          3    interested in how that worked and whether you

          4    think it would contribute to the process if

          5    you bring other team members into the

          6    interviewing activities.

          7              I was interested in your result

          8    that some results are out of date before the

          9    summaries are available.  In other words I'm

         10    wondering why whether it's the nature of the

         11    data that you're collecting that maybe the

         12    data are so volatile that it's almost

         13    impossible to get a summary out quick enough

         14    or maybe the summary process can be adapted

         15    to get a summary out more quickly, maybe

         16    fewer interviewers and more time summarizing

         17    in between them so that you can get data out

         18    while they are still useful.  So that was

         19    something I wondered about, what makes them

         20    on a date.  Is it the nature of the data or

         21    is it the nature of the reporting process?

         22              And I'll finish up by saying Johnny

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           306

          1    mentioned on asking tough questions and FCSM

          2    suggested that you should be asking some

          3    tough questions and I wondered whether in a

          4    self-assessment there is really such a thing

          5    as a tough question.  I mean, I understand

          6    the difference between objective and

          7    subjective but I think maybe the toughness of

          8    the question really is in the eyes of the

          9    beholder and in the attitude of the

         10    respondent; that is, a relatively subjective

         11    question can still get really good data if

         12    the person is seriously invested in the

         13    process and can still uncover potential

         14    problems in quality if the person is really

         15    interested in identifying problems in

         16    quality.

         17              And so I'm not sure that toughness

         18    is in the question alone.  It might be part

         19    of the process.  And maybe "toughness" isn't

         20    the right word, maybe it's "honesty" or

         21    "self-examination," but I'm not sure the

         22    questions alone are going to influence what

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           307

          1    they were describing as toughness.

          2              MR. BROENE:  Well, why don't I

          3    respond to a couple of those?

          4              MS. FORSYTH:  I think that's it

          5    except, I mean, it really is good to have, I

          6    think, as Johnny said, a systematic approach

          7    for assessing and improving quality.

          8              MR. BROENE:  A couple of responses

          9    and explanations, in terms of a separate

         10    group that's what we saw being done in Europe

         11    and so that's what we did.  Also our group is

         12    separate, the Statistics and Methods Group.

         13    The biggest drawback is we don't know enough

         14    about the individual survey so like some of

         15    the surveys I had worked on and knew a great

         16    deal about them and Shawna and some others

         17    and so we knew about some of them.  Others we

         18    had never looked at them and so that's a

         19    drawback.

         20              The out of date was mainly for

         21    doing it right at the turn of the calendar

         22    year and they were making changes in

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           308

          1    procedures and so the documentation I had was

          2    out of date.  And even though I got that one

          3    out pretty quick it was out of date by the

          4    time I got it out.

          5              MS. FORSYTH:  So it was just the

          6    timing?

          7              MR. BROENE:  The Stats Sweden

          8    approach would have been nice.  They did

          9    everything in one week flat.  And so I guess

         10    that's all the auditors work on and that's a

         11    much more intensive time commitment for the

         12    survey managers than what we were doing.  We

         13    were having a much shorter time commitment,

         14    maybe a day total.

         15              DR. NEERCHAL:  This morning we had

         16    a breakout session on the documentation and

         17    how often to update and things like that.  It

         18    seems to me that that question has to be part

         19    of this self-assessment process.  I think

         20    this is a great time to really look at that

         21    issue and for all the products coming out of

         22    that particular this may be the right time to

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           309

          1    develop how frequent updating and all the

          2    questions coming up and very complicated

          3    ones, really, but this seems like a good time

          4    to do that because you are anyway focused on

          5    that survey for a few hours or few days.

          6              And the other one in terms of

          7    addressing the burden issue, would it be

          8    possible to have two levels of details?  You

          9    have a two-page questionnaire asking very

         10    important questions at a more frequent basis.

         11    And every seven years you do a your thorough

         12    thing.  Is it possible so that you have a

         13    seven-year cycle so every year you're only

         14    doing one-seventh of them, perhaps?

         15              But on a yearly basis you still

         16    keep track of some important issues and that

         17    could be self-reported.  That could be

         18    something self-reported.  And I want to

         19    really appreciate what you said about

         20    toughness.  Someone asked me how are you

         21    doing.  It depends on who is asking me what I

         22    am going to say.  On the way if someone says

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           310

          1    hi I'm just going to say hi and if a friend

          2    asks me hi I'm going to stop and say do you

          3    have time, a couple of hours.

          4              MS. FORSYTH:  What a tough

          5    question.

          6              DR. HENGARTNER:  Jae?

          7              DR. EDMONDS:  In trying to think

          8    about this I go back one level and ask the

          9    question well, what's the purpose of all

         10    this, and actually Barbara took us back there

         11    in her comments and I would say that the

         12    purpose that I would put forward is that

         13    we're trying to improve the quality of the

         14    data, the product that this organization

         15    delivers, and so I think that it's really

         16    important to begin there because of many of

         17    your questions about how do you go forward

         18    out of this experience, I think, really need

         19    to go back and set up a test, ask yourself

         20    what was I trying to do and did this activity

         21    get me what I wanted as efficiently as

         22    possible and in a sense what were the costs

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           311

          1    and what were the benefits.

          2              Obviously you've pointed out it

          3    takes a lot of time and a lot of energy and

          4    in a sense you've diverted resources from

          5    potentially other tracks.  And so I think

          6    it's really important to ask what did you

          7    learn out of all of this.

          8              Because you're going very fast and

          9    over this it looks like essentially a blanket

         10    survey of all the individuals that were

         11    involved in producing these data products.

         12    And you're going and asking them how are

         13    things going here, what's going well, what's

         14    not going well.  And out of that you're

         15    hoping to generate some new information that

         16    you didn't already have.

         17              And so I think one of the things

         18    you want to go back and ask yourself is what

         19    did you learn.  And one of the things that

         20    you said I think is useful.  You said we took

         21    this up to Howard and Howard said I knew

         22    that, I knew that, I knew that.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           312

          1              And so the question is what is it

          2    that you are learning and how much time and

          3    energy did you divert to learning that.  So

          4    the first thing is where is it that we are

          5    doing well and where is it that we are doing

          6    badly, and then I think the next question is

          7    did this survey process generate some

          8    solutions that we hadn't already identified.

          9    And it may be that those solutions were down

         10    there in the people who managed the product

         11    but for some reason had never bubbled up to

         12    the surface and you were getting that.

         13              But I think that what I would

         14    recommend and maybe you've already done this

         15    and just in the time that you've got to

         16    present I know you just get this little

         17    sliver of time, maybe that's already

         18    happened, but that needs to happen.  You need

         19    to go back and you ask yourself that

         20    question.

         21              And you probably want to put a

         22    couple of hard questions to your European

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           313

          1    friends.  What do you guys get out of this?

          2    You spent a lot of money.  What did you learn

          3    from this?  What are you doing differently

          4    today than you were doing before you did this

          5    assessment because you're putting a lot of

          6    resources into it.

          7              The other thing that others have

          8    brought up that I'll just put out as a bullet

          9    is what are the incentives to the people

         10    you're interviewing.  In some ways it

         11    reminded me of these Maoist circles where you

         12    do self-criticism and that's probably not

         13    going to work in this environment.  And so is

         14    there an incentive?  Am I saying my data is

         15    not that good, does that imply that I'm not

         16    doing a very good job, and so what are the

         17    incentives to get a true reading, assuming

         18    that in fact you want to go forward with it,

         19    that is, you feel that this is the best way

         20    to generate information you weren't getting.

         21              And then at the same time you think

         22    about what are the steps if I go ahead on

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           314

          1    this trajectory ask yourself well, is there a

          2    parallel path that gets me to the same goal

          3    and can I do let's lay up two or three

          4    options and then say which look like the most

          5    efficient given that we had now this

          6    experience that we had.  So those are some

          7    thoughts that I had just listening to the

          8    presentation this afternoon.

          9              DR. HENGARTNER:  Johnny?

         10              MR. BLAIR:  Before I make my

         11    comment I want to underline Jae's point about

         12    asking follow-up questions of the European

         13    agencies.  I think keeping those contexts

         14    open and following up with them is something

         15    that will be very very valuable.

         16              A couple of things I wanted to

         17    mention, I think he mentioned that sometimes

         18    you're going in and you know a fair amount

         19    about the survey and sometimes you don't have

         20    very much knowledge about it.  That's an

         21    important thing to go back to and address,

         22    how much you know about the survey before you

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           315

          1    go into it, I think, both in terms of the

          2    reception of staff to you.  You're asking

          3    questions and particularly when you have to

          4    ask follow-up questions or probe or whatever

          5    it's going to be much better.

          6              I also think that having a better

          7    understanding would also affect the reporting

          8    process in writing up the results afterwards

          9    and also even before writing up the results

         10    structuring what the report is going to be

         11    like.  So I think the whole process is

         12    affected by how much knowledge you bring to

         13    it about the particular survey.

         14              The second thing I wanted to go

         15    back to, this issue of how you ask the

         16    questions has a number of effects but I just

         17    want to touch on two of them.  One is that a

         18    respondent in addition to being cooperative

         19    and so forth and wanting to be forthcoming

         20    has to understand what you're looking for,

         21    has to understand what level of detail that

         22    do these folks want in my response.  A more

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           316

          1    specific question conveys that better than a

          2    less specific question.

          3              So if you're asking about training

          4    and just saying do you have a training

          5    program as opposed to do you have written

          6    procedures for training will help them

          7    respond more specifically to that question

          8    and hopefully give you more uniform responses

          9    across your survey but will also convey to

         10    the respondent what you're looking for and

         11    what level of detail you would like for them

         12    to respond at.  So respondent attitude is key

         13    and is the fundamental thing that they want

         14    to be cooperative and forthcoming but I think

         15    that the question, and this is why you want

         16    it to be specific rather than "tough," that

         17    how you ask the question can affect how they

         18    respond not just to that question but to the

         19    later questions in the survey.

         20              DR. HENGARTNER:  Thank you.  Randy?

         21              DR. SITTER:  I'm done.

         22              DR. HENGARTNER:  I'd actually like

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           317

          1    to ask one of those tough questions.  It's

          2    all good to do those self-assessments and I

          3    find it's wonderful we're doing that.  But

          4    then in the back of my mind it's, like, what

          5    are actionable results.  What are we going to

          6    do with it?  And I think one way to motivate

          7    people to participate in that is if indeed

          8    you can benefit them doing something in the

          9    future.  So are there any results you can

         10    already say yes, we've identified this and,

         11    lo and behold, we've changed that and look at

         12    that, our response rate is better?  Do you

         13    have any success story so far?

         14              MR. BROENE:  The only thing that's

         15    close is something that was already underway

         16    and that was training on how to deal with

         17    respondents.  We have instituted some

         18    training.  That's all I know of it.  I

         19    believe it's a one-day course of how to deal

         20    with respondents and that was something that

         21    some people have already had training in

         22    that.  Others don't really train the people

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           318

          1    that deal with our respondents.  A lot of

          2    them are contractors, some of them are very

          3    knowledgeable, some of them are not

          4    knowledgeable.  And so that was one of the

          5    things we have found, that there are big

          6    differences among our surveys and the amount

          7    of training for either federal staff or

          8    contractors.  But then that training

          9    procedure was already underway.

         10              DR. HENGARTNER:  Thank you.  I also

         11    have something I'd like to throw out for the

         12    committee.  I know that Johnny is not going

         13    to be here tomorrow so that's why I'm

         14    bringing it up.  I was wondering if there

         15    would be a sense of actually having a

         16    USA-Euro Cup on the data quality essentially

         17    at the next JSM or maybe JSM 2007, i.e.,

         18    invite some of our colleagues from Sweden,

         19    Canada and have a discussion at JSM on this,

         20    if that would be fruitful for the EIA we can

         21    sponsor a session and if you think that the

         22    general statistical community might be

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           319

          1    interested in something like that?

          2              Yes, Johnny?

          3              MR. BLAIR:  To respond to that, my

          4    answer is yes.  I think that would a good

          5    idea and I think that's something that people

          6    would definitely be interested in.  And I

          7    also just want to make another comment to

          8    follow up on your previous point and that is

          9    when you do this self-assessment or whatever

         10    assessment that you do it's important to keep

         11    in mind that it sets up expectations of the

         12    people that are working on these surveys and

         13    that it's important that they see something

         14    come out of this on the other end

         15    particularly for the next round.  When you

         16    come around the second time to do this if

         17    they haven't seen some results come out the

         18    first time you'll notice it in the response

         19    you get.

         20              DR. HENGARTNER:  We don't need to

         21    make a decision but I'd like to throw it out

         22    in the air and depending on the response we

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           320

          1    get I'll try to see if it's worthwhile doing

          2    it.

          3              MR. SINGPURWALLA:  What do you

          4    mean, World Cup?

          5              MS. FORSYTH:  Competition.

          6              DR. HENGARTNER:  A competition.

          7    What I'm thinking is essentially the

          8    Europeans have one way of doing their quality

          9    control, we seem to have a different take

         10    although we're somewhat inspired, and what we

         11    want to know is what works and what doesn't.

         12              MR. SINGPURWALLA:  Well, what works

         13    better.

         14              DR. HENGARTNER:  Well, another way

         15    of putting it.  Good.  At this stage I'd like

         16    to invite the members of the general public

         17    if anyone feels compelled to go to the

         18    microphone and wants to raise questions.

         19              If not I'd like to adjourn this

         20    meeting until tomorrow.  I remind you dinner

         21    is at 6:00 o'clock.  Those who are going back

         22    to the hotel, depart at 5:30 from the lobby.

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382


                                                           321

          1    That should give us enough time.  Or you can

          2    fend for yourself.

          3                   (Whereupon, at 4:24 p.m., the

          4                   PROCEEDINGS were continued.)

          5                       *  *  *  *  *

          6

          7

          8

          9

         10

         11

         12

         13

         14

         15

         16

         17

         18

         19

         20

         21

         22

                              BETA COURT REPORTING
                              www.betareporting.com
                  202-464-2400                     800-522-2382