AMERICAN STATISTICAL ASSOCIATION





                  COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY

                            INFORMATION ADMINISTRATION







                                     DAY 2 OF 2



















                                  WASHINGTON, D.C.

                               FRIDAY, APRIL 20, 2007





                                                              2


           1    PARTICIPANTS:

           2       NAGARAJ K. NEERCHAL, Ph.D., Chair

           3       DEREK BINGHAM, Simon Frasier University

           4       EDWARD A. BLAIR, University of Houston

           5       JAE EDMONDS, Ph.D., Pacific Northwest Laboratory

           6       MOSHE FEDER, D.Sc., Research Triangle Institute

           7       BARBARA FORSYTH, Westat

           8       WALTER W. HILL, St. Mary's College of Maryland

           9       NEHA KHANNA, Binghamton University

          10       EDWARD KOKKELENBERG, Ithaca, New York

          11

          12

          13                       *  *  *  *  *

          14

          15

          16

          17

          18

          19

          20

          21

          22




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                              3


           1                   P R O C E E D I N G S

           2                                             (8:40 a.m.)

           3              DR. NEERCHAL:  What I'd like to do

           4    -- I think you start with -- I will ask any

           5    Committee member who is not here and -- but

           6    here today to go ahead and introduce

           7    themselves and also the audience members,

           8    anybody in the audience who is -- who was not

           9    here, but who is here today to go to the mike

          10    and introduce themselves.

          11              And also I want to remind you that

          12    please register and put your e-mail down when

          13    you're coming in because of -- on the

          14    registration table there.

          15              Any new persons here?  I do.

          16              MS. CARLSON:  Mary Carlson.

          17              DR. NEERCHAL:  You have to come to

          18    the mike.

          19              MS. CARLSON:  All right.

          20              SPEAKER:  You have like those

          21    foreign national fingerprint IDs to get into

          22    the building.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                              4


           1              MR. BURNS:  Hello.  I'm Eugene

           2    Burns.  I'm with the Office of Energy Markets

           3    and End Use in the Energy Consumption

           4    Division.

           5              MR. CONRAD:  I'm Lenny Conrad with

           6    SMG.

           7              MS. BATTLES:  I'm Stephanie

           8    Battles.  I'm Director of the Energy

           9    Consumption Division.

          10              MS. CARLSON:  Mary Carlson, Office

          11    of Oil and Gas, Natural Gas Division.

          12              MS. MICHAELS:  Joelle Michaels,

          13    Energy Consumption Division.  I'm the survey

          14    manager for the Commercial Buildings Energy

          15    Consumption Survey.

          16              MS. O'BRIEN:  Eileen O'Brien,

          17    Survey Manager of the Residential Energy

          18    Survey in the Energy Consumption Division,

          19    EMEU.

          20              MR. SWENSON:  Alan Swenson, Energy

          21    Consumption Division.

          22              DR. NEERCHAL:  Before we start off




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                              5


           1    with (off mike) today, I think we should take

           2    a moment and remember the people who are

           3    affected by the Virginia Tech tragedy.  Today

           4    is a national day of mourning, and, you know,

           5    we all feel for them and that our colleagues,

           6    and, you know, fellow citizens of Virginia

           7    there.  So maybe we should observe a moment

           8    of silence and then.

           9              (Moment of silence.)

          10              DR. NEERCHAL:  Thank you.  Today's

          11    first -- we have two plenary sessions today,

          12    and the first plenary session speaker is

          13    Jacob Bournazian?

          14              DR. BOURNAZIAN:  Yes.

          15              DR. NEERCHAL:  And he's going to

          16    talk about the microdata.

          17              DR. BOURNAZIAN:  Yes.

          18              DR. NEERCHAL:  Okay.

          19              DR. BOURNAZIAN:  Well, Members of

          20    the Committee on Energy Statistics, Fellow

          21    EIA staff and guests, I'm happy to be here

          22    this morning because this topic is an area




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                              6


           1    that's developing and it's on a fast pace

           2    with the federal statistical agencies.

           3              It's also an area that's been

           4    growing with the federal statistical

           5    agencies, and it's a long time since we've

           6    been able to talk about this here at EIA, and

           7    so my presentation today really comes at a

           8    timely moment.  We're talking about expanding

           9    research or access to microdata or EIA's

          10    microdata.

          11              All right, Larry.  EIA is

          12    considering three options for expanding

          13    research or access.  There's a program that

          14    Ruey-Pyng spoke about yesterday afternoon

          15    with the National Institute on Statistical

          16    Sciences.  How many Committee members heard

          17    Ruey-Pyng's presentation?

          18              One, two, three, four.  Okay.  A

          19    majority of you.  All right.  So I'll be

          20    going each one of these options in detail,

          21    but the NIST-EIA student research program, a

          22    nice program on site here at the Forrestal




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                              7


           1    Building.  I'll be going over that again.

           2              A second option, though, is to

           3    place our residential energy consumption

           4    survey data and our commercial building

           5    energy consumption survey at the Census

           6    Bureau's RDC facilities.

           7              There's nine facilities and the

           8    Census Bureau conducts the Manufacturing

           9    Energy Consumption Survey, MECS, which is

          10    already housed at the Census Bureau's RDC.

          11    So by placing these two surveys with the

          12    Census Bureau, you'd have the whole sweep of

          13    consumption surveys in one RDC facilities set

          14    up.

          15              There's a third option, and that's

          16    to place the same two surveys at the National

          17    Opinion Research Council's data enclave

          18    that's going live May 1st, in a few weeks.

          19              The big feature of NORC's data

          20    enclave program is remote access, where the

          21    researcher stays right there in their office,

          22    right from their PC, dials in and can do the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                              8


           1    research.

           2              It's a private VPN network.  No

           3    cutting and pasting allowed, no downloading,

           4    no printing.  You just -- you can do your

           5    research and connect to those company-level

           6    data files.

           7              So we'll be talking about these

           8    three options and at the end of my

           9    presentation, I'm going to be asking you for

          10    some input to help us evaluate which is most

          11    appropriate and be effective for EIA.

          12              Now, EIA has never engaged in this

          13    activity to any significant extent as far as

          14    allowing researcher access to the

          15    company-level files.  But there are benefits

          16    and many federal, if not all of the federal

          17    statistical agencies have moved in this

          18    direction for this very reason.

          19              There are substantial benefits by

          20    allowing researcher access.  First, you're

          21    fostering an open research environment.

          22    Basically, this benefit here is replication




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                              9


           1    of your results, the transparency of the

           2    data, and of the information you're putting

           3    out.

           4              However, in every single agency's

           5    case we've talked with that has allowed

           6    researcher access, they found another

           7    by-product of benefit here:  It improves data

           8    quality.  Well, if you heard Ruey-Pyng's

           9    presentation, he mentioned a few selected

          10    topics dealing with statistical methodology.

          11              Agencies learn about their data by

          12    allowing researcher access.  They evaluate

          13    their statistical methods, but they also

          14    learn a great deal about their data.  In

          15    fact, talking with RDC Administrator, a lot

          16    of outliers, misreporting of problems with

          17    the data get identified.  Not that you can do

          18    anything about it five years after the fact,

          19    but you learn about your data and it has --

          20    and it impacts you on your analysis.

          21              Now, in the old days at EIA, we

          22    used this, okay?  We went down through




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             10


           1    output, company by company, and we analyzed

           2    what those companies reported.

           3              We don't do that anymore.  Resource

           4    constraints.  We have automated edits.  We're

           5    going to sort the data, look for companies

           6    are sufficient distance away from the crowd.

           7    We're going to look at what companies make an

           8    impact on a cell.  We don't have time to call

           9    three -- make 300 phone calls to validate our

          10    data, and the automated edits work well.

          11              But what does a researcher do?

          12    Researcher wanders right into the forest,

          13    looks at every tree, every branch, leaf on

          14    the, you know, branch.  Right, he's going to

          15    look at a small company's reporting history

          16    regardless if it falls in the geographic area

          17    they're interested in.

          18              So if every other agency has

          19    realized these benefits, no doubt EIA will

          20    benefit by the same way.

          21              Strongest reason down here at the

          22    bottom:  Microdata they permit in-depth




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             11


           1    analysis for many policy-relevant issues that

           2    the agency just is not able to analyze.  We

           3    don't have the resources to do that.  As

           4    Ruey-Pyng pointed out on his research

           5    projects, there's a lot of methodology

           6    issues.  We just don't have the time.

           7    They're wonderful research projects, but

           8    staffing and time is limited.

           9              However, if you heard Guy Caruso's

          10    opening remarks yesterday, he talked about

          11    service reports.  Well, we get a lot of

          12    questions, besides questions from Congress.

          13    We get questions on whether electric power

          14    producers from Montana are leveraging

          15    residential electricity consumers in Idaho.

          16    You got questions why gasoline prices in

          17    Georgia are more expensive than Florida when

          18    Georgia's tax is only seven cents and

          19    Florida's is 20.

          20              Okay.  We don't have time to get

          21    into that real local state, not local, but

          22    state analysis.  In fact, what Guy said was




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             12


           1    he has to prioritize.  He can't do every

           2    single service request.  And what is he

           3    doing?

           4              He's regionalizing, nationalizing

           5    it, and if there are other requests that

           6    match up, because we're going to focus on the

           7    national supply picture or regional.

           8              Who is going to be able to analyze

           9    the difference between Arkansas and

          10    Tennessee?  Maybe a professor from the

          11    University of Tennessee would be interested

          12    in that, but certainly EIA doesn't have the

          13    analysis resources to go after every single

          14    analysis question.  So we would benefit a

          15    great deal by using our data in this fashion.

          16              Okay.  Now I've made this point in

          17    my paper, but I have to admit I probably

          18    didn't do it effectively because my

          19    discussant approached me at lunch time

          20    yesterday and he said, you know, what do you

          21    mean by this statement expanding researcher

          22    access is going to increase EIA's investment




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             13


           1    in the data collection.  In fact, I used Ray

           2    to return to my paper.  All right?

           3              Well, I promised that.  I said, I'm

           4    going to make it up to the whole Committee

           5    this morning with this one slide.  And I may

           6    have failed in my paper; we'll come back with

           7    a picture.

           8              Well, we looked at where EIA spends

           9    its resources.  Now, Guy Caruso mentioned

          10    yesterday that it was a strong wind blowing

          11    through Congress yesterday or this year and

          12    we got $90 million in the budget.

          13              We didn't always have that strong

          14    wind.  Okay.  Budgetary financing was more of

          15    the '80s.  Let's round down to $80 million.

          16    And what's two- thirds of $80 million?  I

          17    think it's a little over $50 million, but I'm

          18    going to round down to $50 million.  And Guy

          19    also mentioned that we've been around for

          20    what?  Thirty years?  What's 30 times $50

          21    million?  $1.5 billion, to be exact.  Where

          22    are our resources and money going?  It's




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             14


           1    going to collect data.  We processed the data

           2    and now we have a bunch of master files with

           3    company-level data on there sitting on the

           4    shelves.  Nobody is using it.  We've got the

           5    aggregate tabular data that's been released

           6    and we're real good about that.  We've got

           7    online query systems that get you that

           8    aggregate tabular data right from the web.

           9    But nothing, nothing is going on with our

          10    company-level data.  It sits there, year

          11    after year, decade after decade.  I don't

          12    even know if we could link up with our

          13    company-level files from the '80s to even do,

          14    you know, some long-term analysis with the

          15    files in the '90s or even in this decade.

          16              But we just keep on going out,

          17    collecting, compiling, and stacking on

          18    shelves, and it sits there.  It sits there,

          19    and so I'm going to point this out, because

          20    I'm going to be talking about some options

          21    that are cost money, tens of thousands of

          22    dollars.  I'm going about spending tens of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             15


           1    thousands of dollars, talking about

           2    increasing our data utility.

           3              And when you compare the tens of

           4    thousands of dollars I'm going to ask this

           5    agency to spend, compared to the opportunity

           6    cost of having over a billion dollars of

           7    resources sit there, the benefits should be

           8    there and the money as well worth spending.

           9              Now, I hoped that addressed on what

          10    I mean by rate of return, because Ed's a New

          11    Yorker.  Right?  So what does he say in the

          12    lunch line?  Are you looking for seven

          13    percent, eight percent -- the typical New

          14    York question.  And I thought about it on my

          15    second pass through the linguini line is that

          16    there's plenty of metrics.  We could look at

          17    users' accessions to the files.  We could

          18    look at research proposals.  We could look at

          19    products produced.  You could even look at

          20    citations on the research.  There's many

          21    metrics we could develop to measure the

          22    usefulness and utility of this activity.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             16


           1              Okay.  Well, then why haven't we

           2    just run out there and done this?  I mean if

           3    this is such a great activity and every other

           4    federal agency is doing it, why aren't we out

           5    there with the pack?

           6              Well, you got some risks.  And so

           7    let's talk about those risks 'cause you want

           8    to -- that's one reason, main reasons, why we

           9    have not engaged in expanding researcher

          10    activity.

          11              Well, this is a general policy that

          12    there's concerns over confidentiality.

          13    Across federal statistical agencies,

          14    confidentiality impacts response rates and

          15    refusal rates, well documented in the

          16    research.  One thing that's unique about EIA,

          17    we've got mandatory data collection

          18    authority.  You have very high response

          19    rates, and so even though there can be

          20    confidentiality problems, our response rates

          21    would never be sucking wind as bad as what we

          22    see in like Census or BLS that are down to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             17


           1    the 40 percent range on voluntary surveys.

           2    Mandatory surveys in general always have a

           3    higher rate.

           4              So do we win automatically?  Can I

           5    dismiss this?  No, not really?  Cause lack of

           6    trust negatively affects data quality.

           7    Actually, lack of trust negatively affects

           8    communication and even in a personal level;

           9    right?

          10              When trust is high, easy to

          11    communicate with one another.  When trust is

          12    low, it's very difficult.  The same in a

          13    business setting.  Okay.  So what happens to

          14    EIA if we cause a breach in confidentiality

          15    or engage in a program that diminishes and

          16    denigrates our pledge?

          17              Well, I'll tell you one thing:

          18    Volume data is going to get rounded down and

          19    cost data is going to get rounded up; right?

          20              That's what we get.  EIA is going

          21    to force me to report, you'll get your file

          22    and report on time.  Is it going to be the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             18


           1    right data?  Maybe not.  Is that going to

           2    impact us?  Yeah.  That's real important.  We

           3    have a good reputation for accuracy.

           4              There's another common risk.  The

           5    perceived risk of a breach can be just as

           6    damaging as an actual breach.  Again, it's

           7    dynamics of human relationships; the same in

           8    a business setting.

           9              However, there's one caveat on the

          10    perceived risk.  It's how you handle it.

          11    It's how you handle the perceived risk versus

          12    an actual breach.  An actual breach you're

          13    stuck.  You're burned on that one.  But on a

          14    perceived -- how do you respond could

          15    boomerang and be a huge deposit with your

          16    trust and goodwill with your respondents.

          17    Getting cast in a false light and then

          18    correcting it with the truth always comes

          19    back for strengthening that bond.

          20              However, it's very important to

          21    handle on your -- we have certain rules now

          22    and notification of breaches and then




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             19


           1    policies and procedures.

           2              Here's a huge, another big risk:

           3    EIA may get blamed for bad research.  Now you

           4    researchers are a suspect group.  Why?

           5    Because you're intermittently playing around

           6    with the data.  You come in and you want to

           7    do energy studies.  Then you want to do a

           8    labor study.  And then you come back and do

           9    another energy study.  Whereas hard-core

          10    staffers, consultants, well, they're always

          11    in the energy industry; right?

          12              Are you guys going to get it right

          13    this time?  Wow, what if you get it wrong and

          14    EIA gets blamed 'cause, after all, you used

          15    EIA data.  Mm.

          16              Well, federal agencies handle it

          17    two ways:  Disclaimers and or extensive

          18    disclosure -- product review.  And you can

          19    pick any flavor or a combination of the two.

          20              A very interesting observation.

          21    Actually this one here has led some program

          22    officers to altogether pull out and say we're




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             20


           1    not going to let any researchers touch the

           2    data because bad research -- the ironic

           3    thing, though, is on the aggregate data, and

           4    I've talked about three core cases in the

           5    Midwest where professors were called in as

           6    expert witnesses, testified on jobbers

           7    getting squeezed by refiners in the upstream

           8    market; used EIA data, petroleum data and

           9    won.  The jobbers won.

          10              The researchers did a great job, a

          11    professional job of explaining that to the

          12    courts and they used it correctly.

          13              So somehow, maybe it's

          14    intrinsically that they're able to be bright

          15    and clever and on point using aggregate data,

          16    but when you come to microdata, you guys just

          17    mess it up every time; right?

          18              You know, just like you can handle

          19    magnitude, but not frequency.  But it's

          20    there.

          21              The cost of providing access --

          22    normally that should not be a risk, except




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             21


           1    for when you work at EIA.  All right.  This

           2    is not always going to be a strong wind

           3    blowing through Congress.  All right?

           4              And if you're going to commit the

           5    agency to an annual expenditure of fifty

           6    grand, ten grand, a hundred thousand dollars,

           7    okay, that is a constraint.

           8              And so it's a risk -- are you going

           9    to be able to continue with these programs?

          10    We have to look at cost.  In fact, cost is

          11    going to be the number one bullet item when

          12    we go through these three options.

          13              But I do want to put in context,

          14    'cause I been talking about what other

          15    federal agencies have been doing, and it's

          16    pretty exciting.  Remember when I opened, I

          17    said this is a fast-developing area.

          18              Well, if I gave this speech five

          19    years ago, I'd have four agencies up here on

          20    the top. But it keeps on growing.  Now, it's

          21    not just that little light bulbs are going on

          22    and more and more agencies are getting into




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             22


           1    this game and engaging in this activity.  You

           2    see -- all right. It's a weak laser pointer,

           3    so good thing I had Chinese food last night.

           4    Chopsticks seems to be in the morning.

           5              Agency for Health Research and

           6    Quality, they have a nice researcher access

           7    program and what do they do?  They host their

           8    data at the National Center for Health

           9    Statistics.  They're -- they have an RDC in

          10    Hyattsville.  They pay an annual fee.  Sort

          11    of similar to the option I'm going to be

          12    explaining for what we're considering with

          13    the Census Bureau.  The Bureau of Economic

          14    Analysis, the Bureau of Labor Statistics

          15    on-site researcher programs.

          16              The Census Bureau, (off mike) they

          17    have the nine research data center

          18    facilities.  National Agriculture Statistics

          19    Service.  They have a data lab right down the

          20    street, and also 40 regional offices that are

          21    connected on a secure LAN for researchers to

          22    come in and access microdata.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             23


           1              The National Center for Education

           2    Statistics.  Well, one of the leaders in it.

           3    It started in the late '80s, mainly using

           4    licensing, but now moving on.

           5              The National Center for Health

           6    Statistics and National Science Foundation.

           7              Now, even the agencies that have

           8    been in this activity for a while are going

           9    through their second and third generation of

          10    evolving; okay?  It's not like they went and

          11    got engaged in it; they're staying with it.

          12    They're moving well ahead forward.

          13              For example, NASS has a data lab,

          14    but now NASS has also signed a contract with

          15    NORC to place their data in their data

          16    enclave.  Why?  They're looking for that

          17    remote access feature.  Okay.

          18              Also, NCHS, they've got a research

          19    data center.  They signed a contract with

          20    Census Bureau and placed their data at the

          21    RDCs in Census.  What are they doing?  Just

          22    increased their RDCs by nine- fold.  Okay?




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             24


           1              Now, besides all this, too many CDs

           2    are probably ending up in researchers'

           3    attics, but I don't know why, but NCES, the

           4    Census Bureau, NASS, NCHS moving to online

           5    query systems.  You got CDC Wonder, in which

           6    you can query a database and get, you know, a

           7    company, household respondent-level records.

           8              The Census Bureau American Fact

           9    Finder query level you can do data restricted

          10    queries, but you can go and access -- do your

          11    research on company-level data.  NASS has the

          12    Agricultural Research Methods Survey.  NCES

          13    just started this in the last year.  Besides

          14    licensing, now they are also putting up, and

          15    they have on their Web site their

          16    company-level files.  Again, buffered access

          17    for approved researchers.

          18              So you can see these agencies are

          19    moving forward in developing multiple modes

          20    for researcher access, not just one.

          21              Okay.  So who's been left behind?

          22    The Bureau of Transportation Statistics, a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             25


           1    very small agency about one-third the size of

           2    EIA.  It's been around for about 10 years.

           3              Internal Revenue Service and that's

           4    administrative data.  I don't think it's a

           5    fair comparison, except they have one unit,

           6    Statistics of Income.

           7              The Social Security Administration,

           8    once again administrative data.  Now look at

           9    this little agency right here.  I'm not

          10    saying we should do this because everybody

          11    else is doing it.  I'm saying everybody else

          12    is doing it because there's real benefits and

          13    they're growing in the area and if EIA is not

          14    going to engage in it, EIA just gets left

          15    behind.  And that's been the story of the

          16    last 10 years.

          17              Okay.  Can we do this?  Maybe

          18    that's why we've never done it.  Well, yes,

          19    we can.  We do have the legal authority to

          20    engage in it, because if we didn't, end of

          21    the presentation.

          22              This is what we're relying on in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             26


           1    our negotiation with the Census Bureau.  They

           2    require legal authority.

           3              All right.  Also, expanding

           4    researcher access promotes EIA's mission.

           5    Now EIA's mission statement is kind of

           6    boring, but if you step back and you look at

           7    it as a trilogy within a trilogy, it's a

           8    really a great mission statement, because why

           9    are we here as an agency?  Well, to provide

          10    the data, the forecasting work, and analysis.

          11              But so why do we provide that?

          12    Well, we want to promote sound policy making,

          13    operating efficient markets, and a public

          14    understanding.  Public understanding of what?

          15    Well, first of all energy.  We want to

          16    promote an understanding of energy --

          17    energy's interaction with the economy;

          18    energy's interaction with the environment.

          19              Now, when I see words like

          20    interaction with the economy, interaction

          21    with the environment, I'm thinking data

          22    linkage.  Right?  That's kind of the analysis




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             27


           1    that provides for that kind of work.  You

           2    want to link the files and really get some

           3    powerful research done.

           4              Now how's that going to benefit

           5    this program, researcher access?  Right here.

           6    Promote sound policy making and public

           7    understanding, two big areas we can get a

           8    boost from.

           9              Got some outside support for this

          10    activity, too.  The External Study Team

          11    issued the report last year, 2006, and

          12    laudable goal.  They recommended that EIA

          13    increase its level of interaction with the

          14    research community to a level that is on par

          15    with EIA's level of interaction with the

          16    energy industry.  Good luck.  Right.  Right.

          17    I agree, because it's wonderful we have these

          18    energy conferences.  It's the only time

          19    really researchers come here on site and we

          20    have this kind of candid interaction.

          21              The only other interaction you guys

          22    doing is presentations at JSM in August, a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             28


           1    minor showing at American Public data users

           2    conference, but this really is a core

           3    activity for us.  Compare it to EIA's

           4    interaction at energy conferences.  Okay.  So

           5    it's going to be a long time, and I don't

           6    think researcher access is going to balance

           7    the equation, but it's a step in the right

           8    direction.

           9              But this quote I felt was really

          10    useful.  Later on in the report, they write,

          11    ideas to promote this goal, right up above,

          12    include providing a program of visiting

          13    scholars to bring academic researchers to

          14    EIA.

          15              Well, that's what Ruey-Pyng was

          16    talking about yesterday.  And expanding

          17    researcher access to microdata made under

          18    appropriate safeguards to protect the

          19    confidentiality of the reporting entities.

          20    No one's looking to undermine our data

          21    protection pledges.  No one's looking to

          22    throw that away, and no one's looking to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             29


           1    undermine the goodwill and trust we have

           2    built up over the last 30 years.

           3              We have the resources to maintain

           4    those data safeguards and yet permit this

           5    activity, and they were highlighting it.

           6    Yes, we recognize the importance of

           7    preserving that trust and goodwill with our

           8    respondent community.

           9              Okay.  So let' go to the three

          10    options and wrap up.

          11              The first option that research data

          12    center at Census Bureau.  A hundred thousand

          13    dollars to start up; $75,000 annually.  Well,

          14    I just want to brag, because a (inaudible).

          15    Well, we started off the negotiations with

          16    $250,000 up front and a $125,000 a year, so

          17    we did the best we could, but that's where

          18    the Census Bureau has drawn the line in the

          19    sand for offering their services.

          20              It's off site.  It's a full service

          21    operation here.  They do everything from, you

          22    know, receive the proposals, monitor the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             30


           1    employee when they're working there, do the

           2    disclosure review, do the product review.

           3    Okay.

           4              As I mentioned, we already have the

           5    Manufacturing Energy Consumption Survey data

           6    there at the same location.  Now, I was

           7    talking with Dan Weinberg (ph.), who is the

           8    Director of the Center for Economic Studies,

           9    and he is in charge of the RDC program at

          10    Census Bureau.  And I said, you know, can't

          11    you come down below a hundred grand because

          12    quite frankly you got competition.  You got

          13    competition, buddy.  You could lose our

          14    business if you can't come down below a

          15    hundred.

          16              And so Dan reminds me, he says,

          17    Jake, I serve on the Board of Directors for

          18    NORC, and I'm helping them build their data

          19    enclave.  We do not view this competition at

          20    all.  In fact, I want to help my colleagues

          21    develop their program.  Okay.

          22              You know why?  Right here.  Right




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             31


           1    here.  He said, Jake, Census Bureau is still

           2    and will always probably have the largest

           3    register of business data available.

           4              So if you're a researcher and

           5    you've got some interesting topic, most

           6    likely this is the place that you're going to

           7    need to go.  So Dan's not worried about

           8    competition.

           9              Now we got some downsides.  When he

          10    was (off mike) working at RDC, you got to

          11    submit a proposal to the Census Bureau.  It

          12    goes to their Disclosure Review Board; a copy

          13    would come over to us.  You got about a month

          14    or two, probably two months time lag there.

          15    Once you get the proposal approved, then you

          16    get -- go in to getting special sworn status.

          17    They're not going to waste time doing a

          18    background check if your proposal is not

          19    accepted.  Right?  This takes six months, the

          20    FBI background check.

          21              What are we talking about?  It

          22    could take (off mike) away, just like eight




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             32


           1    months, eight months right down there, and

           2    it's not for free.  Okay.  We're paying a

           3    hundred grand.  Researchers got pay $3,750 a

           4    month.  That's about the average cost of a

           5    researcher.  On top of it is $18,500.  The

           6    National Center for Health Statistics has a

           7    similar fee, about $18,000 per researcher to

           8    use that facility.

           9              So -- it's plus or minus in there.

          10    Okay.  Ruey-Pyng talked about this from the

          11    National Institute of Statistical Sciences --

          12    $10,000.  Great program.  Nice deep

          13    affiliations than this has with its other

          14    universities.  They've got -- it would be

          15    done here on site, so EIA needs to provide a

          16    work area, dedicated PC, some staff to (off

          17    mike).  Selected research topics, so it may

          18    involve other survey data besides RECS and

          19    CBECS data.  That's pretty appealing.  Also

          20    you got a list of selected research topics

          21    that, you know, we can channel out for (off

          22    mike).  But it goes to a selected group of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             33


           1    researchers, a selected group of researchers.

           2              How many people in this room got

           3    their Ph.D. after the year 2000?  That's what

           4    I'm talking about.  See.  That's right.  See,

           5    you know, if you woke up this morning, and

           6    said, you know, I'm getting kind of old to do

           7    a lot of things in life.  When you came here

           8    to this meeting, I bet you never thought

           9    you're too obsolete to do any research on

          10    company-level data at EIA, but, in fact,

          11    you're all discriminated against.  You're

          12    out.  You're axed.  You're off the list.

          13              All right.  This goes to grad

          14    students; right?  That's why we have selected

          15    topics, because that's what grad students are

          16    for.  They, you know, they like to do what

          17    they're told.  Give them a topic, and it may

          18    work.  But it's a great program, and it's a

          19    nice little module that adds on.

          20              Okay.  How do we control that?  The

          21    researcher is an agent of EIA.  It's

          22    important.  I'll be pointing this out in each




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             34


           1    one of my options, because you have to look

           2    at what control mechanisms are in place to

           3    make sure there is no diminishment of the

           4    confidentiality pledge.  What safeguards are

           5    in place?  What rules, remedies, you know,

           6    regs do you have?

           7              DR. NEERCHAL:  Jacob, just a

           8    clarification.

           9              DR. BOURNAZIAN:  Mm hmm.

          10              DR. NEERCHAL:  The (off mike) what

          11    this option means or include what you're

          12    going to (off mike), at that time (off mike)

          13    just a list.

          14              DR. KIRKENDALL:  (off mike).

          15              DR. BOURNAZIAN:  Right.  How much

          16    do grad students make; right?  I'm just

          17    joking.  No.

          18              DR. NEERCHAL:  (off mike)

          19              DR. BOURNAZIAN:  No, that works

          20    through NISS.  Right.  There is a payment,

          21    right, that goes with that.

          22              DR. NEERCHAL:  (off mike)




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             35


           1              DR. BOURNAZIAN:  Right.

           2              DR. NEERCHAL:  (off mike)

           3              DR. BOURNAZIAN:  Good point,

           4    Nagaraj.  In fact, you might as well say $10K

           5    plus, $10K plus.

           6              Last option is from the National

           7    Opinion Research Council.  Annual fees 50,000

           8    plus.  Why?  Because they're offering a

           9    portfolio approach.  They're saying, okay, we

          10    got this remote access feature in which

          11    researchers can use it, but what other

          12    services do you need as an agency?  You need

          13    disclosure protection training.  You need

          14    file work -- coordinating with the

          15    researcher, working with him on that.  We can

          16    give them training.  How about archival

          17    services?  You got files from 10 years ago

          18    and your documentation for that and link them

          19    up.  All right.  Is there any archival work

          20    that needs to be done?

          21              Typical private sector consultant.

          22    They do it cost plus; right?  But it allows




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             36


           1    you to choose your boutique selection of

           2    services you want.  But the baseline $50,000.

           3              The researcher is also going to get

           4    charged.  I think it was about, (off mike),

           5    but it was somewhere around $100 a week.

           6    They were going to do it in weekly units of

           7    measurement, being that when you offer it for

           8    free to researcher, it's not valued as much

           9    if they maybe have to pay a de minimis fee.

          10    So they wanted something applied (off mike)

          11    to them.

          12              Oh, they do have two off-site

          13    facilities -- Chicago and D.C.

          14              Now, we've got to apply those data

          15    safeguards, right, because we don't want to

          16    change the playing field.  We keep the level

          17    of risk the same.

          18              And so you have monitored -- (off

          19    mike) argues buffered remote access.

          20              We're going to apply some filters

          21    to the queries and have some queries

          22    restricted.  Everything else at the default




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             37


           1    is let the query go to the data.

           2              Okay.  Now, a lot of authentic --

           3    you have two-way authentication for the IP

           4    addresses.  Not only is the researcher

           5    approved by NORC to log into the system

           6    through that VPN network, there's an

           7    authentication data safeguard for the IP

           8    address.

           9              And like I said, they have a

          10    features (off mike), you can't download (off

          11    mike) through the system.  It's like your own

          12    private VPN network (off mike) research.

          13              The researcher now becomes a

          14    subcontractor or agent.  (off mike) the

          15    contractor.  Now, oh, also NASS, ERS has

          16    signed up and they're putting their data

          17    there.  They're using a -- working as a

          18    subcontractor, but EIA can go either way --

          19    subcontractor or agent.

          20              Okay.  I showed you some cost data.

          21    That was the first bullet on every slide.  I

          22    just want to summarize that.  NISS is coming




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             38


           1    in at $10,000; NORC $10,000 plus.

           2    Everybody's plus.  NORC $50,000 plus except

           3    the Census Bureau.  I have it at $75,000, but

           4    I didn't tell you it's a $100,000 in the

           5    first year.  They had some start-up costs

           6    getting a -- purchase additional equipment

           7    and service to host our data, but you got to

           8    say if we engage in this activity, it's not

           9    for a year or two.  It's a commitment to this

          10    activity.  And so this really would be the

          11    recurring annual cost to the agency who did

          12    that.

          13              You got -- (off mike) do some work.

          14    It's not something we can just jump into

          15    outside of the NISS program, which, like I

          16    said, a nice contained module (off mike)

          17    organization.

          18              We got to document those files.  A

          19    lot of our file are (off mike) on shelves

          20    (off mike) the documentation is pretty

          21    sparse.  If you want a researcher to work

          22    with your files, they got to have the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             39


           1    documentation to tell them what the file

           2    structure is.

           3              Okay.  Researcher training has to

           4    go along with this or else the research is

           5    going to get bogged down real quickly.

           6              Disclosure protection training can

           7    be modules, a training module, but that has

           8    to be addressed.  You got to let the

           9    researcher know how you treat confidentiality

          10    at your agency, how it applies to you.

          11              Cyberinfrastructure work.  I've

          12    told you how NORC is establishing their data

          13    enclave, but, you know, every time you think

          14    you got something set up and your IT people

          15    have a solution for you, there's some hacker

          16    out there waiting to unlock the key.  It just

          17    happens.  It's the nature of this business

          18    and in cybersecurity.  EIA has received -- it

          19    receives a lot of cyberattacks on a daily

          20    basis (off mike) 300 in the first three

          21    months of this year.  But we're also getting

          22    what they call this root matrices where the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             40


           1    hacker comes in and places a false IP address

           2    on an employee's PC.  They don't know it, and

           3    then can either steal data or try to corrupt

           4    the database and through that the employee's

           5    PC.  We've had about 10 attacks on that.

           6              So maybe that dual IP

           7    authentication that NORC is proposing may not

           8    work.  What do they have?

           9              They have solutions to that, too,

          10    but once again more cyberinfrastructure would

          11    probably be needed to put in place the remote

          12    access researcher program.

          13              Resource constraints.  You got to

          14    be looking at money and where the funding is

          15    going to come from to do that.

          16              Okay.  So let's summarize and turn

          17    it over to you Committee members (off mike)

          18    want to get some input.  We put on some

          19    questions.

          20              On these three options that I've

          21    just been talking about this morning, which

          22    modes of researcher access should EIA provide




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             41


           1    and be providing or getting into developing?

           2    Which mode of access works best with the

           3    research community?  (off mike) not only the

           4    ability to do the research, but the type of

           5    research.  You remember that trilogy within

           6    our mission statement regarding energy,

           7    energy's interaction with the economy,

           8    energy's interaction with the environment.

           9    What kind of research and not only the

          10    substantive areas, but also statistical

          11    methodology.

          12              What kind of research would people

          13    be interested in, and if they had access to

          14    company- level files here at EIA.

          15              And then what I'd like what I'd

          16    like to hear from -- if anyone is engaged in

          17    accessing microdata from federal statistical

          18    agency, a nice war story, your anecdote that

          19    sets, laid out what you found useful or

          20    beneficial or what you found as an obstacle

          21    or a difficulty so that maybe EIA wouldn't

          22    have to repeat that same mistake and then




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             42


           1    develop once again an effective researcher

           2    access program.

           3              So in summarizing, I turn it back

           4    to you Nagaraj, our Chair, for comments.

           5              DR. NEERCHAL:  Thank you.  Thank

           6    you, Jacob.

           7              I will go to Moshe for the first

           8    (off mike) discussion.

           9              MR. FEDER:  First of all, I wanted

          10    to say great presentation.  In fact, we have

          11    two great ideas the last two days.  Ruey-Pyng

          12    spoke about the N-I-S-S plan, and Jake spoke

          13    about -- and I think both are great ideas and

          14    I'm looking forward to hearing more about

          15    (off mike).

          16              I just had one question before you

          17    went to summary.  Since NISS and the option

          18    is relatively very cheap, why not do that and

          19    consider yet another option?  Is there

          20    anything in the contract that would be signed

          21    that would preclude the other from being

          22    done?




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             43


           1              DR. BOURNAZIAN:  No.  No, in fact,

           2    that is one of the options.  We can do

           3    multiple modes.

           4              MR. FEDER:  So even if I let's say

           5    go with the data enclave, why not do that at

           6    the NISS as well?  I mean I don't -- I don't.

           7              DR. BOURNAZIAN:  They're not

           8    mutually exclusive.

           9              MR. FEDER:  (off mike) Why is it so

          10    cheap and --

          11              DR. KIRKENDALL:  It's not -- no

          12    matter -- it's actually not a competition

          13    with NISS.  We are going to pursue that one.

          14              MR. FEDER:  Yeah.

          15              DR. KIRKENDALL:  Because it's no

          16    particular additional cost.

          17              MR. FEDER:  No, no, no.  I meant

          18    the NISS data arrangement.

          19              DR. KIRKENDALL:  That's part of the

          20    same thing.

          21              MR. FEDER:  A part of it.  Okay.

          22    Great.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             44


           1              DR. KIRKENDALL:  So it's a -- we

           2    are doing that and I think on that one, this

           3    came from a little fellowship program we had

           4    through ASA, and we actually did have a few

           5    researchers to come through that program that

           6    worked here at EIA on site.  They

           7    particularly accessed the Electric Power

           8    data, and they thought that it worked really

           9    well for them.  And, you know, agreeable to

          10    do some work.

          11              Electric power thought it was good,

          12    too.

          13              MR. FEDER:  I remember seeing Joe

          14    Guspid (ph.) here.  Was he working (off

          15    mike)?

          16              DR. KIRKENDALL:  No, Joe wasn't.

          17    But anyhow, that's the kind of program we

          18    were thinking about where the program

          19    officers review the proposals and (off mike)

          20    to, and because they're the ones who have to

          21    provide the access and make sure the data are

          22    protected according to their specifications.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             45


           1              MR. FEDER:  Now, Jake, you

           2    mentioned this is a developing area.  Indeed,

           3    it is, and about a week and a half ago, I saw

           4    an announcement for a new journal on data

           5    confidentiality.  I know one of the editors

           6    is Christina, and I can send you -- I think I

           7    saw Paula here.  She's also an expert in this

           8    area, so I'll send you guys information on

           9    it.  In fact, I would consider publishing a

          10    paper in that journal when you make your

          11    choice of the considerations that might be

          12    useful, either that or (off mike), but

          13    because it's a very good piece of information

          14    for other agencies and government agencies

          15    across the world to know what you're doing,

          16    because you were very right about all the

          17    benefits to EIA and to the research community

          18    of giving people access to the data that

          19    cannot be used otherwise because of

          20    constraints, and the constraints are not

          21    necessarily because your budget is not (off

          22    mike) enough.  It's because there's so much




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             46


           1    data.  Even if you had billions of dollars,

           2    you would still be able to get a lot of

           3    things out of the data.

           4              So I think it's worthwhile to

           5    consider publishing once you've (off mike) on

           6    this -- on that.

           7              Let's see.  Now, my experience has

           8    been mostly with data collected from human

           9    subjects, which has particular constraints.

          10    We have to get every piece of data collection

          11    approved by the IRB.  That's the

          12    Institutional Review Board.  And everything

          13    that's collected has to be -- any use of that

          14    data has to be approved.  So if we cannot

          15    even get someone to be our agent, so to

          16    speak, and use the data, if that has not been

          17    approved, and usually it has to be done prior

          18    to collection.

          19              Now, in your case, it's mostly

          20    business data, but I do believe you have some

          21    household-level data, so you have to check on

          22    the legalities and if you can iron this out,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             47


           1    again that would be worth publishing so

           2    people will know what is -- what can and

           3    cannot be done, I think that's one thing to

           4    consider.

           5              I really liked your comments about

           6    the perceived risk of breaching of the

           7    confidentiality pledge.  It's something

           8    that's been a lot on my mind because I think

           9    all it takes is one case in the Washington

          10    Post, let's say about an agency that released

          11    information that was collected under pledge

          12    of confidentiality with response rates,

          13    accuracy of reporting and so on would be in

          14    jeopardy.

          15              So it's a good point, and I think

          16    has to be kept in line.

          17              So in summary, I liked -- Ed will

          18    probably talk more about more substantive

          19    aspects of use of the data, but I liked the

          20    idea of leveraging researches, you know,

          21    resource (off mike) students, whatever they

          22    have to make more of the data.  I think the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             48


           1    first NISS option should be done regardless

           2    of which other options you chose, because

           3    it's cheap and probably very effective, and,

           4    again, I think when you develop your approach

           5    and make your own choices, I would encourage

           6    you to consider publishing it in the

           7    disclosure literature.

           8              Yeah, that's about everything I

           9    wanted to say.

          10              DR. NEERCHAL:  Thank you, Moshe.

          11              DR. KOKKELENBERG:  You know, a lot

          12    of different observations, but let me give

          13    you as background.  I was one of the first

          14    people over the falls with the longitudinal

          15    establishment of data files at Census, and

          16    (off mike) how many years ago that was.

          17    Somebody asked me how working on that file

          18    was, and after a year of working on it, I

          19    said did you ever read Thomas Mann's Der

          20    Zauberberg, the Magic Mountain, where time

          21    all of a sudden goes extremely slowly and

          22    nothing happens.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             49


           1              Well, that was the Bureau of the

           2    Census.  I did want to underscore one of the

           3    things that Moshe said right at the end about

           4    the Washington Post, not that I have it in

           5    for the Washington Post, but last fall, if

           6    you will recall, there were some misplaced

           7    laptops in various agencies, and there was a

           8    great, great media uprising on that, but not

           9    quite as bad as some we've seen elsewhere and

          10    other times, but a lost laptop in an agency

          11    that has eve 50 or 60 computers is not an

          12    uncommon thing; right?

          13              And so if that is going to be the

          14    kind of reaction that happens, there is a

          15    catastrophic risk if somebody does somehow

          16    breach the confidentiality rules.  So that

          17    really has to be a concern.

          18              Since I did this work, there was a

          19    lot -- there's been a lot of developments and

          20    basically because of these off-site centers

          21    like the Census off-site centers, the

          22    researcher needs to come to these problems




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             50


           1    with an awful lot of background.  I didn't

           2    have it, and I learned that I had to have it,

           3    and I got some of it.

           4              First of all, the researcher really

           5    needs to know what the underlying --

           6    understand underlying data generating process

           7    is and what the questions were that were

           8    asked to get this data.

           9              Without that, you just got numbers.

          10    And there's an awful lot of economists who

          11    will take any dataset and run it through

          12    their algorithm and go and say, see, I showed

          13    the efficiency of or the cost function is or

          14    the utility function is, and most of it is

          15    not read very much deservedly.

          16              This agency is really good on

          17    statistical reliability.  It's probably one

          18    of the leaders in the area, so I think that

          19    when you hand out data for whatever purpose,

          20    emphasizing your understanding of the

          21    reliability of the data is useful.

          22              One of the big things that datasets




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             51


           1    have in them at the micro level are an awful

           2    lot of imputations, and they generally are

           3    flagged if the agency is doing the work

           4    properly.  And the researcher has to

           5    understand that particular aspect of things.

           6              Now, the agency unfortunately is

           7    going to have to have virtually a consultancy

           8    to be able to help the researcher use the

           9    data.

          10              The researchers come in and they

          11    don't know what the data -- I mean they have

          12    an idea, and they spend a lot of time working

          13    on a research plan.  They submit it.  It gets

          14    approved.  It has a basic understanding of

          15    the problem, but it really doesn't understand

          16    what data is there and how it will be used

          17    and how to use it.

          18              And so there's a lot of backing and

          19    forthing.  You can't say submit your request

          20    for the variables you want or the 15 firms

          21    you want for the time period that you want,

          22    and we'll give you the data, and then expect




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             52


           1    the person to go away fully satisfied.

           2    They're going to be back saying don't you

           3    have something and (off mike) so yes, and

           4    they don't know what the -- the rules of

           5    engagement that you lay out are going to have

           6    -- be able to have them put a request in

           7    advance knowing that the data exists and that

           8    means they have to have access to the data

           9    before they ask the question.

          10              So one thing that has been proposed

          11    is synthetic data sets.  Okay.  Are you

          12    familiar with those?

          13              DR. BOURNAZIAN:  Yes.

          14              DR. KOKKELENBERG:  Okay.  That

          15    would be the way to go I think.  If you gave

          16    a synthetic dataset, which is all hokey

          17    numbers, and the person -- but it's complete.

          18    It shows all the variables that you really do

          19    have and in the loving detail you have, and

          20    then the researcher takes the synthetic

          21    dataset and works on the problem, understands

          22    what variables they need, what the strengths




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             53


           1    and weaknesses of the models they have are,

           2    but then gets access to the real live data.

           3    And it's a one- shot or maybe two-shot access

           4    to the data.  But they're not --

           5              DR. BOURNAZIAN:  So they first do

           6    research on the synthetic data?

           7              DR. KOKKELENBERG:  On the synthetic

           8    data to try to understand what the

           9    limitations of the data are and what are the

          10    variables that are available and their

          11    strengths and weaknesses of each variable and

          12    the time pass that they can look at.

          13              So I understand some of the

          14    research data center stuff is going to be

          15    that way out at Census.

          16              The other thing you're going to

          17    need is you're going to need to be able merge

          18    this data with other datasets, and that means

          19    that this cut and paste prohibition for --

          20    might be legitimate.  You've got to work your

          21    way around that.  But you want to match it up

          22    with Census data or Ag data that's available,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             54


           1    maybe also microdata, but without being able

           2    to match it up, you're going to have problems

           3    with that, or limitations.

           4              Now, I would agree with Moshe when

           5    he said that you should try multiple methods

           6    of researcher access.  The best way to

           7    control confidentiality is only the access

           8    through the agencies with computers that are

           9    not connectable to the Internet and don't

          10    have USB ports on them, you know, that sort

          11    of thing.

          12              DR. BOURNAZIAN:  Mm hmm.

          13              DR. KOKKELENBERG:  And can't burn a

          14    disk.  The next best step is probably these

          15    RDCs that the Census Bureau has, because

          16    they've had a lot of experience with this

          17    problem.

          18              I don't know anything about the

          19    National Opinion --

          20              DR. BOURNAZIAN:  Research Council?

          21              DR. KOKKELENBERG:  Yeah, that --

          22    what their centers are going to be like so I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             55


           1    couldn't say anything about them.

           2              DR. KIRKENDALL:  That's relatively

           3    new.

           4              DR. KOKKELENBERG:  Right.  But even

           5    the Research Data Centers take years to set

           6    up.  I know Cornell has been trying to get a

           7    Research Data Center, and the best they could

           8    do was get one I think in Buffalo and in one

           9    in New York City, where they have access.  In

          10    fact, they teach a whole course on how to use

          11    these Research Data Centers in the graduate

          12    level.  It's called Information 747, and John

          13    Abowd teaches it, and I think he's a person

          14    you should really strongly talk to, because

          15    he's had a lot of experience working with the

          16    data centers.  He was Arnold Zelmer's

          17    student.

          18              DR. BOURNAZIAN:  I know John.

          19              DR. KOKKELENBERG:  You do know him?

          20              DR. BOURNAZIAN:  Yep.

          21              DR. KOKKELENBERG:  Okay.  He's an

          22    expert in the -- he knows what




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             56


           1    confidentiality laws apply and so forth, and

           2    so he's the kind of guy you can (off mike) to

           3    bring into have as a discussant when you get

           4    to the point where you really want to make

           5    some decisions on these (off mike), and get

           6    some better advice.

           7              Let's see.  I think that's about

           8    (off mike).  Well, one other point:  The

           9    Census have one man, Jim Monahan was his name

          10    who knew the dataset that they were making

          11    available.  He had spent years working it, so

          12    it -- this takes a heavy- duty investment of

          13    resources on the part of the agency to have

          14    somebody who can shepherd the research or

          15    into accessing the data and what's out there

          16    and what isn't out there.

          17              DR. BOURNAZIAN:  Right.

          18              DR. KOKKELENBERG:  That's all.

          19              DR. NEERCHAL:  Neha?

          20              DR. KHANNA:  I have a question, one

          21    question and another comment.  My first

          22    question is before the options that you




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             57


           1    considered the three different options, does

           2    it -- for any of those options does it matter

           3    whether the researcher is a foreign national

           4    and located in a U.S. university?

           5              DR. BOURNAZIAN:  Yes.  If they're a

           6    foreign national, they're going to have a

           7    very difficult time coming on site here, and

           8    then if it's Census Bureau RDC that FBI

           9    background check could take a year.

          10              DR. KHANNA:  That's what I thought,

          11    because that's a big disadvantage for an

          12    onsite kind of arrangement because you're

          13    losing more than half your research community

          14    right there.

          15              DR. KOKKELENBERG:  But then you'll

          16    lose it even for off-site --

          17              DR. BOURNAZIAN:  Yeah.

          18              DR. KOKKELENBERG:  -- if they have

          19    to go through this background check.

          20              DR. KHANNA:  Yes, but if you're

          21    going -- the background check has to be done

          22    anyway, whether you're coming on-site or




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             58


           1    off-site, but on-site is -- and it's (off

           2    mike) today, because I'm a foreign national

           3    and they didn't like where I worked before.

           4    I don't know.  And I just -- it makes it more

           5    difficult to me whether the load I think some

           6    good fraction (off mike) because a foreign

           7    national grad students from Georgia (off

           8    mike) foreign national.  That's just

           9    something I wanted to put on the table

          10    because not everyone in this room has to deal

          11    with it as much as maybe I have to.

          12              Another question that I -- another

          13    question or comment that I had was I think

          14    Ed's point about being able to match the data

          15    with other publicly available data is really

          16    important.  An incubator by itself may be

          17    nice, but it may not be very useful to do the

          18    research project.

          19              And there are two ways of doing

          20    that.  I mean if you went through let's say

          21    the off-site VPN access route, because of the

          22    fact that NORC might have let's say other




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             59


           1    Census data also available that matching

           2    issue may not actually be as great; whereas,

           3    if you have it on-site and come and work on

           4    our computer her, then I kind (off mike) the

           5    Census data, and I have to --

           6              DR. BOURNAZIAN:  Right.

           7              DR. KHANNA:  And I have to be

           8    physically --

           9              DR. KIRKENDALL:  I'm not sure

          10    you'll be able to access Census data from

          11    NORC either.

          12              DR. KHANNA:  No?

          13              DR. FORSYTH:  But you can from

          14    Census.

          15              DR. KIRKENDALL:  But you certainly

          16    could at Census.

          17              DR. KHANNA:  No, but if I have an

          18    off- site, if I have the VPN, let's -- if I

          19    have, you know, approval and I can use the

          20    VPN from my computer at home or in my office,

          21    and through the (off mike) can I access both

          22    the Census and the EIA data?




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             60


           1              DR. BOURNAZIAN:  No.  Right now,

           2    NASS and ERS are putting their agriculture

           3    data with NORC in a -- but, no, there's --

           4    they don't have the depth of the business

           5    registries at the Census RDC has.

           6              DR. KHANNA:  But then another

           7    question is:  If a researcher, let's say, has

           8    some publicly available Census data, let's

           9    say at the Census track level or block group

          10    level, something that you can just go to the

          11    Census Web site and download, would it be

          12    possible for the researcher to bring those

          13    data to the NORC Web site?

          14              DR. BOURNAZIAN:  Yes.  NORC

          15    provides those portfolio services to a

          16    researcher can bring in exogenous or

          17    administrative data.

          18              DR. KHANNA:  Okay.

          19              DR. BOURNAZIAN:  And match to

          20    whatever they're accessing.

          21              DR. KHANNA:  Okay.  So that --

          22              DR. BOURNAZIAN:  Enhance it.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             61


           1              DR. KHANNA:  Enhance it a little

           2    bit.  But okay.

           3              DR. NEERCHAL:  Moshe.

           4              DR. FEDER:  I wanted to comment on

           5    the synthetic data.  It's an appealing idea,

           6    but it has a number of limitations.  Number

           7    one is the cost.  To do it, you really need

           8    to spend way more than a hundred thousand

           9    dollars to develop it, but the main problem

          10    is how do you do that?  You hypothesize a

          11    certain model.  You feed the model parameters

          12    from your data and then you generate data

          13    from that model.

          14              And the problem is if the

          15    researchers were interested in validating

          16    that same very model that was used to

          17    generate the data, they cannot, because the

          18    data were using that model.  And that has

          19    been raised many times.

          20              So synthetic data has this inherent

          21    limitation that cannot be overcome, well,

          22    because maybe use some semi-parametric then




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             62


           1    approach, then you would really develop some

           2    methods that haven't been created yet.

           3              I don't think we have been working

           4    on it, but it really -- real data is real are

           5    -- sorry -- real data and simulated data

           6    costly and not as good.  So that's why I

           7    still like the idea of direct access to the

           8    data, plus what Neha was talking about, being

           9    able to link it with Census data, which you

          10    would not be able to do with synthetic data,

          11    because you need some --

          12              DR. KOKKELENBERG:  The Census data

          13    is only used for you to develop your research

          14    proposal pertinent to the points of where you

          15    know what the real data is, but that's

          16    available and then use it in that sense, not

          17    to fit any models necessarily but to try to

          18    understand data availability.

          19              DR. FEDER:  I would use Census data

          20    to get company variables that are not

          21    available in the EIA data.

          22              DR. KOKKELENBERG:  Oh, yeah, well,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             63


           1    that's a good point.

           2              DR. FEDER:  Yeah.

           3              DR. KOKKELENBERG:  You know one way

           4    to that I would have thought of doing this is

           5    let's say you have the electricity generation

           6    by generator.  So if you take this vector of

           7    just add to it or multiply it or take it to a

           8    power of a random number, and there's your

           9    synthetic data.

          10              DR. NEERCHAL:  I think that one

          11    quick point on that one I think knowing the

          12    data structure, even if I don't know what the

          13    numbers are in the beginning really help a

          14    researcher focus their data request a lot.

          15              DR. BOURNAZIAN:  Absolutely.

          16              DR. NEERCHAL:  I think in terms of

          17    that you can just basically, you're going to

          18    give me a blank matrix with all the data --

          19    you know, the variables in there and what the

          20    components are and so on.  That will be

          21    extremely useful.

          22              DR. FEDER:  Well, you could make




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             64


           1    the code book available.

           2              DR. NEERCHAL:  No, no.  On the

           3    other hand, I think Moshe is absolutely

           4    right.  I think really doing -- putting

           5    synthetic data out there, which is kind of,

           6    you know, unbeatable is a very difficult

           7    task.  I think that that's a -- but I do want

           8    to make one comment, you know.

           9              From the presentations, the way I

          10    understood is that, you know, you have two

          11    important objectives.  One is to address all

          12    these questions, many of these questions that

          13    come in and you don't have time to answer it.

          14    You just say, you know, that information is

          15    over there; you know, help yourself.  That

          16    kind of, you know, you don't need to spend

          17    too much time on this.

          18              The other objective is to promote

          19    research on your data.  I think the -- maybe

          20    the first one should be the, you know, the

          21    first objective.  I think that maybe you want

          22    to kind of prioritize it and say how can we




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             65


           1    really, like, cut down some of this work that

           2    we need to do periodically and not -- don't

           3    really enjoy doing it.  At the same time, we

           4    don't want to say no.  I think this is the

           5    negotiation that Guy was talking about

           6    yesterday.

           7              I think, you know, come with

           8    something that really helps you to say oh, it

           9    will be very hard, we have been thinking

          10    about what you would ask and we put this

          11    information out there.  Help yourself.  So,

          12    but, you know, that's about it.

          13              On the other hand, promoting

          14    researchers (off mike) certainly it's not

          15    possible for us to sit down here and think

          16    about all the things a researcher might ever

          17    ask for.  No matter what you put out there,

          18    somebody is going to come and say, I need

          19    more.  Okay.  I think that that's something

          20    -- that is why what promoting research, you

          21    really want to go the "proposal" route.  If

          22    someone wants data, let them write down what




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             66


           1    they want.  You do whatever you can to the

           2    best of your available resources.

           3              You may want to kind of bifurcate

           4    your approach like that.

           5              DR. KIRKENDALL:  Actually, all of

           6    these proposals involve us responding to a

           7    proposal for research by a researcher.  A

           8    person would have to send in a proposal

           9    either to NISS or to NORC or to the Census

          10    Bureau and that would proposal would have to

          11    be reviewed any number of different places

          12    and accepted in all of those places before

          13    they would actually get the access.

          14              So everybody would have to look at

          15    it and make the sure the research was okay,

          16    according to the plan and then the people who

          17    owned the data would have to be sure it was a

          18    good idea, too.

          19              DR. NEERCHAL:  And I think we need

          20    to -- we may have to (off mike) better.

          21    Barbara, who invested (off mike)?

          22              DR. BINGHAM:  No, go ahead, because




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             67


           1    I think she wanted to --

           2              DR. FORSYTH:  It was, but I think

           3    it might be less important (off mike) what

           4    you just said, Nancy.

           5              What I was going to suggest is, you

           6    know, the synthetic data and giving --

           7    helping people to understand the structure of

           8    the data.  I liked your idea of a course for

           9    users, teaching (off mike) of the structure

          10    of the microdata, and you have a ready

          11    outlet.  You know, JPSM, for example, could

          12    help you put together like a short course, a

          13    two-day course, and possibly in the context

          14    of that course, depending on how you

          15    structure it, maybe give people some kind of

          16    access to some kind of subset or limited set;

          17    at least give them some experience,

          18    opportunities to experience the structure of

          19    the microdata.  So that's what I was going to

          20    suggest.

          21              DR. KIRKENDALL:  Actually, both of

          22    these surveys already have public use




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             68


           1    datasets, so we have -- it's just that

           2    there's been some top coating and maybe not

           3    all the variables are there --

           4              DR. FORSYTH:  Right.  Right.

           5              DR. KIRKENDALL:  But other methods

           6    to (off mike) of it.  That's at least a way

           7    you could get started understanding the data.

           8              DR. FORSYTH:  Right.  The

           9    microdata, though, would have a somewhat

          10    different structure, because it would be more

          11    of it; right?

          12              DR. KIRKENDALL:  Yeah.  It would --

          13              SPEAKER:  The actual --

          14              DR. FORSYTH:  Right.  And the

          15    matching across datasets.

          16              DR. BINGHAM:  So I'm curious (off

          17    mike) change the subject.  I'm curious about

          18    what the -- I'm curious what the -- how easy

          19    is a breach of confidentiality insofar as

          20    when you're looking -- so what is

          21    confidentiality in this case?  I mean that's

          22    the first question and does that mean I'm




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             69


           1    able to identify a respondent's exact

           2    information and lots of detailed information

           3    about them and is that bad?

           4              DR. KIRKENDALL:  Well, we tell them

           5    we're not going to give that data out, and

           6    that is the kind of identification we want to

           7    protect against, and we really don't really

           8    want somebody to say that they've identified

           9    it, even if they're wrong.

          10              DR. BINGHAM:  Right.  (Laughter.)

          11              DR. BINGHAM:  Okay.  So --

          12              DR. KIRKENDALL:  And that's a

          13    little harder to protect against.

          14              DR. BINGHAM:  So I know that you --

          15    the temptation is always there to do that;

          16    right?  If you're working with that level of

          17    data.  Okay.  So this happens -- so there's a

          18    lot of work that goes on with confidentiality

          19    in surveys, for instance, where

          20    health-related things and they collect things

          21    all the way down to the county level and

          22    suppose you're looking at HIV counts and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             70


           1    there are two people who live in the county;

           2    right?  And so the other person knows that

           3    they don't have HIV, so they know who does;

           4    right?

           5              DR. KIRKENDALL:  Mm hmm.

           6              DR. BINGHAM:  And so there are lots

           7    of efforts to either aggregate to another

           8    level or randomly switch around counties and

           9    things like that and my question is any of

          10    this taking place at -- you know, on this

          11    microdata when you're going to send it out,

          12    you know, because you know whether it's a

          13    graduate student or it's another company,

          14    just a guy in a company who says, you know

          15    what, I know enough about this.  I should be

          16    able to identify these individual records

          17    here, and is this fiddling?  Is it pos --

          18    what's being done?

          19              DR. BOURNAZIAN:  Well, the --

          20    there's a lot in that area.  The pledge is

          21    that you're not -- there's not going to be

          22    unauthorized disclosure of reported value of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             71


           1    a respondent by EIA.

           2              That's why in a lot of the slides

           3    you're saying that this researcher is going

           4    to be an agent or a subcontractor.  We all

           5    look at company-level data.  In fact, 70

           6    percent of Americans can be identified just

           7    by zip code and birth date.  So you've got

           8    these survey records that have a lot of

           9    direct identifiers, even though you can strip

          10    off the name and company ID.

          11              The researcher, he's interested in

          12    some kind of statistical analysis, so he

          13    might be in a small geographic area.  He

          14    knows this is a certain type of company.  He

          15    might not know the name, and then, yes, you

          16    know, if he's well, you know, involved in

          17    this research, he knows this market and he

          18    knows the players in the market, and so he

          19    knows really who he's looking at when he's

          20    looking at the record.

          21              The point is when he finishes his

          22    research and he writes up his paper, is he




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             72


           1    going to be talking about, you know, X, Y, Z

           2    power generator, you know, or is he going to

           3    be talking about this, the power generating

           4    industry in this area.

           5              So it still has to be statistical

           6    analysis.  That's what Nancy was saying.

           7    Those proposals get reviewed quite

           8    rigorously.  What's the purpose for allowing

           9    access?  What do they want to study?

          10              And if it has a nice statistical

          11    purpose, yes, it can go forward.  Researcher

          12    is going to have access to it, but he can't

          13    take it out.  He can't take it home.  And he

          14    can't be publishing, you know, a

          15    company-level responses in his paper.

          16              But the access is there.

          17              DR. KIRKENDALL:  And he's signed an

          18    agreement that he won't reveal the data.  So

          19    there has to be an agreement with the

          20    researcher that they, you know, I'm sure --

          21    even though they may have looked at something

          22    and they have figured out that it's company




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             73


           1    X, Y, Z, he can't -- he's not supposed to

           2    make use of that information.

           3              DR. BOURNAZIAN:  Right.

           4              DR. KIRKENDALL:  And he's not

           5    supposed to reveal it, and, in fact, if it's

           6    CIPSEA data, then he's subject to big

           7    penalties if he does.

           8              DR. BINGHAM:  Okay.  So I guess the

           9    only thing -- so in that case, so, like I

          10    think I said, I suspect with somebody

          11    fiddling will figure something out.

          12              You can also guess that if

          13    somebody's doing a detailed enough micro

          14    micro analysis on something via, I guess

          15    somebody has to vet this in order to make

          16    sure that something inadvertently doesn't get

          17    published.

          18              You know, I don't know to what

          19    level people are using the information, what,

          20    you know, like how fine down something that's

          21    going to appear in a publication, how far

          22    down somebody will have dug, and, if they




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             74


           1    will have -- I guess it's the only thing that

           2    I would -- the only question I have is okay,

           3    can something -- okay, can confidentiality be

           4    inadvertently breached, and) I guess it seems

           5    like from the spirit of what you're saying,

           6    the answer is no.  But --

           7              DR. KIRKENDALL:  We hope the answer

           8    is no, and there are safeguards.  You know,

           9    the papers are reviewed -- they'd be reviewed

          10    either by the Census Bureau Research Data

          11    Center or-and or by the NORC folks.  It would

          12    also be reviewed by EIA people, and so you

          13    hope that you haven't allowed something that

          14    breaches confidentiality to get out.

          15              And that's what the level of review

          16    is about.

          17              DR. BOURNAZIAN:  And this has been

          18    going on for many years.

          19              DR. KIRKENDALL:  But that doesn't

          20    guarantee either, of course.

          21              DR. BOURNAZIAN:  But it's been

          22    going on for almost 20 years, and we have not




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             75


           1    found any confidentiality breaches from

           2    researcher access among the federal

           3    statistical agencies.  There's been breaches,

           4    but it hasn't come from this activity where a

           5    researcher has gone in and done something

           6    intentional or even foolish or deliberate.

           7              They're usually -- well, they're

           8    all dedicated in good faith and in those

           9    reviews is your final check.  And so you got

          10    to say, look at the track record.  Zero

          11    breaches across eight, nine agencies engaging

          12    in this activity over the last 20 years.

          13              DR. KIRKENDALL:  Well, it's bound

          14    to happen like tomorrow.

          15              DR. BOURNAZIAN:  Right.  You can

          16    never say it will never happen.  Right.

          17    Well, we're building up to it.

          18              DR. NEERCHAL:  Ed, you had your --

          19              DR. KOKKELENBERG:  Well, much of

          20    what I was going to respond to, Derek brought

          21    up and it was responded to by Nancy and Jake.

          22    One thing I did want to add, though, is that




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             76


           1    you do know, as a researcher, you know that

           2    company's name?

           3              DR. KIRKENDALL:  Mm hmm.

           4              DR. KOKKELENBERG:  And you -- I

           5    don't care whether you strip the name out or

           6    not, but in the Census data, it was not -- at

           7    the establishment level, I knew that company,

           8    and I knew its quarterly labor bills and its

           9    production, every -- you know, the details.

          10              So that's available to me.  All

          11    these other safeguards are there, plus that

          12    there are a lot of penalties under the laws

          13    that enable the Census, which, you know,

          14    could put you in prison if you really wanted

          15    to go out and breach the confidentiality

          16    arrangement.

          17              And the final review of your paper

          18    is before -- even with Moshe raised the issue

          19    about confidential data in human-or for

          20    individuals.

          21              I know I've been working with

          22    student- level records, and I get that paper




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             77


           1    vetted by the FERPA enforcer, I guess I even

           2    called her, at SUNY Binghamton before I can

           3    let the paper go out the door or what have

           4    you.

           5              So it's the same with Census.  But

           6    you really want to go look over those papers

           7    very carefully to make sure that I wasn't

           8    revealing anything to anybody.  That's all.

           9              You just have to stand on the, you

          10    know, trust the ethical responsibilities of

          11    the individual researcher that requires the

          12    -- perhaps this long vetting of the

          13    researcher to do that; that the FBI does for

          14    me to commit to do it.

          15              DR. NEERCHAL:  Thank you.  Thank

          16    you and the presenters, discussants and (off

          17    mike).  I think we break and get back here

          18    around 10 maybe?  We're a little bit behind,

          19    but that should be okay.  We need a little

          20    break, I think.

          21              DR. KIRKENDALL:  Okay.  Yeah.

          22    (Recess.)




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             78


           1              DR. NEERCHAL:  So we are onto our

           2    last plenary session.  Can I just go and ask

           3    any audience member who was not here before

           4    to go to the mike and mention your name and

           5    affiliation, please.

           6              SPEAKER:  You may even need to turn

           7    your switch on.

           8              MR. SIMBALSKI:  John Simbalski.  I

           9    work for EIA, and I'm the NEMS residential

          10    (off mike), and I'm here to listen to how

          11    George is going to pick apart our work.

          12              (Laughter.)

          13              MS. BOEDECKER:  Erin Boedecker.  I

          14    work on the commercial module of the NEMS

          15    system.

          16              MR. McDOWNEY:  Preston McDowney.

          17    DIA, Systems and Methods Group.

          18              MR. RADICH:  Tony Radich.  I work

          19    on the Petroleum Market Model in NEMS.

          20              MR. SKELLY:  Dan Skelly, EIA.

          21              DR. NEERCHAL:  Thank you.  So the

          22    last session speaker is George Lady.  It's on




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             79


           1    Forecast Evaluation Methodology.

           2              DR. LADY:  Thank you very much.  I

           3    very much appreciate a chance to talk about

           4    my project and John is wrong.  In fact,

           5    without his help, a good deal of what I'm

           6    going to report on today couldn't have been

           7    done.  So.

           8              In fact, I should say in advance,

           9    just to be clear, OIF has more and more taken

          10    an interest and supported what I am doing and

          11    by the time we're done today, you'll see my

          12    point of view in terms of what I recommend is

          13    going to depend very much on that.

          14              The project that I'm reporting on

          15    is at this point taking thought on what would

          16    be a good way to evaluate the accuracy of

          17    NEMS forecasts.

          18              The truth is that EIA has published

          19    for some years comparisons of the NEMS

          20    projections with the eventual values of the

          21    energy statistics, and I think they reported

          22    on something like 17 or 18 of the major




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             80


           1    series that they're forecasting.

           2              There's a literature on this, and

           3    part of the literature found that, although

           4    the forecasts would often look quite

           5    accurate, if you disaggregated them, for

           6    example, if you took the consumption forecast

           7    and re-expressed it in terms of residential,

           8    commercial, transportation, and industrial

           9    sector consumption that the errors would be

          10    somewhat larger and then offsetting when

          11    added together in the total.

          12              My idea was to pursue sort of that

          13    point of view and to look at the forecasts

          14    and try to understand what were the major

          15    influences that would lead to the differences

          16    between what was forecast and what eventually

          17    happened.

          18              And here on the slide is the list I

          19    came up with of the sorts of things that

          20    would explain why, in the end, the projection

          21    would not be accurate.

          22              From the standpoint of what I can




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             81


           1    do and have been trying to develop a point of

           2    view about, I can analyze the weather.  I'm

           3    trying to develop a method where I can

           4    organize major explanatory variables for

           5    supply and demand, like price and gross

           6    domestic product.  And I think in the end, I

           7    can evaluate structural change by comparing

           8    different versions of NEMS and can detect

           9    whether or not there has been a change in the

          10    model logic.

          11              What has been pointed to me is that

          12    there are a number of other things that can

          13    make a big difference, too, perhaps the most

          14    important one is the fact that the data

          15    definitions can change over time, and this

          16    can make the forecasts appear inaccurate in

          17    terms of historical values, but really the

          18    world is changing underneath of the model.

          19              And I think that's very important,

          20    and I'm hoping really that this project

          21    starts an enterprise of worrying about all of

          22    the possible sources of forecast error, but




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             82


           1    some of them, such as changes in data, I

           2    can't speak to.  I hope that a tradition

           3    starts where that particular issue is taken

           4    care of.

           5              As far as what I'm doing, the

           6    methods at stake really are three:

           7              The point is to understand at the

           8    level of explanation why the supply and

           9    demand forecasts in NEMS are not accurate,

          10    and the idea is to take the errors that are

          11    found and to try to partition them across

          12    such as price forecasts which are inaccurate,

          13    driver forecasts, like GDP, that are

          14    inaccurate, innocent things like the weather,

          15    which can be different, in fact, than was

          16    assumed for NEMS; and try to parse out all of

          17    these influences.

          18              The ideal -- and I received some

          19    very nice written comments from the Committee

          20    earlier in the week -- the ideal would be to

          21    save each version of NEMS, to archive it, and

          22    when the time comes to rerun the model using




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             83


           1    actual values for assumed values and maybe

           2    even reconfiguring the model logic to account

           3    for changes in laws or other structural

           4    changes in the energy system.

           5              In fact, just to show you how old I

           6    am, back in 1980, I actually, when I worked

           7    here, initiated a program to archive the

           8    versions of all of the models that were used.

           9    At that time, one of the more, one of the

          10    things that was in mind was that when a

          11    forecast was published, the archive model

          12    could be given to a third party, who could

          13    then inspect what was the basis for the

          14    agency's projections.

          15              But strictly speaking, it could be

          16    used in this project.  The problem is that

          17    it's very hard to do, and it's been well over

          18    a decade I think since there has been any

          19    routine archive program.  It's very expensive

          20    and as far as I know, there hasn't been any

          21    initiative to use archives versions of the

          22    model to speak to the kinds of matters that I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             84


           1    think I would like to organize.

           2              But it obviously would be the best

           3    thing, and that's what the reviewers say, and

           4    I agree with that.  But I fear that it is too

           5    hard to do.

           6              What I will propose today is that

           7    there is a good second best that we can do;

           8    that I think is a lot easier as far as

           9    forecast evaluation compared to archiving the

          10    model, and subject to the errors of linear

          11    approximation will do probably as nearly a

          12    good job.

          13              And in particular it's my item two

          14    here.  I'm advocating now that each major

          15    version of NEMS, the AEO versions would be

          16    the first to come to mind, have a structure

          17    of runs made which are designed to pick out

          18    major influences.

          19              OIF has already done this several

          20    times with respect to prices, and John's

          21    group, very nicely, gave me over 70 runs that

          22    they did a little over a year ago where they




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             85


           1    changed each fuel price one by one and just

           2    in terms of the demand models were able

           3    through the solution to see what difference

           4    the price changes made.

           5              Well, I believe this can be done in

           6    general, and that what I would advocate in

           7    the end is that there be some routine

           8    protocol developed, some experimental design

           9    agreed to, where some number of solutions are

          10    created, designed in a comparative static

          11    sense to pick out influences and rather than

          12    archive the models, we'll archive the

          13    solutions.  Much, much simpler.  And then a

          14    statistical analysis of the solutions can be

          15    used to pick out the sources of error in a

          16    fashion that I'm proposing.

          17              For what I did, not having, except

          18    in the one case, such an inventory of

          19    solutions was to use the solutions that are

          20    published for the Annual Energy Outlook,

          21    which I have back through 1998.

          22              The reviewers quite rightly in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             86


           1    considering the statistical approach I

           2    brought to those solutions worried that there

           3    is what in economics is called an

           4    identification problem.  In other words, when

           5    I was picking out the influence of price, the

           6    truth is that price is an endogenous variable

           7    in the model.  And as a result, in general,

           8    if you took market data, you wouldn't be able

           9    to just run a simple least squares and

          10    isolate the price effect.

          11              And here is the graphic that the

          12    reviewers provided demonstrating that.

          13              So here is supply and demand

          14    shifting around and in terms of the

          15    elasticity that you want to estimate, there's

          16    no way from these solutions to identify the

          17    slope here, which is what you need.

          18              But since NEMS is not the world, we

          19    can control it more.  And the truth of the

          20    matter is that the principal scenarios for

          21    the AEO were designed by me, in fact, in 1977

          22    to show you how old I am, to speak to the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             87


           1    identification problem.  So specifically if

           2    you consider the oil price variation cases,

           3    what they are doing in principle is freezing

           4    the demand configuration and then shifting

           5    supply.

           6              So the elasticity estimates that

           7    I'll show you before we're done came from

           8    these solutions pooled together.  And

           9    although the reviewers were quite right,

          10    there's a simultaneity problem in general, I

          11    believe it is very successfully spoken to by

          12    the data that I pooled.

          13              But my proposal number two I think

          14    is even better.  If in the end, which has

          15    been done, we generate solution sets where

          16    the variables are changed one by one in terms

          17    of the supply and demand components

          18    individually, then when you pool those data,

          19    you will have completely avoided the

          20    simultaneity problem at least in terms of the

          21    first) order subject to the errors of a

          22    linear approximation.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             88


           1              So in terms of method, what I am

           2    proposing in the end is that we -- I --

           3    somebody agree to an experimental design and

           4    EIA devote the resources so that each model

           5    version can be run through this drill and the

           6    solution saved to then be used in later years

           7    to assess the accuracy of the forecast.

           8              Now for the rest of my talk, I want

           9    to just go through the sorts of things that I

          10    was able to do in the several months that

          11    I've had a chance to work on this.  All of

          12    this is with -- is prospective.  In other

          13    words, my words my purpose was to investigate

          14    how this might be done.  Nothing that I will

          15    provide you was intended to be, and whether

          16    it was or not is certainly not definitive in

          17    terms of the way in which the evaluation

          18    should be conducted, but rather it's my

          19    effort to try to get all of the methods in

          20    hand and to show examples of how they might

          21    be used.

          22              Certainly, for adjusting forecasts




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             89


           1    with respect to differences in the historical

           2    values, weather would be an obvious influence

           3    on energy consumption, so I looked at the

           4    weather data that EIA keeps and found some

           5    things I think probably should be attended

           6    to.

           7              The Annual Energy Review publishes

           8    heating degree day and cooling degree day

           9    time series, which are the major weather

          10    variables in terms of understanding what

          11    weather might have to do with energy

          12    consumption.

          13              NEMS I learned -- and I didn't

          14    really know this until I got into the project

          15    -- does have weather variables in it, and in

          16    terms of my proposal two OIF very nicely ran

          17    two weather cases for me, where they changed

          18    in one case the heating degree day variable

          19    and in the other case the cooling degree day

          20    variable each by 10 percent that I could then

          21    compare to the base case for the current

          22    version of NEMS and then could quite readily




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             90


           1    estimate what the weather effect is.

           2              These data come from those cases

           3    that were provided me.

           4              The short-term model also uses

           5    weather, and these data come from oh,

           6    probably eight weeks ago what was on the

           7    Internet in terms of what the short-term

           8    model was doing.

           9              The point of my remarks is when you

          10    look at the numbers, where you would assume

          11    they would be the same -- here's the AER

          12    definition of normal.  Here is NEMS forecasts

          13    for normal weather in this year, and here's

          14    the short-term models forecast.  They're all

          15    different.  And I don't know why they're

          16    different, but I think it's probably not good

          17    that they're different.  So a byproduct of

          18    this particular discussion is my thought that

          19    this should be somehow cleared up.

          20              When I went ahead and used the data

          21    to calculate the sensitivity, I went about it

          22    three ways just to see what difference it




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             91


           1    would make.  I have now cases where I'm doing

           2    discreet changes.  It's perfect.  It's

           3    absolutely just what you want.  So the base

           4    percent of elasticity, as I understood it,

           5    was the way in which OIF would process the

           6    data, which was to simply take the actual

           7    changes and then use the base case as the

           8    percentage basis for calculating elasticity.

           9              A more standard economic approach

          10    would use the average of the two cases.  But

          11    I decided, because of what I wanted to do

          12    with the data that I would use the constant

          13    elasticity form as the mathematical form to

          14    estimate the sensitivity.

          15              I then looked at what difference it

          16    made, and it made almost no difference as to

          17    which method was used, so this is the method

          18    I used, and what I'll show you in a minute,

          19    and these are actual elasticities that come

          20    out of the 2007 Annual Energy Outlook version

          21    of NEMS.  It's very interesting.

          22              I see that they change over time.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             92


           1    I don't know why, but it's very interesting

           2    to me.

           3              In any case, I took the

           4    elasticities and calculated them and then

           5    compared them to the elasticities in the

           6    short-term model.

           7              And as you see, I found

           8    differences.  I'm not sure why they're

           9    different.  It's just -- was out of my scope

          10    to be able to track that down.  There's even

          11    a sign difference, although I suspect that

          12    these small numbers really should be nil

          13    rather than an actual number.  But still,

          14    there is a sign difference.

          15              I think that this should also be

          16    cleared up.  The methods are different, and

          17    here we all know that the NEMS method versus

          18    the more econometric method that the

          19    short-term model uses can lead to different

          20    results, but outside, say, if I'm explaining

          21    this to my class at Temple, they don't

          22    understand why they're different.  They think




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             93


           1    there should just be one sensitivity.  So I'm

           2    recommending that that somehow be dealt with

           3    as well.

           4              In any case, I did use the results

           5    that I was nicely given and in the results

           6    I'm going to show you in a minute, I took the

           7    constant elasticity form that I estimate from

           8    the data and then applied it to the

           9    historical versus the forecast values in the

          10    following way:  When I have the actual data

          11    and I have the elasticity and I have

          12    historically what the consumption number was,

          13    I can back out the value of the

          14    multiplicative constant in this functional

          15    form.

          16              And given that, I can plug in what

          17    the normal weather consumption would be and

          18    then difference them and this would give me

          19    my adjustment due to the weather that

          20    happened versus the assumption of normal

          21    weather.

          22              And in a minute, this particular




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             94


           1    method will be showing up in the impact

           2    analysis that I'll show you.

           3              One of the most interesting things

           4    I had an opportunity to do was to see the

           5    degree to which I could look at the price

           6    sensitivity of NEMS across years worth of

           7    NEMS versions, 1998 all the way through this

           8    year's 2007 version.

           9              I used a very austere specification

          10    and the reviewers quite correctly pointed out

          11    that when you got down to doing this really,

          12    the specification that I used was certainly

          13    too austere, and I agree with that entirely

          14    and, in fact, my very next order of business

          15    is to I hope get some advice from the

          16    modelers as to what the proper specification

          17    would be for some of these regressions.

          18              In the meantime, to get on with it,

          19    I ran this regression for the data pooled

          20    across the base case and oil price cases for

          21    all 10 years worth of versions of NEMS.  And

          22    because EIA published elasticities in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             95


           1    long-run and short-run cases, since I have

           2    the lagged endogenous variable in this

           3    specification, I went ahead and backed out

           4    the one year and the two year and the long

           5    run elasticities as well using these formula

           6    just to compare them.

           7              There is a problem potentially,

           8    which is to say the regression doesn't have

           9    to come out right.  Specifically, the price

          10    elasticity certainly should be negative and

          11    the driver elasticity certainly should be

          12    positive, and for the long-run adjustment to

          13    converge the parameter that you estimate for

          14    the lag term has to be positive but less than

          15    one for the whole business to work.

          16              And although it generally worked,

          17    it didn't always work, and here is for the

          18    four fuel types for the 10 years and for both

          19    residential and commercial sector

          20    consumption.  Here is a scorecard, and, as

          21    you can see, distillate really didn't work

          22    very well at all.  The others worked most of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             96


           1    the time.  I think the problem is probably

           2    partly that the data series that I'm using

           3    are a little sparse and if I get the custom

           4    runs, that this will be better.  But I think

           5    probably the problem is more that I've left

           6    something out.  There's something excluded in

           7    my simple specification that is causing the

           8    trouble.

           9              But most of them did work.  And

          10    there's a hard copy.  This is fun to look at,

          11    so we can't really enjoy this as much as it's

          12    worth.  But here, think of it:  Here are the

          13    elasticities for the versions of NEMS between

          14    1998 and this year for these four fuel type,

          15    both long-run and short-run.  All of the rows

          16    which are labeled SIM are my regressions.

          17    The rows that are labeled AEO9903 and are

          18    from the much better experimentally designed

          19    data that OIF created, so there was a report

          20    that gave me the elasticities for 1999 and

          21    2003 and then, although there wasn't a

          22    report, I was given the solutions and can




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             97


           1    read out of the spreadsheets where they've

           2    done it what the values would be for 2006.

           3              As you can see, they move around a

           4    lot.  I think part of that is because the

           5    specification I'm using is too austere, but

           6    the better designed values also move around a

           7    lot, so I think there's a lot of interesting

           8    groundwork that we can go through to see

           9    what's going as far as the model is

          10    concerned.

          11              So these are all available to you.

          12    I have a lot of fun looking at them, and it

          13    is a very interesting result.

          14              But in terms of the bottom line,

          15    what I wanted to do was to take what I had

          16    done and assuming that it was good enough, to

          17    go ahead and formally partition the

          18    differences between the NEMS projections and

          19    historical data insofar as my statistical

          20    specification would allow me to do it.

          21              Here's an example of the way it

          22    would work:  Here is a regression for the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             98


           1    year 2000 version of NEMS, where I'm looking

           2    at commercial- delivered energy, and the

           3    regression is from data pooled across the

           4    base case in the two oil price cases.  The

           5    outcome of the regression in terms of the

           6    implied elasticities is very plausible;

           7    certainly consistent.  And this then was the

           8    mathematical underpinnings of how and then I

           9    would look at the projections from that

          10    version of NEMS and try to see what might

          11    explain the differences between the

          12    projections and historical data.

          13              Well, this is a good one.  The

          14    projection is very close.  It's only two

          15    percent low.  So the first line here shows

          16    the difference between what was projected by

          17    NEMS and what actually happened.

          18              The other lines in the table take

          19    each of these influences -- the price, the

          20    driver, which for commercial energy

          21    consumption that I used I used floor space --

          22    the lag endogenous variable and then weather




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                             99


           1    adjusting in the fashion that I mentioned

           2    earlier and went ahead and looked at what

           3    actually happened compared to what was

           4    assumed and then determined how that would be

           5    expected to influence the forecasts.

           6              So, for example, the price that was

           7    assumed in the year 2000 for the year 2005

           8    was 24 percent low.  So you would think as a

           9    result that the NEMS forecast would be high

          10    compared to what actually happened with the

          11    price a lot higher.  So that's what the 4.41

          12    percent is -- how high you would think the

          13    forecast would be given the fact that

          14    historically the price was 24 percent higher

          15    than was assumed.

          16              Then almost perfectly offsetting

          17    that is the fact that the actual drivers

          18    values were larger than those assumed.  So

          19    for that you would think the forecast would

          20    be low.  And so these two influences offset.

          21              In any case, by the time you take

          22    each of these influences identify them and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            100


           1    add them all up, you would think the forecast

           2    would be one percent low, but instead it was

           3    a little over two percent low, so the

           4    uncertainty number that I present is what

           5    balances that out.  In other words, the

           6    forecast was a little lower than you would

           7    have thought given the explanatory variables

           8    that I was keeping track of.

           9              So I went ahead and did this and

          10    again you have the hard copy and you can

          11    enjoy this more at leisure than we can in

          12    this talk, but I did it for five variables

          13    across the AEO versions up through 2004 for

          14    the year 2005.

          15              I'm advised that for the versions

          16    of NEMS which are quite close to the year

          17    that's being projected that actually the way

          18    the projections are published relates to the

          19    short-term model to some degree as well as

          20    NEMS and it's a little bit not right to view

          21    this as accuracy of NEMS per se.  But for the

          22    ones that are five and six years into the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            101


           1    past, I think that is fine for NEMS.

           2              Interestingly, I found that using

           3    my specification and the adjustment process

           4    that I've outlined, I found for four of the

           5    five fuels that I kept track of that the

           6    forecast ended up being lower than I would

           7    have thought, consistently, year by year,

           8    which could be because I'm just leaving

           9    something out, except for commercial

          10    electricity where it was just the other way

          11    around.

          12              And although I think there is a lot

          13    of room for improvement in terms of the

          14    particular statistical methods that I'm

          15    using, I think, nevertheless, this kind of

          16    result is indicative as to what can be

          17    brought out of this kind of analysis to

          18    assist the modelers in understanding what the

          19    model is doing.

          20              Accordingly, is my proposal that we

          21    proceed along the following lines.

          22              First, I really do think that the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            102


           1    weather business should be at least

           2    explained.  I could just not understand

           3    something, which is very clear, but the way

           4    it looks from the outside, there's too many

           5    differences.

           6              In any case, otherwise, I believe

           7    that we should move towards the direction of

           8    until we do archive NEMS, which I suspect is

           9    not going to happen, we should move towards

          10    developing an experimental design to run

          11    versions of the models through so that the

          12    solution sets that you bring together from

          13    that design can be used to isolate the

          14    influences of the explanatory variables.

          15              My idea would be that at least for

          16    the time being as these solutions are

          17    created, I can then test the specifications

          18    that I'm given and then there can be

          19    interaction -- will it work or won't, to some

          20    degree -- and we can iterate through that

          21    analysis.  But the target will be to bring

          22    into being I would think an annual report




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            103


           1    which presents -- and there's much to con --

           2    in the template that's missing yet -- but

           3    still the general idea would be to present

           4    how NEMS is doing and to try to understand

           5    the major sources of differences between what

           6    the projections said and what actually

           7    happened, and not only cope with the things

           8    that I can talk about, but also try to deal

           9    with data differences, and in the end have a

          10    good, solid quality control handle on what

          11    that model is doing, which would be a

          12    credential to the users of the mode and I

          13    also believe a diagnostic tool that would

          14    help the modelers.  So my efforts are to try

          15    to point towards something much better than

          16    I've been able to do, but try to demonstrate

          17    that this is an important and useful

          18    approach.

          19              DR. NEERCHAL:  Thank you.  Can we

          20    start with the discuss -- (off mike)

          21    discussions and then we'll open it up.  Jae,

          22    you want to go first?




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            104


           1              DR. EDMONDS:  I thought you were

           2    listed first.

           3              DR. NEERCHAL:  And the thing is

           4    that a good gesture for me to say go first.

           5              DR. EDMONDS:  Well, let me just

           6    first say that -- two things.  One is that I

           7    was part of that session last time, where we

           8    sort of began looking at this issue, which

           9    was an opportunity to sort of follow up with

          10    George's career.  I've been learning from

          11    George for as long as I've been working in

          12    the energy business, so it's always a

          13    pleasure to get back in touch.

          14              In terms of trying to reflect back

          15    and give you some helpful comments, I wasn't

          16    -- I read through the material, and I wasn't

          17    quite clear exactly what the -- you know,

          18    what the questions you'd like us to deal with

          19    were, so what I'm going to say may or may not

          20    be at all helpful, so, you know, I apologize

          21    if that's the case.

          22              And just thinking through it, I was




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            105


           1    having, you know, trouble getting my mind

           2    wrapped around exactly what we're trying to

           3    do.  The basic idea is to try to understand

           4    how do you make NEMS a more accurate forecast

           5    or if there is a -- you know, if there is --

           6    will always irreducible uncertainty; that is,

           7    if you're given a set of input drivers and,

           8    you know, they're coming over from the

           9    Council of Economic Advisors and it's 25

          10    percent higher than, you know, your

          11    assessment, you know, it's coming from the

          12    Council of Economic Advisors and this is what

          13    you're going to run, and so, you know, some

          14    of that is just going to be inevitable; you

          15    know, things that are just going to be

          16    outside your control.

          17              But the model itself is inside your

          18    control and so, you can actually go back and

          19    say well, what if I had the right, you know,

          20    gross domestic product?  What if I had, you

          21    know, the right -- you know, any set of

          22    exogenous variables that go into NEMS, and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            106


           1    you're going to ask, you know, how -- you

           2    know, how well did I do if I had that

           3    information correct, and how do I make it a

           4    better forecast.  And that seems to me to be

           5    what this activity is really geared at and

           6    when you just sort of go back to the

           7    foundations.  How do you make the NEMS a

           8    better modeling system and so with that in

           9    mind, you know, NEMS, of course, to anyone

          10    who has tried to, you know, to -- has

          11    experience with NEMS knows that it's a --

          12    it's not a simple model.  It's a

          13    sophisticated system of models and so how do

          14    you get your arms around it?

          15              I guess the first thing to

          16    recognize is that, as George has pointed out,

          17    the forecast accuracy really depends on the

          18    variable that you're looking at.  I mean it's

          19    going to be different depending upon whether

          20    you're looking at, you know, total primary

          21    energy or whether you're asking yourself what

          22    was the household consumption of natural gas,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            107


           1    and each one of those is going to give you a

           2    different problem to deal with and the

           3    strategy that, as I understand it, you've

           4    laid out is to say I don't always want to

           5    have to deal with the entirety of NEMS.  I

           6    actually want to be able to get the essential

           7    NEMS and so there is this process by which

           8    you're trying to create a reduced form

           9    representation; is that correct?  So you're

          10    actually going back and saying, you know, I

          11    want, you know, I want the simplified

          12    version.

          13              And the right simplified version,

          14    at least in my experience -- of course,

          15    depends on that prior question -- that is,

          16    what is it I'm trying to explain.  If I'm

          17    trying to explain the price, I'm going to

          18    have one reduced form model.  But if I'm

          19    trying to explain total primary energy, I'm

          20    going to have a different reduced form model,

          21    and so forth.

          22              From the perspective of these




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            108


           1    questions then the question as I would see it

           2    is if I'm looking at the aggregate

           3    performance of NEMS, I have to kind of -- and

           4    I've tried to create a reduced form version

           5    of that model -- then, in fact, I have to

           6    kind of put a wrapper around the model with

           7    its, you know, what's exogenous and what's en

           8    -- and what is the projection, and everything

           9    else then kind of becomes a black box in the

          10    middle.  If you're looking at sub-modules,

          11    as, in fact, you're doing here, where you're

          12    kind of taking out a piece and actually

          13    asking how's the performance of that piece,

          14    you actually end up going back to the same

          15    kind of question.  What's the wrapper around

          16    that piece and how do you actually get your

          17    arms around that wrapper.

          18              From the presentation, I couldn't

          19    quite tell -- I mean there are well developed

          20    methodologies for creating reduced form

          21    models, and I'm going to presume that the,

          22    you know, the sensitivity uncertainty




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            109


           1    analyses that you would do to actually pull

           2    -- you know, create the parameterizations

           3    (ph.) and to do the attributions are being

           4    done below that wrapper, but it might be --

           5    and it might be a good, you know, exercise to

           6    just sort of go through and, in fact, discuss

           7    how that, you know, that happens.

           8              But once you've done that, then, of

           9    course, you do begin to get those insights.

          10    How much of the -- of the error was, in fact,

          11    attributable to the fact they got the price

          12    wrong?  How much of it was attributable to

          13    the fact that they just didn't have the

          14    square feet of floor space right?  How much

          15    of it was the fact that they missed -- they

          16    were using an average winter and, in fact,

          17    the winter turned out to be much warmer than

          18    normal and so forth?

          19              And then, of course, you're left

          20    with that uncertainty piece, which I presume

          21    is what leads you to try to do this on an

          22    annual basis to figure out how much of that




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            110


           1    uncertainty is, in fact, coming out of the

           2    structure, and, as the structure changes --

           3    the structure of the model changes, is, in

           4    fact, that structure leading to a more

           5    accurate representation of the behavior or is

           6    that structure actually, you know, it's

           7    evolving.  It may be more attractive in some

           8    -- in some aesthetic way, but it's not

           9    actually performing better, and I think that,

          10    you know, that actually is a really

          11    interesting piece because, in fact, in the

          12    world of modeling, what you always

          13    implicitly, if not explicitly, assume is that

          14    the model is, in fact, the true

          15    representation of the world, and it's only

          16    the data that carries the uncertainty.

          17              Well, of course, you know, there

          18    may be occasions when you actually had the

          19    wrong structure of the world, and so this is,

          20    in fact, a way of asking as the structure of

          21    the world, according to NEMS, evolves is

          22    that, in fact, evolving in a way that is




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            111


           1    improving the accuracy or not.

           2              And those are just -- those are the

           3    thoughts that came to me in sort of trying to

           4    reflect back.  If there's some specific

           5    questions you would have for us about how to

           6    -- you know, that we could help use to help

           7    you, you know, just let us know.

           8              DR. NEERCHAL:  Thanks.  Yeah, I

           9    want to make a couple of comments basically

          10    starting with an apology.  I do not know a

          11    whole lot about -- I know very little about

          12    NEMS, except that I know it's a very

          13    important model, you know, it comes up every

          14    other presentation that I've been through

          15    this current year.  I think it's a very

          16    important thing, and I definitely think it

          17    deserves full-scale sensitivity analysis, the

          18    kind of thing you're proposing.

          19              And, you know, from what I heard, I

          20    think who else should be doing it, the one

          21    who has created it.  I think it seems like a

          22    good choice.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            112


           1              But I have a couple of comments.  I

           2    think that one thing when I look at or when I

           3    hear about accuracy about something that I

           4    don't know much about, so I'm going to be

           5    doing it at the totally at the top level --

           6    actual, predicted (off mike).  So there is

           7    very layman's level assessment of accuracy.

           8    I think that's what you're sort of referring

           9    to your proposed action number one.  I didn't

          10    say look at a table and say here's an actual

          11    column and here's the proposed column and if

          12    there is a big difference, put a star saying

          13    that this happened because of this weather

          14    event that is not part of the NEMS.

          15              So it's a very top level, if you

          16    will.  A kind of top level explanation of why

          17    some things were not captured by NEMS'

          18    forecast.  I think that will be very useful.

          19              The other thing would be I would

          20    say the other extreme, and for, as a

          21    researcher in NEMS, they might want to (off

          22    mike) have a whole lot more and very detailed




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            113


           1    sensitivity analysis, and I think that what

           2    you're proposing, coming up with a design,

           3    you know, properly chosen, I think that's the

           4    way to do it, and I think in this (off mike)

           5    literature there are papers -- I think Gerry

           6    Sachs and Cohen -- but many papers on how to

           7    analyze computer models, how to analyze the

           8    uncertainty in computer models.  There are

           9    quite a few papers.  I can give you a couple

          10    of references on that work.

          11              Statistical Sciences about maybe

          12    six or eight years ago had a, you know,

          13    leading article by Gerry and (off mike), a

          14    very useful reference.

          15              DR. BINGHAM:  '93.

          16              DR. NEERCHAL:  '93.

          17              DR. BINGHAM:  Sachs is the first

          18    author of that.

          19              DR. NEERCHAL:  And I think --

          20    something like that would be very useful in

          21    deciding what the design points should be and

          22    so on.  That's one thing.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            114


           1              I do want to ask you a question:

           2    How much -- what do you mean by a full

           3    archive of NEMS and how much effort is

           4    involved in it?  I was curious about that.

           5              What do you mean by full archive?

           6    And I was thinking is the code itself it

           7    takes three hours to run or 10 hours?

           8              DR. LADY:  Well, the idea of --

           9    originally was that in some sense a third

          10    party could run the model and (off mike) the

          11    basis for any published projection.  In terms

          12    of the reviewers of this report and what I

          13    would say, too, and did say, it would be a

          14    version of the model that can be run when the

          15    time comes with historical data substituted

          16    for assumed data and then further run against

          17    an experimental design where you can isolate

          18    each influence.

          19              Again, while that clearly is the

          20    best thing to do, I suspect it's not

          21    practical, at least for the moment, and I

          22    further argue that the proposal that I've




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            115


           1    made will probably do very well in terms of

           2    serving the same purpose and be much simpler

           3    to work with.

           4              DR. NEERCHAL:  I will open.  Yes.

           5              DR. KOKKELENBERG:  How many -- you

           6    mentioned you had a set of AEOs back in '98.

           7              DR. LADY:  Yes.

           8              DR. KOKKELENBERG:  But how -- when

           9    did the AEOs start?  When was the first year

          10    it was published?

          11              DR. LADY:  1974 would be the first

          12    time that the --

          13              DR. KOKKELENBERG:  Okay.  So that

          14    data from '74 to '97 is not available or?

          15              DR. LADY:  That I don't know.

          16              DR. KOKKELENBERG:  Okay.

          17              DR. LADY:  I -- there's a graphic

          18    interface for NEMS that allows -- that first

          19    of all requires that the solutions be

          20    generated in a form that the program can

          21    read, and then I can use that program to take

          22    data out to do statistics with or do anything




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            116


           1    I want with, and that starts with the '98

           2    version.

           3              DR. KOKKELENBERG:  I see.  Okay.

           4    So then when you're doing your tests and

           5    you're calculating your elasticities, how

           6    many observations did you have for let's say,

           7    you calculated the elasticity of residential

           8    natural gas consumption?

           9              DR. LADY:  All I'm probably --

          10              DR. KOKKELENBERG:  How many years

          11    do you have?

          12              DR. LADY:  I probably did it -- I

          13    would do it for three solutions, 20 years per

          14    solution, so I would have 60 -- 60.

          15              DR. KOKKELENBERG:  So you would

          16    have it for -- those three solutions were

          17    let's say the high, low, and medium GDP

          18    growth or something like that?

          19              DR. LADY:  Well, the oil price

          20    cases would be the ones that would be --

          21              DR. KOKKELENBERG:  Okay.

          22              DR. LADY:  -- proper.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            117


           1              DR. KOKKELENBERG:  So you'd have 20

           2    observations --

           3              DR. LADY:  Per solution.

           4              DR. KOKKELENBERG:  Per solution.

           5              DR. LADY:  And then I would pool

           6    them.

           7              DR. KOKKELENBERG:  Okay.  So you'd

           8    pool those.  Would that maybe explain why

           9    you're getting this high variability in your

          10    elasticities.

          11              I mean first of all, you have a

          12    short number of years, 20 observations, and

          13    then you're pooling them and so the standard

          14    error on an individual elasticity, evaluated

          15    at any particular year, is probably going to

          16    be pretty high?

          17              DR. LADY:  Well, not in the

          18    regressions they're not.

          19              DR. KOKKELENBERG:  You mean the

          20    standard error -- do you calculate the

          21    standard error of the elasticity, not the

          22    coefficients but -- it would be if you used




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            118


           1    the log form to do that.

           2              DR. LADY:  No, I didn't do that.  I

           3    could use -- I didn't use the log form, so

           4    I'd only have the P-statistic on the

           5    coefficient.

           6              DR. KOKKELENBERG:  Okay.

           7              DR. LADY:  Not on elasticity.

           8              DR. KOKKELENBERG:  Okay.  If you --

           9    you can calculate the elasticity standard

          10    errors, but you have to recognize whatever

          11    form you have and how many times that

          12    variable enters in.  But that may be one way

          13    to looking at why you get these -- I'm really

          14    disturbed by these elasticities that jump all

          15    over the place, from 1.4 down to negatives

          16    for some (off mike).

          17              DR. LADY:  Well, I suspect that the

          18    specification is too austere.  I don't think

          19    -- I think that there's some reason to think

          20    that this is my fault for not quite using a

          21    sufficiently detailed specification.

          22              DR. KOKKELENBERG:  I'm not sure




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            119


           1    that -- I don't know.  What -- do you think

           2    that would help?  I don't think it would.

           3              DR. KHANNA:  It's hard to say.

           4              DR. KOKKELENBERG:  It's hard to say

           5    if it would help, but I think, particularly

           6    since you're using the lagged value, which

           7    should be capturing everything that

           8    influences the lagged value.  Theoretically,

           9    that model makes a lot of sense.

          10              DR. LADY:  I agree.

          11              DR. KOKKELENBERG:  And I believe

          12    people have said this for years that they

          13    can't forecast what influences electricity

          14    demand, but if you use last year's

          15    electricity demand, it's a good proxy for all

          16    that equipment that's in place that demands

          17    electricity.

          18              And --

          19              DR. LADY:  Well, I can tell you

          20    that I have run the log form, these

          21    regression utility that makes it easy to do,

          22    and the T-statistics on the coefficients,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            120


           1    which are the elasticities in that form are

           2    very high.  In other words, the regression

           3    doesn't make it look like there's a large

           4    variation for each case.

           5              DR. EDMONDS:  Well, the first

           6    question is, you know, in that module, did

           7    you actually do a full sensitivity analysis

           8    so that you know every, you know, every ex --

           9    in that module, every exogenous variable and

          10    what its potential contribution to -- you see

          11    that, you know, that's what may be what he

          12    means when he says too austere --

          13              DR. LADY:  Yes.

          14              DR. EDMONDS:  -- that, in fact, if

          15    you went back and you did a -- you know, if

          16    you systematically went through and just --

          17    which is kind of the first step and when

          18    you're trying to do this reduced form model

          19    is you just take everything that's outside

          20    the model, outside that module, and you just

          21    perturb it, you know, one percent or whatever

          22    the appropriate value and just see what's the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            121


           1    impact on the dependent variable of interest

           2    and that's kind of where you start and then

           3    you have to put weights on those to get a

           4    sense of, you know, are some of these things

           5    just, you know, they never change, you know,

           6    it's the solar constant or something, which

           7    actually does change, but let's set that

           8    aside.

           9              But anyway, so, you know, that, in

          10    fact, may be what you're referring to in it's

          11    too austere.  There may be something in there

          12    which actually is more important than you are

          13    giving it credit for a priori.

          14              DR. NEERCHAL:  Derek.

          15              DR. BINGHAM:  So I'm going to

          16    change the topic here slightly so.  Just

          17    regarding kind of your overall plan, what

          18    you're describing of, describing in the

          19    statistics literature would be called a

          20    design and analysis of (off mike) which

          21    Nagaraj was referring to.  So there's been a

          22    thematic year this year at SAMSI and at also




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            122


           1    at NCAR on this.  And I work in this area, so

           2    I can actually tell you a little bit about

           3    this.

           4              So let me ask you a quick question:

           5    Is NEMS deterministic?  I mean if you put the

           6    same inputs in, do you get the same output?

           7              DR. LADY:  I don't believe so.

           8    It's a numerical algorithm, so there is a

           9    tolerance definition to solution.

          10              DR. BINGHAM:  So if I were to run

          11    it and you were to run it, we would get

          12    different answers?

          13              DR. LADY:  No, you should get the

          14    same answer, yeah.  It's deterministic.

          15              DR. BINGHAM:  Okay.  So any

          16    regression function that you fit to this,

          17    okay, is going to be a wrong model because

          18    all of the model -- your errors are not

          19    independent because everything comes from --

          20    there's no randomness.  It's all model mis-

          21    specification, and that's why typically what

          22    people will do instead is use something like




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            123


           1    a Gaussian process model where you kind of

           2    view the function as a random function.

           3              And so there's a whole literature

           4    out there on this, and there are also design

           5    experiments that kind of go along with this

           6    as well to basically what you want to -- the

           7    worst thing you could do is do a one at a

           8    time sort of thing.

           9              What you want to do is you have an

          10    input space and you want to fill that space;

          11    right?  So there are inputs into your model

          12    and what you want to do is somehow fill this

          13    as best as possible and run the model at a

          14    whole bunch of different places, and you can

          15    either keep those runs as archive or you

          16    could try to build surrogate models from

          17    those.  So rather than keeping all the runs,

          18    you could build a series of surrogate models,

          19    either for component wise or for a global

          20    model, based on -- you wouldn't try a

          21    regression model.  It would be -- the order

          22    of the polynomial or something that you would




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            124


           1    need.  It would be way too high.  And you

           2    would use something called a Gaussian process

           3    model or some sort of non- parametric

           4    regression model.

           5              And right and so -- what the folks

           6    at NASA and Los Alamos and NCAR will do is

           7    they'll archive the -- that model and use it

           8    as a surrogate for the computer code later

           9    on, and also with associated uncertainties.

          10              And if you're looking through that

          11    literature, there's a slight disconnect, so

          12    one of the things that you mentioned that you

          13    wanted to do was design experiments and also

          14    identify the important drivers and in that

          15    literature it would be called -- that would

          16    be called screening; right?

          17              So you're looking for the active --

          18    the big guys, the things that are really

          19    driving the system.  Whereas sensitivity

          20    analysis is typically a post-hoc thing, so

          21    I'm just giving you the definitions in that

          22    literature.  If you decide to dive into it,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            125


           1    the sensitivity analysis is usually deciding,

           2    you know, what is a functional form.  So

           3    would something have a quadratic effect or,

           4    you know, some sort of periodic effect or

           5    linear effect on any type of output.  So that

           6    would fall under sensitivity analysis and

           7    there's a set of references -- there's

           8    actually a book by Cytelli on sensitivity

           9    analysis, which has a different -- that

          10    phrase has a different meaning in almost

          11    everybody's field.

          12              When you're looking at computer

          13    models and doing experiments on computer

          14    models and simulate or computer simulators of

          15    physical processes so sensitivity analysis is

          16    usually a post talk, you know, an

          17    explanatory, you know, explaining what's

          18    going on; whereas screening is usually

          19    identifying the big guys.  And there is a

          20    broad literature on that, and we can give you

          21    a list of references offline if you want.  No

          22    need to --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            126


           1              DR. LADY:  Okay.

           2              DR. BINGHAM:  -- list them all now.

           3    But I also think that that parts of this

           4    would be so given that there's been a lot of

           5    effort to try to get -- there was a summer

           6    school that was held on computer experiments

           7    last year that was part of this thematic year

           8    at SAMSI, which is an applied math and

           9    statistics institute affiliated with NISS.

          10    There's a lot of people out there who are now

          11    trained in it, young people, graduate

          12    students, who might fit very well into this

          13    graduate program that you're trying to set

          14    up.  It would be a perfect place for an

          15    internship to kind of transfer some of that

          16    knowledge both on the design and on the

          17    analysis side.

          18              So I think this is a methodology

          19    that would be very useful for you that's kind

          20    of sitting out there, and there's lots of

          21    code available --

          22              DR. LADY:  Okay.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            127


           1              DR. BINGHAM:  -- on the Web, so you

           2    could probably steal their code and start

           3    there.

           4              DR. NEERCHAL:  Good refer (off

           5    mike) in the summer.  Advice.  I'll say.

           6    Keep them off the street.  Walter?

           7              DR. HILL:  Nagaraj, you introduced

           8    that you know very little about these models.

           9    I know less.  This has been a great break out

          10    session.

          11              DR. NEERCHAL:  You mean (off mike)?

          12              DR. HILL:  Yeah, I've made a note

          13    in reference to your comment that it looks

          14    like their first model of observation, so

          15    that may partially explain the reason why

          16    you're getting a fairly large error.

          17              You also brought the -- your list

          18    of unknowns.  It looks like those -- a number

          19    of those will be unknown that you will not

          20    know the number of (off mike) or accidents

          21    (off mike) by definition.  It won't be known

          22    ahead of time.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            128


           1              DR. LADY:  I can't hear.  I'm

           2    sorry.

           3              DR. HILL:  That number of (off

           4    mike) -- some of these -- that number of

           5    accidents will not be known ahead of time,

           6    and so it will be difficult to put that in

           7    the original model, although you did mention

           8    I think (off mike) in the current model.

           9              DR. LADY:  You mean that (off mike)

          10    definition model?

          11              DR. HILL:  Some of these transitory

          12    influences, like accidents, for example, you

          13    will not --

          14              DR. LADY:  Right.

          15              DR. HILL:  -- you will not know.

          16    You either do not know.  You will not know

          17    how many accidents that will be in the summer

          18    of this year.

          19              DR. LADY:  Sure.  I don't know

          20    about (off mike) Katrina.

          21              DR. HILL:  Yes.

          22              DR. LADY:  That's right.  But I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            129


           1    think it should be part of what's done?

           2              DR. HILL:  Right.  And (off mike)

           3    that, but in --

           4              DR. LADY:  Mm hmm.

           5              DR. HILL:  -- eventually that's put

           6    in the (off mike).

           7              DR. NEERCHAL:  You want to use the

           8    microphone for the transcription.  Yeah.

           9              SPEAKER:  There's a problem.  He

          10    can't hear Walter until she stands there, and

          11    then she can't hear him when he talks.

          12              SPEAKER:  There's a lavaliere mike.

          13              SPEAKER:  Hello.  Hello.

          14              SPEAKER:  Speak louder perhaps.

          15              DR. LADY:  I -- it is my proposal

          16    that the institutionalization of this idea

          17    would include accounting for what you might

          18    call offline influences as well as the sort

          19    of analytical things that we've been talking

          20    about, like Katrina; yes.

          21              DR. HILL:  Perhaps those are put in

          22    the later model.  Those are not in.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            130


           1              DR. LADY:  I'm not so sure that

           2    that's -- by model you mean what's done to

           3    generate --

           4              DR. HILL:  Post hoc.  Post hoc.

           5    What is the effect of the Katrina?

           6              DR. LADY:  I would view that

           7    probably to be simply offline, and I'm not so

           8    sure you would want to try to model that.

           9              DR. HILL:  But it must affect

          10    energy use?

          11              DR. FORSYTH:  The star.

          12              DR. LADY:  Mm?

          13              DR. HILL:  It must affect --

          14              DR. NEERCHAL:  The star, yeah.

          15    Star.  That's -- you know, you're talking

          16    about the weather and I think something that

          17    you want a, you know, a regular web browser

          18    will catch saying wait, this actual doesn't

          19    fit the predicted outcome rate.

          20              DR. LADY:  Mm hmm.

          21              DR. NEERCHAL:  I think those are

          22    easy -- I mean (off mike) easy, but you can




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            131


           1    have some identifiable causes saying if this

           2    Katrina; this is, you know --

           3              DR. FORSYTH:  The or --

           4              DR. NEERCHAL:  -- or whatever, you

           5    know.  Whatever else is there -- I think that

           6    something, you know, understandable in a

           7    layperson's language; whereas, elasticity and

           8    those kinds of things are not understandable

           9    by laypersons' language.  If there is a

          10    problem due to issues like Ed is raising and

          11    common people cannot really -- but they don't

          12    really get into that kind of level, so that

          13    is a different level of, you know, accuracy

          14    check.  You know, so I was trying to divide

          15    them into groups in my mind to understand

          16    this NEMS accuracy check one, from a very top

          17    level -- top or bottom, I don't know --

          18    easier perspective; right?  I don't want to

          19    get into that.

          20              So at one extreme of very low level

          21    knowledge like me about NEMS and the other

          22    one is someone who is more -- (off mike) more




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            132


           1    expert level knowledge of NEMS.  You know,

           2    how do you -- how do you convince them?

           3              So the expert people will not

           4    bother with the small change.  They know that

           5    it's a model.  It's -- I mean it's supposed

           6    to be wrong sided most of the time.  It's an

           7    accident if it keeps right on the target.

           8    You know, so experts shouldn't really be

           9    excited about matching.  Experts should

          10    expect a certain error, because it's a model.

          11    But what the expert will be divided about,

          12    for example, if you look at the NEMS -- those

          13    are beautiful runs -- the last table, if you

          14    look at 1998, '99, 2004, and actual listed in

          15    the next column, most of the time the numbers

          16    are actually moving towards the actual,

          17    almost always.  You know it starts with a big

          18    gap and then slowly they converge.

          19              I mean those are the things that,

          20    you know, an expert might notice, but not a

          21    layperson.  You know, he's going to -- oh, I

          22    was right on 2001, because, you know, I put a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            133


           1    thousand dollars in the Vegas or something

           2    like that.  The layperson is going to worry

           3    about instantaneous issues.

           4              So for those people when they --

           5    when you are not matching or when you're

           6    significantly different, you may want to be

           7    able to explain the identifiable cause and

           8    say this was not -- this is not (off mike)

           9    this way.  That's what I was saying.  I don't

          10    know whether you are referring to the same?

          11              DR. HILL:  Yes.

          12              DR. KIRKENDALL:  I think I sort of

          13    --

          14              DR. NEERCHAL:  I think Neha has

          15    been waiting for a while and then I think now

          16    I forgot we tell the two Eds.

          17              DR. KHANNA:  So my comment is

          18    possibly completely off the (off mike)

          19    because I may have (off mike) misunderstood

          20    what you do, but it's actually is related to

          21    what Derek was trying to say.  The different

          22    outcomes that you get you essentially say




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            134


           1    it's the weather.  Heating degree days were

           2    higher than what we had modeled.  You know,

           3    here's one heating degree day where we look

           4    at the output in electricity demand, et

           5    cetera.

           6              There's another heating degree day

           7    now going to look at what electricity demand

           8    number NEMS spits out for a different value

           9    heating degree day; right?

          10              And that's how you calculate (off

          11    mike).  So because the model is

          12    deterministic, the only reason that those

          13    numbers differ and you can't predict a priori

          14    how they're going to differ or exactly what

          15    the difference is going to be, it's because

          16    of the inherent non-linearity in the model;

          17    right?

          18              And we don't know exactly what the

          19    shape of that non-linearity is.

          20              DR. LADY:  I suspect not, because

          21    my experience is that what you might call the

          22    variable parameter manifold that describes




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            135


           1    the model is pretty flat and, in fact, for

           2    many of the major series, just using linear

           3    relationships, you could get a very good fit

           4    with the NEMS solution.  So I suspect that

           5    the differences in the elasticities that

           6    we're commenting upon are due, in part, to

           7    the need for a better specification than the

           8    one I used.  But even in the NEMS experiments

           9    themselves across the three years where it

          10    was done properly, (off mike) statically with

          11    the modules, you got big differences as well.

          12    So the modelers would be the ones who could

          13    explain why the model is changing that way I

          14    think.

          15              I mean part of this is to reveal

          16    what should excite interest and explanation.

          17              DR. NEERCHAL:  Sorry.

          18              DR. BLAIR:  Actually, I'll defer to

          19    him.

          20              DR. NEERCHAL:  Okay.

          21              DR. KOKKELENBERG:  Oh, well, I

          22    guess I'm totally confused.  The NEMS itself




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            136


           1    is deterministic and that -- evaluating the

           2    computer models was what our deterministic

           3    was addressed by Derek as far as I can tell.

           4              But what you're doing, the way I

           5    understand it, is taking the forecasts that

           6    NEMS made and then comparing that to the

           7    actual, which is a test of forecast accuracy.

           8    So here there is a stochastic element.  So

           9    it's -- you've got about three different ways

          10    you're trying to evaluate NEMS.  One is the

          11    way that Derek suggested, taking a hard look

          12    at how deterministic computer models arrive

          13    at solutions and what could be done to

          14    enhance that.

          15              The second is to look at forecast

          16    accuracy, and see, you know, take the worst

          17    things you forecast and why are they so bad.

          18              And then the third thing is what

          19    Neha was talking about was scenarios

          20    essentially; tweak a variable, and then see

          21    if that comes closer to the actual in your

          22    model.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            137


           1              But then you need a rather complex

           2    -- well, you really need the NEMS model to do

           3    the tweaking.  You can't just use the similar

           4    equations.

           5              So and there's probably five other

           6    way to test the accuracy that I haven't

           7    immediately thought of.  But I think you're

           8    going to have to -- well, didn't we publish

           9    or EIA publish a study of the NEMS

          10    forecasting accuracy very recently as part of

          11    the --

          12              DR. KIRKENDALL:  One of the NEMS

          13    people want to speak to that.

          14              SPEAKER:  Yeah.  We can clear up a

          15    lot of this stuff right here.

          16              DR. KIRKENDALL:  Well, come up.

          17              DR. KOKKELENBERG:  Okay.

          18              DR. KIRKENDALL:  Come on up to the

          19    table and talk.

          20              DR. KOKKELENBERG:  That's what I

          21    was going to say.  What's their experience?

          22              DR. NEERCHAL:  Come.  Come, come,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            138


           1    come.

           2              DR. KIRKENDALL:  There's a couple

           3    of seats here.

           4              DR. NEERCHAL:  Grab one of the

           5    chairs here.

           6              SPEAKER:  So I (off mike) spoke up

           7    earlier about it.  I didn't know what the

           8    protocol was.

           9              Okay.  So the reason the

          10    elasticities change over time, the portion of

          11    heating and cooling to total energy, which is

          12    what you're looking at is changing.  So in

          13    2030, the amount of energy attributable to

          14    heating and cooling is less than it was in

          15    the beginning of the forecast.  So you're

          16    going to get a smaller elasticity, and you

          17    can see that on your handout.

          18              DR. LADY:  I think it was the price

          19    elasticities, not the weather elasticities

          20    that they were talking about.

          21              DR. KOKKELENBERG:  Well, both

          22    elasticities do vary immensely.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            139


           1              DR. LADY:  Well, I'm sure there's a

           2    good reason.

           3              SPEAKER:  And it still applies.

           4              DR. LADY:  The weather elasticity

           5    -- I'm sure that you would know.  I didn't

           6    think that was an anomaly.  It was just I

           7    didn't know why that (off mike).  Yeah.

           8              SPEAKER:  No.  All right, and we

           9    did publish the forecast.  The evaluation

          10    just came out a couple of weeks ago?

          11              So we do that, and we do the

          12    tweaking and I think I've shared with you the

          13    table where I actually went with the

          14    residential model went back and changed like

          15    four or five key variables; put in the

          16    history that I knew; and the forecast -- and

          17    then the rest we would consider model error.

          18    So we got down to about one to three percent.

          19              DR. LADY:  Right.  I propose that

          20    what you did in that one case be generalized

          21    and done regularly.  Yeah.  You're right.

          22    That's --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            140


           1              SPEAKER:  Yeah.  I mean you got to

           2    understand that the models run.  It's not a

           3    regression model.  It's an NU structural

           4    model and so the uses that are affected by

           5    weather and price are different now than they

           6    are forecasted so the elasticities will

           7    definitely change.

           8              Refrigeration, if your electricity

           9    price goes up, there's nothing in the short

          10    run you're going to do to change your

          11    refrigerator energy consumption.

          12              However, heating you can:  You can

          13    change the thermostat, so the concept here I

          14    think is very, you know, for me, 'cause I do

          15    it, it's easy to understand, but there's no

          16    reason to think that the elasticity would not

          17    change over time, whether price or -- weather

          18    or price.

          19              DR. LADY:  I was just measuring it.

          20    I didn't --

          21              SPEAKER:  And then why the weather

          22    is different?  I'll address that real quick




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            141


           1    while I'm up here.  What I do is I use the

           2    last 30 historical years, so that's '76 to

           3    2005.  That's my normal.  The NOAA official

           4    normal is '71 to 2000, and then from 2000 on

           5    they used fixed population weights for the

           6    states.  I actually used the actual from

           7    Census that they have.  So that's why they

           8    don't mesh.

           9              DR. BLAIR:  Ed Blair.  I had some

          10    comments along the lines of what the other Ed

          11    just made, but I -- the main problem I have

          12    is I lost track somewhere along the way of

          13    just where we're going and what the end

          14    objective of the exercise is.  I did get the

          15    idea that our goal was to in effect try to

          16    partition the forecasting error and try to

          17    figure out where it was coming from.  But

          18    then for what purpose?  You know, simply to

          19    have that, and it seemed to me that in a

          20    sense where we were heading toward was the

          21    possibility of targeting model inputs for

          22    measure improvement potentially of saying,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            142


           1    Jeez, you know, we're consistently taking a

           2    beating here.  You know, this is a place

           3    where we really need to get better.

           4              Or potentially revising the model,

           5    and I don't know if this language will even

           6    make sense to better statisticians in the

           7    room, but that in a sense what happens is as

           8    we take our in sample information and make

           9    predictions, you know, well, we're always

          10    making predictions to an out of sample

          11    environment, you know, which the new year.

          12    And it turns out that a variable that

          13    performs very well, you know, on a backcast

          14    basis just, you know, it's sort of a mean

          15    square error kind of context.  It's - - you

          16    know, it's got a lot of variance associated

          17    with it and we, you know, we'd be better with

          18    a model specification that just doesn't rely

          19    so heavily on that variable or a different

          20    form of the model that is more robust and I'm

          21    not sure if I should be saying less sensitive

          22    or --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            143


           1              (Laughter.)

           2              DR. BLAIR:  -- or I'm not sure

           3    which word, you know, which word I should be

           4    using, but in a sense it seemed to me that

           5    that was the direction we were going.  In

           6    that sense, it was not clear to me that the

           7    screening, sensitivity, I'm not sure what

           8    word to use -- it wasn't sure to me that the,

           9    you know, sort of the experimental design of

          10    varying the parameters and seeing how the

          11    model responds to these -- I mean the only

          12    thing I could see that you were going to get

          13    out of that was that you would identify which

          14    variables of the model showed responsiveness

          15    to or the forecasts showed responsiveness to

          16    and when you sort of thought about which of

          17    those are frail, either because they're

          18    inherently high variance or because they're

          19    poorly measured, that you would say, whoa,

          20    you know, we've got a problem there.  Okay.

          21    But if we just on a sort of a historical

          22    basis go back and say where, you know, in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            144


           1    effect try to partition where did we get

           2    problems, that's sort of built in.  It's all

           3    built in.

           4              So I wasn't clear what the

           5    experimental design would do for us at the

           6    end.  I'm sorry if I've been confusing as

           7    I've wandered through that.

           8              DR. LADY:  My meaning of it was

           9    that -- and I would have to look at the

          10    literature that I've been advised to look at

          11    -- was that the design would create a dataset

          12    in essence that would allow an analysis of

          13    the sources of error differentially.  That's

          14    what I meant in essence.

          15              And that isn't.  I didn't present a

          16    resolved such design, and I've been told that

          17    I should go look and I will at the work

          18    that's already -- there's already a

          19    literature on this.  So that's what I meant,

          20    just at the -- an overview level.  A proper

          21    design so that the dataset that was gotten

          22    would serve the purpose that the archive




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            145


           1    model would have served, and yet it would be

           2    much cheaper and easier to accomplish.

           3              DR. NEERCHAL:  Jae.  Last one.

           4              DR. EDMONDS:  I was just going to

           5    go down that same, you know, path.  We've

           6    talked about some specific representations.

           7    My thought -- my question is sort of stepping

           8    back one step and what is -- you know, which

           9    of these variables have been chosen for

          10    examination so that you do a better job of

          11    representing them in the future and what was

          12    that process?  In other words, you're

          13    creating reduced, you know, you're creating a

          14    set of reduced form models to predict certain

          15    -- to -- for particular outputs of NEMS.

          16    That is, you're trying to be able to say what

          17    would the NEMS model have predicted if I'd

          18    run the whole thing and, you know, but I'm

          19    not -- I'm going to actually use a reduced

          20    form representation of it.

          21              So that's the problem that you --

          22    you know, you're addressing, which is how do




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            146


           1    I do that and, you know, how do I learn from

           2    that so that, in fact, I reflect back to the

           3    real NEMS information that helps that real

           4    NEMS do a better job.

           5              Well, the very first question I

           6    have is what are the variables that have been

           7    selected that you're trying to say, all

           8    right, NEMS has got a vector of outputs and

           9    there are N of them -- maybe N is eight;

          10    maybe N is 10; I don't know what it is -- how

          11    were those chosen and, you know, why were

          12    those chosen and, you know, what was the

          13    process by which this whole arg -- the whole

          14    design, you know, the whole project

          15    architecture was created?

          16              DR. LADY:  Well, they haven't been

          17    chosen at this point, and the specification I

          18    used, as was mentioned, is just a traditional

          19    one that an economist would come up with.

          20              My proposal is to talk to the

          21    modelers.  They know how it works, and to get

          22    their estimate as to the right set of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            147


           1    important parameters to keep track of and

           2    base the specification on their

           3    recommendations.  And that is the very thing

           4    that I hope and propose to start doing next

           5    week.

           6              In other words, they know.  I don't

           7    know.  I'm good at some things, but I -- but

           8    they know the model, and they know far better

           9    than I how to set up a proper regime of

          10    sensitivities.  Now, there may be some

          11    mathematical twists that I need to learn as

          12    to what to do with that, but they are the

          13    experts.

          14              DR. NEERCHAL:  Yes, just a quick

          15    question, more of a clarification.  Any model

          16    that produces an output is essentially

          17    producing a point estimate because of the

          18    parameters that are built into the model.

          19              So one of the issues to keep in

          20    mind is when you're talking is when you're

          21    talking about forecast accuracy, we've got a

          22    point estimate and we can (off mike) that




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            148


           1    point estimate with non- stochastic

           2    observation.  And I mean I would worry about

           3    the forecast accuracy of the model if it was

           4    systematically underpredicting or

           5    overpredicting.  But if it is randomly

           6    predicting differently, I don't know if we

           7    have a problem at all.

           8              DR. LADY:  From my own perspective,

           9    to have some protocol that assesses accuracy

          10    and understands in some sense the underlying

          11    reasons for it is proper science, whether it

          12    does the modelers any good or not.  In other

          13    words, as a credential for published

          14    information, I believe that that should be

          15    known, although I do believe that if this is

          16    done in the end that the diagnostic

          17    assistance that can come from it could be

          18    significant as well.

          19              But from an outsider's standpoint,

          20    to make these projections and to do it with

          21    quite a substantial analytical engine, you

          22    should know how it's working at the level of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            149


           1    what the model is trying to do simply as a

           2    credential for the information.

           3              DR. NEERCHAL:  I think we should

           4    wrap up - - oh, you know, well, one of --

           5    regardless of what the objective is, I think

           6    what Derek said I think -- because it is

           7    computer model.  It's a numerical

           8    deterministic model, there is a literature

           9    out there how statisticians can look at it

          10    and they actually go through these different

          11    scenarios, you know, what they are looking at

          12    and so on where the answer (off mike) is

          13    coming from.

          14              Sometimes the inputs are

          15    deterministic, some numbers that are pretty

          16    viable, and sometimes there is measurement

          17    error in the inputs themselves.  You know,

          18    they have to be treated differently in this

          19    context.

          20              So I think it would be really

          21    helpful to take a look at that literature I

          22    think.  So.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            150


           1              Please identify yourself just for

           2    the record.

           3              MR. SKELLY:  I'm Dan Skelly.  You

           4    know, we publish five main scenarios each

           5    year.  High macro.  Low macro.  High price.

           6    Low price.

           7              The impact analysis George

           8    presented, was for the reference case.  Now,

           9    if you did that impact analysis for the other

          10    cases, I can imagine you might get two types

          11    of results.  One it would be similar across

          12    all cases and maybe that would validate the

          13    approach.  But what if you got different

          14    impact analyses across these five scenarios?

          15    What would that tell you?

          16              DR. LADY:  Well, I would be alarmed

          17    if it was very different across the

          18    scenarios.  In other words, I would think

          19    that the approximation, if that's the right

          20    word to use, that's being used to parse the

          21    errors would not show great differences, so I

          22    would worry about the method if I found that.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            151


           1              MR. SKELLY:  So doing that would be

           2    a --

           3              DR. LADY:  Yep.

           4              MR. SKELLY:  Something you should

           5    do as part of this?

           6              DR. LADY:  Sure.

           7              DR. NEERCHAL:  Ed.

           8              DR. BLAIR:  Ed Blair.  I'm really

           9    hesitant to speak because I feel I'm better

          10    to remain silent than let people think you're

          11    a fool than open your mouth and (off mike).

          12              (Laughter.)

          13              DR. NEERCHAL:  That's not stopping

          14    us, by the way.

          15              (Laughter.)

          16              DR. BLAIR:  Going back to storing

          17    the model, going back to the idea as an

          18    alternative to storing the model, running

          19    conditions and retaining a record of those,

          20    it seems to me that if the -- how to put this

          21    -- that at one level, I don't need to know

          22    what the model was.  I just need to know what




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            152


           1    the forecast was and I can go back and I can

           2    look at differences between the inputs that

           3    were used in the model and the now known

           4    correct inputs.

           5              Now, I would appreciate that some

           6    of those inputs which may be in here are

           7    intermediate inputs, which is going to be a

           8    problem.  But I can go back and weather,

           9    strikes, you know, accidents, embargos, you

          10    know, changes in the laws.  I can go back and

          11    I can correct and I can in a sense run the

          12    correction against model error and that gives

          13    me a direct sense at the variable measurement

          14    level of which variables are killing me and

          15    then everything that rolls to the residual or

          16    the uncertainty is a candidate for

          17    specification error.  I don't know if this

          18    makes any sense to anybody else.

          19              But then this is a candidate for

          20    specification error and is the objective here

          21    to try to get a window into specification

          22    error, because it seems to me that where we




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            153


           1    were going was we were getting essentially

           2    variable-level results, which I could have

           3    gotten directly.

           4              So that's why I'm just not getting

           5    this.

           6              DR. HILL:  I was trying to raise a

           7    point like that, that these things are

           8    unknown ahead of time.  If there -- if the

           9    model is properly specified, the essence of

          10    the (off mike).  So it's just along a

          11    variation (off mike).

          12              DR. BLAIR:  So to me, I would not

          13    be running scenarios, well, then and I guess

          14    this is where I lost track of exactly what's

          15    happening here.  I would not be running

          16    scenarios of imagine a particular input being

          17    10 percent high or 10 percent low anything

          18    like that.  In a sense, I'd be running

          19    variations in the specification and then

          20    being able to go back and see how the

          21    variation in specification related to the

          22    ultimate error.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            154


           1              DR. LADY:  In my opinion, this does

           2    not necessarily imply the project itself that

           3    there's anything wrong with NEMS.

           4              DR. BLAIR:  No, it doesn't.

           5              DR. LADY:  Or that it's

           6    misspecified at all.  In fact, it could be

           7    perfect.  I just think that when the time

           8    comes you should be able to report why the

           9    projection was different than the event and

          10    to do it in a way which conforms to the

          11    economic logic of NEMS and can speak to the

          12    supply and demand interactions that NEMS is

          13    trying to project.  There doesn't have to be

          14    anything wrong with NEMS specification at

          15    all.  If the price was reasonably assumed in

          16    the year 2005 to be 24 percent less than it

          17    turned out to be five years later, which is

          18    not anybody's fault, you should have a way of

          19    seeing the degree to which this would explain

          20    the fact that the forecast that was made was

          21    higher than it should have been, because it

          22    didn't take into account that higher price.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            155


           1              Now I believe that there's a lot of

           2    learning by doing that this project will give

           3    rise to and I think that from the modeler's

           4    standpoint if this is done right, that it can

           5    be of help to things like specification.  But

           6    there is no presumption in my mind that there

           7    is such a problem.  In other words, there's

           8    nothing wrong -- as far as I'm concerned NEMS

           9    is one of the most sophisticated models

          10    anywhere outside of defense anyway that I

          11    know anything about, and I want a credential

          12    for its results rather than a criticism of

          13    its nature.

          14              DR. NEERCHAL:  Walter?

          15              DR. HILL:  Well, this could be and

          16    since it's a verification of the model is

          17    correct if you --

          18              DR. FORSYTH:  Right.  So it's just

          19    random.

          20              DR. HILL:  Yeah.  That there's

          21    random variation.  You won't be able to do

          22    any better, because you can't make (off




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            156


           1    mike).  You don't know ahead of time what the

           2    value of some of these variables are.

           3              DR. LADY:  You can't predict the

           4    weather.

           5              DR. HILL:  You don't know how many

           6    heating degree days will be in the summer,

           7    this summer.  And who (off mike) this summer.

           8              DR. NEERCHAL:  Yes.  I think so

           9    we'll go.  Thank you very much and thank you

          10    for the discussants and all the comments, and

          11    so let's -- next time we have in the agenda

          12    here Committee suggestions for topics at the

          13    fall 2007 meeting.  I think, you know what?

          14    Spending a few minutes.  Email to Nancy.

          15    (off mike).  So you don't have to think that

          16    this is where you have to come up with a

          17    topic right off the top of your head.

          18              DR. KIRKENDALL:  The other thing

          19    just to remind you last -- yesterday, Guy

          20    talked about the 30th anniversary of EIA,

          21    which is also the anniversary of the

          22    Committee, and if you have any ideas for what




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            157


           1    we could do that would involve the Committee.

           2              DR. KOKKELENBERG:  Oh, for the 30th

           3    anniversary?

           4              DR. KIRKENDALL:  Yeah, for the 30th

           5    anniversary.  John Paul is the Chair of the

           6    History Committee and he's going to be taking

           7    an active role in planning things, and so he

           8    was particularly interested in any ideas you

           9    might have, so I thought I'd bring it up

          10    again in case you've forgotten that (off

          11    mike) that you were invited.

          12              DR. KOKKELENBERG:  Can we have one

          13    conversation?

          14              DR. NEERCHAL:  Yeah.  I think --

          15              DR. WEINIG:  George, Derek.  Would

          16    you take that off mike so that our

          17    transcription --

          18              DR. KHANNA:  Well, one thing I can

          19    (off mike) I didn't think about is somehow

          20    incorporate the celebration, it would be

          21    great at least for the Committee members

          22    because you know the history of the Committee




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            158


           1    -- what exactly -looking back 30 years, what

           2    are the great things this Committee has done

           3    here?  Why do you insist on having us around?

           4              DR. NEERCHAL:  We are very

           5    expensive?

           6              DR. EDMONDS:  I certainly want to

           7    ask that question.

           8              DR. NEERCHAL:  I think another

           9    question, a related question will be that is

          10    it possible to have the celebration around

          11    the same time as the Committee meeting so

          12    that it reduces travel.

          13              DR. KIRKENDALL:  So you could

          14    participate more broadly --

          15              DR. NEERCHAL:  Right.

          16              DR. KIRKENDALL:  If you wanted to?

          17              DR. NEERCHAL:  Exactly.  That may

          18    be a -- I mean for me, it's okay.  I have a

          19    short travel (off mike).  In terms of topics,

          20    one thing I thought, you know, I (off mike)

          21    the most exciting thing I learned in this

          22    meeting is fellows and the graduate students




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            159


           1    (off mike) that you are thinking,

           2    undoubtedly, the most exciting thing I think.

           3    As a (off mike) I think there's no doubt

           4    about it in my mind it's the most exciting

           5    thing and the most important thing you can do

           6    is to train the workers.

           7              So I think that could be a kind of

           8    a regular thing in your agenda, make a place

           9    for it, instead of people, when you have

          10    identified those people, if they're working

          11    here, make sure that they have some

          12    presentation time, and because they are

          13    students, don't put them in the exact same

          14    format as everybody else.  You know, make it

          15    a little more friendly looking, because it

          16    almost -- this is very much like a Ph.D.

          17    defense.

          18              DR. KIRKENDALL:  Yeah.

          19              DR. NEERCHAL:  And (off mike) I

          20    don't know that that's good or not.  It could

          21    be good also, you know.  You know, I put my

          22    kids through that weekly.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            160


           1              DR. KIRKENDALL:  Oh, no, your poor

           2    kids.

           3              DR. NEERCHAL:  We have a graduate

           4    student seminar where the student presents

           5    and then you have the faculty criticizing

           6    their presentation right in front of

           7    everybody else.  Every week, there are two

           8    (off mike) for this torture, and our kids are

           9    very good because of this.  It has been going

          10    on for about 10 years, and I think that will

          11    be nice.

          12              And also I think before you go on

          13    ahead, I'm sure, you know, this might come to

          14    you with a specific plan.  I think this (off

          15    mike) good place to really (off mike).  Many

          16    of us actually have thought about this early.

          17              DR. KOKKELENBERG:  I was going to

          18    add John Paul might be able to get a hold of

          19    (off mike) of the names of all the past

          20    Committee members and the chairs and so

          21    forth.  I know you can't afford to invite

          22    them all here, and they won't all come here




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            161


           1    even if you invited them, but you might

           2    consider at least having some sort of

           3    biograph (off mike) since it's been in

           4    existence for 30 years and you're probably

           5    talking upwards of 500 people, because, you

           6    know --

           7              DR. WEINIG:  It's a six-year span.

           8              DR. KOKKELENBERG:  Well, some of

           9    them are only three-year span.  Some of them

          10    are (off mike).

          11              Maybe -- okay 300 so far.

          12              DR. KIRKENDALL:  Still, it's a

          13    reasonable number by this time.

          14              DR. KOKKELENBERG:  But also it

          15    might be useful to build on what has been

          16    said already.  Nagaraj may have been going

          17    there, and I may have missed this, but if the

          18    Committee has some participation or has a

          19    meeting around that time, it might be useful

          20    to allow the Committee to invite -- and I'm

          21    not going to talk about out of who's pocket

          22    book this goes out of -- maybe one of their




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            162


           1    grad students that, if appropriate, could

           2    come to the meeting to see what goes on a

           3    these kinds of meetings.  It might help you

           4    for this research fellowship or whatever

           5    we're going to be calling it, and it may also

           6    -- I don't think we'd invite so many that

           7    there wouldn't be any room for the staff to

           8    come to the meeting.  But, because if you

           9    send about -- well, you could invite one if

          10    you thought it was appropriate and why it's

          11    appropriate.  But I'm not sure that that's

          12    the best idea, but it's an idea.

          13              DR. FORSYTH:  This is different

          14    (off mike), but actually I liked Moshe's idea

          15    earlier this morning about publishing how you

          16    structure your decision about or your

          17    approach to, if I understood correctly, your

          18    approach to making decisions about how to

          19    release data, and I was thinking of it as

          20    kind of like the cost-benefit structure that

          21    you work through.  And if you do something

          22    like that, then I'd love to hear about it --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            163


           1    like that; further structure (off mike).

           2              DR. NEERCHAL:  And the one place,

           3    easy place for it to be published, there's a

           4    section on Garman statistics of (off mike)

           5    thing.  You have the electronic kind of (off

           6    mike) kind of, and they actually publish, you

           7    know, at the bottom of that electronic news

           8    listserv, you know, they put a -- you know

           9    put some slightly technical information at

          10    the end, and you (off mike) like a couple of

          11    pages how you did it, that will be very

          12    useful for a lot of people and (off mike).

          13              DR. FORSYTH:  The factors to

          14    consider.

          15              DR. NEERCHAL:  I can give you the

          16    person who is actually -- who sends out that

          17    (off mike).

          18              DR. FORSYTH:  Okay.

          19              DR. HILL:  I mentioned an offhand

          20    comment that it might be of interest to look

          21    at the effect of the weather.  I don't know

          22    if that's -- we should think about it a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            164


           1    little bit more whether or not that's worth

           2    pursuing.  So maybe, maybe.

           3              DR. NEERCHAL:  I think it's my time

           4    for inviting the public for comments.

           5              DR. KOKKELENBERG:  Could?  I --

           6              DR. NEERCHAL:  Oh, sorry.

           7              DR. KOKKELENBERG:  I guess I got

           8    confused.  I thought we were discussing the

           9    30th anniversary and now you want topics?

          10              DR. KIRKENDALL:  Yeah.  I guess for

          11    future --

          12              DR. KOKKELENBERG:  Okay.

          13              DR. KIRKENDALL:  -- yeah, it was

          14    sort of both.  I was --

          15              DR. KOKKELENBERG:  Transmission

          16    reliability data issues.  I don't know if you

          17    have data on transmission, but electricity

          18    transmission and pipeline transmission

          19    reliability issues I think are -- we had that

          20    blackout.  What two years ago?  And there was

          21    a huge study on why we had that blackout.

          22              The main thing they pointed up was




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            165


           1    that the reliability responsibility has been

           2    smeared out all over the place and we talked

           3    about it this time when we talked with -- we

           4    just alluded to it, when we talked about the

           5    electricity transmission problems as being a

           6    constraint on what kind of plants come in and

           7    out.

           8              So it seems to me that perhaps

           9    transmission, which is not a regulated

          10    industry (off mike) as far as I understand

          11    it, is something that maybe we should be

          12    paying some attention to or the EIA should be

          13    paying attention to.

          14              Another topic I had is that when

          15    you collect solar energy data here, all we

          16    collect, it was my understanding, is the

          17    shipment of solar panels.  We don't have any

          18    idea of how much electricity is generated by

          19    solar devices.

          20              DR. KIRKENDALL:  We have some idea.

          21              DR. KOKKELENBERG:  Okay.  I didn't

          22    -- I wasn't aware of that.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            166


           1              DR. KIRKENDALL:  Yeah.  But that

           2    has to be from big solar establishments.  We

           3    certainly don't have data on the amount

           4    generated by home solar panels.

           5              DR. KOKKELENBERG:  Or even

           6    industrial.

           7              DR. KIRKENDALL:  Or -- well, I

           8    don't know whether (off mike).

           9              DR. KOKKELENBERG:  Or commercial?

          10              DR. KIRKENDALL:  Yeah, I don't know

          11    whether RECS, MECS, CBECS have any of that or

          12    not.

          13              DR. KOKKELENBERG:  Yeah.  Well, it

          14    may be an (off mike) topic because --

          15              DR. KIRKENDALL:  Okay.

          16              DR. KOKKELENBERG:  -- this is one

          17    of the things that is really touted and the

          18    same for wind generation, although I think we

          19    have much more on wind generation.

          20              Forced outage modeling.  I was

          21    surprised to hear Phillip say that the forced

          22    outages were such a high percentage that he




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            167


           1    had to reduced capacities essentially by as

           2    much as seven or eight percent just from

           3    forced outages.  That works out to be about

           4    three weeks out of the year for a plant, and

           5    that's a lot of time to be down on a non-

           6    scheduled I would assume he meant by forced

           7    outages.

           8              And I just want to underline what

           9    Walter said.  My work that I've done on a

          10    variety of things suggests that things like

          11    Katrina really didn't have a whole heck of a

          12    lot of effect on the economy as a whole.  It

          13    certainly did on the South economy.  It

          14    certainly had an effect on the energy prices

          15    for a while, but on as far as the economy as

          16    a whole, it didn't have a whole heck of a lot

          17    of impact.  And so when you get to the NEMS

          18    modeling system, it's got to take, you know,

          19    maybe a global rise in temperature or a

          20    global winter for a year before it's going to

          21    have the impact that is felt in the GDP

          22    estimates, and, hence, the energy (off mike).




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            168


           1    And so the sensitivity to the weather of the

           2    NEMS, it's high -- Neha and I were just

           3    discussing that -- it's high when it comes

           4    to, say, fuel prices and natural gas prices,

           5    but it may not be high in the sense that it

           6    feeds back through the GDP as much as

           7    sometimes we would think it might.  And so

           8    the incomplete NEMS modeling sensitivity to

           9    modest changes in weather might be revealing.

          10    And you probably want (off mike).  But would

          11    that be useful to discuss that or not?

          12    Where's Dan when you want him; right?

          13              MS. BOEDECKER:  We're off?  We're

          14    off?

          15              COURT REPORTER:  No, you're on.

          16              MS. BOEDECKER:  I'm on?  Okay.  We

          17    do run scenarios where we do weather

          18    sensitivities. In fact two AEOs ago, I think

          19    there was a piece in the Issues and Focus

          20    about it.  Excuse me.

          21              But in the residential and

          22    commercial modules, when we have weather




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            169


           1    affecting our fuel use, we have it only

           2    affecting the heating and cooling fuel use

           3    and not the other uses.  And because the

           4    heating and cooling are becoming a smaller

           5    percentage of the energy use, you're not

           6    going to have as great an effect as you would

           7    for, say, a price change, which affects all

           8    fuel uses.

           9              DR. KOKKELENBERG:  So maybe that

          10    isn't' a topic that's worth looking at?

          11              MS. BOEDECKER:  It's been looked at

          12    --

          13              DR. KOKKELENBERG:  Yeah, that's why

          14    I'm saying this.  That's why --

          15              MS. BOEDECKER:  But it's up to the

          16    Committee I guess as far whether they want to

          17    see something on it, but there is something

          18    published on it already.

          19              DR. NEERCHAL:  Thanks.  I mean it

          20    is something -- if you think of something,

          21    send an e- mail to Nancy or Bill.  It's an

          22    ongoing discussion.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            170


           1              So now, it's my pleasure to invite

           2    the public for their comments.  Last chance.

           3    I think -- and it doesn't have to be this

           4    presentation.  It could be the previous

           5    presentation or any of the (off mike) day and

           6    a half.  Please identify yourself for the

           7    transcription.

           8              MS BATTLES:  Okay.  Stephanie

           9    Battles.  I'm the Division Director for the

          10    Energy Consumption Division.  We're the home

          11    of the RECS and the CBECS.

          12              I just have a few comments.  As far

          13    as the NISS it's quite intriguing, because I

          14    do have a student right now from Stafford who

          15    would love to get to historic data, because

          16    we do not have very much on the Internet at

          17    the moment.

          18              What we are doing, though, is I do

          19    have one staff.  Right now, the staff is like

          20    half the size of what it was 10 years ago,

          21    but I do have one who is working on the 1990

          22    RECS microdata, but this is public data,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            171


           1    putting it up on the Internet.

           2              And my goal is to get every last

           3    microdata set for the RECS and CBECS up on

           4    the Internet, but it does take time and it

           5    takes staff.

           6              We find that the public data is

           7    used a lot, a lot by researchers, especially

           8    students and what have you.  We take very few

           9    variables off of the public data.  I mean

          10    like for the RECS we take state identifiers

          11    off.  The main reason we do that is because

          12    the sample size is so small.  CBECS we've had

          13    to, because we have problems with large

          14    buildings, we've had to top code the number

          15    of stories and the floor space because big

          16    buildings you can point them out.  You know

          17    everybody knows the Empire State Building as

          18    an example. I'm not saying it's in our

          19    sample.  But that's just an example there.

          20              And so that is -- to me, it's low

          21    hanging fruit to get as much data out as

          22    possible.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            172


           1              Another thing that we've done we're

           2    going to be getting it up on the Net in I

           3    think within a month -- I've had every last

           4    report that we've ever done scanned into PDFs

           5    and so this is all, but this is aggregated

           6    data.  All the tables going back like the

           7    RECs to '78 and any other reports, you know,

           8    that we've done, and we're getting this up on

           9    the Internet.

          10              So we've done things that we can;

          11    we can afford to do.  Yes, we do want to get

          12    the researchers to get the unmasked data.  We

          13    get very few requests for it, but they are

          14    available.  But a lot of times they're

          15    requests because they want one or two

          16    variables.  You know, they want -- they want

          17    that -- we had to take some large buildings

          18    off in the CBECS.  They want those large

          19    buildings because of sample size and things

          20    of that nature; like the number of stories we

          21    had to do.  So they do want that data for the

          22    monthly building data that we get, but I say




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            173


           1    it does take a lot of time and money to do

           2    that.

           3              So, anyway, you know, we're just

           4    trying to do what we can do with the

           5    resources we have at the moment.  And just as

           6    an aside, we know -- we do the MECS also.

           7    And we've never had the opportunity to go to

           8    the (off mike) of the RDC at the Census,

           9    because basically we can't afford it, so

          10    we've never -- we don't have access to our

          11    historic MECS data basically unless we have

          12    the funds.

          13              Thank you very much for the

          14    opportunity.

          15              DR. NEERCHAL:  Thank you.  I mean

          16    just include it in public if you want to make

          17    a comment.

          18              MS. BOEDECKER:  This is Erin

          19    Boedecker again.  Commercial module and NEMS.

          20              I did want to make one slight

          21    clarification.  We do archive the model every

          22    year, the Annual Energy Outlook version of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            174


           1    the model.

           2              But it is an evolving model when

           3    over time there are changes made and we just

           4    haven't -- don't have the resources for one

           5    thing to go back and make changes to input

           6    what the actuals were instead of what we

           7    assumed at the time I think, because some

           8    models are more modular than others, and in

           9    some cases, you're changing the actual code

          10    instead of just inputs.  So I believe it

          11    would be very resource intensive, even if you

          12    just chose a limited number of variables.

          13    But we do have archives of the AEO version of

          14    the NEMS module every year.

          15              DR. KOKKELENBERG:  Can I say -- I

          16    don't think you should be changing it.  I

          17    think if you made a forecast based on the

          18    best knowledge you had at that time, the best

          19    specification you had at that time, then

          20    that's the forecast.  Revisiting that and

          21    saying, well, really temperatures were higher

          22    or a (off mike) changed, you really don't




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            175


           1    serve much of a useful purpose unless you're

           2    trying to identify why there was an error in

           3    the model.  And that should be left to the

           4    individual investigator rather than

           5    rearchiving the model with those changes.

           6              MS. BOEDECKER:  Okay.  Thank you.

           7    And to some extent, we do do that, as John

           8    Cymbalsky mentioned earlier.

           9              DR. FEDER:  But, Ed, if you want to

          10    improve the model, you want to know where the

          11    error is.

          12              DR. KOKKELENBERG:  Oh, yes.  Yes,

          13    yes.  That's right.  I really forecast that,

          14    but this is what I really (off mike) forecast

          15    that you're kind of (off mike).

          16              DR. NEERCHAL:  I don't know.  Do

          17    you have any (off mike) on it?

          18              With that, I think we are ready for

          19    adjourning to -- adjourning this meeting and

          20    go to lunch I guess.  So lunch is like

          21    yesterday, downstairs the same room.

          22              DR. WEINIG:  Yes.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382





                                                            176


           1              DR. FEDER:  Could you remind us

           2    which room?  Which floor?

           3              DR. NEERCHAL:  2E-081.  081.  It's

           4    the same corridor, second floor.

           5                   (Whereupon, at 11:45 a.m., the

           6                   PROCEEDINGS were adjourned.)

           7                       *  *  *  *  *

           8

           9

          10

          11

          12

          13

          14

          15

          16

          17

          18

          19

          20

          21

          22




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382