**[Measuring the Quality of EIA Analysis: A Revised
Approach,]{.underline}** Bill Weinig, SMG, EIA This revised survey
approach was developed by a team of Stan Freedman, Team Leader, Howard
Bradsher-Fredrick, Tom Broene, Inderjit Kundra, Herb Miller, Renee
Miller, Joseph Sedransk, Phillip Tseng and Bill Weinig, SMG, EIA.

**1. Background:**

The first of three goals in EIA's Strategic Plan for 2004-2008 states
that, "**EIA\'s information program is relevant, reliable and consistent
with changing industry markets and EIA's information products will
retain or improve their high quality and timeliness.**

EIA has been conducting surveys and employing other methodologies to
assess the quality, timeliness and other attributes of EIA products for
a number of years. However, a systematic quantitative assessment of EIA
analytical products had not been conducted. In a paper presented to the
Committee last fall, we proposed to survey a limited number of subject
matter experts who review EIA draft products. These we would contact
through the Independent Expert Review program and 2 reviewers under
contract to the Statistics and Methods Group. Seven questions with
definitions of terms were designed to solicit responses from these
reviewers. At the fall 2003 meeting, we shared this approach and the
proposed questions, and asked the Committee for advice. [^1]

The Committee response was that the survey questions were comprehensive
and balanced. However, the questions were fairly complex and open to
some range of interpretation. EIA's use of definitions in the proposed
questionnaire suggested this too. The small sample size of outside
reviewers could lead to fairly unstable results. While the survey form
supported tabular results, the results were not likely to provide rich
information. If this limited survey approach was important, the
Committee suggested EIA consider open-ended questions, and a qualitative
evaluation. Independent reviewers might be good at qualitative
evaluations, and might be surveyed through Independent Expert Review
contracts at intervals after their reviews. The committee was mixed on
the value of keeping the reviews anonymous, and was generally not
supportive of the survey in its current methodology or form.[^2]

**2. EIA reactions to the ASA Committee's advice:**

**2. A. A Summary of EIA's Response to Advice at the Fall 2003,
Meeting**

Since the fall 2003, meeting, we have:

\(1\) replaced the original 7 questions with 17 more narrowly focused
questions and 2 open-ended questions, 1 at the end of each survey
section,

\(2\) eliminated the imbedded definitions in the original questions, and

\(3\) otherwise maintained the scope of the original survey.

Also, we reviewed the previous survey methodology, and replaced the
limited number of intended survey recipients (approximately 10 reviewers
of EIA products) with a much larger group of EIA's analytical customers
(those registering to attend the spring 2004 National Energy Modeling
System Annual Energy Outlook Conference in March, 2004).

**2. B. EIA's Detailed Responses to ASA Advice**

This paper is about the revised survey methodology. (The revised
questions may be found in Attachment 1.)

Responding to the Committee's advice to broaden the survey, the team
chose to survey users of EIA's two main analytical publications, the
"Annual Energy Outlook" and the "International Energy Outlook." These 2
publications are considered to represent EIA's broader range of
analytical products, and recognized and used by the widest audience of
EIA's analytical customers.

(Other important analytical products, such as EIA's Service Reports,
typically respond to Congressional requests to study proposed
legislation. These studies typically compare proposed legislation to
EIA's base case in the Annual Energy Outlook (AEO). This audience is
narrow, difficult to reach and survey, and is thought to be closely akin
to EIA's other analytical customers anyway.)

The AEO is based on forecasts from the National Energy Modeling System,
EIA's mid-term analytical model, forecasting to 2025. The AEO and the
International Energy Outlook are discussed at an annual 1-day conference
and agency sponsored. The conference, called the National Energy
Modeling System/ Annual Energy Outlook, was held in Washington, D.C. on
March 23, 2004.

Based on past experience, EIA expected approximately 500 people at the
conference. To register, one was asked to provide name, title,
organization, address, phone, fax and email. The conference was free.

Registrant names, email addresses and phone numbers were provided to the
Survey Team electronically by EIA's Office of Integrated Analysis and
Forecasting, who managed the conference.

So an effort was made to hold the survey mailing list cut-off as long as
possible. This (1) minimized those potentially "different" participants
from the main group, and (2) kept the survey population as close as
possible to the total expected conference attendance.

Survey names, email addresses and telephone numbers were transferred to
the team electronically. This allowed last minute transfer, insured the
accuracy of the information and minimized the time needed for the
information transfer.

The electronic survey mailing and the survey instrument were pre-tested
March 1, to approximately 12 EIA staff to see if the survey could be
received, if questionnaires could be completed, if the responses could
be returned and if responses could be read. On March 2, the pre-test was
evaluated and judged successful. No revisions were needed.

On March 4, the survey was sent to 397 conference registrants received
as of March 1, 2004. There were 42 responses in 3 days for a 10.6
percent response rate. After several days, the list was purged of
respondents and a second emailing went to the reduced list. The results
of these and a third wave of mailings may be seen in Table 1 below.

Table 1: Respondents to March EIA Analytical Survey Mailings

  ----------- --------------- -------------- -------------- --------------
  Mailing:    Number of       Percent        Cumulative     Cumulative
              Response        Response       Respondents    Percent
                                                            Response

  March 4     42              10.6           42             10.6

  March 9     39              9.8            81             20.4

  March 11    47              11.8           128            32.2
  ----------- --------------- -------------- -------------- --------------

Answers from the first round of respondents were compared to those from
the rest of the respondents and no significant differences were found.

We had considered sampling at the conference. But this was considered
very awkward. The logistics associated with finding and surveying
specific individuals in a meeting of 500 individuals without a central
registration desk or a location to pick up materials (done in previous
years) led us to believe there was no effective way to contact a sample
audience. Further, the schedule was very compressed leading participants
to have limited time to complete surveys. The team decided that no
further sampling should be conducted at or following the AEO/IEO/NEMS
Conference.

Some team members thought it would be worthwhile to attempt one
follow-up using an alternative mode. Thus, on March 19 and 22, a sample
of approximately every 6^th^ non-respondent was selected and their names
and phone numbers were given to two SMG contract employees to do
follow-up calls. The sample selection and results to the follow-up calls
follow in Table 2:

Table 2: Post Email Sample Selection and Survey Results

3.  **The Survey Results and Findings:**

The survey results may be found in **Attachment 2: Table of Results:
Abbreviated Questions and Descriptive Statistics**. Highlighting these
results, the survey found that the respondents evaluated EIA's
analytical products as follows:

1.  132 Respondents answering the survey (email and by phone)

2.  19 stated they were not familiar with the AEO and the IEO

3.  Therefore, 113 respondents provided at least some information.

4.  On a 5-point Likert scale, averages varied from a low of 3.37 on the
    AEO ("The methodology is suitable to the analysis Petroleum
    Refining.") to a high of 4.31 ("The AEO is relevant."). This range
    of answers is indicative of products that are generally well liked
    and acceptable to the evaluators.

5.  Survey responses by mailing:

-   1^st^ mailing: 42 respondents

-   2^nd^ mailing: 39 respondents

-   3^rd^ mailing: 47 respondents

-   4^th^ attempt (phone calls): 4 respondents

6.  Comparison of responses to all questions of those submitting answers
    after 1^st^ mailing & before 2^nd^ mailing to those submitting
    answers following the 2^nd^ mailing:

-   No significant differences (at 0.05 significance level)

-   This provides some evidence that non-respondents are not different
    from respondents.

7.  Comparison of experts[^3] (n = 44) with non-experts for the AEO:

-   One question was significant at the .05 level: Q1. "The AEO is used
    extensively in my work."

-   Averages: 4.11 experts,

-   3.70 non-experts.

-   This should be expected.

8.  Comparison of experts[^4] with non-experts for the IEO. (13 experts
    and 19 non- experts). Questions resulting in significant differences
    are the following:

-   Question #1: "The IEO is used extensively in my work."

    a.  Averages: Expert: 3.82

    b.  Non-Expert: 2.89

    c.  Difference significant at the 0.10 level

-   Question #2: "The IEO is clearly written."

    a.  Averages: Expert: 4.33

    b.  Non-Expert: 3.75

    c.  Difference significant at the 0.01 level

    d.  

**4. Two Questions to the Committee:**

1.  Do EIA's responses appear adequate to meet suggestions and
    criticisms at the fall 2003 meeting?

2.  If this survey is needed every year, is there a way to avoid
    overworking our respondents?

**Attachment 1:**

OMB No. 1901-0302

The Energy Information Administration is conducting an e-mail survey of
customers who registered for our upcoming NEMS Conference.  This short
survey will take about 5 - 10 minutes to complete.

 

The purpose of the survey is to determine the level of satisfaction with
two of EIA\'s major products:  The *Annual Energy Outlook* and the
*International Energy Outlook*.  We are interested in your feedback.

 

To take this voluntary survey now, please **click the reply button** to
this e-mail and add your responses to the questions below.  Your
participation is voluntary and your responses are confidential. 

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

EIA PRODUCT SURVEY

> ** **

**Are you familiar with the *Annual Energy Outlook*?**

**If Yes, continue at A.  *Annual Energy Outlook*.**

**If No, go to B, *International Energy Outlook.    ***

*** ***

**If unfamiliar with either publication, stop here.  Please** **click
the reply button and send your e-mail reply to EIA.**

**Thank you for your time.    **

> ** **

 

A.  ***Annual Energy Outlook*** 

> Because analyses by the Energy Information Administration (EIA) are
> required to be policy-neutral, the projections in the *Annual Energy
> Outlook 2004* (*AEO2004*) are based on Federal and State laws and
> regulations in effect on September 1, 2003. The potential impacts of
> pending or proposed legislation, regulations, and standards---or of
> sections of legislation that have been enacted but that require funds
> or implementing regulations that have not been provided or
> specified---are not reflected in the projections.

Use a scale from 1 to 5, with 5 meaning that you strongly agree with the
statement; 4 meaning you agree; 3 meaning you are neutral, 2 meaning you
disagree; 1 meaning you strongly disagree with the statement. If the
answer is "Don't Know", please answer using a 6. 

 

**Strongly                                    Strongly**

**[Disagree           Neutral           
Agree\_\_\_\_\_\_\_\_\_\_]{.underline}**

      1          2          3          4            5          6 (Don't
Know)

 

1.  The *AEO* is **[used extensively in my work.]{.underline}**

    Enter number here:  

    (If your answer is 1 or 2, please tell us why.)

 

2.  The *AEO* is **[clearly written]{.underline}**.

     Enter number here:

>  (If your answer is 1 or 2, please tell us why.)

3.  The *AEO* is **[relevant.]{.underline}**

     Enter number here:

(If your answer is 1 or 2, please tell us why.) 

 

4.  Do you use **[other sources]{.underline}** with information similar
to the *AEO? (Please check*.)

\_\_Yes

\_\_ No

If yes, please list one or more of those sources.

 

**If you are familiar with the modeling details behind the *AEO*, then
please continue with questions 5 thru 9.  If not, please provide the
name and e-mail address of someone in your organization who is
knowledgeable about the *AEO* and go on to Section B on the *IEO*.**

 

Name: 

E-mail Address: 

 

5\. The *AEO* **incorporates the changing industry structure.**

**    ** Enter number here:

(If your answer is 1 or 2, please tell us why.)  

 

6.  **[Appropriate assumptions]{.underline}** (e.g. GDP, world oil
prices, environment) are used in the *AEO*.

     Enter number here:

(If your answer is 1 or 2, please tell us why.)  

 

7.   **Methodology used in the *AEO*** *- Are* you familiar with the
NEMS model structure?

**      If the answer is "Yes" continue,**

**      If "No", go to the next question.**

 

Use a scale from 1 to 5, with 1 = Strongly Disagree, 5 = Strongly Agree;
Answer using 6 if you Don't Know.

> **Strongly                                    Strongly**
>
> **[Disagree           Neutral           
> Agree\_\_\_\_\_\_\_\_\_\_]{.underline}**
>
>       1          2          3          4            5          6
> (Don't Know)

 

The methodology is suitable to the analysis of the following:

>  

**[(7a.) End-Use Sectors ]{.underline}**

> (i)Residential Sector

-   Enter number here:

> (ii)Commercial Sector

-   Enter number here:

> (iii)Industrial Sector

-   Enter number here

> (iv)Transportation  

-   Enter number here

    -   (If your answer to i -- iv is 1 or 2, please tell us why.)

**[(7b.) Supply]{.underline}**

> (i)Petroleum

-   Enter number here:

> (ii)Natural Gas

-   Enter number here:

> \(iii\) Coal

-   Enter number here:

> \(iv\) Renewables

-   Enter number here:

\(v\) Electric Power Generation

-   Enter number here:

    -   (If your answer to i -- v is 1 or 2, please tell us why.)

**[(7c.) Petroleum Refining]{.underline}**

-   Enter number here:

    -   (If your answer is 1 or 2, please tell us why.)

8\. The **[conclusions]{.underline}** reached in the report are
**[supported]{.underline}** by the analysis in the *AEO* and its
companion report, \"Assumptions to the AEO\".

     Enter number here:

(If your answer is 1 or 2, please tell us why.)

 

9 The information in the *AEO* is of **[high quality]{.underline}**.

     Enter number here:

(If your answer is 1 or 2, please tell us why.)

.

 

 

Is there anything else you would like to tell us about the A*EO*?

 

 

 

 

 CONTINUE WITH *IEO* -- IF FAMILIAR

B.  *International Energy Outlook*

> Because analyses by the Energy Information Administration (EIA) are
> required to be policy-neutral, the projections in this *International
> Energy Outlook 2004* (I*EO2004*) are based on Federal and State laws
> and regulations in effect on September 1, 2003. The potential impacts
> of pending or proposed legislation, regulations, and standards---or of
> sections of legislation that have been enacted but that require funds
> or implementing regulations that have not been provided or
> specified---are not reflected in the projections.

Use a scale from 1 to 5, with 5 meaning that you strongly agree with the
statement; 4 meaning you agree; 3 meaning you are neutral, 2 meaning you
disagree; and 1 meaning you strongly disagree with the statement. If the
answer is "Don't Know", please answer using a 6. 

**Strongly                                    Strongly**

**[Disagree           Neutral           
Agree\_\_\_\_\_\_\_\_\_\_]{.underline}**

      1          2          3          4            5        6 (Don't
Know)

  

1.  The *IEO* is **extensively used** in my work**.**

     Enter number here:

> (If your answer is 1 or 2, please tell us why.)

 

2.  The *IEO* is **[clearly written]{.underline}**.

     Enter number here:

(If your answer is 1 or 2, please tell us why.)  

 

3.  The *IEO* is **[relevant.]{.underline}**

     Enter number here:

> (If your answer is 1 or 2, please tell us why.)

 

4.  Do you use **[other sources]{.underline}** with information similar
to the *IEO*?   (Please check)

\_\_Yes

\_\_ No

If yes, please list one or more of those sources.

**If you are familiar with the modeling details behind the *IEO*, then
please continue with questions 5 thru 8.  If not, please provide the
name and e-mail address of someone in your organization who is
knowledgeable about the *IEO*.**

 

Name:

E-mail Address:

 

**Thank you for your participation in this study.**

 

 

5\. The *IEO* **incorporates the changing industry structure.**

     Enter number here:

(If your answer is 1 or 2, please tell us why.)

 

6.  **[Appropriate assumptions]{.underline}** (e.g. GDP, world oil
prices, environment) are used in the *IEO*.

     Enter number here:

(If your answer is 1 or 2, please tell us why.)

 

7\. The **[conclusions]{.underline}** reached in the report are
**[supported]{.underline}** by the analysis in the *IEO*.

     Enter number here:

(If your answer is 1 or 2, please tell us why.)

 

8\. The information in the *IEO* is of **[high quality]{.underline}**.

     Enter number here:

(If your answer is 1 or 2, please tell us why.)

 

Is there anything else you would like to tell us about the *IEO?*

 

**"Thanks for responding."**

**Attachment 2: Survey Results**

Table of Results: Abbreviated Questions and Descriptive Statistics

  --------------------------------------- ---------- ---------- ----------
  Abbreviated Questions                   Average    \% 1s&2s   %4s&5s

  AEO Section:                                                  

  1\. The AEO is used extensively in my   3.85       12.5%      68.8%
  work.                                                         

  2\. The AEO is clearly written.         4.08       1.8%       80.1%

  3\. The AEO is relevant.                4.32       2.8%       88.0%

  4\. Do you use other sources with                             
  information similar to the AEO?                               

  5\. The AEO incorporates changing       3.51       7.2%       49.2%
  industry structure.                                           

  6\. Appropriate assumptions (e.g., GDP, 3.64       7.5%       59.7%
  world oil prices, environment) are used                       
  in the AEO.                                                   

  7.a. The methodology is suitable to the                       
  analysis of the following end-use                             
  sectors:                                                      

  i\. Residential                         3.70       12.1%      66.7%

  ii\. Commercial                         3.79       6.1%       66.7%

  iii\. Industrial                        3.46       11.4%      48.6%

  iv\. Transportation                     3.41       17.2%      55.2%

  7.b. Supply                                                   

  i\. Petroleum                           3.76       6.1%       66.7%

  ii\. Natural Gas                        3.56       16.7%      52.8%

  iii\. Coal                              3.88       0%         68.8%

  iv\. Renewables                         3.45       20.7%      51.7%

  v\. Electric Power Generation           3.80       12.5%      72.5%

  7.c. Petroleum Refining                 3.37       14.8%      40.7%

  8\. The conclusions reached in the      3.95       5.0%       76.7%
  report are supported by the analysis in                       
  the AEO and its companion report,                             
  "Assumptions to the AEO."                                     

  9\. The information in the AEO is of    4.13       2.9%       82.3%
  high quality.                                                 

  Is there anything else you would like                         
  to tell us about the AEO?                                     

  IEO Section:                                                  

  1\. The IEO is used extensively in my   3.34       14.3%      42.9%
  work.                                                         

  2\. The IEO is clearly written.         3.94       2.8%       77.8%

  3\. The IEO is relevant.                4.11       0%         83.3%

  4\. Do you use other data sources with                        
  information similar to the IEO?                               

  5\. The IEO incorporates the changing   3.64       7.1%       64.2%
  industry structure.                                           

  6\. Appropriate assumptions (e.g., GDP, 4.0        0%         85.7%
  world oil prices, environment) are used                       
  in the IEO.                                                   

  7\. The conclusions reached in the      4.0        0%         73.3%
  report are supported by the analysis in                       
  the IEO.                                                      

  8\. The information in the AEO is of    4.13       0%         73.3%
  high quality.                                                 

  Is there anything else you would like                         
  to tell us about the IEO?                                     
  --------------------------------------- ---------- ---------- ----------

**Scale instruction:** Indicate your evaluation "Using a scale from 1 to
5, with 5 meaning that you strongly agree with the statement; 4 meaning
you agree; 3 meaning you are neutral; 2 meaning you disagree; and 1
meaning you strongly disagree with the statement. If the answer is
"Don't know (DK)", please answer DK."

[^1]: The paper and proposed questions may be reviewed on last fall's
    ASA/EIA meeting home page at:
    <http://www.eia.doe.gov/smg/asa_meeting_2003/fall/>

[^2]: The complete summary of advice and response to the fall 2003
    meeting may be found on EIA's Home page under Energy Events, Summary
    of Advice from the American Statistical Association:
    <http://www.eia.doe.gov/calendar/asa_overview.htm>

[^3]: Experts were operationally defined as respondents answering one or
    more questions evaluating the AEO NEMS methodology.

[^4]: Experts were operationally defined as respondents answering one or
    more questions evaluating the IEO NEMS methodology.