# Presentation: 718931

**Bill Weinig & Howard Bradsher-Fredrick  Statistics & Methods Group**
**Presentation to the ASA Committee on Energy Statistics **
**April 22, 2004**

**Measuring the Quality of EIA Analytical Products**

## Outline of Presentation

**Goal 1 of EIA’s Strategic Plan**
**Review of Presentation from Fall, 2003 and Committee’s Suggestions**
**Revised Methodology**
**Evaluating *****Annual Energy Outlook***** (AEO) & *****International Energy Outlook *****(IEO)**
**Results**
**Questions to Committee**

## Acknowledgement of Support 

**Team Members (SMG):  **
**Stan Freedman, Team Leader, Howard Bradsher-Fredrick, Tom Broene, Inderjit Kundra, Herb Miller, Renee Miller, Joseph Sedransk, Phillip Tseng, Bill Weinig**
**Support (OIAF):**
**Paul Holtberg, Peggy Wells**

## EIA’s Strategic Plan: Goal 1

**“****EIA’s information program is relevant, reliable and consistent with changing industry markets and EIA’s information products will retain or improve their high quality and timeliness.”**

## Approach Described at Fall, 2003 Meeting 

**7-question survey with clarifying definitions**
**Respondents: 10 reviewers of EIA’s draft analytical products.**

## Comments by Committee	

**The scope of the questions seemed comprehensive (acceptable)**
**Complex questions open to some range of interpretation (vague questions)**
**Small sample size**
**Survey format supported tabular results**
**Would probably return unreliable results**** **

## Revised Research Design

**Focused on evaluating only the AEO and IEO.**
**Used 17 narrowly focused questions closed-ended questions and 2 open-ended questions**
**Eliminated embedded definitions**
**Maintained 5-point Likert scale. **

## Revised Methodology

**Surveyed registrants at Spring, 2004 Nat’l Energy Modeling Conference. **
**Pre-tested electronic format & logistics**
**Requested comments from OIAF/SMG on survey questions**
**Obtained list of 397 non-EIA pre-March 1****st**** registrants who provided e-mail addresses.**
**Used a Likert Scale:**
**1: Strongly disagreed**
**2:  Disagreed**
**3:  Neutral**
**4:  Agreed**
**5:  Strongly Agreed**

## Number of Respondents

| Date | No. | %  | Cum No. | Cum. % |
| --- | --- | --- | --- | --- |
| Mar. 4 | 42 | 10.6 | 42 | 10.6 |
| Mar. 9 | 39 | 9.8 | 81 | 20.4 |
| Mar. 11 | 47 | 11.8 | 128 | 32.2 |
| Mar. 19 (phone) | 4 | 1.0 | 132 | 33.2 |

## Commentary on Response Rates

**33.2% response rate-- higher than we expected****	**
**Difficult to infer anything about the 66.8% non-respondents**
**However, statistical tests showed no significant differences (@ .10 level) for the responses to any question between the 1****st**** mailing and subsequent mailings.**

## Results

**Average evaluation scores varied from:**
**Low: 3.37 (AEO: “The methodology is suitable to the analysis of Petroleum Refining.”)**
**High: 4.31 (“The AEO is relevant.”)**
**This range of averages appears to be indicative of products that are generally well-liked by the evaluators.**

## Table of Results: Descriptive Statistics

**The AEO does best with the “relevancy” and “high quality” questions.**
**The AEO does least well on the “methodology being suitable to the analysis” questions.**
**The IEO does well on almost all questions.**
**The IEO does least well on the “used extensively in my work,” question.****   **

## Results: Experts vs. Non-Experts

**Experts were operationally defined as those respondents who answered “yes” to filter question, “(Are you) familiar with modeling details behind the AEO/IEO.” **
**Significant differences with the “used extensively in my work,” questions.**
**For the IEO, “The IEO is clearly written,” question (@ .01 significance level).**

## Questions for the Committee

**Do EIA’s responses appear adequate to meet your suggestions and criticisms at the Fall, 2003 meeting?**
**If this survey is needed every year, is there a way to avoid overworking our respondents.**
**Could we vary our survey population from NEMS conference registrants to a Listserve.**
**What changes should we make?**
**Is a 33.2% response rate adequate for a customer survey.**