# Presentation: 502884

##  A Customer Evaluation of the Short Term Energy Outlook (STEO)

**Howard Bradsher-Fredrick, Stan Freedman, Tom Broene, Inderjit Kundra, Herb Miller, Renee Miller, Joseph Sedransk, Phillip Tseng, Bill Weinig**
**Energy Information Administration**
**October 28, 2004**

**Notes:**
Good morning and welcome to our session.  We’re scheduled for about 70 minutes, but we may not need that much time.

The title of our presentation is, “A Customer.....”

I want to begin by thanking my co-authors who are listed, many of whom are in the room here.

## Outline of Presentation

- Background
- Methodology
- Results
- Response rate comparison with the AEO/IEO
- Demographic characteristics of respondents
- Evaluation of the STEO customer data
- Comparison to the AEO/IEO
- Analysis of earlier versus later respondents
- Conclusions
- Questions for Committee

**Notes:**
This is the outline of my presentation.  

## Background

- Customer evaluations are part of EIA strategic planning efforts.
*Annual Energy Outlook* (AEO), 3/2004
*International Energy Outlook* (IEO), 3/2004
- At the April ASA-Energy Meeting, the committee suggested a web-based approach using the STEO Listserv.

**Notes:**
Customer evaluations have become part of EIA’s strategic planning efforts and their performance measures.  

Last April, our team presented the findings of our customer survey of the Annual Energy Outlook and the International Energy Outlook.

Also at the April meeting, the committee suggested a web-based approach using the STEO Listserv.  We acted upon that advice.

## Methodology

- Use of STEO Listserv to select a random sample of STEO customers.
- 4,000 e-mail addresses in the Listserv
- 500 selected using MS-EXCEL software (EIA staff excluded)
- 471 usable addresses
- Pre-test of the software was conducted
- 50 e-mail addresses selected for an early 1st mailing
- We received 4 responses within the 1st 4 hours.
- Software appeared to be operating properly.

**Notes:**
We randomly selected 500 non-EIA e-mail addresses from the 4,000 e-mail addresses listed on our STEO Listserv

There appeared to be 471 usable addresses; this is based upon the number of “Undeliverable” messages received from the initial mailing.

Due to the fact that we could not really test the software before mail-out, we decided to pre-test the software by selecting 50 e-mail addresses for an early 1st mailing.

We received 4 responses within the 1st four hours with the software apparently operating correctly.  The spreadsheet appeared to be written to correctly and the e-mail respondent and non-respondent lists appeared okay.  

## Methodology (cont.)

- 5-point Likert Scale was used. (1 = S.D., 2 = D,    3 = Neutral, 4 = A, 5 = S.A.)
- Questions used followed the AEO/IEO customer evaluation questions as closely as possible.
- 5 of 13 questions the same.
- EMEU staff members were consulted for relevancy of all questions.
- 23 (18.9%) agreed to a follow-up interview.

**Notes:**
We employed a 5-point Likert scale, beginning with 1 for strongly disagree and ending with 5 for strongly agree.

2nd bullet.

We also consulted with the Office of Energy Markets and End-Use; that office has responsibility for the STEO.

Unlike with the AEO and IEO, we asked respondents if they wouldn’t mind being contacted again.  23 (18.9%) agreed to a follow-up interview – 21 via telephone and 2 via e-mail.  We asked this question so that a subset of respondents could be interviewed in depth if we desire to do that.

## Response Rate Comparison to AEO/IEO	

| Mailing | STEO Cum. # of Resp. | STEO Cum. Resp. % | AEO/IEO Cum. Resp. % |
| --- | --- | --- | --- |
| 1st | 36 | 7.6% | 10.6% |
| 2nd (3 bus. days later) | 83 | 17.6% | 20.4% |
| 3rd (2 bus. days later) | 111 | 23.6% | 32.2% |
| 4th (2 bus. days later) | 122 | 25.9% | 33.2% |

**Notes:**
Here is the response rate comparison to the AEO/IEO.  The final number of respondents was 122 (or 25.9%) for the STEO.  This was slightly lower than the 33.2% achieved for the AEO/IEO.

In both cases we conducted a total of 4 mailings over a 2-week period.

It should be remembered that we used a list of conference registrants to survey customers of the AEO and IEO last spring.

## Demographic Characteristics of Respondents

| Group Affiliation | No. | % |
| --- | --- | --- |
| Academia | 2 | 1.6% |
| Research & Consulting | 18 | 14.8% |
| Gov’t (Fed, State, & Local) | 13 | 10.7% |
| Banking & Investment | 16 | 13.1% |
| International Organizations | 2 | 1.6% |
| Library | 1 | 0.8% |
| Business & Industry | 62 | 50.8% |
| Media & Press | 1 | 0.8% |
| Other | 7 | 5.7% |
| Total | 122 | 100% |

**Notes:**
This chart shows the demographic characteristics of the respondents.  As you can see, we have a good cross section of group affiliations, but more than half were from “business and industry.”

## Results: Evaluation of the STEO Customer Data

| Question | Avg. | % 1s , 2s | % 4s, 5s |
| --- | --- | --- | --- |
| 1. The STEO is used extensively in my work. | 3.42 | 21.2% | 53.4% |
| 2. The STEO is clearly written. | 4.18 | 3.5% | 88.6% |
| 3. The STEO is relevant. | 4.34 | 0.8% | 93.2% |
| 4. The STEO incorporates the most up-to-date market information. | 4.02 | 7.4% | 81.5% |
| 5. Appropriate data inputs are used for the projections in the STEO. | 3.91 | 1.0% | 71.4% |
| 6. Appropriate assumptions (e.g., GDP, world oil prices) are used in the STEO.  | 3.93 | 0.0% | 79.8% |

**Notes:**
The next two slides show the results for each question – the average, the % 1s and 2s and the % 4s and 5s.

As you can see, the highest scores are from Q3 (relevancy) and Q2 (clearly written).

The lowest score comes from Q1, the “used extensively in my work” Question.

## Evaluation of the STEO Customer Data (cont.)

| Question | Avg. | % 1s, 2s | % 4s, 5s |
| --- | --- | --- | --- |
| 7. The analysis is suitable to the following:
(n = 28) |  |  |  |
| a. Summer gasoline supply. | 4.00 | 3.7% | 85.2% |
| b. Winter heating fuel supply. | 4.00 | 3.8% | 80.8% |
| c. Natural gas demand and supply. | 4.10 | 3.7% | 88.9% |
| d. Summer and winter electricity market. | 3.90 | 4.8% | 81.0% |
| 8. The projections provided in the report are in-line with other independent sources (from profit and non-profit organizations). | 3.70 | 6.3% | 69.5% |
| 9. The information in the STEO is of high quality. | 4.09 | 0.9% | 81.4% |
| 10. It would be useful to add regional data for prices, demand and supply to STEO. | 4.15 | 6.9% | 76.7% |

**Notes:**
As you can see, Question #10 (regionalizing the model) also receives a high score as does Question #9, “The information in the STEO is of high quality.” 

I should also mention that only respondents who answered “yes” to a screener question answered the 4 sections (a thru d) of Question #7.  The screener question was, “Are you familiar with the STEO model structure?”

## Results: Comparison of the STEO to the AEO & IEO 

| Question | STEO 
n=122 | AEO n=133 | IEO n=35 |
| --- | --- | --- | --- |
| Q1. The publication is used extensively in my work. | 3.42* | 3.85 | 3.34 |
| Q2. The publication is clearly written. | 4.18 | 4.08 | 3.94 |
| Q3. The publication is relevant. | 4.34 | 4.32 | 4.11 |
| Q6. Appropriate assumptions are used in the publication. | 3.93# | 3.64 | 4.0 |
| Q9. The information in the publication is of high quality. | 4.09 | 4.13 | 4.2 |

## Results: Analysis of Earlier vs. Later Respondents

| Question # | 1st Mailing Average 
(n = 36) | 2nd – 4th Mailing Average (n = 86) |
| --- | --- | --- |
| Q1 | 3.28 | 3.48 |
| Q2 | 4.08 | 4.23 |
| Q3 | 4.28 | 4.37 |
| Q4 | 3.88 | 4.08 |
| Q5 | 3.67* | 4.01 |
| Q6 | 3.88 | 3.96 |
| Q7a | 4.17 (n = 7) | 3.95 (n = 21) |
| Q7b | 4.00 (n = 7) | 4.00 (n = 21) |
| Q7c | 3.80 (n = 7) | 4.30 ( n = 21) |
| Q7d | 3.33 (n = 7) | 4.00 (n = 21) |
| Q8 | 3.69 | 3.70 |
| Q9 | 4.00 | 4.13 |
| Q10 | 3.94 | 4.24 |

**Notes:**
The question always arises as to whether or not the response rate achieved is good, bad, or otherwise. In other words, we achieved a 25.9% response rate, but what about the 74.1% of respondents who did not respond?

In order to provide some evidence to answer this question, we compared the results by question of the 1st mailing versus the 2nd – 4th mailings and obtained the results shown above.

You will notice that the early respondents scored the questions fairly uniformly lower than the later respondents.  However, only one question resulted in a statistically significantly lower result at the .05 level and that was question 5.

## Conclusions

- Listserv was useful.
- Web-based survey was successful.
- 25.9% response rate was achieved.
- STEO found to be generally satisfactory to customers.
- Earlier respondents may have scored the STEO slightly lower than later respondents.

## Questions for Committee

- Any comments about the methodology?
- Any other suggestions?