# Presentation: 789718

## Program to Evaluate Form EIA-920, Combined Heat and Power Plant Report

- ASA Committee on Energy Statistics
- October 28, 2004

**Notes:**
Good morning, I am Bob Rutchik of the Statistics and Methods Group of EIA. I want to welcome you to the third breakout session in the past year and a half that we have on the EIA-920, Combined Heat and Power Plant Report. The previous two were in April 2003 on the EIA-920 pre-survey design visits and in October 2003 on proposed respondent level and aggregate level edits for the form.

Purpose of this session is to get the committee’s feedback on EIA’s plan to evaluate the effectiveness of Form EIA-920. 

We picked the EIA-920 to apply this program because we had extensive experience in the survey’s design and testing.  We are very familiar with it.

## Criteria for Evaluation of Form EIA-920

- Timeliness of response
- Ability to provide data
- Accuracy of response
- Reduction in processing time for EIA

**Notes:**
This is the plan.  There are four criteria which EIA propose to use to evaluate the EIA-920. 

			Timeliness of response
Ability to provide data
Accuracy of response
Reduction in processing time for EIA

Shortly, I will go over measures to operationalize each of the four criteria.  

We also hope to use this proposal as a template to evaluate other surveys to which EIA has applied its survey design and testing program, so EIA will not just have a design and testing program, but a design, testing, and evaluation program.

## The EIA-920

- Collects data on
- Total fuel consumed
- Fuel consumed to generate electricity
- Generation
- Fossil fuel stocks
- Major focus is fuel consumed to generate electricity

**Notes:**
Now, I just want to review the Form, EIA-920.

The EIA-920, which EIA began collecting data on early this year, is a monthly form of a sample selected from all CHP facilities

There is also an annual form that is completed by CHP facilities that are not selected to respond monthly.
It collects these variables.  The major focus is fuel consumed to generate electricity. If the respondent has understood the survey, the reported quantity for it should be less than total fuel consumed. 

## Variables to Operationalize Criteria

- Timeliness
- More respondents are reporting within the reporting period
- Fewer respondents getting reporting extensions
- Ability to provide data
- Response rates
- Reporting errors
- Analysis of comments
- Expert review with survey staff
- Follow-up with respondents

**Notes:**
As noted, for each of the four criteria, we have several variables to operationalize each.

Timeliness
More respondents are reporting within the reporting period (than in the EIA-906).
Fewer reporting extensions given to respondent
Ability to provide data
Response rates
Reporting errors (Houston, literally in some cases, we have a problem)
Analysis of comments
Expert review with survey staff
Follow-up with respondents

## Variables to Operationalize Criteria - Continued

- Accuracy of response
- Compare data on EIA-920 to data collected on EIA-906
- Subgroup analysis by prime mover and  industry
- Reduction in processing time for EIA
- Decrease in respondent call backs
- Less imputation for item nonresponse

**Notes:**
Accuracy of response
Compare fuel used to generate electricity on EIA-920 to fuel consumption data on EIA-906, Power Plant Report. There is a pitfall here.  On the EIA-906, EIA collected data for Useful Thermal Output, which is fuel used for manufacturing and other process in combined heat and power facilities. From UTO, EIA then calculated fuel used to generate electricity as the difference between UTO and  total fuel consumed. It was EIA’s weariness about the accuracy of UTO and the calculated fuel numbers that led to the EIA-920.
Subgroup analysis: by prime mover configuration, industry.
Reduction in processing time for EIA
Decrease in respondent call backs 
Less imputation for item nonresponse

## Preliminary Analysis on Ability to Provide Data

- Analysis of comments
- Fuel used to generate electricity less than total fuel consumed?
- 74 respondents reported that fuel used to generate electricity data was less than total consumption
- 68 respondents did not

**Notes:**
We have done some work on the ability to provide data criterion. For the January to May data, we looked at 140 respondents of the approximately 300 responses EIA got for that period. 
 Respondent and EIA comments 
Compared Question 1 and to Question 2 data to assess if the respondents understood the basic survey concept. This was not assessment of the data’s accuracy. 

Comments yielded little systematic information.

Comparison of Question 2 to Question 1 data provided this breakdown 
Little over half, 74, seem to have understood the survey’s basic concept; that fuel used to generate electricity data should be less than total consumption.
68 did not. For most of these, fuel used to generate electricity data equaled  total fuel consumption. The remainder did not reported consumption data, or any data at all, or had other problems with responses. 

.

## Other Findings

- Waste heat fuel consumption not reported
- Larger respondents appear to report correctly
- Pre-survey design respondents reported correctly
- Cognitive interview respondent reporting mixed

**Notes:**
Describe waste heat problem – from pre-survey design visits.

Very preliminary, but larger companies appear to be reporting correctly by a little higher proportion than the rest of the “sample.” Decentralized reporting.

Four pre-survey design visit respondents and 6 cognitive interview respondents.  Pre-survey respondents got it; cognitive interviews did not. But

## Questions for the Committee

- Are the criteria that EIA has proposed for the evaluation valid?
- Are there other criteria that EIA can use?
- Are there measures to operationalize the criteria other than those presented that EIA can use?

**Notes:**

Here are our questions for the Committee.
To sum up these questions, is EIA on the right track to assessing survey effectiveness.