# Presentation: 745962

## Using Performance Statistics to Design the 2006 Manufacturing Energy Consumption Survey

- April 6, 2006
**Robert Adler**
- (202-586-1134)
- _[Robert.Adler@eia.doe.gov](mailto:Robert.Adler@eia.doe.gov)_
**Tom Lorenz**
- (202-586-3442)
- _[Thomas.Lorenz@eia.doe.gov](mailto:Thomas.Lorenz@eia.doe.gov)_

## Acknowledgements

- 	Census Bureau staff:
- Richard Hough
- Vicki Haitot
- Stacey Cole

##                 Objective

- To improve data quality in the Manufacturing Energy Consumption Survey by examining performance statistics on:
- Data status flags
- Differential response rates

## Discussion Topics

- Boosting participation in Internet electronic form
- Other performance statistics that would be useful to track
- Risks in targeting non-response follow-up to larger establishments

## Organization

- Background on MECS
- Examination of data status flags to evaluate effectiveness of questionnaire
- Comparison of status flag data: respondents who completed an Excel version vs a written version
- Examination of differential response rate to suggest other changes to improve data quality in the MECS

## Background

- 	MECS is a quadrennial survey of manufacturing industries based on the North American Industry Classification System (NAICS)
- 3 survey forms
- “A”:  Mailed to most manufacturers
- “B”:  Petroleum refineries only
- “C”:  Sent to Chemical, Paper, Iron and Steel, and other NAICS industries that are energy intensive or have complicated energy flows
- Self-administered written questionnaire

## Background - continued

- Designed and sponsored by EIA and conducted by the U.S. Census Bureau because data are confidential by Title 13 of the U.S. Code
- Mandatory by federal law
- Statistical sample from list frame (15,500 sample cases in 2002)

## Data Status Flags

- 	**R**:Reported data; either through questionnaire or follow-up telephone or e-mail
- 	**A**:Analyst correction of reporting error
- 	 **I**:Imputed data; missing data imputed by price or other information (rare)
- 	**S**:	Source data: An entirely different set of data was used instead of MECS for the item
- 	**E**:	System edit; data changed due to systematic edit because of common mistake in reporting

**Notes:**
1. Generally, a greater proportion of R’s and fewer A’s, E’s, and I’s would indicate better data quality. 
2. An “S” flag may indicate good data quality but would cast some doubt on the quality of the questionnaire item.
3. R-flags may be an of good data quality in general, but tend to overstate the effectiveness of questionnaire items as even data that were retrieved through follow-up would get an “R.”
4. A way to get at “pure” R’s would be to compare unedited versions of the data with the edited—both versions are kept in StEPS. Future status flags will make this distinction.

## Figure 1: Status Flags of Selected Data Items, Manufacturing Energy Consumption Survey, 2002

**Notes:**
The numbered bars are the ones I want to talk about.
Although Figure 1 is crowded, it does provide a quick way of evaluating some of the MECS items. 
#1: As it turned out, the percentage of R’s that required analyst intervention was less than 10 percent for most items. 
One noteworthy exception was the LPG non-fuel from non C-form respondents, who would be very unlikely to have non-fuel use of LPG. For this item, 23 percent of the R’s actually required analyst intervention.
What’s really striking here is the percentage of A flags: 88 percent. Most of these corrections involved zeroing out the non-fuel and making it fuel. 
--------------------------------------------------------------------------------------------------------
#2: Natural gas used as a fuel is one of the most reported data items in the MECS. In 2002, 71 percent of the 200,000 establishments in manufacturing used natural gas as a fuel. 
At that high level of reporting (approximate 9,000 sample cases), it is important to minimize costly analyst intervention to any extent possible.
Yet, as can be seen in Figure 1, 74 percent of the cases used in the final MECS dataset actually come from directly reported data (93 percent of those R’s did not require analyst intervention.) The 22 percent “A” flags represent mostly conversions of the reporting units after failure of a price range check.
We believe that most of those analyst corrections could have been avoided if the respondents were warned of an edit failure while at the establishment site

## Table 1: Type of Status Flag as Percentage of Total, Manufacturing Energy Consumption Survey, 2002

|  | 
Reported | Analyst Correction | Source | E (System) | Imputed | Total Flags |
| --- | --- | --- | --- | --- | --- | --- |
| Electricity Generated Onsite | 87.0% | 11.9% | 1.0% | 0.0% | 0.2% | 523 |
| Total Consumption of Electricity | 86.5% | 10.3% | 0.8% | 1.3% | 1.1% | 11,199 |
| LPG Fuel | 80.3% | 15.6% | 0.2% | 3.6% | 0.4% | 3,345 |
| LPG Nonfuel (non C-form) | 12.0% | 88.0% | 0.0% | 0.0% | 0.0% | 183 |
| Natural Gas Fuel | 74.1% | 22.2% | 0.0% | 3.0% | 0.7% | 9,057 |
| Natural Gas Nonfuel | 32.1% | 67.6% | 0.2% | 0.0% | 0.2% | 623 |
| Mixtures of Butane, Ethane, Propane Fuel | 27.3% | 66.7% | 6.1% | 0.0% | 0.0% | 33 |
| Mixtures of Butane, Ethane, Propane Nonfuel | 38.5% | 61.5% | 0.0% | 0.0% | 0.0% | 13 |
| Other LPG's (-ene's) Fuel | 58.6% | 39.7% | 1.7% | 0.0% | 0.0% | 58 |
| Other LPG's (-ene's) Nonfuel | 45.9% | 11.5% | 41.8% | 0.0% | 0.8% | 122 |
| Butane Fuel | 74.3% | 20.0% | 5.7% | 0.0% | 0.0% | 35 |
| Butane Nonfuel | 75.9% | 20.7% | 3.4% | 0.0% | 0.0% | 29 |
| Ethane Fuel | 62.5% | 37.5% | 0.0% | 0.0% | 0.0% | 8 |
| Ethane nonfuel | 76.0% | 12.0% | 12.0% | 0.0% | 0.0% | 25 |
| Propane fuel | 74.5% | 20.2% | 0.4% | 4.6% | 0.3% | 1,589 |
| Propane nonfuel | 21.6% | 75.9% | 1.2% | 1.2% | 0.0% | 162 |

**Notes:**
#3: 
A different type of problem emerges in the preponderance of “S” flags in the nonfuel use of different types of LPG. In Figure 1, the nonfuel use of “Other LPG’s (-enes),” which includes propylene, ethylene, and butylene, has an especially high rate of that flag, 42 percent. 
The “S” flag for LPG nonfuel represents the substitution of Economic Census—Manufacturing (ECM) data for MECS when the MECS might have missing or questionable data. When data editors contacted the respondents to reconcile the discrepancy between the MECS and the ECM concerning the LPG nonfuel use, they discovered that Chemical respondents did not interpret the MECS reporting requirements correctly. Therefore, we accepted the ECM data as true.
Apparently, “nonfuel” or “feedstock” use was not seen as a “material input,” the ECM term that induced them to report correctly on that survey. Thus, as part of the 2006 MECS, we intend to use that term along side the other traditional MECS terms. 

## Results for Selected Items 

- Natural gas used as a fuel one of the most reported data items in the MECS
- Important to minimize costly analyst intervention
- The 22 percent “A” flags represent mostly conversions of the reporting units after failure of a price range check
- Most of those analyst corrections could have been avoided if the respondents were warned of an edit failure while at the establishment site

## Results for Selected Items - LPG Non-fuel

- “S” flag for LPG nonfuel represents the substitution of Economic Census—Manufacturing (ECM) data for MECS
- Data editors discovered that Chemical respondents did not interpret the MECS reporting requirements correctly
- “Non-fuel” or “feedstock” use was not seen as a “material input,” the ECM term. Thus, as part of the 2006 MECS, we intend to use that term along side the other traditional MECS terms

## Creation of Excel Version 

- EIA developed an Excel workbook version of the 2002 MECS C-Form
- Excel version looked like the written questionnaire
- Added capabilities to Excel version include:
- price and value range checks of major energy sources
- automatically calculates derived data items (e.g., total consumption of electricity)
- automatically copies reported and derived data to later sections

## Limitations

- 	Time and other constraints prevented it from optimal development
- Not all desired onsite edits or screeners could be included because respondents had to save data to a CD and/or diskette
- Data from the Excel version was not fully integrated with Census data capture routines
- CD and/or diskette had to be mailed to the same Census address as the paper questionnaires
- Responses were keyed the same as with the normal paper-and-pencil mode

## Participation Rate

- 323 responses from the Excel version (8.9% of the total number of C-form responders)
- Excel responders accounted for 12.4% of the total consumption reported from the C-form
- 23 respondents failed reporting using Excel version:
- Incorrectly formatted diskette
- Bad diskette

## Electricity: Electronic vs. Paper

| 
Type of Reporter | Flag | Total Electricity Purchased | Total Electricity Expenditures | Total Electricity Consumption |
| --- | --- | --- | --- | --- |
| Electronic | R | 96.0% | 97.5% | 96.3% |
|  | A | 3.4% | 1.9% | 3.1% |
|  | Other | 0.6% | 0.6% | 0.6% |
| Paper | R | 92.5% | 96.2% | 85.7% |
|  | A | 6.2% | 2.8% | 11.9% |
|  | Other | 1.3% | 1.0% | 2.3% |

**Electricity: Electronic vs. Paper**

## Natural Gas: Electronic vs. Paper

**Natural Gas: Electronic vs. Paper**

| 
Type of Reporter | Flag | Total Natural Gas Purchased | Total Natural Gas Expenditures | Total Natural Gas Fuel Consumption |
| --- | --- | --- | --- | --- |
| Electronic | R | 87.8% | 96.7% | 88.6% |
|  | A | 11.3% | 3.3% | 8.1% |
|  | Other | 0.9% | 0.0% | 3.3% |
| Paper | R | 76.0% | 95.9% | 74.9% |
|  | A | 21.7% | 3.6% | 16.8% |
|  | Other | 2.3% | 0.6% | 8.3% |

## Differential Response Rates

- As in the case of other establishment surveys, the problem of nonresponse in the MECS has been growing. Possible causes:
- Downsizing and greater emphasis on “core functions”
- Attitude toward surveys
- On the other hand, the budgets necessary to conduct large government surveys are also being squeezed
- To best use the resources available to us, the non-response follow-up for the 2002 MECS was targeted to have the greatest effect on the survey data quality

## Table 4: Response and Coverage Rates by Cell Size and Type.

| 
Cell Size | Target Coverage Rate | Type | Number in Cell | Reporters in Cell | Non-reporters in Cell | Response Rate | Achieved Average Coverage Rate |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Small | 70% | Certainty | 73 | 50 | 23 | 68% | 85% |
|  |  | Noncertainty | 134 | 89 | 45 | 66% |  |
| Medium-Small | 75% | Certainty | 566 | 457 | 109 | 81% | 84% |
|  |  | Noncertainty | 856 | 600 | 256 | 70% |  |
| Medium | 80% | Certainty | 1,751 | 1,452 | 299 | 83% | 87% |
|  |  | Noncertainty | 1,987 | 1,457 | 530 | 73% |  |
| Medium-Large | 85% | Certainty | 1,884 | 1,624 | 260 | 86% | 87% |
|  |  | Noncertainty | 4,511 | 3,294 | 1,219 | 73% |  |
| Large | 90% | Certainty | 552 | 483 | 69 | 88% | 92% |
|  |  | Noncertainty | 3,212 | 2,422 | 790 | 75% |  |
| Grand Total |   |   | 15,526 | 11,928 | 3,600 | 77% | 90% |

**Notes:**
1. The target rate is based on the total MOS of the cell under consideration
2. As can be seen in Table 4, the achieved coverage rate exceeded the target in all cases. This was true, even though the response rate in terms of valid submissions was relatively low
3. Non-response follow-up was stopped after the desired coverages were achieved
4. Coverage rates expressed this way gives a better indicator than the simple ratio of returns over total mailed of how much of the data produced by MECS is actually from respondents 

## Conclusions and 2006 MECS

- 2006 MECS will shift to an Internet-based electronic reporting method for its main mode of collection
- Much of the 2002 MECS late- and non-response was due to not having the right contact persons at the establishment in the beginning
- Status flags will be amended to further define the occurrence and type of analyst intervention

## Questions for the Committee

- Given that only 10 percent of the eligible respondents chose to use the electronic Excel form in 2002, have we introduced bias into the comparisons? Are we still justified in reporting the results?
- How do we boost participation in the Internet electronic form, given that respondents will have a back-up available to them?

## Questions for the Committee - continued

- What other performance statistics would be the most useful to track?
- What are the risks associated with targeting non-response follow-up to larger establishments to minimize a non-response adjustment?