**Frames Evaluation**

**of Five CNEAF Surveys**

**with Establishments in the Manufacturing Sector of the 2002 Economic
Census**

**October 15, 2004**

**Purpose of Frames Evaluation:** EIA contracted with the Census Bureau
to conduct evaluations for five CNEAF surveys, each with establishments
in the manufacturing sector of the 2002 Economic Census. The results of
the frame evaluation will indicate the percentage of coverage, both by
count and volumes where possible, for these surveys. The analysis will
cover the following surveys:

*Coal Industry*

-   EIA-3, "Quarterly Coal Consumption and Quality Repot, Coke Plants"

-   EIA-5, "Quarterly Coal Consumption and Quality Repot, Manufacturing
    Plants"

*Renewable Energy Industry*

-   EIA-63a, "Annual Solar Thermal Collector Manufacturing Survey"

-   EIA-63b, "Annual Photovoltaic Module and Cell Manufacturing Survey"

> *Electric Power Industry*

-   EIA-860, "Annual Electric Generator Report" for Combined Heat and
    Power (CHP) Plants and Independent Power Producers (IPPs) with NAICS
    in manufacturing sector.

Key variables selected for analysis of coverage include an input, an
output, or both. These surveys are intended to evaluate coverage of coal
consumption (EIA-3); coke and breeze production (EIA-5), solar thermal
collector production (EIA-63a); photovoltaic cells/module production
(EIA-63b); and fuel consumption and/or electricity generation (EIA-860).

Individual establishment data collected during the Economic Census are
protected by Title 13, US Code. Therefore, the results of these
assessments are subject to the Census Bureau's disclosure policy. The
Census Bureau has agreed to analyze and provide as much detail as
allowed by law.

EIA will use the information provided to establish an action plan for
improving coverage where necessary.

**Matching and Analysis**

*Matching*

1.  Identify establishments on EIA frames.

2.  Identify establishments on Census frame (using Business Register)

3.  Match establishments

4.  Research establishments that do not match

*Analysis*

1.  Summarize results of analysis by count and by volume

2.  Summarize differences between frames which impact coverage
    (classification or boundary of establishments)

3.  Characterize establishments missing from EIA frame (e.g. location,
    size, type of establishments)

**Schedule**

By December 2004, Census will provide final results of evaluations for
EIA-5, EIA-63a, and EIA-63b. Census will also provide preliminary
results of matching for EIA-3 and EIA-860.

By July 2005, Census will provide final results of evaluation for EIA-3
and EIA-860

**Attachments**

I.  Summary of EIA and Census Frames (page 2)

II. Proposed Frame Comparison (page 3)

III. Preliminary EIA-63a Report (page 4)

**Attachment I: Summary of EIA and Census Frames**

+------+--------------------+--------------------+--------------------+
| **   | **Name and         | **Population**     | **Frame size**     |
| Surv | frequency of       |                    |                    |
| ey** | survey**           |                    |                    |
+------+--------------------+--------------------+--------------------+
| EC   | *Economic Census*  | The 2002 Economic  | There are          |
|      |                    | Census covers      | approximately 7    |
|      | Conducted every 5  | nearly all of the  | million            |
|      | years              | U.S. economy in    | establishments     |
|      |                    | its basic          | contained in the   |
|      |                    | collection of      | Census Bureau's    |
|      |                    | establishment      | Business Register, |
|      |                    | statistics.        | approximately      |
|      |                    | Censuses of        | 250,000            |
|      |                    | agriculture and    | establishments are |
|      |                    | governments are    | classified as      |
|      |                    | conducted at the   | manufacturing.     |
|      |                    | same time.         |                    |
+------+--------------------+--------------------+--------------------+
| MECS | *Manufacturing     | Manufacturing      | Sample Frame of    |
|      | Energy             | sector of 2002     | 15,500 drawn from  |
|      | Consumptions       | Economic Census    | the 250,000        |
|      | Survey*            |                    | establishments in  |
|      |                    |                    | manufacturing      |
|      | Conducted every 4  |                    | sector of Economic |
|      | years              |                    | Census.            |
+------+--------------------+--------------------+--------------------+
| CIR  | *Current           | This survey covers | Approximately      |
|      | Industrial Report* | the manufacture of | 3,000              |
|      | (MA334Q)           | semi-conductors,   | manufacturing      |
|      |                    | printed circuit    | establishments     |
|      |                    | boards,            | report to the      |
|      |                    | connectors,        | MA334Q. (This      |
|      |                    | capacitors and     | includes           |
|      |                    | other electronic   | establishments     |
|      |                    | components (except | that produce       |
|      |                    | relays), in the    | photovoltaic cells |
|      |                    | US.                | and modules.)      |
+------+--------------------+--------------------+--------------------+
| EIA  | *Annual Solar      | Census of          | 20                 |
| -63a | Thermal Collector  | manufacturers,     |                    |
|      | Manufacturers      | wholesale,         |                    |
|      | Survey*            | import/export of   |                    |
|      |                    | solar thermal      |                    |
|      |                    | collectors         |                    |
+------+--------------------+--------------------+--------------------+
| EIA  | *Annual            | Census of          | 20                 |
| -63b | Photovoltaic       | manufacturers,     |                    |
|      | Module and Cell    | wholesale,         |                    |
|      | Manufacturers      | import/export of   |                    |
|      | Survey*            | photovoltaic       |                    |
|      |                    | modules and cells  |                    |
+------+--------------------+--------------------+--------------------+
| E    | *Quarterly Coal    | Census of active   | 21                 |
| IA-5 | Consumption and    | coke plants        |                    |
|      | Quality Report,    |                    |                    |
|      | Coke Plants*       |                    |                    |
+------+--------------------+--------------------+--------------------+
| E    | *Quarterly Coal    | Census of          | 507                |
| IA-3 | Consumption and    | manufacturers      |                    |
|      | Quality Report,    | which consume coal |                    |
|      | Manufacturing      |                    |                    |
|      | Plants*            |                    |                    |
+------+--------------------+--------------------+--------------------+
| EIA  | *Annual Electric   | Combined Heat and  | 221 IPPs           |
| -860 | Generator Report*  | Power Plants and   |                    |
|      |                    | Independent Power  | 625 CHPs           |
|      |                    | Producers in       |                    |
|      |                    | manufacturing      |                    |
|      |                    | sector, NAICS 31   |                    |
|      |                    | to 33              |                    |
+------+--------------------+--------------------+--------------------+

Below are both the EIA and Census surveys used for conducting the frame
evaluations. This list includes five EIA surveys and three surveys
conducted by Census -- Economic Census (EC), Current Industrial Report
(CIR), and the Manufacturing Energy Consumption Survey (MECS), an EIA
sponsored survey.

**Attachment II: Proposed Frame Comparisons**

+-----------------+-----------------+-----------------+----------------+
| **EIA Survey**  | **Census        | **Key           | **Frame        |
|                 | Survey**        | variables:**    | Size:** number |
| Population      |                 | Input (I) and   | of             |
|                 | EC/MECS or CIR  | Output (O)      | establishments |
+-----------------+-----------------+-----------------+----------------+
| EIA-3           | EC/MECS         | Coal, tons (I)  | 507            |
|                 |                 |                 |                |
| Manufacturing   |                 |                 |                |
| Plants          |                 |                 |                |
+-----------------+-----------------+-----------------+----------------+
| EIA-5           | EC/MECS         | Coal coke, tons | 21             |
|                 |                 | (O)             |                |
| Coke Plants     |                 |                 |                |
|                 |                 | Breeze, tons    |                |
|                 |                 | (O)             |                |
+-----------------+-----------------+-----------------+----------------+
| EIA-63a         | EC              | Solar Panels,   | 20             |
|                 |                 | \$ (O)          |                |
| Solar           |                 |                 |                |
| Collectors      |                 |                 |                |
+-----------------+-----------------+-----------------+----------------+
| EIA-63b         | Current         | Photovoltaic    | 20             |
|                 | Industrial      | cell, megawatts |                |
| Photovoltaic    | Report (CIR)    | (O)             |                |
| Cells/Modules   | MA334Q          |                 |                |
+-----------------+-----------------+-----------------+----------------+
| EIA-860         | EC/MECS         | Electricity     | 221 IPPs       |
|                 |                 | generation (O)  |                |
| CHPs and IPPs   |                 |                 | 625 CHPs       |
+-----------------+-----------------+-----------------+----------------+

> **Attachment III:** **Report on Frames Coverage for EIA-63a**

Background: The EIA-63a is an annual survey that publishes data
collected from manufacturers, wholesalers, and importer/exporters of
Solar Collector Panels within the United States. There are 20
establishments in 5 states on EIA-63a frame.

To compare the frame of the EIA-63a to the 2002 Economic Census, it was
necessary to determine where each of the 20 establishments was
classified in the Census Bureau's Business Register. (A more detailed
account of the matching done for this comparison is below under the
methodology section.) The Business Register classifications of
establishments are based on primary activity performed at the
establishment. This implies that an establishment may have some
manufacturing capabilities but if the primary activity were not
manufacturing then the manufacturing information would be reported as
secondary activity. In this case, detailed information at the 10-digit
product as to the manufacturing portion of the establishment's total
receipts may not be available.

For establishments that are classified in manufacturing, the Economic
Census collects and publishes general statistics such as employment,
payroll, inventories, etc. at the 6-digit NAICS industry level. The EC
also publishes product detail regardless of the industry classification
of the establishments that manufacture the given product. Establishments
that have less than 5 employees are not mailed an Economic Census form;
the Census Bureau uses administrative information obtained from other
Federal Agencies to impute data for these establishments. These
establishments account for approximately three percent (3%) of the total
receipts published for the manufacturing sector of the 2002 Economic
Census.

**What is Methodology for evaluating coverage?**

1.  Matching: Matching establishments on the EIA-63a frame to
    establishments on the 2002 Economic Census was conducted in three
    phases. First, establishments on the EIA-63a frame were matched to
    establishments on the Census Bureau's Business Register to determine
    whether or not detailed information was available about solar panel
    manufacturing. Establishments were matched on EIA-63a frame with
    establishments on the Economic Census. Second, all establishments
    which produced solar panels (Product Code 333414A36) were identified
    and these establishments were matched to the EIA-63a frame. Finally,
    for establishments within the non-mailed portion of the Economic
    Census that reported production of solar panels to the EIA-63a, we
    researched and verified that they manufactured solar panels then
    added the data reported to the EIA-63a to the Product Code
    333414A136. We researched establishments that did not match to the
    EIA-63a; where we found response error present we made corrections
    to product code 333414A136.

2.  Coverage analysis: Analysis is provided both by counts and by
    volume. The more significant of the two is the coverage by volume
    since it reflects the percent of the total value of shipments.

*Coverage by counts:* The coverage by number of establishments was
measured as the percent of the total number of matched establishments on
EIA's frame (numerator) with the total number of in-scope establishments
on the frame for the MECS or the Census of Manufacturers (denominator).
For the EIA-63, the following equation applies to percent of coverage by
count:

*% =* [Number of Matched Establishments on Economic Census that produced
solar panels]{.underline}

Number of Active Establishments Inscope on Economic Census that produced
solar panels

*Coverage by volume:* The value of shipments (dollars) that was used to
calculate the coverage ratio by volume was the same in the denominator
and numerator for establishments that match since the value was from the
Economic Census. The denominator also includes values for nonmatched
"in-scope" establishments on Economic Census. Not all the establishments
on the Economic Census are "in-scope" due to differences between the
surveys. For the EIA-63, the following equation applies to percent of
coverage:

*% =* [Unweighted Value of Shipments of solar panels for matched active
Establishments on Economic Census]{.underline}

Unweighted Value of Shipments of solar panels for all active
establishments Inscope on Economic Census

**Is the Coverage Sufficient?**

The following tables provide a template for the proposed results.

Table I: Regional counts

  --------- ----------------------- --------- ------------ --------- --------- --------- ---------
            **Counts/Percentage**   **State   **State2**   **State   **State   **State   **U.S.
                                    1**                    3**       4**       5**       Total**

  a         Both CM/EIA-63a         \#        \#           \#        \#        \#        \#

  b         EC only                 \#        \#           \#        \#        \#        \#

  c         EIA only                \#        \#           \#        \#        \#        \#

  b-c       Difference              \#        \#           \#        \#        \#        \#

  a/(a+b)   Coverage by count       \%        \%           \%        \%        \%        \%
  --------- ----------------------- --------- ------------ --------- --------- --------- ---------

**Table II: Regional Volumes** (Value of Shipments)

  --------- ----------------------- ---------- ---------- ---------- ---------- ---------
            **Counts/Percentage**   **Region   **Region   **Region   **Region   **U.S.
                                    I**        II**       III**      IV**       Total**

  a         Both CM/EIA-63a         \$         \$         \$         \$         \$

  b         EC only                 \$         \$         \$         \$         \$

  c         EIA only                \$         \$         \$         \$         \$

  b-c       Difference              \$         \$         \$         \$         \$

  a/(a+b)   Coverage by volume      \%         \%         \%         \%         \%
  --------- ----------------------- ---------- ---------- ---------- ---------- ---------

**What are potential reasons for differences between EIA and Economic
Census?**

Potential causes for differences between surveys may include:
differences in boundaries of establishments and other factors regarding
construction of the two frames, differences in question-wording
(measurement) which impact establishment counts and key variables used
to evaluate coverage.

**What are Characteristics of Missing Establishments?**

Table I (above) is intended to identify location of establishments which
may be missing from EIA-63a frame. Additional characteristics of missing
establishments might include size, NAICS classification, and other
potential differences.