**EIA FORM 906 [DRAFT]{.underline} EVALUATION REPORT FOR** **STEPS 1B
AND 2B**

**Allied Technology Group, Inc.**

**Task Order \#** DE-AT01-03EI90602

**August 30, 2003**

![](media/image3.png){width="2.05in" height="0.6375in"}

**1803 Research Blvd.**

#### Suite 601

**Rockville, MD 20850**

**T: 301.309.1234**

**F: 301.309.0978**

[**www.alliedtech.com**](http://www.alliedtech.com/)

**TABLE OF CONTENTS**

[SECTION]{.underline} [Page]{.underline}

1.0 INTRODUCTION 1

2.0 TABLE OF EDIT SPECIFICATIONS 2

3.0 TEST PLAN METHODOLOGY 8

3.1 Assumptions 8

3.2 Methodology 8

4.0 COMMENTS ON INITIAL VIEW OF THE DATA 16

**1.0 INTRODUCTION**

EIA tasked Allied Technology Group, Inc. to review their current EIA
Form 906 Edits and the proposed Form 906 redesign and evaluate the edits
used to validate submitted data. The purpose of these edits is to
validate submitted respondent data thereby minimizing the number of
potential errors that must be researched by EIA staff, errors which
result from edits that identify submitted data as unreasonable that are
actually correct.

The objective of work steps 1b and 2b is to conduct evaluations of the
proposed respondent and summary level edits submitted by Allied
Technology Group under steps 1a and 2a. The edits proposed under steps
1a and 2a were reviewed by EIA staff prior to this evaluation. Allied
Technology then revised the proposed edits based on comments by EIA. It
is the revised proposed edits that are being evaluated under steps 1b
and 2b. This report presents the results of the first stage of this
evaluation effort. Specifically this report presents the:

1.  Table of Specifications of the revised proposed edits that are being
    evaluated;

2.  Methodology of the test plan to be used for the evaluation; and

3.  Some comments based on an initial view of the data.

The second phase of the evaluation for steps 1b and 2b will culminate in
a final report of evaluation findings. The methodology used to construct
the edits was included in the deliverable for steps 1a and 2a. This
methodology will be summarized in the final evaluation report after the
second phase is complete.

This report is divided into four sections:

-   Section 1.0 -- INTRODUCTION -- describes the purpose and background
    of this effort;

-   Section 2.0 -- TABLE OF EDIT SPECIFICATIONS -- presents the revised
    edits proposed for use in the FORM 906 process;

-   Section 3.0 -- TEST PLAN METHODOLOGY -- presents an overview of the
    test methodology; and

-   Section 4.0 -- COMMENTS ON INITIAL VIEW OF THE DATA -- provides a
    listing data inconsistencies that may affect the edit testing.

> **2.0 TABLE OF EDIT SPECIFICATIONS**

This section presents the table of specifications for the revised
proposed Form 906 edits that are being evaluated. Techniques for how a
particular edit might be implemented, however, are not presented. This
approach facilitates analysis of the validity of the edits without
biasing the process by proposing an implementation methodology.
Suggested implementation methodologies will be provided after the
proposed edits are validated for applicability.

> ![](media/image4.wmf){width="7.759722222222222in"
> height="6.0472222222222225in"}
>
> ![](media/image5.wmf){width="7.729861111111111in"
> height="4.856944444444444in"}
>
> ![](media/image6.wmf){width="7.840277777777778in"
> height="6.206944444444445in"}

![](media/image7.wmf){width="7.729861111111111in"
height="4.7756944444444445in"}

![](media/image8.wmf){width="7.720138888888889in" height="3.99375in"}

# 3.0 TEST PLAN METHODOLOGY

To fulfill the requirements of steps 1b and 2b of this task, Allied
Technology will evaluate the respondent level and summary level edits as
defined in the revised proposed edit table above. The objective of the
evaluation is to conduct analysis testing rather than validity testing.
In other words the evaluation will not establish the validity of the
edits but will establish the effectiveness of the results of the edit
processing. This draft report only presents the plan to be used to test
these edits.

The testing will be conducted by applying the edits to the last 10 years
of F906 data.

1.  **Assumptions**

> Evaluation of the proposed F906 edits will be conducted under the
> following assumptions:

There are pre-defined lists available from EIA or other organizations as
noted for:

-   Prime Mover groupings: ( PM by non-combustible fuel and PM by
    combustible fuel)

-   Fuel type groupings: (non-combustible, combustible & stock able,
    combustible & non stock able)

-   Valid combinations of Prime Mover and fuel type

-   State codes (Federal Information Processing Standards)

-   Valid zip code list (Postal Service)

-   

-   Plant Identification numbers

```{=html}
<!-- -->
```
-   Unit of measure for reporting fuel quantities by fuel type

```{=html}
<!-- -->
```
-   Heat content (high and low range) per unit of fuel

> **3.2** **Methodology**
>
> The F906 proposed edits will be tested using a four-phased process:
> first, available F906 historical data for the past ten years will be
> collected; second, a preliminary review of the data will be conducted
> to identify data anomalies that might impact edit testing; third, the
> data will be processed against the set of proposed edits; and fourth,
> the results of the edit processing will be analyzed and their
> effectiveness evaluated. The following is a listing of proposed edits
> and the preliminary test methodology.

![](media/image9.wmf){width="4.059722222222222in"
height="6.153472222222222in"}

![](media/image10.wmf){width="4.070138888888889in" height="5.075in"}

![](media/image11.wmf){width="4.059722222222222in"
height="6.385416666666667in"}

![](media/image13.wmf){width="4.070138888888889in"
height="4.166666666666667in"}

## Time Series Methodology

> In the table above, the test approach for the quantitative variables
> of generation 2a(2), consumption 2a(4), and stock amount (3b) include
> range checks and regression techniques. In addition to using these
> testing approaches, time series smoothing techniques can also be
> employed.
>
> A primary approach to reducing the number of reported potential errors
> while maintaining data accuracy is to develop a time series-based
> estimating process. This approach assumes that the overall data values
> are consistent, if one can identify and account for outside "drivers."
>
> There are two methods to apply the a smoothing constant in exponential
> smoothing of a time series:

1.  Solve for the smoothing constant by using the criterion (other
    criteria are available) of minimizing mean absolute error (MAE);

2.  Subjectively set the smoothing constant (.9 if there have been some
    recent major changes in the way a company does business, or .1 if
    the series has been relatively stable over time).

> The first method is empirical and thus will indicate whether the
> series fluctuates a lot (i.e., a high smoothing constant), or is
> relatively stable (i.e., a low smoothing constant).
>
> The purpose in the F906 edits is to use the notion of the smoothing
> constant in the exponential smoothing equation to establish the
> production of an edit message. In this regard a two-step procedure
> will be tested:
>
> **STEP 1**

Solve for the smoothing constant using the MAE criterion. If the
smoothing constant is above .9 produce an edit to determine whether
recent fluctuation is due to: a) recent major changes in the way a
company does business, or b) error in recording the data.

### STEP 2

> If the smoothing constant that has been found in STEP 1 is below .9,
> determine the distance that the newest observation is from the
> previous observation in terms of MAE units\*. If the absolute value of
> this distance is greater than .5\*\* then produce an edit to determine
> whether recent fluctuation is due to: a) recent major changes in the
> way a company does business, or b) error in recording the data.
>
> \* distance = (data at time t) - (data at time t-1)/MAE
>
> The table below presents the results of this two step process on an
> analysis of monthly generation (GEN) data for one power plant for the
> period of 1992 thru 2002.
>
> The data illustrate the impact on solving for the smoothing constant
> versus setting the smoothing constant on MAE, the predicted value and
> the residual relative distance, which is the absolute difference
> between the predicted and actual divided by the MAE.

  --------------- ------------ ----------------- ------------------------
  Smoothing       Via MAE      Set at:           Set at:
  constant                                       
  criterion:                                     

  smoothing       .8358        .9                .1
  constant                                       

  MAE             116,346      117,328           142,375

  Forecast of     995,169      994,374           906,987
  60^th^ value                                   

  Actual 60^th^   1,019,451    1,019,451         1,019,451
  value                                          

  Absolute MAE    .21          .21               .79
  deviation                                      
  --------------- ------------ ----------------- ------------------------

> Note that as the smoothing constant approaches .9 and exceeds it using
> the minimizing the MAE criterion, the smoothing equation weights more
> recent observations as more important because of the fluctuations
> throughout the series. In contrast, when the exponential smoothing
> equation (using the minimizing MAE criterion) results in a low
> smoothing constant, the series is viewed as more stable. Thus one
> could set the smoothing constant at .1 if the series was considered
> stable. However in this example setting the smoothing constant at .1
> results in a much higher MAE and an absolute difference between the
> predicted and actual divided by the MAE greater than .5. The important
> point is that the two step process described here first determines the
> relative stability of the process by solving for the smoothing
> constant using the minimizing
>
> the MAE criterion, and at step two checks the process labeled as
> stable from step 1 for relative deviation of latest observed data with
> the previous value. This two step process thus identifies the series
> as \"stable\" or \"fluctuating\" (step 1: produce an edit if
> \"fluctuating\"), and then at step 2, if \"stable\", check relative
> distance between latest
>
> observation at time t, and observation at t-1.

4.0 COMMENTS ON INITIAL VIEW OF THE DATA

An initial review of the F906 ten-year historical data showed that there
were data inconsistencies that will affect the edit testing. The data
inconsistencies will be reported as errors. The data inconsistencies
appear to be confined to the most recent years, giving the appearance
that there was a system change that allowed or created the
inconsistencies. The specific inconsistencies are:

-   Starting in 2001, it appears that the F906 system began accepting
    dual case data. Thus, fields such as plant code, fuel name, state
    name, and county name are now shown in upper and lower case.

-   Starting in 2001, there are different names for the same fuel type.
    The duplication is shown by reviewing the relation between fuel name
    and the numeric fuel type.

-   Prior to 1999, a 2 digit year was apparently used. When the change
    was made to a four-digit year, it appears that the data for prior
    years was not adjusted. This difference appears to have resulted in
    a secondary inconsistency. The stockage data for four digit years
    has been redacted by inserting all 9's. Where a two-digit year is
    present, the stockage data has not been redacted.