# Presentation: 671883

## ASA/IEA Meeting
 October 19, 2007

William Watson
Office of Coal, Nuclear, Electric & Alternate Fuels

**The Processes CNEAF Follows  To Assure Data Quality: **
**The Case of Coal** *

**ASA/IEA Meeting**
** ****October 19, 2007**
**William Watson**
**Office of Coal, Nuclear, Electric & Alternate Fuels**

-  * This is a working document prepared by the Energy Information Administration (EIA) in order to solicit advice and comment on statistical matters from the American Statistical Association Committee on Energy Statistics. This topic will be discussed at EIA's fall 2007 meeting with the Committee to be held October 18 and 19, 2007.”

## CIDC Deployment

| Survey | Population | Frequency | Deployed | Survey Cycle |
| --- | --- | --- | --- | --- |
| EIA-6A | Distributors | Annually | 12/04 | 2005 |
| EIA-7A | Coal Mines | Annually | 12/03 | 2003 |
| EIA-3 | Mnfg. Plants | Quarterly | 03/04 | 2004 Q1 |
| EIA-5 | Coke Plants | Quarterly | 03/04 | 2004 Q1 |

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## Survey Response Rates

| Survey | Period | % of Frame | % IDC | % of Current Year Production |
| --- | --- | --- | --- | --- |
| EIA-7A | 2002 | - | 0% | - |
|  | 2003 | - | 55% | - |
|  | 2004 | 90% | 63% | 98% |
|  | 2005 | 94% | 75% | 99% |
|  | 2006 | 94% | 74% | 99% |
| EIA-6A | 2004 | 94% | 0% | - |
|  | 2005 | 94% | 53% | - |
|  | 2006 | 99% | 64% | - |
| EIA-3 | All Qs since 2004 | 100% | >95% | - |

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## Frame Design & Maintenance

- Specify threshold
- Establish a base frame
- Revise frame
- Births
- Deaths

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## Data Survey Forms

- Instructions
- User-friendly interface and flow to simplify process and minimize human error
- For example, drop-down lists, standardized names
- Automated accounting checks
- Respondents resolve imbalances
- Automated edit flag checks
- Respondents resolve invalid data edit flags by entering correct value or an override comment

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## Respondent Contact Record Tool (RCRT)

- Tracks unresolved edits & their resolution
- All forms with data edit flags are given a default “Pending” status
- Includes every IDC form submitted with an override comment
- Contractors review comment, perform “Fast Acceptance” or follow up with respondent by phone or e-mail
- Federal staff review “Fast Acceptance” cases and identify forms for recheck
- Survey closed when all RCRT statuses are “Completed”

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## Level 3 Checks

- Survey data sorted by specific fields from lowest to highest value to identify outliers
- Example: Respondent reports 0.04 as sulfur value, value fails edit range check of 1<S<5, respondent enters override comment: “value as reported by lab,” contractor and Federal staff accept comment as valid. The value should have been entered as 4, representing 4 percent.

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## Level 3 Checks

- Data survey fields searched for “null” entries
- Page 1 – page 2 comparables checked
- Page 1 has total receipts
- Page 2 has receipts at detailed level
- Sum of page 2 receipts should equal page 1 receipts

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## QA Guidance Documents & Process

- Checklist of all process steps, QA, etc.
- Track progress of survey
- Document issues & problems
- Recommend changes
- Sign-off at end of reporting cycle
- Submit to CNEAF QA officer

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## QA Guidance Documents & Process

- Weekly or bi-weekly status review
- Response rate charts
- Clean-up progress charts
- End-of-cycle wrap-up review
- Review of what worked
- Review of what didn’t work
- Review of what needs fine tuning

**Notes:**
Top 6 countries produce > 82%
China & US alone produce 60%
In 2030, top 6 will produce 85% and China & US will produce 64%

## EIA-3 Responses

**EIA-3 Manufacturing Survey 2007 Q1**

**Days Into Reporting Cycle**

## EIA-7A Responses

**EIA-7A Coal Mine Survey 2006**

**Days Into Reporting Cycle**

## Edit Resolution in Practice

**Fast Acceptance**
**672**

**Respondent Contact**
**248**

**Fast Acceptance**
**553**

**Respondent Contact**
**1057**

**Fast Acceptance**
**1328**

**Respondent Contact**
**289**

**920 Edit Flags**

**1609 Edit Flags**

**1617 Edit Flags**

**EIA 3**
**2000 Respondents**
**(on an annual basis)**

**EIA 7A**
**1540 Respondents**

**EIA 6A**
**960 Respondents**

**Well-Honed**
**System**

**Low-IDC**
**Enrollment**

**Unneeded Edit**
**Flags; Need **
**More Edits **
**Cleared By **
**Respondent**

## IDC enrollment > 95%
Self-editing by respondent
Complete and balance data to progress through form
Data edits against past values and industry standards
Fast compilations of edit flag status
Fast review/resolution of pending edits
Cross survey comparison

- Self-editing by respondent
- Complete and balance data to progress through form
- Data edits against past values and industry standards
- Fast compilations of edit flag status
- Fast review/resolution of pending edits
- Cross survey comparison

- Micro Strategies to Improve Quality

## Frame maintenance – EIA3
Industry newsletters and reports (new coal users)
Inventories to implement environmental regulations
Cross reference to other surveys
Aggregate Balance
Production + Imports + Stock Change = Consumption + Exports
OR
Supply = Demand
Challenge: Supply routinely exceeds demand

- Industry newsletters and reports (new coal users)
- Inventories to implement environmental regulations
- Cross reference to other surveys
- Aggregate Balance
- Production + Imports + Stock Change = Consumption + Exports
- OR
- Supply = Demand
- Challenge: Supply routinely exceeds demand

- Macro Strategies to Improve Quality

## What weaknesses do you see in our QA process?
What can we learn about data survey methods from academic sources?
Are there best practices we should know and follow?
How can we assure continued quality improvement?

- What can we learn about data survey methods from academic sources?
- Are there best practices we should know and follow?
- How can we assure continued quality improvement?

- Questions to ASA Committee