# Presentation: 950009

## The AMP255 and Statistical Data AssessmentsAugust 19, 2008

*Presented by Jonathan Miller*
*EPA - Office of Air Quality Planning and Standards*
*Presentation Created by Louise Camalier & Jonathan Miller*

*CV*

## Agenda

- How We Got Here (10 Minutes)
- Brief History of the AMP255
- Intended Usage of the AMP255
- Review of 40 CFR 58 Appendix A (60 Minutes)
- The Statistics
- Data Quality Indicators (DQI’s)
- How to Generate the AMP255 (10 Minutes)
- BREAK (15 Minutes)
- Morning Review (10 Minutes)
- What Am I Looking At?!?! (60 Minutes)
- Walkthrough Each Report
- Field & Value Definitions
- Known Bugs & Future Enhancements (10 Minutes)
- Questions and Answers (10 Minutes)

## Part I:  Review of 40 CFR Part 58 Appendix A

- Quantifying “Good ‘nuff”

## AMP255 History

- February 2004:  EPA Releases “2002-2004 Criteria Pollutant Precision and Bias Data”
- 1st Publication by EPA Showing Results of Proposed Changes in the P&B Statistics
- Subsequent Report Released in December 2004

## AMP255 History (Cont)

- OAQPS QA Team Proposed Funding the Development of the Annual Report as a Standard Report in AQS
- Development Starts in May 2005
- Development Completes in October 2005
- Used in Annual Assessment Report December 2005
- January 17, 2006:  40 CFR Part 58 Appendix A Revisions Go FINAL
- Last Minute Revisions NOT Accounted for in AMP255

## AMP255 Intended Usages

- Save $$$:  Report was Based on the Annual Assessment Report Which Was Created by Contractors
- Needed a Mechanism to Produce Assessments in AQS to be Consistent with the New Appendix A Rules
- Replacement of the AMP240 Report Beginning in 2008 for 2007 Data Annual Certification

## Limitations with AMP255

- Only for Criteria Pollutants
- Only Covers What is in Appendix A
- Non-Criteria Report Under Development
- Known Issues
- Some Calculations are Based on the Proposal and Not What Was Finalized
- Formatting Issues

## Percent Difference &Relative Percent Difference

- Basis for All Statistical Calculations
- For standard gas
- For Collocated Samples,

## What Are We Trying to Determine?

*Precision*

*Bias*

- “A measure of mutual
- agreement among
- individual measurements
- of the same property,
- usually under prescribed
- similar conditions”

- “A systematic or persistent
- distortion of a
- measurement process
- which causes errors in one
- direction”

## Another Way of Looking at It...

*# of Observations*

*% Difference*

*0*

*+10*

*-10*

*Ideal:  Precise w/o Bias*

*Precise w Negative Bias*

*Precise w Positive Bias*

*Low Precision w/o Bias*

*Low Precision w Negative Bias*

*Low Precision w Positive Bias*

## Precision

***Gaseous Estimate***

**Precision is more conservative when evaluated at**
**the 90% one-sided upper confidence level**

*Defines the “Width” of the Bell Curve*

*Denominator changes because we do not have “truth”*

*90**th** Percentile of a Chi-Squared Distribution **** *_***More Conservative***_

## Bias

*Central Tendency*

*95**th** Quartile of a Student-t Distribution w/ n-1 df*

*Standard Error*

## Assigning a Sign to Bias

***# of Observations***

***% Difference***

***0***

***+10***

***-10***

*25**th** < 0 < 75**th**  **	**** “+/-”*
*25**th** Percentile is >0 **** “+”*
*75**th** Percentile is <0 **** “-”*

*25**th **%tile        75**th** %tile*
*‘**+/-’ (no sign)*

*25**th **%tile        75**th** %tile*
*‘**+’ (positive sign)*

*25**th **%tile        75**th** %tile*
*‘**-’ (negative sign)*

## Validation of Bias

*Lower Limit*

*Upper Limit*

*Upper Probability Limit = m + 1.96*S*

*Lower Probability Limit = m + 1.96*S*

*“**d”s for the Results of the *
*Annual Performance Evaluation*

*95%*
*of data*

*2.5% of data in both tails*

## CV= 0%

*Signed Bias= 0%*
*Upper Probability Limit= 0%*
*Lower Probability Limit= 0%*

*CV= 0%*
*Signed Bias= -5%*
*Upper Probability Limit= -5%*
*Lower Probability Limit= -5%*

*CV= 7.3%*
*Signed Bias= +/- 5%*
*Upper Probability Limit=  10.24*
*Lower Probability Limit= -10.24*

## The DQI’s

- The Statistics
- Gaseous Precision & Bias Assessments
- Precision Estimate from Collocated Samples
- PM2.5 Bias Assessment
- PM2.5 & PMc Absolute Bias Assessment
- One-Point Flow Rate Bias Estimate
- Semi-Annual Flow Rate Audits
- Lead Bias Assessments

## What is Performed

| Method | Pollutants | Frequency | MQO |  |
| --- | --- | --- | --- | --- |
| One-Point QC | SO2, NO2, O3, CO | Every 2 Weeks | O3 : 
Precision =  7%                    Bias = 7% | SO2, NO2, CO :
Precision = 10%
Bias = 10% |
| Annual Performance Evaluation | SO2, NO2, O3, CO | Once per Year @ 3 Levels | <= 15% for each audit concentration |  |
| Flow Rate Verification | PM10, PM2.5, PMc | Once per Month (Quarterly for TSP & PM10 High Vols) | <= 4% of Standard & 5% of Design Value |  |
| Semi-Annual Flow Rate Audit | PM10, PM2.5, PMc | Every 6 Months | <= 4% of Standard & 5% of Design Value |  |
| Collocated Sampling | PM10 (Manual), TSP, PM2.5, PMc | 15% of Network Every 12 Days | PM2.5, TSP, PM10 :
Precision = 10% | PMc:
Precision = 15% |
| PM PEP Program | PM2.5, PMc | If <= 5 Sites: 5/yr
If > 5 Sites: 8/yr
All monitors covered in 6 years | PM2.5:
Precision = 10% | PMc:
Precision = 15% |

## How it is Entered

| Method | Pollutants | Frequency | MQO |  |
| --- | --- | --- | --- | --- |
| One-Point QC | SO2, NO2, O3, CO | Every 2 Weeks | O3 : 
Precision =  7%                    Bias = 7% | SO2, NO2, CO :
Precision = 10%
Bias = 10% |
| Annual Performance Evaluation | SO2, NO2, O3, CO | Once per Year | <= 15% for each audit concentration |  |
| Flow Rate Verification | PM10, PM2.5, PMc | Once per Month | <= 4% of Standard & 5% of Design Value |  |
| Semi-Annual Flow Rate Audit | PM10, PM2.5, PMc | Every 6 Months | <= 4% of Standard & 5% of Design Value |  |
| Collocated Sampling | PM10, TSP, PM2.5, PMc | 15% of Network Every 12 Days | PM2.5, TSP, PM10 :
Precision = 10% | PMc:
Precision = 15% |
| PM PEP Program | PM2.5, PMc | If <= 5 Sites: 5/yr
If > 5 Sites: 8/yr
All monitors covered in 6 years | PM2.5:
Precision = 10% | PMc:
Precision = 15% |

*“**Precision”*

*“**Accuracy”*

*“**Precision”*

*“**Accuracy”*

*“**Precision”*

*“**Precision” – Only Entered by RTI...  Not by all Users*

## The AMP255

- Making Sense of the Madness...

*Or adding to it...*

## Generating an AMP255

*Zip File*

*1 XLS File (AMP255.xls)*
*1 README File*
*1 PDF File (your selection)*
*35 CSV files*

*Download to*
*Local Directory*

*Open “AMP255.xls”*

*Agencies*

*Gaseous QI*
*Completeness*
*Tables*

*Lead*
*Tables*

*PM2_5*
*Tables*

*PM10*
*Tables*

*Precision*
*And Accuracy*
*Tables*

*R31 –*
*Standard Report*
*Selection Form*

*To Use Excel*

## Things to Do Beforehand

- Check Your Excel Security Settings
- Enable the Macros When Prompted
- Indicate you want to Load the Summary Data When Prompted (why else are we here?!?!)

## Common Errors on Opening

- You Tried to Open the AMP255.xls From the Zip File
- At Least 1 of the 35 csv files are missing from the Directory.

## Live Demonstration

- The Point in the Program Where it Can all go Horribly Wrong...

## Part II – Going Through the AMP255

## Common Terminology

- PQAO
- Primary Quality Assurance Organization
- Organization Responsible for the QA Activities for a Monitor
- NSP vs Other
- NSP = “NAMS / SLAMS / PAMS” Monitor Type
- Anything other than “NAMS / SLAMS / PAMS” Monitor Type
- Applies only to the “Highest Ranking” Monitor Type Defined for the Monitor
- The “M” After a PQAO Organization
- The Agency Operates in Multiple States (National Park Service for Example)

## Gaseous QI Completeness

- Single Point Quality Check Completeness
- Based on One-Point QC Checks
- AQS: PRECISION
- Performance Evaluation Check Completeness
- Based on Annual Performance Evaluations
- 3 Consecutive Levels
- AQS: ACCURACY
- Reports Repeated for O3, CO, SO2, NO2

## Single Point QC Completeness

- Primary Monitor Type:
*NAMS / SLAMS / PAMS = “SLAMS”*
*Anything Else without one of those 3 = “OTHER*

*Date Range Based on:*
*Sample Periods*
*Monitor Type*
*PQAO Dates*

*Number Required = **Number of Days Between (“Start Date” & “End Date”)/ 14*

*Number Submitted = *
*Number Precision Checks Between “Start Date” & “End Date”*

*Count of “Comp %” < 50*

## PE Check Completeness

*The (x) value indicates the number of Levels required*

*Count of the number of Audits Performed.*
*(Count of the number of Levels Performed)*

*Number of Sites*
*Within the Group*

*Number Audits Performed per Quarter Within the Group*

## Precision and Accuracy Tables

- Single Point P&B Estimates
- One Sheet per Pollutant / MT Group
- Summarized by PQAO
- PQAO Accuracy Summary
- One Sheet per Pollutant / MT Group
- One Line per PQAO
- Reports Repeated for
- O3, CO, SO2, NO2
- NSP / Non-NSP Monitor Types

## Single-Point P&B Estimates

*Upper Probability Limit = m + 1.96*S*

*Lower Probability Limit = m - 1.96*S*

## PQAO Accuracy Summary

*Upper Probability Limit = m + 1.96*S*

*Lower Probability Limit = m + 1.96*S*

## PM2.5 Tables

- PM2.5 Collocation Summaries
- Only Shows for “Primary Monitors”
- PM2.5 Flow Rate Completeness

## PM2.5 Collocation Summaries

*Number Required = **Number of Days Between (“Start Date” & “End Date”)/ 12*

*Number Submitted = *
*Number Collocation Precision Checks Between “Start Date” & “End Date”*

*Number of Sites*
*Within the Group*

*Number of Sites with “Primary” Designation*

*Number of Pairs wher both value > 6 ug/m3*

## PM2.5 Flow Rate Completeness

*Number Required = **Number of Months Between (“Start Date” & “End Date”)/ 6*

*Number Submitted = **Number of Accuracy Flow Audits in the Period*

***Number Audits Performed per Quarter Within the Group***

***Number Sites Within the Group***

## PM10 Tables

- Automated PM10 Precision Completeness
- Based on Flow Audits
- Manual PM10 Precision Completeness
- Based on Collocation
- Automated / Manual PM10 Accuracy

## Automated Precision

*Number Required = **Number of Months Between “Start Date” & “End Date”*

*Number Submitted = **Number of Precision Flow Check Days in the Period*

*Count of “Comp %” < 50/# Sites*

## Manual Precision

*Number Required = Number of Days Between (“Start Date” & “End Date”)/ 12”*

*Number Submitted = **Number of Precision Flow Checks in the Period*

*Count of “Comp %” < 50/# Sites*

***# Sites That Submitted Data (Should be 15%)***

***Number Sites Within the Group***

*Number of Pairs wher both value > 20 ug/m3*

***Upper Probability Limit = ***
***m + 1.96*S***

***Lower Probability Limit = ***
***m - 1.96*S***

## Accuracy Completeness Tables

*Number Required = **Number of Months Between (“Start Date” & “End Date”)/ 6*

*Number Submitted = **Number of Accuracy Flow Audit Days in the Period*

***Number Audits Performed per Quarter Within the Group***

***Number Sites Within the Group***

## Lead Tables

- Lead Collocation Completeness
- Based on Collocated Precision Data
- Lead Flow Rate Completeness
- Based on Accuracy Flow Data
- Lead Filter Strip Completeness

## Lead Collocation Completeness

*Number Required = Number of Days Between (“Start Date” & “End Date”)/ 12*

*Number Submitted = **Number of Precision Flow Checks in the Period*

*Count of “Comp %” < 50/# Sites*

***Number Sites Within the Group***

***# Sites That Submitted Data (Should be 15%)***

## Flow Rate Completeness

*Number Required = **Number of Months Between (“Start Date” & “End Date”)/ 6*

*Number Submitted = **Number of Accuracy Flow Audits in the Period*

***Number Audits Performed per Quarter Within the Group***

***Number Sites Within the Group***

## Filter Strip Completeness

*Number Required = *
*3 x Number of Months Between (“Start Date” & “End Date”)/ 4 @ Level 1*

*Number Submitted = **Number of Accuracy Audits in the Period @ Level 1*

*Number Submitted = **Number of Accuracy Audits in the Period @ Level 2*

*Number Required = *
*3 x Number of Months Between (“Start Date” & “End Date”)/ 4 @ Level 2*

## Known Issues

| Item | Indicator | Issue | What AMP255 Does | What App A Does |
| --- | --- | --- | --- | --- |
| Automated PM10 | Completeness | Required # Flow Rate Verifications | Every 2 Weeks | One / Month |
| Manual PM10 & PM2.5 | Precision Value | Minimum value to be used in Calculations | PM10 = 20
PM2.5 = 6 | PM10 = 15
PM2.5 = 3
 |
| Manual PM10 & PM2.5
 | Completeness | Flow Rate Audits
 | Once / Quarter | Every 6 Months |
| Manual PM10 , PM2.5, Lead
 | Collocation Completeness | Frequency and Number of Sites | 1 in 6 days; based on # sites in PQAO | 1 in 12 Days; 15% of the PQAO Network.  For PM, this is based on method designation |

## Known Issues

| Item | Indicator | Issue | What AMP255 Does | What App A Does |
| --- | --- | --- | --- | --- |
| Lead | Completeness | # Required Flow Rates | 1 per Year | 1 every 6 months |
| O3, SO2, NO2, CO |  | New Tables Need to be Used | Pre-2007 tables with only 4 levels | 5 Levels using new level ranges |
| All | Display | Formatting Issues with Upper/Lower Probability Limits | Program assumes the “,” between the values means it is two values. |  |