# Presentation: 674383

## AIRNow Forecast Verification Program

- Prepared by:
- Neal D. Conatser
- Sonoma Technology, Inc.
- Petaluma, California
- Richard A. “Chet” Wayland
- John E. White
- U.S. Environmental Protection Agency
- Research Triangle Park, North Carolina
- Presented to:
- U.S. EPA 2006 National Air Quality Conference
- San Antonio, Texas
- February 5-8, 2006

- STI-2857

## Outline

- Introduction, history, and present
- The AIRNow-Tech Forecast Verification System
- The future – feedback and additional considerations
- Questions, comments, and discussion

## Introduction	

- What is forecast verification?
- A comparison between a forecast and an observation
- Attempts to measure
- Forecast correctness compared to observed conditions
- The tendency for over predicting and under predicting
- Performance at predicting certain events (e.g., high AQI days)
- Helps ensure a high-quality product
- Learn from missed forecasts
- Quantify the improvements from new forecasting tools and techniques
- Establish program goals

## History

- Initial efforts of the AIRNow program were to score and rank agencies based on _percent correct_ (PC) and _submittal frequency_ of *next-day* forecasts (ozone and/or PM2.5)
- 2005 National Air Quality Conference
- AIRNow Forecasting Subcommittee

**Notes:**
Initial efforts too limited.  Why?
Only evaluated next day forecasts, ignoring forecasts issued multiple days in advance (e.g. over weekends).
Focused on scoring and ranking agencies against each other.
Only evaluated ability to forecast the exact AQI categories.
Did not provide a diverse array of statistics.
Did not offer agencies / forecasters  the ability to choose what was important to them.

2005 National Air Quality Conference
Presentation: Evaluating State/Local Forecast Performance: Lessons Learned and Future Direction
Introduced new statistics
Sought input on how forecast performance tools could be better implemented  into the AIRNow program

## Present

- Feedback revealed that forecasters wanted a more flexible measure of forecast performance and the ability to _choose_ statistics and outputs they considered important.
- Solution—creation of the new AIRNow-Tech forecast verification tool.

## Forecast Verification (1 of 2)

- AIRNow-Tech forecast verification tool
- Web-based query tool
- Accessible if you have an AIRNow account with forecaster privileges
- Select ***Forecasts*** from main menu, then select ***Verification*** from the submenu

## Statistical Methods:

Threshold - how well forecasts predict observations above and below a certain AQI category or AQI number

Categorical - how well forecasts predict observations in a particular AQI category or group of AQI categories

Discrete - how well numerical forecasts predict observed AQI numbers 

_**Threshold**_ - how well forecasts predict observations above and below a certain AQI category or AQI number
_**Categorical**_ - how well forecasts predict observations in a particular AQI category or group of AQI categories
_**Discrete**_ - how well numerical forecasts predict observed AQI numbers

- Forecast Verification (2 of 2)

**Notes:**
Can very based on what type of forecasts you make (I.e. categorical versus numerical).
Discrete forecasts are only applicable to those forecasts specifying an AQI number.
When launched, only threshold statistics will be provided, with the possibility of the addition of other methods in the future.
Either way, you will have the data at your fingertips, which will give you the ability to perform additional statistics on your own.

## Forecast Verification – Inputs

**Notes:**
Users can specify
Agency (one or all)
Date range
Pollutants (ozone, PM2.5, PM10, maximum AQI)
Threshold (e.g., Moderate/USG)
Forecasts to evaluate (e.g., next day)
For PM, observation method (e.g., midnight to midnight average or maximum 24-hr rolling average)

## Forecast Verification – Outputs (1 of 2)

**Notes:**
Outputs for each forecast city
Five statistics (percent correct, probability of detection, false alarm rate, critical success index, bias)
Threshold contingency matrix

## Forecast Verification – Outputs (2 of 2)

**Notes:**
Raw data (exportable as CSV file)
Forecast AQI and category
Observed AQI and category

## Metrics require thresholds.
Matrix will provide the number of forecast/observation pairings for each of the four possible combinations.
User has the option of choosing four different thresholds.  Also, the AQI number can be specified.
Metrics help answer the question, “How do we forecast above and below a certain level?”

- Matrix will provide the number of forecast/observation pairings for each of the four possible combinations.
- User has the option of choosing four different thresholds.  Also, the AQI number can be specified.
- Metrics help answer the question, “How do we forecast above and below a certain level?”

- Threshold Contingency Matrix

_**Observed**_

- Above

_**Forecast**_

- Below

- Above

- Below

- 19

- 109

- 16

- 9

## Forecast Verification – Metrics (1 of 5)

- _Percent correct (PC):_
- 	Shows the percent of all forecasts that match observations.  Includes easy low-pollution forecasts.  Also known as accuracy when calculating discrete statistics.

- 19 + 109

- 19 + 109 + 9 + 16

- X 100

-  = 84%

## Forecast Verification – Metrics (2 of 5)

- _Bias:_
- 	Indicates, on average, whether forecasts are under predicted (false negatives) or over predicted (false positives).

- 16 + 109

- 9 + 109

-  = 1.06

## Forecast Verification – Metrics (3 of 5)

- _False alarm rate (FAR):_
- 	Shows the percent of cases when a forecast of high pollution (at or above the threshold) was wrong.

- 16

- 16 + 109

- X 100 = 13%

## Forecast Verification – Metrics (4 of 5)

- _Critical success index (CSI):_
- 	Shows how well high-pollution events were predicted; unaffected by correctly forecasted, low-pollution events.

- 109

- 16 + 9 + 109

- X 100 = 81%

## Forecast Verification – Metrics (5 of 5)

- _Probability of detection (POD):_
- 	Indicates the ability to correctly predict high-pollution events (i.e., at or above a certain threshold).

- 109

- 9 + 109

- X 100 = 92%

## Metrics – Additional Information

- Perfect Statistics
- PC, POD, CSI = 100%
- FAR = 0%
- Bias = 1%
- Notes
- POD can be raised by forecasting high events all the time, but this could increase FAR and Bias, and lower CSI.
- FAR can be lowered by NOT forecasting high events at all, but this could decrease POD, Bias, and CSI.

## Threshold Contingency (1 of 2)

- Threshold Contingency (1 of 2)

- %

## Threshold Contingency (2 of 2) 

- Threshold Contingency (2 of 2)

- %

## Example Application (1 of 2)

- Example Application (1 of 2)

## Example Application (2 of 2)

- Example Application (2 of 2)

- Year

- Percent

## Future Direction

- Additional statistics
- Discrete statistics based on AQI numbers
- AQI category statistics
- Skill score
- Statistics based on last issue date, any issue date, and day of week
- New features
- Agency- and forecaster-specific statistics
- Measurement of forecasts against FRM data (in addition to existing options)
- Graphics
- Anticipated release mid-February 2006

## Resources

*Guidelines for Developing an Air Quality (Ozone and PM**2.5**) Forecasting Program*
- Section 5.6 – Forecast Verification
- Available for download from the AIRNow website.
- Side Menu > Resources > Publications > Outreach
- http://www.epa.gov/airnow/aq_forecasting_guidance-1016.pdf

## Questions and Discussion

- Thank you for your input.