# Presentation: 465991

## PART 2: Tool Development

- Joe Cassmassi
- South Coast Air Quality Management District

## Tool/Model Background
Developing a Database
Checking the Database
Creating a Model
Regression
Decision Trees
Model Evaluation
Alternative Models
Daily Operation

- Overview

- Tool/Model Background
- Developing a Database
- Checking the Database
- Creating a Model
- Regression
- Decision Trees
- Model Evaluation
- Alternative Models
- Daily Operation

**Notes:**
For each subject discussed, an example will be provided with its URL.  At the end of the prepared slides are a series of pages containing lists of links for each category.

## Model Background –Why Build or Revise a Model?

- Provides a tool for the forecaster that translates weather to a pollution prediction
- Speeds up the forecast process
- Levels the playing field
- Algorithms set bounds on the prediction
- Minimize bias among a group of forecasters
- Addresses the changing air quality trend – empirical relationships are outdated
- Incorporates new sources of meteorological or air quality data

## Model Background – Basic Model Structure

- Subjective day-in-advance prediction (phenomenological forecasting using conceptual models)
- Objective same-day prediction with perfect meteorological forecast
- Objective day-in-advance prediction (with/without same-day model)
- Subjective override – fine-tune prediction

**Notes:**
Nothing improves forecasts better than experience informed by hindsight.  A good daily image archive is critical to the forecast improvement effort.

## Model Background – General Conceptual Model 

- Define the mechanics of the pollutant episode by evaluating previous events
- Identify key predictor variables
- Surface: T, Td, RH, dd, ff
- Aloft: HHH, T, Td, RH, dd, ff
- Air quality persistence
- Determine timing of data availability
- Observations before forecast preparation
- Prognostic forecast verification periods

## Model Background – Conceptual PM2.5 Model for L.A. Basin

- Gradient of particulates highest in east basin
- PM2.5 dominated by secondary PM2.5 (nitrate, ammonium and organic carbon [OC]) formed through photochemistry or heterogonous chemistry
- Sulfate, elemental carbon (EC), and crustal contribute less

##  Conceptual PM2.5 Model for L.A. Basin Spatial Distribution 

## Conceptual PM2.5 Model for L.A. Basin Temporal 

- Seasonal
- All species highest during fall stagnation
- OC also high mid- and late summer
- Weekend effect
- Altered VOC/NOx ratio Saturday and Sunday
- Lower EC (diesel) on weekend days

## Basin PM/Ozone Model – Data Requirements

- Surface:  O3,  NO2, PM10, PM2.5, T, RH,  PPP-Gradients
- Aloft: Height (HHH), T, Td, RH (950, 900, 850, 700, 500 mb surfaces)
- Inversion characteristics (top and base HHH and T), stability, thickness
- Prognostic Grid Point FOUS and MOS surface T and RH predictions

## Developing a DatabaseHow Much and What Types of Data?

- For statistical analysis, two years minimum
- Data obtained from many sites
- Determined by size and characteristics of forecast domain
- Follow the conceptual model definition
- Use all available data and let the analysis                       sort the key predictors
- Include predictor variables for multiple pollutants  (common predictors for O3, PM10 and PM2.5)
- Include other pollutants (may act as surrogates)

## Developing a Database Locate Sources of Data

- Internally generated air quality
- FSL – upper-air
- Universities  (e.g., Wyoming, OSU, PSU, Plymouth State)
- Sister agencies (APCDs, water districts, RAWS, BLM)
- NCDC

## FSL RAOB

## Decoded University of Wyoming RAOB

## Decoded Surface Observations

## AQMD Temperature Data

## FOUS Data  

## AQMD Hi-Vol PM10 Data

## AQMD Hi-Vol PM2.5 Data

## Developing a Database Common Linear Format

- Most statistical software require a linear format – one line of data for each day (see Session 1B Part 1)
- The data must be structured (lined up) based on time considerations
- Air quality data for preceding, current, and next day (d-1, d0, d+1)
- Meteorological observational data for time periods  prior to expected forecast preparation/issuance time
- Prognostic data (or surrogate prognostic data) for the next day

## Developing a Database Common Predictor/Stratifying Variables

- Day-of-week indicator		        (numerical 1,2...7 or binary 0,1)
- Julian day of year (1-365)
- Number of hours in day 	           (function of Julian day and latitude)
- Holiday marker (binary 0,1)

## Developing a Database Translating and Reformatting the Data

- Only critical rule is to line up the data chronologically (remember one day’s data on one record)
- Write simple programs to extract and massage data (Fortran, Basic, C++,...)
- Spreadsheets and databases are good platforms for merging individual or groups of products and aligning dates
- Use as many tools as necessary – use programs and spreadsheets

## Developing a Database – Example 

- Extracting Upper-Air Data From an FSL-formatted RAOB Archive

## Base FSL Format 

## Strip Header, Grab Time/Date, Eliminate Unneeded Data

## Convert Date to Numeric

## Excel Date Formatting

## Extracting Data Using Numeric Date,  Time, and Pressure Tags

- Extrapolate data
- vertically (surface to 800 mb)
- with time (00Z and 12Z soundings)
- Interpolate throughout year for missing data

## Relative Humidity,Temperature, and Heights

## Align Air Quality Data 

## Missing Data

- Missing data must be marked in the data set
- If possible, fill in spurious gaps in the data through extrapolation
- Minimize the use of variables that have large or routine gaps in the data
- Statistical software typically eliminates a record from the analysis if it includes a variable with missing data

## Checking the Database

- Use a spreadsheet or canned programs to assess the data
- Basic statistics
- Maximum, minimum, and range
- Average
- Correlation matrices
- Graphical presentations to identify unique data and outliers
- Time series plots
- Bivariate data plots

## Checking the Database Data Analysis

- Examine trend
- Scan data for outliers
- Note unusual or exceptional cases

## Checking the Database Analyze Scatter Plots

## Creating a Model

- Overview
- Stepwise Multivariate Regression
- Decision Trees
- Model Evaluation

## Overview –Selecting Statistical Software

- Availability, platform, and cost
- What statistical algorithms are provided?
- Data transformation
- Fixed format (flat ASCII)
- Spreadsheet or variable format (.xls)
- Translation software (DBMS Copy)
-  Windows-based?
-  Output formats
-  Ease of use

## Overview –Common Statistical Packages

- Excel (Basics)
- SYSTAT
- BMDP
- SASS
- SPSS
- CART
- Neural Networks

## Stepwise Multivariate Regression (1 of 2)

- Data Considerations
- The number of cases is more important than the number of predictors
- Two years of data at minimum
- Select the appropriate predictor variables
- Missing data limits number of cases
- Sort data into dependent and independent data sets for validation

## Algorithm Fitting Criteria 
Software sets a default criteria for developing empirical model (probabilistic, f-level to enter...)
Lowering the default criteria to allow variables into the algorithm development will result in over-fitting the data
Small data samples (only a few records) can result in over-fitting as well

- Software sets a default criteria for developing empirical model (probabilistic, f-level to enter...)
- Lowering the default criteria to allow variables into the algorithm development will result in over-fitting the data
- Small data samples (only a few records) can result in over-fitting as well

- Stepwise Multivariate Regression (2 of 2)

## Regression Model Evaluation

- Evaluate R, R2, and error
- Considerations
- What variables are included in the algorithm?
- Does the algorithm make physical sense?
- Variables selected consistent with conceptual model?
- Directional sign of coefficients consistent?
- Assess variable contribution to prediction
- Value of intercept
- Weight of coefficients

## Stepwise Multivariate Regression – Excel Database (1 of 2)

## Stepwise Multivariate Regression – Excel Database (2 of 2)

## Stepwise Multivariate Regression –  Data Imported into SYSTAT

## SYSTAT – Variable Sorting

## SYSTAT – Multivariate Regression

## SYSTAT – Adding Stepwise Option

## SYSTAT – Regression Equation, Fit, and Statistics

## Interpreting the Equation

| Variable | Coefficient/Intercept | Contribution to PM2.5 Formation Process |
| --- | --- | --- |
| CONSTANT | -60.216 |  |
| HOURS | -3.111 | Compensates for summer insolation |
|        H8VBG | 0.634 | Stagnation |
|        T8VBG | 0.512 | Stagnation |
| SUMPG | 1.568 | Westerly onshore flow |
| LAXSFO | -0.782 | Southerly onshore flow |
| SANLAS | -3.379 | Competing with SUMPG |
| T1000 | -1.265 | Warm temps greater mixing |
| T950 | -0.626 | Competing T8VBG or RH1 |
| RH950 | 0.130 | Stratus  or  saturated layer  |
| RH850 | -0.150 | Very deep mixing |
| RIVRO32 | 0.503 | Summer ozone (up to 10 μg/m3) |
| RIVRNO22 | 3.623 | Fall NO2 contribution |
| RIVRRH1 | 0.237 | Surface heterogeneous chemistry |
| R1 | 0.105 | Surface heterogeneous chemistry |
| T2 | 0.471 | Photochemistry indicator |
| UCRIT2 | 0.117 | PM10 trend contribution  |

## Decision Tree – Options to Consider

- Variable transformations
- Case selection
- New variables
- Missing data less an issue
- Address algorithm fitting criteria

## SYSTAT – Growing the Tree

## SYSTAT – Tree Statistics, Splits, and Nodes

## Deep Mixing

*Deep Mixing*

*Sea Breeze Flow*

*Dry Offshore Flow*

*Limited Persistence*

*Nov - Feb*

*Mar - Oct*

*Moist Surface Air Mass*

*Dry Air Mass*

*Moderate*
*Persistence*

## Understanding the Tree

- Do the splits make physical sense?
- Does the structure fit the conceptual model?
- Can the tree be further pruned?
- Does the tree need to be grown?
- Can the tree be used directly as a forecast tool?

## Model Evaluation – Evaluating the Branching and Terminal Nodes

| Node | PM2.5 μg/m3  | AQI | Percent Distribution | Description |
| --- | --- | --- | --- | --- |
| 1 | 13.7 | G | 18.6 | Deep mixing with limited persistence  |
| 2 | 10.1 | G | 4.0 | Dry offshore flow with limited persistence (Santa Ana)  |
| 3 | 25.7 | M | 1.7 | Onshore flow with moderate mixing and limited persistence  |
| 4 | 10.0 | G | 4.0 | Dry offshore low with moderate persistence  |
| 5 | 31.1 | M | 39.4 | Nov-Feb heterogeneous nitrate chemistry  |
| 6 | 43.4 | US | 5.7 | Mar-Oct combined photochemical and heterogeneous chemistry  |
| 7 | 45.3 | US | 10.6 | Stagnant air mass with fog |

## Model Evaluation – General

- Assess performance using development and independent data sets
- Set a target for model performance
- Contingence matrix for threshold analyses (>50%)
- Error (10% of maximum observed concentration)
- Bias as near to zero as desired

## Model Evaluation –  PM2.5 Regression Model Performance

- Model tendency:
- underpredict

## Alternative Models

- Combinations of CART and regression
- Discriminate or cluster analyses to group similar air quality events
- Factor analysis
- Nearest neighbor
- Neural networks

## Daily Operation

- The model/algorithm is a tool!
- Bad or missing input data will be   reflected in the prediction
- Adjustments are required for special circumstances and timing
- Forecasters blend experience with model output to generate the final product
- Have fun!