# Presentation: 272491

## Evaluate Earth ScienceModels for What They are- Cartoons of Reality

- John S. Irwin, NOAA Meteorologist
- EPA OAQPS
- Air Quality Modeling Group
- RTP, NC 27711

## Model Evaluation Background

- Linear regressions (Clarke, 1964), Marin (1971).
- EPA Guideline on Air Quality Models, EPA-450/2-78-027, (OAQPS, 1978), Revised (1980, 1994, proposed 2001).
- National Commission on Air Quality Panel examines uses and limitations of air quality models, (Fox and Fairobent, 1981) BAMS(62):218-221.
- September, 1980: Woods Hole Workshop.  Judging air quality model performance, (Fox, 1981) BAMS (62):499-609.
- Framework for evaluating air quality models, (Venkatram, 1982) BLM (24):371-385.

## Model Evaluation Background(cont.)

- 1984
- Uncertainty in air quality modeling, (Fox, 1984) BAMS (65):27-36.
- Review of the attributes and performance of 10 rural diffusion models, (Smith, 1984) BAMS (65):554-558.
- Potentially useful additions to the rural model performance evaluation, (Irwin and Smith) BAMS (65):599-568.
- 1988: Air quality model evaluation and uncertainty (Hanna, 1988) JAPCA (38):406-412.
- 1989: Confidence limits for air quality model evaluations as estimated by bootstrap and Jackknife resampling methods.  (Hanna, 1989) AE(23):1385-1398.

## Model Evaluation Background(cont.)

- 1990: A statistical procedure for determination of the best performing air quality simulation model, (Cox and Tikvart) AE(24A):2387-2395.
- 1994: Verification, validation, and confirmation of numerical models in the earth sciences.  (Oreskes et. al., 1994): Science (263):641-646.
- 2000: Standard Guide for statistical evaluation of atmospheric dispersion model performance, ASTM, D 6589-00, 16 pages.

## Examples

- Examples:
- Gaussian Plume (Irwin et al., 1987)
- Grid Model (Hanna et al., 1998, 2001)
- Transport Direction (Weil et al., 1992)

## Slide 6

## Slide 7

## Computed correlation coefficints (r) forRun 26 Green Glow, 2100-2130 PST,August 28, 1959.

## Unexplained Variability& Simple versusComplex Modeling

## Evaluation

## My answer: We need an evaluation procedure,that uses statistical evaluation test methods to:

- Determine which of several models in performing best (for the data available) _the physics within the model_:
- This will be defined as the “best performer.”
- Determine whether the differences seen in the  performance of the other models is statistically significant (in light of stochastic variations present in the data comparisons).
- This will identify other models that may be performing as well (a set of ‘best’ performers).
- I call this the _Olympics “High Bar”Analogy_.  This is an open (hopefully fair) competition, in which the rules are known and the conclusions reached are objectively determined.

## My answer: (cont.)

- Once the ‘set of best performers’ has been defined, then a new set of statistical evaluation test methods would be used to determine, which of these models _best performs the user-defined tasks_.
- Again, define the ‘best’ performer.
- Then, test to see if differences in performance are statistically significant.
- This sequence recognizes that models are used for situations for which they _do not_ have the requisite physics.
- _If you test ONLY for the user defined tasks, you likely will end up with perverted models whose results are “tuned” to the data at hand, that may well provide erroneous results when used operationally_.
**Example**: Pasquill dispersion sigmas have a 3-minute averaging time.  They are tested in their ability to replicate 1-hr, 3-hr concentration extremes, and then applied to produce annual averages.

## Promising Test methods:

- _Grouped Data_:
- (Irwin and Smith, 1984), ASTM (2002).
- _Decomposed Time Series_:
- Eskridge, R.E., Ku, J.Y., Rao, S.T., Porter,P.S., Zurbenko, I.G., (1997)BAMS (78):1473-1483.
- Rao, S.T., Zurbenko, I.G., Neagu, R., Porter, P.S., Ku, J.Y., Henry, R.F., (1997), BAMS, (78):2153-2166
- _Process Analysis_:
- Dennis R.L. (1986), Air Pollution Modeling and its Application V, Plenum Press, pp. 411.-424.
- Dennis, R.L., Arnold, J.R., Tonnesen, G.S., Y. Li (1999): Computer Physics Communications, 117:99-112.

## All Models of PhysicalProcesses are Cartoons ofReality

**Models Simulate Only a Portion of the Natural Variability.  They Do Not Simulate What Is Directly Seen.**
**FIRST:** Test a Model to Accurately Perform the Physics Within It
**THEN:** Test a Model to Perform Some User-defined Task (Which More Often Then Not Is Beyond the Capabilities of the Physics Within the Model).
- All “test methods” should provide a test of  whether differences between several models are statistically significant.
- _All “test methods” and test data sets should be peer reviewed and public domain_.