DRAFT FOR INTERNAL REVIEW May 27, 2009

EPA-454/R-09-XXX

Draft 2009



## Reassessment of the Interagency Workgroup on Air Quality Modeling (IWAQM) Phase 2 Summary Report:

Revisions to Phase 2 Recommendations

U.S. Environmental Protection Agency Office of Air Quality Planning and Standards Air Quality Analysis Division Air Quality Modeling Group Research Triangle Park, North Carolina 27711

National Park Service

Air Resources Division Denver, Colorado 80225

U.S. Fish and Wildlife Service Air Quality Branch Denver, Colorado 80225

Draft 2009DRAFT FOR INTERNAL REVIEW May 27, 2009

ii NOTICE

The information in this document is in DRAF T form and is still under review by the U.S. Environmental Protection Agency (EPA) and the Federal Land Managers (FLMs). The draft revisions to the IWAQM Phase 2 recommenda tions presented herein are still undergoing internal testing to assess their viability for mee ting the technical objectives of this reassessment. Some sections are still under development and will be incorporated in future updates to the

DRAFT document.

This DRAFT document is being made available at this time to provide additional technical information in support of the May 15, 2009 M odel Clearinghouse recommendations to U.S. EPA Region 8 regarding the Otter Tail BART modeling protocol, to inform the modeling community of our concerns regarding the CALP UFF modeling system for long range transport (LRT) applications, and to notify the community of our plans for addressing these concerns.

Mention of trade names, products, or services doe s not convey, and should not be interpreted as conveying official EPA approval, endorsement, or recommendation.DRAFT FOR INTERNAL REVIEW May 27, 2009

iii PREFACE

The Interagency Workgroup on Air Quality Modeling (IWAQM) was formed to provide a focus for development of technica lly sound recommendations regarding assessment of air pollutant source impacts on Federal Class I and Wilderness areas. Meetings were held with personnel from interested Federal agencies, viz. the U.S. Environmental Protection Agency, the U.S. Forest Service, the National Park Service, and the U.S. Fish and Wildlife Service. The purpose of these meetings was to review respective modeling programs, to develop an organizational framework, and to fo rmulate reasonable objectives and plans that could be presented to management for suppor t and commitment. The members prepared a memorandum of understanding (MOU) that incor porated the goals and objectives of the workgroup and obtained signatures of management officials in each participating agency.

The IWAQM recommended the use of the CA LPUFF modeling system for use in Class I increment and air quality related values ( AQRV) analyses required under the Prevention of Significant Deterioration of Air Quality (PSD ) major source permitting program. In the ten years since the publication of the original IWAQM Phase 2 recommendations, the CALPUFF modeling system has continually evolved. Experience within the modeling community has also expanded with numerous applications of CALP UFF for PSD and Regiona l Haze Best Available Retrofit Technology (BART). However, the IWAQM guidance did not evol ve to reflect the changes in modeling technology an d experience gained since the original publication in 1998.

In 2005, the EPA convened a federal workgr oup to discuss ongoing issues with the development and management of the CALPUFF modeling system recommended for use by the IWAQM. Members of the fede ral CALPUFF workgroup include representatives from the Environmental Protection Agency, the National Park Service, and the U.S. Fish and Wildlife

Service. These recommendations reflect the collective experience of these agencies and extensive research on emerging issues which we re not foreseen during the publication of the original IWAQM recommendations.

As with the previous IWAQM document, this document will be released as a publication of the Environmenta l Protection Agency (EPA). The document updates IWAQM’s recommendations for modeling methods that might be used to estimate Prevention of Significant Deterioration (PSD) air quality imp acts, National Ambient Air Quality Standards (NAAQS) air quality impacts, and Best Avai lable Retrofit Technology (BART) air quality impacts associated with long-ra nge transport of pollutant emi ssions to Class I and Wilderness

areas.

The revised recommendations to the Interagency Workgroup on Air Quality Modeling (IWAQM) Phase 2 Summary Report and R ecommendations for Modeling Long Range Transport Impacts (EPA-454/R-98-019) contained in this document are considered technical guidance tailored for use in assessing air qua lity impacts associated with PSD and BART applications of the CALMET/CALPUFF modeli ng. These recommendations are intended to supersede the existing IWAQM Phase 2 recomme ndations for the application of CALMET.DRAFT FOR INTERNAL REVIEW May 27, 2009

iv ACKNOWLEDGMENTS

The members of federal CALPUFF workgroup ac knowledge the special efforts by Tim Allen

of the U.S. Fish and Wildlif e Service; John Notar and John Vimont of the National Park Service; Bret Anderson, Roger Brode, Tyler Fox, Kevin Golden, and Herman Wong of the U.S. Environmental Protection Agency (EPA) for thei r input and suggestions on assembling this document and their subsequent review.DRAFT FOR INTERNAL REVIEW May 27, 2009

v CONTENTS

PREFACE........................................................................................................................ ...........III

ACKNOWLEDGMENTS........................................................................................................... iv

1.0 INTRODUCTION............................................................................................................... 1

2.0 METEOROLOGICAL MODEL ISSUES........................................................................... 5

2.1 HORIZONTAL GRID RESOLU TION CONSIDERATIONS........................................... 5

2.2 ABILITY OF DWM ALGORITHMS TO ENHANCE NWP DATA TO ADEQUATELY REPLICATE METEOROLOGI CAL FEATURES OF INTEREST .... 11

2.3 REVIEW OF RECENTLY PUBLIS HED MM5/CALMET “HYBRID”

APPLICATIONS............................................................................................................... 15 2.4 CALMET “NO-OBSERVATIONS” (NOOBS) OPTIONS ............................................. 17

2.5 INCORPORATION OF OBSERVATIONAL DATA WITH NWP DATA WITHIN

CALMET......................................................................................................................... .. 21

3.0 MODEL EVALUATION PHILOSOPHY AND METHODOLOGY .............................. 27

3.1 MODEL EVALUATION PROTOCOL OBJECTIVES ................................................... 27

3.2 OVERALL EVALUATION PHILOSOPHY.................................................................... 27

3.3 METEOROLOGICAL MODEL EVALUATION COMPONENT .................................. 28

3.3.1 Statistical Measures fo r Meteorological Fields ................................................................. 29

3.3.2 Statistical Benchmarks ................................................................................................... ... 30

3.4 LONG RANGE TRANSPORT DISPER SION MODEL EVALUATION

COMPONENT .................................................................................................................. 35 3.4.1 LRT Model Evaluation Philosophy................................................................................... 35 3.4.2 Irwin Evaluation Methodology.......................................................................................... 36 3.4.3 Statistical Evaluation Methodology .................................................................................. 37 3.4.3.1 Spatial Analysis .............................................................................................................. 37 3.4.3.2 Global Statistical Analysis ............................................................................................. 38

3.5 GRAPHICAL METHODOLOGIES ................................................................................. 41

4.0 EVALUATION STUDIES AND FINDINGS .................................................................. 42

5.0 REFERENCES ............................................................................................................... .... 43

APPENDIX A. CALMET RECOMMENDATIONS ................................................................ 50

APPENDIX B. SUMMARY COMPARIS ON OF CALPUFF MODELING SYSTEM

RESULTS FOR VERSION 4.0 AND VERSION 5.8....................................................... 51DRAFT FOR INTERNAL REVIEW May 27, 2009

1 1.0 INTRODUCTION

The CALPUFF modeling system, consisting of the CALPUFF dispersion model, CALMET meteorological processo r, and CALPOST postprocessor, was promulgated by EPA in April 2003 as the preferred model for l ong-range transport (LRT) regulatory modeling applications for purposes of demonstrating compli ance with Class I PSD increments and is also recommended by the Federal Land Managers (FLM ) for Air Quality Related Values (AQRV) analyses. In 1998, EPA published the Interagency Workgroup on Air Quality Modeling (IWAQM) Phase 2 Summary Report and R ecommendations for Modeling Long Range Transport Impacts (EPA-454/R-98-019) (USEPA, 1998). The IWAQM Phase 2 report provides a series of recommenda tions concerning the applicat ion of the CALPUFF model for use in regulatory LRT modeling. This guida nce document correctly offered no concrete formula for determining certain user specified model control options su ch as grid resolution

and/or radii of influence for CALMET simulations . Rather, this document assumed that expert user judgment would determine the appropr iateness of certain CALMET/CALPUFF model control options, including grid re solution and radius of influen ce options which are central to proper wind field development in the CALMET meteorological model. The IWAQM Phase 2 report (USEPA, 1998) stated that:

“The control of the CALMET options requi res expert understanding of mesoscale and microscale meteorological effects on meteorol ogical conditions, and finesse to adjust the available processing contro ls within CALMET to develop the desired effects. The IWAQM does not anticipate the lessening in this required expertise in the future.”

Likewise, former NOAA meteorologist John Irwin summarized this philosophy at the 7 th
Conference on Air Quality Modeling (USEPA, 2000)

“Inevitably, some of the model control opti ons will have to be set specific for the application using expert judgment and in consultation with the relevant reviewing authorities. .... This is a modeling system that demands experience and judgement,”

The CALPUFF modeling system has conti nuously evolved since the publication of these recommendations in 1998; however, this guidance has not evolved and may not reflect current state-of-the-practice of the application of the model. Recognizing the need to update the existing guidance, EPA’s Office of Air Qu ality Planning and Standards (OAQPS) convened a CALPUFF Users Workgroup, consisting of air qu ality modelers from States, EPA Regional Offices, and the Federal Land Ma nagers, in the summer of 2005 whose charge was to identify areas for evaluation and update in the existing IWAQM guidance. Some of the key issues identified by the group included the disper sion coefficients, puff-splitting, and CALMET settings.

EPA envisioned that the required expertise for application of the CALPUFF modeling system would evolve through deve lopment of applicatio n-specific protocols, consultation with appropriate reviewing authori ties, and through consultation with EPA’s Model Clearinghouse as provided under Section 3.3 of the GAQM . At that time, EPA believed that a “cookbook”DRAFT FOR INTERNAL REVIEW May 27, 2009

2 approach to options settings was ‘premature , problematic, and counter-productive’ (USEPA, 2003). As time elapsed, it was anticipated that a growing body of knowledge would emerge regarding the appropriate mode l control options for applicat ions of the CALMET/CALPUFF system. However, only one (1) CALPUFF rela ted issue has been brought to the EPA Model Clearinghouse since the model was promulga ted in 2003 (i.e., the 2006 Region 4 request regarding to PG vs. turbulence dispersion options in CALPUFF).

Despite the lack of Model Clearinghouse cases, a range of issues have emerged regarding application of the CALPUFF modeling system, as documented in EPA’s “clarification memo” regarding th e regulatory status of CALPUFF for near-field applications (USEPA, 2008a), a subsequent memo addressing technical issues related to near-field applications (USEPA, 2008b), as well as the results of EPA’s assessment of the VISTAS version of CALPUFF (USEPA, 2008c). EPA now finds itself in a position that requires a fundamental reevaluation of the philosophical ap proach cited above. This reevaluation also acknowledges that it is increas ingly evident that a gulf of knowledge exists between the meteorological modeling community and the di spersion modeling community. Expertise in mesoscale meteorological modeling, cited as a critical prerequisi te by the IWAQM for CALMET applications, still only exists in a se lect number of air quality agencies, with meteorological modeling staff typi cally dedicated to chemical tr ansport modeling in support of ozone, fine particulate, and regional h aze implementation plan development.

The required expertise and collective body of knowledge in mesoscale meteorological models has never fully emerged from within the dispersion modeling co mmunity to support the necessary expert judgment on se lection of CALMET model contro l options. The lack of a sufficient body of knowledge with respect to mesoscale meteorological models, model evaluation procedures, and related issues has resulted in a process whereby the dispersion modeling community typically obtains the most readily available numerical weather prediction (NWP) dataset for applications of CALMET/CALPU FF without regard to its suitability, creates a three year CALMET dataset, and performs no additional assessment of the resulting CALMET meteorological fields. As a result of this process, the end user (e.g. dispersion modeler) typically has li ttle knowledge of choices made in NWP model physics options or the suitability of either the NWP or CALMET datasets used in LRT model applications. This has also created the unenviable posi tion for reviewing aut horities of having to make judgments of the suitability of NWP datasets for specific LRT applications, with little or no experience in the application of mesoscale meteorological mode ls and an incomplete understanding of the practical limitations of diagnosti c meteorological models such as CALMET in relation to their usage for air dispersion modeling.

In a regulatory context, this situation has often resulted in an ‘anything goes’ process, whereby model control option selection can be leve raged as an instrument to achieve a desired modeled outcome, without regard to the scientific legitimacy of the options selected. The BART experience has shown that many appli cations of the CALMET/ CALPUFF model for the same geographic region and time frame can yield di vergent model results so lely on the basis of which model control options are selected (H awkins et al., 2008). From a public policy perspective, this creates the untenable situation for reviewing authorities of having to determine which model application is ‘most correct.’ Th ese determinations are often made without theDRAFT FOR INTERNAL REVIEW May 27, 2009

3 benefit of the requisite expe rience and expertise previously mentioned, and without the necessary model performance eval uations to provide an objective basis for the determination.

At the 8 th Conference on Air Quality Modeling in September 2005, EPA discussed the necessity of updating the IWAQM Phase 2 guidan ce. One of key elements to updating the existing guidance was to update the historical pe rformance evaluations us ed in the original CALPUFF evaluation process to te st the enhancements to the CA LPUFF system that occurred after the publication of the Phase 2 guidance. Additionally, the AWMA Air Pollution Meteorology (AB-3) listed a methodology to eval uate CALMET wind fields and determine the appropriateness of horizontal grid reso lution as high priority issues.

In January 2008, EPA initiated the CALP UFF reassessment project in support of updating the IWAQM Phase 2 guidance. With this project, the EPA is pe rforming four tasks: (a) assemble a tracer and meteorological data base for use with LRT model evaluations; (b) develop a comprehensive evaluation fram ework (methodologies and tools) for both meteorological (prognostic and diagnostic) and LRT models; (c) exercising and testing meteorological and LRT models for the assemb led tracer database; and (d) updating existing EPA LRT modeling guidance to reflect le ssons learned from this project.

EPA also received comments on a number of technical issues related to the CALPUFF modeling system at the 9 th Conference on Air Quality Modeling in October 2008. Among these issues were the effects of horizontal grid resolution of both pr ognostic and diagnostic meteorological models on the accuracy of wind fields, development of objective methods for evaluating prognostic and diagnostic meteorological model output used in dispersion modeling, development of a methodology for determini ng how best to incorporate meteorological observations in a diagnostic meteorological mode l and set appropriate ra dii of influence for such observations, etc.

The situation described above and public comments have compelled the EPA to reassess the existing guidance and standard practices for the app lication of CALMET. Whereas in the past it was deemed to be both ‘prematu re and counter-productive ’ to recommend specific CALMET model control options, the EPA now be lieves it is both timely and necessary to specify such items to promote scientific inte grity and restore balan ce to the public decision making process.

Section 2 of this document presents a number of meteorological modeling issues

identified at the 8 th and 9 th Conferences on Air Quality Modeling and proposes interim

solutions to address these issues . These interim methods are inte nded to preserve as much of the integrity of the orig inal prognostic meteorological fields as is practical within the CALMET diagnostic meteorological model. Briefly summarized, the revisions to the Phase 2

recommendations include:

• Preservation of original prognostic data La mbert Conformal grid specifications and

horizontal resolution in CALMET simulations unless performance evaluation clearly indicates that original prognostic data used as the first-guess wind field for CALMETDRAFT FOR INTERNAL REVIEW May 27, 2009

4 does not adequately represent relevant meteor ological features wh ich are important to source-receptor relationships associated with long range tran sport (LRT) modeling.

• One-to-one vertical layer matching between prognostic and diagnos tic meteorological models between the surface to 5000 meters above ground.

• Elimination of CALMET diagnostic adjust ments to first-guess wind field unless performance evaluation clearly indicates that diagnostic adjustment s increase objective accuracy of final wind fields and are rele vant to plume transport and dispersion.

• Continuation of incorporation of surface obs ervations for Radii of influence (RMAX1, RMAX2, RMAX3, R1, R2, R3) set to mini mal value (0.001 km) to preserve the integrity of prognostic meteorological da ta used as the first-guess wind field.

• Recommendation against the use of th e “no-observation” methods for CALMET (NOOBS=1, 2).

Section 3 of this document presents a comp rehensive protocol for the evaluation of both meteorological and long range transport (LRT) dispersion models. Sta tistical and graphical

methods for evaluation of both me teorological and LRT models are presented. Section 4 of this document presents results from the ongoing EP A performance evaluation of the CALPUFF modeling system, which are used to form the ba sis of some of the recommendations contained

within this document.DRAFT FOR INTERNAL REVIEW May 27, 2009

5 

### 2.0 METEOROLOGICAL MODEL ISSUES

This section addresses a numb er of meteorological modeling issues that have emerged since the promulgation of the CALPUFF modeli ng system in April 2003, including horizontal grid resolution, limitations of diagnostic wind models (DWMs) such as CALMET to simulate complex meteorological flows, and methods for utilizing NWP and/or observational data to generate three-dimensional wi nd fields in CALMET. The discussion includes a summary of relevant scientific literature, as well as m odel evaluation studies, which provide technical support for the revisions to the IWAQM Phase 2 recommendations presented in Appendix A [to be provided]

.



### 2.1 HORIZONTAL GRID RESOLU TION CONSIDERATIONS

At the 8 th Conference on Air Quality Modeling in September 2005, the Air and Waste Management Association AB-3 Meteorology Comm ittee offered comments listing horizontal grid resolution as a priority issue for the CA LPUFF modeling system. Similarly, the American Petroleum Institute (API) listed grid reso lution and model performance evaluations among several issues for the CALPUFF modeling system at the 9 th Conference on Air Quality Modeling in October 2008. This section discus ses the relevant considerations regarding horizontal grid resolution based upon reviews of the available scientific literature and recent performance evaluations.

Traditionally, NWP data generated by mesos cale meteorological models such as MM5 have been used in conjunction with rou tinely available NWS observations in CALMET applications for air quality studies. This appr oach is most commonly referred to as the “hybrid” approach, reflecting a hybrid meteorological field c onsisting of a first-guess wind field supplied by NWP data, supplemented with ob servations to enhance the performance of the resulting diagnostic wind fields. Typically, CALMET has been exercised at a much higher

resolution than the input NWP data used as the first-guess wind field. The “hybrid” approach, as described by Scire and Robe (1998), provides the advantage of reducing the simulation times relative to what would be needed for high reso lution prognostic meteorolog ical simulations run at the same resolution. The philosophy behi nd the “hybrid” modeling approach with CALMET is to incorporate higher resolution topographic and/or land use features that would not be adequately represented in coarser scale prognos tic meteorological model runs. Earth Tech (2001) summarized the philosophy as follows:

“It is attractive to use or include MM5 da ta in the CALMET initi al guess wind field relative to the data from typical meteorol ogical observation networks. However, it is

common that the coarse-scale MM5 data are not adequate to fully-resolve the fine-scale

terrain effects that can domina te the flow field near a part icular source and control the design concentrations produced by the mode l. Increasing MM5 grid resolution would increase costs in cubic, not linear, since the time step of integration needs to be reduced in order to keep the integr ation stable. On the other ha nd, CALMET offers a practical,

cost-effective solution to this problem, by adjusting the coarse scale flow fieldsDRAFT FOR INTERNAL REVIEW May 27, 2009

6 produced by MM5 model so that they repr esent the fine-scale terrain seen by the

CALMET and CALPUFF models.”

From a historical perspective, it appeared that the “hybrid” ap proach could offer a viable alternative to the necessity of having to run multiple years of prognostic data at high resolutions, considering the prac tical barrier that the enormous computational costs have to generating such data sets. Results from Irwin et al. (1996) suggested that inclusion of NWP data along with observational data improved th e performance of CALPUFF compared to the construction of CALMET datasets using obser vations alone. In th is study, CALMET was operated at an 18 km resolution in both obser vation-only and “hybrid” mode with 80 km MM4 data used as the Step 1 wind field. These re sults showed that the “hybrid” mode of CALMET performed better than either MESOPAC II or CALMET in observation-only mode. The horizontal resolution of the NWP data was very coarse in comparison to present day NWP applications. With an 80 km resolution, the NW P data would not adequa tely characterize manycomplex terrain features; therefore, the prevailing paradigm was that supplementing the firstguess field with observations would e nhance the final CALMET solution.

The IWAQM Phase 2 report (USEPA, 1998) offered no concrete formula for determining the appropriate gr id resolution for CALMET simula tions. Grid resolutions of

various studies contained w ithin the IWAQM Phase 2 repo rt are 18 km for the CrossAppalachian Tracer Experiment (CAPTEX) (Irwin et al., 1996), 10 km for the Idaho Falls Tracer Study (Irwin, 1997), and 250 meters for the near-field Columbia River Gorge study (Scire and Robe,1997). Traditionally, th e FLMs have recommended a CALMET grid resolution of approximately 4 km (Tim Allen, personal communication).

NWP modeling technology has evolved drama tically since the publication of results from Irwin et al. (1996) and the IWAQM Phas e 2 summary report (EPA, 1998). Higher spatial and temporal resolution of NWP data is av ailable for routine use in LRT modeling. Theoretically, this should result in more re alistic LRT simulations. LRT model performance evaluations conducted by Van D op et al (1998), Nasstrom et al . (1998), and Deng et al. (2004) have shown that higher spatial and temporal reso lution of model data typically results in more accurate LRT model simulations. However, th e relationship between increased horizontal resolution of NWP data and enhanced model pe rformance does not necessarily apply without

limitation to all reso lutions. Mass et al. (2002) suggested that a “law of diminishing returns” may exist for accuracy of NWP forecasts when increasing the horizontal resolution of NWP model simulations, indicating that the point of diminishing return s is around 10 to 15 km in the northwestern U. S., but considerably larger (20 to 40 km) in the eastern half of the U. S. where topographic relief is less dramatic . Mass et al. (2002) further s uggested that only in cases of highly complex terrain, e.g., the Columbia Rive r gorge, was it necessary to operate a NWP model at an ultra-high resolution (0.5 km – 1 km resolution) to increase the objective accuracy of the NWP wind field solution. Similarly, Deng et al. (2006) found that increasing horizontal resolution of NWP models does not always produce better simulations, especially in areas of convective instability. Weygandt and Seaman (1994) further noted that increased grid resolution may actually lead to decreased mode l skill for some parameters. In addition to higher resolutions made possible by significan t advances in computational resources, significant advances have also been made in coupling NWP and air quality models, includingDRAFT FOR INTERNAL REVIEW May 27, 2009

7 more advanced physics options to account for boundary layer processes of importance to air quality modeling applications. De ng et al. (2004) indicated that introduction of more advanced physics within the NWP model produced much gr eater reductions of simulation errors than increasing grid resolution.

Mass et al. (2002) noted that decreasing horizontal grid spacing may increase the structural detail of the atmosphere simula ted by the NWP model, but does not necessarily increase the accuracy of predicte d variables. In a similar sens e, the higher resolution CALMET simulations may increase the structur al detail of the final wind fi elds; however, the majority of CALMET evaluations to date have been subj ective in nature a nd have relied upon the perceived increase in structural de tail (i.e. “realism”). In a fre quently referenced example used in CALMET training classes, Scire (2008) show s significant structural detail of a high

resolution CALMET simulation in Pocatello ID. This evaluation relies upon the perceived

increase in structural detail wi thout any form of a statistical performance evaluation to verify the objective accuracy of high resolution wind fields. In short, a subjective assessment that a

wind field is “realistic” is not su fficient to support the assumption that the wind field accurately reflects reality.

Given the limitations of diagnostic models to ensure dynamically consistent wind fields (Seaman, 2000), there is legitimate concern that the increased structural detail in the horizontal wind fields resulting from application of CA LMET at higher grid resolutions may lead to spurious effects on plume dispersi on which may not be obvious, even from a detailed review of horizontal wind fields. In particular, Seaman (2000) noted that the technique employed in CALMET and other diagnostic wind models (DW Ms) of adjusting vert ical velocities by imposing mass conservation to account for horizontal divergences which result from diagnostic adjustments to the wind fields “can lead to unreali stic ‘residual’ vertical velocities at the top of the modeling domain.” He points out that sinc e the divergence is severa l orders of magnitude smaller than the wind, small errors due to inte rpolation or other diagnostic adjustments can cause much larger errors in the divergence, and in turn the diagnosed vertical velocities. While

limited evaluations of CALMET wind fields have typically focused on horizontal wind components, some researchers (Chang et al ., 2003; Wang et al., 2008) have noted that CALMET may not simulate vertic al velocities well compared to more refined NWP models, showing less skill than exhibited for horizon tal winds. Based on standard CALMET options currently in use (LCALGRD = .TRUE.), CALMET will pass a 3-dimensional grid of vertical velocities generated from the mass conser vation adjustment to CALPUFF. Although

CALPUFF does not use the vertical velocities directly to vert ically displace the puffs, the vertical velocity gradient may lead to enhanced vertical puff spread in CALPUFF, with some vertical redistribution of puff ma ss. The potential magnitude of the impact of this effect on

CALPUFF modeled concentratio ns is not well documented.

EPA conducted a limited statistical performa nce evaluation of four separate CALMET ‘no-observational’ analyses with different ho rizontal grid resolutions for both MM5 and CALMET, utilzing the fifth trace r release of the Cross-Appa lachian Tracer Experiment (CAPTEX). The PSU/NCAR MM5 mesoscale mete orological model Version 3.73 was used to produce NWP data fields for the CALPUFF tracer experiments. MM5 was initialized using 6- hourly NCEP Reanalysis data (available on a 2.5o x 2.5o resolution). MM5 physics optionsDRAFT FOR INTERNAL REVIEW May 27, 2009

8 selected included the Pleim-Xu planetary boundary layer and land surface model scheme, KainFritsch cumulus parameterization, simple ice mi crophysics, and the RRTM radiation scheme. Three nested domains of 108 km, 36 km, and 12 km were utilized for this experiment.

CALMET was initialized with both 36 km a nd 12 km MM5 data sets for 18 km, 12km, and 4 km CALMET simulations (Figure 2.1.1). EPA generated a full suite of model performance statistics using its prototype CALMETSTAT software (Anderson, 2006). These

metrics are discussed in Secti on 3.3.1 of this document. To analyze the impact of grid resolution on meteorological model performan ce, the evaluation focused upon wind statistics from four simulations represen ting application of CALMET with a higher grid resolution than the MM5 data used for the Step 1 wind field. The four CALMET simulations are defined in Table 2.1.1. As shown in Figure 2.1.2 [to be provided] , the gross error and index of agreement (IOA) for wind speed, and gross error for wind direction showed little sensitivity to the resolution of the first guess field or th e final CALMET field reso lution, with nearly identical performance statistics across each of the simulations.

EPA also conducted a statistical perfor mance evaluation of the CALPUFF model response to changes in the horizontal grid resolution of both the MM5 and CALMET models

based on the four CALMET simulations. EPA us ed performance evaluation metrics described

in Sections 3.4.3.1 and 3.4.3.2 of this document. The CALPUFF evaluation results were consistent with the statistical performance evaluation of the various CALMET simulations, exhibiting nearly identical perf ormance statitics. The final composite model performance

RANK ranged between 1.84 – 1.86 (higher number re presents better model performance) for each of the CALMET simulations (Table 2.1.2), showing little, if a ny, sensitivity to the increase in grid resolution within CALMET re lative to the MM5 grid resolution. The full meteorological and LRT model performance evalua tion results from this study are presented in Appendix B of this document [to be provided]

.

This evaluation underscores several critical elements suggested by Mass et al. (2002). First, for many areas of the country, there does in fact exist a ‘law of diminishing returns’ where there is little perfor mance benefit observed by arbitr arily increasing the horizontal resolution of the meteorological model. In this example, key statistics for wind showed little sensitivity whether initialized with 36 km or 12 km MM5, and there was no augmentation of model performance by increasing the horizontal resolution of CALMET from 18 km to 4 km. Second and equally important is th e necessity of determining the adequacy of the NWP data set prior to assimilation within DWMs such as CALMET. In this experiment, it is shown that the first guess field largely determines the outcome of the statistical results, since the CALMET

‘no-observational’ simulations ar e largely insensitive to the increase of horizontal resolution from 18 km to 4 km, indicating that the CA LMET diagnostic adjustments are of minor importance to overall model performance. The re sults of this experime nt are not universally applicable as some areas of the country, as indicated by Mass et al. (2002), may require higher

resolution than 36 km or 12 km NWP data.DRAFT FOR INTERNAL REVIEW May 27, 2009

9 Figure 2.1.1 – Topography of 4 km modeling domain for CAPTEX Release 5 experiments.

Table 2.1.1 – CALMET ‘NOOBS’ experi ments for CAPTEX Release 5 used for meteorological performance

evaluation.

Experiment MM5 Resolution (km) CALMET Resolution (km)

EXP1D 36 18

EXP3D 36 12

EXP5D 36 4

EXP6D 12 4

.

Table 2.1.2 – Final CALPUFF model ranks from global statistical analysis of five CALMET

‘no-observations’ simulations.
Experiment Rank

EXP1D 1.86

EXP3D 1.84

EXP5D 1.85

EXP6D 1.85DRAFT FOR INTERNAL REVIEW May 27, 2009

10 The main lesson drawn from these studies is that while increasing the horizontal grid resolution of NWP data has generally yielded be tter LRT model verificati on scores, the benefit to objective accuracy of both NW P and LRT model simulations doe s not necessarily increase as one continues to decrease horizontal grid sp acing. While these studies have examined the sensitivity of NWP models to gr id resolution, there is no obvious reason to assume that a DWM like CALMET will respond any better to increa sing horizontal grid resolution than NWP models. In fact, the lack of adequate physics in CALMET to simulate complex meteorological flows and also ensure the dynamical consistency of the adjusted wind fields raises the concern that a possible effect of increasing grid resolu tion may be propagation of errors, in the sense

that an error at one location al ong the plume trajectory affects all subsequent time steps in the simulation of the plume. Any systematic erro r that might exist within the modeling system could result in a significant cumulative error in the overall impact of the plume, even if the localized magnitude of the error is small. Given that LRT applications such as this are focused on simulating the plume impact inside limited area s within a much larger domain, errors which are relatively small viewed in isolation may coll ectively introduce significant uncertainty in the

overall result. As a result of these uncertainti es, it is essential that the objective accuracy of

final CALMET wind fields be established thr ough appropriate performance evaluations.

As discussed in Chandrasekar et al. ( 2003) and Wang et al. ( 2008), CALMET has been shown to produce reasonable wind fields when us ing either a highly resolved NWP data set as the first-guess wind field or with a higher number of observations in areas of relatively modest terrain. Wang et al. (2008) found that differe nces between CALMET and the reference winds tended to be reduced with data sampled from more stations or from more uniformly distributed stations. However, both of these studies also em phasize the fact that the ability of CALMET to produce wind fields with objective accuracy is di rectly tied to the density of the observational data set used to construct the wind field, not si mply to increasing the horizontal resolution of CALMET and relying upon its diagnostic wind flow algorithms to accurately simulate complex

flows. Chandrasekar et al. (2003) correctly stated that

“...regions of complex terrain can introduce additional diffi culties like inadequate density of observations, limitations of a diagnostic model to reproduce the observed features over complex terrain, and difficulties in fully resolving terrain features by using a coarse prognostic model over a complex terr ain. The effectiveness of this approach may therefore be different for a region of complex terrain.”

In the last several years, th ere has been an increasing tr end of using higher horizontal resolution CALMET simulations (i.e. less than 4 k ilometers), especially in areas of moderate topographic relief. In many of these cases, th e higher resolution creates complications for planning model simulations due to limitations of co mputational capacity. In order to overcome these computational limitations, it is not uncommon to propose multiple high resolution

domains to cover all Class I areas of interest. Ho wever, the consensus of scientific literature provides no clear basis for extending the CALMET/CALPUFF grid resolution much beyond the resolution of the NWP model used to spec ify the first-guess wind field. Therefore, the IWAQM guidance is being revised in such a way to preserve as much of the integrity of the original NWP model as is practical.DRAFT FOR INTERNAL REVIEW May 27, 2009

11 In summary, there is little scientific ev idence to support the clai m that higher CALMET resolutions increase the objective accuracy of the final wind field, especially in areas of relatively modest topographic relief. The preponderance of scientific literature is consistent in

the conclusion that there is a limitation to the be nefit of higher resolution NWP data, especially for areas of modest topographi c relief. Higher resolution datadoes not necessarily improve model performance, but may in fact degr ade model performance for some predicted meteorological parameters. Second, CALMET has limited ability to independently capture the full three-dimensional structure of complex flow s. Without the benef it of high resolution NWP data or a high density of repr esentative observational data, the ability of the DWM to accurately

simulate these conditions is limited. Section 2.2 of this document discusses the limitations of DWM diagnostic algorithms in further detail.



### 2.2 ABILITY OF DWM ALGORITHMS TO ENHANCE NWP DATA TO ADEQUATELY REPLICAT E METEOROLOGICAL FEATURES OF

INTEREST

Robe and Scire (1998) suggest ed that CALMET can serve as an effective tool for construction of wind fields in complex terrain environments, by using a coarse scale NWP data set as the first-guess wind field and allowing CALMET to make diagnostic adjustments to

reflect the fine scale features of the wind field not resolved by the NWP model. A fundamental assumption of this paradigm is that the DWM diagnostic adjustments can replicate the threedimensional structures of complex meteorological flows. Therefore, it is important to establish which aspects of complex terrain are important for source-receptor relationships and then evaluate the scientific algorithms of the DWM to determine if it has the ability to independently simulate the complex meteorological flow.

As noted in the EPA memorandum “ Technical Issues Related to CALPUFF Near-field Applications ” (USEPA, 2008b), there are known limita tions to any DWM, including CALMET, which need to be considered when ap plying such a model in complex terrain. For example, CALMET only contains algorithms for certain aspects of the valley wind system (drainage flows). Other portions of the wind sy stem (cross-valley, up/down valley circulations) are neglected in the algorithms (Scire et al., 200 0a). Currently, these components of the valley wind system can only be introduced through hi gh resolution NWP data or strategicallypositioned surface and upper atmospheric observation stations that capture the complex threedimensional structure of the valley wind system.

An accurate treatment of energy balances is an essential el ement of meteorological field

construction. In the current version of CALMET, diagnostic wi nd field adjustments such as slope flows attempt to examine the local sensible heat flux ( Q h ) and temperatures. In the CALMET subroutine SLOPE, the variable tinf represents the domain representative temperature which is defined by the user contro lled variable ISURFT. ISURFT is the surface

station number to use for surface temperature (defined as 1 to NSST). If there are no temperature measurements within the area of complex terrain where the unique thermal structure has evolved, then CALMET has no knowle dge of the local thermal structure. This places extreme importance on insuring that surface observations actually ex ist within areas ofDRAFT FOR INTERNAL REVIEW May 27, 2009

12 complex terrain. If the user sets ITPROG equal to two (ITPROG=2), then CALMET relies upon the temperature from the first layer of the NWP model data assimilated into CALMET. This approach automatically infers that the NWP data is of a high enough resolution to actually represent the local surface temper atures accurately in areas of complex terrain. In almost all cases when using coarse resolution NWP data, these models will not have adequately represented the fine scale thermal structure in areas of complex terrain. This approach effectively nullifies the “hybrid” pr incipal that it is practical to use coarse scale NWP data and

allow for local refinements based upon CALMET diagnostic adjustments.

The sensible heat flux, Q h , is supplied to the SLOPE subroutine from the HEATFX

subroutine in CALMET. Local Q h as computed by CALMET is subject to important limitations often not considered in complex terr ain modeling. First is the impact of “terrain shadowing” on local Q h and surface temperature is neglec ted in the publically available versions of CALMET. In valleys with north-sou th axes of orientation, the incident incoming

solar radiation will strike one side wall of the va lley while the other is essentially ‘shadowed.’ In the mornings during ‘warmer’ months, the western side walls of valleys will receive the majority of incoming solar radiation, whereas the eastern side wall re mains ‘shadowed’ by the terrain. The process creates an energy imbalance and is an important fa ctor in developing the daytime thermally driven wind system. As the da y progresses, the proces s essentially reverses itself with the eastern-side wall of the valley receiving the majority of the incoming solar radiation. According to Bella sio et al. (2005), a special ve rsion of CALMET (“m-CALMET”) has been developed which incorporates the eff ects of “terrain shadow ing” upon the radiation balance and surface temperatures. Since these e nhancements have not been introduced into any of the publically available versions of CALMET, this remains as a technical deficiency in CALMET complex terrain adjustments. Second is the impact of clouds on model radiation balance. Normally, clouds ar e introduced to CALMET through surface observations. The 2-D cloud cover is constructed using the value from the nearest valid reporting surfa ce station for a given time step. However, when the user selects the full ‘no-observation’ approach for CALMET (NOOBS=2), thus relying completely upon the assimilated NWP data to provide all necessary information, cloud cover is estimated from the NWP hydrometeors available from

the assimilated NWP data. Due to incorrect implementation of the prognostic cloud fraction (Teixeira, 2001) and subsequent underestima tion of total cloud cover, CALMET often will overestimate the amount of incoming shortwave radiation, resulting in overestimates of the

sensible heat flux. This issue is discussed in greater detail in Secti on 2.4 of this document.

Terrain blocking in CALMET is determin ed by calculating the local Froude ( Fr ) number (Scire et al., 2000a). The Froude number is a measure of the ratio of kinetic energy to potential energy. In atmospheric motions, the kine tic energy is represen ted by the velocity of the horizontal wind ( U ) and the potential energy is repres ented by the Brunt-Vaisala frequency ( N

), a measure of atmospheric static stability, multiplied by the height of a terrain obstacle ( Δ h ):

h N

U Fr Δ = (1)DRAFT FOR INTERNAL REVIEW May 27, 2009

13 Note that the height of the terrain obstacle ( Δ h ) is derived from the gr idded terrain elevations input to CALMET, based on the terrain radius of influence (TERRAD) specified by the user.

Since the value of Δ h is determined without regard to the direction of the maximum terrain elevation relative to the reference grid cell, it may not be representa tive of actual terrain

features of interest in some cases.

The Brunt-Vaisala frequency ( N

) is the frequency of osc illation of an air parcel produced by the restoring force (net force of buoyancy and gravity) acting on the air parcel which has been displaced from its equilibrium level in an unsaturated, stably stratified atmosphere. Brunt-Vaisala frequency is given by the following equation:

5

. 0 ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛

= dz d g N θ

θ (2)

where N is the Brunt- Vais ala frequency (1/s)

g

is the acceleration due to gravity (m s -2 )

θ

is the potential temperature (o K)

dz d

θ is the potential temperature lapse rate (o K/ m)

When the local Froude number is less than the user-specified cr itical Froude number (default = 1), a parcel of air has insufficien t kinetic energy to overcome the gravitational potential imposed by the height of the terrain obstacle and flow is blocked, causing the parcel

of air to be deflected. If th e local Froude number is greater than 1, then the air parcel has sufficient kinetic energy to overcome the gravit ational potential imposed by the terrain obstacle and flows over the top of the obstacle (UCAR, 2001).

The basic response of the atmosphere when flow is blocked by a terrain obstacle is to

either flow around the obstacle or be turned back. During normal atmospheric flow, the wind flow is essentially governed by a balance of for ces, primarily the pressure gradient force (PGF) and the Coriolis force. As an air parcel approa ches and begins its ascen t of a terrain obstacle, its speed is reduced as it must work against th e gravitational potential of the obstacle. When the speed is reduced, it also reduces the Coriolis force, which in turn throws the wind out of

balance with the PGF. When th is occurs, the parcel of air begins flowing along the pressure gradient force, from higher pressure to lower pre ssure. In essence, as an air parcel is blocked because it does not have sufficient kinetic energy to overcome the gravitational potential of the obstacle, it is deflected a nd flows along the PGF as a result of the development of the imbalance of competing forces (UCAR, 2001).

The first concern with the CALMET Froude number implementation is that the potential temperature ( θ ) and the potential temperature lapse rate ( d θ /dz) are specified by two different mechanisms within CALMET based upon the user-specified value for the ITPROG option. When the user specifies ITPROG equals zero (ITPROG=0), the potential temperature lapse rate is determined from the domain representative upper ai r station specified by the user

with variable IUPT in the CALM ET control file. In many cases, the upper air station is locatedDRAFT FOR INTERNAL REVIEW May 27, 2009

14 far away (several hundred kilometers or more in some cases) from the local terrain features of interest, and thus the temperatur e lapse rate would not be repres entative of the local thermal structure. Since the upper-air soundings are typi cally available only twice per day (at 12Z and 00Z), the hourly temperature laps e rate is determined by a linear temporal interpolation, further diminishing its representativeness for purposes of terrain adjustments. When the user specifies

ITPROG ≥ 1, CALMET calculates a domain mean temperature lapse rate based on the NWP data to provide lapse rate information for te rrain blocking calculati ons handled by the FRADJ

subroutine in CALMET. Neither of these a pproaches are representative of actual local thermodynamic conditions which govern the blocki ng effects of terrain obstacles, and this limitation can significantly affect the main diagnostic adjustments to the wind fields that form

the basis for use of CALMET.

Several other potential areas of concern exist with the implementation of Froude number adjustments within CALMET. First, ac cording to the CALMET User’s Guide (Scire et al., 2000a), the wind speed remains unchanged as it in teracts with a terrain obstacle. Recall the basic principle that kinetic energy of an air pa rcel is reduced as it must work against the gravitational potential of the terrain. In this sense, the wind velocity ( U ) must reduce as it works to ascend the obstacle. CALMET does not adjust the velocity ( U ) to represent the decrease in kinetic energy of an air parcel as it works to as cend the barrier. This creates another concern regarding the directionality of the wind de termined by the Froude number adjustment. CALMET assumes that the resultan t wind vector will flow tangentially to the terrain obstacle. This assumpti on is only valid for isolated te rrain obstacles. When conducting LRT modeling studies on the mesoscale, terrai n obstacles are more commonly represented as long chains of hills or mountains. In these case s, wind vectors will not simply flow tangentially to the terrain. Recall as the speed of an air pa rcel is reduced, the PGF becomes greater than the Coriolis force, and wind begins to flow from higher pressure to lower pressure. CALMET lacks a three-dimensional pressure field in order to calculate the PGF; therefore, in mesoscale simulations where terrain is represented by long chains of mountains rather than isolated obstacles, CALMET will simply modify the local flow field by adjusting the vector to flow tangentially to the terrain. In these cases, it is unrealistic to assume that thermodynamic blocking simply results in tangential flow.

Second, there is a finite distance upstream of the obsta cle where the flow can be blocked by a terrain obstacle, not simply when th e air parcel interfaces with the terrain obstacle (UCAR, 2001). This distance is de termined by the following equation:



## ( )

f U h N L − Δ

= (3)

where L is the distance upstream of the terr ain obstacle where flow is blocked

N

is the Brunt-Vaisala frequency

h Δ is the terrain height

U

is the wind velocity f

is the Coriolis parameterDRAFT FOR INTERNAL REVIEW May 27, 2009

15 CALMET simply assumes that the flow is impede d at the interface between the air parcel and

the terrain obstacle when Fr is less than 1. It neglects the fact that the flow field a finite distance upstream ( L ) of the obstacle is also infl uenced and is “blocked.”

In summary, it is EPA’s technical judgment th at there are substantial limitations in the complex terrain parameterizations in the CALMET model and that understanding these limitations is a critically important component in the decision to apply CALMET at higher resolutions. If the performance evaluation of the NWP data set es tablishes the unsuitability of the NWP data for characterizing the dominant complex terrain meteorol ogical features, the application of a DWM at a highe r resolution would sti ll require its own statistical evaluation, focusing upon the ability of the DWM to provide superior wind fields compared to the NWP

data. Statistical evaluations of diagnostic wind models such as CALMET may be misleading or of limited value if not designed properly. DWMs rely almost exclusively upon observations and will always exactly reproduce the obser ved wind at surface meteorological sites represented in the model. Therefore, one faces an “autocorrelation” issue when attempting to conduct a statistical performance evaluation of the diagnostic models wind fields as the diagnostic model exactly reproduce s observations at th eir respective locations in the modeling domain. The purpose of such an evaluation is no t to analyze the performance of the diagnostic model over broad regions of the target area(s) representing th e synoptic scale features, but rather is to evaluate the diagnos tic features of the model which th e protocol states will enhance the NWP data used as the first-guess field repr esenting the synoptic scal e. A properly designed performance evaluation for such an applicati on of the diagnostic model would necessitate “degradation” or withholding key observations in areas of complex terrain to determine if the diagnostic adjustments that the model makes are physically realistic and show agreement with those observations made in areas of complex te rrain that are withheld from the diagnostic

model run.



### 2.3 REVIEW OF RECENTLY PUBLI SHED MM5/CALMET “HYBRID”

APPLICATIONS

At the time of publication of the IWAQM Phase 2 summary report in 1998, there was little collective experience on the application of the “hybrid” method introduced by Scire and Robe (1998). Since the date of publication of the IWAQM Phase 2 summary report, members of the user community have gained experien ce on its application a nd have published their findings. EPA has reviewed a number of th ese studies and has summarized the relevant

information below.

A study published by the CALPUFF develope rs (Earth Tech, 2001) focused on the operation of MM5 and CALMET at various resoluti ons in areas of highly complex terrain and

varied land use characteristics in Alaska. In general, this stud y found that representative, sitespecific meteorological data are needed to ad equately capture wind fields for the complex terrain situations. The use of just MM5 data at 20 km or 4 km resolution (NOOBS=2), a hybrid of 20 km MM5 data with remote NWS data (NOOBS=0), and remote NWS data only (“obs only”) were all insufficient and produced wind characte ristics that did not match the observed

winds.DRAFT FOR INTERNAL REVIEW May 27, 2009

16 Similarly, in a presentation entitled “ Modeling in a Complex Terrain Environment at High Latitudes ” (Scire, 2009), it was demonstrated that , in a complex terrain and sea-breeze environment in Iceland, extremely high horizonta l resolution NWP data (1 km) was necessary to adequately resolve the local flows. The pres entation showed that NWP data alone at a 2 km

resolution was insufficient to resolve the sea- breeze phenomena, requiring that the NWP data be run at a 1 km resolution to adequately simulate the sea-breeze environment.

RWDI (2002) published a report for the Bri tish Columbia Ministry of Water, Land, and

Air Protection entitled Final Report: Using Mesoscal e Models to Support Regulatory Dispersion Modelling . RWDI cited two studies in the Paci fic Northwest and Alberta, Canada. The first study was conducted for a proposed powe r plant in Washington. MM5 was available

at a 12 km resolution and CALMET was run at a 4 km resolution. Observations of cloud cover, temperature, and relative humidity were assimilated into CALMET from 94 sites. Surface

winds were not assimilated. Obse rved and predicted winds were evaluated for three sites in southern British Columbia. Wind roses gene rally showed poor agreement, with both wind speed and direction distributions showing poor agreement.

The second study cited by RWDI concerne d the application of MM5 and CALMET to produce meteorological fields near Fort McMurr ay, Alberta. MM5 was run at a resolution of

20 km and CALMET was run at 2.5 km. Observ ations from three surface stations were

assimilated into the runs. Results indicated th at the simulation produced reasonable results for winds aloft, while the number of surface observi ng sites incorporated into the analysis was insufficient to fully resolve the wind fl ows in the Athabasca River Valley.

All of these studies illustra te a key point in the genera l application of DWMs: the DWM class of wind model lacks the physics necessary to adequately simulate complex flows. Without more highly resolved NWP data or a sufficiently dense and strategically positioned surface and upper atmospheric meteorological netw ork, it is likely that most DWMs will have great difficulty simulating orographically indu ced wind flows or lake/sea breeze circulations independently . The incorporation of NWP data as the first-guess wind field itself does not guarantee that the meteorological fe atures of interest will be captured in the final DWM wind field. The NWP data itself must capture the general features of interest.

It is unlikely that the higher resolution CALMET domain will result in any benefit to the simulation of lake breezes. While inge stion of NWP data by CALMET provides the capability of introducing flow features such as lake breezes that may not be captured by the surface observational data, typical 36 km NWP data sets generated by the Regional Haze Regional Planning Organizations (RPOs) likely w ill not have resolved either local complex

terrain flows or lake breeze circulations. If surface observations exist in the surface meteorological database that are heavily influenced by water bodi es and exhibit characteristics of the lake/sea breeze not resolved by the NWP data, this may result in disagreement between the coarse resolution first-guess wind field and the observations introduced in the Step 2 wind field, creating the possibility of unrealistic physic al discontinuities in the wind field (Scire, 2006; Scire, 2008). Irrespective of radius of influence settings for surface stations (R1 and RMAX1) that one may chose for CALMET wind fiel d construction (unless R1 is set so smallDRAFT FOR INTERNAL REVIEW May 27, 2009

17 that it essentially eliminates the influence of a surface station), physic al discontinuities may develop when observations introduced in the Step 2 wind field disagree with the first-guess wind field (NIWA, 2004; Scir e, 2006; Scire, 2008).



### 2.4 CALMET “NO-OBSERVATIO NS” (NOOBS) OPTIONS

As discussed in previous sections, there is some evidence to suggest that higher spatial and temporal frequency of NWP data used in LRT modeling generally results in better LRT

model verification statistics. Therefore, in theory, the NOOBS approach in CALMET could offer the opportunity to take advantage of higher temporally and spatially resolved initial guess

wind fields from NWP data than could otherw ise be achieved through the exclusive use of twice-daily RAOB soundings. Howe ver, it is important to note that CALMET does not merely pass through the majority of the information fr om the NWP model to CA LPUFF. Much of the original NWP data (e.g., planet ary boundary layer (PBL) heights and scaling parameters) is recomputed within CALMET. Therefore, careful consideration must be given to how these rediagnostic procedures are implemented within CALMET. As also noted above, CALMET does not fully utilize the 3-dimensi onal temperature fields when a pplying diagnostic adjustments to the wind fields under the regulatory default optio n, although the full temperature field is passed to CALPUFF (along with the vertical velocities ) if the LCALGRD option is selected. Aside from the documented limitations of the modeling sy stem to properly utili ze the full benefits of current state-of-the-practice prognostic modeling capabilities, there are few, if any, objective evaluations of model performance on which to base acceptance of these NOOBS options that have not previously been approved by EPA for regulatory applications.

EPA’s assessment of several recent enhancem ents to the CALPUFF system has yielded

mixed results. In the Assessment of the “VISTAS” Versi on of the CALPUFF Modeling System (USEPA, 2008c), EPA identified significant area s of concern regarding modifications to the treatment of the convective boundary layer (CBL ) over land. EPA tests identified that these modifications led to spurious collapses and rege nerations of the CBL. EPA tests showed that the collapse of the CBL and asso ciated changes in dispersion had varying, albeit in some cases significant, impacts on surface concentrations depending upon source type.

Similarly, in the EPA 2001 Philadelphia Air Toxics Study (Touma et al. 2007), the diagnostic cloud cover algorithm from CALMET was used to estimate cloud cover for constructing AERMOD meteorological files solely from MM5 data. Cloud cover is an essential element of the Holtslag and van Ulden ( 1982) energy budget model contained within CALMET. Incoming shortwave radiation infl uences many meteorologi cal variables CALMET calculates, such as Monin-Obukhov length, convec tive velocity scale, and the CBL height. Opaque cloud cover is a parameter required by CALMET, normally introduced through surface observational data. When using CALMET in its ‘no-observations’ mode, CALMET calculates a diagnostic cloud cover from the 850 mb prognostic relative humidity value derived from the MM5 hydrometeoric mixing ratio data based upo n an algorithms from Teixeira (2001). However, the CALMET implementation of this algorithm incorrectly assumed that the equation from Teixeira (2001) should only consider progno stic relative humidity from the 850 mb level, and that this value in turn repr esents total cloud cover. As not ed in Teixeira and Hogan (2002),DRAFT FOR INTERNAL REVIEW May 27, 2009

18 the algorithm actually only re presents the diagnostic cloud fraction for cumuliform clouds implemented in the Naval Research La boratory’s NOGAPS model (Hogan and Rosmond, 1991), and that stratiform clouds may be significantly underestimated (Duynkerke and Teixeira, 2001). Cumuliform clouds are typically a small, subgrid -scale feature and often play less of a role in large scale radiation balan ce represented in most climate models. More important to the global radiation balance ar e large scale stratiform cloud cover which is neglected in the CALMET implementation of its cloud diagnostic algorithm. Large scale stratiform clouds are a prominent feature of cl imate systems because of their high albedo and large areal coverage. The NOGAPS cloud scheme is a combination of the diagnostic cloud fraction for cumuliform (Teixeira, 2001) and stratiform clouds (Teixeira and Hogan, 2002). Normally, prognostic cloud cover is derived at all model levels and then a total cloud cover is calculated (Xu and Randall, 1996a, 1996b). The current implementation of diagnostic cloud cover in CALMET Version 5.8 potentially misses cumuliform cloud cover that exists both

below and above the 850 mb level as well as neglecting the larger scale stratiform clouds (Anderson, 2007b).

Comparisons with ASOS observed clouds fo r the Philadelphia area showed that the diagnosed cloud cover was on average 30% lower than the ASOS cloud cover (Evangelista, 2005). The net result is that, unde r periods of higher daytime cl oudiness as indicated by ASOS observations, insolation and sens ible heat flux estimates from the CALMET diagnosed cloud cover would be significantly higher because CALMET is only diagnosing cumuliform cloud

cover at one model level. This would result in greater atmospheric “instability” or enhanced mixing when compared to boundary layer para meter estimates when using ASOS observed clouds. Theoretically, this coul d translate into lower ground leve l concentrations as compared to ASOS derived estimates, depending on source characteristics and transport distance. The opposite effect would be occur at night, with more stable conditions expected based on CALMET diagnosed cloud cover.DRAFT FOR INTERNAL REVIEW May 27, 2009

19 

# Cloud cover classes

0 500 1000 1500 2000 2500 3000 3500 4000

<= 1

1 -2

te

n th

s 2-3 tenths

3-

4 t

e nt

h

s

4 -5

te

n th

s 5-6 tenths

6 -7

te

n th

s 7-8 tenths

8 -9

t

e n th

s 9-10 ten t hs mis s



## Cloud Cover (tenths)Number of hours



## NWS

MM5

Figure 2.3.1 – Comparison of cloud cover classes, CALMET derived v. ASOS observed (taken from Evangelista, 2005).

In order to test the impacts from these di fferences, EPA created the equivalent of a single column model by extracting radiation and boundary layer modules from CALMET and supplied both ASOS and diagnosed cloud es timates from the 2001 Philadelphia Study (Evangelista, 2005) to the off-line single column model. The resulting boundary layer parameters responded as theorized, with the enhanced insolation and sensible heat flux estimates resulting from lower cloud cover estimat es. As a result, the atmosphere was often times “less stable” during the daytime as compar ed to the ASOS cloud case, meaning that puff growth will often be enhanced using the NOOB S approach, as compared to the ASOS cloud case. Hourly Pasquill-Gifford (PG) stability classes were estimated from the Monin-Obukhov lengths based upon the work of Golder (1972). When EPA examined the downstream impact of this, it was shown that PG stability classes for the full “NOOBS” case were often times lower (less stable) during the da ytime as compared to the ASOS cloud case, and hourly stability class estimates differed on average by 1 class, but differed by as much as 4 PG classes in the same hour between the two approaches (Anderson, 2006).DRAFT FOR INTERNAL REVIEW May 27, 2009

20 Figure 2.3.2 – Comparison of cloud fraction and insolation from CALMET diagnosed cloud cover and ASOS observed cloud cover from EPA 2001 Philadelphi a Air Toxics Study (taken from Anderson, 2006).

McEwen and Murphy (2004) documented th e same behavior with CALMET for deriving PG classes when using CALMET in th e NOOBS=2 mode with NWP data from either the RAMS or MC2 NWP models. In this study, the frequencies of unstable and stable PG classes were significantly higher and the neutra l PG class frequency was significantly lower when using CALMET diagnosed cloud cover fo r both RAMS and MC2, compared to use of

measured cloud cover.

EPA concludes from these analyses that at mospheric stability derived from CALMETdiagnosed cloud covers in the full “NOOBS” approach may often differ significantly as compared to observations, and could significan tly affect modeled c oncentrations within CALPUFF, with a poten tial bias towards underprediction in some cases. Therefore, EPA cannot recommend the application of CALMET in NOOBS=2 mode. Due to the lack of adequate documentation and performance te sting of the NOOBS=1 approach, the IWAQM cannot recommend the use of this CALMET option either.DRAFT FOR INTERNAL REVIEW May 27, 2009

21 

### 2.5 INCORPORATION OF OBSERVATIO NAL DATA WITH NWP DATA

WITHIN CALMET

Traditionally, the primary method for “blendin g” observational data with NWP data, as recommended under Section 8.3.1.2(d) of the GAQM , has been through using CALMET in its “hybrid” mode (Scire and Robe, 1998) described in Section 2.1 of this document. As discussed in Section 2.3 of this document, ther e are periods when the NWP data used as the first-guess wind field substantially differs from the observation data that is incorporated into the Step 2 wind field. CALMET will incorporate all observations included during the objective analysis (OA) phase irrespective of any differe nces that may exist between the observation and the first-guess field. This can o ccur particularly if the prognostic model does not resolve terrain

effects or sea breeze circ ulations which would require much higher horizontal grid resolution to adequately simulate (NIWA, 2004). These differences can result in severe physical discontinuities in the wind field (NIWA, 2004; Scire, 2006; Anders on, 2007b; Scire, 2008). Therefore, great care must be taken to insure that the final CALMET wind fields are physically realistic. Statistical performance evaluati ons typically will not de tect these wind field

discontinuities due to the “auto correlation” issue described in Section 2.2 of this document. Visualization techniques are the only viable method to de tect these discontinuities. The IWAQM noted that “...to review and critique the CALMET results requ ires strong computer

skills for visualization of the CALMET result s... (USEPA, 1998b).” CALMET performance assessments have also been a subject of a numb er of presentations at the annual EPA Regional, State, and Local Modelers’ Workshop. Anders on (2005, 2006) laid out a paradigm for a twostep evaluation of CALMET wind fields.

“Expert judgment is required in determin ing if the prognostic meteorological model output is suitable for your domain or locat ion of interest. Visual and statistical performance evaluations are essential elem ents in determining if a data set is appropriate your area of concer n. It is necessary to perfor m both statistical performance evaluations and a visual insp ection of your prognostic data and derivative input data.”

Historically, it has been extremely difficult to determine the frequency of occurrence of these discontinuities due to the experience defici t of many end users previously discussed as well as a lack of adequate software tools to visualize the three (3 ) years of CALMET data usually generated as recommende d under Section 8.3(d) of the GAQM . The EPA must strongly emphasize that graphical analysis techniques ar e an integral part of any assessment of CALMET performance. This issue is discussed further in Section 3.3.5 of this document.

Anderson (2007a, 2007b) presented wind field sn apshots from recent applications of

CALMET/CALPUFF for BART determ inations by the state of Kansas to emphasize the need for visual inspection of CALMET “hybrid” wind fields. Anderson (2007b) sampled a onemonth period from one of the Kansas BART app lications to perform a gr aphical analysis. The result indicated a number of periods when physic al discontinuities deve loped because of the disagreements between the background NWP fiel d and the surface observations used in the CALMET OA phase. Similarly, Hawkins et al ( 2008) presented a similar issue with anotherDRAFT FOR INTERNAL REVIEW May 27, 2009

22 CALPUFF application for BART determinations . Two CALMET analyses were performed for the same geographic region and the same time periods, one incorpora ting observations using the “hybrid” method previously discussed and th e other using the “no- observations” feature (NOOBS=2). Each of the two analyses used the same MM5 data and same domain of interest. Figure 2.4.1 shows a vector difference plot based on the two CALMET wind fields. The resulting visibility estimates on a number of da ys were significantly different, sufficient enough to potentially change the outcome of a BART determination. Ka nsas attributed the introduction of observations in the “hybrid ” analysis as the primary cause of the differences.

Figure 1.4.1 – Vector difference plot between two CA LMET fields. The first field is a CALMET “hybrid”

field and the second is a “no-observation” field from CALMET. Length and directio nality of vectors displayed depict magnitude of differences between “hybri d” fields and “no-observat ion” fields (taken from Hawkins et al, 2008).

One area of potential concern that these analyses raised by these particular analyses is that physical discontinuities also appear in areas with only modest topographic relief. It has long been recognized that the potential for di scontinuities to develop in CALMET wind fields is increased in highly complex areas such as mountains or coastal environments where NWP data did not adequately reso lve important meteorological f eatures and when observations reflected these features were assigned too large of R1/R2 values in the model. The results from limited inspection of CALMET wind fields used for BART applications give rise to the

concern that the discontinuities are more prevalent than previ ously conjectured. Scire (2008) suggested that such behavior is an indication of poorly selected values of R1/R2 with station

winds that clash with MM5 fiel ds. Since CALMET applies the same radii of influence for all stations input to the model, the user must reso rt to other techniques th at have been developedDRAFT FOR INTERNAL REVIEW May 27, 2009

23 over the years, such as defining barriers, to rest rict the influence of certain observations on the interpolation of wind fields. Such techniques ar e indicative of the lim itations of DWMs to effectively utilize meteorologica l observations which may have value with respect to complex flow patterns within the modeling domain without inappropriately influencing other parts of the

domain. These difficulties and limitations are even more pronounced when multi-level observations are available (EPA, 2008b).

2.6 POTENTIAL SOLUTIONS

The issues of grid resolution and physical discontinuities were briefly discussed at

EPA’s 9 th Conference on Air Quality Modeling in October 2008. Scire (2008) summarized several potential CALMET options relative to the issue of physical discontinuities. The first option is to run CALMET in pure observation mode. The second option is to run CALMET in NOOBS mode using NWP fields only. The third option is to configure CALMET in such a way as to pass through as much of the NWP data unaltered by optimizing selection of radii of influence and minimizing changes ca used by CALMET diagnostic features.

The first option, running CALMET in pure obser vation mode, is an alternative allowed under Section 8.3.1.2(d) of the GAQM . A minimum of five (5) year s of meteorological data is required if this option is exercised. However, studies such as Irwin et al. (1996) have shown that this approach was the least desirable fr om a LRT model performance perspective when conducting mesoscale modeling. Therefore, EPA ac tively encourages the trend towards a more full use of NWP data in LRT modeling studies.

In order to reduce the potential for wind fiel d discontinuities, the New Zealand Ministry for the Environment suggested that the second option, to run CALMET in its ‘no-observations’ mode (NOOBS=2), would be a safer approach (NIWA, 2004). In its context, NIWA (2004)

stated

“It must be assumed that over a 12-month period the prognostic model will not predict some days well in (probably) all regions. If the intention is to run a dispersion model for 12 months and examine annual statistics, it may be safely assumed that the meteorological model will predict the right types of weather and at the right annual frequency, even if not on the correct day a ll the time. It is perhaps safer to use the

observations to validate the modelled meteor ology, rather than assimilating them and potentially generating unrealistic model resu lts. Extra care must be taken if the dispersion modeller wishes to use the meteorol ogical model to simulate a particular day. In that case, the meteorology has to be corr ect and must be validated against suitable

observed data.”

EPA agrees with New Zealand’s rationale toward a more full reliance upon NWP data to drive LRT model applications. However, EPA cannot endorse the NOOBS=2 approach due to the findings of Anderson (2006, 2007b) and McEwen and Murphy (2004) discussed in Section 2.4

of this document.DRAFT FOR INTERNAL REVIEW May 27, 2009

24 The revisions to the IWAQM Phase 2 reco mmendations contained in this document reflect the third option, configura tion of CALMET model control options in such a way as to preserve as much of the integrity of the orig inal NWP meteorological fi elds as is practical within the CALMET diagnostic meteorological m odel. The recommendations still encourage use of observations, but recommends assimilation of observations in such a way as to minimize the potential for the development of discontinuities in the fina l CALMET wind fields. Due to

the aforementioned technical concerns with the ‘no-observations’ a pproach in CALMET,

observations are essential to inco rporating data fields such as cloud cover which are critical for proper energy balance calculations . Likewise, since EPA’s reco mmendation is to maintain the original horizontal grid resolution of the NWP data in most situations, it would be inappropriate to apply CALMET with any diagnostic adjustme nts, unless the improved performance of the CALMET wind fields can be objectively demons trated. The CALMET firs t-guess field likely already reflects the relevant meteorological fe atures of interest at that resolution.

The revised IWAQM recommendati ons strictly imply that the candidate NWP data used should appropriately characterize the key meteorological features that govern source-receptor relations for the specific a pplication. This places a highe r emphasis on ensuring that the

candidate NWP dataset is at the a ppropriate horizontal grid resolution and that the dataset captures the key meteorological features for the specific application. Therefore, the recommendation for establishing the suitability of NWP dataset under Section 8.3(d) of the GAQM is a critical component for planning a succe ssful LRT model application. In light of these concerns, the appropriate ness and adequacy of the CALMET/CALPUFF grid resolution, as well as any prognostic model data used as i nput to CALMET, should be adequately justified based on the specific needs of the application, and measures should be taken to objectively assess the resulting meteorological fields, including both horizontal a nd vertical velocity fields, prior to their acceptance for use in CALPUFF. In accordance with Section 8.3(d) of the GAQM , EPA must reemphasize that acceptance of a prognostic da ta set is contingent upon concurrence from the appropriate reviewing aut hority. Therefore, at a minimum, any protocol should include an evaluation of the performa nce of the candidate NWP dataset prior to acceptance by the reviewing authority. Mode l performance evaluation procedures are

discussed in Section 3 of this document. Furt her, if the intent is to apply CALMET at resolutions much higher than the original NWP data set, the suitability of the resultant datasets should also be examined through the appropriate statistical and graphica l analytical methods.

Section 3.3 discusses evaluation metrics a nd procedures when combining NWP and

observational data in DWMs.

An alternative approach for incorporation of observations is via the OA preprocessors of routinely used NWP models such as MM5 and ARPS. Section 8.3.1.2(d) of the GAQM

recommends that standard NWS data be used in conjunction with NWP data for LRT model applications; however, the GAQM does not specify that CALMET must be the sole mechanism for incorporation of observations with NWP data. It is EPA’s view that NWP data prepared with OA preprocessors and FDDA satisfies the recommendation of S ection 8.3.1.2(d) of the GAQM . Recognizing the significant a dvances that have occurred w ith NWP models in the last decade and the increasing availability of multiy ear, high resolution NWP datasets, it is the federal CALPUFF workgroup’s intention to trans ition to allowing for direct coupling of the

LRT model to NWP models as an alternative to CALMET. EPA discussed this goal at the 8 thDRAFT FOR INTERNAL REVIEW May 27, 2009

25 Conference on Air Quality Modeling in 2005 (Evangeli sta, 2005a). This approach is consistent with the state-of-the-pra ctice for other LRT models such as SCIPUFF (Sykes et al., 1998) or HYSPLIT (Draxler and Hess, 1998). At the 9 th Conference on Air Quality Modeling in 2008, EPA discussed an ongoing software development project that allows for CALPUFF to be directly coupled to NWP models such as MM5 and WRF (Wong, 2008). In contrast to the CALMET OA procedures, quality assurance proc edures are applied in the OA preprocessors

that are used to “blend” obs ervations with NWP models su ch as MM5 and ARPS. These quality assurance procedures ar e applied to determine if significant differences exist between the first-guess field and the obser vations. If a difference value (first-guess value subtracted from observation value) exceeds a certain thre shold, the observation is discarded from the objective analysis (Dudhia et al., 2005). These procedures offe r the potential advantage of reducing or eliminating the di scontinuities that may devel op in the simplified CALMET OA

scheme.

This position is founded as much on a recognitio n of significant scientific advances in prognostic meteorological modeling over the past decade since the original IWAQM Phase 2 report as it is on a growing recognition of the limitations of the CALMET diagnostic model to adequately simulate the complex meteorological features of importance to LRT applications. Based on these two trends, we are further of the opinion that a prope rly applied prognostic meteorological model can provide a more scientifi cally sound and reliable basis for application of the CALPUFF dispersion model than CA LMET-derived wind fields for most LRT applications. A complete transition to this paradigm for LRT modeling will commence as soon as practicable. In the interim, EPA is recomm ending specific modifications to the original IWAQM Phase 2 recommendations.

2.7 INTERIM IWAQM RECOMMENDA TIONS FOR CALMET MODEL

CONTROL OPTIONS

In the interim until the modeling community completes the transition to the direct coupling of NWP models to LRT models as discussed in Section 2.6, the IWAQM is recommending specific model control settings for CALMET which are intended to pass through as much of the existing prognostic da ta as possible without alteration. These recommendations are formulated based upon the e xperiences of the IWAQ M in the application of the CALMET and review of the model computer code.

These recommendations include:

• CALMET Input Group 1: General run c ontrol parameters – MREG set 1, conforming to EPA guidance for IMIXH, ICOARE, and THRESHL.

• CALMET Input Group 2: Grid Control Parameters – RLON0, RLAT0, XLAT1, XLAT2, and DGRIDKM set to match grid sp ecifications of NWP data used for

STEP 1 wind field.

• CALMET Input Group 2: Grid Control Para meters – NZ set to match the exact

number of vertical le vels of NWP data betw een the surface to 4,000 - 5,000DRAFT FOR INTERNAL REVIEW May 27, 2009

26 meters above ground level. ZFACE values set to match exact layer heights of NZ layers from NWP data.

• CALMET Input Group 4: Meteorological Data Options – NOOBS set to 0.

• CALMET Input Group 5: Wind Field Options and Parameters – IWFCOD must

be set to 1. Use NWP data as initial guess wind field.

• CALMET Input Group 5: Wind Field Options and Parameters – IPROG set to 14, NWP used as initial guess wind field.

• CALMET Input Group 5: Wind Field Opti ons and Parameters – Diagnostic model control options IFRADJ, IKINE, IO BR, ISLOPE, IEXTR, BIAS are to be individually disabled. Individual values for parameters are delineated in Appendix A of this document.

• CALMET Input Group 5: Wind Field Options and Parameters – Radii of influence values for RMAX1, RMAX2, R1, and R2 are to be set to a nominal value of 0.001 km or equivalent.DRAFT FOR INTERNAL REVIEW May 27, 2009

27 

### 3.0 MODEL EVALUATION PH ILOSOPHY AND METHODOLOGY

3.1 MODEL EVALUATION PR OTOCOL OBJECTIVES

This section offers the guidance to rigor ously test the performance of CALPUFF and other LRT models in conjunction with the meteor ological fields that are used to drive the transport simulations. The objective of this evalua tion is two fold. First it is to determine whether and to what extent confidence may be placed in both a prognostic and diagnostic meteorological model’s output fields (e.g., wind, temperature, mixing ratio, diffusivity, clouds/precipitation, and radiation) that will be used as input to LRT models. This assessment centers on the reliability of output from the National Center for Atmospheric Research (NCAR)/Penn State University (PSU) Fifth Generation Mesoscale Meteorological Model (MM5), NCAR Weather Research and Fo recasting Model (WRF) and the CALMET Diagnostic Meteorological Model. Model fi eld reliability will be addressed from phenomenological (i.e., does the model simulate key processes correctly) and regulatory perspectives. In most cases the scientific eval uation of the model will have been concluded its suitability of the prognostic model for a particular application, and one of the most important questions addressed in an evaluation con cerns whether the prognostic or diagnostic meteorological fields are adequate for their inte nded use in supporting a variety of air quality modeling exercises.

These guidelines are not meant to establis h a bright line for meteorological and model performance and acceptability; however, a signifi cant amount of information can be developed by following these evaluation procedures that will enable the analyst to quantify the adequacy of the MM5 and CALMET modeling and to judge its suitability for use in modeling studies. Likewise, these guidelines are not meant to establis h bright line criteria for suitability of LRT models. As with the meteorological model eval uation process, these guid elines are intended to provide useful, quantitative assessments of the adequacy of the meteorological fields for a variety of regional air quality modeling studies and the suitabili ty of LRT models for those studies. This protocol outlines a formal model evaluation process that EPA plans to implement in future EPA guidance for both mete orological and LRT model evaluation.



### 3.2 OVERALL EVALUATION PHILOSOPHY

The objective of the model evaluation process in the regulatory context is to determine the suitability of a particular model or disti nguishing between different models for a specific regulatory niche. The framework for evaluati ng models in the long range transport (LRT) regulatory niche consists primarily of two se parate, yet related asp ects of the evaluation process. These primarily consis t of an operational evaluation and a diagnostic evaluation. The operational evaluation consists of various graphical and statistical techniques to help determine if estimates of predicted values of model variables are comparable to measured variables. The diagnostic evaluation focuses upon analyses to help determine if individual components of the modeling system are working properly.DRAFT FOR INTERNAL REVIEW May 27, 2009

28 Previous evaluation studies of LRT mode ls indicated LRT model performance was highly sensitive to both the input resolution and type of meteorologi cal fields introduced (Brandt et al, 1998). Therefore, a very impor tant, albeit often overlooked component of the diagnostic evaluation is th e meteorological evaluation.

It is a logical extension of the current evaluation philosophy to include a step in the diagnostic evaluation component for meteorological model evaluation, prior to evaluating the LRT model, since overall performance of the LRT model is inexorably linked to the input meteorology. Therefore, this protocol outline s methodologies and metrics for evaluating the LRT modeling system, including both the mete orological and dispersi on modeling components of the system.



### 3.3 METEOROLOGICAL MODEL EVALUATION COMPONENT

CALPUFF, like most LRT models , is typically coupled dire ctly to some form of a meteorological model, prognostic or diagnosti c. The CALMET diagnostic meteorological model (Scire et al, 2000a) is th e primary tool used to supply three-dimensional meteorological

data to the CALPUFF model. Current EP A guidance concerning th e application of the CALMET diagnostic meteorological model centers on the “hybrid approach.” In the “hybrid approach,” coarser scale prognosti c data is used as the initial guess field for CALMET and the wind fields are diagnostically adjusted for terrai n and slope flow effect s, producing the Step 1 wind field. In the Step 2 wind field, surface a nd upper atmospheric observations are “blended” with the background prognostic fiel d to produce a final wind field.

The preferred “hybrid” approach for CALMET does not lend itself to easy evaluation. Since the “first guess” wind field is typically a coarser scale prognostic wind field from models

such as NCAR/PSU MM5 model or the National Center for Environmental Prediction (NCEP) Rapid Update Cycle Model (RUC) that is i ngested into the CALMET diagnostic model and diagnostically changed due to terra in effects or “blended” with su rface observations to form the Step 2 wind field, it is not possible to sepa rate performance issues between the prognostic meteorological model and the diagnostic meteor ological model once diagnostic adjustment or “blending” has occurred. Th is implies that the meteorologi cal evaluation should actually encompass two phases to isolate any potential performance issues associated with the prognostic data.

Anderson (2005) outlined an evaluation paradigm for the CALMET diagnostic meteorological model. This approach consists of a two-phase evaluation process in which the prognostic meteorological data first is statistically evaluated. If the prognostic meteorological data is within proposed statistical benchmarks , then this data can be used for the CALMET diagnostic meteorological model. After running the diagnostic model, the CALMET output can be evaluated using the same statistical be nchmarks, and should, at a minimum, perform as well as, if not better, than the original prognosti c meteorological data. Th is approach reflects the previously identified paradigm that “adjusting the coarse scale flow fields produced by the MM5 model so that they represent the fine-sca le terrain seen by the CALMET model.” AsDRAFT FOR INTERNAL REVIEW May 27, 2009

29 previously stated, the CALMET simulation should, at a minimum, perform as well as the MM5 model data, and should never dete riorate the quality of the mete orological fields beyond the original MM5 data.

Typical meteorological variables used for model performance include wind speed, wind direction, temperature, and humidit y. However, since the CALMET diagnostic meteorological model uses simple interpolati on techniques to constr uct three-dimensional temperature fields and two-dimensional humidity fields, EPA believes th at the evaluation of these meteorological parameters is of s econdary importance when compared to wind

parameters.



### 3.3.1 Statistical Measures for Meteorological Fields

Key statistical parameters for evaluating th e wind from diagnostic meteorological are bias, gross error, root mean squa re error, and index of agreement. Bias (B) represents the mean difference between the model prediction and the observed data pairings within a given time period:

Gross error (E) is calculated as the absolute difference between predicte d and observed pairings for a given time period:

Root mean square error (RMSE) represents the square root of the mean squared difference in predicted and observed pairings for a given time period.

RMSE in general is considered a good overall pr edictor of model performance. But analyzing

the RMSE S and the RMSE U can identify whether the error is in the model itself or results from random influences upon the model. RMSE S is calculated as the square root of the mean squared difference between the regressed predic tion and observation pairings for a given time 

# ( ) ∑ ∑

=

= − = I

i i
j i
j J

j O P IJ B

1 1 1 (4)



# ( ) ∑ ∑

=

= − = I

i i
j i
j J

j O P IJ E

1 1 1 (5)



# ( ) 2 / 1

1 1 2 ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

− = 

## ∑∑

= = J

j I

i i
j i
j O P IJ

i RMSE (6)DRAFT FOR INTERNAL REVIEW May 27, 2009

30 period. The RMSE S estimates the model’s linear error th rough the use of the least squares regression analysis below.

The regressed prediction, P ˆ , is calculated from the linear least squares regression analysis.

RMSE U is calculated as the square root of the mean squared difference between the prediction and regressed prediction pairings fo r a given time period. The RMSE U estimates the amount of

error attributable to random influences on the model.

The final statistical parameter is the Index of Agreement (IOA). The IOA combines all of the previous metrics into one parameter to provide a measure of the match between the prediction and observation values. IOA is calculated as the ratio of the total RMSE to the sum of two differences, the difference between the predictions and observed mean (M O ) and the

difference between the observations and observed mean.



### 3.3.2 Statistical Benchmarks

There are no currently accepted performa nce criteria for prognostic meteorological models that have been established by the EPA. As noted by Tesche (2002), there is valid

concern that establishment of such criteria, unless accompan ied with a careful evaluation process such as the one outline in this section might lead to the misuse of such goals as is ⎥ ⎥ ⎥ ⎥

⎦ ⎤

⎢ ⎢ ⎢ ⎢

⎣ ⎡

− +

− ⋅

− = 

## ∑∑

= = J

j I

i O i
j O i
j M O M P RMSE

IJ IOA

1 1 2 1 (10) 

# ( ) 2 / 1

1 1 2 ˆ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

− = 

## ∑∑

= = J

j I

i i
j i
j S O P IJ

i RMSE (7)

i
j i
j bO a P + = ˆ (8)



# ( ) 2 / 1

1 1 2 ˆ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

− = 

## ∑∑

= = J

j I

i i
j i
j U P P IJ

i RMSE (9)DRAFT FOR INTERNAL REVIEW May 27, 2009

31 occasionally the case with the accuracy, bias, and error statistics recommended for judging photochemical dispersion models. In spite of th is concern, there rema ins nonetheless the need for some benchmarks against which to co mpare new prognostic and diagnostic model

simulations.

In two recent studies (Tesche et al., 2001b; Emery et al., 2001), an attempt has been

made to formulate a set mesoscale model evaluation benchmarks based on the most recent MM5/RAMS performance evaluation literature. Th e purpose of these benchmarks is not to assign a passing or failing grade to a particular meteor ological model applicat ion, but rather to put its results into a useful context. These benchmarks may be helpful to analysts in understanding the quality of their results are relati ve to the range of othe r model applications in other areas of the U.S. As Tesche (2002) noted , often lost in routine statistical ozone model

evaluations is the need to cri tically evaluate all aspects of the model via the diagnostic and process-oriented approaches. The same must be stressed for the meteorological performance evaluation. Thus, the appropria teness and adequacy of the following benchmarks should be carefully considered based upon the results of the specific meteorological model application being examined.

Based upon the above considerations, the benchmarks suggested from the studies of Emery et al, (2001) and Tesche et al., (2001) are as follows:

Table 3.3.2 – Statistical benchmarks for MM5/CALMET performance.
Parameter Metric Benchmark Wind Speed RMSE ≤ 2 m/s

Bias ≤ ± 0.5 m/s

IOA ≥ 0.6

Wind Direction Gross Error ≤ 30 deg

Bias ≤ ± 10 deg Temperature Gross Error ≤ 2 K

Bias ≤ ± 0.5 K

IOA ≥ 0.8 Humidity Gross Error ≤ 2 g/kg

Bias < ± 1 g/kg

IOA ≥ 0.6



### 3.3.3 MM5 Evaluation Methodology

Typically, in most CALMET/CALPUFF simulations, the modeler will not be responsible for the MM5 simulations that are us ed as the first guess wind field for CALMET. This reality does not relieve th e CALPUFF modeler of the res ponsibility of understanding the performance of the underlying MM5 simulation, si nce if CALMET is being run in its “hybrid” mode, its performance is ultim ately linked to the quality of the prognostic data sets.DRAFT FOR INTERNAL REVIEW May 27, 2009

32 In general, the MM5 evaluation will have been performed with both scientific and policy perspectives in mind. While the EPA has not explicitly established methodologies or benchmarks for meteorological model evaluatio ns, our experience has shown that the air quality modeling community that the majority of th e evaluations utilize, in some or fashion, the

methods and metrics outlined in the previously cited literature (Tesche et al., 2001b; Emery et al., 2001), which is also reflected in EPA’s Guidance on the Use of Models and Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, PM2.5, and Regional Haze (EPA-454/B-07-002) (USEPA, 2007).

Typically, the prognostic model will have been evaluated the model over the continental United States with a number of subregional an alyses. The most common subregions usually correspond to the five Regional Planning Orga nization (RPO) domains . The goal of these additional subregional and sub-tem poral evaluations is to build confidence in the use of the model for regulatory air quality decision-making and to iden tify potential problem areas (should they exist) in the MM5 meteorological fi elds. These subregiona l evaluations will be aimed at elucidating the model’s ability to predic t key processes at the smaller time and scales (e.g. coastal circulation regimes) associated with specific RPO regions. The most routine source of prognostic meteorological data availabl e in the regulatory co mmunity is MM5 data generated by the five RPO’s. However, it is also anticipated that additional data sets will

become available from both the EPA and FLMs.

While in most cases EPA believes that the performance evaluation conducted by the RPO’s for their purposes would be sufficient to establish suitability for use as the first guess wind field in CALMET, there are special cases when this may not be the case, such as modeling in coastal environments or where rapid terrain and/ or landuse changes exist that cannot be adequately represented. This repr esents the primary motivation for employing the CALMET diagnostic model at higher resolutions. In those cases, it is recommended that the MM5 performance analysis be redone with a subdomain corresponding directly to the CALMET domain under evaluation. In this manner, the analyst is obtaini ng direct information

about the MM5 data in the area of interest be fore it is ingested into the CALMET model, allowing for isolation of potenti al issues in the development of the CALMET wind fields.



### 3.3.4 CALMET Evaluation Methodology

As described in Section 2.2.2, CALM ET should, at a minimum, meet the MM5 benchmarks to demonstrate acceptability of th e CALMET data over the region of interest. EPA used its CALMETSTAT (Anderson, 2006) software developed for statistical evaluation of

CALMET wind fields. The statistical benchmar ks were also applied to the CALMETSTAT daily average output for each of the experiments to determine the wind field with the statistical performance most closely in keep ing with the benchmarks. As this wind field will be input for the CALPUFF dispersion model, the wind dire ction and wind speed are the parameters of importance. In theory, good performance on wi nd direction should yi eld a predicted plume where the center of mass closely matches th e observed plume. Also, good performance on wind speed should yield a plume with the same timing as the observed plume so that the concentrations of tracer gas ali gn in time as well as space. Both tabular and graphical displays of statistical performance measures will be utilized.DRAFT FOR INTERNAL REVIEW May 27, 2009

33 

### 3.3.5 Graphical Evaluation Tools

Over the years, a rich variety of graphi cal analysis and display methods have been developed to evaluate the pe rformance of mesoscale meteor ological models. Besides the

statistical measures described in the preceding section, there are a number of procedures for graphically representing model results and observations that allow for direct comparison between them. For graphical evaluation of pr ognostic meteorological data such as from MM5, time series and bar chart comparisons of st atistical measures ar e a common method for displaying such data (Figure 4.1). The para meters to be emphasized include but are not necessarily limited to bias, relative error, root mean square error, and index of agreement. These measures will be plotted in various ways for temperature, wind speed, wind direction, water vapor mixing ratio for th e prognostic data, but wind is the primary concern for the CALMET system. Currently, the graphical tools used to examine model performance examine conditions only at the surface. The EPA and FLMs are currently developing software to graphically evaluate CALMET performance aloft.

An almost equal portion of the evaluation of the diagnostic meteorological model is the graphical evaluation. If the an alyst is using prognostic data fr om MM5 as the first guess wind field in CALMET and the prognostic data differs significantly from th e observations in the vicinity of where observations are incorporated into CALMET, the resulting wind fields will be unrealistic (Figure 4.2). It is not always possi ble to determine the frequency with which such disagreements between the first guess wind field and the observati ons will occur, but graphical analysis of the wind fields is the only practic al method to detect such errors, because the statistical analysis likely will not detect the erro rs due to “autocorrelation” – e.g. incorporating the same data in the CALMET solution and the statistical analysis.

A number of options exist for graphica lly displaying CALMET wind fields, with advantages and disadvantages of each opti on. The most common method for displaying

CALMET wind field data is th rough producing vector plots from the CALPOST package and displaying static vector plots in commercial packages such as Golden Software’s SURFER package. This has the distinct advantage of the ease of use and the wide spread use of SURFER in the air quality modeling community. However, it is largely impractical for very large data sets because individual plots of hour ly winds must be generated in the SURFER package. In recent distributions of the CA LPUFF graphical user interface (GUI), another package called CALVIEW has been implemen ted to read SURFER files generated from CALPOST to provide a seamless time series view of winds. This is one potential feature that bears further investigation for gra phical evaluation of CALMET winds.

Additional software is available for disp laying and animating CA LMET wind field data. In recent years, EPA has adapted programs de veloped by the US Forest Service for their BlueSky smoke dispersion forecasting system. Software has been developed to convert CALMET output into either MODELS3 IOAPI format or into Vis5D format. MODELS3 IOAPI format data can be readily displayed in programs such as the Package for Analysis and Visualization of Environmental Data (PAVE). Vis5D is a package which allows full threedimensional view of the CALMET wind fields, a feature which no othe r package currentlyDRAFT FOR INTERNAL REVIEW May 27, 2009

34 offers. The primary disadvantage of either of these options is that they currently are only available on computer platforms running the Li nux OS with the majority of the CALPUFF modeling community operati ng in the Windows OS.

No recommendations can be offered for se lecting the graphical evaluation tools for portraying the CALMET simulation results, we will draw from among several approaches

which are best suited for indivi dual needs. However, given cu rrent regulatory requirements for use of three years worth of prognostic data fo r air quality modeling with CALMET/CALPUFF,

careful consideration should be given the visualization techni ques which allow for rapid and easy visualization of CALMET wind fields for ex tended periods (e.g. animations) rather than the construction of individual snap shots of wind field behavior.

Observed/Predicted Windspeed

0 5 10

7/ 8 7/ 9 7/10m/s ObsWndSpd PrdWndSpd

Bias Windspeed

-2 0 2

7/ 8 7/ 9 7/10m/s BiasWndSpd

RMSE Windspeed

0 2 4

7/ 8 7/ 9 7/10m/s RMSEWndSpd RMSESWndSp RMSEUWndSp

IOA Windspeed

0 0.5

1

7/ 8 7/ 9 7/10 IOAWndSpd Observed/Predicted Windspeed

0 5 10

7/ 8 7/ 9 7/10m/s ObsWndSpd PrdWndSpd

Bias Windspeed

-2 0 2

7/ 8 7/ 9 7/10m/s BiasWndSpd

RMSE Windspeed

0 2 4

7/ 8 7/ 9 7/10m/s RMSEWndSpd RMSESWndSp RMSEUWndSp

IOA Windspeed

0 0.5

1

7/ 8 7/ 9 7/10 IOAWndSpd

Figure 3.3.1 – Example of hourly wind statistics from M ETSTAT/CALMETSTATDRAFT FOR INTERNAL REVIEW May 27, 2009

35 Figure 3.3.2 – Example graphical analysis of the resul tant “hybrid” CALMET wind field illustrating effect when disagreement between MM5 first-gu ess wind field and the observations.



### 3.4 LONG RANGE TRANSPORT DISP ERSION MODEL EVALUATION

COMPONENT

3.4.1 LRT Model Evaluation Philosophy

Irwin (1997) focused his evaluation of the CA LPUFF modeling system on its ability to replicate centerline concentra tions and plume widths, with more emphasis placed upon these

factors than data such as modeled/observed plume azimuth, plume arrival time, and plume

transit time. The Great Plains and Savannah Ri ver tracer evaluations (EPA, 1998) followed the methodology of the INEL Study (Irwin, 1997).

Given the unique role that the CALPUFF m odeling system has within the hierarchy of EPA dispersion models for conducting LRT a nd the typical methodology for conducting LRT simulations, greater emphasis on the evaluation of the spatial and temporal metrics is warranted. The typical regulat ory application of the CALP UFF modeling system is for Prevention of Significant Deterioration of Air Qu ality (PSD) Class I air quality related values (AQRVs) (visibility, depositi on, etc.) and increment analyses. When employed for theseDRAFT FOR INTERNAL REVIEW May 27, 2009

36 purposes, it is customary to only model discre te receptors defined within the boundaries of national parks and wilderness areas (federal manda tory Class I areas with specially protected air quality related values) and compare modeled concentrations against short-term averaging periods with few exceedance periods. This implies a fundamentally different philosophy to model evaluation for a dispersion model than other EPA workhorse models such as the Industrial Source Complex (ISC) model or AE RMOD employed for evaluations within 50 kilometers. When evaluating these models, EPA focuses upon a model’s ability to replicate the highest end of the concentration distribution, regardless of tem poral or spatial pairing.

This philosophy is embodied in its Guidel ine on Air Quality Models (EPA, 2005) with

the statement “ the models are reasonably reliable in es timating the magnitude of the highest concentrations occurring sometime, somewhere within an area. ” However, the methodology employed with the CALPUFF modeling system is fundamentally different , and better spatial and temporal correlation than other EPA dispersi on models is implicit in its use. Therefore, these analyses place equal emphasis upon a model’ s ability to simulate spatial and temporal pairing through analysis of plume centerline azimuths, arrival times, and plume arc transit

times.



### 3.4.2 Irwin Evaluation Methodology

There are a number of visual and statistical measures that can be employed to evaluate model performance. In the previous sect ion, an integrated methodology for evaluating prognostic and diagnostic meteorological modeli ng results within a common framework based upon a set of routinely used statistical measures was introduced. In this study, a variation of the method employed by Irwin (1998) is used . Irwin examined CALPUFF performance by calculating the cross-wind integrated concentrat ion (CWIC), azimuth of plume centerline, and

the second moment of tracer concentration (lateral di spersion of the plume ( σ y )). The CWIC is calculated by trapezoidal integr ation across average monitor con centrations along the arc. By assuming a Gaussian distribution of concentrat ions along the arc, a fi tted plume centerline concentration (Cmax) can be calculated by the following:

The measure σ y describes the extent of plume horizon tal dispersion. This is important to understanding differences between the variou s dispersion options avai lable in the CALPUFF modeling system. Additional measures for tem poral analysis include plume arrival time and the plume transit time on arc.

Table 3.4.2 - Model Performance Metrics from 1998 EPA Evaluation. Spatial Temporal Performance

Azimuth of Plume

Centerline Plume Arrival Time Crosswind Integrated

Concentration Plume Sigma-y Transit Time on Arc Observed MaximumDRAFT FOR INTERNAL REVIEW May 27, 2009

37 The measures employed by Irwin (1998) a nd EPA (1998) provide useful diagnostic information about the performa nce of LRT modeling systems su ch as CALPUFF, but they do not always lend themselves easily to spatiotemporal analysis or direct mode l intercomparison.

For tracer studies such as the Great Pl ains Tracer Experiment and Savannah River

where distinct arcs of monito rs were present, the Irwin ev aluation approach was used. In addition to the Irwin methodology, EPA em ployed statistical measures focusing upon spatiotemporal comparisons of model-observati on pairings. After an extensive literature

review of recent LRT model performance evaluations, the EPA decided to employ model performance metrics adopted for the second Atmospheric Model Evaluation Study (ATMESII).



### 3.4.3 Statistical Evaluation Methodology

The model evaluation methodology employed fo r this project was designed following the procedures of Mosca et al . (1998) and Draxler et al. (2001) . Mosca et al. (1998) defined three types of statistical analyses:

• Spatial analysis – concentrations at a fixed time are considered over the entire domain. Useful for determining differen ces spatial differences between predicted

and observed.

• Temporal analysis – concentrations at a fi xed location are considered for the entire analysis period. This can be useful for determining differences bew

• Global analysis – all concentr ation values at any time and location are considered in this analysis. The global analysis considers the distribution of the values (probability), overall tendency toward s overestimation or underestimation of measured values (bias and error).



### 3.4.3.1 Spatial Analysis

To examine similarities between the predicted and observed ground level concentrations, the figure of merit in space (FMS) is calculated at a fixed time and for a fixed

concentration level. The FMS is defi ned as the ratio be tween measured ( A M ) and predicted ( A P ) areas above a significant concen tration level and their union:

The more that the predicted and measured trace r clouds overlap one another, the greater the FMS values are. A high FMS value co rresponds to better model performance. % 100 ×

∪ ∩

=

P M P M A A A A FMS (11)DRAFT FOR INTERNAL REVIEW May 27, 2009

38 EPA decided to augment the FMS statistic with additional spatial performance measures of probability of detection (POD), fals e alarm rate (FAR), and threat score (TS). Typically used as a method for meteorological fo recast verification, these three interrelated statistics are useful descriptions of an air quali ty model’s ability to spatially forecast a certain

condition. The forecast condition for the model is the predicted concentration above a userspecified threshold (at the 0.1 ngm -3 level for ATMES-II study). In these equations, A represents the number of times a condition that has been foreca st, but was not observed (false alarm). B represents the number of times the c ondition was correctly forecasted (hits). C represents the number of times the nonoccurrence of the condi tion is correctly forecasted (correct negative) and D represents the number of times that the condition was observed but not forecasted (miss).

The FAR (Equation 12) is described as a meas ure of the percentage of times that a condition was forecast, but was not observed. The range of the score is 0 to 1 or 0% to 100%,

with the ideal FAR score of 0.

The POD is a statistical measure which desc ribes the fraction of observed events of the condition forecasted was correctly forecasted. Equation 13 shows that POD is defined as the

ratio of “hits” to the sum of “hits” and “misses.” The range of the POD score is 0 to 1 (or 0%to 100%), with the ideal score of 1 (100%).

The TS (Equation 14) is described as the measure describing how well correct forecasts corresponded to observed conditions. The TS doe s not consider correctly forecasted negative conditions, but penalizes the score for both false alarms and misses. The range of the TS is the same as the POD, ranging from 0 to 1 (0% to 100%), with th e ideal score of 1 (100%).



### 3.4.3.2 Global Statistical Analysis

Following Draxler et al. (2001), four broad cate gories were used for model evaluation. These broad categories are 1) scatter, 2) bias, 3) spatial distribution of predictions relative to % 100 × ⎟ ⎠ ⎞ ⎜ ⎝ ⎛

+ + = d b a b TS (14) % 100 × ⎟ ⎠ ⎞ ⎜ ⎝ ⎛

+ = d b b POD (13) % 100 × ⎟ ⎠ ⎞ ⎜ ⎝ ⎛

+ = b a a FAR (12)DRAFT FOR INTERNAL REVIEW May 27, 2009

39 measurements, and 4) differences in the dist ribution of unpaired measured and predicted

values. One or more statistical measures are us ed from each of the four categories in the global analysis. These include the percent over-predicti on, number of calculations within a factor of 2 and 5 of the measurements, normalized mean sq uare error, correlation coefficient, bias, fractional bias, figure of merit in space, a nd the Kolomogorov-Smirnov parameter representing

the differences in cumulative dist ributions (Draxler et al., 2001).

Factor of Exceedance : In the scatter category, bette r model performance is observed when the FOEX measure is close to zero and FA2 has a high percentage. A high positive FOEX and high percentage of FA5 would indi cate a model’s tendency towards overprediction when compared to observed values.

where N in the numerator is the number of pairs where the prediction ( P ) exceeds the measurement ( M ) and the N in the denominator is the total numb er of pairs in the evaluation. In FOEX, all 0-0 pairs are excluded from the anal ysis. FOEX can range from -50% to +50%.

Factor of α (FA α ) : FA α represents the percen tage of predicted values that are within a

factor of α where α = 2 or 5. As with FOEX, in FA α all 0-0 pairs are excluded.

Normalized Mean Square Error (NMSE) : Normalized mean square error is the average of the square of the differences divided by the pr oduct of the means. NMSE gives information about the deviations, but does not yield estimations of model overprediction or underprediction.

Pearson’s Correlation Coefficient (PCC) : Also referred to as the linear correlation coefficient, its value ranges between -1 and +1 . A value of +1 indi cates “perfect positive correlation” or having all pairings of (M i , P i ) lay on straight line on a scatter diagram with a positive slope. Conversely, a value of -1 indica tes “perfect negative corr elation” or having all 

## ( ) 2 1 

## ∑

− = i i M P PM N NMSE (17) ( ) % 100 5

. 0 × ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

− = >
N N FOEX i i Ni P (15)



## [ ] ( ) 100 0 0
× ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

− =

−

= N x x y y N FA α α (16)DRAFT FOR INTERNAL REVIEW May 27, 2009

40 pairings of (M i , P i ) lie on a straight line with a negative slope. A value of near 0 indicates the

clear absence of relationsh ip between the model predic tions and observed values.

Fractional Bias (FB): Calculated as the mean differen ce in prediction-observation pairings

with valid data.

Kolmogorov-Smirnov Parameter (KS) : The KS parameter is defined as the maximum

difference between two cumulative distributions. The KS parameter provides a quantitative

estimate where C is the cumulative distribution of the measured and predicted concentrations over the range of k. The KS is a measure of how well the model reproduces the measured concentration distribution rega rdless of when or where it occurred. The maximum difference between any two distributions cannot be more than 100%.

Draxler et al. (2001) correc tly opined that a single m easure describing the overall performance of a model would be highly valuable. Stohl et al . (1998) evaluated many of the

above measures and discovered ratio based st atistics such as FA2 and FA5 were highly susceptible to measurement errors . Draxler proposed a single metric which is the composite of

one statistical measure from each of the four broad categories.

The final score, model rank ( RANK ), provides a combined meas ure to facilitate model intercomparison. RANK is the sum of four of th e statistical measures fo r scatter, bias, spatial coverage, and the unpaired dist ribution. RANK scores ra nge between 0 and 4 with 4 representing the best model rank ing. Using this measure allows for direct intercomparison of

models across each of the four broader statistical categories. 

## ( ) ( ) 100 / 1 100 / 2 / 1 KS FMS FB R RANK

− + +

− + = (21) 

## ( ) ()

k k P C M C Max KS

− = (20) 

# ( ) M P B FB + = 2 (19) 

# ( ) ( )

( ) ( ) ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

− ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

− − •

−

= 

## ∑ ∑ ∑

2 2 P P M M P P M M

R

i i i i i (18)DRAFT FOR INTERNAL REVIEW May 27, 2009

41 

### 3.5 GRAPHICAL METHODOLOGIES

In addition to the statistical measures descri bed in Sections 3.4.3.1 and 3.4.3.2, scatter plots of model/observed pairs and other graphical methods fo r assessing model performance should also be employed.

Figure 3.5.1 – Example global scatter plot for LRT model performance evaluations. Solid line represents 1:1 line, dashed lines are the FA2 and dotted lines are the FA5.

Figure 3.5.2 – Example of azimuth of fitted plume on receptor arc.DRAFT FOR INTERNAL REVIEW May 27, 2009

42 

### 4.0 EVALUATION STUDI ES AND FINDINGS



## TO BE PROVIDEDDRAFT FOR INTERNAL REVIEW May 27, 2009

43 

### 5.0 REFERENCES

Anderson, B.A., 2005: Use of Prognostic Meteorological Modeling Data for the CALMET/CALPUFF Modeling System: Alternativ es. Presentation at the 2005 EPA Regional, State, and Local Modeler’s Meeting, New Orleans, LA, May 16 – 20, 2005. http://cleanairinfo.co m/modelingworkshop/pres entations/MM5_Anderson.pdf

Anderson, B.A., M. Evangelista, and D. Atkins on, 2006. Evaluating Recent Enhancements to the CALPUFF Modeling System: Phase I – MM5/CALMET Performance Evaluations. Presented at Guideline on Air Quality Models: Applications and FLAG Developments – An AWMA Specialty Conference, De nver, CO, April 26 – 28, 2006.

Anderson, B.A., 2006: Use of Prognostic Meteorol ogical Model Output in Air Quality Models: The Good, the Bad, and the Ugly. Presentati on at the 2006 EPA Regional, State, and Local Modeler’s Meeting, San Di ego, CA, May 16-18, 2006. http://www.cleanairinfo.com/regionalstatelo calmodelingworkshop/old/2006/documents/RSL1P RESENTATION.pdf

Anderson, B.A., 2007a: Evaluation of Prognos tic and Diagnostic Meteorological Data. Presentation at the 2007 EPA, Regional, State, and Local Modeler’s Meeting, Virginia Beach, VA, May 15 – 17, 2007. http://www.cleanairinfo.com/regionalstatelo calmodelingworkshop/arch ive/2007/presentations/ Wednesday%20-%20May%2016%202007/ Performance_Evaluation.pdf

Anderson, B.A., 2007b: Illustration of Mete orological Issues – CALMET Diagnostic Meteorological Model. Presentation at the 2007 EPA, Regional, State, and Local Modeler’s Meeting, Virginia Beach, VA, May 15 – 17, 2007. http://www.cleanairinfo.com/regionalstatelo calmodelingworkshop/arch ive/2007/presentations/ Wednesday%20-%20May%2016%202007/Met_Example.pdf

Anderson, B.A., 2008: The USEPA MM5CALPUFF Software Project. Presented at 12 th
Annual George Mason University Conferen ce on Atmospheric Transport and Dispersion Modeling, Fairfax, VA, July 8-10, 2008.

Anderson, B.A., and R.W. Brode, 2009a: Evaluati on of Results of Four Lagrangian Dispersion Models against the 1994 European Tracer Experiment.

Bellasio, R., Maffeis, G., Scire, J.S., M.G. Longoni, R. Bianconi, and N. Quaranta, 2005: Algorithms to account for topographic shading effects and surface temperature dependence on terrain elevation in diagnostic meteorological models. Boundary-Layer Meteorology

, 114 , 595-

614.

Chan, S.T., and G. Sugiyama, 1997: A New Model for Generating Mass-Consistent Wind Fields Over Complex Terrain. Preprints Ameri can Nuclear Society’s Sixth Topical Meeting on Emergency Preparedness and Response, 22- 25 August 1997, San Francisco, CA, 4 pp.DRAFT FOR INTERNAL REVIEW May 27, 2009

44 Chandrasekar, A., C.R. Philbrick, R. Clark, B. Doddridge, P. Ge orgopoulos, 2003: Evaluating the performance of a comput ationally efficient MM5/CALMET system for developing wind field inputs to air quality models. Atmos. Environ.

, 37 , 3267-3276.

Chang, J.C., P. Franzese, K, Chayantrakom, a nd S.R. Hanna, 2003: Eval uations of CALPUFF, HPAC, and VLSTRACK with two mesoscale field datasets. J. App. Meteor.

, 42 , 453-466.

D’Amours, R., 1998: Modeling the ETEX Plume Di spersion with the Canadian Emergency Response Model. Atmos. Environ.

, 32 , 4335 – 4341.

Deng, A., N.L. Seaman, G.K. Hunter, and D.R. Stauffer, 2004: Evalua tion of Interregional Transport Using the MM5-SCIPUFF System. J. App. Meteor.

, 43 , 1864-1885.

Deng, A., and D.R. Stauffer, 2006: On Im proving 4-km Mesoscal e Model Simulations. J. App.

Meteor.

, 45 , 361-381.

Draxler, R.R., and G.D. Hess, 1997: Descriptio n of the Hysplit_4 modeling system, Tech. Rep. NOAA Tech Memo ERL ARL-224, National Oceanic and Atmospheric Admi nistration, Silver Springs, MD, 24 pp.

Draxler, R.R., and G.D. Hess,1998: An overvie w of the HYSPLIT_4 modelling system for trajectories, dispersi on, and deposition. Australian Meteorological Magazine

, 47 : 295-308.

Draxler, R.R., J.L. Heffter, and G.D. Ro lph, 2001: DATEM: Data Archive of Tracer Experiments and Meteorology, National Oceanic and Atmospheric Admi nistration, Silver Springs, MD, 27 pp. [Available online at http://www.arl.noaa.gov/datem ]

Dudhia. J., D. Gill, K. Manning, W. Wang, a nd C. Bruyere, 2005: Tutorial Class Notes and User’s Guide: MM5 Modeling System Version 3, Tech. Rep. National Center for Atmospheric Research (NCAR), Boulder, CO. http://www.mmm.ucar.edu/mm5/documents/tuto rial-v3-notes-pdf/objective_analysis.pdf

Duynkerke, P. G., and J. Teixeira, 2001: A comp arison of the ECMWF Reanalysis with FIRE I

observations: Diurnal variation of marine stratocumulus. J. Climate, 14, 1466–1478.

Earth Tech, 2001: CALPUFF/MM5 Study Report: Final Report. Tech Rep. Earth Tech, Inc., Concord, MA 40 pp. www.dec.state.ak.us/ai r/ap/docs/finrep.pdf

Emery, C., E. Tai, and G. Yarwood, 2001: “Enhanced Meteor ological Modeling and

Performance Evaluation for Two Texas Ozone Episodes”, report to the Texas Natural

Resources Conservation Comm ission, prepared by ENVIRON, International Corp, Novato, CA.

Evangelista, M., 2005a: CALPUFF: Updates, Applications, and Recommendations.

Presentation at 8 th Conference on Air Quality Modeling, Research Triangle Park, NC, September 22-23, 2005.DRAFT FOR INTERNAL REVIEW May 27, 2009

45 http://www.epa.gov/ttn/scram/8thmodconf/pre sentations/day1afternoon/calpuff-iwaqm.ppt

Evangelista, M., 2005b: Use of Prognostic Mete orological Output in Dispersion Models.

Presentation at 8 th Conference on Air Quality Modeling, Research Triangle Park, NC, September 22-23, 2005. http://www.epa.gov/ttn/scram/8thmodconf/prese ntations/day1afternoon/ prognosticmetdispersio

n.ppt

Golder, D., 1972, Relations among stability parameters in the surface layer, Boundary Layer Meteorology

, 3 , 47 – 58.

Hakwins, A., Y. Tang, and B.A. Anderson, 2008: Regional Haze & BART: The Kansas Experience. Presentation at th e 2008 EPA Regional, State, Lo cal Modeler’s Meeting, Denver, CO, June 10 – 12, 2008. http://www.cleanairinfo.com/regionalstatelo calmodelingworkshop/arch ive/2008/presentations/ Andy_The%20Kansas%20BART%20Experience.pdf

Hogan T. F., and T. E. Rosmond, 1991: The desc ription of the U.S. Navy Operational Global Atmospheric Prediction System 's spectral forecast model. Mon. Wea. Rev.

, 119 , 1786–1815.

Holtslag, A.A.M., and A.P. van Ulden, 1982: A simple scheme for daytime estimates of the

surface fluxes from routine weather data. J. Clim. And Appl. Meteor

., 22 , 517-529.

Irwin, J.S., J.S. Scire, and D.G. Stri maitis, 1996: A Comparison of CALPUFF Modeling Results with CAPTEX Field Data Results. Air Pollution Modeling and Its Application XI. Edited by S.E. Gryning and F.A. Schiermeier. Plenum Press, New York, NY., pg 603-611.

Johnson, J., Y. Jia, C. Emery, R. Morris, Z. Wang, and G. Tonneson, 2006: Comparison of 36 km and 12 km MM5 Model Runs for 2002. Presentation to CENRAP Modeling Workgroup, May 23, 2006. http://pah.cert.ucr.edu/aqm/cenra p/ppt_files/CENRAP _2002_36km_vs_12km_MM5_May22_2 006.ppt

Kemball-Cook, S., Y. Jia, C. Emery, R. Morri s, Z. Wang, and G. Tonnesen, 2004: Comparison of CENRAP, VISTAS, and WRAP 36 km MM5 Mode l Runs for 2002, Task 3: Meteorological Gatekeeper Report. Presentation to CENR AP Modeling Workgroup, December 14, 2004. http://pah.cert.ucr.edu/aqm/cenra p/ppt_files/CENRAP_VISTAS_WRAP_2002_36km_MM5_e val.ppt

Mass, C.F., D. Ovens, K. Westrick, and B. A. Colle, 2002: Does Increasing Horizontal

Resolution Produce More Skillful Forecasts? Bull. Amet. Meteor. Soc.

, 83 , 407-430.

McEwen, B., and B. Murphy (200X): The Use of High Resolution Numerical Fields for Regulatory Dispersion Modeling: An Analysis of RAMS and MC2 fields over Kamloops, B.C.

Presentation at the 13 th Conference on the Applications of Air Pollution Meteorology with the Air & Waste Management Association, Vancouve r, B.C., Canada, August 23 – 26, 2004, 4 pp.DRAFT FOR INTERNAL REVIEW May 27, 2009

46 Nasstrom, J.S., and J.C. Pace, 1998: Evaluati on of the Effect of Meteorological Data Resolution on Lagrangian Particle Dispersi on Simulations Using the ETEX Experiment. Atmos.

Environ.

, 32 , 4187-4194.

NIWA, 2004: Good Practice Guide for Atmosp heric Dispersion Modelling, Tech. Rep. Ministry for the Environment, Wellington, NZ, 152 pp.

NPS, 2000: Phase I Report of the Federal Land Managers’ Air Quality Related Values Workgroup (FLAG). National Park Service, Air Resources Division ; U.S. Forest Service, Air Quality Program; U.S. Fish and Wildlife Service, Air Quality Branch. http://www.nature.nps.gov/air/Pubs/pdf/flag/FlagFinal.pdf

Robe, F.R. and J.S. Scire, 1998: Combini ng Mesoscale Prognostic and Diagnostic Wind Models: A Practical Approach for Air Quality A pplications In Complex Terrain. Preprints 10th Joint Conference on the Appli cations of Air Pollution Mete orology, 11-16 January 1998, Phoenix, Arizona, 4 pp.

Scire, J.S., F.R. Robe, 1997. Fine-scale appl ication of the CALMET me teorological model to a complex terrain site. 90 th Annual Meeting of A&WMA, 8 -1 3 June, Toronto, Canada,12 pp.

Scire, J.S., F.R. Robe, M.E. Fernau, and R. J. Yamartino, 2000a: A User’s Guide for the CALMET Meteorological Model (Ver sion 5). Tech. Rep., Earth T ech, Inc., Concord, MA 332

pp. http://www.src.com/calpuff/download

Scire, J.S., D.G. Strimaitis, and R.J. Yama rtino, 2000b: A User’s Guide for the CALPUFF Dispersion Model (Version 5), Tech. Rep., Earth Tech, Inc., Concord, MA, 521 pp. http://www.src.com/calpuff/download

Scire, J.S., 2006: VISTA’s BART Flowchart – Status of FLAG and Class I Area Impact Modeling Panel Session. Presentation at the Gu ideline on Air Quality Models: Applications and FLAG Development Specialty Conference of the Air & Waste Management Association, Denver, CO, April 26-28, 2006. http://www.awma.org/events/confs/AQMODELS06/Intro/0- Panel_Scire.pdf

Scire, J.S., 2008: Development, Maintenance and Evaluation of CALPUFF. Presentation at 9 th
Conference on Air Quality Modeling, Research Tr iangle Park, NC, October 9 -10, 2008. http://www.epa.gov/scram001/9t hmodconf/scire_calpuff.pdf

Scire, J.S., 2009: Modeling in a Complex Terrai n Environment at High Latitudes. Presentation at the The Latest Developments in Air Mode ling Specialty Conferen ce of the Air & Waste Management Association, Toronto, Ontario, Canada, January 19-22, 2009. http://www.awma.org/proceedings/CDNairmodeling2009.html

Seaman, N.L., 2000. Meteorological m odeling for air-quality assessments. Atmos. Environ.

, 34 , 2231-2259.DRAFT FOR INTERNAL REVIEW May 27, 2009

47 Sykes, R.I., S.F. Parker, D.S. Henn, C.P. Cerasoli, and L.P. Santos, 1998: PC-SCIPUFF Version 1.2PD, Technical Documentation. ARAP Report 718, Titan Research and Technology Division, Titan Corp., Princeton, NJ, 172 pp.

Teixeira, J., 2001: Cloud Fract ion and Relative Humidity in a Prognostic Cloud Fraction

Scheme. Mon. Wea. Rev.

, 129, 1750-1753.

Teixeira, J., and T.F. Hogan, 2002: Boundary Layer Clouds in a Global Atmospheric Model: Simple Cloud Cover Parameterizations. J. Clim.

, 15 , 1261 – 1276.

Tesche, T.W., 1994. Evaluation Procedures for Regional Emissions , Meteorological, and

Photochemical Models. Presented at the 86th Annual Meeting of the Air and Waste Management Association, 14-18 June, Denver, CO.

Tesche, T.W., D.E. McNally, C.A. Emery, E. Tai. 2001. Evaluation of the MM5 Model Over the Midwestern U.S. for Three 8-hour Oxidant Ep isodes. Prepared for the Kansas City Ozone Technical Workgroup, by Alpine Geophyisi cs, LLC, Ft. Wright , KY, and ENVIRON International Corp., Novato, CA.

Tesche, T.W., D.E. McNally, and C. Trem back, 2002. Operational Evaluation of the MM5 Meteorological Model Over the Continenta l United States: Protocol for Annual and Episodic Evaluation. Prepared for US EPA by Alpine Geophysics, LLC , Ft. Wright, KY, and ATMET, Inc., Boulder, CO. http://www.epa.gov/scram001/reports /tesche_2002_evaluation_protocol.pdf

Tiedtke, M., 1993: Representation of Clouds in Large Scale Models. Mon. Wea. Rev.

, 121,

3040-3061.

Touma, J.S., V. Isakov, A.J. Cimorelli, R.W. Brode, and B.A. Anderson, 2007: Using Prognostic Model – Generated Mete orological Output in the AERMOD Dispersion Model: An Illustrative Application in Philadelphia, PA. J. Air & Waste Manage. Assoc

., 57 , 586-594.

TRC Environmental Corporation, 2009: Modeling Protocol for a BART Assessment of the Big Stone I Coal-Fired Power Plant, Big Stone City, South Dakota. Prepared for Otter Tail Power Company, Big Stone City, SD, 39 pp.

USEPA, 1998a: A Comparison of CALPUFF Modeling Results to Two Tracer Field Experiments. Tech. Rep., EPA-454/R-98-009, Research Triangle Park, NC, 48 pp. http://www.epa.gov/scram001/7thconf/calpuff

USEPA, 1998b: Interagency Workgroup on Air Quality Modeling (IWAQM) Phase 2 Summary Report and Recommendations for Modeli ng Long Range Transport Impacts. Tech Rep., EPA-454/R-98-009, Research Triangle Park, NC, 160 pp. http://www.epa.gov/scram001/ 7thconf/calpuff/phase2.pdfDRAFT FOR INTERNAL REVIEW May 27, 2009

48 USEPA, 2000: Transcripts from 7 th Conference on Air Quality Modeling, Washington, D.C., June 28 – 29, 2000. http://www.epa.gov/ttn/scram/7thc onf/information/proc6-28.pdf

USEPA, 2003: Summary of Public Comments and EPA Responses, 7 th Conference on Air Quality Modeling, Washington, D.C., June 28 – 29, 2000. http://www.epa.gov/ttn/scram/guidance/guide/response.pdf

USEPA, 2005: EPA, 2005. Guideline on Air Qu ality Models, 40 CFR Part 51, Appendix W.

Published in the Federal Register , Vol. 70, No. 216, November 9, 2005. http://www.epa.gov/scram001/ guidance/guide/appw_05.pdf

USEPA, 2007: Guidance on the Use of Models and Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, PM 2.5 , and Regional Haze. Tech Rep., EPA-454/B07-002, Research Triangle Park, NC, 262 pp. http://www.epa.gov/scram001/guidance/ guide/final-03-pm-rh-guidance.pdf

USEPA, 2008a: Clarification of Regulatory Status of CALPUFF fo r Near-field Applications. Staff Memorandum, Research Triangle Park, NC, 16 pp. http://www.epa.gov/ttn/scram/clarification% 20of%20regulatory%20status%20of%20calpuff.pd

f

USEPA, 2008b: Technical Issues Releated to Use of the CALPUFF Modeling System for Nearfield Applications. Staff Memorandum, Research Triangle Park, NC, 16 pp. http://www.epa.gov/scram001/7thconf/calpuff/ calpuff_near-field_technical_issues_092608.pdf

USEPA, 2008c: Assessment of the “VISTAS” Version of the CALPUFF Modeling System. Tech Rep., EPA-454/R-08-007, Research Triangle Park, NC, 32 pp. http://www.epa.gov/ttn/scram/reports/cal puff_vistas_assessment_report_final.pdf

Van Dop, H., R. Addis, G. Fraser, F. Girardi, G. Graziani, Y. Inoue, N. Kelly, W. Klug, A. Kulmala, K. Nodop, and J. Pretel, 1998: ETEX : A European Tracer Experiment: Observations, Dispersion Modelling and Emergency Response. Atmos. Environ.

, 32 , 4089 – 4094.

Wang, W., W.J. Shaw, T.E. Seiple, J.P. Ri shel, and Y. Xie, 2008: An Evaluation of a Diagnostic Wind Model (CALMET). J. App. Meteor. and Clim.

, 47 , 1739 – 1755.

Weygandt, S.S., and N.L. Seaman, 1994: Quantifi cation of predictive skill for mesoscale and synoptic-scale meteorological features as a function of horizontal gid resolution. Monthly

Weather Review

, 122 , 57-71.

Willmont, C.J. 1981. On the Validation of Models. Phys. Geogr.

, 2 , 168-194.

WindLogics, 2004a: RUC Analysis-based CALMET Meteorological Data for the State of North Dakota. Tech. Rep., Prepared for North Dakota Department of Health, Saint Paul, MN, 14 pp.DRAFT FOR INTERNAL REVIEW May 27, 2009

49 WindLogics, 2004b: A Comparis on of NOAA RUC Analysis Surface Winds and ADASEnhanced RUC Analysis Winds with Surface Ob servations. Tech. Rep., Prepared for North Dakota Department of Health, Saint Paul, MN, 21 pp.

Wong, H., 2008: Mesoscale Model Refo rmatter Program. Presentation at 9 th Conference on Air Quality Modeling, Research Triangle Park, NC, October 9 -10, 2008. http://www.epa.gov/scram001/9thmodconf/m esoscalemodeldatareformatterprogram.pdf

WRAP, 2006: CALMET/CALPUFF Protocol for BART Exemption Screening for Class I

Areas in the Western United States. WRAP Air Quality Modeling Forum, Regional Modeling Center, 43 pp.

Xu, K., and D.A. Randall, 1996a: A Semiempiri cal Cloudiness Parameterization for Use in

Climate Models. J. Atmos. Sci.

, 53 , 3084 – 3102.

Xu, K., and D.A. Randall, 1996b: Evalua tion of Statistically Based Cloudiness

Parameterizations Used in Climate Models. J. Atmos. Sci.

, 53 , 3103 – 3119.DRAFT FOR INTERNAL REVIEW May 27, 2009

50 APPENDIX A.

CALMET RECOMMENDATIONS



## TO BE PROVIDEDDRAFT FOR INTERNAL REVIEW May 27, 2009

51 APPENDIX B.

SUMMARY COMPARISON OF CALPU FF MODELING SYSTEM RESULTS

FOR VERSION 4.0 AND VERSION 5.8



## TO BE PROVIDED