CALPUFF: Status and UpdateDennis Atkinson*Meteorologist (NOAA)*in partnership with US EPA, OAQPSTyler Fox and Roger BrodeUSEPA/OAQPSRegional/State/Local Modelers WorkshopMay 16, 2007Outline• CALPUFF Modeling System Background– Scorecard of CALPUFF/CALMET versions• First update of EPA regulatory version ofCALPUFF/CALMET• Initial efforts to update to VISTAS version– Engaging TRC through VISTAS• Results of update process• Findings and next stepsScorecard of CALPUFFVersions- 5.7 April, 2003 (promulgated version)- 5.711a July, 2004 (current regulatory)- 5.711b December, 2005- 5.754 February, 2006 (original VISTAS)- 5.756 August, 2006 (latest VISTAS)- 6.112 April, 2006 (MMS)April, 2003July, 2004Dec., 2005Feb., 2006April, 2006August, 20065.7 5.711a 5.711b5.7545.7566.112CALPUFF/CALMET VersionLinkages- 5.7.......... 5.53- 5.711a.... 5.53a- 5.711b.... 5.53b- 5.754...... 5.724- 5.756...... 5.726- 6.112...... 6.211Regulatory Approval ofCALPUFF Modeling System• Question about regulatory status of CALMET?(by community, model developer)• EPA approves each component of theCALPUFF modeling system• Regulatory approval is for both, e.g. currentregulatory version is 5.711a/5.53aCALPUFF Update Process• Incumbent upon EPA to perform an independentassessment of CALPUFF when updating to newversions• CALPUFF requires extensive assessment andunderstanding of changes made; approvalsmade by EPA, not developer• CALPUFF Update Tool – Introduced at 8thModeling Conference in Sept 2005CALPUFF Update Tool• Developed by MACTEC• Compares 2 CALPUFF versions;proposed (beta) vs. current version (base)• Determines code changes and differencesin results across 10 scenarios• Provides a standardized methodology forevaluating changes in model codesAnalysisToolSourcesMeteorologyTerrainDifferences?Resolve DocumentBETABASECALPUFF Update ToolDescription of Scenarios1. Large-scale domain in Pacific NW-NWS data only2. Same as 1 with MM5 NOOBS option3. Medium-scale domain in Pacific NW withMM5/NWS4. Medium-scale domain near Shenandoah NP5. Small-scale complex flow with deep valley6. Idealized hill with steady-state met – similarity7. Idealized hill with steady-state met – PG8. Flat terrain with steady-state met – similarity9. Flat terrain with steady-state met – PG10. Idealized hill with simulated wind shearDescription of Sources• Four core sources included in allscenarios:(1) Ground-level area source (20m x200m)(2) 10m volume source(3) 30m non-buoyant point source(4) 65m buoyant point sourceCALPUFF Regulatory Update:Version 5.7 to 5.711a• Model Change Bulletin-A provideddescription of changes– Only minor changes in coding• Application of Update Tool showedminimal differences/impacts onconcentrations– Clearly attributed differences to bug fixes• Approved 5.711a - June 15, 2006CALPUFF Regulatory Update:Version 5.7 to 5.711aCALPUFF Regulatory Update:Version 5.7 to 5.711aFEDs Discussions on CALPUFF:Spring 2006• Reason: EarthTech sells CALPUFF rights to TRC (April2006) and EPA experiences difficulties in updating toVISTAS version 5.754• Representatives from Federal Agencies discussedcurrent situation with CALPUFF (status and regulatoryupdates)– EPA– FLM– MMS• Discussed the following issues:– lack of adequate documentation– response time from developer slow– developer requires funding toaddress any issues from Federalcommunity, despite regulatory classification of the modelEPA contacts TRC:June 30, 2006• CALPUFF must meet Appendix Wrequirements for regulatory models,Section 3.1.1(b)-(d):– *.exe to run on a common platform– current model documentation– input/output test data w/model– users can operate model from availabledocumentation– documentation must include comparisonw/tracer data– model/code cannot be proprietaryTRC to EPA response:July 11, 2006• TRC provides copy of Use Agreement• States that ... “Permission to modify theFortran computer codes is granted tousers with the provision that modifiedversions of the model codes must beprovided promptly to TRC, along withappropriate written documentation andinstructions for use”TRC to EPA response:July 11, 2006• Use Agreement also states...“It is the user's responsibility to determineif the CALPUFF modeling system isappropriate for any particular application,and the user assumes all responsibilityand liability for its use.”CALPUFF Regulatory Update• Began update process with v5.754 in June,2006• Engaged with TRC in discussions to understand5.711a vs. 5.754 differences• August 15, 2006—Received e-mail from TRCindicating a new VISTAS v5.756 would bereleased• OAQPS decided to focus on update for newVISTAS v5.756– newest features, latest corrections, etc. for usercommunity (including MMS updates for over water)CALPUFF Regulatory Update:5.711a -> 5.756• Found large ‘unresolved’ differencesbetween 5.711a/5.756– CALMET versions (5.53a vs. 5.726)—up to83% for scenarios 2, 3, (Pacific NW), 5 (deepvalley)– CALPUFF versions (5.711a vs. 5.756)— up to7% for scenario 5 (deep valley)• PROBLEM = incomplete documentation– Missing Model Change Bulletins (MCBs)– No in-code comparisons– Inadequate model documentationWhile we wait for response . . .• BART usage– “Good faith” effort to use “best” availablemodel version– Attempt to be “consistent” with App W and‘wall off’ any potential bad precedents• OAQPS becomes increasingly concernedabout NSR/PSD permit modeling– Appendix W provides “rules of the game”Clearing the Air on CALPUFF• OAQPS makes clear position statement atmonthly call with RO modelers (January 2007)• Region 4 communicates with VISTAS so PatBrewer contacts OAQPS and facilitatesdiscussion with TRC (Feb 2007)• Request for documentation from TRC viaVISTAS (Feb. 2007)• Received documentation from TRC on March15, 2007CALPUFF Update Process:Initial Assessment of v5.756• Significant differences in high ranked valuesbetween v5.711a and v5.756– Largest differences (up to 83%) attributable toCALMET (v5.53a vs. v5.726) for Scenarios 2 &3 (Pacific NW), and 5 (near field/complex flow)– Smaller differences attributable to CALPUFF(v5.711a vs. v5.756); up to 7% for Scenario 5(near field/complex flow)• Much larger differences found across fullmodeling domain – LRT distancesCALPUFF Update Process:Initial Assessments of v5.756• Results from initial assessment of VISTAS version -range of percent differences of high ranked valuesacross all sources and averaging periods byscenario:– Scenario 1: -22.0 to +23.3– Scenario 2: -0.1 to +83.4– Scenario 3: -17.7 to +60.8– Scenario 4: -13.6 to +28.1– Scenario 5: -46.0 to +21.1– Scenario 6: -10.3 to +6.3– Scenario 7: -1.7 to +1.0– Scenario 8: -10.0 to +5.4– Scenario 9: -1.2 to +1.0– Scenario 10: No differences**Maximum Percent Differences by Averaging Period** **Due to CALMET: Short Term Averages:** **Max % Diffs by Aver P eriod for Scenario 1:** **ARE1 VOL1 P301 P651 ARE2** **VOL2 P302 P652** **----- ----- ----- ----- ----- ----- ----- ----- 1 -0.019 0.107** **0.675 15.212 1.641 1.879** **-15.804 19.750 3 0.011 0.081** **-1.132 -10.811 0.218 -1.139 -22.012 -1.440** **24 -1.757 -0.272 2.609 12.242** **0.676 0.8 00 12.437 11.659** **8 -0.041 0.075 1.985 9.376 -0.062** **0.908 21.914 23.260** **Max % Diffs by Aver P eriod for Scenario 2:** **ARE1 VOL1 P301 P651 ARE2** **VOL2 P302 P652** **----- ----- ----- ----- ----- ----- ----- ----- 1 22.278 54.430** **0.153 42.447 -0.084 -0.091 -0.133 7.295** **3 60.666 51.456 0.137 34.825 -0.047** **0.393 -0.070 2.704 24 59.770** **83.390 3.771 46.302** **0.837 0.803 0.185 10.634** **8 68.987 39.749 1.448 63.646 -0.202** **0.732 0.472 15.409** **Max % Diffs by Aver P eriod for Scenario 3:** **ARE VOL P30 P65 P99** **----- ----- ----- ----- ----- 1 0.008 0.** **023 2.704 60.830** **5.198 3 0.177 -0.141** **0.175 2 5.405 17.063** **24 0.715 0.440 5.879 2** **1.892 -7.893 8 -0.025 2.368** **4.788 -17.7 22 11.605** Maximum Percent Differences for High Ranked Values byScenario and Source Due to CALMET v5.53a vs. v5.726**Maximum Percent Differences by Averaging Period** **Due to CALMET: Short Term Averages:** **Max % Diffs by Aver P eriod for Scenario 4:** **ARE VOL P30 P65 BAR** **----- ----- ----- ----- ----- 1G 4.244 6.878** **-3.408 7.026 -6.991** **3G 16.207 9.008 17.049 4.782 -11.002** **24G 15.403 16.381 1 6.672 13.3** **79 3.346 8G 18.695** **16.323 18.239 5.697 -13.627** **1D -6.718 -8.323 6.937 6.1** **76 3.585 3D 17.231 20.079 2** **2.897 8.9 38 -3.995** **24D 22.332 23.710 16.899 2** **1.473 -9.353 8D 21.152 12.227** **19.777 28.060** **-3.349 Max % Diffs by Aver P** **eriod for Scenario 5: ARE1 VOL1 P301** **P651 ARE2 VOL2** **P302 P652 ----- ----- ----- ----- ----- ----- ----- -----** **1 -0.020 -0.188 2.522 -33.268 -19.460** **-0.015 16 .664 -40.994** **3 -0.003 -0.069 3.798 -21.227 -5.749** **-0.058 11 .722 -33.893** **24 -0.558 2.006 19.485 -27.681 11.601** **1.498 11.318 -28.363** **8 0.005 -0.300 17.110 -31.322** **0.133 0.304 21.106 -46.021** Maximum Percent Differences for High Ranked Values byScenario and Source Due to CALMET v5.53a vs. v5.726Scenario 2 - 65m Point Source, Salem LocationPercent Difference - 1st Highest, 1-Hour Averages-400 -300 -200 -100 0 100 200 300 400-400-300-200-1000100200300400710 712 714 716 718 720 722 724524252445246524852505252Scenario 5, Area SourceWest Side of River Near Rock Island DamPercent Differences - 1stHighest, 1-Hour AverageCALPUFF Update Process:Additional Assessments of v5.756• Under auspices of VISTAS, TRC providedadditional documentation regarding model changes– New MCB-C and MCB-D (not released yet)– Tables of in-code documentation annotated to indicatecategory of change and whether change may affectresults (previous MCB’s only included bug fixes)– Summary of TRC model tests, claiming that “the VISTAScode is equivalent to the EPA-approved code once thecoding errors in the EPA code are corrected.”CALPUFF Update Process:Additional Assessments of v5.756• Annotated in-code documentation identifiedseveral categories of changes, including:– Bug fixes– Non-optional technical enhancements– Optional technical enhancements– Non-technical enhancements– Enhancement adjustments– Coordinate conversion fixesCALPUFF Update Process:Additional Assessments of v5.756• TRC provided the following interim versions,to facilitate isolating impacts due to (1) bugfixes and (2) non-optional technicalenhancements:– CALMET v5.53c = CALMET v5.53a + bug fixes– CALMET v5.53c2 = CALMET v5.53c + non-optionaltechnical enhancements– CALPUFF v5.711c = CALPUFF v5.711a + bug fixesCALPUFF Update Process:Additional Assessments of v5.756• Additional potential source of differences identified(indirectly) by TRC, namely “new defaultparameters for optional technical enhancements”• User must manually override these new defaultparameters to maintain consistency with EPAapproved version• CALMET and CALPUFF output list filesdo notdocument these new parameters unless the userhas specified them in the input fileCALPUFF Update Process:Additional Assessments of v5.756• New default parameters for optional technicalenhancements in CALMET include:–IMIXH= -1 for convective mixing height option based on MaulCarson for land cells only and original OCD mixing height overwater(default value of 1 will select Maul-Carson for land and water cells ifIMIXH is omitted);–THRESHL= 0.0 for threshold buoyancy energy flux per meter ofboundary layer required for Mixing Growth over land (default valueof 0.05 W/m2/m will be selected if THRESHL is omitted);–THRESHW= 0.0 for threshold buoyancy energy flux per meter ofboundary layer required for Mixing Growth over water (default valueof 0.05 W/m2/m will be selected if THRESHW is omitted); and–ICOARE= 0 for option to use delta-T method for the CoupledOcean Atmosphere Response Experiment (COARE) bulk flux modelfor computing Zo, u*, L and QHoverwaterCALPUFF Update Process:Additional Assessments of v5.756• TRC later identified the following new defaultparameter for an optional technical enhancementin CALPUFF:–SVMIN(7:12)= 0.50 for minimum sigma-vvalue (m/s) for over watercells for stability classes 1 through 6 (default value of 0.37 m/s willbe used for each stability class forover water grid cells if theseparameters are omitted)CALPUFF Update Process:Additional Assessments of v5.756• EPA conducted several tests in order to compareand isolate affects of the following categories ofchanges:– Bug fixes– Non-optional technical enhancements– New default parameters for optional technicalenhancements• Test No. 1 designed to test TRC’s assertion that“the VISTAS code is equivalent to the EPAapproved code once the coding errors in the EPAcode are corrected”– CALPUFF v5.711c/CALMET v5.53c vs. current VISTASversion (CALPUFF v5.756/CALMET v5.726)CALPUFF Update Process:Additional Assessments of v5.756• Results from Test No. 1 - range of percentdifferences of high ranked values across allsources and averaging periods by scenario:– Scenario 1: -22.0 to +24.5– Scenario 2: -21.7 to +81.8– Scenario 3: -17.9 to +60.3– Scenario 4: -3.3 to +30.5– Scenario 5: -41.8 to +19.5– Scenario 6: -9.7 to +6.3– Scenario 7: -1.7 to +1.0– Scenario 8: -9.3 to +5.6– Scenario 9: -1.2 to +1.0– Scenario 10: No differencesCALPUFF Update Process:Additional Assessments of v5.756• Conclusion? Range of differences similar to initialassessment so at odds with the assertion that it’sall about the bug fixes.• Moving on, Test No. 4 was designed to eliminate alldocumented causes of differences relative to theVISTAS version– 1) bug fixes; 2) non-optional technical enhancements; 3)new default parameters for optional technicalenhancements– CALPUFF v5.711c/CALMET v5.53c2 vs. current VISTASversion (CALPUFF v5.756/CALMET v5.726), withmodified inputs to override new default parametersCALPUFF Update Process:Additional Assessments of v5.756• Results from Test No. 4 - range of percentdifferences of high ranked values across allsources and averaging periods by scenario:– Scenario 1: -17.7 to +0.6– Scenario 2: -20.9 to +10.3– Scenario 3: -0.4 to +5.1– Scenario 4: No differences– Scenario 5: No differences– Scenario 6: No differences– Scenario 7: No differences– Scenario 8: No differences– Scenario 9: No differences– Scenario 10: No differencesCALPUFF Update Process:Additional Assessments of v5.756• Conclusion? Some progress made (no differencesfor Scenarios 4 – 10), but significant unresolveddifferences remain for Scenarios 1 – 3• While range of differences is smaller than previoustests, much larger differences are found acrossdomain for LRT distances• The following example shows one source fromScenario 2Figure 1: Contour Plot of Percent Differences for Scenario 2Jordan Valley 65m Point Source; 4th-Highest 24-Hour AveragesTest 4 - CALPUFF (5.711c) and CALMET (5.53c2) vs. VISTAS (5.756/5.726)(with modified inputs to override new default parameter)0 100 200 300-200-1000100200CALPUFF Update Process:Additional Assessments of v5.756• Given results from Tests 1 through 4, additionaltests were performed to further isolate differencesdue to each of the three factors contributing todifferences• Results from Tests 5 through 7 are shown in thefollowing table as the range of high ranked valuesby scenarioCALPUFF Update Process:Additional Assessments of v5.756Table 1. Range of Maximum Absolute Percent Differences by ScenarioTest 5 – Bug FixesTest 6 – Non-OptionalTechnical EnhancementsTest 7 – New DefaultParametersScenario 1 -1.0 to +22.6 -18.6 to +1.9 -22.0 to +24.4Scenario 2 -15.4 to +27.8 -30.5 to +0.7 -21.8 to +118.3Scenario 3 -5.1 to +6.0 -0.8 to +1.7 -19.3 to +60.8Scenario 4 -18.4 to +8.4 -0.7 to +4.2 -3.2 to +30.5Scenario 5 -32.4 to +9.5 -9.2 to +0.6 -41.8 to +20.9Scenario 6 -1.4 to +0.6 -2.0 to +6.3 -12.6 to +5.5Scenario 7 No differences > |0.01| -1.7 to +1.0 0.0 to +0.4Scenario 8 -1.4 to +0.6 -2.0 to +3.1 -12.1 to +5.5Scenario 9 No differences > |0.01| -1.2 to +1.0 No differences > |0.01|Scenario 10 No differences No differences No differencesCALPUFF Update Process:Additional Assessments of v5.756• Conclusions?– All three factors cause significant differences– New default parameters appear to cause the largestdifferences overall– Differences vary significantly across different scenariosand source types, with no significant overall bias evident• Reminder:These summaries of differences forhigh ranked values are only the tip of the iceberg,and may significantly understate magnitude ofdifferences for LRT portions of domainCALPUFF Update Process:Additional Assessments of v5.756• Given the unresolved differences found in Test 4,EPA sent a copy of Test 4 to TRC on April 18, 2007• TRC identified four (4) additional bug fixes and two(2) additional non-optional technical enhancementsin the VISTAS version that were not included in theprevious interim versions “c” and “c2”• EPA received a DVD from TRC on May 11, 2007containing test results showing no differencesacross all scenarios once these additional changeswere incorporatedCALPUFF Update Process:Additional Assessments of v5.756• TRC also sent a new version of CALMET, v5.53c2– Level 070501, containing the six (6) additionalchanges• EPA has requested that TRC provide a new version5.53c of CALMET containing only the four new bugfixes in order to isolate differences due to eachcategory of change• Once new version 5.53c – Level 07???? ofCALMET is received, at a minimum EPA will redoTests 5 through 7 to document impact of eachcategoryCALPUFF Update Process:Additional Assessments of v5.756• Beyond all of the tests and multiple versions ofCALMET and CALPUFF, EPA has serioustechnical concerns regarding how the optionaltechnical enhancements for mixing height havebeen implemented in CALMET• These concerns are related to the new defaultparameters THRESHL and THRESHW for thethreshold buoyancy energy flux over land andwater, respectively• According to TRC, these new thresholds wereimplemented as part of the MMS-sponsoredenhancements, to prevent unlimited growth of theconvective boundary layer over waterCALPUFF Update Process:Additional Assessments of v5.756• These new thresholds are defined (in the limiteddocumentation available) as the “thresholdbuoyancy flux required to sustain convective mixingheight growth”• However, the code indicates that when the sensibleheat flux falls below the threshold, the convectivemixing height is immediately assigned to 0m forthat grid cell, eliminating convective turbulence• A new convective boundary layer may form onsubsequent hoursCALPUFF Update Process:Additional Assessments of v5.756• This behavior is masked somewhat by otherdefaults within CALMET, including the defaultminimum mixing height of 50m, the use of themaximum of convective and mechanical mixingheights as the overall mixing height (similar toAERMOD), and the default option for upwindaveraging of mixing heights• The default threshold is 0.05 W/m2/m. Forexample, if Zic = 3000m at 2pm, and H = 149 W/m2at 3pm, then Zic = 0m at 3pm. A new CBL willlikely form at 4pm.• The following figures from Scenario 4 illustrate thisbehaviorConvective Mixing Heights forScenario 4 – 10/18/1992 Hour 13Convective Mixing Heights forScenario 4 – 10/18/1992 Hour 14Convective Mixing Heights forScenario 4 – 10/18/1992 Hour 15Convective Mixing Heights forScenario 4 – 10/18/1992 Hour 16Convective Mixing Heights forScenario 4 – 10/18/1992 Hour 17Current Options forRegulatory Approval• Approval of latest VISTAS version “as is”is simply NOT an option at present so . . .• Step 1: Approve “de-bugged” version5.711c– Addressed main concern by all which is useof version with bugs & have understanding ofimplications• Step 2: Continue process to approvelatest VISTAS version with pre-setregulatory defaults by resolving issues andproblems w/ implementation of techenhancementsHow we expect to proceed• Step 1: Approve “debugged” version 5.711c –Level 07????– Replicate and document review and apply update toolto revised versions from TRC• Step 2: Engage with model developer and usercommunity on appropriateness of technicalenhancements– Approval process will include revised MOU restatingresponsibilities and expectations of model developerwrt regulatory version and interaction with usercommunityLessons Learned