9-1

9.0 QUALITY ASSURANCE 

 

9.1 Objective of the QA Program 

 A quality assurance (QA) program is needed to ensure that products produced in the BART 

exemption or BART control applications satisfy the regulatory objectives of the program. The scope 

of the QA program for a particular application should be defined in the protocol. Common elements 

include the configuration, data base development, setup and execution of the CALPUFF air quality 

model and processing of modeling results to determine compliance with visibility thresholds. In most 

if not all screening cases (see Chapter 6), states or source operators will use the CALMET datasets 

provided by CENRAP. Applications involving source-specific or alternative modeling that use 

different datasets, modeling options or tools will need to ensure that an appropriate QA program is 

defined in their protocols. More extensive quality assurance will be required in these latter 

applications. It is the responsibility of the state ensure that an adequate QA program, defined in the 

protocol, is implemented faithfully. 

 The CALPUFF modeling system contains features to facilitate quality assurance. These 

include the automatic production of “QA” files for various datasets, including geophysical fields, 

sources and receptors, and imbedded tracking of model options and switches within the output files 

from the major modules. The CALPUFF Graphical User Interface system (GUI) allows these QA files 

to be displayed graphically. In addition, a software management system is available to track version 

and level numbers associated each program and utility. This information reproduced in all of the 

output files, creating an audit trail of software versions and major model options. 

 The BART modeling process involves multiple organizations. As mentioned earlier, the states 

have overall responsibility for the process and may also execute some or all of the modeling. CENRAP 

is contributing general guidance via these guidelines and has funded the development of 

meteorological fields for use in screening applications. BART-eligible source operators will need to 

provide process information and emissions data for use in the analyses. Furthermore, ‘subject to 

BART’ sources will need to be actively involved in control technology decisions and assessments. 

Finally, some or all of the modeling steps may be carried out by contractors on behalf of a state or a 

source operator. 

 Each of these organizations has a responsibility to ensure that it is providing correct 

information to others and to evaluate the quality of any analyses it is performing, whether with data of 

its own or from others. This chapter provides general guidance and information on those aspects of 

quality assurance that are specific to the CALPUFF modeling effort, regardless of which organization 

is carrying out the effort. The focus is on the CALPUFF screening analyses described in Chapter 6. 

More comprehensive QA is needed for the unique aspects of the CALPUFF source-specific or the use 

of alternative models described in Chapters 7 and 8, respectively. 

9.2 Quality Assurance Procedures 

 Chapter 6 recommends procedures for conducting CALPUFF screening modeling to determine 

the whether a particular source or group of sources is subject to BART controls. For initial 

applications, the regional 6 km resolution CALMET data files developed for CENRAP are available to 

modelers for the screening assessments. The development of these CALMET datasets was subject to a 

QA program. Thus, the amount of effort for modelers performing QA of these pre-defined 
 

9-2

meteorological fields is less than that required when developing source-specific meteorological fields 

at finer scale and/or over tailored domains. 

 9.2.1 Input Data 

 The input data required by the model depends on the application. At a minimum, source data is 

required by CALPUFF (see Section 6.3.4) along with specification of the model configuration (choices 

of options and switches). Most of the modeling option choices are specified or recommended by 

regulatory guidance and default values (see Appendix A through Appendix D). If an application uses 

CENRAP’s CALMET files and the CALPUFF screening configuration recommended in Chapter 6, the 

quality assurance of input data will be straightforward. More detailed steps are needed for the setup of 

modeling files for CALPUFF source-specific applications using domains and meteorological fields 

that are not already developed. 

 The basic procedures that will apply to all CALPUFF model applications include a 

confirmation of the source data, including units, verification of the correct source and receptor 

locations, including datum and projection, confirmation of the switch selections relative to modeling 

guidance, checks of the program switches and file names for the various processing steps, and 

confirmation of the use of the proper version and level of each model program. An independent 

modeler should review the CALPUFF input files to confirm the switches and data entry in the model 

input files are correct. The independent reviewer should also exercise the CALPUFF modeling system 

for one or a few corroboration simulations (e.g., run of the worst case event) as a confirmation check. 

 The protocol should stipulate that a set of DVDs be created that contain all of the data and 

program files needed to reproduce the results presented in the final report. The model list files from 

each step should be included on the project DVD. This information allows independent checking and 

confirmation of the modeling process and facilitates archival of the study results. 

 9.2.2 MM5 and CALMET Applications 

 The CALMET meteorological data sets are available on external USB2 or IEEE 1394 

(Firewire) hard drives in a format ready for use with CALPUFF. The QA steps used in the 

development of the CENRAP common datasets are described in the documentation attending the 

distribution to CENRAP. While these QA steps need not be repeated, some testing should be 

performed to show that the data are indeed suitable for the application for which it is being used. This 

is discussed in more detail below. 

 Appendix A contains a list of recommended CALMET switch settings for screening and 

source-specific modeling. Except as modified in a source-specific protocol, these configurations 

should be used in setting up the CALMET simulations. The CALMET model obtains the switch 

settings from an ASCII control file with a default name of CALMET.INP. Whether the model is run 

using a GUI or from the control line in DOS or Linux, it is essential that the control file be reviewed as 

part of the CALMET QA analysis. The CALMET GUI retains all of the input descriptive information 

that is part of the standard CALPUFF.INP file structure. This includes the default value for each 

variable, a text description of the variable, the meaning of each variable option, the units of the 

variable and inter-relationships among variables indicating if/when the variable is used. If CALMET is 

set up and run from the command line or if other user interfaces are employed, this QA step may 

become somewhat more difficult. It is essential nonetheless. 
 

9-3

 Part of the CALPUFF modeling system’s built-in QA capability is a variable tracking system 

that retains the control file inputs for CALMET and CALPUFF in the output files create by the models. 

This information includes the Version and Level numbers of the processor codes and main model 

codes used in the simulations as well as the control files from the main models (CALMET and 

CALPUFF). The information from the preprocessing steps and the CALMET and CALPUFF model 

simulations is all carried forward and saved in the CALPUFF/postprocessor output files so that the 

final concentration/flux files contain a history of the model options and switch settings. This allows a 

user or reviewing agency to confirm the switch settings provided in a control file with that actually 

used in the model simulations. An optional switch in the CALPOST processor creates a complete 

listing of the QA data. This step requires access to the output CALPUFF concentration and/or flux 

files, which are normally practical to store on CDs or DVDs and to provide a part of the project 

CD/DVD set. 

 9.2.3 CALPUFF Applications 

 The minimum source data required by CALPUFF includes: 

 - Number of sources 

 - Locations (e.g., UTM coordinates, UTM zone and datum) 

 - Stack heights above the ground 

 - Stack diameters 

 - Exit velocities 

 - Exit temperatures 

 - Emission rates (SO2, H2SO4, NOx and PM10). 

There are additional requirements for building dimension information (building width, length, height 

and corner locations) for short stacks that are less than Good Engineering Practice (GEP) height. This 

information is used in providing effective structure dimensions for building downwash calculations. 

The requirement to conduct building downwash modeling may be waived by individual States where 

the pertinent transport distance to the nearest Class I area is large. 

 The source coordinates must be expressed in the coordinate system used to define the 

CALMET and CALPUFF modeling domains. For CENRAP screening applications, a Lambert 

Conformal Conic (LCC) coordinate system is used. The required parameters include two matching 

parallels, latitude/longitude of the projection origin, coordinate datum, and false Easting and Northing 

(if used) of the projection origin. For source-specific CALMET/CALPUFF domains, either an LCC or 

UTM projection may be used. The CALPUFF user interface includes software (e.g., COORDS) to 

compute to/from latitude/longitude, LCC and UTM coordinates for a large number of datum. In 

addition, the CALVIEW graphics feature allows the use of georeferenced satellite or aerial 

photographs to be used as base maps to confirm source locations. Links to sources of suitable base 

maps can be found on the CALPUFF data site (www.src.com) in the section on “Aerial Photos”. The 

CALPUFF QA files provide the source coordinates used by the model in a CALVIEW-ready format 

for plotting. Many errors are found in source coordinates and related projection/datum parameters, so 

confirmation of the source location is an important part of the model QA. 

 The PM10 emissions should be broken into filterable and condensable components. The 

filterable PM10 emissions should be speciated into elemental carbon (EC), fine particulate matter 

(PMF) less than 2.5 μm diameter and coarse particulate matter (PMC) between 2.5 to 10 μm diameter. 

The condensable emissions should be divided into a primary H2SO4 component and organic and non-
 

9-4

H2SO4 inorganic components. (The sources of this information, such as source-specific data and AP42 tables, are not addressed in this protocol.) The speciation of the PM10 emissions is important 

because the light extinction efficiency varies by more than a factor of 16 for different species, from 0.6 

for coarse particulate matter to 10.0 for elemental carbon. Plus, hygroscopic aerosols are subject to a 

humidity growth factor (f(RH)), which can increase the light scattering from dry conditions by more 

than an order of magnitude in highly humid conditions. Thus, careful QA of the source and emissions 

data is an important component of the modeling. 

 The locations of the Class I area receptors may be obtained from the National Park Service 

(NPS) receptor dataset. Although the latitude and longitude of each receptor point is provided, it is 

necessary to ensure that the proper UTM or LCC coordinates have been computed for computational 

domain selected. In particular, the datum of the NPS conversion software is not specified, so it is 

recommended that coordinates be checked using the CALPUFF GUI’s COORDS software or another 

comparable coordinate translation software package that recognizes various datums. 

 Most of the CALPUFF input variables contain default values. Appendix B contains a list of 

recommended CALPUFF switch settings for CENRAP screening applications. CALPUFF is 

configured using the ASCII “control file”, CALPUFF.INP file. As with CALMET, whether 

CALPUFF is exercised GUI or from the command line in a DOS, Linux, or Unix window, it is 

essential that the control file be reviewed manually as part of the QA analysis. The CALPUFF GUI 

retains all of the input descriptive information that is part of the standard CALPUFF.INP file structure. 

This includes the default value for each variable, a text description of the variable, the meaning of each 

variable option, the units of the variable and inter-relationships among variables indicating if/when the 

variable is used. 

 9.2.4 Application of CALPOST and POSTUTIL 

 CALPOST is run separately for each Class I area in order to obtain the necessary visibility 

statistics for evaluating compliance with the BART screening and refined modeling thresholds. The 

inputs to CALPOST involve selection of the visibility method (Method 6), entry of Class I areaspecific data for computing background extinction and monthly relative humidity factors for 

hygroscopic aerosols. CALPOST contains a receptor screening that allow subsets of a receptor 

network modeling in CALPUFF to be selected for processing in a given CALPOST run. This is how 

receptors within a single Class I area are selected for processing from a CALPUFF output file that may 

contain receptors from several Class I areas. CALPOST contains options for creating plot files that 

will help in the confirmation that the proper receptor subset is extracted. 

 The CALPOST output file contains a listing of the highest visibility impact each day of the 

model simulation over all receptors included in CALPOST analysis. Receptors will normally be 

selected in each CALPOST run so that each CALPOST run represents the impacts at a single Class I 

area. For a screening assessment, the peak value of the change in extinction is presented near the end 

of the CALPOST output file. For a source-specific studies where the 98th percentile value (8th highest 

day) is used for comparison against the BART threshold of 0.5 deciviews, the standard CALPOST 

output must be additionally processed. One may either obtain special-purpose processing software 

(see, for example CDPHE, 2005) or perform the processing manually. That is, one may import the 

results of the CALPOST table into a sorting program such as a spreadsheet to rank the daily change in 

extinction values in order to identify the 98th percentile or 8th highest day visibility increment. 

 
 

9-5

 CALPOST inputs that require particular QA review include: 

 

 > Visibility technique (Method 6); 

 > Monthly Class I-specific relative humidity factors for Method 6; 

 > Background light extinction values; 

> Inclusion of all appropriate species from modeled sources (e.g., sulfate, nitrate, 

organics, (as SOA), coarse and fine particulate matter and elemental carbon; 

 

 > Appropriate species names for coarse PM used; 

 > Extinction efficiencies for each species; 

 > Appropriate Rayleigh scattering term (10 Mm-1 for screening modeling but Class I area 

specific value for source-specific modeling); and 

 

 > Screen to select appropriate Class I receptors for each CALPOST simulation. 

The CALPOST program produces plot files compatible with CALVIEW that allow confirmation of 

receptor locations that is useful in evaluating the receptor screening step. 

 POSTUTIL allows the user to sum the contributions of sources from different CALPUFF 

simulations into a total concentration file. In addition, it contains options to scale the concentrations 

from different modeled species (e.g., different particle sizes) into species- dependent size distributions 

for the particulate matter. For example, PM is often simulated with unit emission rates for each 

particle size category and, in the POSTUTIL stage, the contributions of each size category based on the 

species being considered (e.g., elemental carbon, coarse particulate matter, etc) is combined to form 

the species concentrations for input into CALPOST. This process, although simple, requires a careful 

review of the weighting factors for each source. 

 POSTUTIL also allows a repartitioning of nitric acid and nitrate to account for the effects of 

ammonia limiting conditions. The four-step procedure for applying the ALM method is summarized 

in Chapter 7 and described in detail in Appendix A of the VISTAS protocol1

. If source-specific 

modeling is performed using different sources of ammonia data or different techniques, the protocol 

should provide justification for all assumptions and a QA plan specific for the application and the data 

bases used. 

 9.2.5 QA Issues for Alternative Source-Specific or Alternative Modeling 

 The level of QA required for application of source-specific or alternative modeling protocols 

will be substantially higher than for the use of datasets that has already been subject to a QA 

procedure. For example, source-specific CALPUFF protocols may include the use of on-site 

meteorological datasets, the use of higher resolution prognostic meteorological (e.g., MM5) datasets, 

alternative visibility calculations, different extinction coefficients, or other changes to the screening 

 

1

 “Comments on the Computation of Nitrate Using the Ammonia Limiting Method in CALPUFF”, (Scire et al., 2005) 
 

9-6

approach. Source-specific must therefore include the development of a QA plan to properly evaluate 

the data used in the modeling. 

 The critical CALMET input parameters depend on the mode in which the model is run 

(observations mode, hybrid mode or no-observations mode), and the location and spatial 

representativeness of any observational data. In a source specific protocol involving the development 

of a meteorological dataset, the elements of the QA process include preparation of wind rose (using 

observed, MM5 and CALMET-derived data), including examination of the data as a function of season 

and time of day (e.g., 4am, 10am, 4pm wind roses), time series analyses, and presentation of 2-D 

vector plots illustrating terrain effects/sea breeze circulation or other features of the flow expected to 

occur within the domain. For example, 2-D vector plots produced during light wind speed stable 

conditions (e.g., early morning such as 4am) are good for assessing the performance of the CALMET 

model configuration and switches in reproducing terrain effects because these conditions are likely to 

maximize the terrain impacts in the model. Seasonal wind roses at 4am, 10am and 4pm would be 

expected to show the development of seas breeze circulations that may be important for certain 

applications. Customization of the QA process for the individual source-specific domain based on the 

availability of data and the physical processes expected to be important at that location should be 

conducted as part of the source-specific QA plan development. 

 If source-specific CALPUFF simulations involving the ALM are conducted, the procedure 

summarized in Chapter 7 should be used. This includes an evaluation of the performance of the model 

in reproducing observed CASTNet or IMPROVE sulfate and nitrate concentrations at measurements 

sites within the source-specific modeling domain. The use of alternative ammonia concentration data 

(e.g., one-atmosphere model output rather than derived ammonia based on aerosol measurements) will 

require a review of the CMAQ or CAMx model performance results developed for CENRAP by the 

regional modeling contractor (Morris et al., 2005d) 

 

9.3 Quality Assessment Summary 

 

 Each BART application requires a modeling report that includes a discussion of the datasets, 

modeling assumptions and a general discussion of limitations and uncertainties of the modeling 

exercise. Where CALPUFF screening analyses are performed followed by source-specific modeling, it 

will be possible to assess the relative conservatism of the screening approach. This may be helpful in 

assessing the impact of uncertainties in screening sources from BART controls. The reliability of 

source-specific CALPUFF modeling is expected to be somewhat better in predicting changes in 

visibility impacts due to BART controls than in predicting absolute visibility values. This is because 

errors and uncertainties in defining meteorological conditions, real-world atmospheric chemistry, and 

plume transport and dispersion rates may be diminished somewhat since the errors are included in both 

the base and sensitivity simulation cases.