UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

 WASHINGTON D.C. 20460 

OFFICE OF THE ADMINISTRATOR 

SCIENCE ADVISORY BOARD 

July 14, 2009 

EPA-CASAC-09-012 

The Honorable Lisa P. Jackson 

Administrator 

U.S. Environmental Protection Agency 

1200 Pennsylvania Avenue, N.W. 

Washington, D.C. 20460 

Subject: Consultation on EPA’s Carbon Monoxide National Ambient Air Quality 

Standards: Scope and Methods Plan for Health Risk and Exposure Assessment

Dear Administrator Jackson: 

The Clean Air Scientific Advisory Committee (CASAC) Carbon Monoxide (CO) Review Panel 

(see Enclosure A) met on May 13, 2009, to conduct a consultation on EPA’s Carbon Monoxide National 

Ambient Air Quality Standards: Scope and Methods Plan for Health Risk and Exposure Assessment 

(April 2009). The CASAC uses consultation as a mechanism for individual technical experts to provide 

comments to guide the Agency on issues early in the development of a document, before the first draft is 

ready for peer review. Written comments provided by the individual members in response to the 

Agency’s charge questions are provided in Enclosure B. In general, the written comments focused on 

exposure characterization. Given that EPA has not yet fully assessed the health issues we did not have 

many comments to offer on health effects for risk assessment. A consultation is conducted under the 

normal requirements of the Federal Advisory Committee Act, which include advance notice of the public 

meeting in the Federal Register. No request for public participation was received. 

As this is a consultation, we do not expect a formal response from the Agency. We thank the 

Agency for the opportunity to provide advice early in the NAAQS review process, and look forward to 

the review of EPA’s First Draft Risk and Exposure Assessment on CO. 

Sincerely, 

/Signed/ 

Dr. Joseph Brain 

Clean Air Scientific Advisory Committee 

Carbon Monoxide Review Panel 

Enclosures 
NOTICE 

This report has been written as part of the activities of the EPA’s Clean Air Scientific Advisory 

Committee (CASAC), a federal advisory committee independently chartered to provide 

extramural scientific information and advice to the Administrator and other officials of the EPA. 

CASAC provides balanced, expert assessment of scientific matters related to issues and 

problems facing the Agency. This report has not been reviewed for approval by the Agency and, 

hence, the contents of this report do not necessarily represent the views and policies of the EPA, 

nor of other agencies within the Executive Branch of the federal government. In addition, any 

mention of trade names of commercial products does not constitute a recommendation for use. 

CASAC reports are posted on the EPA website at http://www.epa.gov/CASAC. 

2 
Enclosure A 

ROSTER 

U.S. Environmental Agency 

Clean Air Scientific Advisory Committee 

Carbon Monoxide Review Panel

CASAC MEMBERS 

Dr. Joseph D. Brain, (Chair) Cecil K. and Philip Drinker Professor of Environmental 

Physiology, Department of Environmental Health, Harvard School of Public Health, Harvard 

University, Boston, MA 

Dr. H. Christopher Frey, Professor, Department of Civil, Construction and Environmental 

Engineering, College of Engineering, North Carolina State University, Raleigh, NC 

Dr. Armistead (Ted) Russell, Professor, Department of Civil and Environmental Engineering, 

Georgia Institute of Technology, Atlanta, GA 

PANEL MEMBERS 

Dr. Thomas Dahms, Professor and Director, Anesthesiology Research, School of Medicine, St. 

Louis University, St. Louis, MO 

Dr. Russell R. Dickerson, Professor and Chair, Department of Meteorology, The University of 

Maryland, College Park, MD 

Dr. Laurence Fechter, Senior Career Research Scientist, Department of Veterans Affairs , 

Research Service (151), Loma Linda VA Medical Center, Loma Linda , CA 

Dr. Milan Hazucha, Professor, Department of Medicine, Center for Environmental Medicine, 

Asthma and Lung Biology, University of North Carolina - Chapel Hill, Chapel Hill, NC 

Dr. Michael T. Kleinman, Professor, Department of Medicine, Division of Occupational and 

Environmental Medicine, University of California, Irvine, Irvine, CA 

Dr. Arthur Penn, Professor LSU School of Veterinary Medicine, Department of Comparative 

Biomedical Sciences, LSU SVM - Room 2425, Louisiana State University, Baton Rouge, LA 

Dr. Beate Ritz, Associate Professor, Epidemiology, School of Public Health, University of 

California at Los Angeles, Los Angeles, CA 

Dr. Paul Roberts, Executive Vice President, Sonoma Technology, Inc., Petaluma, CA 

Dr. Stephen R. Thom, Professor, Institute for Environmental Medicine, 1 John Morgan 

Building, University of Pennsylvania, Philadelphia, PA 

3 
SCIENCE ADVISORY BOARD STAFF 

Dr. Ellen Rubin, Designated Federal Officer, 1200 Pennsylvania Avenue, NW, Washington, 

DC, Phone: 202-343-9975, Fax: 202-233-0643, (rubin.ellen@epa.gov) 

4 
 

Enclosure B

Compendium of Comments 

CASAC Carbon Monoxide Review Panel on 

CO NAAQS Scope and Methods Plan for Health Risk and Exposure 

Assessment (April 2009) 

Dr. Milan Hazucha........................................................................................................................ 6 

Dr. Michael Kleinman .................................................................................................................. 7 

Dr. Stephen R. Thom,................................................................................................................... 8 

Dr. H. Christopher Frey............................................................................................................... 9 

Dr. Russell R. Dickerson ............................................................................................................ 17 

Dr. Paul T. Roberts..................................................................................................................... 18 

Dr. Armistead (Ted) Russell ...................................................................................................... 20 

5 
Dr. Milan Hazucha 

This chapter outlines carefully prepared and reasoned approach to Health Effects and Risk 

Characterization. The supporting evidence will be based on controlled human exposure studies 

(unfortunately almost all of them decades old), epidemiologic, and toxicologic studies. The 

toxicologic studies should be limited to exposure under 500ppm CO. 

The outlined plan for “integrative synthesis” seems to depend heavily on 1991 and 2000 ACQD 

evidence and conclusions (p. 8). As for the integration of new epidemiologic studies published 

since 2000 the Staff paper acknowledges that “it is difficult to determine from this groups of 

studies the extent to which CO is independently associated with cardiovascular disease outcomes 

or if CO is a marker for the effects of another traffic-related pollutant or mix of pollutants” and 

that ”this complicates the effort to disentangle specific CO-related health effects”. Indeed, these 

concerns raise the key questions that have to be specifically answered before any 

recommendations and conclusions can be made (see Fig. 5-5, page 5-43 ISA 2009). The Staff 

nevertheless concludes that a robust CO association in co-pollutant models and the endpoint 

coherency with evidence from human studies “support a direct effect of short-term CO exposure 

on cardiovascular morbidity at ambient concentrations below the current NAAQS level.” 

However, the robust association with co-pollutants and the coherence with experimental 

evidence do not necessarily provide supporting evidence for the above statement particularly 

when many target physiologic and clinical endpoints for CO and co-pollutant may be the same. 

Particularly the interpretation of health effects of mixed pollutants, e.g., CO and PM10 should be 

an integrated assessment based on evaluation and conclusions reached in respective ISA 

documents (CO and PM) so that the outcome of the same study reached in PM ISA is not 

interpreted differently in CO ISA. For the key studies discussed in CO ISA I suggest that the 

reference is made to the PM ISA section where the same study is discussed and the key 

conclusions are included in the CO ISA. 

The Risk Characterization section (2.2) appropriately suggests more cautious approach in studies 

interpretation and notes that currently we do not have enough data to “conduct a quantitative risk 

assessment for this health endpoint”, i.e., the quantitative dose-response relationship. Therefore, 

the risk characterization will be based solely on the controlled human exposure literature with the 

proposed benchmark COHb levels at 2.0, 2.5 and 3.0%. However, the conditions under which 

these levels will be reached are not given (at rest, exercise, static CO levels, etc. ?). As 

subsequently stated the calculation of dose will be based on the “well-established” CFK equation 

from 1965. I am almost sure that it is not the original CFKE (1965) but an enhanced model 

which is being utilized. This should be properly referenced here and in other sections as well 

when the applied model is cited, e.g., p. 15. Some multicompartment models that claim to be 

even more precise particularly under dynamic conditions of exposure should be considered as 

well. 

The Staff has appropriately expressed a number of concerns regarding Risk Characterization for 

cardiovascular effects in epidemiologic studies. As suggested the issues of concern need to be 

discussed a depth at this or subsequent meetings. 

6 
Dr. Michael Kleinman 

General Comments: 

The plan is well written and comprehensive. 

Specific Comments: 

Pg. 7. Although it is true that most healthy individuals can physiologically compensate for COinduced reductions in tissue oxygen (O2) levels (e.g. through increased blood flow, blood vessel 

dilation), some susceptible individuals may not be able to benefit from compensatory responses 

because they have arteriosclerosis or impaired cardiac output (either from vascular or cardiac 

tissue inflammation or damage). 

Pg 9. Would it be appropriate to use meta-analytical methods to establish exposure-response 

relationships for the endpoints used in the controlled human exposure studies? 

Pg 10. It would be appropriate to show the results of the zero CO data for these studies (i.e. 

extend the X-Axis). 

Pg 11. It would be appropriate to draw a distinction between precision and accuracy with respect 

to the CO-oximeter responses. The word “variable” is used in this document with multiple 

meanings. 

Pg 18. Does the dose algorithm not take breathing rate into consideration? The diagram shows 

that parameter as a separate unincluded track. 

Pg 19. In discussion of the probabilistic sampling approach, has the degree to which the 

precision of estimate will depend on the number of individuals being simulated been 

determined? This could be considered to be in the nature of a power calculation. 

Pg 23. The terms in the column labeled “Method” need to be better defined. 

Pg 25 and 26. Rewrite the equation. “d” appears to be used for 2 parameters; a monitor index 

and “district d”. 

Pg 26. Some explanation of the normalized variable Lmd and its SD would be helpful. 

Pg 27. The number of significant figures in Table 3-2 could be reduced. Is the value for Vehicle 

exposure realistic, given the much higher values cited in the ISA for measured in vehicle 

exposures. 

Pg 28. “Uncertainty” is used in this document and the ISA with different connotations. It would 

be better to define uncertainty in terms of precision, accuracy, bias etc. 

Pg 29. Is the commute distance is incorporated into the census tract data? 

Pg 30. What is a ‘tornado’ graph? 

Pg 31. The discussion of sensitivity could be improved by including some concrete examples. 

7 
Dr. Stephen R. Thom, 

We were asked to submit written comments on the Carbon monoxide national ambient air 

quality standards: Scope and Methods Plan for Health Risk and Exposure Assessment. My 

comments are merely a recapitulation of ideas I stated at the CASAC meeting in Chapel Hill. 

The reasoning and rationale for the Health Effects and Approach to Risk Characterization plans 

are quite clear. It is also understandable why the plan uses carboxyhemoglobin (COHb) as the 

biomarkers for human exposure. I think a rationale justification for using COHb is that it is the 

only quantifiable assessment of environmental exposure currently available, rather than stating 

that the health effect of greatest concern from CO exposure is hypoxia caused by elevated COHb 

(opening line of section 2.1). Clearly, the COHb levels shown to hasten onset of angina in at-risk 

individuals cannot be explained by mere hypoxia. That is, I suggest the EPA might consider 

COHb as a surrogate for magnitude of CO exposure and leave the question of CO 

pathophysiology to further study. 

Responses to questions pertaining to chapter 2: 

1 a) Overall planned approach is reasonable and valid. 

1b) The range of potential health effect benchmark COHb levels should include 1.5 % COHb 

based on the multicenter study of Allred, et al. 

2a) It is reasonable, based on current data, to use CO as a surrogate for multi-component air 

pollution exposure. It is not scientifically valid to interpret health risks identified as a 

quantitative assessment solely for CO. That is, pathophysiological effects are likely to be due to 

the combined presence of agents and may not be reproducible with exposures to any single 

agent. 

2b) Results from co-pollutant models provide a qualitative – not quantitative – assessment of CO 

risk. 

 

8 
Dr. H. Christopher Frey 

The comments provided here focus on Chapter 3 and the charge questions related to Chapter 3. 

Chapter 3 – Scope and Approach for Population Exposure/Dose Analysis

1. We plan to build upon the basic structure and design of the exposure assessment 

conducted in the previous review. Since that time there have been major 

improvements in the exposure model and in the data for input to the model. Are the 

Panel members aware of information sources that would help inform further 

improvements that would be worth considering in the current review? 

The 1st draft of the ISA provides a reasonable overview of the state-of-science pertaining to 

exposure assessment, particularly in the context of CO. The use of APEX as a basis for 

estimating exposure is reasonable. The document appropriately identifies variability, 

uncertainty, and limitations associated with the general approach. 

The comments here for Charge Question 1 pertain to material contained in the scope and 

methods document. During the discussion at the May 13, 2009 CASAC CO Panel meeting to 

review this document, EPA indicated that they would not be using the approach indicated in 

Equation (1) of page 25. EPA staff did not provide detail on the approach that will be used; 

therefore, it is not possible at this time to provide comment on the scope and method of the 

proposed approach. Nonetheless, I choose to keep the comments made prior to the meeting just 

to illustrate the kinds of issues that might need to be addressed for other modeling approaches. 

P 25. What is the validity or evaluation of Equation (1)? The reference cited, Johnson et al. 

(2000) is available in a 1999 draft word perfect document at 

http://www.epa.gov/ttn/caaa/t1/meta/m2154.html. I could not readily locate the final version. It 

appears that Section 2.4.1 of this document presents an approach that is similar to Equation (1) 

but not identical. In particular, I am looking for an explanation of how and why the factors M, L, 

and T were chosen and why Cdt is raised to an exponent of 0.621. I am also interested in how the 

constant multipliers for Mm were estimated, how the lognormal distributions for L and T were 

estimated, and how the exponent for Cdt was estimated. Furthermore, I would like to know the 

performance of this model in terms of goodness of fit measures such as R2, slope of a parity plot, 

and standard error of the estimate, and whether any evaluation was done in which this model was 

applied to independent data. Moreover, since L and T are depicted as lognormal distributions 

with median values of 1.0, one suspects that a log transformation of Equation (1) might have 

been fit to data. In back-transforming, this introduces a bias that needs to be characterized – was 

this taken into account when estimating the value of M? In summary, more justification and 

detail is needed on this equation, how it was developed, and its evaluation/validity. 

The explanation in the last full paragraph of Page 26 seems incomplete. In addition to the 

personal exposure monitor (PEM), were there measurements made outdoors directly outside the 

microenvironment? How were the ratios for M estimated? What is their distribution of 

variability? Why are they treated here as point estimates? Some explanation and interpretation 

is warranted. For example, the value of approximately 1.0 for a residence seems reasonable for a 

steady state ratio of indoor to immediately outdoor concentration. The value of 2.97 for an auto 

service station is plausible since there may be local emission sources of CO (cars) that lead to 

much higher CO concentrations compared to a location some distance away. 

The meaning or interpretation of M seems straightforward. The interpretation of L seems a little 

unclear, especially regarding how it is distinct from that of M. This needs further explanation. 

Why is only one distribution used to represent all districts and microenvironments? 

9 
Why is the time of day multiplier the same for all microenvironments and districts? Wouldn’t 

the temporal profile be different for near-roadway versus a location distant from a roadway? 

The statement in the middle of Page 26 that comparisons were made with hourly CO 

concentrations measured simultaneously at the nearest fixed-site monitor raise some questions. 

Fixed site monitors are located for different purposes, as described in the ISA, with some 

intended to represent near-roadway conditions and others to be more representative of area-wide 

conditions. This would imply the need to stratify the analysis with respect to different purposes 

or objectives of fixed monitors. 

How is Equation (1) compatible with the latest available information regarding near-roadway CO 

concentrations that are influenced by local traffic? How does the performance of Equation (1) 

for near roadway applications compare to other approaches, such as estimating incremental local 

near-roadway concentrations using a model such as CALINE4 or AERMOD (using emission 

factors either from Mobile6 or Draft MOVES 2009)? 

2. One of the main issues in this analysis is how to estimate ambient CO concentrations 

on and near roadways, which can be significant contributors to ambient CO 

exposures. The relationship between CO levels measured at ambient fixed site 

monitors is highly variable due to the spatial and temporal variability of on- and 

near-roadway CO concentrations. In the previous review, proximity factors were 

used to adjust the concentrations measured at monitors to estimate roadway-related 

concentrations of CO. We plan to conduct a review of the literature and draw upon 

the results of near-road studies to update the proximity factor distributions. Do the 

Panel members have recommendations for improvements or alternatives to this 

approach? 

There is typically very little correlation between the ambient concentration measured at a fixed 

site monitor and the concentration of the same pollutant if measured immediately outside of a 

vehicle. The latter is a more useful basis for estimating in-vehicle concentration. Thus, the use 

of a methodology that attempts to estimate concentration immediately outside of a vehicle is 

worthy of serious consideration. 

The recent literature indicates that a mass balance approach to modeling in-vehicle concentration 

merits consideration. There is significant variability in in-vehicle concentration attributable to 

factors such as: (a) status of windows (open, partially open, closed); (b) status of the heating, 

ventilation, and air conditioning (HVAC) system (on, off, recirculation mode, outside air intake 

mode); (c) vehicle speed; (d) in-vehicle emission sources (e.g., smoking); (e) filter efficiency; (f) 

deposition; and (g) concentration of the pollutant immediately outside of the vehicle. Factors 

such as (a) through (c) influence the air exchange rate. 

The concentration of a pollutant immediately outside of a vehicle that is operating on a roadway 

can be conceptualized as having at least two main components: (1) an area-wide concentration 

such as obtained from a community-based monitor; and (2) an incremental increase in 

concentration when comparing on-road versus area-wide concentration. The latter can be 

conceptualized as influenced solely by the local contribution of vehicle emissions and their 

dispersion over the roadway. However, there is not as yet an accepted methodology for 

predicting onroad concentrations. Existing modeling tools that are used to estimate nearroadway concentrations, such as CALINE4, model a mixing zone over the road way that extends 

several meters to either side of the road. In practice, CALINE4 is typically not used for 

receptors any closer than several meters from the edge of the roadway. AERMOD could also be 

10 
used to estimate near-roadway concentrations. The estimation of the incremental concentration 

attributable to traffic flow would be sensitive to factors such as: (a) roadway geometry (number 

of lanes, road width); (b) vehicle emissions; (c) atmospheric stability; (d) wind speed and 

direction; and (e) terrain or features near the roadway, such as sound barriers or vegetation. 

Vehicle emissions depend on factors such as: (a) traffic volume; (b) vehicle fleet distribution 

among vehicle sizes and fuels; (c) vehicle speeds and accelerations; (d) potential existence of 

cold start (depending on proximity to trip origins) or fuel enrichment effects; (e) road grade; and 

(f) ambient temperature. 

It is tempting to conclude that given so many sources of variability it is not worth trying to 

implement an approach for estimating in-vehicle concentration using vehicle emissions and near 

roadway dispersion models. However, not all factors are equally important, and some factors are 

likely to emerge as the most important ones that should be the focus of attempts to develop 

accurate characterizations. Consideration should be given to whether the use of an improved 

methodology would lead to more accurate estimates, even if random components of variability in 

the estimates are not perfectly captured. Alternatively, consideration should be give to whether 

there is a more reliable theoretical and empirical basis for a more robust fundamentals-based 

approach than the use of empirical factors that may be inapplicable to situations other than for 

the calibration data from which they were derived. 

As an example, we have recently been exploring a methodology for estimating in-vehicle 

exposure to PM2.5. This methodology is described in a paper that has been accepted for 

presentation at the Annual Meeting of the Air & Waste Management Association (AWMA) in 

June 2009: 

Liu, Z., H.C. Frey, Y. Cao, and B. Deshpande, “Modeling of In-vehicle PM2.5 Exposure 

Using the Stochastic Human Exposure and Dose Simulation Model,” Paper 2009-A-238-

AWMA, Proceedings, 102nd Annual Conference and Exhibition, Air & Waste Management 

Association, Detroit, Michigan, June 16-19, 2009. 

This paper demonstrates the use of a mass balance approach for modeling in-vehicle 

concentrations, based on an approach described by Abadie et al. (2006) and Ott et al. (2007) 

combined with the use of CALINE4 to estimate near roadway concentration increments. The 

methodology is briefly illustrated based on variations in atmospheric stability, windspeed, and 

roadway type. A more extensive treatment of this topic is in progress. 

It may be reasonable to create somewhat stylized but representative default assumptions for the 

typical combination of roadway types that comprise typical commuting trips based on 

transportation data. For example, currently, models such as APEX make stylized assumptions 

regarding commuting between census tracts without attempting to account for site-specific 

characteristics of the commuting. The use of a default-based approach to estimating nearroadway and in-vehicle concentrations from a more fundamental basis could be a significant 

improvement, even though it may not be perfect. Furthermore, sensitivity analysis can be used 

to explore the factors that are estimated to most affect in-vehicle concentrations and exposures, 

and efforts to improve the modeling framework by developing distributions of variability in such 

factors could be prioritized. Of course, there would be a need for evaluation of any new 

modeling approach. 

Abadie, M., Limam, K.; Builly J.; Genin, D.; Atmospheric Environment, 2006, 38, 2017-2027 

Ott, W.; Klepeis, N.; Switzer, P.; Journal of Exposure Analysis and Environmental 

Epidemiology, 2007, 18, 312-325 

11 
3. The planned approach for addressing uncertainty is primarily qualitative with a 

focus on sensitivity analysis and a limited quantitative analysis for those variables 

determined to be most influential with respect to exposure and/or dose estimation 

and where supporting data are available. 

a. What are the Panel members’ views concerning this general approach? 

b. Spatial and temporal gradients in ambient CO relative to CO concentrations 

measured at fixed-site monitors are potentially a major source of uncertainty 

in the exposure and dose estimates. Do the Panel members have suggestions 

for how best to characterize the uncertainties in this relationship? 

EPA is correct to state that the APEX exposure model is designed to explicitly account, 

quantitatively, for many sources of variability. However, as noted in the comments above, there 

may be additional sources of variability that need to be quantified, such as variability in the ratio 

M (if the approach of Equation (1) is to be used). 

Regarding the general approach proposed to dealing with uncertainty, conceptually it is fine. It 

is plausible to deal with some uncertainties qualitatively, some via sensitivity analysis, and some 

via probabilistic analysis. The key to how successfully this is implemented is the degree to 

which each method is used. Wherever possible, a quantitative approach using probabilistic 

methods is preferable if it serves a decision making purpose. Sensitivity analysis is a secondbest alternative to probabilistic analysis. Qualitative analysis is the least preferred among the 

three, because ultimately it is the least informative or most subject to ambiguity in interpretation. 

There may be situations in which the qualitative approach is the best approach. However, it 

should not be the default if there is a need for more information about uncertainty that can be 

conveyed quantitatively using plausible input assumptions. Since the NAAQS must be set taking 

into account a margin of safety, it should not be presumed that quantitative acknowledgement of 

uncertainty would undermine the decision making process. If anything, it could enhance the 

decision making process. 

For consistency, some comments are repeated here that were offered as part of my review of the 

SOx REA, with some minor revisions to be applicable to the CO 1st draft ISA. The weight of 

evidence and the uncertainties associated with the state-of-science have implications for the 

decision making process. Weight of evidence involves a qualitative determination of causality 

and supports conclusions regarding relationships between air quality, exposure, and adverse 

effects. Uncertainty implies that scientists are not entirely sure of the numerical values that 

precisely and accurately quantify these relationships. However, in many cases these quantities 

can be bounded or described using plausible ranges, based either on analysis of empirical data or 

encoding of expert judgment. Based on quantitative analysis and reasonable and informed expert 

judgments, information regarding uncertainty can be used to inform explicit or implicit choices 

of the margin of safety with which to develop a standard that protects public health. 

The ISA refers to WHO (2008) as the basis for the qualitative uncertainty analysis approach that 

is used by EPA. However, EPA should explain why it chose a qualitative approach rather than a 

more quantitative approach, and it should be careful to distinguish or justify situations in which 

the qualitative approach is appropriate versus when quantitative methods can be used instead. 

Generally, from a scientific perspective, it is preferred to quantify uncertainties wherever 

possible. As WHO (2008) explains (p. 31): 

Determination of an appropriate level of sophistication required from a 

particular uncertainty analysis depends on the intended purpose and scope of a 

12 
given assessment. Most often tiered assessments are explicitly incorporated within 

regulatory and environmental risk management decision strategies. The level of 

detail in the quantification of assessment uncertainties, however, should match 

the degree of refinement in the underlying exposure or risk analysis. Where 

appropriate to an assessment objective, exposure assessments should be 

iteratively refined over time to incorporate new data, information and methods to 

reduce uncertainty and improve the characterization of variability. Lowest-tier 

analyses are often performed in screening-level regulatory and preliminary 

research applications. Intermediate tier analyses are often considered during 

regulatory evaluations when screening-level analysis either indicates a level of 

potential concern or is not suited for the case at hand. The highest tier analyses 

are often performed in response to regulatory compliance needs or for informing 

risk management decisions on suitable alternatives or trade-offs. 

Hence, the Tier 1 (Qualitative) approach is not a default. It should be a justified choice that is 

consistent with the purpose and scope of the assessment. 

WHO specifies a structured approach to qualitative assessment of uncertainty that includes 

1) qualitatively evaluate the level of uncertainty of each specified source; 

2) define the major sources of uncertainty; 

3) qualitatively evaluate the appraisal of the knowledge base of each major source; 

4) determine the controversial sources of uncertainty; 

5) qualitatively evaluate the subjectivity of choices of each controversial source; and 

6) reiterate this methodology until the output satisfies stakeholders 

Hence, there are three dimensions to the qualitative approach, as depicted in Figure 6 of WHO 

(2008). In other documents recently reviewed by CASAC, such as EPA’s draft REA for SOx, 

EPA seems to have created a different approach in which the level of uncertainty and the 

appraisal of the knowledge base are combined, and it is less clear as to the role of subjectivity of 

choice in the framework. Given the significance of the ISA and the apparent differences in 

approach from that in the WHO Guidelines, further explanation is needed, or the WHO approach 

should be adopted with less modification. 

The use of low, medium, and high categories of uncertainty can be problematic in that the 

interpretation of these is vague and thus may be made differently by different readers or 

stakeholders. 

In developing the uncertainty analysis, a clear definition of uncertainty and its components (i.e. 

bias, imprecision) should be provided. 

Sensitivity analysis is a quantitative technique, and there are many variations of sensitivity 

analysis. There are recent evaluations of sensitivity analysis with specific focus on their 

applicability to exposure assessment models, which build upon some concepts offered by Saltelli 

and coworkers, but with more specific application to exposure modeling: 

Frey, H.C., and S.R. Patil, “Identification and Review of Sensitivity Analysis Methods,” 

Risk Analysis, 22(3):553-578 (June 2002). 

Patil, S.R., and H.C. Frey, “Comparison of Sensitivity Analysis Methods Based Upon 

Applications to a Food Safety Risk Model,” Risk Analysis, 23(3):573-585 (June 2004). 

13 
Mokhtari, A., and H.C. Frey, “Recommended Practice Regarding Selection of Sensitivity 

Analysis Methods Applied to Microbial Food Safety Process Risk Models,” Human and 

Ecological Risk Assessment, 11(3):591-605 (2005). 

Mokhtari, A., H.C. Frey, and J. Zheng, “Evaluation and recommendation of sensitivity 

analysis methods for application to Stochastic Human Exposure and Dose Simulation 

(SHEDS) models,” Journal of Exposure Science and Environmental Epidemiology, 

16(6):491-506 (Nov 2006). 

Mokhtari, A., and H.C. Frey, “Sensitivity Analysis of a Two-Dimensional Probabilistic 

Risk Assessment Model Using Analysis of Variance,” Risk Analysis, 25(6):1511-1529 

(2005). 

Mokhtari, A., and H.C. Frey, “Evaluation of Sampling-Based Methods for Sensitivity 

Analysis: Case Study for the E. coli Food Safety Process Risk Model,” Human and 

Ecological Risk Assessment, 12(6):1128-1152 (Dec 2006). 

EPA appears to be proposing a variation on nominal range sensitivity analysis (NRSA) or 

differential sensitivity analysis (DSA). However, by proposing to vary each selected input by 

plus or minus 5 percent, EPA is apparently opting for something more like DSA. If only an 

arbitrary range is to be used, it may be better to vary each input by plus or minus 1 percent or to 

use an elasticity metric. However, NRSA is often more informative. In NRSA, each selected 

input is varied over a plausible range of values, with the range differing from input to input. The 

insights obtained from NRSA can be quite different than those obtained from DSA, as noted in 

many of the references listed above. Both of these are local sensitivity analysis methods, that 

assess the response of the model based on perturbations around a specific point. 

Even more informative than NRSA are global sensitivity analysis methods, in which many inputs 

are varied simultaneously over plausible ranges. Global sensitivity analysis can be conducted in 

combination with probabilistic simulation methods such as Monte Carlo simulation, using 

techniques such as correlation coefficients, regression analysis, Analysis of Variance (ANOVA), 

or categorical and regression trees (CART). Global sensitivity analysis methods take into 

account nonlinearity and interaction among inputs. There are some advanced methods, such as 

Sobol’s method and the Fourier Amplitude Sensitivity Test (FAST) that enable characterization 

not only of the first order sensitivities (i.e. the variation in the output directly attributable to 

variation in an input) but also interaction effects in which the co-variation of two or more inputs 

contributes to variation in an output. Furthermore, if the focus of sensitivity analysis is to assess 

inputs and their ranges that lead to high exposure outcomes, then a method such as CART might 

be more useful. 

In the discussion of the use of global sensitivity analysis as described on the bottom of page 30 

and top of page 31, appendix A is cited. However, Appendix A does not appear to define global 

sensitivity analysis nor offer a practical approach to conduct it. The references listed above, as 

well as the works of Andrea Saltelli, should be consulted for additional methodological 

approaches. Jon Helton has also written extensively on the use of sensitivity analysis methods in 

conjunction with Monte Carlo simulation and Latin Hypercube Sampling. 

On Page 30, sensitivity analysis is implied for many parameters of variability distributions. EPA 

may find it helpful to look at the following articles, which explore some of the statistical 

properties for characterizing uncertainty based on sampling distributions for parameters of 

variability distributions, taken into account sample sizes and variability in the available data: 

14 
Frey, H.C., and D.S. Rhodes, “Characterizing, Simulating, and Analyzing Variability and 

Uncertainty: An Illustration of Methods Using an Air Toxics Emissions Example,” Human 

and Ecological Risk Assessment: an International Journal, 2(4):762-797 (December 1996). 

Frey, H.C., and D.S. Rhodes, “Characterization and Simulation of Uncertain Frequency 

Distributions: Effects of Distribution Choice, Variability, Uncertainty, and Parameter 

Dependence,” Human and Ecological Risk Assessment: an International Journal, 4(2):423-

468 (April 1998). 

Frey, H.C., and S. Li, “Quantification of Variability and Uncertainty in AP-42 Emission 

Factors: Case Studies for Natural Gas-Fueled Engines,” Journal of the Air & Waste 

Management Association, 53(12):1436-1447 (December 2003). 

Zheng, J., and H.C. Frey, “Quantification of Variability and Uncertainty Using Mixture 

Distributions: Evaluation of Sample Size, Mixing Weights and Separation between 

Components,” Risk Analysis, 24(3):553-571 (June 2004). 

Zhao, Y., and H.C. Frey, “Quantification of Variability and Uncertainty for Censored Data 

Sets and Application to Air Toxic Emission Factors,” Risk Analysis, 24(3):1019-1034 (2004). 

Zheng, J., and H.C. Frey, “Quantitative Analysis of Variability and Uncertainty with Known 

Measurement Error: Methodology and Case Study,” Risk Analysis, 25(3):663-676 (2005). 

A key point in many of these papers is that the parameter sampling distributions for some 

variability distributions can be treated as statistically independent, but for some others they 

cannot be. However, it is relatively straightforward to use bootstrap simulation to estimate the 

sampling distributions of parameters and the dependency between them. A prototype software 

tool developed for EPA, AuvTool, can be used to assist with this. A version of this, with its 

documentation, is publicly available at http://www.foodrisk.org/exclusives/AuvTool/. AuvTool 

was developed to support the developing of distributions for uncertainty in the parameters of 

distributions of variability for input to exposure models such as SHEDS. 

In recent years, EPA has been exploring and applying methods for encoding expert judgments 

regarding uncertainties. An example is an expert elicitation for concentration-response functions 

related to PM2.5. An EPA guidance document on expert elicitation has recently been reviewed 

by an SAB panel. Based on publicly available information, this guidance document has received 

strong support from the SAB panel (see 

http://yosemite.epa.gov/sab/SABPRODUCT.NSF/81e39f4c09954fcb85256ead006be86e/f4ace05

d0975f8c68525719200598bc7!OpenDocument&TableRow=2.2#2.). The final report from the 

panel will need to be reviewed and approved by the SAB before final transmittal to EPA. 

However, the purpose of mentioning the positive reaction to the guidance document is to point 

out that expert elicitation is a viable technique that can be used along with statistical analysis of 

empirical data to develop input assumptions for quantitative analyses of uncertainty. This is a 

methodology that OAQPS should consider applying, when appropriate, for other applications. 

The bottom of page 31 appropriately mentioned that activity data in CHAD may not be 

representative of Denver or Los Angeles. However, in conducting the REA, EPA should attempt 

to identify, characterize, and where possible quantify, the ways in which lack of 

representativeness, if significant, might lead to biases in results. Furthermore, EPA could 

consider making adjustments to attempt to correct for biases. For example, if there are biases in 

the CHAD commuting activity patterns that might not reflect city-specific behaviors, it could be 

appropriate to make adjustments. Any such adjustments should be documented to facilitate peer 

review. 

Other Comments

15 
Page 19 – have improvements been made to the characterization of near-roadway and in-vehicle 

CO concentrations? Are improvements planned? 

Page 20 – later mention is made that the CHAD data might not be representative of activity 

patterns specifically in Denver and Los Angeles. In what significant ways are CHAD data not 

representative of these cities, and how will this affect the results? 

Page 21. How are commuting times estimated, and are they internally consistent with the 

commuting distances implied by the distance between home and work census tracts? How are 

transportation modes selected? Is this an example of where the CHAD data might not be 

representative of the study locations? 

Page 24, Table 3-1. For the indoor microenvironments, will any distinctions be made in the 

estimated outdoor concentration adjacent to the indoor microenvironment taking into account 

proximity to roadways? Will it be possible to use site-specific data for LA and Denver regarding 

the proximity to actual schools to roadways, and so on? Does the in vehicle- heavy truck 

category imply occupational exposure? 

16 
 

Dr. Russell R. Dickerson 

Comments on the “Plan for Health Risk” 

The section on microenvironments in could be more specific on how the modeled concentrations 

of CO compare to observations. The uncertainty/sensitivity measures (Appendix A) look good, 

but how accurate are personal monitors and what are their detection limits; are they adequate? 

At least refer to section 3.6.3.2 of the ISA. Do wood burning stoves of fireplaces (popular 

around Denver) contribute to personal exposure? 

17 
Dr. Paul T. Roberts 

I focused my review on Chapter 3 and the presentation made to the Panel on May 13, 2009. 

As mentioned during the presentations and following discussions, much of what was 

written in Chapter 3 of this Plan on the exposure modeling will not be followed and a 

different approach will be used (especially subsections 3.3.1 through 3.3.4. We were told 

that they would use spatial interpolation using central-site data with relationships between 

near-road (and in-vehicle) concentrations with central-site data. This approach is not 

discussed in the ISA, nor in the Scope and Methods Plan - thus it is not possible to evaluate 

the approach and determine if the proposed approach is supported by the scientific 

literature. In fact, as I mentioned in my comments on the ISA (copied below), the science 

discussion in the ISA, Chapter 3.6.7 in particular, is not sufficient. 

PTR comments on the ISA, Chapter 3.6.7, which also apply to the Scope and Methods Plan: 

“Chapter 3.6.7: In light of the currently-planned method for preparing the CO concentration 

fields for the exposure model (as discussed at May 13 panel meeting and in slide 13 of the 

presentation), the discussion and references cited on pages 3-70 and 3-71 (in Chapter 3.6.7) are not 

sufficient to support the methods plan and should be significantly expanded. There is only one 

reference cited for concentration surfaces, which will be a major tool in the analysis; many more 

are needed. A few additional references that I can easily find are listed at the bottom of my 

comments. Note that many of these references are for pollutants other than CO, since few studies 

are currently being done on CO; however, the methods can be reviewed and used as guidance for 

similar applications for CO. In addition, I think that the exposure modeling Chapter ( 3.6.7) 

should include much more specifically about the methods that will be used to address in-vehicle 

and near-road exposures. A recent HEI report is now available on the web at: 

http://pubs.healtheffects.org/view.php?id=306; this report has an excellent summary of the 

current literature and thinking on near-roadway exposures and a good reference list. 

Regarding the Chapter 3.7 conclusion that central-site monitor concentrations is generally 

a good indicator for the ambient component of personal CO exposure: Total personal 

exposure to CO is the time weighted sum of exposure to all microenvironments including 

multiple outdoor environments (not just multiple indoor environments). Therefore the centralsite monitor concentration is not viewed as ‘a good general indicator for the ambient component 

of personal CO exposure’. Equation 3.4 should be reformulated to include multiple outdoor 

microenvironments, including at least near roadway exposures (ref section 3.5.1.3 and Figure 3-

34). Equation 3.4 should also distribute the concentration term to both outdoor and indoor 

microenvironments as a concentration within both the sum of the indoor components and the 

sum of the outdoor components (into a new summation term) specifically as the concentration in 

each microenvironment, Ci for both indoor and outdoor. This will also require that the following 

sections (and any others) be modified to reflect that more-complex exposure: Lines 30-31, page 

3-57; lines 7-10, page 3-65 and page 3-74 lines 10-11.” 

18 
In addition, there are many new references on near-road and in-vehicle pollutant concentrations, 

including those in the HEI report referenced above and the recent paper by Barzyk and others at 

EPA (Atmospheric Environment 43 (2009) 787-797 and references therein). 

The exposure modeling to be performed will need to result in significantly higher CO 

concentrations in various near-road and other microenvironments, as illustrated in the M factors 

in Table 3-2 of the Scope and Methods Plan. 

Regarding the planned approach for addressing uncertainty, I suggest that actual realistic ranges 

be used rather than a standard plus/minus 5% type of approach. Using realistic ranges for the 

important variables will result in an evaluation that will be closer to the types of situations that 

occur in the real world. 

In summary, I look forward to reviewing an expanded literature review and state of the science 

on exposure estimation in the 2nd draft of the ISA and as part of the 1st draft of the Risk and 

Exposure document. 

Selected, easy for me to find, references for spatial mapping (see above discussion for Chapter 3.6.7): 

Gauderman, Avol, Lurmann, Kuenzli, Filliland, Peters, and McConnell “Childhood Asthma and 

Exposure to Traffic and Nitrogen Dioxide, Epidemiology 2005; 16, 737-743. 

Ross, Jerrett, Ito, Tempalski, and Thurston “A land use Regression for predicting fine particulate 

matter concentrations in the New York City region”, Atmospheric Environment 41 (2007) 2255-

2269. 

Hoek, Beelen, Hoogh, Vienneau, Gulliver, Fischer, and Briggs “A review of land-use regression 

models to assess spatial variation of outdoor air pollution” Atmospheric Environment 42 (2008) 

7561-7578. 

Henderson, Beckerman, Jerrett, and Brauer “Application of Land Use Regression to Estimate 

Long-Term Concentrations of Traffic-Related NItrogen Oxides and Fine Particulate Matter 

ES&T 2007, 41, 2422-2428. 

Molitor, Jerrett, Chang, Molitor, Gauderman, Berhane, McConnel, Lurmann, Wu, Winer, and 

Thomas “Assessing Uncertainty in Spatial Exposure Models for Air Pollution Health Effects 

Assessment EHP vol 115,no 8, August 2007. 

Popawski, Gould, Setton, Allen, Su, Larson, Henderson, Brauer, Hystad, LIghtowlers, Keller, 

Cohen, Silva, and Buzzelli “Intercity transferability of land use regression models for estimating 

ambient concentrations of nitrogen dioxide” J Exposure Science & Environmental Epidemiology 

(2008), 1-11 

19 
Dr. Armistead (Ted) Russell 

Given that the Scope and Methods (SM) document describes the approach to be taken mainly in 

general terms, it succeeds in laying out a reasonable plan to achieve the objective of providing a 

quantitative assessment of exposure to CO using various target levels. The use of Los Angeles 

and Denver make sense, though Fairbanks or Anchorage would be of interest as well given the 

extreme conditions found there. The use of APEX makes sense, though this again argues for a 

more robust evaluation of APEX across pollutants. 

A second concern I have is that it is not apparent how the results from LA and Denver will be 

generalized to a national scale. In the end, I suspect the panel will want to know what the 

resulting exposures will be at a national level to various level/forms of the standard. 

In replying to the given questions: 

1. As discussed in the Plan, at this time there does not appear to be sufficient controlled 

human exposure data to support development of quantitative dose-response relationships 

for the health effects reported in subjects with angina. Following the same overall 

approach used in prior CO NAAQS reviews, the planned approach is to characterize risks 

associated with these effects by estimating exposures and resulting dose (i.e., COHb 

levels) and estimating the number and frequency of occurrences over several potential 

health effect benchmark levels for the cardiovascular disease population. The potential 

health effect benchmark levels are expressed in terms of COHb levels and are based on 

the evaluation of the controlled human exposure studies in the draft ISA. With regard to 

this planned approach for risk characterization for cardiovascular related health effects 

reported in controlled human exposure studies reporting decreased time to onset of 

angina, what are the Panel members’ views on: 

a. The overall planned approach, which is to estimate the number and percent of the 

population with cardiovascular disease that would exceed potential health effect 

benchmark levels upon just meeting various CO air quality scenarios; 

Answer: This is appropriate if done at the national scale, or a strong linkage can be made to 

what is found for Denver and LA with the rest of the country. 

b. The range of potential health effect benchmark COHb levels (i.e., 2.0, 2.5, and 3.0 

percent COHb) that staff plans to use to characterize these health risks. 

2. While the first draft ISA reaches the conclusion that the overall health effects evidence 

supports the judgment that ambient CO concentrations are likely causal for 

cardiovascular morbidity as a category, the document recognizes the uncertainties that 

exist with respect to evaluating studies of the association between emergency room visit 

and hospital admissions, respectively, for cardiovascular effects and ambient CO 

concentrations. In particular, the ISA raises the question of whether ambient CO levels 

are serving as a surrogate for one or more elements of the traffic-related air pollution mix. 

With regard to the approach for risk characterization, the Plan raises several study-related 

20 
issues affecting judgments about whether the evidence is supportive of developing 

quantitative risk estimates for emergency department visits and hospital admissions for 

cardiovascular effects related to ambient CO concentrations. 

a. What are the Panel members’ views on whether the concerns raised about ambient 

CO levels potentially serving as a surrogate for one or more components of the 

overall traffic-related air pollutant mixture limit the utility of a quantitative risk 

assessment for these health endpoints? 

Answer: I do view this as a serious limitation, and while the ISA does bring up the co-exposures 

to other pollutants, I do not view that the ISA has gone as deeply as it should. I think this 

uncertainty does make a quantitative risk assessment of less value. 

b. Given the potential for CO at ambient levels to act as a marker for the effects of 

another traffic-related pollutant or mix of pollutants, what are the Panel members’ 

views on whether or not the results of co-pollutant models provide sufficient 

evidence to support a quantitative risk assessment for CO effects at ambient 

levels? 

Answer: I do not see that there are sufficient studies that have investigated the range of copollutants to adequately model the impacts of these co-pollutants, and thus, I would be hesitant 

to conduct a quantitative risk assessment. 

Chapter 3 – Scope and Approach for Population Exposure/Dose Analysis

4. We plan to build upon the basic structure and design of the exposure assessment 

conducted in the previous review. Since that time there have been major improvements 

in the exposure model and in the data for input to the model. Are the Panel members 

aware of information sources that would help inform further improvements that would be 

worth considering in the current review? 

Answer: No. 

5. One of the main issues in this analysis is how to estimate ambient CO concentrations on 

and near roadways, which can be significant contributors to ambient CO exposures. The 

relationship between CO levels measured at ambient fixed site monitors is highly variable 

due to the spatial and temporal variability of on- and near-roadway CO concentrations. 

In the previous review, proximity factors were used to adjust the concentrations measured 

at monitors to estimate roadway-related concentrations of CO. We plan to conduct a 

review of the literature and draw upon the results of near-road studies to update the 

proximity factor distributions. Do the Panel members have recommendations for 

improvements or alternatives to this approach? 

Answer: It would appear that, if you have the time and resources, using a more detailed 

Gaussian model would be good, or possibly a land use based model. In the end, resource 

constraints will dominate, and while I think a proximity-based modeling approach is not 

scientifically the best, it likely produces results that will get over the bar. Such a proximity 

model should take in to account traffic intensity and fleet mix, not just distance to road. 

21 
22 

6. The planned approach for addressing uncertainty is primarily qualitative with a focus on 

sensitivity analysis and a limited quantitative analysis for those variables determined to 

be most influential with respect to exposure and/or dose estimation and where supporting 

data are available. 

a. What are the Panel members’ views concerning this general approach? 

Answer: I think that staff should be as quantitative as possible right up front, and provide best 

estimates of the uncertainties. These estimates can be caveated and discussed, but without going 

through this exercise, it is possible to mis-label an uncertainty as to be low, medium or high. 

b. Spatial and temporal gradients in ambient CO relative to CO concentrations 

measured at fixed-site monitors are potentially a major source of uncertainty in 

the exposure and dose estimates. Do the Panel members have suggestions for 

how best to characterize the uncertainties in this relationship? 

 

I would use the results of the Gaussian modeling and the various detailed monitoring 

experiments that have been conducted. Some sort of synthesis of the two would be of interest.