5A-1

APPENDIX 5A: COMPREHENSIVE CHARACTERIZATION OF UNCERTAINTY IN O3

BENEFITS ANALYSIS

Overview

As noted in Chapter 5, the benefits analysis relies on an array of data inputs—including air 

quality modeling, health impact functions and valuation estimates among others—which are 

themselves subject to uncertainty and may also in turn contribute to the overall uncertainty in this 

analysis. The RIA employs a variety of analytic approaches designed to reduce the extent of the 

uncertainty and/or characterize the impact that uncertainty has on the final estimate. We strive to 

incorporate as many quantitative assessments of uncertainty as possible (e.g., Monte Carlo 

assessments, sensitivity analyses); however, there are some aspects we are only able to characterize 

qualitatively. 

To more comprehensively and systematically address these uncertainties, including those we 

cannot quantify, we adapt the WHO uncertainty framework (WHO, 2008), which provides a means for 

systematically linking the characterization of uncertainty to the sophistication of the underlying health 

impact assessment. EPA has applied similar approaches in peer-reviewed analyses of PM2.5-related 

impacts (U.S. EPA, 2010b, 2011, 2012) and O3-related impacts (U.S. EPA, 2014). EPA’s Science 

Advisory Board (SAB) has supported using a tabular format to qualitatively assess the uncertainties 

inherent in the quantification and monetization of health impacts, including identifying potential bias, 

potential magnitude, confidence in our approach, and the level of quantitative assessment of each 

uncertainty (U.S. EPA-SAB, 1999, 2001, 2004a, 2004b, 2011a, 2011b). The assessments presented 

here are largely consistent with those previous peer-reviewed assessments. 

This appendix focuses on uncertainties inherent in the O3 benefits estimates. For more 

information regarding the uncertainties inherent in the PM benefits estimates, please see the 2012 PM 

NAAQS RIA (U.S. EPA, 2012).

5A.1 Description of Classifications Applied in the Uncertainty Characterization

Table 5A-1 catalogs the most significant sources of uncertainty in the O3 benefits analysis and 

then characterizes four dimensions of that uncertainty (listed below). The first two dimensions focus 

on the nature of the uncertainty. The third and fourth dimensions focus on the extent to which the 

analytic approach chosen in the benefits analysis either minimizes the impact of the uncertainty or 

quantitatively characterizes its impact.
5A-2

1) The direction of the bias that a given uncertainty may introduce into the benefits 

assessment if not taken into account in the analysis approach; 

2) The magnitude of the impact that uncertainty is likely to have on the benefits estimate if 

not taken into account in the analysis approach; 

3) The extent to which the analytic approach chosen is likely to minimize the impact of 

that uncertainty on the benefits estimate; and

4) The extent to which EPA has been able to quantify the residual uncertainty after the 

preferred analytic approach has been incorporated into the benefits model. 

5A.1.1 Direction of Bias

The “direction of bias” column in Table 5A-1 is an assessment of whether, if left unaddressed, 

an uncertainty would likely lead to an underestimate or overestimate the total monetized benefits. In 

some cases we indicate that there are reasons why the bias might go either direction, depending upon 

the true nature of the underlying relationship. Where available, we base the classification of the 

“direction of bias” on the analysis in the Integrated Science Assessment for Ozone and Related 

Photochemical Oxidants (hereafter, “O3 ISA”) (U.S. EPA, 2013a). Additional sources of information 

include advice from SAB and the National Academies of Science (NAS), as well as studies from the 

peer-reviewed literature. In some cases we indicate that there is not sufficient information to estimate 

whether the uncertainty would likely lead to under or overestimation of benefits; these cases are 

identified as “unable to determine.” 

5A.1.2 Magnitude of Impact

The “magnitude of impact” column in Table 5A-1 is an assessment of how much plausible 

alternative assumptions about the underlying relationship about which we are uncertain could influence 

the overall monetary benefits. EPA has applied similar classifications in previous risk and benefit 

analyses (U.S. EPA, 2010b, 2011, 2014), but we have slightly revised the category names and the cutoffs here.1 The definitions used here are provided below. 

 

1

In The Benefits and Costs of the Clean Air Act from 1990 to 2020 (U.S. EPA, 2011), EPA applied a classification of 

“potentially major” if a plausible alternative assumption or approach could influence the overall monetary benefit 

estimate by 5% percent or more and “probably minor” if an alternative assumption or approach is likely to change the 

total benefit estimate by less than five percent. In the Quantitative Health Risk Assessment for Particulate Matter (U.S. 

EPA, 2010b), EPA applied classifications of “low” if the impact would not be expected to impact the interpretation of 

risk estimates in the context of the PM NAAQS review, “medium” if the impact had the potential to change the 

interpretation; and “high” if it was are likely to influence the interpretation of risk in the context of the PM NAAQS 

review.
5A-3

 High—if the uncertainty associated with an assumption could influence the total 

monetized benefits by more than 25%.

 Medium—if the uncertainty associated with an assumption could influence the total 

monetized benefits by 5% to 25%.

 Low—if the uncertainty associated with an assumption could influence the total 

monetized benefits by less than 5%.

 For each uncertainty, we provide as much quantitative information as is available in the 

table to support the classification. 

Although many of the sources of uncertainty could affect both morbidity and mortality 

endpoints, because mortality benefits comprise over 94% of the monetized benefits that we are able to 

quantify in this analysis, uncertainties that affect the mortality estimate have the potential to have 

larger impacts on the total monetized benefits than uncertainties affecting only morbidity endpoints. 

One morbidity-related uncertainty that could have a significant impact on benefits estimate is the 

extent to which omitted morbidity endpoints are included in the benefits analysis. Including additional 

morbidity endpoints that are currently not monetized would reduce the fraction of total benefits from 

mortality. Ultimately, the magnitude classification is determined by professional judgment of EPA 

staff based on the results of available information, including other U.S. EPA assessments of 

uncertainty (U.S. EPA, 2010b, 2011) 

Based on this assessment, the uncertainties which we classified as high or medium-high impact 

are: the causal relationship between long-term and short-term ozone exposure and mortality, shape of 

the concentration-response function for both categories of ozone-related mortality, and mortality 

valuation, specifically for long-term exposure-related mortality. The classification of these 

uncertainties as “high magnitude” is generally consistent with the results of EPA’s Influence Analysis

(Mansfield et al., 2009), the Quantitative Health Risk Assessment for Particulate Matter (U.S. EPA, 

2010b), and the Benefits and Costs of the Clean Air Act 1990 to 2020 (U.S. EPA, 2011).

5A.1.3 Confidence in Analytic Approach

The “confidence in analytic approach” column of Table 5A-1 is an assessment of the scientific 

support for the analytic approach chosen (or the inherent assumption made) to account for the 

relationship about which we are uncertain. In other words, based on the available evidence, how 

certain are we that EPA’s selected approach is the most plausible of the potential alternatives. Similar 
5A-4

classifications have been included in previous risk and benefits analyses (U.S. EPA, 2010b, 2011).2

The three categories used to characterize the degree of confidence are: 

 High—the current evidence is plentiful and strongly supports the selected approach;

 Medium—some evidence exists to support the selected approach, but data gaps are 

present; and

 Low—limited data exists to support the selected approach.

Ultimately, the degree of confidence in the analytic approach is EPA staff’s professional 

judgment based on the volume and consistency of supporting evidence, much of which has been 

evaluated in the O3 ISA (U.S. EPA, 2013a) and EPA’s independent Science Advisory Board (SAB). 

The O3 ISA evaluated the entire body of scientific literature on O3 science and was twice peerreviewed by EPA’s Clean Air Scientific Advisory Committee (CASAC). In general, we regard a 

conclusion in the O3 ISA or specific advice from SAB as supporting a high degree of confidence in the 

selected approach. 

Based on this assessment, we have low or low-medium confidence in the evidence available to 

assess exposure error in epidemiology studies, morbidity valuation, baseline incidence projections for 

morbidity, and omitted morbidity endpoints. However, because these uncertainties have been classified 

as having a low or low-medium impact on the magnitude of the benefits, further investment in 

improving the available evidence would not have a substantial impact on the total monetized benefits. 

5A.1.4 Uncertainty Quantification

The column of Table 5A-1 labeled “uncertainty quantification” is an assessment of the extent to 

which we were able to use quantitative methods to characterize the residual uncertainty in the benefits 

analysis, after addressing it to the extent feasible in the analytic approach for this RIA. We categorize 

the level of quantification using the four tiers used in the WHO uncertainty framework (WHO, 2008). 

The WHO uncertainty framework is a well-established approach to assess uncertainty in risk estimates 

that systematically links the characterization of uncertainty to the sophistication of the health impact 

assessment. The advantage of using this framework is that it clearly highlights the level of uncertainty 

quantification applied in this assessment and the potential sources of uncertainty that require methods 

 

2 We have applied the same classification as The Benefits and Costs of the Clean Air Act from 1990 to 2020 (U.S. EPA, 

2011a) in this analysis. In the Quantitative Health Risk Assessment for Particulate Matter (U.S. EPA, 2010b), EPA 

assessed the degree of uncertainty (low, medium, or high) associated with the knowledge-base (i.e., assessed how well 

we understand each source of uncertainty), but did not provide specific criteria for the classification.
5A-5

development in order to assess quantitatively. Specifically, EPA applied this framework in multiple 

risk and exposure assessments (U.S. EPA, 2010b, 2014), and it has been recommended in EPA 

guidance documents assessing air toxics-related risk and Superfund site risks (U.S. EPA, 2004 and 

2001, respectively). Ultimately, the tier decision is professional judgment of EPA staff based on the 

availability of information for this assessment. The tiers used in this assessment are defined below.

 Tier 0—screening level, generic qualitative characterization.

 Tier 1—Scenario-specific qualitative characterization.

 Tier 2—Scenario-specific sensitivity analysis.

 Tier 3—Scenario-specific probabilistic assessment of individual and combined 

uncertainty.

Within the limits of the data, we strive to use a more sophisticated approaches (e.g., Tier 2 or 3) 

for characterizing uncertainties that have the largest magnitudes and could not be completely addressed 

through the analytic approach. The uncertainties for which we have conducted probabilistic (Tier 3) 

assessments in this analysis are mortality causality, the shape of the concentration-response function, 

and mortality and morbidity valuation. For lower magnitude uncertainties, we include qualitative 

discussions of the potential impact of uncertainty on risk results (WHO Tier 0/1) and/or completed 

sensitivity analyses assessing the potential impact of sources of uncertainty on risk results (WHO Tier 

2). 

5A.2 Organization of the Qualitative Uncertainty Table

Table 5A-1 is organized as follows. The uncertainties are grouped by category (i.e., 

concentration-response function, valuation, population and baseline incidence, omitted benefits 

categories, and exposure changes). Within each category, the uncertainties are sorted by magnitude of 

impact (i.e., high to low) then by confidence in our approach (i.e., low to high). In the table, red (bold) 

text is used to indicate the uncertainties that likely have a high magnitude of impact on the total 

benefits estimate. This organization highlights the uncertainty with the largest potential impact and the 

lowest confidence at the top of each category.
5A-6

Table 5A-1. Summary of Qualitative Uncertainty for Key Modeling Elements in O3 Benefits

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Uncertainties Associated with Concentration-Response Functions

Causal relationship 

between short-term 

O3 exposure and 

premature mortality

Overestimate, if short-term O3

exposure does not have a causal 

relationship with premature 

mortality.

High High Tier 1 (qualitative)

Mortality generally dominates 

monetized benefits, so small 

uncertainties could have large 

impacts on the total monetized 

benefits.

Our approach is consistent with the O3

Integrated Science Assessment (ISA), 

which determined that premature mortality 

has a likely causal relationship with shortterm O3 exposure based on the collective 

body of evidence (p. 6-264). In addition, the 

NAS recommended that EPA “should give 

little or no weight to the assumption that 

there is no causal association between 

estimated reductions in premature mortality 

and reduced ozone exposure” (NRC, 2008). 

In 2010, the Health Effects Subcommittee 

of the Advisory Council on Clean Air 

Compliance Analysis, while reviewing 

EPA’s The Benefits and Costs of the Clean 

Air Act 1990 to 2020 (U.S. EPA, 2011), 

also confirmed the NAS recommendation to 

include O3 mortality benefits (U.S. EPASAB, 2010).

Either Medium-High Medium Tier 2 (sensitivity analysis)
5A-7

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Shape of the C-R 

functions, 

particularly at low 

concentrations for 

short-term ozone 

exposure-related 

mortality

The direction of bias that assuming 

linear-no threshold model or 

alternative model introduces 

depends upon the “true” functional 

from of the relationship and the 

specific assumptions and data in a 

particular analysis. For example, if 

the true function identifies a 

threshold below which health 

effects do not occur, benefits may 

be overestimated if a substantial 

portion of those benefits were 

estimated to occur below that 

threshold. Alternately, if a 

substantial portion of the benefits 

occurred above that threshold, the 

benefits may be underestimated 

because an assumed linear nothreshold function may not reflect 

the steeper slope above that 

threshold to account for all health 

effects occurring above that 

threshold.

The magnitude of this impact 

depends on the fraction of benefits 

occurring in areas with lower 

concentrations. Mortality 

generally dominates monetized 

benefits, so small uncertainties 

could have large impacts on total 

monetized benefits.

The O3 ISA did not find any evidence that 

supports a threshold in the relationship 

between short-term exposure to O3 and 

mortality within the range of O3

concentrations observed in the United 

States, and recent evidence suggests that the 

shape of the O3 -mortality C-R curve 

remains linear across the full range of O3

concentrations (p. 6-257). Consistent with 

the O3 ISA, we assume a log-linear nothreshold model for the concentrationresponse functions for short-term O3

mortality. However, the ISA notes that there 

is less certainty in the shape of the C-R 

function below 20 ppb due to the low 

density of data in this range (p. 6-254-255). 

The comparison of short-term 

mortality against the associated 

distribution of (ozone seasonaveraged) 8hr max ozone levels 

(see Appendix 5C, section 5C.2) 

suggests that the vast majority of 

predicted reductions in mortality 

are associated with days having 

8hr max values that fall within the 

range of increased confidence in 

specifying the nature of the 

mortality response (as identified in 

the Ozone ISA). 

Causal relationship 

between long-term 

O3 exposure and 

Overestimate, if short-term O3

exposure does not have a causal 

relationship with premature 

mortality.

High High Tier 1 (qualitative)
5A-8

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

premature 

respiratory mortality

The total dollar benefits of 

reducing ozone levels are 

comprised mostly of the value 

placed on reducing the risk of 

premature death, so small 

uncertainties could have large 

impacts on the total monetized 

benefits.

While the Ozone ISA concludes that 

evidence is suggestive of a causal 

association between total mortality and 

long-term ozone exposure (section 7.7.1), 

specifically with regard to respiratory health 

effects (including mortality), the ISA 

concludes that there is likely to be a causal 

association (section 7.2.8). Furthermore, in 

their review of the HREA completed for the 

O3 NAAQS review, the CASAC expressed 

their support for our plan to include a nonthreshold-based modeling of long-term 

exposure-related respiratory mortality in the 

core estimate. The CASAC also supported 

EPA’s plan to consider the potential for 

thresholds in the response (in the range of 

40-60ppb) as a sensitivity analysis in the 

HREA (see section 5.6.3.1). This same 

approach (with regard to the core approach 

and sensitivity analysis for long-term 

exposure-related morality) has been adopted 

for the RIA (see below). 

Shape of the C-R 

functions, 

particularly at low 

concentrations for 

long-term ozone 

exposure-related 

respiratory mortality

Either Medium-High Medium Tier 2 (sensitivity analysis)
5A-9

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

The direction of bias that assuming 

linear-no threshold model or 

alternative model introduces 

depends upon the “true” functional 

from of the relationship and the 

specific assumptions and data in a 

particular analysis. For example, if 

the true function identifies a 

threshold below which health 

effects do not occur, benefits may 

be overestimated if a substantial 

portion of those benefits were 

estimated to occur below that 

threshold. Alternately, if a 

substantial portion of the benefits 

occurred above that threshold, the 

benefits may be underestimated 

because an assumed linear nothreshold function may not reflect 

the steeper slope above that 

threshold to account for all health 

effects occurring above that 

threshold.

The magnitude of this impact 

depends on the fraction of benefits 

occurring in areas with lower 

concentrations. Note, that due to 

limitations in our ability to predict 

the lag structure associated with 

reductions in long-term ozone 

exposure-related mortality, we are 

not including dollar benefits 

associated with this endpoint in 

the core benefits analysis (which

significantly reduces the role of 

this source of uncertainty in 

impact core benefit estimates).

In their memo (see Sasser 2014) clarifying 

the results of their study (Jerrett et al., 2009) 

regarding long-term ozone exposure-related 

respiratory mortality, the study authors note 

that in terms of goodness of fit, long-term 

health risk models including O3 clearly 

performed better than models without O3, 

indicating the improved predictions of 

respiratory mortality when O3 is included. 

In the article proper, the authors state that, 

“There was limited evidence that a 

threshold model specification improved 

model fit as compared with a nonthreshold 

linear mode...”. Furthermore, in the memo 

referenced above, the authors conclude that 

considerable caution should be exercised in 

using any specific threshold, particularly 

when the more stringent statistical test 

indicates there is no significantly improved 

prediction. The CASAC was supportive of 

the approach we used in the HERA of using 

a non-threshold C-R function based on this 

study to generate core estimates and 

consider the impact of potential thresholds 

(ranging from 40-60ppb) as a sensitivity 

analysis. 

As part of the sensitivity analyses 

completed for the RIA, we did 

examine the potential impact of 

thresholds (from 40 to 60ppb) in 

the response function for longterm exposure-related morality 

(see Appendix 5B, section 5B.1). 

That analysis suggested that 

thresholds between 55 and 60ppb 

would have a substantial impact on 

overall modeled benefits, while 

thresholds above below 50ppb 

would have a minor impact on 

predicted benefits (see Appendix 

5B, Table 5B-1). 

Exposure error in 

epidemiology studies

Underestimate (generally) Medium Low-Medium Tier 1 (qualitative)

The O3 ISA states that exposure 

measurement error can also be an 

important contributor to 

uncertainty in effect estimates 

associated with both short-term and 

long-term studies (p. 1xii). 

Together with other factors (e.g., 

low data density), exposure error 

can smooth the C-R functions and 

obscure potential thresholds (p. 

lxix). In addition, the O3 ISA states 

that exposure error can bias effect 

estimates toward or away from the 

null and widen confidence intervals 

(p. lxii). 

Recent analyses reported in 

Krewski et al. (2009) demonstrate 

the potentially significant effect 

that this source of uncertainty can 

have on effect estimates. These 

analyses also illustrate the 

complexity and site-specific 

nature of this source of 

uncertainty.

Although this underestimation is well 

documented, including in the O3 ISA, the 

SAB has not suggested an approach to 

adjust for this bias.

(No quantitative method available)

Unknown Medium Medium Tier 1 (qualitative)
5A-10

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Adjustment of risk 

coefficients to 8-

hour maximum

Several of the mortality 

epidemiological studies were 

reported for a 24-hour average or 

1-hour maximum ozone level. 

These metrics are not the most 

relevant to characterizing 

population-level exposure. Thus, 

we have converted ozone mortality 

health impact functions that use 

these metric to maximum 8-hour 

average ozone concentration using 

standard conversion functions. The 

conversion ratios are based on 

observed relationships between the 

24-hour average and 8-hour 

maximum in the underlying 

studies.

This conversion also does not 

affect the relative magnitude of 

the health impact function. 

However, we do note that the 

pattern of 8hr max concentrations 

for a particular location over an 

ozone season could differ from the 

pattern of 1hr max or 24 hour 

average metrics for that same 

location. Consequently, estimates 

of incidence reductions (and 

associated dollar benefits) could 

differ for a particular location 

depending on the metric used for 

risk calculations.3

This practice is consistent both with the 

available exposure modeling and with the 

form of the current ozone standard. 

However, in some cases, these conversions 

were not specific to the ozone “warm” 

season which was the period used in the 

benefits analysis, which introduces 

additional uncertainty due to the use of 

effect estimates based on a mixture of warm 

season and all year data in the 

epidemiological studies.

(No quantitative method available)

Confounding by 

individual risk 

factors, other than 

SES—e.g., smoking, 

or ecologic factors, 

which represent the 

neighborhood, such

as unemployment

Either, depending on the factor and 

study Medium Medium Tier 2 (sensitivity analysis)

Individual, social, economic, and 

demographic covariates can bias 

the relationship between particulate 

air pollution and mortality, 

particularly in cohort studies that 

rely on regional air pollution levels.

Because mortality dominates 

monetized benefits, even a small 

amount of confounding could 

have medium impacts on total 

monetized benefits.

To minimize confounding effects, we use 

risk coefficients that control for individual 

risk factors to the extent practical.

(Quantitative methods available 

but not assessed in this analysis.)

Confounding and 

effect modification 

by co-pollutants

Either, depending upon the 

pollutant. Medium Medium Tier 1 (qualitative)

Disentangling the health responses 

of combustion-related pollutants 

(i.e., PM, SOx, NOx, ozone, and 

CO is a challenge. The O3 ISA 

defines a confounder as the true 

cause of the association observed 

between the exposure and the 

outcome; in contrast, an effect 

modifier changes the magnitude of 

the association between the 

exposure and the outcome (p. lxii).

Because this uncertainty could 

affect mortality and because 

mortality generally dominates 

monetized benefits, even a small 

uncertainties could have medium 

impacts on total monetized 

benefits.

The O3 ISA states that there is high 

confidence that unmeasured confounders 

are not producing the findings when 

multiple studies are conducted in various 

settings using different subjects or 

exposures, such as multi-city studies (p. 

lxi). When modeling effects of pollutants 

jointly (e.g., PM and O3), we apply multipollutant effect estimates when those 

estimates are available to avoid doublecounting and satisfy other selection criteria. 

In addition, we apply multi-city effect 

estimates when available.

(No quantitative method available)

 

3 We note that if metric adjustment ratios were derived for individual point locations (reflecting the relationship between metrics for each location) and those metrics 

were used to generated adjusted C-R functions for each location, then some of this potential uncertainty could be reduced, however available data does not support the 

application of highly location-specific adjustment ratios (with cross-city or national adjustment ratios being typically used instead).
5A-11

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Application of C-R 

relationships only to 

the original study 

population

Underestimate Low High Tier 2 (sensitivity analysis)

Estimating health effects for only 

to the original study population 

may underestimate the whole 

population benefits of reductions in 

pollutant exposures.

Mortality generally dominates 

monetized benefits, so further age 

range expansions for morbidity 

endpoints would have a small 

impact on total monetized 

benefits.

Following advice from the SAB (U.S. EPASAB, 2004a, pg. 7) and NAS (NRC, 2002, 

pg. 114), we expanded the age range for 

childhood asthma exacerbations beyond the 

original study population to ages 6-18.

(Quantitative methods available 

but not assessed in this analysis.)

Uncertainties Associated with Economic Valuation

Mortality Risk 

Valuation / Valueof-a-Statistical-Life 

(VSL)

Unknown High Medium Tier 3 (probabilistic)

Some studies suggest that EPA’s 

mortality valuation is too high, 

while other studies suggest that it is 

too low. Differences in age, 

income, risk aversion, altruism, 

nature of risk (e.g., cancer), and 

study design could lead to higher or 

lower estimates of mortality 

valuation.

Mortality generally dominates 

monetized benefits, so moderate 

uncertainties could have a large 

effect on total monetized benefits.

The VSL used by EPA is based on 26 labor 

market and stated preference studies 

published between 1974 and 1991. EPA is 

in the process of reviewing this estimate and 

will issue revised guidance based on the 

most up-to-date literature and 

recommendations from the SAB-EEAC in 

the near future (U.S. EPA, 2010a, U.S. 

EPA-SAB, 2011c).

Assessed uncertainty in mortality 

valuation using a Weibull 

distribution.

Cessation lag 

structure for long 

term O3 mortality

Underestimate Medium-High Low Tier 2 (sensitivity analysis)

Jerrett et al. (2009) notes that, 

“Allowing for a 10-year period of 

exposure to ozone (5 years of 

follow-up and 5 years before the 

follow-up period) did not 

appreciably alter the risk 

estimates...” We acknowledge 

substantial uncertainty associated 

with specifying the lag for longterm respiratory mortality.

Although the cessation lag does 

not affect the number of premature 

deaths attributable to long-term O3

exposure, it affects the timing of 

those deaths and thus the 

discounted monetized benefits. 

Mortality generally dominates 

monetized benefits, so moderate 

uncertainties could have a large 

effect on total monetized benefits.

As discussed in section 5.6.4.1, in 

presenting dollar benefit estimates as part of 

the sensitivity analysis (presenting dollar 

benefits for long-term ozone-related 

morality), we include both an assumption of 

zero lag and a lag structure matching that 

used for the core PM2.5 estimate (the SAB 

20 year segmented lag). Inclusion of the 

zero lag reflects consideration for the 

possibility that the long-term respiratory 

mortality estimate captures primarily, an 

accumulation of short term mortality effects 

across the ozone season. The use of the 20 

year segmented lag reflects consideration 

for advice provided by the HES (USEPASAB, 2010).

As shown in sensitivity analysis 

results presented Tables Appendix 

5-A, the use of the 20 year 

segmented lag results in a 10-20% 

reduction in the total dollar benefit 

(using a 3% and 7% discount rate, 

respectively) relative to the 

alternative approach of applying 

no lag (i.e., assuming all of the 

mortality reductions occur in the 

same year).

Either Medium Medium Tier 2 (sensitivity analysis)
5A-12

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Income growth 

adjustments

Income growth increases 

willingness-to-pay (WTP) 

valuation estimates, including 

mortality, over time. From 1997 to 

2010, personal income and GDP 

growth have begun to diverge. If 

this trend continues, the 

assumption that per capita GDP 

growth is a reasonable proxy for 

income growth may lead to an 

overstatement of benefits. (IEc, 

2012).

Income growth from 1990 to 2020 

increases mortality valuation by 

20%. Alternate estimates for this 

adjustment vary by 20% (IEc, 

2012). Because we do not adjust 

for income growth over the 20-

year cessation lag, this approach 

could also underestimate the 

benefits for the later years of the 

lag.

Consistent with SAB recommendations 

(U.S. EPA,-SAB, 2000, pg. 16), we adjust 

WTP for income growth. Difficult to 

forecast future income growth. However, in 

the absence of readily available income data 

projections, per capita GDP is the best 

available option.

As shown in Appendix 5-A, the 

use of alternate income growth 

adjustments would change the 

monetized benefits by +33% to 

−14%.

Morbidity valuation

Underestimate Low Low Tier 3 (probabilistic), where 

available

Morbidity benefits such as hospital 

admissions are calculated using 

cost-of-illness (COI) estimates, 

which are generally half the 

willingness-to-pay to avoid the 

illness (Alberini and Krupnick, 

2000). In addition, the morbidity 

costs do not reflect physiological 

responses or sequelae events, such 

as increased susceptibility for 

future morbidity.

Even if we doubled the monetized 

valuation of morbidity endpoints 

using COI valuation that are 

currently included in the RIA, the 

change would still be less than 5% 

of the monetized benefits. It is 

unknown how much including 

sequelae events could increase 

morbidity valuation.

Although the COI estimates for 

hospitalizations reflect recent data, we have 

not yet updated other COI estimates such as 

for school loss days. The SAB concluded 

that COI estimates could be used as 

placeholders where WTP estimates are 

unavailable, but it is reasonable to presume 

that this strategy typically understates WTP 

values (U.S. EPA-SAB, 2004b, pg. 3).

Assessed uncertainty in morbidity 

valuation using distributions 

specified in the underlying 

literature, where available (see 

Table 5-10).

Uncertainties Associated with Baseline Incidence and Population Projections

Population estimates 

and projections

Either Low–Medium Medium Tier 1 (qualitative)

The monetized benefits would 

change in the same direction as the 

over- or underestimate in 

population projections in areas 

where exposure changes.

Monetized benefits are 

substantially affected by 

population density. Comparisons 

using historical census data show 

that population projections are 

±5% nationally, but projection 

accuracy can vary by locality. 

Historical error for Woods & 

Poole’s population projections has 

been ±8.1% for county-level 

projections and ±4.1% for states 

(Woods and Poole, 2012). The 

magnitude of impact on total 

monetized benefits depends on the 

specific location where PM is 

reduced.

We use population projections for 5-year 

increments for 304 

race/ethnicity/gender/age groups (Woods 

and Poole, 2012) at Census blocks. 

Population forecasting is well-established 

but projections of future migration due to 

possible catastrophic events are not 

considered. In addition, projections at the 

small spatial scales used in this analysis are 

inherently more uncertain than projections 

at the county- or state-level.

(No quantitative method available)

Unknown Low-Medium Medium Tier 1 (qualitative)
5A-13

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Uncertainty in 

projecting baseline 

incidence rates for 

mortality

Because the mortality rate 

projections for future years reflect 

changes in mortality patterns as 

well as population growth, the 

projections are unlikely to be 

biased.

Because mortality generally 

dominates monetized benefits, 

small uncertainties could have 

medium impacts on total 

monetized benefits.

The county-level baseline mortality rates 

reflect recent databases (i.e., 2004–2006 

data) and projected for 5-year increments 

for multiple age groups. This database is 

generally considered to have relatively low 

uncertainty (CDC Wonder, 2008). The 

projections account for both spatial and 

temporal changes in the population.

(No quantitative method available)

Uncertainty in 

projecting baseline 

incidence rates and 

prevalence rates for 

morbidity

Either, depending on the health 

endpoint Low Low-Medium Tier 1 (qualitative)

Morbidity baseline incidence is 

available for current year only (i.e., 

no projections available). 

Assuming current year levels can 

bias the benefits for a specific 

endpoint if the data has clear trends 

over time. Specifically, asthma 

prevalence rates have increased 

substantially over the past few 

years while hospital admissions 

have decreased substantially.

The magnitude varies with the 

health endpoint, but the overall 

impact on the total benefits 

estimate from these morbidity 

endpoints is likely to be low.

We do not have a method to project future 

baseline morbidity rates, thus we assume 

current year levels will continue. While we 

try to update the baseline incidence and 

prevalence rates as frequently as

practicable, this does not continue trends 

into the future. Some endpoints such as 

hospitalizations and ER visits have more 

recent data (i.e., 2007) stratified by age and 

geographic location. Other endpoints, such 

as respiratory symptoms reflect a national 

average. Asthma prevalence rates reflect 

recent increases in baseline asthma rates 

(i.e., 2008).

(No quantitative method available)

Uncertainties Associated with Omitted Benefits Categories

Unquantified O3

health benefit 

categories, such as 

worker productivity 

and long-term 

mortality

Underestimate Medium-High Low Tier 2 (sensitivity analysis)

EPA has not included monetized 

estimates of these benefits 

categories in the core benefits 

estimate.

Although the potential magnitude 

is unknown, including all of the 

additional endpoints associated 

with O3 exposure that are 

currently not monetized could 

increase the total benefits by a 

large amount.

Current data and methods are insufficient to 

value national quantitative estimates of 

these health effects. The O3 ISA determined 

that respiratory effects (including mortality) 

are causally associated with long-term O3

exposure (p. 2–17). The O3 ISA also 

determined that outdoor workers have an 

increased risk of O3 -related health effects 

(p. 1-15), and that studies on outdoor 

workers show consistent evidence that 

short-term increases in ambient O3 exposure 

can decrease lung function in healthy adults 

(p.6-38). Additional studies link short-short 

term O3 exposure to reduced productivity in 

outdoor workers (Graf Zivin and Neidell, 

2013; Crocker and Horst, 1981).

We include sensitivity analyses 

reflecting long-term mortality 

which shows that this endpoint 

could add substantially to the total 

core benefits range (see Tables 5-

20 and 5-27). We are still 

considering options for updating 

the worker productivity analysis 

and including it as a sensitivity 

analyses.

Uncertainties Associated with Estimated Exposure Changes

Unknown Unknown Low Tier 1 (qualitative)
5A-14

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Spatial matching of 

air quality estimates 

from epidemiology 

studies to air quality 

estimates from air 

quality modeling

Epidemiology studies often assume 

one air quality concentration is 

representative of an entire urban 

area when calculating hazard 

ratios, while benefits are calculated 

using air quality modeling 

conducted at 12 km spatial 

resolution. This spatial mismatch 

could introduce uncertainty.

We have not controlled for this potential 

bias, and the SAB has not suggested an 

approach to adjust for this bias.

(No quantitative method available)

Uncertainties Associated with the Dollar-per-ton Approach Used in Modeling PM2.5 Co-benefits

Derivation of dollarper-ton estimates for 

PM2.5

Unknown Unknown Medium Tier 1 (qualitative)

In the analysis used to generate the 

dollar-per-ton values we assume 

that all fine particles, regardless of 

their chemical composition, are 

equally potent in causing premature 

mortality. However, the scientific 

evidence is not yet sufficient to 

allow differentiation of effect 

estimates by particle type. We also 

assume that the health impact 

function for fine particles is linear 

down to the lowest air quality 

levels modeled in this analysis. 

Thus, the estimates include health 

benefits from reducing fine 

particles in areas with varied 

concentrations of PM2.5, including 

regions that are in attainment with 

fine particle standard.

While there are concerns regarding the 

assumption of a uniform PM2.5 toxicity 

across sectors and a linear C-R function all 

the way down to zero, these sources of 

uncertainty impact benefits modeling for 

PM2.5 in general and have been discussed as 

part of earlier RIAs (see section 5.7.2, of the 

final PM RIA, U.S. EPA. 2012). We do 

recognize that, as discussed below (and on 

pp. 24-25 of US. EPA, 2013), there is

increased uncertainty when dollar-per-ton 

benefits are applied outside of the specific 

scenario used in their derivation. 

(No quantitative method available)

Application of Unknown Unknown Medium
5A-15

Potential Source of 

Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

dollar-per-ton 

estimates in the 

current Ozone 

NAAQS review

As discussed in section 5.4.4, we 

used a method to calculate the 

regional benefit-per-ton estimates 

that is a slightly modified version 

of the national benefit-per-ton 

estimates described in the TSD: 

Estimating the Benefit per Ton of 

Reducing PM2.5 Precursors from 17 

Sectors (U.S. EPA, 2013b). The 

national estimates were derived 

using the approach published in 

Fann et al. (2012c), but they have 

since been updated to reflect the 

epidemiology studies and Census 

population data first applied in the 

final PM NAAQS RIA (U.S. EPA, 

2012). These dollar-per-ton 

estimates were applied to sectorspecific NOx emissions reductions 

modeled as part of the current 

ozone NAAQS review. 

While we acknowledge 

uncertainty associated with 

applying dollar-per-ton estimates 

in the context of this ozone 

NAAQS review (and outside of 

the scenario in which they were 

derived), we are not in a position 

to characterize the magnitude of 

direction of any bias that might 

result from that application.

All benefit-per-ton have inherent 

limitations, including that the estimates 

reflect the geographic distribution of the 

modeled sector emissions, which may not 

match the emission reductions anticipated 

by the proposed guidelines, and they may 

not reflect local variability in population 

density, meteorology, exposure, baseline 

health incidence rates, or other local factors 

for any specific locations reflected in 

benefits modeling. However, the fact that 

we are modeling regional/national benefits 

rather than attempting a more spatially 

refined application of the dollar-per-ton 

values does reduce uncertainty in the 

benefits estimates that are generated.

(No quantitative method available)
5A-16

5A.3 References

Alberini, Anna and Alan Krupnick. 2000. “Cost-of-Illness and Willingness-to-Pay Estimates of the Benefits of 

Improved Air Quality: Evidence from Taiwan.” Land Economics

Vol. 76, No. 1 (Feb), pp. 37-53.

Centers for Disease Control and Prevention (CDC). 2008. National Center for Health Statistics. National Health 

Interview Survey, 1999-2008.

Fann, N.; K.R. Baker and C.M. Fulcher. 2012. “Characterizing the PM2.5-related health benefits of emission 

reductions for 17 industrial, area and mobile emission sectors across the U.S.” Environment International 49 

41–151. 

Graff Zivin, J., Neidell, M. (2012). “The impact of pollution on worker productivity.” American Economic Review, 

102, 3652-3673.

Industrial Economics, Incorporated (IEc). 2012. Updating BenMAP Income Elasticity Estimates—Literature 

Review. Memo to Neal Fann. March. Available on the Internet at 

<http://www.epa.gov/ttn/ecas/regdata/Benefits/Income%20Elasticity%20Update_Recommendations_noAppD.p

df>

Jerrett M, Burnett RT, Pope CA, III, et al. 2009. “Long-Term Ozone Exposure and Mortality.” New England 

Journal of Medicine 360:1085-95.

Krewski D, Jerrett M, Burnett RT, Ma R, Hughes E, Shi, Y, et al. 2009. “Extended follow-up and spatial analysis of 

the American Cancer Society study linking particulate air pollution and mortality.” HEI Research Report, 140, 

Health Effects Institute, Boston, MA.

Mansfield, Carol; Paramita Sinha; Max Henrion. 2009. Influence Analysis in Support of Characterizing Uncertainty 

in Human Health Benefits Analysis: Final Report. Prepared for U.S. EPA, Office of Air Quality Planning and 

Standards. November. Available on the internet at 

<http://www.epa.gov/ttn/ecas/regdata/Benefits/influence_analysis_final_report_psg.pdf>.

National Research Council (NRC). 2002. Estimating the Public Health Benefits of Proposed Air Pollution 

Regulations. Washington, DC: The National Academies Press. Washington, DC.

National Research Council (NRC). 2008. Estimating Mortality Risk Reduction and Economic Benefits from 

Controlling Ozone Air Pollution. National Academies Press. Washington, DC.

Sasser, E. 2014. Response to Comments Regarding the Potential Use of a Threshold Model in Estimating the 

Mortality Risks from Long-term Exposure to Ozone in the Health Risk and Exposure Assessment for Ozone, 

Second External Review Draft. Memorandum to Holly Stallworth, Designated Federal Officer, Clean Air 

Scientific Advisory Committee from EPA/OAQPS Health and Environmental Impacts Division.

Schwartz J, Coull B, Laden F. 2008. “The Effect of Dose and Timing of Dose on the Association between Airborne 

Particles and Survival.” Environmental Health Perspectives 116: 64-69.

U.S. Environmental Protection Agency (U.S. EPA). 2001. Risk assessment guidance for Superfund. Vol. III, Part A. 

Process for conducting probabilistic risk assessment (RAGS 3A). Washington, DC, United States 

Environmental Protection Agency (EPA 540-R-02-002; OSWER 9285.7-45; PB2002 963302. Available on the 

Internet at <http://www.epa.gov/oswer/riskassessment/rags3adt/index.htm>.

U.S. Environmental Protection Agency (U.S. EPA). 2004. EPA’s risk assessment process for air toxics: History and 

overview. In: Air toxics risk assessment reference library. Vol. 1. Technical resource manual. Washington, DC, 
5A-17

United States Environmental Protection Agency, pp. 3-1–3-30 (EPA-453-K-04-001A. Available on the Internet 

at <http://www.epa.gov/ttn/fera/data/risk/vol_1/chapter_03.pdf>.

U.S. Environmental Protection Agency (U.S. EPA). 2010a. Valuing Mortality Risk Reductions for Environmental 

Policy: A White Paper (SAB Review Draft). December. Available on the Internet at 

<http://yosemite.epa.gov/ee/epa/eerm.nsf/vwAN/EE-0563-1.pdf/$file/EE-0563-1.pdf>.

U.S. Environmental Protection Agency (U.S. EPA). 2010b. Quantitative Health Risk Assessment for Particulate 

Matter—Final Report. EPA-452/R-10-005. Office of Air Quality Planning and Standards, Research Triangle 

Park, NC. September. Available on the Internet at 

<http://www.epa.gov/ttnnaaqs/standards/pm/data/PM_RA_FINAL_June_2010.pdf>.

U.S. Environmental Protection Agency (U.S. EPA). 2011a. The Benefits and Costs of the Clean Air Act 1990 to 

2020: EPA Report to Congress. Office of Air and Radiation, Office of Policy, Washington, DC. March. 

Available on the Internet at <http://www.epa.gov/oar/sect812/feb11/fullreport.pdf>.

U.S. Environmental Protection Agency (U.S. EPA). 2012. Regulatory Impact Analysis for the Final Revisions to the 

National Ambient Air Quality Standards for Particulate Matter. EPA-452/R-12-003. Office of Air Quality 

Planning and Standards, Health and Environmental Impacts Division. December. Available at: 

<http://www.epa.gov/pm/2012/finalria.pdf>. 

U.S. Environmental Protection Agency (U.S. EPA). 2013a. Integrated Science Assessment of Ozone and Related 

Photochemical Oxidants (Final Report). U.S. Environmental Protection Agency, Washington, DC, EPA/600/R10/076F. Available at: <http://cfpub.epa.gov/ncea/isa/recordisplay.cfm?deid=247492#Download>. 

U.S. Environmental Protection Agency (U.S. EPA). 2013b. Technical Support Document: Estimating the Benefit 

per ton of Reducing PM2.5 Precursors from 17 sectors. Office of Air Quality Planning and Standards, Research 

Triangle Park, NC. February. Available at: 

<http://www.epa.gov/airquality/benmap/models/Source_Apportionment_BPT_TSD_1_31_13.pdf>.

U.S. Environmental Protection Agency (U.S. EPA). 2014. Health Risk and Exposure Assessment for Ozone Final 

Report. EPA-452/R-14-004a. Office of Air Quality Planning and Standards, Health and Environmental Impacts 

Division. August. Available at: < http://www.epa.gov/ttn/naaqs/standards/ozone/data/20140829healthrea.pdf>. 

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 1999. An SAB Advisory: The 

Clean Air Act Section 812 Prospective Study Health and Ecological Initial Studies. EPA-SAB-COUNCILADV-99-005, Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/1AFDEBB45C57F6E5852571930058C1A5/$File/coua9905.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2000. The Clean Air Act 

Amendments (CAAA) Section 812 Prospective Study of Costs and Benefits (1999): Advisory by the Advisory 

Council on Clean Air Compliance Analysis: Costs and Benefits of the CAAA. EPA-SAB-COUNCIL-ADV-00-

002. October. Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/E992F918DF32131A852571B9004571F7/$File/coua0002.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2001. REVIEW OF THE 

DRAFT ANALYTICAL PLAN FOR EPA’S SECOND PROSPECTIVE ANALYSIS - BENEFITS AND 

COSTS OF THE CLEAN AIR ACT 1990- 2020. EPA-SAB-COUNCIL-ADV-01-004, Available on the 

Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/572DC1989A3986CC8525718D006BAB8B/$File/councila01004.

pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2004a. Advisory on Plans for 

Health Effects Analysis in the Analytical Plan for EPA’s Second Prospective Analysis—Benefits and Costs of 

the Clean Air Act, 1990–2020. EPA-SAB-COUNCIL-ADV-04-002. March. Available on the Internet at 
5A-18

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/08E1155AD24F871C85256E5400433D5D/$File/counc

il_adv_04002.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2004b. Review of the Draft 

Analytical Plan for EPA’s Second Prospective Analysis—Benefits and Costs of the Clean Air Act, 1990-2020: 

An Advisory by the Advisory Council for Clean Air Compliance Analysis. EPA-SAB-COUNCIL-ADV-04-

004. May. Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/7CCBBFE15CD4C8B185256F17005E3079/$File/council_adv_04

004.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2004c. Advisory Council on 

Clean Air Compliance Analysis Response to Agency Request on Cessation Lag. EPA-COUNCIL-LTR-05-001. 

December. Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/0/39F44B098DB49F3C85257170005293E0/$File/council_ltr_05_

001.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2010. Review of EPA’s 

DRAFT Health Benefits of the Second Section 812 Prospective Study of the Clean Air Act. EPA-COUNCIL10-001. June. Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/9288428b8eeea4c885257242006935a3/72D4EFA39E48CDB2852

5774500738776/$File/EPA-COUNCIL-10-001-unsigned.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2011a. CASAC Review of 

Quantitative Health Risk Assessment for Particulate Matter—Second External Review Draft (February 2010). 

EPA-CASAC-10-008. Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/BC4F6E77B6385155852577070002F09F/$File/EPA-CASAC-10-

008-unsigned.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2011b. Review of the Final 

Integrated Report for the Second Section 812 Prospective Study of the Benefits and Costs of the Clean Air Act 

(August 2010). EPA-COUNCIL-11-001. Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/1E6218DE3BFF682E852577FB005D46F1/$File/EPA-COUNCIL11-001-unsigned.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (U.S. EPA-SAB). 2011c. Review of Valuing 

Mortality Risk Reductions for Environmental Policy: A White Paper (December 10, 2010). EPA-SAB-11-011. 

July. Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/298E1F50F844BC23852578DC0059A616/$File/EPA-SAB-11-

011-unsigned.pdf>.

Woods & Poole Economics, Inc. 2012. Complete Economic and Demographic Data Source (CEDDS). CD-ROM. 

Woods & Poole Economics, Inc. Washington, D.C.

World Health Organization (WHO). 2008. Part 1: Guidance Document on Characterizing and Communicating 

Uncertainty in Exposure Assessment, Harmonization Project Document No. 6. Published under joint 

sponsorship of the World Health Organization, the International Labour Organization and the United Nations 

Environment Programme. WHO Press: Geneva, Switzerland. Available on the Internet at 

<www.who.int/ipcs/methods/harmonization/areas/uncertainty%20.pdf>.