## 5A

- 1

APPENDIX 5A: COMPREHENSIVE CHARACTERIZATION OF UNCERTAINTY IN O 3 

## BENEFITS ANALYSIS

Overview As noted in Chapter 5, the benefits analysis relies on an array of data inputs

— including air quality modeling, health impact functions and valuation estimates among others

— which are themselves subject to uncertainty and may also in turn contribute to the o verall uncertainty in this analysis. The RIA employs a variety of analytic approaches designed to reduce the extent of the uncertainty and/or characterize the impact that uncertainty has on the final estimate. We strive to incorporate as many quantitative assessments of uncertainty as possible (e.g., Monte Carlo assessments, sensitivity analyses); however, there are some aspects we are only able to characterize qualitatively.

To more comprehensively and systematically address these uncertainties, including those we cannot quantify, we adapt the WHO uncertainty framework (WHO, 2008), which provides a means for systematically linking the characterization of uncertainty to the sophistication of the underlying health impact assessment. EPA has applied similar a pproaches in peer

- reviewed analyses of PM 2.5 

## - related impacts (U.S. EPA, 2010b, 2011, 2012) and O 3 

## - related impacts (U.S. EPA, 2014). EPA’s Science Advisory Board (SAB) has supported using a tabular format to qualitatively assess the uncertainties
inherent i n the quantification and monetization of health impacts, including identifying potential bias, potential magnitude, confidence in our approach, and the level of quantitative assessment of each uncertainty (U.S. EPA

- SAB, 1999, 2001, 2004 a , 2004b, 2011a, 201 1b). The assessments presented here are largely consistent with those previous peer

- reviewed assessments.

This appendix focuses on uncertainties inherent in the O 3 

## benefits estimates. For more information regarding the uncertainties inherent in the PM be nefits estimates, please see the 2012 PM NAAQS RIA (U.S. EPA, 2012).

5A.1 Description of Classifications Applied in the Uncertainty Characterization

Table 5A

- 1 catalogs the most significant sources of uncertainty in the O 3 

## benefits analysis and
then charac terizes four dimensions of that uncertainty (listed below). The first two dimensions focus on the nature of the uncertainty. The third and fourth dimensions focus on the extent to which the analytic approach chosen in the benefits analysis either minimizes the impact of the uncertainty or quantitatively characterizes its impact.

## 5A

- 2

1) The direction of the bias that a given uncertainty may introduce into the benefits assessment if not taken into account in the analysis approach;
2) The magnitude of the impact that u ncertainty is likely to have on the benefits estimate if not taken into account in the analysis approach;
3) The extent to which the analytic approach chosen is likely to minimize the impact of that uncertainty on the benefits estimate; and
4) The extent to whi ch EPA has been able to quantify the residual uncertainty after the preferred analytic approach has been incorporated into the benefits model.

5A.1.1 Direction of Bias

The “direction of bias” column in Table 5A

- 1 is an assessment of whether, if left unadd ressed, an uncertainty would likely lead to an underestimate or overestimate the total monetized benefits. In some cases we indicate that there are reasons why the bias might go either direction, depending upon the true nature of the underlying relationshi p. Where available, we base the classification of the “direction of bias” on the analysis in the Integrated Science Assessment for Ozone and Related
Photochemical Oxidants (hereafter, “O 3 

## ISA”) (U.S. EPA, 2013 a ). Additional sources of information
include a dvice from SAB and the National Academies of Science (NAS), as well as studies from the

peer

- reviewed literature. In some cases we indicate that there is not sufficient information to estimate whether the uncertainty would likely lead to under or overestim ation of benefits; these cases are identified as “unable to determine.”

5A.1.2 Magnitude of Impact The “magnitude of impact” column in Table 5A

- 1 is an assessment of how much plausible alternative assumptions about the underlying relationship about which we are uncertain could influence the overall monetary benefits. EPA has applied similar classifications in previous risk and benefit analyses (U.S. EPA, 2010b, 2011 , 2014), but we have slightly revised the category names and the cut

offs here. 1 

## The definit ions used here are provided below.

1 

### In The Benefits and Costs of the Clean Air Act from 1990 to 2020 (U.S. EPA, 2011 ), EPA applied a classification of “potentially major” if a plausible alternative assumption or approach could influence the overall monetary benefit estimate by 5% percent or more and “proba bly minor” if an alternative assumption or approach is likely to change the total benefit estimate by less than five percent. In the Quantitative Health Risk Assessment for Particulate Matter (U.S. EPA, 2010b), EPA applied classifications of “low” if the i mpact would not be expected to impact the interpretation of risk estimates in the context of the PM NAAQS review, “medium” if the impact had the potential to change the interpretation; and “high” if it was are likely to influence the interpretation of risk in the context of the PM NAAQS review.

## 5A

- 3

 High

— if the uncertainty associated with an assumption could influence the total monetized benefits by more than 25%.

 Medium

— if the uncertainty associated with an assumption could influence the total monetized benefits by 5% to 25%.

 Low

— if the uncertainty associated with an assumption could influence the total monetized benefits by less than 5%.

 For each uncertainty, we provide as much quantitative information as is available in the table to support the classification.

Al though many of the sources of uncertainty could affect both morbidity and mortality endpoints, because mortality benefits comprise over 94% of the monetized benefits that we are able to quantify in this analysis, uncertainties that affect the mortality est imate have the potential to have larger impacts on the total monetized benefits than uncertainties affecting only morbidity endpoints. One morbidity

- related uncertainty that could have a significant impact on benefits estimate is the
extent to which omitte d morbidity endpoints are included in the benefits analysis. Including additional morbidity endpoints that are currently not monetized would reduce the fraction of total benefits from mortality. Ultimately, the magnitude classification is determined by pro fessional judgment of EPA staff based on the results of available information, including other U.S. EPA assessments of uncertainty (U.S. EPA, 2010b, 2011)

Based on this assessment, the uncertainties which we classified as high or medium

- high impact

are: t he causal relationship between long

- term and short

- term ozone exposure and mortality, shape of
the concentration

- response function for both categories of ozone

- related mortality, and mortality valuation, specifically for long

- term exposure

- related mortalit y. The classification of these uncertainties as “high magnitude” is generally consistent with the results of EPA’s Influence Analysis (Mansfield et al., 2009 ), the Quantitative Health Risk Assessment for Particulate Matter (U.S. EPA, 2010b), and the Benefi ts and Costs of the Clean Air Act 1990 to 2020 (U.S. EPA, 2011).

5A .1.3 Confidence in Analytic Approach The “confidence in analytic approach” column of Table 5A

- 1 is an assessment of the scientific support for the analytic approach chosen (or the inherent assumption made) to account for the relationship about which we are uncertain. In other words, based on the available evidence, how certain are we that EPA’s selected approach is the most plausible of the potential alternatives. Similar

## 5A

- 4

classifications hav e been included in previous risk and benefits analyses (U.S. EPA, 2010b, 2011). 2 

## The three categories used to characterize the degree of confidence are:

 High

— the current evidence is plentiful and strongly supports the selected approach;

 Medium

— some eviden ce exists to support the selected approach, but data gaps are present; and

 Low

— limited data exists to support the selected approach. Ultimately, the degree of confidence in the analytic approach is EPA staff’s professional judgment based on the volume and consistency of supporting evidence, much of which has been
evaluated in the O 3 

## ISA (U.S. EPA, 2013a ) and EPA’s independent Science Advisory Board (SAB).
The O 3 

## ISA evaluated the entire body of scientific literature on O 3 

## science and was twice peer

reviewed by EPA’s Clean Air Scientific Advisory Committee (CASAC). In general, we regard a
conclusion in the O 3 

## ISA or specific advice from SAB as supporting a high degree of confidence in the selected approach.

Based on this assessment, we have low or low

- medium confidence in the evidence available to assess exposure error in epidemiology studies, morbidity valuation, baseline incidence projections for morbidity, and omitted morbidity endpoints. However, because these uncertainties have been classified as having a low or low

- medium impact on the magnitude of the benefits, further investment in improving the available evidence would not have a substantial impact on the total monetized benefits.

5 A .1.4 Uncertainty Quantification

The column of Table 5A

- 1 labeled “un certainty quantification” is an assessment of the extent to which we were able to use quantitative methods to characterize the residual uncertainty in the benefits analysis, after addressing it to the extent feasible in the analytic approach for this RIA. We categorize the level of quantification using the four tiers used in the WHO uncertainty framework (WHO, 2008). The WHO uncertainty framework is a well

- established approach to assess uncertainty in risk estimates that systematically links the characteriz ation of uncertainty to the sophistication of the health impact assessment. The advantage of using this framework is that it clearly highlights the level of uncertainty quantification applied in this assessment and the potential sources of uncertainty that require methods

2 

### We have applied the same classification as The Benefits and Costs of the Clean Air Act from 1990 to 2020 (U.S. EPA, 2011a) in this analysis. In the Quantitative Health Risk Assessment for Particulate Matter (U.S. E PA, 2010b), EPA assessed the degree of uncertainty (low, medium, or high) associated with the knowledge

- base (i.e., assessed how well we understand each source of uncertainty), but did not provide specific criteria for the classification.

## 5A

- 5

development in order to assess quantitatively. Specifically, EPA applied this framework in multiple risk and exposure assessments (U.S. EPA, 2010b, 2014), and it has been recommended in EPA guidance documents assessing air toxics

- related r isk and Superfund s ite risks (U.S. EPA, 2004 and 2001, respectively). Ultimately, the tier decision is professional judgment of EPA staff based on the availability of information for this assessment. The tiers used in this assessment are defined below.

 Tie r 0

— screening level, generic qualitative characterization.

 Tier 1

— Scenario

- specific qualitative characterization.

 Tier 2

— Scenario

- specific sensitivity analysis.

 Tier 3

— Scenario

- specific probabilistic assessment of individual and combined uncertainty.

Withi n the limits of the data, we strive to use a more sophisticated approaches (e.g., Tier 2 or 3) for characterizing uncertainties that have the largest magnitudes and could not be completely addressed through the analytic approach. The uncertainties for whic h we have conducted probabilistic (Tier 3) assessments in this analysis are mortality causality, the shape of the concentration

- response function, and mortality and morbidity valuation. For lower magnitude uncertainties, we include qualitative
discussions of the potential impact of uncertainty on risk results (WHO Tier 0/1) and/or completed sensitivity analyses assessing the potential impact of sources of uncertainty on risk results (WHO Tier 2).

5A.2 Organization of the Qualitative Uncertainty Table

Table 5A

- 1 is organized as follows. The uncertainties are grouped by category (i.e.,
concentration

- response function, valuation, population and baseline incidence, omitted benefits categories, and exposure changes). Within each category, the uncertainties are s orted by magnitude of impact (i.e., high to low) then by confidence in our approach (i.e., low to high). In the table, red (bold) text is used to indicate the uncertainties that likely have a high magnitude of impact on the total
benefits estimate. This or ganization highlights the uncertainty with the largest potential impact and the lowest confidence at the top of each category.

## 5A

- 6

Table 5A

- 1

. Summary of Qualitative Uncertainty for Key Modeling Elements in O 3 

## Benefits
Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification
Uncertainties Associated with Concentration

- Response Functions

Causal relationship between short

- term O 3 exposure and premature mortality Overestimate, if short

- term O 3

exposure does not have a causal relationship with premature mortality. High High Tier 1 (qualitative)

Mortality generally dominates monetized benefits, so small uncertainties could have large impacts on the total monetized benefits. Our approach is consistent with the O 3 Integrated Science Assessment (ISA), which determined that premature mortality has a likely causal relationship with short

ter m O 3 exposure based on the collective body of evidence (p. 6

- 264). In addition, the NAS recommended that EPA “should give little or no weight to the assumption that there is no causal association between estimated reductions in premature mortality and redu ced ozone exposure” (NRC, 2008). In 2010, the Health Effects Subcommittee of the Advisory Council on Clean Air Compliance Analysis, while reviewing EPA’s The Benefits and Costs of the Clean Air Act 1990 to 2020 (U.S. EPA, 2011), also confirmed the NAS reco mmendation to include O3 mortality benefits (U.S. EPA

SAB, 2010). Either Medium

- High Medium Tier 2 (sensitivity analysis)

## 5A

- 7

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Shape of the C

- R functions, particularly at low concentrations for short

- term ozone exposure

- related mortality The direction of bias that assuming

linear

- no threshold model or alternative model introduces depends upon the “true” functional from of the relationship and the specific assumptions and data in a particular analysis. For example, if the true function identifies a threshol d below which health effects do not occur, benefits may be overestimated if a substantial portion of those benefits were estimated to occur below that threshold. Alternately, if a substantial portion of the benefits occurred above that threshold, the benef its may be underestimated because an assumed linear no

threshold function may not reflect the steeper slope above that threshold to account for all health effects occurring above that threshold. The magnitude of this impact depends on the fraction of benef its occurring in areas with lower concentrations. Mortality generally dominates monetized benefits, so small uncertainties could have large impacts on total monetized benefits. The O 3 ISA did not find any evidence that supports a threshold in the relations hip between short

- term exposure to O 3 and mortality within the range of O 3 concentrations observed in the United States, and recent evidence suggests that the shape of the O 3 - mortality C

- R curve remains linear across the full range of O 3 concentrations (p . 6

- 257). Consistent with the O 3 ISA, we assume a log

- linear no

threshold model for the concentration

response functions for short

- term O 3 mortality. However, the ISA notes that there is less certainty in the shape of the C

- R function below 20 ppb due to t he low density of data in this range (p. 6

- 254

- 255). The comparison of short

- term mortality against the associated distribution of (ozone season

averaged) 8hr max ozone levels (see Appendix 5C, section 5C.2) suggests that the vast majority of predicted re ductions in mortality are associated with days having 8hr max values that fall within the range of increased confidence in specifying the nature of the mortality response (as identified in the Ozone ISA).

Causal relationship between long

- term O 3 exposure and Overestimate, if short

- term O 3

exposure does not have a causal relationship with premature mortality. High High Tier 1 (qualitative)

## 5A

- 8

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification premature respiratory mortality

The total dollar benefits of reducing ozone levels are comprised mostly of the value placed on reducing the risk of premature death, so small uncertainties could have large impacts on the total monetized benefits. While the Ozone ISA concludes that evidence is suggestive of a causal association between total mortality and long

- t erm ozone exposure (section 7.7.1), specifically with regard to respiratory health effects (including mortality), the ISA concludes that there is likely to be a causal association (section 7.2.8). Furthermore, in their review of the HREA completed for the O3 NAAQS review, the CASAC expressed their support for our plan to include a non

threshold

- based modeling of long

- term exposure

- related respiratory mortality in the core estimate. The CASAC also supported EPA’s plan to consider the potential for thresholds in the response (in the range of 40

- 60ppb) as a sensitivity analysis in the HREA (see section 5.6.3.1). This same approach (with regard to the core approach and sensitivity analysis for long

- term exposure

- related morality) has been adopted for the RIA (se e below). Shape of the C

- R functions, particularly at low concentrations for long

- term ozone exposure

- related respiratory mortality Either Medium

- High Medium Tier 2 (sensitivity analysis)

## 5A

- 9

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

The direction of bias that assuming linear

- no threshold model or alternative model introduces depends upon the “true” functional from of the relationship and the specific assumptions and data in a particular analysis. For example, if the true function identifies a threshold below which health effects do not occur, be nefits may be overestimated if a substantial portion of those benefits were estimated to occur below that threshold. Alternately, if a substantial portion of the benefits occurred above that threshold, the benefits may be underestimated because an assumed linear no

threshold function may not reflect the steeper slope above that threshold to account for all health effects occurring above that threshold. The magnitude of this impact depends on the fraction of benefits occurring in areas with lower concentrati ons. Note, that due to limitations in our ability to predict the lag structure associated with reductions in long

- term ozone exposure

- related mortality, we are not including dollar benefits associated with this endpoint in the core benefits analysis (which significantly reduces the role of this source of uncertainty in impact core benefit estimates). In their memo (see Sasser 2014) clarifying the results of their study (Jerrett et al., 2009) regarding long

- term ozone exposure

- related respiratory mortality, the study authors note that in terms of goodness of fit, long

- term health risk models including O3 clearly performed better than models without O3, indicating the improved predictions of respiratory mortality when O3 is included. In the article proper, the authors state that, “There was limited evidence that a threshold model specification improved model fit as compared with a nonthreshold linear mode...”. Furthermore, in the memo referenced above, the authors conclude that considerable caution should be exer cised in using any specific threshold, particularly when the more stringent statistical test indicates there is no significantly improved prediction. The CASAC was supportive of the approach we used in the HERA of using

a non

- threshold C

- R function based o n this study to generate core estimates and consider the impact of potential thresholds (ranging from 40

- 60ppb) as a sensitivity analysis. As part of the sensitivity analyses completed for the RIA, we did examine the potential impact of thresholds (from 4 0 to 60ppb) in the response function for long

term exposure

- related morality (see Appendix 5B, section 5B.1). That analysis suggested that thresholds between 55 and 60ppb would have a substantial impact on overall modeled benefits, while thresholds above below 50ppb would have a minor impact on predicted benefits (see Appendix 5B, Table 5B

- 1).

Exposure error in epidemiology studies Underestimate (generally) Medium Low

- Medium Tier 1 (qualitative)

The O 3 ISA states that exposure measurement error can also be an important contributor to uncertainty in effect estimates associated with both short

- term and long

- term studies (p. 1xii). Together with other factors (e.g., low data density), exposure error

can sm ooth the C

- R functions and obscure potential thresholds (p. lxix). In addition, the O 3 ISA states that exposure error can bias effect estimates toward or away from the null and widen confidence intervals (p. lxii). Recent analyses reported in Krewski et a l. (2009) demonstrate the potentially significant effect that this source of uncertainty can have on effect estimates. These analyses also illustrate the complexity and site

- specific nature of this source of uncertainty. Although this underestimation is we ll documented, including in the O 3 ISA, the SAB has not suggested an approach to adjust for this bias. (No quantitative method available)

Unknown Medium Medium Tier 1 (qualitative)

## 5A

- 10

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Adjustment of risk coefficients to 8

- hour maximum Several of the mortality

epidemiological studies were reported for a 24

- hour average or 1

- hour maximum ozone level. These metrics are not the most relevant to characterizing population

- level exposure. Thus, we have converted ozone mortality health impact functions that u se these metric to maximum 8

- hour average ozone concentration using standard conversion functions. The conversion ratios are based on observed relationships between the 24

- hour average and 8

- hour maximum in the underlying studies. This conversion also does not affect the relative magnitude of the health impact function. However, we do note that the pattern of 8hr max concentrations for a particular location over an ozone season could differ from the pattern of 1hr max or 24 hour average metrics for that sam e location. Consequently, estimates of incidence reductions (and associated dollar benefits) could differ for a particular location depending on the metric used for risk calculations. 3 This practice is consistent both with the available exposure modeling and with the form of the current ozone standard. However, in some cases, these conversions were not specific to the ozone “warm” season which was the period used in the benefits analysis, which introduces additional uncertainty due to the use of effect est imates based on a mixture of warm season and all year data in the epidemiological studies. (No quantitative method available)

Confounding by individual risk factors, other than SES

— e.g., smoking, or ecologic factors, which represent the neighborhood, such as unemployment Either, depending on the factor and

study Medium Medium Tier 2 (sensitivity analysis) Individual, social, economic, and demographic covariates can bias the relationship between particulate air pollution and mortality, particularly in cohort studies that rely on regional air pollution levels. Because mortality dominates monetized benefits, even a small amount of confounding could have medium impacts on total monetized benefits. To minimize confounding effects, we use risk coefficients t hat control for individual risk factors to the extent practical. (Quantitative methods available but not assessed in this analysis.)

Confounding and effect modification by co

- pollutants Either, depending upon the

pollutant. Medium Medium Tier 1 (qualitative)

Disentangling the health responses of combustion

- related pollutants (i.e., PM, SOx, NOx, ozone, and CO is a challenge. The O 3 ISA defines a confounder as the true cause of the association observed between the exposure and the outcome; in contrast, an effect modifier changes the magnitude of the association between the exposure and the outcome (p. lxii). Because this uncertainty could affect mortality and because mortality generally dominates monetized benefits, even a small uncertainties could have medium impacts on total monetized benefits. The O 3 ISA states that there is high confidence that unmeasured confounders are not producing the findings when multiple studies are conducted in various settings using different subjects or exposures, such as multi

- city studies (p. lxi). When modeling effects o f pollutants jointly (e.g., PM and O 3 ), we apply multi

pollutant effect estimates when those estimates are available to avoid double

counting and satisfy other selection criteria. In addition, we apply multi

- city effect estimates when available. (No quanti tative method available)

3 

### We note that if metric adjustment ratios were derived for individual point locations (reflecting the relationship between metrics for each lo cation) and those metrics were used to generated adjusted C

- R functions for each location, then some of this potential uncertainty could be reduced, however available data does not support the application of highly location

- specific adjustment ratios (with cross

- city or national adjustment ratios being typically used instead).

## 5A

- 11

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Application of C

- R relationships only to the original study population Underestimate Low High Tier 2 (sensitivity analysis)

Estimating health effects for only to the original study population may underestimate the whole population benefits of reductions in pollutant exposures. Mortality generally dominates monetized benefits, so further age range expansions for morbidity endpoints would have a small impact on total monetized benefits. Following advice from the SAB (U.S. E PA

SAB, 2004a, pg. 7) and NAS (NRC, 2002, pg. 114), we expanded the age range for childhood asthma exacerbations beyond the original study population to ages 6

- 18. (Quantitative methods available but not assessed in this analysis.)

Uncertainties Associated with Economic Valuation

Mortality Risk Valuation / Value

of

- a

- Statistical

- Life (VSL) Unknown High Medium Tier 3 (probabilistic)

Some studies suggest that EPA’s mortality valuation is too high, while other studies suggest that it is too low. Differences in age, income, risk aversion, altruism, nature of risk (e.g., cancer), and study design could lead to higher or lower estimates of mortality valuation. Mortality generally dominates monetized benefits, so moderate uncertainties could have a la rge effect on total monetized benefits. The VSL used by EPA is based on 26 labor market and stated preference studies published between 1974 and 1991. EPA is in the process of reviewing this estimate and will issue revised guidance based on the most up

- to

- date literature and recommendations from the SAB

- EEAC in the near future (U.S. EPA, 2010a, U.S. EPA

- SAB, 2011c). Assessed uncertainty in mortality valuation using a Weibull distribution.

Cessation lag structure for long term O 3 mortality Underestimate Medium

- High Low Tier 2 (sensitivity analysis)

Jerrett et al. (2009) notes that, “Allowing for a 10

- year period of exposure to ozone (5 years of follow

- up and 5 years before the follow

- up period) did not appreciably alter the risk estimates...” We acknowledge substantial uncertainty associated with specifying the lag for long

term respiratory mortality. Although the cessation lag does not affect the number of premature deaths attributable to long

- term O 3 exposure, it affects the timing of those deat hs and thus the discounted monetized benefits. Mortality generally dominates monetized benefits, so moderate uncertainties could have a large effect on total monetized benefits. As discussed in section 5.6.4.1, in presenting dollar benefit estimates as par t of the sensitivity analysis (presenting dollar benefits for long

- term ozone

- related morality), we include both an assumption of zero lag and a lag structure matching that used for the core PM 2.5 estimate (the SAB 20 year segmented lag). Inclusion of the zero lag reflects consideration for the possibility that the long

- term respiratory mortality estimate captures primarily, an accumulation of short term mortality effects across the ozone season. The use of the 20 year segmented lag reflects consideration f or advice provided by the HES (USEPA

SAB , 2010). As shown in sensitivity analysis results presented Tables Appendix 5

- A, the use of the 20 year segmented lag results in a 10

- 20% reduction in the total dollar benefit (using a 3% and 7% discount rate, respec tively) relative to the alternative approach of applying no lag (i.e., assuming all of the mortality reductions occur in the same year).

Either Medium Medium Tier 2 (sensitivity analysis)

## 5A

- 12

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Income growth adjustments Income growth increases

willingness

- to

- pay (WTP) valuation estimates, including mortality, over time. From 1997 to 2010, personal income and GDP growth have begun to diverge. If this trend continues, the assumption that per capita GDP growth is a reasonable proxy for income growt h may lead to an overstatement of benefits. (IEc, 2012). Income growth from 1990 to 2020 increases mortality valuation by 20%. Alternate estimates for this adjustment vary by 20% (IEc, 2012). Because we do not adjust for income growth over the 20

- year cess ation lag, this approach could also underestimate the benefits for the later years of the lag. Consistent with SAB recommendations (U.S. EPA,

- SAB, 2000, pg. 16), we adjust WTP for income growth. Difficult to forecast future income growth. However, in the absence of readily available income data projections, per capita GDP is the best available option. As shown in Appendix 5

- A, the use of alternate income growth adjustments would change the monetized benefits by +33% to −14%.

Morbidity valuation Underestimate Low Low Tier 3 (probabilistic), where available Morbidity benefits such as hospital admissions are calculated using cost

- of

- illness (COI) estimates, which are generally half the willingness

- to

- pay to avoid the illness (Alberini and Krupnick, 2000). In addition, the morbidity costs do not reflect physiological responses or sequelae events, such as increased susceptibility for future morbidity. Even if we doubled the monetized valuation of morbidity endpoints using COI valuation that are currently included in the RIA, the change would still be less than 5% of the monetized benefits. It is unknown how much including sequelae events could increase morbidity valuation. Although the COI estimates for hospitalizations reflect recent data, w e have not yet updated other COI estimates such as for school loss days. The SAB concluded that COI estimates could be used as placeholders where WTP estimates are unavailable, but it is reasonable to presume that this strategy typically understates WTP

va lues (U.S. EPA

- SAB, 2004b, pg. 3). Assessed uncertainty in morbidity valuation using distributions specified in the underlying literature, where available (see Table 5

- 10).

Uncertainties Associated with Baseline Incidence and Population Projections

Population estimates and projections Either Low

– Medium Medium Tier 1 (qualitative)

The monetized benefits would change in the same direction as the

over

- or underestimate in population projections in areas where exposure changes. Monetized benefits are substantially affected by population density. Comparisons using historical census data show that population projections are ±5% nationally, but projection accuracy can vary by locality. Historical error for Woods & Poole’s population projections has been ± 8.1% for county

- level projections and ±4.1% for states (Woods and Poole, 2012). The magnitude of impact on total monetized benefits depends on the specific location where PM is reduced. We use population projections for 5

- year increments for 304 race/ethni city/gender/age groups (Woods and Poole, 2012) at Census blocks. Population forecasting is well

- established but projections of future migration due to possible catastrophic events are not considered. In addition, projections at the small spatial scales use d in this analysis are inherently more uncertain than projections at the county

- or state

- level. (No quantitative method available)

Unknown Low

- Medium Medium Tier 1 (qualitative)

## 5A

- 13

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Uncertainty in projecting baseline incidence rates for mortality Because the mortality rate

projections for future years reflect changes in mortality patterns as well as population growth, the projections are unlikely to be biased. Because mortality generally dominates monetized benefits, small uncertainties could have medium i mpacts on total monetized benefits. The county

- level baseline mortality rates reflect recent databases (i.e., 2004

– 2006 data) and projected for 5

- year increments for multiple age groups. This database is generally considered to have relatively low uncertai nty (CDC Wonder, 2008). The projections account for both spatial and temporal changes in the population. (No quantitative method available)

Uncertainty in projecting baseline incidence rates and prevalence rates for morbidity Either, depending on the health

endpoint Low Low

- Medium Tier 1 (qualitative)

Morbidity baseline incidence is available for current year only (i.e., no projections available). Assuming current year levels can bias the benefits for a specific endpoint if the data has clear trends over time. Specifically, asthma prevalence rates have increased substantially over the past few years while hospital admissions have decreased substantially. The magnitude varies with the health endpoint, but the overall impact on the total benefits estima te from these morbidity endpoints is likely to be low. We do not have a method to project future baseline morbidity rates, thus we assume current year levels will continue. While we try to update the baseline incidence and prevalence rates as frequently as practicable, this does not continue trends into the future. Some endpoints such as hospitalizations and ER visits have more recent data (i.e., 2007) stratified by age and geographic location. Other endpoints, such as respiratory symptoms reflect a nationa l average. Asthma prevalence rates reflect recent increases in baseline asthma rates (i.e., 2008). (No quantitative method available)

Uncertainties Associated with Omitted Benefits Categories

Unquantified O 3 health benefit categories, such as worker productivity and long

- term mortality Underestimate Medium

- High Low Tier 2 (sensitivity analysis)

EPA has not included monetized estimates of these benefits categories in the core benefits estimate. Although the potential magnitude is unknown, including all of the additional endpoints associated with O 3 exposure that are currently not monetized could increase the total benefits by a large amount. Current data and methods are insufficient to value national quantita tive estimates of these health effects. The O 3 ISA determined that respiratory effects (including mortality) are causally associated with long

- term O 3 exposure (p. 2

– 17). The O 3 ISA also determined that outdoor workers have an increased risk of O 3 - related health effects (p. 1

- 15), and that studies on outdoor workers show consistent evidence that short

- term increases in ambient O 3 exposure can decrease lung function in healthy adults (p.6

- 38). Additional studies link short

- short term O 3 exposure to reduced productivity in outdoor workers (Graf Zivin and Neidell, 2013; Crocker and Horst, 1981). We include sensitivity analyses reflecting long

- term mortality which shows that this endpoint could add substantially to the total core benefits range (see Ta bles 5

- 20 and 5

- 27). We are still considering options for updating the worker productivity analysis and including it as a sensitivity analyses.

Uncertainties Associated with Estimated Exposure Changes

Unknown Unknown Low Tier 1 (qualitative)

## 5A

- 14

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification

Spatial matching of air quality estimates from epidemiology studies to air quality estimates from air quality modeling Epidemiology studies often assume

one air quality concentration is representative of an entire urban area when calculating hazard ratios, while benefits are calculated using air quality modeling conducted at 12 km spatial resolution. This spatial mismatch could introduce uncertainty. We have not controlled for this potential bias, and the SAB has not suggested an approach to adjust for thi s bias. (No quantitative method available)

Uncertainties Associated with the Dollar

- per - ton Approach Used in Modeling PM 2.5 Co

- benefits

Derivation of dollar

per

- ton estimates for PM 2.5 Unknown Unknown Medium Tier 1 (qualitative)

In the analysis used to generate the dollar

- per

- ton values we assume that all fine particles, regardless of their chemical composition, are equally potent in causing premature mortality. However, the scientific evidence is not yet sufficient to allow differentiation of effect

es timates by particle type. We also assume that the health impact function for fine particles is linear down to the lowest air quality levels modeled in this analysis. Thus, the estimates include health benefits from reducing fine particles in areas with var ied concentrations of PM 2.5 , including regions that are in attainment with fine particle standard. While there are concerns regarding the assumption of a uniform PM 2.5 toxicity across sectors and a linear C

- R function all the way down to zero, these sourc es of uncertainty impact benefits modeling for PM 2.5 in general and have been discussed as part of earlier RIAs (see section 5.7.2, of the final PM RIA, U.S. EPA. 2012). We do recognize that, as discussed below (and on pp. 24

- 25 of US. EPA, 2013), there is increased uncertainty when dollar

- per

- ton benefits are applied outside of the specific scenario used in their derivation. (No quantitative method available)

Application of Unknown Unknown Medium

## 5A

- 15

Potential Source of Uncertainty Direction of Potential Bias Magnitude of Impact on Monetized Benefits Confidence in Analytical Approach Uncertainty Quantification
dollar

- per

- ton estimates in the current Ozone NAAQS review As discussed in section 5.4.4, we

used a method to calculate the regional benefit

- per

- ton estimates that is a slightly modified version of the national benefit

- per

- ton estimates described in the TSD: Estimating the Benefit per Ton of Reducing PM 2.5 Precursors from 17 Sectors (U.S. EPA, 2013b ). The national estimates were derived using the approach published in Fann et al. (2012c), but they have since been updated to reflect the epidemiology studies and Census population data first applied in the fin al PM NAAQS RIA (U.S. EPA, 2012 ). These dollar

- per

- ton estimates were applied to sector

specific NOx emissions reductions modeled as part of the current ozone NAAQS review. While we acknowledge uncertainty associated with applying dollar

- per

- ton estima tes in the context of this ozone NAAQS review (and outside of the scenario in which they were derived), we are not in a position to characterize the magnitude of direction of any bias that might result from that application. All benefit

- per

- ton have inhere nt limitations, including that the estimates reflect the geographic distribution of the modeled sector emissions, which may not match the emission reductions anticipated by the proposed guidelines, and they may not reflect local variability in population d ensity, meteorology, exposure, baseline health incidence rates, or other local factors for any specific locations reflected in benefits modeling. However, the fact that we are modeling regional/national benefits rather than attempting a more spatially refi ned application of the dollar

- per

- ton values does reduce uncertainty in the benefits estimates that are generated. (No quantitative method available)

## 5A

- 16

5A .3 References
Alberini, Anna and Alan Krupnick. 2000. “Cost

- of

- Illness and Willingness

- to

- Pay Estimates of the Benefits of Improved Air Quality: Evidence from Taiwan.” Land Economics Vol. 76, No. 1 (Feb), pp. 37

- 53.
Centers for Disease Control and Prevention (CDC). 2008. National Center for Health Statistics. National Health Inte rview Survey, 1999

- 2008.
Fann, N.; K.R. Baker and C.M. Fulcher. 2012. “Characterizing the PM 2.5 

### - related health benefits of emission reductions for 17 industrial, area and mobile emission sectors across the U.S.” Environment International 49 41

– 151.

Graff Zivin, J., Neidell, M. (2012). “The impact of pollution on worker productivity.” American Economic Review , 102, 3652

- 3673.
Industrial Economics, Incorporated (IEc). 2012. Updating BenMAP Income Elasticity Estimates

— Literature Review. Memo to Neal Fann. Mar ch. Available on the Internet at <http://www.epa.gov/ttn/ecas/regdata/Benefits/Income%20Elasticity%20Update_Recommendations_noAppD.p df>
Jerrett M, Burnett RT, Pope CA, III, et al. 2009. “Long

- Term Ozone Exposure and Mortality.” New England Journal of Medic ine 360:1085

- 95.
Krewski D, Jerrett M, Burnett RT, Ma R, Hughes E, Shi, Y, et al. 2009. “ Extended follow

- up and spatial analysis of the American Cancer Society study linking particulate air pollution and mortality. ” HEI Research Report, 140, Health Effects Institute, Boston, MA.
Mansfield, Carol; Paramita Sinha; Max Henrion. 2009. Influence Analysis in Support of Characterizing Uncertainty in Human Health Benefits Analysis: Final Report. Prepared for U.S. EPA, Office of Air Quality Planning and Standards. N ovember. Available on the internet at <http://www.epa.gov/ttn/ecas/regdata/Benefits/influence_analysis_final_report_psg.pdf>.
National Research Council (NRC). 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington, DC : The National Academies Press. Washington, DC.
National Research Council (NRC). 2008. Estimating Mortality Risk Reduction and Economic Benefits from Controlling Ozone Air Pollution . National Academies Press. Washington, DC.
Sasser, E. 2014. Response to Comments Regarding the Potential Use of a Threshold Model in Estimating the Mortality Risks from Long

- term Exposure to Ozone in the Health Risk and Exposure Assessment for Ozone, Second External Review Draft. Memorandum to Holly Stallworth, Designated Fede ral Officer, Clean Air Scientific Advisory Committee from EPA/OAQPS Health and Environmental Impacts Division.
Schwartz J, Coull B, Laden F. 2008. “The Effect of Dose and Timing of Dose on the Association between Airborne Particles and Survival.” Environme ntal Health Perspectives 116: 64

- 69.
U.S. Environmental Protection Agency (U.S. EPA). 200 1

. Risk assessment guidance for Superfund. Vol. III, Part A. Process for conducting probabilistic risk assessment (RAGS 3A). Washington, DC, United States Environmental Protection Agency (EPA 540

- R

- 02

- 002; OSWER 9285.7

- 45; PB2002 963302. Available on the Inte rnet at <http://www.epa.gov/oswer/riskassessment/rags3adt/index.htm>.
U.S. Environmental Protection Agency (U.S. EPA). 200 4

. EPA’s risk assessment process for air toxics: History and overview. In: Air toxics risk assessment reference library. Vol. 1. Techn ical resource manual. Washington, DC,

## 5A

- 17

United States Environmental Protection Agency, pp. 3

- 1

– 3

- 30 (EPA

- 453

- K

- 04

- 001A. Available on the Internet at <http://www.epa.gov/ttn/fera/data/risk/vol_1/chapter_03.pdf>.
U.S. Environmental Protection Agency (U.S. EPA) . 2010a. Valuing Mortality Risk Reductions for Environmental Policy: A White Paper (SAB Review Draft). December. Available on the Internet at <http://yosemite.epa.gov/ee/epa/eerm.nsf/vwAN/EE

- 0563

- 1.pdf/$file/EE

- 0563

- 1.pdf>.
U.S. Environmental Protection Ag ency (U.S. EPA). 2010b. Quantitative Health Risk Assessment for Particulate Matter

— Final Report. EPA

- 452/R

- 10

- 005. Office of Air Quality Planning and Standards, Research Triangle Park, NC. September. Available on the Internet at <http://www.epa.gov/ttnnaaq s/standards/pm/data/PM_RA_FINAL_June_2010.pdf>.
U.S. Environmental Protection Agency (U.S. EPA). 2011a. The Benefits and Costs of the Clean Air Act 1990 to 2020: EPA Report to Congress. Office of Air and Radiation, Office of Policy, Washington, DC. March. Available on the Internet at <http://www.epa.gov/oar/sect812/feb11/fullreport.pdf>.
U.S. Environmental Protection Agency (U.S. EPA). 2012. Regulatory Impact Analysis for the Final Revisions to the National Ambient Air Quality Standards for Particulate Matt er. EPA

- 452/R

- 12

- 003. Office of Air Quality Planning and Standards, Health and Environmental Impacts Division. December. Available at: <http://www.epa.gov/pm/2012/finalria.pdf>.
U.S. Environmental Protection Agency (U.S. EPA). 2013 a . Integrated Science As sessment of Ozone and Related Photochemical Oxidants (Final Report). U.S. Environmental Protection Agency, Washington, DC, EPA/600/R

10/076F. Available at: <http://cfpub.epa.gov/ncea/isa/recordisplay.cfm?deid=247492#Download>.

U.S. Environmental Protectio n Agency (U.S. EPA). 2013b. Technical Support Document: Estimating the Benefit per ton of Reducing PM2.5 Precursors from 17 sectors. Office of Air Quality Planning and Standards, Research Triangle Park, NC. February. Available at: <http://www.epa.gov/airqu ality/benmap/models/Source_Apportionment_BPT_TSD_1_31_13.pdf>.
U.S. Environmental Protection Agency (U.S. EPA). 2014. Health Risk and Exposure Assessment for Ozone Final Report. EPA

- 452/R

- 14

- 004a. Office of Air Quality Planning and Standards, Health and En vironmental Impacts Division. August. Available at: < http://www.epa.gov/ttn/naaqs/standards/ozone/data/20140829healthrea.pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S. EPA

- SAB). 1999. An SAB Advisory: The Clean Air Act Section 81 2 Prospective Study Health and Ecological Initial Studies. EPA

- SAB

- COUNCIL

ADV

- 99

- 005, Available on the Internet at <http://yosemite.epa.gov/sab/sabproduct.nsf/1AFDEBB45C57F6E5852571930058C1A5/$File/coua9905.pdf>.
U.S. Environmental Protection Agency

— Scien ce Advisory Board (U.S. EPA

- SAB). 2000

. The Clean Air Act Amendments (CAAA) Section 812 Prospective Study of Costs and Benefits (1999): Advisory by the Advisory Council on Clean Air Compliance Analysis: Costs and Benefits of the CAAA. EPA

- SAB

- COUNCIL

- ADV

- 0 0

- 002. October. Available on the Internet at <http://yosemite.epa.gov/sab/sabproduct.nsf/E992F918DF32131A852571B9004571F7/$File/coua0002.pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S. EPA

- SAB). 2001. REVIEW OF THE DRAFT ANALYTICAL PLAN FOR EPA’S SECOND PROSPECTIVE ANALYSIS

- BENEFITS AND COSTS OF THE CLEAN AIR ACT 1990

- 2020. EPA

- SAB

- COUNCIL

- ADV

- 01

- 004, Available on the Internet at <http://yosemite.epa.gov/sab/sabproduct.nsf/572DC1989A3986CC8525718D006BAB8B/$File/councila01004. pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S. EPA

- SAB). 2004 a . Advisory on Plans for Health Effects Analysis in the Analytical Plan for EPA ’ s Second Prospective Analysis

— Benefits and Costs of the Clean Air Act, 1990

– 2020 . EPA

- SAB

- COUNC IL

- ADV

- 04

- 002

. March. Available on the Internet at

## 5A

- 18

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/08E1155AD24F871C85256E5400433D5D/$File/counc il_adv_04002.pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S. EPA

- SAB). 2004 b

. Review of the Draft Analytical Plan for EPA’s Second Prospective Analysis

— Benefits and Costs of the Clean Air Act, 1990

- 2020: An Advisory by the Advisory Council for Clean Air Compliance Analysis . EPA

- SAB

- COUNCIL

- ADV

- 04

- 00 4. May

. Available on the Internet at <http://yo semite.epa.gov/sab/sabproduct.nsf/7CCBBFE15CD4C8B185256F17005E3079/$File/council_adv_04 004.pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S. EPA

- SAB). 2004 c

. Advisory Council on Clean Air Compliance Analysis Response to Agency Request on Cessation Lag. EPA

- COUNCIL

- LTR

- 05

- 001. December. Available on the Internet at <http://yosemite.epa.gov/sab/sabproduct.nsf/0/39F44B098DB49F3C85257170005293E0/$File/council_ltr_05_ 001.pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S . EPA

- SAB). 2010. Review of EPA’s DRAFT Health Benefits of the Second Section 812 Prospective Study of the Clean Air Act. EPA

- COUNCIL

10

- 001. June. Available on the Internet at <http://yosemite.epa.gov/sab/sabproduct.nsf/9288428b8eeea4c885257242006935a3/72 D4EFA39E48CDB2852 5774500738776/$File/EPA

- COUNCIL

- 10

- 001

- unsigned.pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S. EPA

- SAB). 2011a. CASAC Review of Quantitative Health Risk Assessment for Particulate Matter

— Second External Review Draf t (February 2010). EPA

- CASAC

- 10

- 008. Available on the Internet at <http://yosemite.epa.gov/sab/sabproduct.nsf/BC4F6E77B6385155852577070002F09F/$File/EPA

- CASAC

- 10

- 008

- unsigned.pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S. EPA

- SAB). 2011b. Review of the Final Integrated Report for the Second Section 812 Prospective Study of the Benefits and Costs of the Clean Air Act (August 2010). EPA

- COUNCIL

- 11

- 001. Available on the Internet at <http://yosemite.epa.gov/sab/sabproduct.nsf/1E6218DE3B FF682E852577FB005D46F1/$File/EPA

- COUNCIL

11

- 001

- unsigned.pdf>.
U.S. Environmental Protection Agency

— Science Advisory Board (U.S. EPA

- SAB). 2011c. Review of Valuing Mortality Risk Reductions for Environmental Policy: A White Paper (December 10, 2010). EPA

- SAB

- 11

- 011. July. Available on the Internet at <http://yosemite.epa.gov/sab/sabproduct.nsf/298E1F50F844BC23852578DC0059A616/$File/EPA

- SAB

- 1 1

- 011

- unsigned.pdf>.
Woods & Poole Economics, Inc. 2012. Complete Economic and Demographic Data Source (CEDDS). CD

- ROM. Woods & Poole Economics, Inc. Washington, D.C.
World Health Organization (WHO). 2008. Part 1: Guidance Document on Characterizing and Communicating Uncertainty in Exposure Assessment, Harmonization Project Document No. 6. Published under joint sponsorship of the World Health Organization, the Internati onal Labour Organization and the United Nations Environment Programme. WHO Press: Geneva, Switzerland. Available on the Internet at <www.who.int/ipcs/methods/harmonization/areas/uncertainty%20.pdf>.