5B-1

APPENDIX 5B: ADDITIONAL SENSITIVITY ANALYSES RELATED TO THE 

OZONE HEALTH BENEFITS ANALYSIS

Overview

The benefits analysis presented in Chapter 5 of this RIA is based on our current 

interpretation of the scientific and economic literature. That interpretation requires judgments 

regarding the best available data, models, and analytical methodologies and the assumptions that 

are most appropriate to adopt in the face of important uncertainties. The majority of the 

analytical assumptions used to develop the main estimates of benefits have been reviewed and 

supported by EPA’s independent Science Advisory Board (SAB). Both EPA and the SAB 

recognize that data and modeling limitations as well as simplifying assumptions can introduce 

significant uncertainty into the estimates of benefits and that alternative choices exist for some 

inputs to the analysis, such as the concentration-response functions for mortality.

This appendix assesses the sensitivity of the core benefits to: (a) the potential impact of 

thresholds in long-term ozone exposure-related mortality in incidence and benefits estimates 

(section 5B.2), (b) alternative response functions developed through expert elicitation (EE) for 

long-term PM2.5 exposure-related mortality (section 5B.3), and (c) alternative assumptions 

regarding income elasticity on benefits derived using willingness-to-pay (WTP) functions 

(section 5B.3).

For the core analysis, we estimated incidence and dollar benefits for two scenarios: 2025 

and post-2025 (see section 5.2 and 5.4.3). However, in conducting these sensitivity analyses, we 

used the 2025 scenario as the basis for making our calculations, since sensitivity analysis 

findings for this scenario would generally hold for the post-2025 scenario.

In addition to the three sensitivity analyses covered in this appendix, we also included 

two sensitivity analyses which are covered in detail in the body of the chapter. The first of these 

are estimates of dollar benefits associated with mortality resulting from long-term exposure to 

ozone. As discussed in section 5.2, while we felt that we had sufficient confidence to include 

incidence estimates associated with long-term ozone exposure (based on effect estimates 

obtained from Jerrett et al., 2009) in the core analysis, limitations in our ability to specify an 
5B-2

appropriate lag for reductions in this endpoint meant that we could not include dollar benefit 

estimates in the core analysis. Instead, we have included those values as sensitivity analyses, 

including consideration of alternative lag structures (they are presented in Tables 5-20 and 5-27, 

respectively for the 2025 and post-2025 scenarios).1 The second of the sensitivity analyses 

already covered in the chapter address alternative models for short-term ozone exposure-related 

mortality. As discussed in section 5.6.3.1, in addition to the two epidemiological studies (Smith 

et al., 2009 and Zanobetti and Schwartz 2008) which provide effect estimates for the core 

benefits estimates, we have also considered seven additional epidemiological studies as 

sensitivity analyses. These additional studies including a mix of multi-city and meta-analysis 

designs. Results of this sensitivity analysis are presented in Tables 5-19 and 5-20 (incidence and 

benefits, respectively for the 2025 scenario) and Tables 5-26 and 5-27 (incidence and benefits, 

respectively for the post-2025 scenario).

5B.1 Threshold Sensitivity Analysis for Premature Mortality Incidence and Benefits from 

Long-term Exposure to O3

In estimating long-term ozone mortality, we employed a continuous non-threshold C-R 

function relating O3 exposure to premature death. However, as discussed in section 5.6.3.1, there 

is uncertainty regarding the potential existence and location of a threshold in the C-R function 

relating mortality and long-term O3 concentrations. Thus, we have included a sensitivity analysis 

exploring the impact of potential thresholds in the C-R relationship on estimates of long-term 

exposure-related mortality that were evaluated in Jerrett et al. (2009), consistent with advice 

from CASAC (Frey, 2014). 

In their memo clarifying the results of their study (Sasser, 2014), the authors note that in 

terms of goodness of fit, long-term health risk models including O3 clearly performed better than 

models without O3, indicating the improved predictions of respiratory mortality when O3 is 

included. In exploring different functional forms, the authors report that the model including a 

threshold at 56 ppb had the lowest log-likelihood value of all models evaluated (i.e., linear 

models and models including thresholds ranging from 40-60 ppb), and thus provided the best 

 

1 The sensitivity analysis-related benefits estimates presented in these tables include (a) benefits estimates reflecting 

application of a zero lag model and (b) estimates reflecting application of the same 20-year segmented lag used in 

modeling benefits for PM2.5, together with application of a 3% and 7% discount rate). See sections 5.6.3 and 5.6.4 

for additional discussion of lags in relation to long-term ozone-related mortality.
5B-3

overall statistical fit to the data. However, they also note that it is not clear whether the 56 ppb 

threshold model is a better predictor of respiratory mortality than when using a linear (nothreshold) model for this dataset. Using one statistical test, the model with a threshold at 56 ppb 

was determined to be statistically superior to the linear model. Using another, more stringent test, 

none of the threshold models considered were statistically superior to the linear model. Under the 

less stringent test, although the threshold model produces a statistically superior prediction than 

the linear model, there is uncertainty about the specific location of the threshold, if one exists. 

This is because the confidence intervals on the model predictions indicate that a threshold could 

exist anywhere from 0 to 60 ppb. The authors conclude that considerable caution should be 

exercised in using any specific threshold, particularly when the more stringent statistical test 

indicates there is no significantly improved prediction. Based on this additional information from 

the authors, we have chosen to reflect the uncertainty about the existence and location of a 

potential threshold by estimating mortality attributable to long-term O3 exposures using a range 

of threshold-based effect estimates as sensitivity analyses. Specifically, we estimate long-term 

O3 mortality benefits using unique risk coefficients that include a range of thresholds from 40 

ppb to 60 ppb in 5 ppb increments, while also including a model with a threshold equal to 56 

ppb, which had the lowest log likelihood value for all models examined.2 Table 5B-1 provides 

the results of these sensitivity analyses (based on modeling incidence) for 60, ppb, 65 ppb and 70 

ppb. We note that the same pattern in terms of relative reductions across thresholds (relative to 

the core estimate) would hold for the dollar benefit estimates generated for this endpoint.

Table 5B-1. Long-term O3 Mortality Incidence at Various Assumed Thresholds a

Threshold 

Concentration 70 ppb 65 ppb 60 ppb

No threshold (core model) 680 2,100 3,900

60 ppb 520 1,600 2,900

56 ppb 410 1,100 1,700

55 ppb 120 280 510

50 ppb 5.6 58 140

45 ppb 3.3 47 105

40 ppb <1 10 13

a All estimates rounded to two significant digits.

 

2 There is a separate effect estimate (and associated standard error) for each of the fitted threshold models estimated 

in Jerrett et al. (2009). As a result, the sensitivity of estimated mortality attributable to long-term O3

concentrations is affected by both the assumed threshold level (below which there is no effect of O3) and the effect 

estimate applied to O3 concentrations above the threshold.
5B-4

The results of the sensitivity analysis based on the suite of threshold-based risk 

coefficients suggests that threshold models can result in substantially lower estimates of O3-

attributable long-term mortality. For example, estimated incidence and dollar benefits for longterm mortality using a model that includes a 55 ppb threshold are approximately 70% less than 

long-term mortality benefits estimated using the core co-pollutant non-threshold model. 

Generally, estimated long-term mortality benefits are progressively reduced when using models 

with increasing thresholds, with the highest threshold considered (60ppb) removing virtually all 

of the estimated incidence reduction and associated benefits.

5B.2 Alternative Concentration-Response Functions for PM2.5–Related Mortality

In modeling PM2.5 cobenefits, we estimate that total dollar benefits are driven largely by 

reductions in mortality (see Table 5-22 and 5-29). Therefore, it is particularly important to 

attempt to characterize the uncertainties associated with reductions in premature mortality as 

modeled in the PM2.5 cobenefits analysis. To better understand the concentration-response 

relationship between PM2.5 exposure and premature mortality, the EPA conducted an expert 

elicitation in 2006 (Roman et al., 2008; IEc, 2006).3In general, the results of the expert 

elicitation support the conclusion that the benefits of PM2.5 control are very likely to be 

substantial. 

Alternative concentration-response functions are useful for assessing uncertainty beyond 

random statistical error, including uncertainty in the functional form of the model or alternative 

study design. In this analysis, we present the results derived from the expert elicitation as 

indicative of the uncertainty associated with a major component of the health impact functions, 

and we provide the independent estimates derived from each of the twelve experts to better 

characterize the degree of variability in the expert responses.

In previous RIAs, the EPA presented benefits estimates using concentration response 

functions derived from the PM2.5 Expert Elicitation (Roman et al., 2008) as a range from the 

lowest expert value (Expert K) to the highest expert value (Expert E). However, this approach 

did not indicate the agency’s judgment on what the best estimate of PM2.5 benefits may be, and 

 

3 Expert elicitation is a formal, highly structured and well documented process whereby expert judgments, usually of 

multiple experts, are obtained (Ayyub, 2002).
5B-5

the EPA’s independent Science Advisory Board (SAB) recommended refinements to the way 

EPA presented the results of the elicitation (U.S. EPA-SAB, 2008). Therefore, we began to 

present the cohort-based studies (Krewski et al., 2009; Laden et al., 2006)4as our core estimates 

in the proposal RIA for the Portland Cement NESHAP (U.S. EPA, 2009a). Using alternate 

relationships between PM2.5 and premature mortality supplied by experts, higher and lower 

benefits estimates are plausible, but most of the expert-based estimates of the mean PM2.5 effect 

on mortality fall between the two epidemiology-based estimates (Roman et al., 2008). In addition 

to these studies, we have included a discussion or other recent multi-state cohort studies 

conducted in North America, but we have not estimated benefits using the effect coefficients 

from these studies (see Appendix 5D). Please note that the benefits estimates results presented 

are not the direct results from the studies or expert elicitation; rather, the estimates are based in 

part on the effect coefficients provided in those studies or by experts. In addition, because we are 

using a dollar-per-ton approach in modeling PM2.5 cobenefits in this RIA, we cannot generate 

confidence intervals reflecting the statistical fit characterized in the expert elicitation-based 

functions.

Even these multiple characterizations based on application of the range of expert 

elicitation-based effect estimates omit the contribution to overall uncertainty from uncertainty in 

air quality changes, baseline incidence rates, and populations exposed. Furthermore, the 

approach presented here does not yet include methods for addressing correlation between input 

parameters and the identification of reasonable upper and lower bounds for input distributions 

characterizing uncertainty in additional model elements. 

The PM2.5 expert elicitation and the derivation of effect estimates from the expert 

elicitation results (used in generating these alternative dollar-per-ton estimates) are described in 

detail in the 2006 PM2.5 NAAQS RIA (U.S. EPA, 2006), the elicitation summary report (IEc, 

2006) and Roman et al. (2008), and consequently, we do not present those effect estimates (and 

associated functional forms) here.

 

4 We have since updated the the Harvard Six Cities cohort study from Laden et al. (2006) to use the most recent 

follow-up publication of this cohort (Lepeule et al, 2012). This study is reflected in the dollar-per-ton values used in 

this RIA.
5B-6

Table 5B-2 presents the results of this sensitivity analysis as completed for the 2025 

scenario (overall conclusions generated from this analysis are transferable to the post-2025 

scenario). The alternative mortality estimates presented in Table 5B-2 were generated similar to 

core cobenefits PM2.5 mortality estimates, but applying dollar-per-ton values (for each expert 

elicitation-based effect estimate) to the sector-level estimates of NOx reductions associated with 

the post-2025 scenario. We have also included the core cobenefits estimates for each alternative 

standard to facilitate comparison against these alternative sensitivity analysis estimates. Because 

application of these effect estimates (using a dollar-per-ton approach) represents a linear 

calculation, results in terms of the relative ranking of the core estimates compared with expert 

elicitation estimates will remain the same for all alternative standards evaluated. Therefore, we 

only present estimates for the 70ppb alternative standard, observing that observations drawn 

from this sensitivity analysis would hold for other alternative standards considered (and for the 

post-2025 scenario – for all alternative standards – as well).

Table 5B-2. Application of Alternative (Expert Elicitation-Based Effect Estimates) to the 

Modeling of PM2.5 Co-benefit Estimates for PM2.5 (avoided incidence)

C-R Function (and effect estimate)b 70 ppb

Krewski et al., (2012) (core model) 45

Lepeule et al., (2012) (core model) 100

Expert K 10

Expert G 54

Expert L 63

Expert D 65

Expert H 67

Expert J 75

Expert F 88

Expert C 92

Expert I 92

Expert B 95

Expert A 120

Expert E 150

a All estimates rounded to two significant digits.

b Expert elicitation-based values ordered by magnitude of incidence reduction

The values presented in Table 5B-2 suggest that the two core incidence estimate fall 

within the range of alternative C-R function based estimates obtained through expert elicitation. 

This increases overall confidence in the core estimates with regard to the form of the functions 
5B-7

and magnitude of the effect estimates. We would expect the relationship between the core 

estimates and the expert-derived estimates to remain constant for the remaining scenarios as 

well.

5B.3 Income Elasticity of Willingness to Pay

As discussed in Chapter 5, our estimates of monetized benefits account for growth in real 

GDP per capita by adjusting the WTP for individual endpoints based on the central estimate of 

the adjustment factor for each of the categories (minor health effects, severe and chronic health 

effects, premature mortality, and visibility). We examined how sensitive the estimate of total 

benefits is to alternative estimates of the income elasticities. Income growth projections are only 

currently available in BenMAP through 2024, so both the 2025 and post-2025 scenario estimates 

use income growth only through 2024 and are therefore likely underestimates. 

Table 5B-4 lists the ranges of elasticity values used to calculate the income adjustment 

factors, while Table 5B-5 lists the ranges of corresponding adjustment factors. The results of this 

sensitivity analysis, giving the monetized benefit subtotals for the four benefit categories, are 

presented in Table 5B-6. 

Table 5B-3. Ranges of Elasticity Values Used to Account for Projected Real Income 

Growth a

Benefit Category

Lower Sensitivity 

Bound

Upper Sensitivity 

Bound

Minor Health Effectb 0.04 0.30

Premature Mortality 0.08 1.00

a Derivation of these ranges can be found in Kleckner and Neumann (1999). COI estimates are assigned an 

adjustment factor of 1.0.

b Minor health effects included in this RIA and valued using WTP-based functions include: upper and lower 

respiratory symptoms, asthma exacerbations, minor restricted activity days, and acute bronchitis.

Table 5B-4. Ranges of Adjustment Factors Used to Account for Projected Real Income 

Growth to 2024 a

Benefit Category

Lower Sensitivity 

Bound

Upper Sensitivity 

Bound

Minor Health Effectb

1.021 1.170

Premature Mortality 1.043 1.705

a Based on elasticity values reported in Table C-4, U.S. Census population projections, and projections of real GDP 

per capita.

b Minor health effects included in this RIA and valued using WTP-based functions include: upper and lower 

respiratory symptoms, asthma exacerbations, minor restricted activity days, and acute bronchitis.
5B-8

Table 5B-5. Sensitivity of Monetized O3 Benefits to Alternative Income Elasticities in 

2025 (Millions of 2011$) a

Benefit Category No adjustment Lower Sensitivity Bound Upper Sensitivity Bound

70 ppb 65 ppb 70 ppb 65 ppb 70 ppb 65 ppb

Minor Health Effect b $66 $200 $67 $210 $77 $240

Premature Mortality c $2,000 $6,400 $2,100 $6,600 $3,500 $11,000

a All estimates rounded to two significant digits. Only reflects income growth to 2024.

b For purposes of completing this sensitivity analysis, we have included minor restricted activity days (MRADS) 

based resulting from short-term ozone exposure as the minor health effect evaluated here.

c Using short-term mortality effect estimate from Smith et al. (2009) and 3% discount rate. Results using other shortterm mortality studies and a 7% discount rate would show the same proportional range.

Consistent with the impact of mortality on total benefits, the adjustment factor for 

mortality has the largest impact on total benefits. The value of mortality in 2025 ranges from 

86% to 133% of the main estimate for mortality based on the lower and upper sensitivity bounds

on the mortality income adjustment factor. The effect on the value of minor health effects is 

much less pronounced, ranging from 96% to 108% of the main estimate for minor effects. These 

observations (in terms of relative impact from alternative elasticities) hold for all three of the 

alternative standard levels evaluated under both the 2025 and post-205 scenarios.

5B.4 References

Frey, C. 2014. CASAC Review of the EPA’s Health Risk and Exposure Assessment for Ozone (Second External 

Review Draft – February, 2014). U.S. Environmental Protection Agency Science Advisory Board. EPACASAC-14-XXX.Gent, J.F.; E.W. Triche; T.R. Holford; K. Belanger; M.B. Bracken; W.S. Beckett, et al. 2003. 

Association of low-level ozone and fine particles with respiratory symptoms in children with asthma. Journal of 

the American Medical Association. 290(14):1859-1867.

Industrial Economics, Incorporated (IEc). 2006. Expanded Expert Judgment Assessment of the ConcentrationResponse Relationship Between PM2.5 Exposure and Mortality. Prepared for: Office of Air Quality Planning and 

Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC. September. Available at 

<http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm_ee_tsd_expert_interview_summaries.pdf>.

Jerrett M, Burnett RT, Pope CA, III, et al. 2009. “Long-Term Ozone Exposure and Mortality.” New England 

Journal of Medicine 360:1085-95.

Krewski D, Jerrett M, Burnett RT, Ma R, Hughes E, Shi, Y, et al. 2009. Extended follow-up and spatial analysis of 

the American Cancer Society study linking particulate air pollution and mortality. HEI Research Report, 140, 

Health Effects Institute, Boston, MA.

Kleckner, N., and J. Neumann. 1999. “Recommended Approach to Adjusting WTP Estimates to Reflect Changes in 

Real Income.” Memorandum to Jim DeMocker, US EPA/OPAR. June 3, Available on the Internet at 

<http://www.epa.gov/ttn/ecas/regdata/Benefits/background/klecknerandneumann1999.pdf>.

Lepeule J, Laden F, Dockery D, Schwartz J. 2012. “Chronic Exposure to Fine Particles and Mortality: An Extended 

Follow-Up of the Harvard Six Cities Study from 1974 to 2009.” Environmental Health Perspectives 120 

(7):965-70.
5B-9

Roman, Henry A., Katherine D. Walker, Tyra L. Walsh, Lisa Conner, Harvey M. Richmond, Bryan J. Hubbell, and 

Patrick L. Kinney. 2008. “Expert Judgment Assessment of the Mortality Impact of Changes in Ambient Fine 

Particulate Matter in the U.S.” Environmental Science & Technology 42(7):2268-2274.

Smith, R.L.; B. Xu and P. Switzer. 2009. Reassessing the relationship between ozone and short- term mortality in 

U.S. urban communities. Inhalation Toxicology. 21:37-61.

U.S. Environmental Protection Agency (U.S. EPA). 2009. Regulatory Impact Analysis: National Emission 

Standards for Hazardous Air Pollutants from the Portland Cement Manufacturing Industry. Office of Air 

Quality Planning and Standards, Research Triangle Park, NC. April. Available at 

<http://www.epa.gov/ttn/ecas/regdata/RIAs/portlandcementria_4-20-09.pdf>.

U.S. Environmental Protection Agency (U.S. EPA). 2012. Regulatory Impact Analysis for the Final Revisions to the 

National Ambient Air Quality Standards for Particulate Matter. EPA-452/R-12-003. Office of Air Quality 

Planning and Standards, Health and Environmental Impacts Division. December. Available at: 

<http://www.epa.gov/pm/2012/finalria.pdf>. 

U.S. Environmental Protection Agency—Science Advisory Board (EPA-SAB). 1999a. The Clean Air Act 

Amendments (CAAA) Section 812 Prospective Study of Costs and Benefits (1999): Advisory by the Health and 

Ecological Effects Subcommittee on Initial Assessments of Health and Ecological Effects. Part 1. EPA-SABCOUNCIL-ADV-99-012. July. Available on the Internet at 

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/F64A21824D19766885257193005E51CA/$File/conadv1

2.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (EPA-SAB). 1999b. The Clean Air Act 

Amendments (CAAA) Section 812 Prospective Study of Costs and Benefits (1999): Advisory by the Health and 

Ecological Effects Subcommittee on Initial Assessments of Health and Ecological Effects. Part 2. EPA-SABCOUNCIL-ADV-00-001. October. Available on the Internet at 

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/7D54675178C12703852571B90044B4F8/$File/coua0001

.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (EPA-SAB). 2004a. Advisory on Plans for Health 

Effects Analysis in the Analytical Plan for EPA’s Second Prospective Analysis—Benefits and Costs of the Clean 

Air Act, 1990-2020: Advisory by the Health Effects Subcommittee of the Advisory Council on Clean Air 

Compliance Analysis. EPA-SAB-COUNCIL-ADV-04-002. March. Available on the Internet at 

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/08E1155AD24F871C85256E5400433D5D/$File/council

_adv_04002.pdf>.

U.S. Environmental Protection Agency Science Advisory Board (U.S. EPA-SAB). 2004b. Advisory Council on 

Clean Air Compliance Analysis Response to Agency Request on Cessation Lag. EPA-COUNCIL-LTR-05-001 

December. Available on the Internet at 

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/39F44B098DB49F3C85257170005293E0/$File/council_l

tr_05_001.pdf>.

U.S. Environmental Protection Agency Science Advisory Board (U.S. EPA-SAB). 2008. Characterizing 

Uncertainty in Particulate Matter Benefits Using Expert Elicitation. EPA-COUNCIL-08-002. July. Available at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/0/43B91173651AED9E85257487004EA6CB/$File/EPACOUNCIL-08-002-unsigned.pdf>.

U.S. Environmental Protection Agency Science Advisory Board (U.S. EPA-SAB). 2010. Review of EPA’s DRAFT 

Health Benefits of the Second Section 812 Prospective Study of the Clean Air Act. EPA-COUNCIL-10-001. June. 

Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/9288428b8eeea4c885257242006935a3/59e06b6c5ca66597852575e7

006c5d09!OpenDocument&TableRow=2.3#2.>.
5B-10

Zanobetti, A; J. Schwartz. 2008. Mortality displacement in the association of ozone with mortality: an analysis of 48 

cities in the United States. American Journal of Respiratory and Critical Care Medicine. 177:184-189.