5B-1

APPENDIX 5B: ADDITIONAL SENSITIVITY ANALYSES RELATED TO THE 

OZONE HEALTH BENEFITS ANALYSIS

Overview

The benefits analysis presented in Chapter 5 of this RIA is based on our current 

interpretation of the scientific and economic literature. That interpretation requires judgments 

regarding the best available data, models, and analytical methodologies and the assumptions that 

are most appropriate to adopt in the face of important uncertainties. The majority of the 

analytical assumptions used to develop the main estimates of benefits have been reviewed and 

supported by EPA’s independent Science Advisory Board (SAB). Both EPA and the SAB 

recognize that data and modeling limitations as well as simplifying assumptions can introduce 

significant uncertainty into the estimates of benefits and that alternative choices exist for some 

inputs to the analysis, such as the concentration-response functions for mortality.

This appendix assesses the sensitivity of the core benefits to: (a) the potential impact of 

thresholds in long-term ozone exposure-related mortality in incidence and benefits estimates 

(section 5B.2), (b) alternative response functions developed through expert elicitation (EE) for 

long-term PM2.5 exposure-related mortality (section 5B.3), and (c) alternative assumptions 

regarding income elasticity on benefits derived using willingness-to-pay (WTP) functions 

(section 5B.3).

For the core analysis, we estimated incidence and dollar benefits for two scenarios: 2025 

and post-2025 (see Chapter 5, Sections 5.2 and 5.4.3). However, in conducting these sensitivity 

analyses, we used the 2025 scenario as the basis for making our calculations, since sensitivity 

analysis findings for this scenario would generally hold for the post-2025 scenario.

In addition to the three sensitivity analyses covered in this appendix, we also included 

two sensitivity analyses that are covered in detail in Chapter 5. The first of these are estimates of 

dollar benefits associated with mortality resulting from long-term exposure to ozone. As 

discussed in Section 5.2, while we felt that we had sufficient confidence to include incidence 

estimates associated with long-term ozone exposure (based on effect estimates obtained from 

Jerrett et al., 2009) in the core analysis, limitations in our ability to specify an appropriate lag for 
5B-2

reductions in this endpoint meant that we could not include dollar benefit estimates in the core 

analysis. Instead, we have included those values as sensitivity analyses, including consideration 

of alternative lag structures (they are presented in Tables 5-20 and 5-27, respectively for the 

2025 and post-2025 scenarios).1 The second of the sensitivity analyses already covered in the 

chapter addresses alternative models for short-term ozone exposure-related mortality. As 

discussed in Section 5.6.3.1, in addition to the two epidemiological studies (Smith et al., 2009 

and Zanobetti and Schwartz 2008) that provide effect estimates for the core benefits estimates, 

we have also considered seven additional epidemiological studies as sensitivity analyses. These 

additional studies include a mix of multi-city and meta-analysis designs. Results of this 

sensitivity analysis are presented in Tables 5-19 and 5-20 (incidence and benefits, respectively 

for the 2025 scenario) and Tables 5-26 and 5-27 (incidence and benefits, respectively for the 

post-2025 scenario).

5B.1 Threshold Sensitivity Analysis for Premature Mortality Incidence and Benefits from 

Long-term Exposure to Ozone

In estimating long-term ozone mortality, we employed a continuous non-threshold 

concentration-response (C-R) function relating ozone exposure to premature death. However, as 

discussed in Section 5.6.3.1, there is uncertainty regarding the potential existence and location of 

a threshold in the C-R function relating mortality and long-term ozone concentrations. Thus, we 

have included a sensitivity analysis exploring the impact of potential thresholds in the C-R 

relationship on estimates of long-term exposure-related mortality that were evaluated in Jerrett et 

al. (2009), consistent with advice from CASAC (Frey, 2014). 

In their memo clarifying the results of their study (Sasser, 2014), the authors note that in 

terms of goodness of fit, long-term health risk models including ozone clearly performed better 

than models without ozone, indicating the improved predictions of respiratory mortality when 

ozone is included. In exploring different functional forms, the authors report that the model 

including a threshold at 56 ppb had the lowest log-likelihood value of all models evaluated (i.e., 

linear models and models including thresholds ranging from 40-60 ppb), and thus provided the 

 

1 The sensitivity analysis-related benefits estimates presented in these tables include (a) benefits estimates reflecting 

application of a zero lag model and (b) estimates reflecting application of the same 20-year segmented lag used in 

modeling benefits for PM2.5, together with application of a 3% and 7% discount rate. See Sections 5.6.3 and 5.6.4 

for additional discussion of lags in relation to long-term ozone-related mortality.
5B-3

best overall statistical fit to the data. However, they also note that it is not clear whether the 56 

ppb threshold model is a better predictor of respiratory mortality than when using a linear (nothreshold) model for this dataset. Using one statistical test, the model with a threshold at 56 ppb 

was determined to be statistically superior to the linear model. Using another, more stringent test, 

none of the threshold models considered were statistically superior to the linear model. Under the 

less stringent test, although the threshold model produces a statistically superior prediction than 

the linear model, there is uncertainty about the specific location of the threshold, if one exists. 

This is because the confidence intervals on the model predictions indicate that a threshold could 

exist anywhere from 0 to 60 ppb. The authors conclude that considerable caution should be 

exercised in using any specific threshold, particularly when the more stringent statistical test 

indicates there is no significantly improved prediction. Based on this additional information from 

the authors, we have chosen to reflect the uncertainty about the existence and location of a 

potential threshold by estimating mortality attributable to long-term ozone exposures using a 

range of threshold-based effect estimates as sensitivity analyses. Specifically, we estimate longterm ozone mortality benefits using unique risk coefficients that include a range of thresholds 

from 40 ppb to 60 ppb in 5 ppb increments, while also including a model with a threshold equal 

to 56 ppb, which had the lowest log-likelihood value for all models examined.2 Table 5B-1 

provides the results of these sensitivity analyses (based on modeling incidence) for 60 ppb, 65 

ppb and 70 ppb. We note that the same pattern in terms of relative reductions across thresholds 

(relative to the core estimate) would hold for the dollar benefit estimates generated for this 

endpoint.

 

2 There is a separate effect estimate (and associated standard error) for each of the fitted threshold models estimated 

in Jerrett et al. (2009). As a result, the sensitivity of estimated mortality attributable to long-term ozone

concentrations is affected by both the assumed threshold level (below which there is no effect of ozone) and the 

effect estimate applied to ozone concentrations above the threshold.
5B-4

Table 5B-1. Long-term Ozone Mortality Incidence at Various Assumed Thresholds a

Threshold 

Concentration 70 ppb 65 ppb 60 ppb

No threshold (core model) 680 2,100 3,900

460 ppb 520 1,600 2,900

4556 ppb 410 1,100 1,700

505 ppb 120 280 510

550 ppb 5.6 58 140

5645 ppb 3.3 47 105

640 ppb <1 10 13

a All estimates rounded to two significant digits.

The results of the sensitivity analysis based on the suite of threshold-based risk 

coefficients suggests that threshold models can result in substantially lower estimates of ozoneattributable long-term mortality. For example, estimated incidence and dollar benefits for longterm mortality using a model that includes a 55 ppb threshold are approximately 70% less than 

long-term mortality benefits estimated using the core co-pollutant non-threshold model. 

Generally, estimated long-term mortality benefits are progressively reduced when using models 

with increasing thresholds, with the highest threshold considered (60 ppb) removing virtually all 

of the estimated incidence reduction and associated benefits.

5B.2 Alternative Concentration-Response Functions for PM2.5–Related Mortality

In modeling PM2.5 cobenefits, we estimate that total dollar benefits are driven largely by 

reductions in mortality (see Table 5-22 and 5-29). Therefore, it is particularly important to 

attempt to characterize the uncertainties associated with reductions in premature mortality as 

modeled in the PM2.5 cobenefits analysis. To better understand the concentration-response 

relationship between PM2.5 exposure and premature mortality, the EPA conducted an expert 

elicitation in 2006 (Roman et al., 2008; IEc, 2006).3In general, the results of the expert 

elicitation support the conclusion that the benefits of PM2.5 control are very likely to be 

substantial. 

Alternative concentration-response functions are useful for assessing uncertainty beyond 

random statistical error, including uncertainty in the functional form of the model or alternative 

study design. In this analysis, we present the results derived from the expert elicitation as 

 

3 Expert elicitation is a formal, highly-structured and well-documented process whereby expert judgments, usually 

of multiple experts, are obtained (Ayyub, 2002).
5B-5

indicative of the uncertainty associated with a major component of the health impact functions, 

and we provide the independent estimates derived from each of the twelve experts to better 

characterize the degree of variability in the expert responses.

In previous RIAs, the EPA presented benefits estimates using concentration-response 

functions derived from the PM2.5 Expert Elicitation (Roman et al., 2008) as a range from the 

lowest expert value (Expert K) to the highest expert value (Expert E). However, this approach 

did not indicate the Agency’s judgment on what the best estimate of PM2.5 benefits may be, and 

the EPA’s independent SAB recommended refinements to the way EPA presented the results of 

the elicitation (U.S. EPA-SAB, 2008). Therefore, we began to present the cohort-based studies 

(Krewski et al., 2009; Laden et al., 2006)4as our core estimates in the proposal RIA for the 

Portland Cement NESHAP (U.S. EPA, 2009a). Using alternate relationships between PM2.5 and 

premature mortality supplied by experts, higher and lower benefits estimates are plausible, but 

most of the expert-based estimates of the mean PM2.5 effect on mortality fall between the two 

epidemiology-based estimates (Roman et al., 2008). In addition to these studies, we have 

included a discussion of other recent multi-state cohort studies conducted in North America, but 

we have not estimated benefits using the effect coefficients from these studies (see Appendix 

5D). Please note that the benefits estimates results presented are not the direct results from the 

studies or expert elicitation; rather, the estimates are based in part on the effect coefficients 

provided in those studies or by experts. In addition, because we are using a dollar-per-ton 

approach in modeling PM2.5 cobenefits in this RIA, we cannot generate confidence intervals 

reflecting the statistical fit characterized in the expert elicitation-based functions.

Even these multiple characterizations based on application of the range of expert 

elicitation-based effect estimates omit the contribution to overall uncertainty from uncertainty in 

air quality changes, baseline incidence rates, and populations exposed. Furthermore, the 

approach presented here does not yet include methods for addressing correlation between input 

parameters and the identification of reasonable upper and lower bounds for input distributions 

characterizing uncertainty in additional model elements. 

 

4 We have since updated the Harvard Six Cities cohort study from Laden et al. (2006) to use the most recent followup publication of this cohort (Lepeule et al, 2012). This study is reflected in the dollar-per-ton values used in this 

RIA.
5B-6

The PM2.5 expert elicitation and the derivation of effect estimates from the expert 

elicitation results (used in generating these alternative dollar-per-ton estimates) are described in 

detail in the 2006 PM2.5 NAAQS RIA (U.S. EPA, 2006), the elicitation summary report (IEc, 

2006) and Roman et al. (2008), and consequently, we do not present those effect estimates (and 

associated functional forms) here.

Table 5B-2 presents the results of this sensitivity analysis as completed for the 2025 

scenario (overall conclusions generated from this analysis are transferable to the post-2025 

scenario). The alternative mortality estimates presented in Table 5B-2 were generated similar to 

the core cobenefits PM2.5 mortality estimates, but applying dollar-per-ton values (for each expert 

elicitation-based effect estimate) to the sector-level estimates of NOx reductions associated with 

the 2025 scenario. We have also included the core cobenefits estimates for each alternative 

standard to facilitate comparison against these alternative sensitivity analysis estimates. Because 

application of these effect estimates (using a dollar-per-ton approach) represents a linear 

calculation, results in terms of the relative ranking of the core estimates compared with expert 

elicitation estimates will remain the same for all alternative standards evaluated. Therefore, we 

only present estimates for the 70 ppb alternative standard, observing that observations drawn 

from this sensitivity analysis would hold for other alternative standards considered (and for the 

post-2025 scenario – for all alternative standards – as well).

Table 5B-2. Application of Alternative (Expert Elicitation-Based Effect Estimates) to the 

Modeling of PM2.5 Co-benefit Estimates for PM2.5 (avoided incidence)

C-R Function (and effect estimate)b 70 ppb

Krewski et al., (2012) (core model) 45

Lepeule et al., (2012) (core model) 100

Expert K 10

Expert G 54

Expert L 63

Expert D 65

Expert H 67

Expert J 75

Expert F 88

Expert C 92

Expert I 92

Expert B 95

Expert A 120

Expert E 150

a All estimates rounded to two significant digits.
5B-7

b Expert elicitation-based values ordered by magnitude of incidence reduction

The values presented in Table 5B-2 suggest that the two core incidence estimate fall 

within the range of alternative C-R function based estimates obtained through expert elicitation. 

This increases overall confidence in the core estimates with regard to the form of the functions 

and magnitude of the effect estimates. We would expect the relationship between the core 

estimates and the expert-derived estimates to remain constant for the remaining scenarios as 

well.

5B.3 Income Elasticity of Willingness-to-Pay

As discussed in Chapter 5, our estimates of monetized benefits account for growth in real 

GDP per capita by adjusting the WTP for individual endpoints based on the central estimate of 

the adjustment factor for each of the categories (minor health effects, severe and chronic health 

effects, premature mortality, and visibility). We examined how sensitive the estimate of total 

benefits is to alternative estimates of the income elasticities. Income growth projections are only 

currently available in BenMAP through 2024, so both the 2025 and post-2025 scenario estimates 

use income growth only through 2024 and are therefore likely underestimates. 

Table 5B-3 lists the ranges of elasticity values used to calculate the income adjustment 

factors, while Table 5B-4 lists the ranges of corresponding adjustment factors. The results of this 

sensitivity analysis, giving the monetized benefit subtotals for the four benefit categories, are 

presented in Table 5B-5. 

Table 5B-3. Ranges of Elasticity Values Used to Account for Projected Real Income 

Growth a

Benefit Category

Lower Sensitivity 

Bound

Upper Sensitivity 

Bound

Minor Health Effectb 0.04 0.30

Premature Mortality 0.08 1.00

a Derivation of these ranges can be found in Kleckner and Neumann (1999). COI estimates are assigned an 

adjustment factor of 1.0.

b Minor health effects included in this RIA and valued using WTP-based functions include: upper and lower 

respiratory symptoms, asthma exacerbations, minor restricted activity days, and acute bronchitis.
5B-8

Table 5B-4. Ranges of Adjustment Factors Used to Account for Projected Real Income 

Growth to 2024 a

Benefit Category

Lower Sensitivity 

Bound

Upper Sensitivity 

Bound

Minor Health Effectb

1.021 1.170

Premature Mortality 1.043 1.705

a Based on elasticity values reported in Table C-4, U.S. Census population projections, and projections of real GDP 

per capita.

b Minor health effects included in this RIA and valued using WTP-based functions include: upper and lower 

respiratory symptoms, asthma exacerbations, minor restricted activity days, and acute bronchitis.

Table 5B-5. Sensitivity of Monetized Ozone Benefits to Alternative Income Elasticities in 

2025 (Millions of 2011$) a

Benefit Category No adjustment Lower Sensitivity Bound Upper Sensitivity Bound

70 ppb 65 ppb 70 ppb 65 ppb 70 ppb 65 ppb

Minor Health Effect b $66 $200 $67 $210 $77 $240

Premature Mortality c $2,000 $6,400 $2,100 $6,600 $3,500 $11,000

a All estimates rounded to two significant digits. Only reflects income growth to 2024.

b For purposes of completing this sensitivity analysis, we have included minor restricted activity days (MRADS) 

based resulting from short-term ozone exposure as the minor health effect evaluated here.

c Using short-term mortality effect estimate from Smith et al. (2009) and 3% discount rate. Results using other shortterm mortality studies and a 7% discount rate would show the same proportional range.

Consistent with the impact of mortality on total benefits, the adjustment factor for 

mortality has the largest impact on total benefits. The value of mortality in 2025 ranges from 

86% to 133% of the main estimate for mortality based on the lower and upper sensitivity bounds

on the mortality income adjustment factor. The effect on the value of minor health effects is 

much less pronounced, ranging from 96% to 108% of the main estimate for minor effects. These 

observations (in terms of relative impact from alternative elasticities) hold for all three of the 

alternative standard levels evaluated under both the 2025 and post-205 scenarios.

5B.4 References

Frey, C. 2014. CASAC Review of the EPA’s Health Risk and Exposure Assessment for Ozone (Second External 

Review Draft – February, 2014). U.S. Environmental Protection Agency Science Advisory Board. EPACASAC-14-XXX.Gent, J.F.; E.W. Triche; T.R. Holford; K. Belanger; M.B. Bracken; W.S. Beckett, et al. 2003. 

Association of low-level ozone and fine particles with respiratory symptoms in children with asthma. Journal of 

the American Medical Association. 290(14):1859-1867.

Industrial Economics, Incorporated (IEc). 2006. Expanded Expert Judgment Assessment of the ConcentrationResponse Relationship Between PM2.5 Exposure and Mortality. Prepared for: Office of Air Quality Planning and 

Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC. September. Available at 

<http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm_ee_tsd_expert_interview_summaries.pdf>.

Jerrett M, Burnett RT, Pope CA, III, et al. 2009. “Long-Term Ozone Exposure and Mortality.” New England 

Journal of Medicine 360:1085-95.
5B-9

Krewski D, Jerrett M, Burnett RT, Ma R, Hughes E, Shi, Y, et al. 2009. Extended follow-up and spatial analysis of 

the American Cancer Society study linking particulate air pollution and mortality. HEI Research Report, 140, 

Health Effects Institute, Boston, MA.

Kleckner, N., and J. Neumann. 1999. “Recommended Approach to Adjusting WTP Estimates to Reflect Changes in 

Real Income.” Memorandum to Jim DeMocker, US EPA/OPAR. June 3, Available on the Internet at 

<http://www.epa.gov/ttn/ecas/regdata/Benefits/background/klecknerandneumann1999.pdf>.

Lepeule J, Laden F, Dockery D, Schwartz J. 2012. “Chronic Exposure to Fine Particles and Mortality: An Extended 

Follow-Up of the Harvard Six Cities Study from 1974 to 2009.” Environmental Health Perspectives 120 

(7):965-70.

Roman, Henry A., Katherine D. Walker, Tyra L. Walsh, Lisa Conner, Harvey M. Richmond, Bryan J. Hubbell, and 

Patrick L. Kinney. 2008. “Expert Judgment Assessment of the Mortality Impact of Changes in Ambient Fine 

Particulate Matter in the U.S.” Environmental Science & Technology 42(7):2268-2274.

Smith, R.L.; B. Xu and P. Switzer. 2009. Reassessing the relationship between ozone and short- term mortality in 

U.S. urban communities. Inhalation Toxicology. 21:37-61.

U.S. Environmental Protection Agency (U.S. EPA). 2009. Regulatory Impact Analysis: National Emission 

Standards for Hazardous Air Pollutants from the Portland Cement Manufacturing Industry. Office of Air 

Quality Planning and Standards, Research Triangle Park, NC. April. Available at 

<http://www.epa.gov/ttn/ecas/regdata/RIAs/portlandcementria_4-20-09.pdf>.

U.S. Environmental Protection Agency (U.S. EPA). 2012. Regulatory Impact Analysis for the Final Revisions to the 

National Ambient Air Quality Standards for Particulate Matter. EPA-452/R-12-003. Office of Air Quality 

Planning and Standards, Health and Environmental Impacts Division. December. Available at: 

<http://www.epa.gov/pm/2012/finalria.pdf>. 

U.S. Environmental Protection Agency—Science Advisory Board (EPA-SAB). 1999a. The Clean Air Act 

Amendments (CAAA) Section 812 Prospective Study of Costs and Benefits (1999): Advisory by the Health and 

Ecological Effects Subcommittee on Initial Assessments of Health and Ecological Effects. Part 1. EPA-SABCOUNCIL-ADV-99-012. July. Available on the Internet at 

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/F64A21824D19766885257193005E51CA/$File/conadv1

2.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (EPA-SAB). 1999b. The Clean Air Act 

Amendments (CAAA) Section 812 Prospective Study of Costs and Benefits (1999): Advisory by the Health and 

Ecological Effects Subcommittee on Initial Assessments of Health and Ecological Effects. Part 2. EPA-SABCOUNCIL-ADV-00-001. October. Available on the Internet at 

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/7D54675178C12703852571B90044B4F8/$File/coua0001

.pdf>.

U.S. Environmental Protection Agency—Science Advisory Board (EPA-SAB). 2004a. Advisory on Plans for Health 

Effects Analysis in the Analytical Plan for EPA’s Second Prospective Analysis—Benefits and Costs of the Clean 

Air Act, 1990-2020: Advisory by the Health Effects Subcommittee of the Advisory Council on Clean Air 

Compliance Analysis. EPA-SAB-COUNCIL-ADV-04-002. March. Available on the Internet at 

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/08E1155AD24F871C85256E5400433D5D/$File/council

_adv_04002.pdf>.

U.S. Environmental Protection Agency Science Advisory Board (U.S. EPA-SAB). 2004b. Advisory Council on 

Clean Air Compliance Analysis Response to Agency Request on Cessation Lag. EPA-COUNCIL-LTR-05-001 

December. Available on the Internet at 

<http://yosemite.epa.gov/sab%5CSABPRODUCT.NSF/39F44B098DB49F3C85257170005293E0/$File/council_l

tr_05_001.pdf>.
5B-10

U.S. Environmental Protection Agency Science Advisory Board (U.S. EPA-SAB). 2008. Characterizing 

Uncertainty in Particulate Matter Benefits Using Expert Elicitation. EPA-COUNCIL-08-002. July. Available at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/0/43B91173651AED9E85257487004EA6CB/$File/EPACOUNCIL-08-002-unsigned.pdf>.

U.S. Environmental Protection Agency Science Advisory Board (U.S. EPA-SAB). 2010. Review of EPA’s DRAFT 

Health Benefits of the Second Section 812 Prospective Study of the Clean Air Act. EPA-COUNCIL-10-001. June. 

Available on the Internet at 

<http://yosemite.epa.gov/sab/sabproduct.nsf/9288428b8eeea4c885257242006935a3/59e06b6c5ca66597852575e7

006c5d09!OpenDocument&TableRow=2.3#2.>.

Zanobetti, A; J. Schwartz. 2008. Mortality displacement in the association of ozone with mortality: an analysis of 48 

cities in the United States. American Journal of Respiratory and Critical Care Medicine. 177:184-189.