### November 2009



# Influence Analysis in Support of Characterizing Uncertainty in Human Health Benefits Analysis

Final Report

Prepared for

Amy Lamson U.S. Environmental Protection Agency Office of Air Quality Planning and Standards (OAQPS) Air Benefit and Cost Group (ABCG) (MD-C439-02) Research Triangle Park, NC 27711

Prepared by

Carol Mansfield
Paramita Sinha RTI International Research Triangle Park, NC 27709

Max Henrion Lumina Decision Systems

EPA Contract Number EP-D-06-003

RTI Project Number 0209897.003.065

### EPA Contract Number EP-D-06-003 RTI Project Number 0209897.003.065



# Influence Analysis in Support of Characterizing Uncertainty in Human Health Benefits Analysis

Final Report

November 2009

Amy Lamson U.S. Environmental Protection Agency Office of Air Quality Planning and Standards (OAQPS) Air Benefit and Cost Group (ABCG) (MD-C439-02) Research Triangle Park, NC 27711

Prepared by

Carol Mansfield
Paramita Sinha RTI International Research Triangle Park, NC 27709

Max Henrion Lumina Decision Systems

### iii CONTENTS

Section Page

1. Introduction ...............................................................................................................

.... 1-1

2. Approach

................................................................................................................... .... 2-1

2.1 Overview of Approach ......................................................................................... 2-1

2.2 Identify and Classify the Uncertainties ................................................................ 2-1

2.2.1 Sources of Uncertainty ............................................................................. 2-2

2.2.2 Types of Uncertainty................................................................................ 2-2

2.3 Assess Ranges for Parameters a nd Alternative Functional Forms ...................... 2-2

2.4 Develop Influence Diagrams ............................................................................... 2-4

2.5 Use of Analytica .................................................................................................. 2-5

2.6 Develop a Simplified Model to Conduct Sensitivity Analysis and Compare to BenMAP .......................................................................................................... 2-5

2.6.1 Calibration to BenMAP ........................................................................... 2-6

2.7 Analysis Strategy ................................................................................................. 2-7

3. Classify Uncertainties and Quantify Parameter Ranges ............................................... 3-1

3.1 Final List of Uncertainties ................................................................................... 3-1

3.2 Quantify Ranges for Parameters .......................................................................... 3-4

3.3 Source for Range Recommendations ................................................................... 3-4

3.4 Alternative Functional Forms .............................................................................. 3-9

4. Influence Diagrams ....................................................................................................... 4-1

4.1 Introduction to Influence Diagrams ..................................................................... 4-1

4.2 Influence Diagrams for Sensitivity Analysis ....................................................... 4-2

### iv 5. Results ....................................................................................................................

....... 5-1

5.1 Assumptions for Analysis .................................................................................... 5-1

5.2 Measures of Sensitivity Used in Analysis ........................................................... 5-1

5.3 Elasticity Results .................................................................................................. 5-2

5.4 Results for Different Concentration- Response Functions for Mortality and Morbidity ............................................................................................................. 5-4

5.4.1 Ranges for Concentration-Response Functions from Published Epidemiological Studies for Mortality and Morbidity............................. 5-4

5.4.2 Ranges from Expert Elicitations of Probability Di stributions on CR Functions .................................................................................................. 5-5

5.5 Range Sensitivity of Mortality Avoided Outcome ............................................ 5-11

5.6 Range Sensitivity for Total Monetized Benefits Outcome ................................ 5-15

6. Discussion and Possible Next Steps ............................................................................. 6-1

6.1 Discussion of Model Results ............................................................................... 6-1

6.1.1 Comments on the Uncertainty Analysis Methods ................................... 6-2

6.2 Potential Next Steps ............................................................................................. 6-3

6.2.1 Update Studies to Support Morbidity Valuation ..................................... 6-3

6.2.2 More Detailed Mortality Valuation Estimates ......................................... 6-4

6.2.3 Framework to Jointly Consider Multiple Health Endpoints .................... 6-4

6.2.4 Add Additional Outcomes and Expand Application to Other Pollutants.................................................................................................. 6-4

6.2.5 Expanding Scope of the Uncertain ty Analysis to Include Air Quality...................................................................................................... 6-5 6.2.6 Investigate Cost Uncertainty .................................................................... 6-5

6.2.7 A More Comprehensive Treatment of Uncertainties ............................... 6-5

6.2.8 Large Changes in Population, Migration and Immigration ..................... 6-6

6.2.9 Additions to Analytica Model .................................................................. 6-6

7. References

................................................................................................................. .... 7-1 Appendix

A: Expert Workshop Memo ..................................................................................... A-1

### v LIST OF FIGURES

Nu mber Page

2-1. Hierarchical Influence Diagram S howing the Top Level Model and Three Selected Subdiagrams ................................................................................................... 2-4
2-2. 36 km Grid Squares over th e Contiguous United States ............................................... 2-6

4-1. Illustrative Influence Diagram with Notation Defined ................................................. 4-1
4-2. Top-Level Influence Diagram ....................................................................................... 4-2
4-3. Influence Diagram for PM 2.5 

### Concentration ................................................................. 4-3
4-4. Influence Diagrams for Incidence Rates ....................................................................... 4-3
4-5. Influence Diagram for Health Impacts CR Functions .................................................. 4-4
4-6. Influence Diagram for Expert Elicitation Distributions ................................................ 4-5
4-7. Influence Diagram for Population Projections ............................................................. 4-5
4-8. Influence Diagram for Valuation Modules ................................................................... 4-6
4-9. Influence Diagram for Inflation and Income Growth ................................................... 4-7
4-10. Influence Diagram for Value of Avoided Morbidity Endpoint .................................... 4-7
4-11. Influence Diagram for Mortality Valuation Lag ........................................................... 4-8 4-12. Mortality Lag ...........................................................................................................

..... 4-9
4-13. Influence Diagram for Air Quality ............................................................................... 4-9

5-1. Elasticity of Mortality Parameters ................................................................................ 5-3
5-2. Elasticity of Total Monetized Benefits to Each Uncertain Parameter .......................... 5-4
5-3. Ranges of Beta for CR Functions ................................................................................. 5-6
5-4. Ranges of Beta for CR functions from Experts A to L ................................................. 5-8
5-5. Range of Avoided Premature Mortal ities and Morbidities (CB and AMI) for Low, Mid, and High Values for Each CR Function ..................................................... 5-9
5-6. Range of Monetized Benefits for Low, Mid, and High Values for Each CR Function ...................................................................................................................... 5-10
5-7. Range Sensitivity of Avoided Mortality for Pope et al. with Zero Threshold ............ 5-12
5-8a. Range Sensitivity of Avoided Mortality for Pope et al. with a 10 ug/m 3


### Threshold .................................................................................................................... 5-12
5-8b. Range Sensitivity of Avoided Mortality for Pope et al. with a 12 ug/m 3


### Threshold .................................................................................................................... 5-13
5-9. Range Sensitivity of Avoided Mortality for Laden et al. with Zero Threshold .......... 5-13
5-10. Range Sensitivity of Avoided Mortality for Pope et al. with Mid Threshold of
7.5 μ g/m 3



### ..................................................................................................................... 5-14

### vi 5-11. Range Sensitivity of Avoided Mort ality for Laden et al. with 10 ug/m 3


### Threshold .................................................................................................................... 5-14
5-12. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Pope et al. with Zero Threshold as the Mid Value ............................. 5-15
5-13. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Pope et al. with Zero Threshold as the Mid Value with an Exponential Declining Mortality Valuation lag Structure .......................................... 5-16
5-14. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Laden et al. Zero Threshold as the Mid Value ................................... 5-17
5-15. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Expert E .............................................................................................. 5-17
5-16. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Expert K .............................................................................................. 5-18
5-17. Discount by Lag Structure .......................................................................................... 5-19
5-18. Range Sensitivity of Total Monetized Benefits to Discount Rate Assumptions for Different Mortality Va luation lag Structures ........................................................ 5-19

### vii LIST OF TABLES

Nu mber Page

3-1. List of Major Sources of Uncer tainty in the BenMAP Model ...................................... 3-1
3-2. Range Values for Uncertain Param eters ....................................................................... 3-5

5-1. Ranges for Epidem iological Studies ............................................................................. 5-5
5-2. Probability Distributions Elic ited from Experts A through L ....................................... 5-7
5-3. Num ber of Cases of Mortality or Morb idity Avoided Using the Low, Mid, and High Values for the Beta from the CR Function in Each Source ............................... 5-11

### 1-1 SECTION 1

INTRODUCTION

The U.S. Environm ental Protection Agency ’s Office of Air Quality Planning and Standards (USEPA OAQPS) prep ares estimates of the benefits and costs of air quality regulations. OAQPS also assesses the sensitivity of the benefits and costs to uncertainties in the data and assumptions used for their analyses. As part of EPA’s efforts to assess uncertainty, RTI and Lumina Decision Systems developed an in fluence and uncertainty analysis model for BenMAP (USEPA, 2008). BenMAP is a software package widely used at the U.S. EPA and elsewhere to estimate the benefits of improvements in air quality.

This document outlines the methods and results of the influence and uncertainty analysis

of the BenMAP estimates of the benefits from air quality improvements. The BenMAP influence analysis focuses on a selection of important but uncertain parameters, functions, and input data

used to calculate the human hea lth benefits of reductions in PM 2.5 

### . For health outcomes, the model examines mortality, acute myocardial in farction (AMI) and chronic bronchitis (CB). The results of this analysis may be used to guide de cisions about how to prioritize future efforts to

characterize these uncertainties and, where possible, reduce them.

The goals of the project were to identify those input parameters and assumptions that have the potential to be significan t contributors to the uncertainty in the results of BenMAP, to compare the potential size of their contributions, and to identify those likely to make the largest

contributions.

By influence analysis, we mean a graphic depi ction of the model stru cture in the form of a hierarchy of influence diagrams. Each vari able, including input assumptions, decisions,
intermediate variables, and result s, is depicted as a labeled no de. Influence arrows between the nodes depict relationships where one variable influences another via a mathematical relationship in the underlying model.

By uncertainty analysis, we mean the process of comparing the effect of the uncertainty in each input variable on a result variable. Range sensitivity analysis is a method of uncertainty analysis that depicts the range of values in a result variable produced by changing each uncertain input from a low to high value, wh ile leaving all the other inputs at a nominal or mid value. We will examine sensitivities by assessing pl ausible ranges on each input and computing deterministic measures of sensitivity to compare their effects on selected output variables.

### 1-2 The focus of this project was to conduct an unc ertainty analysis of BenMAP as a tool to estimate the benefits of improvements in air quality. In order to keep the level of effort for the project within the resources available, the scope was defined to include the effects of uncertainty in the explicit inputs to BenMAP, including:

 Particulate matter under 2.5 microns (PM 2.5 

### ), and no other types of atmospheric pollutant.

 Annual average concentration of PM 2.5 

### , ignoring any effects of short-term peak concentration except insofar as they affect annual averages.

 Improvement in air quality was specified in terms of current annual average concentration of PM 2.5 

### and a future, usually lowe r, concentration of PM 2.5 

### at a future date, typically 2020, and excluding any preced ing variables or factors that might affect PM 2.5 

### , such as changes in emissions, or climate, as modeled in CMAQ and other computer models.

 Spatial resolution was to cover the contiguous United States (excluding Alaska, Hawaii), in 36-kilometer grid squares, a nd no smaller regions such as counties.

 Affected population was by age range, ignoring effects of gender, race, socioeconomic status, or other differentiators.

 Effects of improved air quality are to be examined on reduction in mortality and two types of morbidity, CB and AMI, and no other end points, such as other diseases or visibility.

 Treatment of uncertainty was explored by defining a range of values for key parameters, and their effects on results while holding all other parameters at their mid values (range sensitivity analysis). Probabili stic treatment of unce rtainties about input parameters using expert probability dist ributions and Monte Carlo analysis to propagate probabilistic uncerta inties were explicitly beyond the scope of this project.

The rest of the report is structured as follo ws: Section 2 presents the approach taken for the sensitivity analysis, followed by a discussion of the uncertain parameters and functions in Section 3 and the influence diagrams associated with the model in Section 4. The results can be found in Section 5, followed by a discussion of th e results and possible implications of the results for characterizing uncertainty in Section 6.

### 2-1 SECTION 2

APPROACH

2.1 Overview of Approach

The goals of the pro ject are to identify thos e input parameters and assumptions that have the potential to be significant contributors to the uncertainty in the results of BenMAP, to compare the potential size of their contributions, and to identify those likely to make the largest contributions. This outline shows the steps invol ved in conducting the influence and uncertainty analysis, which are described in more detail below.

 Identify and classify the uncertainties

– Develop an inventory of possi ble sources of uncertainty

– Classify these sources in terms of source and type of uncertainty

– Select which of these uncertainties s hould be treated quantitatively in the sensitivity analysis

– Assess ranges of plausible values for selected parametric uncertainties and ranges of plausible alternative fo rms for model uncertainties

 Build simplified model and conduct sensitivity analysis

– Develop influence diagrams to trace which sources of uncertainty have effects on which results

– Develop a simplified or reduced-form model

– Compare the simplified model with BenMAP for selected scenarios to validate and calibrate it.

– Conduct sensitivity analysis using ranges to identify the relative importance of the uncertainty associated with different parameters and model assumptions on the results using various measures of sensitivity

 Provide the results and discussion of the re sults to EPA, so that EPA can use the information to assess which uncertainties are most important to investigate in more detail

2.2 Identify and Classify the Uncertainties

The f irst step is to identify and classify th e sources of uncertainty in BenMAP. The team developed an inventory of sources of uncertainty by reviewing Be nMAP, including its input files

and documentation. Where BenMAP offers different functional forms, we identified an explicit source of uncertainty. Where it uses a particular functional form and some other functional form appeared reasonable, we identified that as an implicit model uncertainty. An exception is the

### 2-2 uncertainties in emissions and air quality whose explicit quantification is outside the scope of this study.

We classified each source of uncertainty according to its source and type, using the following categories:

2.2.1 Sources of Uncertainty

 Explicit par ameters inside BenMAP, where BenMAP makes it easy to select a value as an input.

 Input parameters generated outside of BenMAP, such as emissions or population distribution.

 Implicit parameters , where BenMAP uses a single value, but there are other plausible values not offered as an explicit input to BenMAP.

 Explicit model forms selectable within BenMAP su ch as the functional form for concentration-response (CR) functions.

 Implicit model forms used in BenMAP where there are other plausible model forms not offered as an expl icit input to BenMAP.

2.2.2 Types of Uncertainty

 Param eter uncertainty

– Measured: Uncertainty in measured quantitie s due to measurement biases and limited sample size, for example, populati on estimate from past census, mortality data, willingness to pay (WTP) to avoid a health outcome or the estimate of beta in CR functions

– Forecast: Uncertainty in forecasts based on e xpert judgment and/or extrapolation from past trends, for example, future population, Gross Domes tic Product (GDP),
or PM 2.5 

### concentrations.

– Values: These are used to quantify outcomes such as avoided premature mortality and morbidity into dollars, for example, th e value of a statistical life (VSL), the discount rate, or cost of illness (COI) estimates.

 Model uncertainty

– Functional form: Due to explicit or implicit choi ce of functions, for example, CR functions or functional form fo r forecast of population growth.

2.3 Assess Ranges for Parameters and Alternative Functional Forms

After preparing a basic list of potential unc ertainties, OAQPS organized a workshop to provide advice on additional sources of uncertainty, approaches to quantifying and evaluating uncertainty, and the range of valu es or functional for ms that coul d be used in the analysis. The

### 2-3 workshop included experts from inside and outsi de EPA on the topics of population forecasting, monetary valuation of morta lity and morbidity endpoints, and techniques for assessing uncertainty. Appendix A contains a memo summarizing the work shop. EPA staff with expertise on the health impacts of PM 2.5 

### and staff involved with esti mating and forecasting baseline emissions and changes in ambient air quality we re also consulted. The team also conducted limited literature reviews, although a full literature review to update the information currently in BenMAP was outside the scope of this project (see discussion in Sect ion 6 on this topic).

After establishing the sources of uncertainty and classifyi ng their source and type, we selected which to treat quantitatively in the se nsitivity analysis. We included those parameters that appear to have signifi cant uncertainty, those model uncertainties that are explicit in BenMAP, and a few implicit model uncertainti es where we could identify a reasonable

alternative to that assumed in BenMAP.

Next, a plausible range was selected for each uncertain parameter. The range for each parameter includes three values:

Mid value: This is a central or nominal value us ed for initial analysis for comparison with BenMAP; usually the same values used for the BenMAP runs provided by EPA to calibrate the model. The values from BenM AP are not necessarily mean or median values, but in all cases the values are within the range we evaluate (between the low and high values). From a probabilistic perspective, the mid value might be the most likely (mode), mean, or median value. However, for most of the values we do not have information about the distribution, and we did not perform an e xplicit probabilistic assessment in this study.

Low value: This is a low value that is plausible, but not the smallest conceivable value or minimum. If we had a distribution, the low va lue might be about the tenth percentile—a value such that someone with expertise on the topic would be surprised if the true value turned out to be smaller. Wit hout a distribution, judgment was used to select values that seemed like plausible lower bounds.
High value: This is a high value that is plausible, but not usually the largest conceivable value or maximum. From a probabilistic pe rspective, it might be about the ninetieth percentile—a value such that someone with e xpertise on the topic w ould be surprised if the true value turned out to be larger. Without a distribution, judgment was used to select values that seemed like plausible upper bounds.

For a few inputs, notably the mortality CR f unctions obtained from expert elicitation, we have their uncertainty assessed explicitly in the form of probability distributions (IEc, 2006). For those quantities, we chose 10 th 

### , 50 th 

### (median), and 90 th 

### percentiles of those distributions as the low, mid, and high values for use in range sensitivity analysis. Assessing probability

### 2-4 distributions for other quantitie s and conducting a probabilistic sens itivity analysis was outside the scope of this project.

For model uncertainties, we compare the eff ects of two or more model formulations (functional forms).

2.4 Develop Influence Diagrams

The initial hierarchy of influence diagram s shows how uncertain input parameters and

model form choices influence the resulting output variables, including avoided premature mortality and net benefits. To avoid creating a single diagram of overwhelming complexity, we organized the diagrams using Analytica modules into a hierarchy of diagrams, each focusing on a particular aspect of the model. Figure 2-1 provides an example of the hierarchy of diagrams. Section 4 presents and discusse s the final influence diagrams.

Figure 2-1. Hierarchical Influence Diagram Showing the Top Level Model and Three Selected Subdiagrams

### 2-5 2.5 Use of Analytica

Analytica is a general-pur pose software environment for building, analyzing, and communicating quantitative m odels ( Analytica User Guide, Release 4.1 , Lumina Decision Systems, Inc. 2008. www.Lumina.com). We used An alytica to create the hierarchy of influence diagrams and to construct a simplified model of BenMAP that we used to conduct the sensitivity analysis described in this report. Analytica lets users draw in influence diagrams by creating and positioning nodes and drawing influence arrows to connect them. It provides a general mathematical modeling language that we used to c onstruct the model. It al so allows probabilistic modeling, defining variables with probability di stributions, and using Monte Carlo or Latin hypercube sampling to propagate uncertainties through the model. Alth ough full probabilistic analysis was outside the project scope, we did use the Monte Ca rlo facilities to obtained low, mid, and high values (as 10th, 50th, and 90th percen tiles) for the slope (beta) of mortality concentration-response functions that had been assessed as a combination of probability distributions as the product of an elicitation of a set of twelve experts (IEc, 2006).

2.6 Develop a Simplified Model to Conduct Sensitivity Analysis and Compare to BenMAP

The next step was to build a model in Anal ytica based on the initial influence diagrams that corresponds to the components of a BenMAP benefits analysis but is simplified so that it can run rapidly enough to perform extensive sensitivit y analysis. The reasons to build the simplified

model include:

 BenMAP is too large and takes too long to compute to be able to perform a large number of runs to support extensive sensitivity and uncertainty analysis.

 Some kinds of sensitivity analysis require modification of the model structure, which is much easier on a simplified model. Comparing some kinds of model form uncertainty and assessing the impreci sion introduced by aggregation are two examples.

Note that the simplified model is not a s ubstitute for BenMAP. The simplified model could only be built and calibrated because Be nMAP exists. The simplified model does not duplicate many of BenMAP’s functions, includi ng aggregating data in to grid squares and generating maps of inputs and results.

The key simplification is to reduce the geographic a ggregation from the 16,576 36km grid squares used for BenMAP runs, displayed in Figure 2-2, to a much sm aller set of bins. The input grid square data incl udes the population for 2020, annual average concentration of PM 2.5 

### at baseline, planned reductions in emissions, and baseline-incidence data. We aggregate the

### 2-6 Figure 2-2. 36 km Grid Squares over the Contiguous United States

population from the grid squares into bins, with a bin for each base concentration from 0 to 1, 1 to 2, up to 15 to 16 micrograms/cubic meter of PM 2.5 

### . In this way, we reduce the computational effort by a factor of about 1,000, from over 16,000 grid squares to just 16 bins. For each bin, we use the population-weighted average for the base c oncentration and the delta concentration in the simplified model.

2.6.1 Calibration to BenMAP

The Analytica version of the m odel approxima tes the results of Be nMAP, but it does not match the results exactly due to its simplifie d representation and there may be some small differences in the implementation of algorithms in BenMAP. These cause modest differences in the estimated number of cases reduced (morbidi ty and mortality) and the total benefits. To increase the accuracy of the sens itivity analysis, we calibrated th e Analytica model to selected

results of BenMAP for both number of cases of mortality and morbidity and the value of those
cases, for each of the CR functions. The calibra tion multiplies each Analytica result by a factor so that it produces identical results to the corresponding BenM AP runs, with two exceptions:

1. For Pope et al. (2002) CR function with threshold of 15 μ g/m 3 

### , the Analytica model generated zero cases, since the single grid square with base co ncentration above 15

μ g/m3 (in Los Angeles at 15.05 μ g/m3) was included in 15 μ g/m3 bin, with a population-weighted average base concentration of 14.87 μ g/m3, and so below the threshold.

### 2-7 2. For Expert K, who specified a likelihood of causality of 0.35, we used a mid value (50th percentile) of zero for beta, where BenMAP uses the mean value. Hence, in this case, the Analytica model produces zero mort ality for mid (nominal) analysis, where BenMAP uses a small positive value for beta.

This calibration process substa ntially improves the correspondence of the results of the sensitivity analysis with what would be obtained were the sensitivity analysis to be carried out

with full runs of BenMAP. The mi d (central values) ar e identical (with the above exceptions). The ranges used in the range sensitivity analysis , as perturbations around this mid value, should

be similar but not identical to what one would obtain if one performed these range sensitivity runs with BenMAP. Thus, the slight approximati on remaining for the range sensitivity results will have negligible on the main conclusions of this study—to identify the relative importance of the various sources of uncertainty.

2.7 Analysis Strategy

We use two m easures of sensitivity in comp aring the effects of each uncertain input X on the output variable Y:

Elasticity: The percent change in Y for a one percent change in X.
Output range: The lower and upper values of th e output corresponding to lower and upper value of input range, holdi ng all other inputs at their nominal values. The standard tornado chart displays output ranges.

These measures of sensitivity focus on one uncer tain input at a time, holding the others at their mid values. This makes the analysis fast an d easy, but it ignores any po ssible interactions in the effects of two or more inputs on the outputs. We can rule out many in teractions—or at least conclude that they are multiplicative (rather than additive)—just by inspecting the form of the equations, without having to run the model. Howe ver, there are some inte ractions that may be worth examining using the model, for example be tween the threshold and beta of CR functions.

We will also examine one or two interactions between parameter uncertainties and functional form uncertainties, notably in the f unctional form of the CR functions.

### 3-1 SECTION 3

CLASSIFY UNCERTAINTIES A ND QUANTIFY PARAMETE R RANGES

3.1 Final List of Uncertainties

The first step in the process involved identify ing a set of uncertain param eters and model assumptions. Table 3-1 lists the sources of uncertain ty identified for potential inclusion in initial sensitivity analysis. In the firs t column, the potential source of uncertainty is categorized as either a BenMAP choice (selected by user from choices in BenMAP) or as a BenMAP default (defined in BenMAP, the user can sometimes im port different data to change assumption). The next two columns identify the source and type of uncertainty using the categories defined in

Section 2.2.

Table 3-1. List of Major Sources of Uncertainty in the BenMAP Model

Modeling Step Uncertainty Source Uncertainty Type Current Formulation in BenMAP

CR FUNCTION AND ISSUES IN THE ESTIMATION OF ADVERSE HEALTH EFFECTS

1 BenMAP choice Functional form of the CR function Model Uncertainty, Functional form  Alternative functional forms are sometimes availableto represent CR relationship including linear, loglinear, logistic, and Cox pr oportional hazard models

2 BenMAP choice Threshold in the CR function Parameter Uncertainty,

Measured  EPA estimates a point estimate threshold below which there is no change in incidence rate associated withchange in air pollution concentration, e.g., for a loglinear CR model:

 Δ y=y 0 ×[exp( β ×(max(PM 1 ,T)

− max(PM 0 ,T)))-1]

where, T is the threshold

3
BenMAP

default Air quality metric in
BenMAP CR

functions Parameter Uncertainty,

Measured  Air quality metrics are matched to the epi studies. For pre-loaded functions, these are already defined in
BenMAP.

 For PM 2.5 , EPA currently uses CR functions that use a

quarterly average air quality concentration. BenMAP creates an annual average by averaging the 4 quarterly

averages

4
BenMAP

default Extrapolation below PM concentration in study Parameter Uncertainty, Extrapolate beyond range  BenMAP does not require that PM concentrations be within the range of PM concentrations used in the epi studies

5 BenMAP

default Current exposure and cumulative

exposure Parameter Uncertainty,

Measured  BenMAP assumes a pulse of exposure to pollutants in the analysis year. The CR functions may apply toshort-term exposure or long-term exposure. For epibased mortality studies, EPA uses the long-term CR functions. (continued)

### 3-2 Table 3-1. List of Major Sources of Uncert ainty in the BenMAP Model (continued)

Modeling Step Uncertainty Source Uncertainty Type Current Formulation in BenMAP

CR FUNCTION AND ISSUES IN THE ESTIMATION OF ADVERSE HEALTH EFFECTS

6 BenMAP default Equal toxicity of PM components Parameter Uncertainty  The functions in BenMAP assume equal toxicity for all PM 2.5 species. Assuming differential toxicity would require defining a new pollutant in BenMAP and obtaining air quality modeling or monitoring data for that species.

7 BenMAP choice Mortality— Coefficient ( β ) in the CR function Parameter Uncertainty,

Measured  EPA currently uses two epi studies for the relationship between exposure to PM and premature mortality over time (i.e., Pope et al., 2002; Laden et al., 2006).

 EPA also uses CR functions generated through an

expert elicitation. Other studies are available.

8 BenMAP choice CB—Coefficient ( β ) in the CR function Parameter Uncertainty,

Measured  EPA currently uses two studies for the relationship between exposure to PM and developing CB over time (i.e., Abbey et al., 1993)

 Parameter of the CR function (b) was specific using a

mean value and a standard error which can be used to form a normal distribution

9 BenMAP choice AMI— Coefficient ( β ) in the CR function Parameter Uncertainty,

Measured  EPA uses one study for the relationship between exposure to PM and AMI (i.e., Peters et al., 2001)

 Parameter of the CR function (b) was specific using a

mean value and a standard error which can be used to form a normal distribution

10
BenMAP

default AMI—Parameter in the CR function Parameter Uncertainty,

Measured  Only considers AMI’s for individuals who do not die within 28 days. Because 6% of male and 8% of female hospitalized heart attack patients die within 28 days, the function includes a correction factor of 0.93 (on average) to estimate the nu mber of non-fatal AMIs (Rosamond et al., 1999)

BASELINE INCIDENCE RATES AND PREVALENCE

1 BenMAP default Allocating baseline incidence to grid cells Model Uncertainty, Aggregation  Baseline incidence rates allocated to grid cells in BenMAP, not clear how

2 BenMAP default Mortality Baseline Incidence Rates Model Uncertainty, Functional form and aggregation  Age, cause, and county-sp ecific mortality rates were obtained from CDC for years 1996 through 1998 by county

 CDC’s age groups include <1, 1–4, 5–9, 10–14, 15–

19, 20–24, 25–34, 35–44, 45–54, 55–64, 65–74, 75– 84, and 85+

 Mortality rates are averag ed across three years,

assumes that rates are uniform ly distributed across all ages in the reported age group (continued)

### 3-3 Table 3-1. List of Major Sources of Uncert ainty in the BenMAP Model (continued)

Modeling Step Uncertainty Source Uncertainty Type Current Formulation in BenMAP

2 BenMAP default Mortality Baseline Incidence Rates Model Uncertainty, Functional form and aggregation  Age, cause, and county-sp ecific mortality rates were obtained from CDC for years 1996 through 1998 by county

 CDC’s age groups include <1, 1–4, 5–9, 10–14, 15–

19, 20–24, 25–34, 35–44, 45–54, 55–64, 65–74, 75– 84, and 85+

 Mortality rates are averag ed across three years,

assumes that rates are uniform ly distributed across all ages in the reported age group

3 BenMAP default Mortality—Year of baseline incidence Parameter uncertainty,

Measured  Age, cause, and county-sp ecific mortality rates were obtained from CDC for years 1996 through 1998 by county

 Mortality rates are averag ed across three years

4 BenMAP choice Mortality— Forecast baseline incidence rates Parameter Uncertainty,

Forecast  Project CDC Wonder coun ty and age-specific mortality rates for 1997 to future years using Census projections of national age-specific mortality ratesfrom 1999 through 2050. Adjust county and agespecific 1997 CDC Wonder rate by ratio of Censusprojection for year Y to the estimated Census agespecific mortality rate for 1997 (1997 Census mortality rate estimated by regressing mortality rates from 1999–2008 on year).

 All segments of the popula tion assumed to grow at

same rate

 User chooses the forecast year

5 BenMAP default AMI Baseline Incidence Rates Model uncertainty,
Parameter uncertainty  AMI incidence rates are based on regional hospitalization rates and hence inherit uncertainties associated with those rates

 Hospitalization rates in 1999 were estimated for four

regions based for 3 age groups (0-18, 18-46, 65+) using National Hospital Discharge Survey and population data.

6 BenMAP default CB Baseline Incidence Rates Parameter uncertainty  For ages 27+, the incidence rate is 0.00378 (Abbey et al., 1993)

 No other age group or geographic breakdown is

available

7 BenMAP default CB Prevalence Parameter uncertainty  Age group specific prevalence estimates are 18-44 (0.0367), 45-64 (0.0505), 65+ (0.0587). For ages 18+, the prevalence is 0.0443

8 BenMAP default Morbidity— Forecast baseline incidence and prevalence rates Model uncertainty,

Forecast  BenMAP contains year 2000 only, no projections currently available

### 3-4 In December 2008, EPA sponsored a workshop to provide expert feedback on the uncertainties used in the analysis, additional sources of uncertainty, the reasons for uncertainty and variability in parameter values, and to provide advice on actual ranges of values or on sources of data to derive ranges. The workshop participants included bot h EPA staff and outside experts in the areas of population forecasting, mortality valuation, morbidity valuation and uncertainty analysis. After the workshop, additi onal discussions took place with EPA health scientists and air qua lity modelers. Appendix A provide s more details on the workshop.

3.2 Quantify Ranges for Parameters

Following the workshop, we selected a set of uncertain param eters and model assumptions from the list in Table 3-1 for in clusion in the sensitivity analysis. For each parameter, we defined a range of values. Ta ble 3-2 presents the range for each uncertain parameter. We selected range s that represent plausible upp er and lower bounds, but not necessarily extreme values. In addition, we selected mid values. The mid values do not necessarily represent the “middl e” of the distribution, especially since we usually lack the

information needed to determine what the distri bution looks like. In most cases, the mid value will be consistent with the value from BenMAP for runs provided by EPA, which is provided in

the last column.

3.3 Source for Range Recommendations

1. VSL:

– Uncertainty in VSL estimates (calculated from an individual’s WTP to reduce the risk of mortality by a given percent) are associated with the stated and revealed preference methods used to calculate VSL, the data, and the assumptions used to make the calculation. VSL is also believed to be characterized by variability due to preferences, baseline age and he alth, and other personal and social characteristics.

– The range from $1,000,000 to $10,000,000 with a midpoint of $5,500,000 covers the range from most of the curren t studies and meta-analyses and was recommended by the experts who particip ated in the workshop. The midpoint represents the VSL that EPA Office of Air and Radiation used for benefits analysis between 2004 and 2009.
2. Elasticity of WTP with respect to income:

– Uncertainty in the income elasticity of WTP come from the methods and data used to estimate the values. In addition, elasticity estimates usually come from cross-sectional data, but they are being used to represen t change in elasticity over time.

– The range comes from the BenMAP ma nual and was supported by the experts who participated in the workshop.

### 3-5 Table 3-2. Range Values for Uncertain Parameters

Low Mid High Value from
BenMAP
Run

1. VSL $1,000,000 $5,500,000 $10,000,000 $5,500,000
2. Elasticity of WTP to income for mortality 0.2 0.40 1 0.40

3. Discount rate 1% 3% 7% 3%, 7%
4. Value of an avoided AMI, 3% discount

rate $11,000 $85,309 $372,000 Varies by age, years of

treatment
5. Value of an avoided case of CB, 3% discount on COI value $11,000 $340,481 $1,749,539 $340,481

6. Uncertainty in Census 2000, percent of baseline from BenMAP run 98% 100% 102% 100%

7. Uncertainty in Census 2020 population forecast 90% 100% 110% 100%

8. Threshold in mortality CR function 0 0 10ug/m 3 BenMAP runs used 0 and 10ug/m 3
9. Beta for mortality CR function with a threshold of 10ug/m 3 Low Mid High Different for each study and expert
10. Beta for mortality CR function with no threshold Low Mid High Different for each study and expert

11. Beta for CB CR function with no threshold 0.004993999 0.0137 0.02240599 0.0137

12. Beta for AMI CR function with a threshold of 10ug/m 3 0.02130709 0.033201 0.04509491 0.033201

13. Beta for AMI CR function with no threshold 0.012227397 0.024121 0.036015217 0.024121307

14. Base mortality rate, percent of value in
BenMAP 95% 100% 105% 100%

15. Incidence rate for CB 0.067872271 0.378 0.798218895 0.378

16. Incidence rate for AMI 0.1391 0.7747 1.6359
17. GDP growth rate 1.72% 2.34% 3.04%

18. Base PM 2.5 concentration, percent difference from BenMAP run 85% 100% 115% 100%

19. Change in PM 2.5 concentration, percent difference from BenMAP run 70% 100% 130% 100%

### 3-6 3. Discount rate: 3% and 7% are required for use in Federal regulatory analysis. The lower end of the range, 1%, was suggeste d because of the long time frame for the model forecasts.

4. Value of an avoided AM I, 3% discount rate:

– Uncertainty in the distribution of values for an avoided AMI is associated with factors such as the lack of WTP values (BenMAP includes only COI values), the age of the studies, the difficulty defining the costs associated with a single AMI separate from other conditions, the impact of co-morbidities on cost, the cost of potential complications after the AMI, and similar issues.

– The expert panel recommended a lower bound of one-half the COI and an upper bound of WTP plus COI. BenMAP currently uses only COI estimates for AMI’s. The COI measures include estimates of di rect medical costs and opportunity cost (lost earnings). In the absence of readily available estimates of WTP, we used the range from BenMAP studies for 5 year a nd 10 year length of costs discounted at 3% (Table H-5, USEPA, 2008).

i. Low: one-half the lowest COI in Table H-5

ii. Mid: the average COI in Table H-5 acro ss all ages (not including 0-24 age group)

iii. High: because there is not WTP value, th e high end of the range was set at two times the highest COI in Table H-5

5. Value of an avoided case of CB, 3% discount rate:

– The uncertainty associated with the valu e of an avoided case of CB comes from factors such as the age and sample of th e existing WTP studies and other factors similar to AMI’s.

– The range of values was derived from th e COI estimates from Table H-3 in the BenMAP User’s Guide and the WTP form ula in Section H.2.1.1 (USEPA, 2008).

i. Low: the lowest COI value in Table H-3 (which is similar in magnitude to one-half the lowest COI for an AMI)

ii. Mid: the median value of th e WTP function in BenMAP

iii. High: WTP value plus the highest COI value from Table H-3 (USEPA, 2008) for the 3% discount rate. WTP was calculated as follows:

1. The formula for WTP in the BenMAP manual is WTP(x) = WTP(13)*exp(-beta(13-x)) (H.2.1.1 USEPA, 2008)

2. x denotes severity and ranges from 1 to 12, where 1 is the least severe; beta is assumed to ~ N(0.18,0.0669). Th e WTP(13), the highest severity, come from Viscusi et al. (1991); estim ates of the beta coefficient and standard error come from Krupnick and Cropper (1992).

### 3-7 3. Using beta=0.18-0.0669 and x=11 and the WTP(13) for the 90 th 

### percentile, the formula above is used to calculate the WTP(x) for the high value. The value of WTP thus obtained is $1,595,117.

6. Census estimate for 2000:

– Uncertainty about national population estimate in 2000

– A range of minus and plus 2% was sp ecified based data from Census and population forecasts for future years. Population projections based on the 1990 Census include low, middle and high se ries estimates (U.S. Department of Commerce 1996). The difference between the high and low series in the projections for 1995 is less than .0001%. Comparing the 10-y ear projection of 2000 population to the populat ion counted by the 2000 Census (U.S. Department of Commerce 2002), the difference between the actual 2000 Census and the middle series forecast for 2000 is about 2.4%. The calculations suggest that the uncertainty in the population estimate for 2000 is low and within a range of 2% of the actual estimate.
7. Population forecast for 2020 and distri bution of population within the U.S.:

– Uncertainty in the population forecast fo r 2020 comes from variety of sources including immigration trends, economic growth, the birth rate and the death rate. The share of the U.S. population in a particular county will depend on similar factors.

– BenMAP uses the 2000 Census population es timate at the block group level that is aggregated and allocated to grid squares that matc h the air pollution data. The population forecast is based on county fo recast data from Woods and Poole (2007) and then allocated to grid squares that match the air pollution data in BenMAP. The Woods and Poole forecast data uses a model that includes economic growth projections. The share of national population in each county based on the Woods and Poole data is used to allocate the 2000 Census estimate across counties. Confidence intervals re ported by Woods and Poole are +/- 6.3% for metropolitan statistical areas and +/ - 4.7% for states based on comparing 10- year forecasts with actual values. Uncerta inty about whether pe ople will move to areas of the country that are more or less polluted will potentially affect the benefit estimates.

– To account for both the uncertainty in the 2020 forecast and the uncertainty in how the population will be spread out across counties, the 2020 population by grid square from BenMAP was adjusted +/- 10%.

8. Threshold in mort ality CR function:

– Guided by the expert elicitation and advice from EPA staff, the threshold for the PM/mortality CR function ranges fro m 0 (no threshold) to 10ug/m 3



### . 9. Beta for mortality CR function with a threshold of 10ug/m 3 

### :

– See Section 5.4 for details on calculating mid, low, and high values for beta for the different studies and experts.

### 3-8 10. Beta for mortality CR function with no threshold:

– See Section 5.4 for details on calculating mid, low, and high values for beta for the different studies and experts.

11. Beta for CB CR function with no threshold:

– Uncertainty in the beta arises from a l ack of studies, possible regional variation, and other sources.

– See Section 5.4 for details on calculating mid, low, and high values for beta for the different studies and experts.

– Beta based on Abbey et al. (1993) using the mean of 0.0137 and a standard deviation of 0.00679645, the low and high are the 10 th 

### and 90 th 

### percentiles of the distribution (1.281 standard de viations from the man) and the mid value is the

mean. 12. Beta for AMI CR function with a threshold of 10ug/m 3 

### :

– Uncertainty in the beta arises from a l ack of studies, possible regional variation, and other sources.

– Beta based on Peters et al. (2001). Us ing the mean of 0.033201 and a standard deviation of 0.009284863, the low and high are the 10 th 

### and 90 th 

### percentiles of the distribution (1.281 standard de viations from the man) and the mid value is the

mean.

13. Beta for AMI CR function with no threshold:

– Uncertainty in the beta arises from a l ack of studies, possible regional variation, and other sources.

– Beta based on Peters et al. (2001). Us ing the mean of 0.024121307 and a standard deviation of 0.009284863, the low and high are the 10 th 

### and 90 th 

### percentiles of the distribution (1.281 standard de viations from the man) and the mid value is the

mean. 14. Base mortality rate:

– Uncertainty arises from possible errors in data collection and compilation, estimating rates for specific population sub-samples and averaging rates across sub-samples.

– Assumed +/- 5% of base rate for mortality

15. Incidence rate for CB:

– Uncertainty arises from lack of studi es that estimate incidence rates.

– BenMAP cites Abbey et al. (1993) and Abbey et al. ( 1995) as the source for the incidence rate values discussed in E. 6.2 of the BenMAP manual (USEPA, 2008). We calculated the percentage change from low to mid to high using the range of AMI incidence rates and a pplied the percent difference to the mid estimate for CB.

i. Low: 82% of the mid value

### 3-9 ii. Mid: consistent with the value from BenMAP run

iii. High: 111% of the mid value

16. Incidence rate for AMI

– Uncertainty arises from lack of studi es that estimate incidence rates.

– Values come from Exhibit E-6 (which cites data from the 1999 National Hospital Discharge Survey and Rosamond et al . [1999]) in BenMAP manual (USEPA, 2008), we used highest and lowest va lues across regions and age groups (excluding 0-18)
17. GDP growth rate

– Used forecasts from Table 20 (Macroec onomic Indicators) of the Annual Energy Outlook 2009 (DOE, 2009). The mid, low and high values were calculated based on the forecasts from the reference case, low and high economic growth cases, respectively. The compound annual averag e growth rate (CAGR) in GDP (real dollars) was computed over the period 2008 to 2020 for these three scenarios. These GDP numbers were used to adjust real income growth from the income growth projections in BenMAP up or down for the two ranges.

18. Base PM 2.5 

### concentration

– Emissions and air quality forecasts were not part of the scope of this project. We assumed +/- 15% of base concentrat ions from BenMAP run to provide information on how uncertainty in the base concentrations might affect the range of outcomes.
19. Change in PM 2.5 

### concentration

– Emissions and air quality forecasts were not part of the scope of this project. We assumed +/- 30% of base concentrat ions from BenMAP run to provide information on how uncertainty in the base concentrations might affect the range of outcomes.

3.4 Alternative Functional Forms

We used alternative functional form assum ptions for:

 Lag structure for mortality valuation

 CR functions

### 4-1 SECTION 4

INFLUENCE DIAGRAMS

4.1 Introduction to Influence Diagrams

Influence diagram s trace how different sources of uncertainty affect different elements of the model and the final results. Figure 4-1 provides an illustrative influen ce diagram of a benefitcost analysis with the variable node types used in Analytica influence diagrams labeled. (This example is purely to illustrate the notation, and no t used in this analysis.) In this influence analysis, decisions (green recta ngles) identify user options, such as choosing the base year for dollars or forecast year. Uncertain parameters (l ight blue ovals) identify variables on which we perform sensitivity analysis. The model is organized as a hierar chy of modules, each with its own diagram. Some variables affect variab les in other modules. A node in italics (like Year in the submodule), is an alias , that is, a copy of a node whose orig inal appears in another diagram, shown to clarify dependency relations.

Figure 4-1. Illustrative Influence Diagram with Notation Defined

### 4-2 4.2 Influence Diagrams for Sensitivity Analysis

The top influence diagram, Figure 4-2, shows the key com ponents in BenMAP as represented in Analytica. Each node with a thick outline represents a module containing an influence diagram shown in more detail below. The influence arrows show how one module or node affects another, and ultimately affects the fi nal result, “Monetized benefits,” shown as a red hexagon representing the fi nal objective variable.

Figure 4-2. Top-Level Influence Diagram

The influence diagram for the PM 2.5 

### concentration, Figure 4-3, generates the base and delta concentration (change in concentration) in PM 2.5 

### for each concentration bin, using the base and delta for each bin computed from BenMAP inputs for each grid square, multiplied by an uncertain parameter with mid valu e 100% (the light bl ue oval nodes) to include uncertainty in these projections.

Figure 4-4 shows how the incidence rates ar e calculated. It is the submodule for “Incidence rates,” the first module in the top ro w of Figure 4-2. This figure includes the key variables determining mortality by bin—that is, the mortality rate for each level of base

concentration of PM 2.5 

### . It also includes the incidence rate for morbidity endpoints CB and AMI.

Figure 4-5 shows the influence diagram fo r the CR functions. The green node “CR function data from epidemiological studies” cont ains the data defining those CR functions. We extract the threshold and beta (along with uncertainty expressed as standard deviations) from Table 5-1. This is combined with thresholds and betas from the expert e licitation distributions,

### 4-3 Base PM25 concentration

Delta PM25 concentration Base PM25 conc % unc

Delta PM25 conc % unc

Base PM25 concentration by bin

Delta concentration



### Figure 4-3. Influence Diagram for PM 2.5 

### Concentration

Figure 4-4. Influence Diagrams for Incidence Rates

### 4-4 Figure 4-5. Influence Diagram for Health Impacts CR Functions

and low, mid and high betas for the CR function (t he three ovals representing the three uncertain parameters used for sensitivity analysis: beta’s for the mortality, CB, and AMI CR functions). Together the threshold and betas for all the CR f unctions feed into the incidence reduction node, also shown on the main Influence Diagram, Figure 4-2.

Figure 4-6 shows the influence diagram for the expert elicitation dist ributions, which feed

into the “Incidence reduction” module in Figure 4-2. The EPA c onducted an expert elicitation on the topic of the CR function for PM 2.5 

### and mortality (IEc, 2006). The data elicited from each expert was incorporated indi vidually into the analysis. The turquoise node “Expert CR functions” contains the parameters defining the pr obability distributions obtained from experts for mortality CR functions for PM 2.5 

### . The remaining nodes process these to generate probability
distributions, to extract mid beta and threshol d, combine segments for piecewise functions and compute low, mid, and high values for range sensitivity analysis as described below.

Figure 4-7 contains the influence diagram for the “Population” module in Figure 4-2. This module show how U.S. population in the for ecast year (2020) is projected from population from the U.S. Census in 2000 and the projected increase to 2020. For this simplified analysis, it assumes the same percent growth in population for each concentration bin.

### 4-5 Figure 4-6. Influence Diagram for Ex pert Elicitation Distributions

Select forecast
year Population in forecast year by bins Population in 2000

Total US population in forecast year Population in 2020

Census

years US pop by

census 2010-50 Pop CAGR to forecast year Population change factor

from 2020 Population by conc bin



### Figure 4-7. Influence Diagram for Population Projections

The influence diagram of valuation modules is presented in Figure 4-8, which feeds into the “Valuation functions” module in Figure 4-2. The diagram includes the key variables and modules used to compute the VSL, the value of avoiding a case of CB, and the value of avoiding an AMI for the forecast year adjusted for lags as necessary. The node “Value per case avoided by end point” combines the valuation of avoided mortality, CB and AMI.

### 4-6 Discount rate Value of a statistical life saved

VSL for forecast
year Elasticity of

WTP to income

GDP growth rate Income growth factor

Discount due to mortality lag VSL adjusted for lag Select forecast y ea r

Valuation of morbidity Inflation and
income growth Inflation factor to base year

Mortality lag Value per

c ase avoided

by end point

Calibrated value per

case



### Figure 4-8. Influence Diagram for Valuation Modules

The diagram in Figure 4-8 includes three sub- modules for inflation and income growth, valuation of morbidity, and the mortality valuation lag. Figure 4-9 is the influence diagram for inflation and income growth, which calculates inco me growth to forecast year for mortality and CB, and the inflation factor to base currency year from standard EPA tables of income growth and inflation index. The income growth factor applies to the WTP functions for mortality and CB, but not to COI functions. It provides two in come growth methods, one the same as BenMAP and the other on projections based on the ratio of GDP to population growth that allows us to use corresponding projections from the Annual Energy Outlook. The numbers below assume the
BenMAP method, unless mentioned otherwise.

Figure 4-10 is the influence di agram for computing the value per case avoided of CB and AMI based on income effect on morbidity (if us ed) and an inflation factor to base year.

### 4-7 Currency
year

EPA inflation index from 2000 EPA inflation

years EPA standard income growth

by endpoint
group EPA income

years EPA income end point group

Inflation factor to base year Income growth factor 2000 to baseline year Income growth for

mortality EPA Income growth 2000 to

forecast year

Income growth for CB Select forecast
year

Select income growth method

Model type



### Figure 4-9. Influence Diagram for Inflation and Income Growth

Figure 4-10. Influence Diagram for Va lue of Avoided Morbidity Endpoint

### 4-8 Figure 4-11, the influence diagram for mortality valuation lags, shows how the model calculates the effect of different assumptions ab out the mortality valuation lag structure. The mortality valuation lag structure defines the distribution of mortality across time based on different assumptions about the delay between e xposure and its effect on mortality. The discount rate over time means that future mortality valu ation is discounted when computing the present value at the time of exposure (or reduction in exposure due to improved air quality).



### Lag structures Mortality percentage by lag year

Lag year

Exponent

for lag Exponential decline by

lag year Select lag structure

Discount for exponential Discount rate Discount due to mortality lag

Figure 4-11. Influence Diagram for Mortality Valuation Lag

The different lag structures are presented in Figure 4-12. This chart shows the percentage of mortality that occurs in each of the 20 years following exposure for six lag structures. None or no lag means all mortality occurs in the year of exposure. 15-year means a ll deaths occur exactly 15 years after exposure. 20-year distributed means 20% of deaths occur in the year of exposure, 50% evenly distributed over years two to five, and 30% evenly distribut ed over the remaining 15

years.

BenMAP takes as an input the estimates or measurements of the current air quality and projections of improved air quality for a set of points or grid squa res over the area of interest, in this case the contiguous United States. The air quality estimates and pr ojections are produced by a series of models that proj ect emissions, atmospheric transp ort, and air quality, which are depicted in the influence diagram in Figur e 4-13. Most commonly, CMAQ provides the air quality projections to BenMAP. We provide this parent diag ram, Figure 4-13, to show the context of the analysis.

### 4-9 Figure 4-12. Mortality Lag

Figure 4-13. Influence Di agram for Air Quality

### 4-10 Analysis of the uncertainty in the air quality projections and the sour ces of uncertainty in those projections are outside the scope of this project. Accordingly, we selected an arbitrary range of uncertainties on the air quality—PM 2.5 

### concentrations—and improvements in air quality—delta reductions in PM 2.5 

### concentrations.

### 5-1 SECTION 5

RESULTS

5.1 Assumptions for Analysis

The sensitiv ity analysis results in this repor t are generated from the Analytica simplified

version of the BenMAP model. Th ey are calibrated to a specific run of BenMAP, so that the overall mortality and morbidity cases avoided, and the monetary benefits of those cases avoided exactly match the BenMAP run. See Section 5.4 for assumptions on the low, mid, and high

values for each CR functions. Section 3 describes how the ranges for the other parameters were

selected.

For simplicity, the primary results assume no mortality valuation lag—i.e., any mortality

reduction is assumed to occur in th e same year as the change in PM 2.5 

### concentration—except where we explicitly present the effects of other mo rtality valuation lag structures. The effects of discounting and mortality valuation lag change th e benefits of mortality reduction by the given mortality valuation lag factor (shown in Figure 5-17). Accordingl y, changing mortality valuation lag would not affect the relative width of range sensitivities that affect mortality.

5.2 Measures of Sensitivity Used in Analysis

We use two m easures of sensitivity to comp are the effects of each uncertain input X on the output variable Y:

 Elasticity: The percentage change in Y fo r a one percent change in X.

 Range sensitivity: The lower and upper values of the output corresponding to lower and upper value of input range, holding all ot her inputs at their nominal values. The standard tornado chart di splays output ranges.

Most of our sensitivity results are based on range sensitivity, si nce, as its name implies, it is sensitive to the range of uncertainty (low to high) for each uncertain variable. However, elasticity does provide some in teresting insights—it depends only on the mathematical structure of the model and the mid values of the inputs, bu t ignores their range of uncertainty. For a model

y = F(x),

the elasticity of y with respect to x is

Elasticity(y, x) = [F(x + dx) / (F(x)] / [(x + dx)/dx]

### 5-2 To provide some insight into this m easure, consider these two examples:

Elasticity additive example Total_benefits = Mortality_bene fits + Morbidity_benefits

With values: $100M = $90M + $10M

Elasticity(Total_benefits, Mo rtality_benefits ) = 90%

Elasticity(Total_benefits, Morbidity_benefits) = 10%

Thus, for a simple additive model, the elasticity of the result is equal to the value of each variable added as a perc entage of the total.

Elasticity multip licative example This multiplicative example is a stylized mode l of the computation of mortality benefits, assuming a linear, no-threshold mortality function, with change in concen tration (Delta_conc), slope (Beta_CR), and value of a statistical life (VSL): 1



### Mortality_benefits = Population × Mortal ity_rate × Delta_conc × Beta_CR × VSL

Elasticity(Morta lity_benefits, Population) = 100%

Elasticity(Mortalit y_benefits, Mortal ity_rate ) = 100%

Elasticity(Mortality_benef its, Delta_conc ) = 100%

Elasticity(Mortality_ben efits, Beta_CR ) = 100%

Elasticity(Mortality_b enefits, VSL) = 100%

The interesting point here is that for a purely multiplicative model, the elasticity of the product to each variable is 100%.

5.3 Elasticity Results

Figure 5-1 s hows the elasticity of mortality to uncertain inputs, for the Pope et al. zero threshold CR function. The elasticity values we re computed using the Analytica function Elasticity(Y, X). The function changes the valu e of X from x1 to x2 by multiplying it by

1 BenMAP and the Analytica model actually use a log-linear mortality CR-function, not this simplified multiplicative form.

### 5-3 Figure 5-1. Elasticity of Mortality Parameters

(1 + 10 − 8 

### ) and computes the resulting value of Y, changing from y1 to y2. It returns the ratio (y2 – y1)/(x2 – y1). Thus, it computes elasticity based on the mid values , ignoring the range on the parameters. The elasticity is 100% with respect to Delta PM 2.5 

### (reduction in PM concentration), population in 2000, population in 2020, beta for the mortality CR function, base mortality rate, and value of a st atistical life suggests that the model is almost exactly purely multiplicative for these variables. (Note that th e population in 2020 is actu ally a multiplier for growth from 2000 to 2020.) It is inte resting also to see that the el asticity is zero with respect to

the base PM 2.5 

### concentration and threshold.

Figure 5-2 shows the elasticity of total monetized benefits (including mortality and morbidity reduction) to each uncertain parameter, using the exponential de clining lag structure. Again, elasticity with respect to Delta PM 2.5 

### , population in 2000 a nd 2020 is still 100%, indicating that total benefits are directly proport ional to these values. The elasticity to Beta on mortality function, base mo rtality, and value of a statistical li fe are 93%. These parameters are multiplicative factors for the bene fits of mortality reduction, whic h is about 93% of the total benefits. The elasticity of the parameters for morbidity—betas, incidence rates, and WTP to avoid—are each about 2% for AMI and 5% for CB, adding up to 7% to account for the remaining total benefit. The GDP growth rate af fects the income and hence VSL via the income elasticity. The discount rate has a negative elasticity because a higher discount rate reduces the benefits of reducing mortality given th e exponential declining lag structure.

### 5-4 Figure 5-2. Elasticity of Total Monetize d Benefits to Each Uncertain Parameter

5.4 Results for Different Concentration- Response Functions for Mortality and Morbidity

CR functions give the response to changes in mortality or morbidity as a function of changes in the atmospheric concen tration of air pollution. In this project, we focus on particulate matter finer than 2.5 microns in diameter (PM 2.5 

### ). BenMAP uses a wide variety of CR functions. All use a log-linear function, some with and so me without a threshol d concentration below

which PM 2.5 

### is assumed to have no health effect. Fo r some cases, the choice of a threshold affects the beta slope, because the regression requires a higher slope to account for observed mortality when assuming a threshold below whic h there are no effects. These studies include several based on epidemiological st udies, notably Pope et al. ( 2002) and Laden et al. (2006) for mortality CR functions, Abbey et al . (1993) for CB and Peters et al . (2001) for AMI. They also include probability distributi ons assessed by twelve experts, who remain anonymous and are identified by letters A through L. In this secti on, we describe the low, mid, and high values for each CR function for use in the range sensitivity analysis.

5.4.1 Ranges for Concentration-Response F unctions from Published Epidemiological Studies for Mortality a nd Morbidity

Table 5-1 provides information on the range va lues from the publishe d studies, including the end point, minimum age of population affected, threshold in μ g/m 3 

### , mean and standard deviation (SD) for beta parameter of the lo g-linear function of the CR functions from the epidemiological studies.

### 5-5 Table 5-1. Ranges for Epidemiological Studies

Note: The suffix "m" after a number means it is multiplied by 10 -3



### We set the mid value for each beta equal to its mean. The low and high values are

selected as the 10 th 

### percentile and 90 th 

### percentile of a normal dist ribution with specified mean and standard deviation—which are plus or mi nus 1.281 standard deviations from the mean.

Figure 5-3 shows the range (low, mid, and high) values for the CR functions for mortality and morbidity from selected epidemiological st udies. The low (left of red bar) and high (right edge of green bar) valu es correspond to the 10 th 

### and 90 th 

### percentiles of the normal distributions
estimated for the beta values, used for range sensitivity analysis.

5.4.2 Ranges from Expert Elicitations of Probability Distribution s on CR Functions

In 2006, EPA commissioned a study to estimate CR functions for PM 2.5 

### from 12 experts. These estimates were elicited in the form of pr obability distributions (IEc, 2006). Table 5-2 from the Analytica model reproduces the value from Exhibit 3-9 “Summary of expert subjective uncertainty distributions for C-R coefficients” (page 3-30 IEc, 2006). Some experts (B, F, K, and L) assessed a piecewise log-li near function with two segments (pieces) and a different distribution on beta on each segm ent. The cutoff value is the PM 2.5 

### concentration (in ug/m 3 

### ) where the segments change from one to the ot her distribution. Where th e expert specified a
custom distribution, we fitted a continuous distribution to the specified minimum, 5 th 

### , 25 th 

### , 50 th



### , 75 th 

### , 95 th 

### , and maximum points on each distribution using a piecewise cubic on the cumulative distribution (using Analytica’s Cu mDist() function). The normal dist ributions were fitted to the

5 th 

### and 95 th 

### percentiles and the triang ular fitted to min, median and max for distributions specified in those forms.

### 5-6 Pope et al 0 Pope et al 7.5 Pope et al 10 Pope et al 12 Pope et al 15
Laden 0
Laden 10
CB 0
CB 7.5

AMI 0

AMI 7.5

0.00 0.01 0.02 0.03 0.04 0.05


### Ranges for CR betas

Low mid and high

Low High



### Figure 5-3. Ranges of Beta for CR Functions

In Table 5-2, “Prob causality” is the likeli hood that each expert assigns to the hypothesis that human exposure to PM 2.5 

### in ranges up to 30 μ g/m 3 

### causes significant mortality. “Includes causality ” is “Y” or yes for experts that include d this probability in their probability

distributions.

For the other experts that did not include causality in the dist ributions they assessed, we treated the complement of the pr obability of causality as a proba bility mass that beta is zero, which we combined with the full distribution in the simulation. We used a Latin Hypercube sampling with 10,000 sample points to represent the combined di stributions. We used these samples to estimate the 10 th 

### , 50 th 

### (median), and 90 th 

### percentile from these distributions, combining the continuous distribu tion with a probability of causal ity when not included in the original distribut ion assessment.

Only expert K expressed a belief that there might be a threshold, w ith a 50% chance that

there is a threshold and 10% chan ce that it is greater than 5 μ g/m 3 

### . Accordingly for the range sensitivity, we set the low and mi d value to 0 (which is the 10 th 

### and 50 th 

### percentile of the threshold distribution) and the high value (90 th 

### percentile) to 5 μ g/m 3



### .

### 5-7Table 5-2. Probability Distributions Elicited from Experts A through LSource: IEc, 2006.

### 5-8 Figure 5-4 shows the range (low, mid, high) values for the CR functions for mortality assessed by experts. For experts B, F, K, and L, there are two ranges since the experts assessed piecewise log-linear functions with two segments. The low (left of red bar) and high (right edge of green bar) values correspond to the 10 th 

### and 90 th 

### percentiles of the distributions assessed by each expert for the beta values, used for range sensitivity analysis. Expert K has a mid and low value of zero reflecting his or her likelihood of causality of only 0.35.

Figure 5-4. Ranges of Beta for CR functions from Experts A to L

Figure 5-5 provides a chart show ing the range of cases of mo rtality or morbidity avoided for each CR function from the published epidemiol ogical studies and from the expert elicitation. The chart includes epidemiological studies (Pope et al. and Laden et al.) and the twelve experts A to L. These ranges reflect the ranges in the beta of the CR functions shown previously. The numbers after Pope et al. and Laden et al. specify th e threshold of PM 2.5 

### in μ g/m 3 

### . Low (left edge of red bar), mid (edge between red and green ba r) and high (right edge of green bar) are the 10 th



### , 50 th 

### , and 90 th 

### percentiles of probability distribution. For Pope et al. and Laden et al., these are distributions from the reported st andard deviation of the estimates of beta. For the Experts A to
L, these distributions are those estimat ed by the expert, as described above.

### 5-9 Figure 5-5. Range of Avoided Premature Mort alities and Morbidities (CB and AMI) for Low, Mid, and High Values for Each CR Function

Pope et al. 15 (with a threshold of 15 μ g/m 3 

### ) has zero mortality and so zero reduction in mortality. BenMAP input data did not contain grid squares with current PM 2.5 

### concentration of greater than the threshold of 15 μ g/m 3 

### . Expert K assessed the likelihood of causality as only 0.35.
So, the 10 th 

### and 50 th 

### percentiles of his or her distributi ons on beta of CR function are zero. Only the high value, the 90 th 

### percentile (right edge of green bar) is nonzero

Figure 5-6 shows the range of monetized bene fits from reduced mortality or morbidity for each CR function. Each range (low, mid, and hi gh value) corresponds with the low, mid, and high value for each CR function estimated as desc ribed above, holding all other parameters at

their mid value. The mid values match th e value from the BenMAP calibration run.

### 5-10All CRfunctions



### Figure 5-6. Range of Monetized Benefits fo r Low, Mid, and High Values for Each CR Function

Finally, Table 5-3 provides the number of cases of mortality or morbidity avoided with the low, mid, and high value for the beta from the CR function for each epidemiological study or expert, where the number after the name of the study or expe rt specifies the threshold.

### 5-11 Table 5-3. Number of Cases of Mortality or Morbidity Avoided Using the Low, Mid, and High Values for the Beta from th e CR Function in Each Source

Source Low Mid High
"Pope et al 0" 1,957 3,727 5,500 "Pope et al 7.5" 1,935 3,685 5,438 "Pope et al 10" 1,391 2,652 3,917 "Pope et al 12" 973 1,856 2,743 "Pope et al 15" 0 0 0 "Laden 0" 6,124 9,521 12,930 "Laden 10" 3,872 6,023 8,186 "CB 0" 961 2,631 4,290 "CB 7.5" 952 2,605 4,248 "AMI 0" 4,461 8,768 13,040 "AMI 7.5" 7,675 11,912 16,116 "Expert A" 5,837 10,268 16,791 "Expert B" 820 7,426 15,934 "Expert C" 5,065 7,691 11,541 "Expert D" 2,308 5,661 8,303 "Expert E" 9,212 12,820 17,786 "Expert F" 4,374 5,955 8,282 "Expert G" 0 5,992 7,582 "Expert H" 1,737 4,479 11,368 "Expert I" 2,503 8,173 12,434 "Expert J" 2,680 5,763 11,182 "Expert K" 0 0 3,191 "Expert L" 0 5,159 9,623



### 5.5 Range Sensitivity of Mortality Avoided Outcome

The following charts show the results of th e range sensitivity analysis on avoided premature mortality. Each chart lists the uncertain parameters down the left axis. The width of each bar shows the effect of changing each parameter from its low value (left edge of red bar) to mid value and high value (right edge of blue bar). A wider bar means higher range sensitivity.

The range sensitivity results in Figure 5-7 use the Pope et al. zero threshold as the CR function for mortality. Delta PM 2.5 

### and beta have the largest range sensitivities. There is no sensitivity to changes in the ba se concentration or th reshold, since this CR function assumes a fixed zero threshold. Figures 5-8a and 5-8b shows the range sensitiv ity for Pope et al. with a 10 and 12 ug/m 3 

### thresholds, respectively. Because of the threshold, the uncertainty in the base PM 2.5 

### concentration has a significant eff ect, larger even than the Delta PM 2.5 

### in this case, because it shifts population in some grid squares below th e threshold for effects. Compared to the zero threshold case, the numbers of avoided premature mortalities is much lower.

### 5-12 Figure 5-7. Range Sensitivity of Avoided Morta lity for Pope et al. wi th Zero ThresholdUncertainparameters formortality



### Figure 5-8a. Range Sensitivity of Avoided Mortality for Pope et al. with a 10 ug/m 3



### Threshold

### 5-13 Figure 5-8b. Range Sensitivity of Avoided Mortality for Pope et al. with a 12 ug/m 3



### Threshold

Figure 5-9 shows the range sens itivity of avoided mortality for Laden et al. with zero threshold. The sensitivity is similar to that for Po pe et al. for zero threshold in terms of relative size of range sensitivities. Again there is no effect of Base PM 2.5



### .

Figure 5-9. Range Sensitivity of Avoided Morta lity for Laden et al. with Zero Threshold

Figure 5-10 shows the range sens itivity if we select a diffe rent Pope et al. CR function assuming a mid threshold of 7.5 μ g/m 3 

### . Thus, there is range sensitiv ity to threshold, with a high threshold (blue) of 10 μ g/m 3 

### leading to lower cases avoided, and a low threshold (red) 0 increasing the cases avoided.

### 5-14Uncertain parameters formortality



### Figure 5-10. Range Sensitivity of Avoided Mortal ity for Pope et al. with Mid Threshold of 7.5 μ g/m 3



### Range sensitivity of avoided mortality for Laden et al. with 10 ug/m 3 

### threshold, Figure 5-11, is similar to that for Pope et al. for high thresholds. Range sensitivity of Base PM 2.5 

### is close to that for Beta.Uncertainparameters formortality



### Figure 5-11. Range Sensitivity of Avoided Mortality for Laden et al. with 10 ug/m 3



### Threshold

### 5-15 5.6 Range Sensitivity for Total Monetized Benefits Outcome

Figure 5-12 shows the range sens itiv ity of total monetized benefits to the full range of uncertain parameters for Pope et al. with zero threshold as the mid value. The zero threshold means no range sensitivity to Base PM 2.5 

### concentration. In this case, a higher threshold (based on the Pope et al. CR function adjusted for the highe r threshold) leads to lower monetized benefits. The largest range sensitivity is the statistical value of a life, followed by Beta for mortality CR
function, and then Delta PM 2.5 

### concentration. All the other range sensitivities are significantly
smaller, with the value of an avoided case of CB being the largest. The discount rate has no effect because we are assuming no mortality valuation lag.

Figure 5-12. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Pope et al. with Zero Threshold as the Mid Value

Figure 5-13, like the previous one, assumes th e Pope et al. zero threshold CR function. The only difference is that we selected an exponen tial declining mortality va luation lag structure. This means that the results are slight ly sensitive to the discount rate.

### 5-16 Figure 5-13. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Pope et al. with Zero Thres hold as the Mid Value with an Exponential Declining Mortality Valuation lag Structure

Figure 5-14 uses the Laden et al. zero threshol d as the mid mortality function. It changes to the Laden et al. 10 threshold with high threshol d. The relative sizes of the range sensitivities is identical to the previous chart using Pope et al. mortality func tion, but the absolute values are more than twice as large—$70 billion instead of $29 billion for mid values.

The final two range sensitivity charts are for experts E and K, who assessed the largest and smallest, respectively, of the mid values for the beta of the mortality CR functions. The chart in Figure 5-15 shows range sensitivities using Expert E. There is no sensitivity to threshold or

Base PM 2.5 

### , since Expert E has no threshold. The mid total benefits are $9 6 billion. Otherwise the pattern of the relative sizes of the range sens itivities is similar to Pope and Laden et al. CR functions. Figure 5-16 shows range sensitivities us ing Expert K. The mid value is only $2 billion per year, which is the benefits of reducing morb idity only, since Expert K’s mid value (median) for the CR beta is zero. The zero mortality reducti on leads to zero range sensitivity for VSL. For the high value of beta, the total benefits exceed $26 billion.

### 5-17 Figure 5-14. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Laden et al. Zero Threshold as the Mid Value

Figure 5-15. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Expert E

### 5-18 Figure 5-16. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain Parameters for Expert K

Finally, uncertainty about the valuation lag structure for mortality has an impact on the outcomes. Figure 5-17 presents the impact on the total outcome of differ ent assumptions about the valuation lag structure and about the discount rate. Figure 5-18 shows the range sensitivity of

total monetized benefits to different discount rate assumptions for each lag structure. This graph is for Pope et al. with no threshold. The high disc ount rate (7%) at left of blue bar gives lower benefits, and low discount rate (1%) at right of red bar gives higher benefits.

### 5-19 None
8-year
15-year
20-year distributed

5-year distributed
Exponential declining

0% 100% 20% 40% 60% 80%

Discount due to mortality lag
Discount rate (%)

1 3 7



### Figure 5-17. Discount by Lag Structure

Figure 5-18. Range Sensitivity of Total Moneti zed Benefits to Discount Rate Assumptions for Different Mortality Valuation lag Structures

### 6-1 SECTION 6

DISCUSSION AND POSSIBLE NEXT STEPS

6.1 Discussion of Model Results

This repor t presents the results from our initia l sensitivity analysis of the benefits from air quality regulations as estimated by BenMAP. The study examined the sources of uncertainty, the possible range of uncertainty for selected sources, the role and influence of the parameters or functions, and the sensitivity of th e final benefit estimates to the different sources of uncertainty all in the simplified Analytica version of the BenM AP model. Here we disc uss the results of the sensitivity analysis and suggesti ons for potentially useful next steps to explor e the role of uncertainty on benefit estimation.

 Comparing the results from the different mortality CR functions, the differences between the functions are larger than the uncertainty of individual CR-functions. In other words, the uncertainty inherent in choosing a CR-function has more impact on results (mortality reduction or total benefits) than the uncertainty within each CRfunction captured by its range sensitivity.

 The elasticity analysis finds that the uncertain parameters delta PM 2.5 

### concentration, the beta for the mortality CR function, population in 2000 and 2020, base mortality rates, and VSL all affect the total mo rtality reduction in an almost purely multiplicative fashion (Figure 5-1). Hence, th e relative sizes of the range sensitivities (ratios of their widths) of these parameters are similar for each expe rt. Hence, most of the key results below on the relative range sensitivity are the same for all mortality CR-functions even if the mid points—mortal ity avoided and total net benefits—differ significantly from one expert to another (e xcepting Expert K because his mid estimate has zero mortality due to low probability of causation).

 In all cases (again, except Expert K), the parameter with largest range sensitivity is the VSL.

 For most of the results from the differe nt mortality CR functions, the beta for mortality has the second larges t range sensitivity analysis.

 The range sensitivity to delta PM 2.5 

### , the reduction in concentration of PM 2.5 

### , is the third largest, and in some cases (experts E and F) comparable to the range sensitivity for the morality CR function beta. Examination of the uncertainty inherent in projections of air quality and improveme nts in air quality was beyond the scope of this project, so we assumed a plus or minus 30% change in this variable to illustrate the possible effects of uncertainty on the resu lts. Again it is not surprising that this quantity has a high sensitivity.

### 6-2  Among the quantities with lower range sens itivity, the most se nsitive include

– Population in 2020—i.e., uncertainty in population projections and the distribution of the population across th e country. Migration and immigration ranked high as sources of uncertainty in the population forecasts and population distribution because of the difficulty in predicting movements of people and the difficulty in forecasting economic activity many years into the future according to experts consulted for the project.

– The value of an avoided case of CB, whic h is highly uncertain at present. The value of avoiding an AMI is also highly uncertain and if WTP studies existed for AMI’s, AMI’s might exhibit larger and more influential range sensitivity.

– Elasticity of WTP to income.

 Discount rate is sensitive when using mort ality valuation lag structures, especially those with longer average lag—and irrelevant with no mortality lag. It is a policy variable usually set by the government for purposes of public policy analysis (typically at 3% or 7% per ye ar) rather than a quantity subject to empirical study. So, it is not clear that further study of this issue would be of value.

6.1.1 Comments on the Uncerta inty Analysis Methods

The developm ent of the influence diagrams in Section 4 provided an intuitive depiction of how the uncertain parameters combine and a ffect the results of interest. These diagrams reflect the actual Analytica model used for the numerical sensitivity analyses. In some cases, these diagrams include variables that are needed for the sensitivity analysis, but perhaps do not add clarity about the influence fr om a purely qualitativ e perspective. In support of transparency of our methods, we chose to display diagrams that reflect all the variables in the underlying quantitative model.

Creating a simplified version of a complex m odel, such as BenMAP, provides a depiction of its essentials, but we did not attempt to repr oduce all the computations in BenMAP, which we felt would detract from clarity and add unhelpful complexity.

The diagrams in Section 4 also do not include the model elements involved in processing and aggregating the 36-km grid square data to the concentration bins , calibrating to BenMAP results, and performing the sensi tivity analysis since th ey do not add clarity to the representation of the essentials of the benefits analysis. All th ese are available in the Analytica model for those

that are interested.

We developed the extremely simple model with 16 concentration bins to make sure that the model would run fast, enabling us to do a wide variety of sensitivity runs. At the end of the project, we also did some tests using the full 36 km grid square data in Analytica. It turns out that

### 6-3 it takes about 1 to 5 seconds for the full analysis, which is faster than BenMAP by 2 or 3 orders of magnitude. We suspect this may be partly due to Analyti ca's high-speed al gorithms for dealing with large arrays, but we would need to do more extensiv e analysis of both models to discover the full reasons. This ope ns the possibility of doing sensit ivity or Monte Carlo analysis

of BenMAP at the full 36-km level of detail using the Analytica model in a future study if

needed.

6.2 Potential Next Steps

Below we discuss potential next steps based on the results f rom this uncertainty analysis. Based on research conducted by EPA and others, so me of the largest sources of uncertainty are not good candidates for additional work. EPA has already devoted considerable resources to characterizing the uncertainty in the CR functi on for mortality, one of the most influential sources of uncertainty. Additiona l expert elicitation does not seem warranted without more and more conclusive empirical evidence. A similar conclusion applies to VSL. VSL is also much
studied, and it is not clear that further study would lead to any significant reduction in its uncertainty.

6.2.1 Update Studies to Support Morbidity Valuation

Additional work on m orbidity outcomes may o ffer important data for BenMAP. Without additional epidemiology studies, WTP values and upda ted studies, it is difficult to know whether the sensitivity analysis captures the full range of values. Th e range sensitivity to WTP to avoid CB is large ($77K to $1.6 million). The magnit ude of the COI component is very small compared to the WTP. This highlights the im portance of incorporating WTP into morbidity

valuation. The CB WTP studies ar e both old and should be update d. Other morbidity endpoints, such as AMIs, have not been the subject of WTP studies. It may be worthw hile reviewing recent literature and investing in new studies to obtain updated estimates of COI and/or WTP for all health effects. The income elasticity of WTP is another parameter that could use additional

research.

In general, much of the valuation data in the current version of BenMAP is potentially out-of-date, especially with changes over the la st decade or two in healthcare and in methods
used to estimate WTP and COI. Thus, there are both (i) uncertainties in the data itself and (ii) in applying them. An example of the former in cont ext of COI would be th at newer estimates may

reflect the advent of newer techno logies that result in shorter hospi tal stays and lower severity of illnesses. Also, individuals may be more educat ed and thus may undertake better preventive and follow-up care. On the other hand, overall increased costs of hea lth care may lead to increased

### 6-4 estimates of COI. True COI estimates for differe nt illnesses in recent years may be smaller than

those in the studies found in Be nMAP or they may be larger.

Uncertainties that stem from applying olde r estimates can also arise since current circumstances may be completely different from those of the studies. For example, the age composition (both current and projected) of highly polluted areas may be different from the ones that were used in the studies underlying BenMAP . This might create uncertainties associated with applying estimates to dem ographic groups that ar e different from the ones included in the study. For example, an older population will be at a higher risk to certa in diseases and also experience more severe symptoms.

The current version of the influence model creates “bins” based on pollution concentrations. One way to assess the sensitiv ity of results to diffe rent demographic groups might be to “bin” by demographic characteristics.

6.2.2 More Detailed Mortality Valuation Estimates

Ideally, W TP and COI estimates would be mo re tailored to specific populations. For example costs should reflect more regional differen ces in healthcare utiliz ation. Patients with comorbidities may have different costs than ot herwise healthy indivi duals who develop CB. Trajectories of costs for different outcomes should reflect common comp lications or sequela,

health conditions that arise af ter an injury or disease.

6.2.3 Framework to Jointly Consid er Multiple Health Endpoints

BenMAP incorporates infor mation on inci dence rates, COI a nd WTP from studies considering each health endpoint separately rather than in conjunction with other co-morbidities. This has two implications. First of all, there ma y be biased estimates due to confounding factors. Also, the general framework and associated assump tions differ across these studies. A review of recent literature to consider studies explori ng multiple endpoints (for example, work being conducted by Cameron and DeShazo on WTP that incorporates multiple outcomes) might provide ideas.

6.2.4 Add Additional Outcomes and Expand Application to Other Pollutants

This study looked at only m ort ality, CB and AMI. Incorporating other outcomes, both health and non-health (such as visibility) would complete the sensitivity analysis. Expansion of the uncertainty analysis to cover some of these additional benefits would require extensions of the benefits model beyond BenMAP.

### 6-5 The results presented in this report arise from a particular air quali ty scenario (the 2006 PM RIA 15/35 option). Application of the model to alternative pollutants, such as ozone, additional regulatory scenarios for PM or to di fferent species of PM were all suggested as possible topics for exploration th at would provide some evidence on how far the results from one specific scenario can be generalized to ot her pollutants or air quality regulations.

6.2.5 Expanding Scope of the Uncertainty Analysis to Include Air Quality

The project scope im posed limitations on the study of uncertainties by excluding uncertainty in the base concentra tion and delta concentration in PM 2.5 

### . These concentrations are estimated and projected by a vari ety of emissions and air quality models, including CMAQ, as we briefly showed in Figure 4-13. An explicit un certainty analysis of the air quality models would be a significant effort, probably larger than the uncertainty analysis of BenMAP undertaken here, due to the grea ter complexity of the emissi ons and atmospheric transport models. In the absence of careful analysis of the uncertainty in the air quality estimates, we chose ranges for these quantities, plus or minus 15% fo r the base concentration, and plus or minus 30% for the deltas. The range sensitivity showed lo w sensitivity to the ba se concentration for nonthreshold CR functions and range sensitivity to the delta was usually third the largest (beta on the mortality CR function and VSL).

A more thorough uncertainty analysis of benefits would require a more careful assessment of the ranges for the base and delta c oncentrations—based on analysis of uncertainty in projections using a simplified model of the ai r quality models, using an approach similar to

that conducted here for BenMAP. It should also include uncertai nty analysis based on comparison of air quality projections with actual observations, something that is not directly possible for benefits calculations.

6.2.6 Investigate Cost Uncertainty

The study does not address the costs of i mprovi ng air quality. Explicit treatment of costs would require uncertainty analysis of the em issions models used to generate emissions inventories used by CMAQ. The addition of econo mic costs would require uncertainty analysis

of the costs of reduced emissions.

6.2.7 A More Comprehensive Treatment of Uncertainties

This project focused on range sensitivity analysis, varying each of the uncertain

param eters one at a time while leaving the othe r parameters fixed at their mid values. This approach does not assess the overall uncertainty in the results from the combined effects of all the uncertainties. One approach would be to exam ine the results when you set all parameters to

### 6-6 their low values—or all to their high values. This "Murphy's law" approach assumes that "everything that can go wrong doe s go wrong"—i.e., all parameters are at their "worst" level simultaneously. Conversely, everything goes right— all parameters are at their best level simultaneously. But the chance of all paramete rs being at one extreme simultaneously is vanishingly small: Given the 17 parameters (in the analysis here), if we assumed that the low value is the 10th percentile, the chance of all 17 being at or less than their low value simultaneously (or all at their high va lue) assuming independence would be 10 -17 

### . Hence, the resulting range would be implausi bly wide. It is for this reason we did not perform and do not recommend this approach.

A second limitation of the project scope was the exclusion of a pr obabilistic analysis. A full probabilistic analysis using Monte Carlo simulation would be the best approach to examine the combined effect of the multiple uncertainties on results. This approach would require an explicit probability distribution be assessed for each parameter—as they already were for the CR-mortality functions by experts A through L. It would also require interdependencies between parameters to be modeled. If EPA desires to obt ain a more complete uncertainty analysis, we would strongly recommend this approach.

6.2.8 Large Changes in Population, Migration and Immigration

Looking back on past im migration or internal migration patterns might provide bounds on the possible size of large immigration or migr ation events. One could develop scenarios that resulted in large movements of population or in creased/decreased immigr ation in response to very low probability events that have a large impact (hurricanes, droughts, changes in climate, political instability).

6.2.9 Additions to Analytica Model

As discussed in this report, the Analyti ca m odel is a simplified version of BenMAP. Further work to add more of the complexity fr om BenMAP into the Analytica model and using the full 16,000 points rather than bins removes some assumptions and increases the range of problems that can be assessed with the simplified model.

### 7-1 SECTION 7

REFERENCES

Abbey, D. E., F. Petersen, P. K. Mill s and W . L. Beeson. 1993. Long-Term Ambient Concentrations of Total Suspended Part iculates, Ozone, and Sulfur Dioxide and Respiratory Symptoms in a Nonsmoking Population. Archives of Environmental Health. Vol. 48 (1): 33-46.

Abbey, D. E., B. L. Hwang, R. J. Burchette, T. Vancuren and P. K. Mills. 1995. Estimated longterm ambient concentrations of PM10 and de velopment of respiratory symptoms in a nonsmoking population. Arch Enviro n Health. Vol. 50 (2): 139-52.

Industrial Economics, Inc. (IEc), 2006. Expanded Expert Judgment Assessment of the Concentration-Response Relationship Between PM 2.5 

### Exposure and Mortality . Prepared for the U.S. EPA, Office of Air Qua lity Planning and Standards, September.

Krupnick, A. J. and M. L. Cropper. 1992. The Effect of Information On Health Risk Valuations. Journal of Risk and Uncertainty. Vol. 5 (1): 29-48.

Laden, F., J. Schwartz, F. E. Speizer and D. W. Dockery. 2006. Reduction in Fine Particulate Air Pollution and Mortality: Extended follow- up of the Harvard Six Cities Study. Am J Respir Crit Care Me d. Vol. 173 (6): 667-72.

Peters, A., D. W. Dockery, J. E. Muller and M. A. Mittleman. 2001. Increased particulate air pollution and the triggering of myocardial in farction. Circulation. Vol. 103 (23): 2810-5. http://www.circulationaha .org/cgi/content/full/103/23/2810.

Pope, C. A., 3rd, R. T. Burnett, M. J. Thun, E. E. Calle, D. Krewski, K. Ito and G. D. Thurston. 2002. Lung cancer, cardiopulmonary mortalit y, and long-term exposure to fine particulate air pollution. Ja ma. Vol. 287 (9): 1132-41.

Rosamond, W., G. Broda, E. Kawalec, S. Rywi k, A. Pajak, L. Cooper and L. Chambless. 1999. Comparison of medical care and survival of hos pitalized patients with acute myocardial infarction in Poland and the United States. Am J Cardiol. Vol. 83 (8): 1180-5.

U.S. Department of Commerce, Economics and Statistics Admini stration, Bureau of the Census. February 1996. CURRENT POPULATION REPO RTS Population Projections of the United States by Age, Sex, Race, and Hispanic Origin: 1995 to 2050, P25-1130.

U.S. Department of Commerce, Economics and Statistics Admini stration, Bureau of the Census. July 2002. U.S. Summary: 2000, Census 2000 Profile, C2KPROF/00-US.

U.S. Department of Energy, Department of Energy (DOE). March 2009. Macroeconomic Indicators , Projection Tables (Table 20), Updated Annual Energy Outlook 2009 Reference Case Service Report ,. Report #:DOE/EIA-0383(2009). http://www.eia.doe.gov/oiaf/aeo/index.html.

### 7-2 U.S. Environmental Protection Agency (EPA). September 2008. BenMAP User’s Manual. Office of Air Quality Planning and Sta ndards Research Triangle Park, NC.

Viscusi, W. K., W. A. Magat and J. Huber. 1991. Pricing Environmental Health Risks—Survey Assessments of Risk—Risk and Risk—Dollar Trade-Offs For Chronic Bronchitis. Journal of Environmental Economics a nd Management. Vol. 21 (1): 32-51.

Woods & Poole Economics Inc. 2007. Complete Demographic Database. Washington, DC. http://www.woodsandpoole.com/index.php.

### A-1 APPENDIX A

EXPE RT WORKSHOPRTI International is a trade name of Research Triangle Institute 

### TO: Amy Lamson (Office of Air Quality Pla nning and Standards, U.S. Environmental Protection Agency)

FROM: Carol Mansfield (RTI International) Amir Moktari (RTI International) Paramita Sinha (RTI International) Max Henrion (Lumina Decisions Systems)

DATE: February 13, 2009

SUBJECT: Influence and Uncertainty Analysis of Human Health Benefits Estimates: Capturing Uncertainty in Input Data and Uncertainty Introduced in BenMAP Modeling Workshop

On December 15 and 16, 2008, the Office of Air Quality Planning and Standards (OAQPS) of the U.S. Environmental Protection Agency (EPA) held an expert workshop to s olicit input on their planned influence analysis of the human health bene fits of reductions in particulate matter (PM). The meeting was held at the OAQPS office in Research Triangle Park, North Carolina.

OAQPS is developing a model to conduct an influen ce analysis on the uncertain parameters, functions, and input data used to calculate the benefits of reductions in PM using BenMAP, the software program OAQPS uses to estimate economic benefits for many re gulatory analyses. The model will be used to help OAQPS identify the degree to which uncertainty about key parameters and functional forms affects the benefits analysis. Long term, this an alysis will help prioritize future research and provide a template for conducting influence analyses on other benefit assessme nts. The influence analysis will use Analytica software to create a simplified representation of BenMAP’s calculations.



### The workshop focused on uncertainty in populat ion forecasts, valuation of mortality, and valuation of acute myocardial infarctions (AMI s) and chronic bronchitis in BenMAP. Outside experts were asked for input on th e sources of uncertainty, struct ure of the influence analysis model, and the range of values that OAQPS should consider.

Specifically, the goals of the workshop were to

• Ensure important sources of uncertainty were not omitted, focusing first on the population data and forecasts and va luation techniques and estimates;

• evaluate the simplified model OAQPS is cr eating to conduct the influence analysis;

• solicit appropriate range data and distri butions for uncertainty parameters and alternative functional form assumptions for mo del uncertainty to use in the sensitivity analysis; and

### Amy Lamson May 21, 2009 Page 2

• solicit input on the methodology fo r the sensitivity analysis.
Before the workshop, OAQPS and RTI Internationa l identified a set of experts on population forecasting, morbidity and mortality valuation, and uncertainty analysis to participate in the workshop. OAQPS also indentified EPA staff to be included. Appendix A provides a list of the participants. The participants we re e-mailed a guide that provided the goals of the workshop, an agenda, background information, and several tables th at listed the sources of uncertainty related to population forecasting, mortality valuation, a nd morbidity valuation identified before the workshop. The Participants Guide is included in Appendix B.



### 1. Overview

The 1.5 day meeting was divided into seven sess ions. The first three se ssions included an introduction by Amy Lamson (EPA), an overvie w of BenMAP by Neal Fann (EPA), and an overview of the simplified Analy tica model proposed for the infl uence analysis by Max Henrion (Lumina Decision Systems). The next three sessi ons each focused on a different topic, including population forecasting, valuation of mortality, and valuation of morbidity. For each discussion, Max Henrion presented the sources of uncertainty identified by the planning team before the workshop and how they fit into the Analytic a model. After the presentation, the workshop participants spent 5 minutes thinking about the topic and additional sources of uncertainty. After 5 minutes, the participants at each table discusse d their thoughts on sources of uncertainty with each other. This discussion was followed by a group discussion where each table reported sources of uncertainty they identified, and Ken Elstein (EPA) summarized the discussion on a computer linked to an overhead projector. Ne xt, each partic ipant spent time thinking about whether each source of uncertainty could be quan tified, the possible range of values or functions that could be used in the influence analysis, and any comments on the uncertainty. Again, after the individual brainstorming, the participants at each table discussed their ideas and then each table reported back to the w hole group. The group also discu ssed areas of agreement and disagreement, which were captured by Ken Elstein. The final session started with a review of the Analytica model by Max Henrion followed by a gr oup discussion on the approach to the current influence analysis and on ways to model the uncertainty in BenMAP.



### 2. Session 2: Overview of BenMAP

Neal Fann presented an overview of BenMAP (slides in Appendix C). The main steps in
BenMAP can be summarized as follows:

### Amy Lamson May 21, 2009 Page 3

• Step 1: Use studies to find relative risk estimates that show the relationship between pollutant concentration and the health e ffect. Find the incremental change in the
health outcome based on a un it change in concentration ( β ) with standard error.

–

Note: BenMAP does not model populati on exposure, assuming all exposure occurs where the population resides.

–

Note: For some functions, EPA includes th reshold-adjusted functions in BenMAP that refit the data for the threshold mode l from the original nonthreshold model (in the original epidemiology study).

• Step 2: Apply the health impact function fr om Step 1 to the incremental air quality improvement for the population and backgr ound incidence rate in each air quality grid square to estimate the ch ange in health outputs.

–

Note: BenMAP includes preloaded national- level health data and applies it to a smaller-scale area such as the grid size associated with the air quality data (typically 12 km square or 36 km square). If local-scale data is available, it can be loaded into BenMAP.

–

Note: County-level population is mapped to the air quality grid squares before being imported into BenMAP using a program called Popgrid.

• Step 3: Value health outcome using willingne ss to pay (WTP) or cost of illness (COI) estimates by multiplying the change in health incidence by WTP or COI. In BenMAP, a distribution of health outcome incidence and a distribution of WTP or COI are created, and random draws from both distributions create the distribution of total benefits.



### 3. Session 3: Overview of Simple Analytica Model of

BenMAP

Max Henrion presented an overview of the simp le model of BenMAP he has created using Analytica software (slides in Appendix C). Th e objectives for the model are to (1) identify sources of uncertainty in BenMAP , (2) visualize how the sources of uncertainty affect the results using an influence diagram, (3) assess the range of values of key uncerta inties, and (4) compare
their relative contributions to the results us ing sensitivity analysis. The simplified Analytica model will be calibrated to BenMAP by compar ing the output of the simplified model at different steps with runs of BenMAP.

BenMAP is a complex program with a lengt hy running time for large- scale analyses, and conducting an extensive influence or sensitiv ity analysis with Be nMAP would be time consuming. With a simplified model of BenMAP, we can conduct initial in fluence analysis to identify parameters or functional form assumpti ons that have the biggest impact on the outcomes of the benefits analysis, including both health incidence and monetary benefits. The simplified model allows us to test parameters over a wi de range of values w ithout requiring excessive

### Amy Lamson May 21, 2009 Page 4

resources. Analytica provides a flexible and tr ansparent format for the analysis. Max Henrion presented an example using tornado graphs to show sensitivity of the simplified model to
different sources of uncertain ty based on fabricated data.

To reduce the size of the data set and the complexity of the Analyt ica model, the air quality grid squares were aggregated into 16 bins based on ini tial ambient air quality concentration. We also evaluated sensitivity to the leve l of aggregation (e.g., instead of 16 bins we can use a smaller number of bins). More concentration bins m eans more detail and decreasing aggregation. A variety of aggregation issues can be addresse d, including spatial and te mporal, over age groups and across endpoints.

The participants discussed some of the assump tions used to create the Analytica model and pointed out some differences be tween the way BenMAP treats cer tain aspects of the model and the way the preliminary Analytica model does. Fo r example, uncertainty varies depending on the control strategy. The control strategy that generate d data used for this example is a more urbanfocused control strategy. A different control strate gy might have a different level of uncertainty if it is a more regional approach. With the binning approach, one may lose the ability to consider the geographic impact of applying policy change s. To address these concerns, the Analytica model will need to be tested using several contro l strategies and different assumptions about bin
size and then calibrated to BenMAP.

The following changes in terminology were requested:

• Use “avoided premature mortalities” instead of “lives saved”

• Use “value of a statistical life” instea d of “value of sta tistical life saved”

• Use “concentration-response” instead of “dose-response”

• Use “health impact function” instead of “health effects function”

• Use “monetized benefits” instead of “ben efits” (unless referring generically to incidence reduction and monetized benefits)

• Use “total monetized benefits” instead of “n et present value of benefits” (because we generally do not evaluate streams of benefits)

• Use “cessation lag” instead of “VSL adjusted for lag” (b ecause we adjust the total mortality benefits, not the VSL specifically)



### 4. Session 4: Population

Max Henrion started the session with a presen tation on how population enters the Analytica model (Appendix C). The experts were asked to th ink about additional sources of uncertainty not captured in the current model and for their judgm ent on the ranges for uncertain parameters. For

### Amy Lamson May 21, 2009 Page 5

the influence analysis, we want plausible low, mid, and high values or perhaps the 10 th


### percentile, median, and 90th percentile.

Much of the discussion centered on the factors th at affect the accuracy of the Census population forecasts. The Census forecasts population at the national level and does not use an economic model to support the forecasts. Migration and im migration emerged as two factors that the Census projections may not capture well. Be nMAP uses Woods and Poole to project the population data at the county level, and this da taset does include an economic component to its
forecasts.

The group also discussed uncertainty introdu ced through the assignment of the population forecasts to the air quality grid squares. The air quality grid squares are at a finer resolution than the underlying health data. The Census data capture where people live, but people do not typically remain in one locati on throughout the day. The finer the resolution, the more likely that exposure will be misclassified if the population travels between grid squares during the day. A higher level of aggregation might be preferable. Participants s uggested a case study (e.g., assign a population based on land use categories or check th e sensitivity of one gr id cell by assigning all of the population to a single grid square, such as the grid square with the highest PM2.5 concentration in the area).

The participants also discussed other measures of sensitivity. Experts were concerned that by defining ranges we would actually introduce anot her source of uncertain ty. If the range is unknown, it may be more appropriate to use elastic ity rather than range since a range can be misleading if one end is a low-probability event.

Discrete events that result in large-scale populat ion and migration shifts within the United States (e.g., Hurricane Katrina) were discussed. Scenario analysis could be used to address the impact of discrete events. The importance of uncerta inty in population foreca sts after 2030 was thought to be most important for climate change and ot her policies where the bene fits may not occur in the near future. Scenario analysis may be th e way to address the uncertainty of population forecasts over a longer time period.

Recommendations
Overall, the population expert a nd others in the group felt that, although uncertainties related to population forecasting were certainly worth expl oring, population uncertainties are perhaps not as large as other uncertainties in the model. For the influence an alysis, the Census low, medium, and high forecasts were thought to provide a goo d range. For allocating population to counties,
the Woods and Poole confidence interval s were thought to provide a good range.

### Amy Lamson May 21, 2009 Page 6



### 5. Session 5: Premature Mortality Valuation

The session started with an overview from Ma x Henrion on how the estimates of premature mortality and the value of reducing the risk of premature mortality (mortality valuation) are generated in the simplified Analytica model of BenMAP (slides in Appendix C). Mortality
valuation is often referred to in the economics literature as the valu e of a statistical life or the VSL, but it is really the value of reducing the ri sk of premature mortality. EPA uses a single VSL function (a normal distribution with a mean of $5.5 million) that does not vary across people (e.g., demographics, health status, or cause of deat h). Because exposure in a particular year will cause premature deaths in the future, the mone tized benefits are discounted to account for the cessation lag, typically over 5 years to 20 years.

There was a lively debate on whether VSL should be treated as a policy d ecision, rather than an uncertain empirical value. VSL measures the va lue of a reduction in mortality risk to an individual. VSL is internally consistent within the utilitarian framework, which is the framework for benefit-cost analysis. Some argued that policy makers should use alternative VSL frameworks (e.g., social choice, altruism, revealed preference for government programs). The
current VSL estimates do not value the distri bution of risk within the population (e.g., environmental justice issues).

The discussion identified a number of factors that influence both the degree of uncertainty (in the strict sense of the word) associated with VS L estimates and the variability in VSL across
individuals or over time as individuals age or wealth increase s. Personal characteristics that may cause VSL to vary across individuals include age, health status, income or wealth, number of children, differing preferences over the charac teristics of the risk (voluntary, unknown, dread factor), cultural f actors, and wealth.

VSL values have been estimated using both re vealed preference (RP) studies and stated preference (SP) studies. The RP studies mostly lo ok at the trade-off between wages and job risk. The population in RP studies (healthy, working-age adults) and the type of risks differs (mostly sudden death from a workplace accident or homicid e) from populations sus ceptible to premature mortality from PM exposure, which, on average, will be older and less healthy). SP studies are more likely to value risks similar to those asso ciated with PM exposure such as the risk of
chronic illnesses like cancer or other chronic diseases. Most SP studies survey the general population and include older adu lts and individuals with comp romised health status. One suggestion from the participants was to use VSL st udies that match more cl osely the type of risk posed by exposure to PM and to compare the ra nge of SP and RP studies. Finally, WTP can be
elicited for risk of death in the future. WTP for risk of death in the future could replace the

### Amy Lamson May 21, 2009 Page 7

assumed lag structure currently used for benefits analysis, which could be described as somewhat
ad hoc based on severa l interpretations of the epidemiology data.

Participants discussed the income elasticity of VSL and thought, in general, that it was not likely to be a large source of uncertainty. However, th ere were questions abou t using cross-sectional estimates of income elasticity to estimate elasticity over time.

Recommendations
Everyone recognizes the importance of the VSL in cal culating the total bene fits of a reduction in PM. For the influence analysis, the range of $1 million to $10 million was supported by most participants, although a lower bound of $700,000 could al so be considered. If we use the range from SP studies, it may be more like $1 million to $3 million.



### 6. Session 6: Morbidity Valuation

The session on morbidity valuation focused on chronic bronchitis and AMI. Max Henrion
discussed how the incidence and value of the di seases enter the Analyt ica model (slides in Appendix C).

The participants discussed the general issue of WTP estimates and COI estimates. Theoretical differences in the costs captured by each measure and how to avoid double-counting were
discussed. We are unaware of WTP estimates for AMIs, and the WTP estimates for chronic bronchitis are from SP studies with known problem s, including the age of the studies, the small and unrepresentative sample, and advances in SP methodology since the studies were conducted. In addition, the studies compared risk/risk trade-offs with risk/ dollar trade-offs and concluded
the risk/risk trade-offs were more stable. Th e COI estimates were also thought to be dated. Medical costs, technology, and publ ic awareness of diseases and treatments have all changed significantly over time.

Some questions raised by the participants rela ted to the health and epidemiology literature. The severity of PM-related AMIs and chronic bronchi tis are not known, which complicates the task of deciding which WTP or COI es timates are appropriate. Participants suggested that the model consider several disease trajectories with di fferent cost implications. Another source of uncertainty relates to sequelae resulting from PM-related AMIs and chronic bronchitis, costs beyond the hospital stay, costs to unpaid caregivers, and long-term impacts on life expectancy and quality of life. Assumptions about the duration of the illne sses and the proper discount rate
were raised. A number of sour ces for uncertainty and variab ility came up in the discussion, including regional and urban/rural variation in medical costs, ge nder, age, and health status.

### Amy Lamson May 21, 2009 Page 8

Finally, the group also discussed th e best methods for pooling the re sults from several studies for morbidity endpoints.

Recommendations
For AMIs, the participants thought that we should search for more recent COI estimates to get a better sense of the range of possible costs and use all the availa ble studies to crea te the range of
values. Several studies have shown that WTP tends to be a factor of 2 larger than COI for the same health endpoint, and those studies together with WTP studies for related diseases could be compared to COI. Wage losses for individuals under age 27 a nd over 65 should be included in the range.

For chronic bronchitis, participants suggested we use the WTP valu es for most severe and least severe in the range. They also suggested that COI estimates be incorporated.

For both morbidity endpoints, the suggested ra nge was 1⁄2 COI in existi ng BenMAP studies (if newer estimates cannot be found) to the top end of COI + WTP + adjustme nts for sequelae costs and other excluded categories.



### 7. Session 7: Discussion and Recommendations on Approach to Influence and Uncertainty Analysis

In the final session, Max Henrion reviewed th e simplified BenMAP model being developed using Analytica software for the influence anal ysis (slides in Appendix C, revised influence diagrams in Appendix D). The discussion focu sed on the approach being proposed for the influence analysis and solicited comments and s uggestions from the workshop participants.

The influence analysis will start by creating a To rnado diagram. For each parameter or functional form assumption, the model is run using the ra nge of possible values for the parameter or functional form holding all the other parameters at their mean value. The Tornado diagram stacks the range of total benefits associated with the range of the parame ter value (or functional form assumptions) from the parameters and functi onal form assumptions that generate the largest range in total benefit estimates down to the smallest.

The Tornado diagram does not account for inte ractions between parameters or between parameters and functional form assumptions. There was a discussion about how to handle interactions. One suggestion by several participants was to use probabilistic sensitivity analysis measures such as rank correlations and regres sion measures. However, one participant felt strongly that we should not make any assumptions about probability distributions where we did

### Amy Lamson May 21, 2009 Page 9

not have information about the distribution. In stead, it was suggested th at a response surface using Latin hyper-cube poi nts could be created.

Recommendations

The list of recommendations summarizes a variet y of perspectives and does not represent a consensus on the part of the participants.

• Conduct analysis both using Tornado di agram and allowing for interactions.

• Conduct comparisons between the results of the simplified Anal ytica model of the BenMAP model and runs from BenMAP itsel f to make sure the Analytica model will provide results that mimic the results that would have been achieved using BenMAP itself.

• Identify parameters and functional form a ssumptions that have the largest impact on the outcomes of interest (monetary benef its and health incidence). Explore these sources in more detail.

• Explore the sensitivity of both the monetary value of benef its and the incidence of the health endpoints, since bot h are important to people.

• After influence analysis identifies potent ially influential parameters and functional form assumptions, EPA will assess the level of resources that might be needed to reduce the uncertainty.

• Use two or three different control scenarios to generate reductions in concentrations. Whether the reductions are local or regional, or just an overall percentage reduction, may affect the relative importance of the parameters. The discussion also included ideas on how to refine the simplified model:

• Initially, the model assumes that the entire exposed population in each base concentration bin experiences the same reduction in PM2.5 concentration levels (averaged over the reductions for all grid s quares in that bin). Later, if time permits, we could expand the bins into a bin array with a second dimension of concentration reductions for each of the concentration bins and see if this reduces the aggregation uncertainty. Therefore, each bin would incl ude grid squares with the same baseline concentration and the same reduction in concentration.

• The model does not differentiate among popul ation segments (i.e., race, ethnicity, or gender). Initially, the model uses an aver age across age groups, but this will be refined for the final analysis.

### Amy Lamson May 21, 2009 Page 10



### Appendix A : Participant List

Soad Benromdhane, OAQPS, HEID Lillian Bradley, OAQPS, HEID Linda Chappell, OAQPS, AQAD Ken Davidson, OTAQ Chris Dockins, OPEI, NCEE Ken Elstein, ORD Ron Evans, OAQPS, HEID Neal Fann, OAQPS, HEID Chris Frey, NCSU Charlie Fulcher, OAQPS, HEID Beth Hassett-Sipple, OAQPS, HEID Brian Heninger, OPEI, NCEE Max Henrion, Lumina Bob Hetes, ORD, NHEERL Bryan Hubbell, OAQPS, HEID Alan Krupnick, RFF, econ Amy Lamson, OAQPS, HEID John Langstaff, OAQPS, HEID Carol Mansfield, RTI Amir Mokhtari, RTI Ellen Post, Abt Pradeep Rajan, OAQPS, HEID Harvey Richmond, OAQPS, HEID Paramita Sinha, RTI Laura Taylor, NCSU, econ Scott Voorhees, OAQPS, HEID Paul Voss, UNC Darryl Weatherhead, OAQPS, HEID

### Amy Lamson May 21, 2009 Page 11



### Appendix B : Participant Guide

Influence and Uncertainty Analysis of Human Health Benefits Estimates: Capturing Uncertainty in Input Data a nd Uncertainty Introduced in BenMAP Modeling

Participants Guide

U.S. Environmental Protection Agency Research Triangle Park, NC December 15–16, 2008

### Amy Lamson May 21, 2009 Page 12

Dear Participant,

On behalf of the U.S. EPA, I would like to welcom e you to the workshop. EPA’s Office of Air Quality Planning and Standards organized this meeting to soli cit your input on our planned influence analysis of the human health benefits of reductions in PM. A 2002 National Academy of Sciences report recommended that EPA conduct an Influence Analysis to improve uncertainty characterization in our benefits analyses and to identify the modeling elemen ts and assumptions that have the largest influence on the human health benefits results. We will present a preliminary model for influence analysis focusing on the valuation of human health benefits associated with mortality and AMIs estimated using BenMAP. As we develop our approach, we want outside advice from experts like you on the sources of uncertainty, structure of the influence analysis model, and the ra nge of values we should consider. The results of this analysis will help to prioritize future efforts to reduce uncertainty. We also plan to take the results of this report to EPA’s Science Advisory Board sometime next year.

We have set three main goals for this workshop:

Desired Goals 1. Ensure we have not left out a ny important sources of uncertainty focusing first on the population data and forecasts and valua tion techniques and estimates. 2. Evaluate the simplified model we are crea ting to conduct the influence analysis 3. Solicit appropriate range data and distributions fo r uncertain parameters a nd alternative functional form assumptions for model uncertainty to use in the sensitivity analysis 4. Solicit input on the methodology for the sensitivity analysis

Desired Outcomes 1. Comprehensive list of uncertainties in benefits analysis arising from population data/forecasts and from valuation approach. 2. Confirmation that simplified influence analysis model of BenMAP is accurate and complete 3. Appropriate range data and/or distributio ns to use in the sensitivity analysis 4. Enhanced understanding of the benefits analys is process and assumptions by participants 5. Methods for incorporating uncertainties into the sensitivity analysis

Your active participation is crucial to achieving these goals.

Please take a few minutes to familiarize yourself with this Participants’ Guide, paying particular note to the “Meeting Principles” and “Lists of Uncertainties. ” If there is anything we can do to improve the workshop, do not hesitate to let me or the facilita tor know. Thank you for your in terest and participation in this workshop.

Sincerely,

Amy Lamson Air Benefits and Cost Group Office of Air Quality Planning and Standards U.S. Environmental Protection Agency

### Amy Lamson May 21, 2009 Page 13

Agenda for Influence and Uncertainty Analysis of Human Health Benefits Estimates: Capturing uncertainty in input data and uncertainty introduced in BenMAP modeling Classroom C112 in EPA Main Building, EPA RTP Campus (109 T.W. Alexander Drive, RTP N.C. 27711)

Day 1: Monday, December 15 th Registration (8:30am to 9:00am) Session 1 (9:00am to 9:30am): Welcome and Introductions—Amy Lamson, OAQPS

Session 2 (9:30am to 10:15am): Introduction to benefits and BenMAP—Neal Fann, OAQPS

Break (10:15am to 10:30am)

Session 3 (10:30am to 12.00pm): Overview of the simp lified influence analysis model and project strategy—Max Henrion, Lumina Decision Systems

Lunch (12:00pm to 12:45pm): EPA cafeteria (on your own)

Session 4 (12:45pm to 2:45pm): Population estimates 12:45pm to 1:00pm: Identified sources of uncertainty in population estimates and uncertainty introduced by BenMAP modeling assumptions—Max Henrion, Lumina Decision Systems 1:00pm to 2:45pm: Discussion on current population estimates in BenMAP

o 5min Quiet brainstorming on the following questions:

 Are there additional sources of uncertainty in population forecasts or introduced by BenMAP’s use of the population data?

 Do you think this will have big impact on benefits results?

o 15min Report out and discussion

o 10min Quiet brainstorming on the following questions:

 Can/should this source of uncertainty be modeled quantitatively in the influence analysis?

 What range of values should we use for these parameters or where can we get the range (i.e., from a study or existing data)?

 Are there other functional forms or modeling approaches that make sense for this component of the model or what would be a good source for this information?

o 30min Small group discussion

 Where do we agree and where do we disagree?

### Amy Lamson May 21, 2009 Page 14

o 45min Report out and discussion

Break (2:45pm to 3:00pm)

Session 5 (3:00pm to 5:15pm): Valua tion of mortality reduction 3:00pm to 3:30pm: Identified sources of uncertainty in valuation of mortality and uncertainty introduced by BenMAP modeling assumptions—Max Henrion, Lumina Decision Systems

• 3:30pm to 5:15pm: Discussion on valu ation of mortality reduction

o 5min Quiet brainstorming on the following questions:

 Are there additional sources of uncertainty in valuation of mort ality reductions or introduced by BenMAP’s use of valuation studies?

 Do you think this will have big impact on benefits results?

o 15min Report out and discussion

o 10min Quiet brainstorming on the following questions:

 Can/should this source of uncertainty be modeled quantitatively in the influence analysis?

 What range of values should we use for th ese parameters or where can we get the range (i.e., from a study or existing data)?

 Are there other functional forms or modeling approaches that make sense for this component of the model or what would be a good source for this information?

o 30min Small group discussion

 Where do we agree and where do we disagree?

o 45min Report out and discussion

Wrap-up (5:15 to 5:30) Dinner with group 6:30pm (optional, otherwise on your ow n) Please let Carol Mansfield know if you are interested in the group dinner

### Amy Lamson May 21, 2009 Page 15

Day 2: Tuesday, December 16th Welcome (8:30am to 8:45am): Introduction, recap from Day 1—Amy Lamson, OAQPS

Session 6 (8:45am to 10:45am): Valuation of morbidity reduction 8:45am to 9:00am: Identified sources of uncertainty in valuation of mortality and morbidity reduction and uncertainty introduced by BenMAP modeling assumptions—Max Henrion, Lumina Decision Sys tems

• 9:00am to 10:00am: Discussion on va luation of morbidity reduction

o 5min Quiet brainstorming on the following questions:

 Are there additional sources of uncertainty in morbidity valua tion or introduced by BenMAP’s use of the morbidity valuation models?

 Do you think this will have big impact on benefits results?

o 15min Report out and discussion

o 10min Quiet brainstorming on the following questions:

 Can/should this source of uncertainty be modeled quantitatively in the influence analysis?

 What range of values should we use for th ese parameters or where can we get the range (i.e., from a study or existing data)?

 Are there other functional forms or modeling approaches that make sense for this component of the model or what would be a good source for this information?

o 30min Small group discussion

 Where do we agree and where do we disagree?

Break (10:00am to 10:15am)

• 10:15am to 10:45am: Discussion on valuation of morbidity reduction, con’t

o 30min Report out and discussion

Session 7 (10:45am to 12:20pm): Discussion and Reco mmendations on Approach to influence and uncertainty analysis—Max Henrion, Lumina Decision Systems

Closing (12:20 to 12:30) Wrap-up and Next Steps

### Amy Lamson May 21, 2009 Page 16

Meeting Principles

1. Everyone shares the responsibility for making the meeting a success : This meeting is a true collaboration between all attendees, including the organizers and facilitators. Your active participation is critical. If you have an idea for a discussion topic or process suggestion, please share it orally and/or in writi ng through a posted note. If you ha ve a criticism, please balance it with a solution.

2. Stay on topic : Start from the “big picture” before moving into details. Jumping to details prematurely can consume a lot of time on a topic that the group may later decide is unnecessary. A “Parking Lot” is available to post ideas/comme nts to ensure that they are addressed at the appropriate time.

3. Listen and understand : All participants bring to this meeting diverse experiences, ideas, knowledge, and perspectives. Inquire of others to understand thei r views before advocating your

own.

4. Be transparent: Our assumption is that all participants are coming to this meeting with the intent of making the workshop more effective. Those w ith individual needs/concerns should make them known to the group so we can develop innovative approaches to meet all needs.

5. First brainstorm, then critique : The most creative ideas emerge when participants can build upon each other’s suggestions. Often the seemingl y wildest ideas stimulate true innovation. Avoid premature critiquing that can unintentionally shut down the creative process.

6. Provide everyone an equal opportunity to speak : Part of our diversity includes variations in how we prefer to express ourselves. Freely offe r your perspectives in whatever way is most comfortable for you, and allow others the opportunity to express theirs.

7. Commit to being fully present : Please turn off cell phones, or put them on vibrate; put away the PDAs and Blackberrys. You can always check them during breaks.

### Amy Lamson May 21, 2009 Page 17

Background Information

For more information regarding the model and t ype of analysis to be discussed during the workshop, please consult the following websites.

• Benefits analysis

o OAQPS Economic Guidelines (1999): http://www.epa.gov/ttn/ecas/econdata/Rmanual2/0.0.html

o EPA Guidelines for Preparing Economic Analyses: http://yosemite.epa.gov/ee/epa/eed.nsf/ webpages/Guidelines.html#download

• EPA benefits analysis for PM

o Regulatory Impact Analysis for Particle Pollution NAAQS (2006): http://www.epa.gov/ttn/ecas/ria.html

o PM Benefits chapter: http://www.epa.gov/ttn/ecas/regda ta/RIAs/Chapter%205—Benefits.pdf

o PM Sensitivity analyses: http://www.epa.gov/ttn/ecas/r egdata/RIAs/Appendix%20J— Additional%20Benefits%20Sen sitivity%20Analyses.pdf

• BenMAP model

o General Introduction: http://www.epa.gov/air/benmap/

o Download version 3.0: http://www.epa.gov/air/benmap/download.html

o BenMAP Manual: http://www.epa.gov/air/benmap/models/BenMapManualSept08.pdf

o 

### BenMAP Manual Appendices: http://www.epa.gov/air/benmap/models/BenMAPappendicesSept08.pdf

### Amy Lamson May 21, 2009 Page 18Table 1. List of Major Sources of Uncertainty (Parametricand Model) for Mortality inDifferent Simulation Stepsof the BenMAP ModelBenMAP choice: Selected by user from choices in BenMAPBenMAP default: Defined in BenMAP, user can sometimes import different data to change assumptionInput: Generated outside BenMAPModeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMAPPopulation DataBenMAP choice Selection of forecast year Parameter Uncertainty,ForecastBenMAP allows the user to select the analysis year. Many recentanalyses use 2020.BenMAP default Estimating populationlevels in non-censusyears—Population levelsbetween 2000 and 2030Parameter Uncertainty,ForecastUsing county-level population scale factors based on, e.g.,age4-9,g,t=age4-9,g,2000×age4-9,county,t/age4-9,county,2000For the gthpopulation grid cell and for 2000<t<2030. Populationforecasts for future years through 2030 basedon a study by Woodsand Poole (2007).Input Estimating populationlevels in non-censusyears—Population levelsafter 2030Model Uncertainty,ForecastEven if user imports a new population dataset, it must be importedthrough Popgrid, which has this assumption built inLinear extrapolation after 2030,e.g.,age4-9,g,2035=age4-9,g,2030+5×(age4-9,2030-age4-9,2029)Input Population distributionwithin an age groupModel Uncertainty,Functional formUniformly distributedwithineach age group, e.g.,age3-12= 1⁄2 × age1-4+ age5-9+ 3/5 × age10-14Input Forecasts of populationfrom Woods and PooleParameter Uncertainty,ForecastWoods and Poole forecasts population based on projected economicconditions (county economies linked), natural increase, and migrationdue to economy at the county level (confidence intervals reported byWoods and Poole ±6.3%for metropolitan statistical areas and±4.7%for states)Input Assigning population datato grid squaresModel Uncertainty,Functional formPopgrid (a separate program) assigns 2000 Census block data to gridsquares for 304 race/ethnicity/gender/age groups, includingpopulation growth weights.(continued)

### Amy Lamson May 21, 2009 Page 19Table 1. List of Major Sources of Uncertainty (Parametricand Model) for Mortality inDifferent Simulation Stepsof the BenMAP Model (continuedModeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMAPValuationBenMAP choice VSL estimateParameter Uncertainty,MeasuredUses mean for Value of Statistical Life (VSL) of $5.5 million(2000$), with range from $1 million to $10 million.Alternative distributions for the mean and range can be used.BenMAP choice Income elasticityof WTP Parameter Uncertainty,MeasuredIncome elasticity estimates for VSLfrom literature, mean of 0.40,lower bound 0.08 and upper bound 1.Input Income growth per person Parameter Uncertainty,ForecastStandard & Poor’s projections of future changes in Gross DomesticProduct (GDP) occurring after the year 2010Divide the projected change in GDP by the Woods & Poole projectedchange in total US population to produce an estimate of the futureGDP per capitaInput Lag structure in mortality Model uncertainty,Functional formEPA considers alternative lag structures for PM-related pre-maturemortality(a)no lag effect(b)8-year: incidences all occur in the 8thyear following year ofchange in exposure(c)15-year: incidences all occur in the 15thyear following year ofchange in exposure(d)Alternative segments: 20% of incidence occur in the 1styear,50% in years 2 to 5, and 30% in years 6 to 20(e)5-year distributed: 50% of incidences occur in years 1 and 2 and50% in years 2 to 5Exponential: incidences occur at an exponentially declining ratefollowing year ofchange in exposure(continued)

### Amy Lamson May 21, 2009 Page 20Table 1. List of Major Sources of Uncertainty (Parametricand Model) for Mortality inDifferent Simulation Stepsof the BenMAP Model (continuedModeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMAPValuation (continuedBenMAP default Income elasticity andincome growthadjustment to WTPModel Uncertainty,Functional form12121221122121112122111221221121122212121212IIIIIIII*WTPWTP)IIII(*WTP)IIII(*WTPWTPIWTPIWTPIWTPIWTPIWTPIWTPIWTPI)WTPWTP(*)II()II(*)WTPWTP(IIWTPWTP−−ε−ε−−ε−ε=−−ε−ε=−−ε−ε−−+=ε−ε−ε+ε+−+−=ΔΔ=εBenMAP default Choice of baselinecurrency year for GDPand WTPParameter Uncertainty,MeasuredBenMAP uses year 2000 baselineBenMAP default Conversion of WTP andGDP to real dollarsParameter Uncertainty,MeasuredBenMAP has inflation indices for all goods, medical costs and wagesusing year 2000 baselineBenMAP choice Discount rate Parameter Uncertainty,MeasuredEPA discounts benefits that occur after the analysis year at 3% and7%Applies when there is a lag between exposure and mortalityC-R Function and Issues in the Estimation of Adverse Health EffectsBenMAP choice Functional form of theC-R functionModel Uncertainty,Functional formAlternative functional forms are sometimes available to representC-R relationship including linear, log-linear, logistic, and Coxproportional hazard models(continued)

### Amy Lamson May 21, 2009 Page 21Table 1. List of Major Sources of Uncertainty (Parametricand Model) for Mortality inDifferent Simulation Stepsof the BenMAP Model (continuedModeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMAPC-R Function and Issues in the Estimationof Adverse Health Effects (continued)BenMAP choice PM Coefficient (β) in theC-R functionParameter Uncertainty,MeasuredEach mortality study including expert opinion assumes a distributionrepresenting uncertainty inβvalue and a mean value. Exampledistributions include normal, weibull, and triangularBenMAP choice Threshold in the C-RfunctionParameter Uncertainty,MeasuredEPA estimates a point estimate threshold below which there is nochange in incidence rate associated with change in air pollutionconcentration, e.g., for a log-linear C-R model:Δy=y0×[exp(β×(max(PM1,T)-max(PM0,T)))-1]Where, T is the thresholdBenMAP choice Select air quality metric inBenMAP to match metricin epi studyParameter Uncertainty,MeasuredFor PM2.5, EPA currently uses C-R functions that use a quarterly averageair quality concentration. BenMAP creates an annual average byaveraging the 4 quarterly averages; quarterly averages are average of daily8 hour average.BenMAP choice Conversion of air qualitymetrics reported in epistudies to common metricParameter Uncertainty,MeasuredCurrently only done for ozone, not PMBenMAP default Extrapolation beyond PMconcentration in studyParameter Uncertainty,Extrapolate beyondrangeBenMAP makes no adjustment if PM levels resulting from regulation areoutside the range of PM levels from the studyBenMAP choice Current exposure andcumulative exposureParameter Uncertainty,MeasuredStudies do not account for effect current exposure versus cumulativeexposure for current and lagged mortalityBenMAP default Differential toxicity of PMcomponentsParameter Uncertainty,MeasuredAssumes equal toxicity, no mechanism in BenMAP for adjusting thisassumption(continued)

### Amy Lamson May 21, 2009 Page 22Table 1. List of Major Sources of Uncertainty (Parametricand Model) for Mortality inDifferent Simulation Stepsof the BenMAP Model (continuedModeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMAPIncidence RatesBenMAP choice Mortality BaselineIncidence RatesModel Uncertainty,Functional form andaggregationAge, cause, and county-specific mortality rates were obtained fromCDC for years 1996 through 1998 by countyCDC’s age groups include <1, 1–4, 5–9, 10–14, 15–19, 20–24, 25–34, 35–44, 45–54, 55–64, 65–74, 75–84, and 85+Mortality rates are averaged across three yearsIt is assumed that rates are uniformly distributed across all ages in thereported age groupBenMAP choice Choice of incidencebaseline yearParameter uncertainty,MeasuredAge, cause, and county-specific mortality rates were obtained fromCDC for years 1996 through 1998 by countyMortality rates are averaged across three yearsBenMAP default Forecast baselineincidence ratesParameter Uncertainty,ForecastProject CDC Wonder county and age-specific mortality rates for1997 to future years using Census projections of national age-specificmortality rates from 1999 through 2050. Adjust county and agespecific 1997 CDC Wonder rate by ratio of Census projection foryear Y to the estimated Census age-specific mortality rate for 1997(1997 Census mortality rate estimated by regressing mortality ratesfrom 1999–2008 on year).All segments of the population assumed to grow at same rateInput Assigning baselineincidence to grid cellsModel Uncertainty,AggregationBaseline incidence rates assigned togrid cells in BenMAP, not clearhowAir Pollution and Exposure Estimation AlgorithmBenMAP choice Assigning air pollutiondata to grid-cellpopulation data,monitoring values versusactual exposureModel Uncertainty,AggregationPeople living within a particularair pollution model grid-cellexperience the same air pollution levels(continued)

### Amy Lamson May 21, 2009 Page 23Table 1. List of Major Sources of Uncertainty (Parametricand Model) for Mortality inDifferent Simulation Stepsof the BenMAP Model (continuedModeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMAPAir Pollution and Exposure Estimation Algorithm (continued)BenMAP choice Size of air quality gridcell/CMAQ grid cell sizeParameter Uncertainty,MeasuredSelect grid cell size based on air quality inputs, typically36km,12 km, or smaller grid cell sizeBenMAP choice Monitor Rollbacks inBenMAP(in analysis of regulation,all monitors are assumedto meet the previousstandard even if modelsforecast that they will not)Model Uncertainty,Functional formPercentage rollback: monitor observations rolled back by a fixedpercentage for values over defined background levelIncremental rollback: monitors observations rolled back by a fixedincrement for values over defined background levelRollback to standard: set values to determine if monitor in attainment,monitors not in attainment rolled back to attainment levels using oneof several methodsInput Monitor rollbacks tostandard prior to BenMAPModel Uncertainty,Functional formMonitor rollbacks conducted using air quality modeling data prior toimporting the data into BenMAPInput Air Quality Modelingfor baseline and controlstrategyParameter Uncertainty,ForecastVarious changes in emissions assumptions, inventories, growth,spatial allocation, control strategy, baselines, spatial and temporaldistributions, etc.Unquantified and non-monetized benefitsWTP to avoid pain andsuffering from the illnessleading to death (not justthe death itself)Pain and suffering byfamily/friends (WTP byfamily and friends toprevent an individual’sillness)

### Amy Lamson May 21, 2009 Page 24Values for Sensitivity Analysis: Population Estimates Reviewer Name:_______________________1Estimating population levels in non-census years—Population levels between 2000 and 2030BenMAP default, Parameter Uncertainty, ForecastAnalyzeQuantitatively?Minimum & Maximum Values or Source for Values;Functional FormsComments2Estimating population levels in non-census years—Population levels after 2030Input, Model Uncertainty, ForecastAnalyzeQuantitatively?Alternative Functional Formsor Sources for AlternativeFunctionsComments3Population distribution within an age groupInput, Model Uncertainty, Functional formAnalyzeQuantitatively?Alternative Functional Formsor Sources for AlternativeFunctionsComments

### Amy Lamson May 21, 2009 Page 254Forecasts of population from Woods and PooleInput, Parameter Uncertainty, ForecastAnalyzeQuantitatively?Minimum & Maximum Values or Source for ValuesComments5Assigning population data to grid squaresInput, Model Uncertainty, Functional formAnalyzeQuantitatively?Alternative Functional Formsor Sources for AlternativeFunctionsComments6Selection of forecast yearBenMAP choice, ParameterUncertainty, ForecastAnalyzeQuantitatively?Minimum & Maximum Values or Source for ValuesComments

### Amy Lamson May 21, 2009 Page 267AnalyzeQuantitatively?Minimum & Maximum Values or Source for ValuesComments8AnalyzeQuantitatively?Minimum & Maximum Values or Source for ValuesComments9AnalyzeQuantitatively?Minimum & Maximum Values or Source for ValuesComments

### Amy Lamson May 21, 2009 Page 27Values for Sensitivity Analysis: MortalityValuation Reviewer Name:_______________________1 VSL estimateBenMAP choice, Parameter Uncertainty, MeasuredAnalyzeQuantitatively?Minimum & Maximum Values or Source for Values;Functional FormsComments2 Income elasticity of WTPBenMAP choice, Parameter Uncertainty, MeasuredAnalyzeQuantitatively?Alternative Functional Forms or Sources for AlternativeFunctionsComments3 Income growth per personInput, Parameter Uncertainty, ForecastAnalyzeQuantitatively?Alternative Functional Forms or Sources for AlternativeFunctionsComments

### Amy Lamson May 21, 2009 Page 284 Lag structure in mortalityInput, Model Uncertainty, Functional formAnalyzeQuantitatively?Minimum & Maximum Values or Source for Values Comments5 Income elasticity and income growth adjustment to WTPBenMAP default, Model Uncertainty, Functional formAnalyzeQuantitatively?Alternative Functional Forms or Sources for AlternativeFunctionsComments6 Choice of baseline currency year for GDP and WTPBenMAP default, Parameter Uncertainty, MeasuredAnalyzeQuantitatively?Minimum & Maximum Values or Source for Values Comments

### Amy Lamson May 21, 2009 Page 297 Conversion of WTP and GDP to real dollarsBenMAP default, Parameter Uncertainty, MeasuredAnalyzeQuantitatively?Minimum & Maximum Values or Source for Values Comments8 Discount rateBenMAP choice, Parameter Uncertainty, MeasuredAnalyzeQuantitatively?Minimum & Maximum Values or Source for Values Comments9AnalyzeQuantitatively?Minimum & Maximum Values or Source for Values Comments10AnalyzeQuantitatively?Alternative Functional Forms or Sources for AlternativeFunctionsComments

### Amy Lamson May 21, 2009 Page 3011AnalyzeQuantitatively?Minimum & Maximum Values or Source for Values Comments12AnalyzeQuantitatively?Minimum & Maximum Values or Source for Values Comments13AnalyzeQuantitatively?Minimum & Maximum Values or Source for Values Comments

### Amy Lamson May 21, 2009 Page 31



### Appendix C: Slides for Sessions 2 through 7Approaches to influence diagramsand sensitivity analysis for BenMAPMax HenrionLumina Decision SystemsLos Gatos, CAC l M fi ld d A i M kht iCarol Mansfieldand AmirMokhtariRTI InternationalResearch Triangle Park, NCUSA EPA,Research Triangle Park, NC15th December 2008Session 3:Overview of the simplifiedinfluence analysis model andproject strategyMax HenrionMaxHenrionLumina Decision SystemsLos Gatos, CAObjectives of theprojectpj•To identify the sources of uncertainty in theassessment of benefits from improvementsin air quality using BenMAP•To visualize how the sources of uncertaintyaffect the results using influence diagrams•To assess the range of values of keyuncertainties•To compare their relative contributions tothe results using sensitivity analysis•To identify which uncertainties are most inneed of more careful characterizationOverview of strawman proposalfor the approach•Identify and classify the uncertainties and theireffectsDevelop an inventory of possible sources of uncertaintyClassify these sources in terms of source and type ofuncertaintyDevelop influence diagrams to visualize their effects onb fienets•Build model and conduct sensitivity analysisDevelop a simplified or reduced-form modelA f l ibl l f iAssessrangesofplausiblevaluesforparametricuncertainties, and ranged of plausible alternative forms formodel uncertaintiesConduct sensitivityanalysis to identifytheir relativey yimportance on results using various measures of sensitivity•Compare the simplified model with BenMAP forselected scenarios to validate and calibrate it.•Make recommendations on which uncertainties aremost important to characterize in more detail andhowExample tornado diagram forsensitivity analysisAny numbers and charts are purely illustrative.Classifyinsources of uncertaintyg y•Uncertainties or assumptions within BenMAP:Choice: user can select value or option – e.g. forecast yearfor analysis discount ratefor analysis,discount rateInput: from input files – e.g. population projectionsDefault: method not easily modifiableOmitted: issue not included in model–e g WTP to avoide.g.WTP to avoidpain and suffering, separate from mortality•Uncertainties thatprecede BenMAP– such as:Inputs and assumptions in emissions and air transportmodels, such as CMAQ,air quality measurementsl i j i h f W d & P l BEApopulationprojections,suchasfromWoods& Poole,BEA,and US Census, by country, aggregated to 30km grid byPopGridWe examine the uncertaintyin these inputs to BenMAPpBut analysis of their sources of uncertainty is out of scopeof this project.Classifyingtypes of uncertaintyg yp y•Parameter uncertaintiesMeasured(estimated)current orpast values –) pe.g. population in recent census, current incomelevels, or incidence ratesForecastor projection of future values–e.g.future population or incomeUser choices- e.g. year for analysisP li i bldi t t VSL(?)Policyvariables– e.g.discountrate,VSL(?)•Model uncertaintiesFunctional form–e g for concentratione.g.for concentrationresponse function, or mortality lag from exposureAggregation– e.g. population from counties to30km gridWhat is an influence diagram?Arrows showinfluencesbetween variablesDecisionChancevariablesSimplevariableObjectiveIndexModuleWhat is Analytica?•Transparency:Visual influence diagramsAnalytica is a visual tool for building and deployinganalytic applications. It offers:make models easier to create,understand, and audit.•Flexibility:Intelligent ArraysTM make ityeasy to build and extend multidimensional models (data cubes)•Scalability:Hierarchical modules,I t llig t A d ltt dIntelligent Arrays,andultra-compaccodelet you manage and run models muchlarger than is practical with spreadsheets.•Risk analysis:Integrated Monte Carlosimulation enables fast evaluation of riskand uncertainty“Everything that’s wrong with the common PCspreadsheet is fixed in Analytica”,PC WeekOverview of BenMAP elementsf B MAP U G idfromBenMAP UserGuideInfluence diagram of BenMAPstructureAir qualitypolicychangeCurrent airqualityfromFuture airqualityfromAir qualitymodelingCensusq ymodelq ymodelgdataEstimatedi liIncidence andprevalenceValuationf ipopulationdataair qualityAdverseprevalenceratesfunctionsPopulationestimatesPopulationexposureAdversehealtheffectsEconomiccostsPopulationprojectionsHealthfunctionsTop level influence diagram:From strawman modelInfluence diagrams forselected submodulesWhy create a simplified model ofBenMAP?•BenMAP is too large and takes too long toperform a large number of runs forsensitivity analysis or Monte Carlo.•For some kinds of uncertaintyanalysis,youy ,yneed to modify model structure to handledependencies appopriately.•To explore the effects of alternativefunctional forms or aggregation, it’s a loteasier to do on a simplified modelN t Th i lifi d d l i t•Note:Thesimplifiedmodel isnotasubstitute for BenMAP: It could only be builtbecause BenMAP exists and is available tocalibrate it. It does not do mapping oranalysis by grid square.Binning grid square datato simplify the analysis•We import data sets by36k id36kmgridsquares:PopulationCurrent PM2.5concentrationDelta concentration wcontrol•We aggregate the gridsquare data into bins bybase concentration levels•This simplifies from16,576 grid squares toabout 16 bins - a factor of~ 1000 times smallerResults of binningPopulation by base concentration binsDelta conc. by base conc. binsSensitivity to the level ofaggregation: By number of binsb dlMore concentrationins means moredetail,and decreasing aggregation.Sensitivity to the level ofaggregation: By number of binsh l ld d hThe sensitivity to aggregationleveldepends on thethreshold of C-R function. For zero threshold, number ofbins has no effect on results. (Illustrative numbers only.)Where are we in this process? And whatare we hoping to get from this workshop?•We have an initial inventory of sources oft i t W t h l t id tifuncertainty:Werequestyourhelptoidentifyoneswe’ve missed, and to review how they’re classifiedand characterizedW h d l d t i fl di•Wehavedevelopedstrawmaninfluencediagrams,for your comments.•We have an initial simplified model capable ofillustrative sensitivity analysis We seek advice onillustrative sensitivity analysis.We seek advice onhow to refine it.•Ranges of parameter values: We request your helpin providing plausible ranges or suggesting how toin providing plausible ranges,or suggesting how toobtain them.•Model form and aggregation uncertaintiesW ld lik d i th t•Wewould likeyouradviceonthestrawmanproposed process for sensitivity analysisSession 4:P l ti E ti t :PopulationEstimates:Identified sources of uncertainty inpopulation estimates and uncertaintyintroduced by BenMAP modelingassumptionsMax HenrionLumina Decision SystemsL G t CALosGatos,CADraft influence diagram forpopulation projectionsRetrospective accuracy ofpopulation projectionsStandard deviation error percent for Woods & Poole projections ofpopulation compared against census data.From 2008 Technical Description of Woods & Poole projections, table on page 20.How should we estimate uncertaintyi US l ti j ti ?inUSpopulationprojections?(age group 30 to 99)Ranges for uncertainparametersp•We want plausible low, mid, and high valuesnot theoretically possible extreme min and max.•Think of them as 10thpercentile, median and90thpercentilelth h ’ t l ki f t b bilitalthough,we’renot lookingforexpertprobabilitydistributions for now.•We’d like your estimates for the rangesbh b hOr suggestions aouthow to obtain them•For functional form, what set of functionsshould be considered?•How can we estimate aggregationuncertainties?e.g. population from county to grid sqaresSession 4: Discussion on populationti tesmates5min: Quiet brainstorming on these questions:•Are there additional sources of uncertainty in population forecasts orintroduced by BenMAP’s use of the population data?•Do you think this will have big impact on benefits results?15min: Report out and discussion10min: Quiet brainstorming on these questions:•Can/should this source of uncertainty be modeled quantitatively in thei fl l i ?influenceanalysis?•What range of values should we use for these parameters or where canwe get the range (i.e., from a study or existing data)?•Are there other functional forms or modeling approaches that makef thi t f th d l h t ld b dsenseforthiscomponentof themodelorwhatwould beagoodsourcefor this information?30min Small group discussion•Where do we agree and where do we disagree?45min Report out and discussionSession 5:Valuation of mortalityreductionIdentified sources of uncertainty invaluation of mortality introduced by BenMAPMax HenrionLumina Decision SystemsLos Gatos, CAValuation influence diagramDraft influence diagramfor Valuation ofstatistical lives saved,includes•Value of a statisticallife•Elasticity of WTP toincome•Income indexGDP growth ratePopulation growth rate•Select forecast year•Inflation index andbaseline currency year•Discount due tomortality lag and lagstructureMortality lag structuresSix lag types representing modeluncertainty about lag betweenexposure and mortality:•None (no lagg in mortality)•15-year: All deaths happen 15years after exposure•20-year distributed:5di t ib t d•None (no lagg in mortality)•8-year: All deaths happen 8 yearsafter exposure•5-yeardistributed:•Exponential declining over 20yearsMortalitylastructureg•Using an annualdiscount rate, wet h lconvereacagstructure into acorresponding totaldiscountrateonbenefitsEffect of lagte on benefitsypRanges for uncertainparametersp•We want plausible low, mid, and high valuesnot theoretically possible extreme min and max.•Think of them as 10thpercentile, median and90thpercentilelth h ’ t l ki f t b bilitalthough,we’renot lookingforexpertprobabilitydistributions for now.•We’d like your estimates for the rangesbh b hOr suggestions aouthow to obtain them•For functional form, what set of functionsshould be considered?•How can we estimate aggregationuncertainties?e.g. population from county to grid sqaresSession 5: Discussion on valuationof mortality reduction5min Quiet brainstorming on:•Are there additional sources of uncertainty in valuation of mortalityreductions or introduced by BenMAP’s use of valuation studies?•Do you think these will have bigimpact on benefits results?15min Report out and discussion10min Quiet brainstorming on:•Can/should this source of uncertainty be modeled quantitatively in theinfluence analysis?Wh f l h ld f h h•Whatrangeofvaluesshouldweusefortheseparametersorwherecanwe get the range (i.e., from a study or existing data)?•Are there other functional forms or modeling approaches that make sensefor this component of the model or what would be a good source for thisinformation?30min Small group discussion•Where do we agree and where do we disagree?45min Report out and discussion5:15 to 5:30: Wrap-upS i 6ession6:Valuation of morbidityreductionIdentified sources of uncertainty invaluation of morbidity introduced by BenMAPValuation of morbiditySession 6: Discussion on valuationof morbidity reduction5min Quiet brainstorming on:•Are there additional sources of uncertainty in valuation of mortalityreductions or introduced by BenMAP’s use of valuation studies?•Do you think these will have bigimpact on benefits results?15min Report out and discussion10min Quiet brainstorming on:•Can/should this source of uncertainty be modeled quantitatively in theinfluence analysis?Wh f l h ld f h h•Whatrangeofvaluesshouldweusefortheseparametersorwherecanwe get the range (i.e., from a study or existing data)?•Are there other functional forms or modeling approaches that make sensefor this component of the model or what would be a good source for thisinformation?30min Small group discussion•Where do we agree and where do we disagree?15 mins Break(10:00am to 10:15am))30min Report out and discussionSession 7:Di i d R d tiDiscussionand Recommendationson Approach to influenceand uncertainty analysisMax HenrionLumina Decision SystemsObjectives of the project:Recap•To identify the sources of uncertainty in theassessment of benefits from improvementsin air quality using BenMAP•To visualize how the sources of uncertaintyaffect the results using influence diagrams•To assess the range of values of keyuncertainties•To compare their relative contributions tothe results using sensitivity analysis•To identify which uncertainties are most inneed of more careful characterizationOverview of proposed approach:Recap•Identify and classify the uncertainties and theireffectsDevelop an inventory of possible sources of uncertaintyClassify these sources in terms of source and type ofuncertaintyDevelop influence diagrams to visualize their effects onb fienets•Build model and conduct sensitivity analysisDevelop a simplified or reduced-form modelA f l ibl l f iAssessrangesofplausiblevaluesforparametricuncertainties, and ranged of plausible alternative formsfor model uncertaintiesConduct sensitivityanalysis to identifytheir relativey yimportance on results using various measures of sensitivity•Compare the simplified model with BenMAP forselected scenarios to validate and calibrate it.•Make recommendations on which uncertainties aremost important to characterize in more detail andhowRefiningthe simplified modelp•Concentration reductionsInitially, we assume that each all exposed population in eachbase concentration bin experiences the same reduction inbaseconcentrationbinexperiencesthesamereductioninPM2.5 concentration levels (averaged over the reductionsfor all grid squares in that bin)Later, if time we could expand it to a bin array with a seconddimension of concentration reductions for eachconcentration bin, and see ifthis reduces the aggregationuncertainty.•Population segmentsPopulationsegmentsInitially, the model ignores population segments (sum overage groups, ethnic groups, and gender).This is only relevant for CR-functions that are disaggregatedb ( l l bidi li f iby segment(currently, only morbidity not mortalityfunctions.Later, if time, we might add a dimension of segments to thebins, with separate population growth rates and incidencerates bysegment.gSensitivityanalysisy•Tornado diagramsWe will conduct deterministic sensitivity analysisll co duct dete st c se s t v ty a alys s(Tornado diagram) over all parameter and modeluncertainties, varying each from low to high values,leaving all others at their mid value.•For selected pairs or groups of uncertain variableswhere knowledge of the functional form leads us tosuspect that interactions may be important and at leastone parameter is individually important, we will alsod l i i i i h i ( lconductmultiwaysensitivity,changingtwo(orrarelymore) parameters at a time.•We may also use probabilistic sensitivity analysismeasures such as rank correlations andmeasures,such as rank correlations andregression measuresassuming a simple uniform or beta distributions overparameter rangesgand uniform discrete distributions over alternativefunctional formsExample tornado diagramgExample “importance analysis” tocompare input uncertaintiesThe relative importance of each uncertain input to the benefitscomputed as the rank correlation of the Monte Carlo samplefor the benefits with respect to each input sampleIterative refinement of model anditi it lsensitivityanalyses•Based on the initial analysis,we will identify a subset of parametersas the most important contributors anddevelop more careful characterizationof their uncertainty, by reviewingrelevant literature and/or moreextended conversation with one or moreexperts on those quantities to refine thelow, mid, and high values.•We will repeat the sensitivityl i th fi d ti tanalysesusingtherefinedestimates.Proposed comparison of simplifiedmodel with BenMAP scenarios•Identify the most important set ofparametersandmodel forms(perhaps 5 to 8),from initial sensitivityp ), yanalysis with the simplified model•Define 5 to 10 scenarios -- i.e. combinations ofvalues of theparameters–toperform sensitivityrunsyof BenMAP.The number of scenarios will depend on how many BenMAPruns are practical given human and computational resourcesC th lt f B MAP ith th i lifi d•Comparethe resultsfromBenMAPwiththe simplifiedmodelDirect values, including incidence, mortality reduction, andbenefitsSensitivities: Relative size and ranking of sensitivitiesDiagnose the source of any differences•If time we may refine the simplified model to toIftime,wemayrefinethesimplifiedmodeltotoimprove correspondence and rerun the sensitivity .Results and recommendations fromthe project•Summarize the results, identifying thewhich sources of uncertainty are mostcritical to uncertainty in the results (andwhich are less so)•Recommend how those uncertainties mightbe refined, e.g.d l f b blBy conducting expert elicitations ofprobabilitydistributions on uncertain quantitiesExperiments on effects of alternativeaggregationsObtaining other sources of data or runs ofpreceding modelsDiscussionRelated models that provide inputsto BenMAP•Sensitivity analysis for CMAQ or other models are outside the scope ofhi j hi h f l B MAPthisproject,which focusesonlyonBenMAP.•Although most models use population projections as shown in thediagram, they currently do not necessarily use the same projections.

### Appendix D: Influence DiagramsInfluence diagrams to supportsensitivity and uncertainty analysisof BenMAPMax HenrionLumina Decision SystemsLos Gatos, CATop level influence diagramto represent BenMap structureInfluence diagrams forselected submodulesDraft influence diagram forpopulation projectionsValuation of mortalityinfluence diagramDraft influence diagramfor Valuation ofstatistical lives saved,includes•Value of a statisticallife•Elasticity of WTP toincome•Income indexGDP growth ratePopulation growth rate•Select forecast year•Inflation index andbaseline currency year•Discount due tomortality lag and lagstructureValuation of morbidityRelated models that provide inputsto BenMAP•Sensitivity analysis for CMAQ or other models are outside the scope ofhi j hi h f l B MAPthisproject,which focusesonlyonBenMAP.•Although most models use population projections as shown in thediagram, they currently do not necessarily use the same projections.