November 2009 

Influence Analysis in Support of 

Characterizing Uncertainty in 

Human Health Benefits Analysis 

Final Report

Prepared for 

Amy Lamson 

U.S. Environmental Protection Agency 

Office of Air Quality Planning and Standards (OAQPS) 

Air Benefit and Cost Group (ABCG) 

 (MD-C439-02) 

Research Triangle Park, NC 27711 

Prepared by 

Carol Mansfield 

Paramita Sinha 

RTI International 

Research Triangle Park, NC 27709 

Max Henrion 

Lumina Decision Systems 

EPA Contract Number EP-D-06-003 

RTI Project Number 0209897.003.065 
EPA Contract Number EP-D-06-003 

RTI Project Number 0209897.003.065 

Influence Analysis in Support of 

Characterizing Uncertainty in 

Human Health Benefits Analysis 

Final Report

November 2009

Amy Lamson 

U.S. Environmental Protection Agency 

Office of Air Quality Planning and Standards (OAQPS) 

Air Benefit and Cost Group (ABCG) 

(MD-C439-02) 

Research Triangle Park, NC 27711 

Prepared by 

Carol Mansfield 

Paramita Sinha 

RTI International 

Research Triangle Park, NC 27709 

Max Henrion 

Lumina Decision Systems 
iii 

CONTENTS 

Section Page

1. Introduction ................................................................................................................... 1-1 

2. Approach ....................................................................................................................... 2-1 

2.1 Overview of Approach ......................................................................................... 2-1 

2.2 Identify and Classify the Uncertainties ................................................................ 2-1 

2.2.1 Sources of Uncertainty ............................................................................. 2-2 

2.2.2 Types of Uncertainty................................................................................ 2-2 

2.3 Assess Ranges for Parameters and Alternative Functional Forms ...................... 2-2 

2.4 Develop Influence Diagrams ............................................................................... 2-4 

2.5 Use of Analytica .................................................................................................. 2-5 

2.6 Develop a Simplified Model to Conduct Sensitivity Analysis and Compare 

to BenMAP .......................................................................................................... 2-5 

2.6.1 Calibration to BenMAP ........................................................................... 2-6 

2.7 Analysis Strategy ................................................................................................. 2-7 

3. Classify Uncertainties and Quantify Parameter Ranges ............................................... 3-1 

3.1 Final List of Uncertainties ................................................................................... 3-1 

3.2 Quantify Ranges for Parameters .......................................................................... 3-4 

3.3 Source for Range Recommendations ................................................................... 3-4 

3.4 Alternative Functional Forms .............................................................................. 3-9 

4. Influence Diagrams ....................................................................................................... 4-1 

4.1 Introduction to Influence Diagrams ..................................................................... 4-1 

4.2 Influence Diagrams for Sensitivity Analysis ....................................................... 4-2 
iv 

5. Results ........................................................................................................................... 5-1 

5.1 Assumptions for Analysis .................................................................................... 5-1 

5.2 Measures of Sensitivity Used in Analysis ........................................................... 5-1 

5.3 Elasticity Results .................................................................................................. 5-2 

5.4 Results for Different Concentration-Response Functions for Mortality and 

Morbidity ............................................................................................................. 5-4 

5.4.1 Ranges for Concentration-Response Functions from Published 

Epidemiological Studies for Mortality and Morbidity............................. 5-4 

5.4.2 Ranges from Expert Elicitations of Probability Distributions on CR 

Functions .................................................................................................. 5-5 

5.5 Range Sensitivity of Mortality Avoided Outcome ............................................ 5-11 

5.6 Range Sensitivity for Total Monetized Benefits Outcome ................................ 5-15 

6. Discussion and Possible Next Steps ............................................................................. 6-1 

6.1 Discussion of Model Results ............................................................................... 6-1 

6.1.1 Comments on the Uncertainty Analysis Methods ................................... 6-2 

6.2 Potential Next Steps ............................................................................................. 6-3 

6.2.1 Update Studies to Support Morbidity Valuation ..................................... 6-3 

6.2.2 More Detailed Mortality Valuation Estimates ......................................... 6-4 

6.2.3 Framework to Jointly Consider Multiple Health Endpoints .................... 6-4 

6.2.4 Add Additional Outcomes and Expand Application to Other 

Pollutants.................................................................................................. 6-4 

6.2.5 Expanding Scope of the Uncertainty Analysis to Include Air 

Quality...................................................................................................... 6-5 

6.2.6 Investigate Cost Uncertainty .................................................................... 6-5 

6.2.7 A More Comprehensive Treatment of Uncertainties ............................... 6-5 

6.2.8 Large Changes in Population, Migration and Immigration ..................... 6-6 

6.2.9 Additions to Analytica Model .................................................................. 6-6 

7. References ..................................................................................................................... 7-1 

Appendix 

A: Expert Workshop Memo ..................................................................................... A-1
v 

LIST OF FIGURES 

Number Page

2-1. Hierarchical Influence Diagram Showing the Top Level Model and Three 

Selected Subdiagrams ................................................................................................... 2-4

2-2. 36 km Grid Squares over the Contiguous United States ............................................... 2-6

4-1. Illustrative Influence Diagram with Notation Defined ................................................. 4-1

4-2. Top-Level Influence Diagram ....................................................................................... 4-2

4-3. Influence Diagram for PM2.5 Concentration ................................................................. 4-3

4-4. Influence Diagrams for Incidence Rates ....................................................................... 4-3

4-5. Influence Diagram for Health Impacts CR Functions .................................................. 4-4

4-6. Influence Diagram for Expert Elicitation Distributions ................................................ 4-5

4-7. Influence Diagram for Population Projections ............................................................. 4-5

4-8. Influence Diagram for Valuation Modules ................................................................... 4-6

4-9. Influence Diagram for Inflation and Income Growth ................................................... 4-7

4-10. Influence Diagram for Value of Avoided Morbidity Endpoint .................................... 4-7

4-11. Influence Diagram for Mortality Valuation Lag ........................................................... 4-8

4-12. Mortality Lag ................................................................................................................ 4-9

4-13. Influence Diagram for Air Quality ............................................................................... 4-9

5-1. Elasticity of Mortality Parameters ................................................................................ 5-3

5-2. Elasticity of Total Monetized Benefits to Each Uncertain Parameter .......................... 5-4

5-3. Ranges of Beta for CR Functions ................................................................................. 5-6

5-4. Ranges of Beta for CR functions from Experts A to L ................................................. 5-8

5-5. Range of Avoided Premature Mortalities and Morbidities (CB and AMI) for 

Low, Mid, and High Values for Each CR Function ..................................................... 5-9

5-6. Range of Monetized Benefits for Low, Mid, and High Values for Each CR 

Function ...................................................................................................................... 5-10

5-7. Range Sensitivity of Avoided Mortality for Pope et al. with Zero Threshold ............ 5-12

5-8a. Range Sensitivity of Avoided Mortality for Pope et al. with a 10 ug/m3

Threshold .................................................................................................................... 5-12

5-8b. Range Sensitivity of Avoided Mortality for Pope et al. with a 12 ug/m3

Threshold .................................................................................................................... 5-13

5-9. Range Sensitivity of Avoided Mortality for Laden et al. with Zero Threshold .......... 5-13

5-10. Range Sensitivity of Avoided Mortality for Pope et al. with Mid Threshold of 

7.5 μg/m3

 ..................................................................................................................... 5-14
vi 

5-11. Range Sensitivity of Avoided Mortality for Laden et al. with 10 ug/m3

Threshold .................................................................................................................... 5-14

5-12. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Pope et al. with Zero Threshold as the Mid Value ............................. 5-15

5-13. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Pope et al. with Zero Threshold as the Mid Value with an 

Exponential Declining Mortality Valuation lag Structure .......................................... 5-16

5-14. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Laden et al. Zero Threshold as the Mid Value ................................... 5-17

5-15. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Expert E .............................................................................................. 5-17

5-16. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Expert K .............................................................................................. 5-18

5-17. Discount by Lag Structure .......................................................................................... 5-19

5-18. Range Sensitivity of Total Monetized Benefits to Discount Rate Assumptions 

for Different Mortality Valuation lag Structures ........................................................ 5-19
vii 

LIST OF TABLES 

Number Page

3-1. List of Major Sources of Uncertainty in the BenMAP Model ...................................... 3-1

3-2. Range Values for Uncertain Parameters ....................................................................... 3-5

5-1. Ranges for Epidemiological Studies ............................................................................. 5-5

5-2. Probability Distributions Elicited from Experts A through L ....................................... 5-7

5-3. Number of Cases of Mortality or Morbidity Avoided Using the Low, Mid, and 

High Values for the Beta from the CR Function in Each Source ............................... 5-11
1-1 

SECTION 1 

INTRODUCTION 

The U.S. Environmental Protection Agency’s Office of Air Quality Planning and 

Standards (USEPA OAQPS) prepares estimates of the benefits and costs of air quality 

regulations. OAQPS also assesses the sensitivity of the benefits and costs to uncertainties in the 

data and assumptions used for their analyses. As part of EPA’s efforts to assess uncertainty, RTI 

and Lumina Decision Systems developed an influence and uncertainty analysis model for 

BenMAP (USEPA, 2008). BenMAP is a software package widely used at the U.S. EPA and 

elsewhere to estimate the benefits of improvements in air quality. 

This document outlines the methods and results of the influence and uncertainty analysis 

of the BenMAP estimates of the benefits from air quality improvements. The BenMAP influence 

analysis focuses on a selection of important but uncertain parameters, functions, and input data 

used to calculate the human health benefits of reductions in PM2.5. For health outcomes, the 

model examines mortality, acute myocardial infarction (AMI) and chronic bronchitis (CB). The 

results of this analysis may be used to guide decisions about how to prioritize future efforts to 

characterize these uncertainties and, where possible, reduce them. 

The goals of the project were to identify those input parameters and assumptions that 

have the potential to be significant contributors to the uncertainty in the results of BenMAP, to 

compare the potential size of their contributions, and to identify those likely to make the largest 

contributions. 

By influence analysis, we mean a graphic depiction of the model structure in the form of 

a hierarchy of influence diagrams. Each variable, including input assumptions, decisions, 

intermediate variables, and results, is depicted as a labeled node. Influence arrows between the 

nodes depict relationships where one variable influences another via a mathematical relationship 

in the underlying model. 

By uncertainty analysis, we mean the process of comparing the effect of the uncertainty 

in each input variable on a result variable. Range sensitivity analysis is a method of uncertainty 

analysis that depicts the range of values in a result variable produced by changing each uncertain 

input from a low to high value, while leaving all the other inputs at a nominal or mid value. We 

will examine sensitivities by assessing plausible ranges on each input and computing 

deterministic measures of sensitivity to compare their effects on selected output variables. 
1-2 

The focus of this project was to conduct an uncertainty analysis of BenMAP as a tool to 

estimate the benefits of improvements in air quality. In order to keep the level of effort for the 

project within the resources available, the scope was defined to include the effects of uncertainty 

in the explicit inputs to BenMAP, including: 

 Particulate matter under 2.5 microns (PM2.5), and no other types of atmospheric 

pollutant. 

 Annual average concentration of PM2.5, ignoring any effects of short-term peak 

concentration except insofar as they affect annual averages. 

 Improvement in air quality was specified in terms of current annual average 

concentration of PM2.5 and a future, usually lower, concentration of PM2.5 at a future 

date, typically 2020, and excluding any preceding variables or factors that might 

affect PM2.5, such as changes in emissions, or climate, as modeled in CMAQ and 

other computer models. 

 Spatial resolution was to cover the contiguous United States (excluding Alaska, 

Hawaii), in 36-kilometer grid squares, and no smaller regions such as counties. 

 Affected population was by age range, ignoring effects of gender, race, 

socioeconomic status, or other differentiators. 

 Effects of improved air quality are to be examined on reduction in mortality and two 

types of morbidity, CB and AMI, and no other end points, such as other diseases or 

visibility. 

 Treatment of uncertainty was explored by defining a range of values for key 

parameters, and their effects on results while holding all other parameters at their mid 

values (range sensitivity analysis). Probabilistic treatment of uncertainties about input 

parameters using expert probability distributions and Monte Carlo analysis to 

propagate probabilistic uncertainties were explicitly beyond the scope of this project. 

The rest of the report is structured as follows: Section 2 presents the approach taken for 

the sensitivity analysis, followed by a discussion of the uncertain parameters and functions in 

Section 3 and the influence diagrams associated with the model in Section 4. The results can be 

found in Section 5, followed by a discussion of the results and possible implications of the 

results for characterizing uncertainty in Section 6. 
2-1 

SECTION 2 

APPROACH 

2.1 Overview of Approach 

The goals of the project are to identify those input parameters and assumptions that have 

the potential to be significant contributors to the uncertainty in the results of BenMAP, to 

compare the potential size of their contributions, and to identify those likely to make the largest 

contributions. This outline shows the steps involved in conducting the influence and uncertainty 

analysis, which are described in more detail below. 

 Identify and classify the uncertainties 

– Develop an inventory of possible sources of uncertainty 

– Classify these sources in terms of source and type of uncertainty 

– Select which of these uncertainties should be treated quantitatively in the 

sensitivity analysis 

– Assess ranges of plausible values for selected parametric uncertainties and ranges 

of plausible alternative forms for model uncertainties 

 Build simplified model and conduct sensitivity analysis 

– Develop influence diagrams to trace which sources of uncertainty have effects on 

which results 

– Develop a simplified or reduced-form model 

– Compare the simplified model with BenMAP for selected scenarios to validate 

and calibrate it. 

– Conduct sensitivity analysis using ranges to identify the relative importance of the 

uncertainty associated with different parameters and model assumptions on the 

results using various measures of sensitivity 

 Provide the results and discussion of the results to EPA, so that EPA can use the 

information to assess which uncertainties are most important to investigate in more 

detail 

2.2 Identify and Classify the Uncertainties 

The first step is to identify and classify the sources of uncertainty in BenMAP. The team 

developed an inventory of sources of uncertainty by reviewing BenMAP, including its input files 

and documentation. Where BenMAP offers different functional forms, we identified an explicit 

source of uncertainty. Where it uses a particular functional form and some other functional form 

appeared reasonable, we identified that as an implicit model uncertainty. An exception is the 
2-2 

uncertainties in emissions and air quality whose explicit quantification is outside the scope of 

this study. 

We classified each source of uncertainty according to its source and type, using the 

following categories: 

2.2.1 Sources of Uncertainty 

 Explicit parameters inside BenMAP, where BenMAP makes it easy to select a value 

as an input. 

 Input parameters generated outside of BenMAP, such as emissions or population 

distribution. 

 Implicit parameters, where BenMAP uses a single value, but there are other 

plausible values not offered as an explicit input to BenMAP. 

 Explicit model forms selectable within BenMAP such as the functional form for 

concentration-response (CR) functions. 

 Implicit model forms used in BenMAP where there are other plausible model forms 

not offered as an explicit input to BenMAP. 

2.2.2 Types of Uncertainty 

 Parameter uncertainty 

– Measured: Uncertainty in measured quantities due to measurement biases and 

limited sample size, for example, population estimate from past census, mortality 

data, willingness to pay (WTP) to avoid a health outcome or the estimate of beta 

in CR functions 

– Forecast: Uncertainty in forecasts based on expert judgment and/or extrapolation 

from past trends, for example, future population, Gross Domestic Product (GDP), 

or PM2.5 concentrations. 

– Values: These are used to quantify outcomes such as avoided premature mortality 

and morbidity into dollars, for example, the value of a statistical life (VSL), the 

discount rate, or cost of illness (COI) estimates. 

 Model uncertainty 

– Functional form: Due to explicit or implicit choice of functions, for example, CR 

functions or functional form for forecast of population growth. 

2.3 Assess Ranges for Parameters and Alternative Functional Forms 

After preparing a basic list of potential uncertainties, OAQPS organized a workshop to 

provide advice on additional sources of uncertainty, approaches to quantifying and evaluating 

uncertainty, and the range of values or functional forms that could be used in the analysis. The 
2-3 

workshop included experts from inside and outside EPA on the topics of population forecasting, 

monetary valuation of mortality and morbidity endpoints, and techniques for assessing 

uncertainty. Appendix A contains a memo summarizing the workshop. EPA staff with expertise 

on the health impacts of PM2.5 and staff involved with estimating and forecasting baseline 

emissions and changes in ambient air quality were also consulted. The team also conducted 

limited literature reviews, although a full literature review to update the information currently in 

BenMAP was outside the scope of this project (see discussion in Section 6 on this topic). 

After establishing the sources of uncertainty and classifying their source and type, we 

selected which to treat quantitatively in the sensitivity analysis. We included those parameters 

that appear to have significant uncertainty, those model uncertainties that are explicit in 

BenMAP, and a few implicit model uncertainties where we could identify a reasonable 

alternative to that assumed in BenMAP. 

Next, a plausible range was selected for each uncertain parameter. The range for each 

parameter includes three values: 

Mid value: This is a central or nominal value used for initial analysis for comparison 

with BenMAP; usually the same values used for the BenMAP runs provided by EPA to 

calibrate the model. The values from BenMAP are not necessarily mean or median 

values, but in all cases the values are within the range we evaluate (between the low and 

high values). From a probabilistic perspective, the mid value might be the most likely 

(mode), mean, or median value. However, for most of the values we do not have 

information about the distribution, and we did not perform an explicit probabilistic 

assessment in this study. 

Low value: This is a low value that is plausible, but not the smallest conceivable value or 

minimum. If we had a distribution, the low value might be about the tenth percentile—a 

value such that someone with expertise on the topic would be surprised if the true value 

turned out to be smaller. Without a distribution, judgment was used to select values that 

seemed like plausible lower bounds. 

High value: This is a high value that is plausible, but not usually the largest conceivable 

value or maximum. From a probabilistic perspective, it might be about the ninetieth 

percentile—a value such that someone with expertise on the topic would be surprised if 

the true value turned out to be larger. Without a distribution, judgment was used to select 

values that seemed like plausible upper bounds. 

For a few inputs, notably the mortality CR functions obtained from expert elicitation, we 

have their uncertainty assessed explicitly in the form of probability distributions (IEc, 2006). For 

those quantities, we chose 10th, 50th (median), and 90th percentiles of those distributions as the 

low, mid, and high values for use in range sensitivity analysis. Assessing probability 
2-4 

distributions for other quantities and conducting a probabilistic sensitivity analysis was outside 

the scope of this project. 

For model uncertainties, we compare the effects of two or more model formulations 

(functional forms). 

2.4 Develop Influence Diagrams 

The initial hierarchy of influence diagrams shows how uncertain input parameters and 

model form choices influence the resulting output variables, including avoided premature 

mortality and net benefits. To avoid creating a single diagram of overwhelming complexity, we 

organized the diagrams using Analytica modules into a hierarchy of diagrams, each focusing on 

a particular aspect of the model. Figure 2-1 provides an example of the hierarchy of diagrams. 

Section 4 presents and discusses the final influence diagrams. 

Figure 2-1. Hierarchical Influence Diagram Showing the Top Level Model and Three 

Selected Subdiagrams 
2-5 

2.5 Use of Analytica 

Analytica is a general-purpose software environment for building, analyzing, and 

communicating quantitative models (Analytica User Guide, Release 4.1, Lumina Decision 

Systems, Inc. 2008. www.Lumina.com). We used Analytica to create the hierarchy of influence 

diagrams and to construct a simplified model of BenMAP that we used to conduct the sensitivity 

analysis described in this report. Analytica lets users draw in influence diagrams by creating and 

positioning nodes and drawing influence arrows to connect them. It provides a general 

mathematical modeling language that we used to construct the model. It also allows probabilistic 

modeling, defining variables with probability distributions, and using Monte Carlo or Latin 

hypercube sampling to propagate uncertainties through the model. Although full probabilistic 

analysis was outside the project scope, we did use the Monte Carlo facilities to obtained low, 

mid, and high values (as 10th, 50th, and 90th percentiles) for the slope (beta) of mortality 

concentration-response functions that had been assessed as a combination of probability 

distributions as the product of an elicitation of a set of twelve experts (IEc, 2006). 

2.6 Develop a Simplified Model to Conduct Sensitivity Analysis and Compare to 

BenMAP

The next step was to build a model in Analytica based on the initial influence diagrams 

that corresponds to the components of a BenMAP benefits analysis but is simplified so that it can 

run rapidly enough to perform extensive sensitivity analysis. The reasons to build the simplified 

model include: 

 BenMAP is too large and takes too long to compute to be able to perform a large 

number of runs to support extensive sensitivity and uncertainty analysis. 

 Some kinds of sensitivity analysis require modification of the model structure, which 

is much easier on a simplified model. Comparing some kinds of model form 

uncertainty and assessing the imprecision introduced by aggregation are two 

examples. 

Note that the simplified model is not a substitute for BenMAP. The simplified model 

could only be built and calibrated because BenMAP exists. The simplified model does not 

duplicate many of BenMAP’s functions, including aggregating data into grid squares and 

generating maps of inputs and results. 

The key simplification is to reduce the geographic aggregation from the 16,576 36km 

grid squares used for BenMAP runs, displayed in Figure 2-2, to a much smaller set of bins. The 

input grid square data includes the population for 2020, annual average concentration of PM2.5 at 

baseline, planned reductions in emissions, and baseline-incidence data. We aggregate the 
2-6 

Figure 2-2. 36 km Grid Squares over the Contiguous United States 

population from the grid squares into bins, with a bin for each base concentration from 0 to 1, 1 

to 2, up to 15 to 16 micrograms/cubic meter of PM2.5. In this way, we reduce the computational 

effort by a factor of about 1,000, from over 16,000 grid squares to just 16 bins. For each bin, we 

use the population-weighted average for the base concentration and the delta concentration in the 

simplified model. 

2.6.1 Calibration to BenMAP 

The Analytica version of the model approximates the results of BenMAP, but it does not 

match the results exactly due to its simplified representation and there may be some small 

differences in the implementation of algorithms in BenMAP. These cause modest differences in 

the estimated number of cases reduced (morbidity and mortality) and the total benefits. To 

increase the accuracy of the sensitivity analysis, we calibrated the Analytica model to selected 

results of BenMAP for both number of cases of mortality and morbidity and the value of those 

cases, for each of the CR functions. The calibration multiplies each Analytica result by a factor 

so that it produces identical results to the corresponding BenMAP runs, with two exceptions: 

1. For Pope et al. (2002) CR function with threshold of 15 μg/m3, the Analytica model 

generated zero cases, since the single grid square with base concentration above 15 

μg/m3 (in Los Angeles at 15.05 μg/m3) was included in 15 μg/m3 bin, with a 

population-weighted average base concentration of 14.87 μg/m3, and so below the 

threshold. 
2-7 

2. For Expert K, who specified a likelihood of causality of 0.35, we used a mid value 

(50th percentile) of zero for beta, where BenMAP uses the mean value. Hence, in this 

case, the Analytica model produces zero mortality for mid (nominal) analysis, where 

BenMAP uses a small positive value for beta. 

This calibration process substantially improves the correspondence of the results of the 

sensitivity analysis with what would be obtained were the sensitivity analysis to be carried out 

with full runs of BenMAP. The mid (central values) are identical (with the above exceptions). 

The ranges used in the range sensitivity analysis, as perturbations around this mid value, should 

be similar but not identical to what one would obtain if one performed these range sensitivity 

runs with BenMAP. Thus, the slight approximation remaining for the range sensitivity results 

will have negligible on the main conclusions of this study—to identify the relative importance of 

the various sources of uncertainty. 

2.7 Analysis Strategy 

We use two measures of sensitivity in comparing the effects of each uncertain input X on 

the output variable Y: 

Elasticity: The percent change in Y for a one percent change in X. 

Output range: The lower and upper values of the output corresponding to lower and 

upper value of input range, holding all other inputs at their nominal values. The standard 

tornado chart displays output ranges. 

These measures of sensitivity focus on one uncertain input at a time, holding the others at 

their mid values. This makes the analysis fast and easy, but it ignores any possible interactions in 

the effects of two or more inputs on the outputs. We can rule out many interactions—or at least 

conclude that they are multiplicative (rather than additive)—just by inspecting the form of the 

equations, without having to run the model. However, there are some interactions that may be 

worth examining using the model, for example between the threshold and beta of CR functions. 

We will also examine one or two interactions between parameter uncertainties and functional 

form uncertainties, notably in the functional form of the CR functions. 
3-1 

SECTION 3 

CLASSIFY UNCERTAINTIES AND QUANTIFY PARAMETER RANGES 

3.1 Final List of Uncertainties 

The first step in the process involved identifying a set of uncertain parameters and model 

assumptions. Table 3-1 lists the sources of uncertainty identified for potential inclusion in initial 

sensitivity analysis. In the first column, the potential source of uncertainty is categorized as 

either a BenMAP choice (selected by user from choices in BenMAP) or as a BenMAP default 

(defined in BenMAP, the user can sometimes import different data to change assumption). The 

next two columns identify the source and type of uncertainty using the categories defined in 

Section 2.2. 

Table 3-1. List of Major Sources of Uncertainty in the BenMAP Model 

Modeling Step 

Uncertainty 

Source 

Uncertainty 

Type Current Formulation in BenMAP 

CR FUNCTION AND ISSUES IN THE ESTIMATION OF ADVERSE HEALTH EFFECTS 

1 

BenMAP choice 

Functional form 

of the CR 

function 

Model 

Uncertainty, 

Functional 

form 

 Alternative functional forms are sometimes available 

to represent CR relationship including linear, loglinear, logistic, and Cox proportional hazard models 

2 

BenMAP choice 

Threshold in the 

CR function 

Parameter 

Uncertainty, 

Measured 

 EPA estimates a point estimate threshold below which 

there is no change in incidence rate associated with 

change in air pollution concentration, e.g., for a loglinear CR model: 

 Δy=y0×[exp(β×(max(PM1,T) − max(PM0,T)))-1] 

where, T is the threshold 

3 

BenMAP 

default 

Air quality 

metric in 

BenMAP CR 

functions 

Parameter 

Uncertainty, 

Measured 

 Air quality metrics are matched to the epi studies. For 

pre-loaded functions, these are already defined in 

BenMAP. 

 For PM2.5, EPA currently uses CR functions that use a 

quarterly average air quality concentration. BenMAP 

creates an annual average by averaging the 4 quarterly 

averages 

4 

BenMAP 

default 

Extrapolation 

below PM 

concentration in 

study 

Parameter 

Uncertainty, 

Extrapolate 

beyond range 

 BenMAP does not require that PM concentrations be 

within the range of PM concentrations used in the epi 

studies 

5 

BenMAP 

default 

Current exposure 

and cumulative 

exposure 

Parameter 

Uncertainty, 

Measured 

 BenMAP assumes a pulse of exposure to pollutants in 

the analysis year. The CR functions may apply to 

short-term exposure or long-term exposure. For epibased mortality studies, EPA uses the long-term CR 

functions. 

(continued) 
3-2 

Table 3-1. List of Major Sources of Uncertainty in the BenMAP Model (continued) 

Modeling Step 

Uncertainty 

Source 

Uncertainty 

Type Current Formulation in BenMAP 

CR FUNCTION AND ISSUES IN THE ESTIMATION OF ADVERSE HEALTH EFFECTS 

6 

BenMAP default 

Equal toxicity of 

PM components 

Parameter 

Uncertainty 

 The functions in BenMAP assume equal toxicity for 

all PM2.5 species. Assuming differential toxicity would 

require defining a new pollutant in BenMAP and 

obtaining air quality modeling or monitoring data for 

that species. 

7 

BenMAP choice 

Mortality—

Coefficient (β) in 

the CR function 

Parameter 

Uncertainty, 

Measured 

 EPA currently uses two epi studies for the relationship 

between exposure to PM and premature mortality over 

time (i.e., Pope et al., 2002; Laden et al., 2006). 

 EPA also uses CR functions generated through an 

expert elicitation. Other studies are available. 

8 

BenMAP choice 

CB—Coefficient 

(β) in the CR 

function 

Parameter 

Uncertainty, 

Measured 

 EPA currently uses two studies for the relationship 

between exposure to PM and developing CB over time 

(i.e., Abbey et al., 1993) 

 Parameter of the CR function (b) was specific using a 

mean value and a standard error which can be used to 

form a normal distribution 

9 

BenMAP choice 

AMI—

Coefficient (β) in 

the CR function 

Parameter 

Uncertainty, 

Measured 

 EPA uses one study for the relationship between 

exposure to PM and AMI (i.e., Peters et al., 2001) 

 Parameter of the CR function (b) was specific using a 

mean value and a standard error which can be used to 

form a normal distribution 

10 

BenMAP 

default 

AMI—Parameter 

in the CR 

function 

Parameter 

Uncertainty, 

Measured 

 Only considers AMI’s for individuals who do not die 

within 28 days. Because 6% of male and 8% of female 

hospitalized heart attack patients die within 28 days, 

the function includes a correction factor of 0.93 (on 

average) to estimate the number of non-fatal AMIs 

(Rosamond et al., 1999) 

BASELINE INCIDENCE RATES AND PREVALENCE 

1 

BenMAP default 

Allocating 

baseline 

incidence to grid 

cells 

Model 

Uncertainty, 

Aggregation 

 Baseline incidence rates allocated to grid cells in 

BenMAP, not clear how 

2 

BenMAP default 

Mortality 

Baseline 

Incidence Rates 

Model 

Uncertainty, 

Functional 

form and 

aggregation 

 Age, cause, and county-specific mortality rates were 

obtained from CDC for years 1996 through 1998 by 

county 

 CDC’s age groups include <1, 1–4, 5–9, 10–14, 15–

19, 20–24, 25–34, 35–44, 45–54, 55–64, 65–74, 75–

84, and 85+ 

 Mortality rates are averaged across three years, 

assumes that rates are uniformly distributed across all 

ages in the reported age group 

(continued) 
3-3 

Table 3-1. List of Major Sources of Uncertainty in the BenMAP Model (continued) 

Modeling Step 

Uncertainty 

Source 

Uncertainty 

Type Current Formulation in BenMAP 

2 

BenMAP default 

Mortality 

Baseline 

Incidence Rates 

Model 

Uncertainty, 

Functional 

form and 

aggregation 

 Age, cause, and county-specific mortality rates were 

obtained from CDC for years 1996 through 1998 by 

county 

 CDC’s age groups include <1, 1–4, 5–9, 10–14, 15–

19, 20–24, 25–34, 35–44, 45–54, 55–64, 65–74, 75–

84, and 85+ 

 Mortality rates are averaged across three years, 

assumes that rates are uniformly distributed across all 

ages in the reported age group 

3 

BenMAP default 

Mortality—Year 

of baseline 

incidence 

Parameter 

uncertainty, 

Measured 

 Age, cause, and county-specific mortality rates were 

obtained from CDC for years 1996 through 1998 by 

county 

 Mortality rates are averaged across three years 

4 

BenMAP choice 

Mortality—

Forecast baseline 

incidence rates 

Parameter 

Uncertainty, 

Forecast 

 Project CDC Wonder county and age-specific 

mortality rates for 1997 to future years using Census 

projections of national age-specific mortality rates 

from 1999 through 2050. Adjust county and agespecific 1997 CDC Wonder rate by ratio of Census 

projection for year Y to the estimated Census agespecific mortality rate for 1997 (1997 Census mortality 

rate estimated by regressing mortality rates from 

1999–2008 on year). 

 All segments of the population assumed to grow at 

same rate 

 User chooses the forecast year 

5 

BenMAP default 

AMI Baseline 

Incidence Rates 

Model 

uncertainty, 

Parameter 

uncertainty 

 AMI incidence rates are based on regional 

hospitalization rates and hence inherit uncertainties 

associated with those rates 

 Hospitalization rates in 1999 were estimated for four 

regions based for 3 age groups (0-18, 18-46, 65+) 

using National Hospital Discharge Survey and 

population data. 

6 

BenMAP default 

CB Baseline 

Incidence Rates 

Parameter 

uncertainty 

 For ages 27+, the incidence rate is 0.00378 (Abbey et 

al., 1993) 

 No other age group or geographic breakdown is 

available 

7 

BenMAP default 

CB Prevalence Parameter 

uncertainty 

 Age group specific prevalence estimates are 18-44 

(0.0367), 45-64 (0.0505), 65+ (0.0587). For ages 18+, 

the prevalence is 0.0443 

8 

BenMAP default 

Morbidity—

Forecast baseline 

incidence and 

prevalence rates 

Model 

uncertainty, 

Forecast 

 BenMAP contains year 2000 only, no projections 

currently available 
3-4 

In December 2008, EPA sponsored a workshop to provide expert feedback on the 

uncertainties used in the analysis, additional sources of uncertainty, the reasons for uncertainty 

and variability in parameter values, and to provide advice on actual ranges of values or on 

sources of data to derive ranges. The workshop participants included both EPA staff and outside 

experts in the areas of population forecasting, mortality valuation, morbidity valuation and 

uncertainty analysis. After the workshop, additional discussions took place with EPA health 

scientists and air quality modelers. Appendix A provides more details on the workshop. 

3.2 Quantify Ranges for Parameters 

Following the workshop, we selected a set of uncertain parameters and model 

assumptions from the list in Table 3-1 for inclusion in the sensitivity analysis. For each 

parameter, we defined a range of values. Table 3-2 presents the range for each uncertain 

parameter. We selected ranges that represent plausible upper and lower bounds, but not 

necessarily extreme values. In addition, we selected mid values. The mid values do not 

necessarily represent the “middle” of the distribution, especially since we usually lack the 

information needed to determine what the distribution looks like. In most cases, the mid value 

will be consistent with the value from BenMAP for runs provided by EPA, which is provided in 

the last column. 

3.3 Source for Range Recommendations 

1. VSL: 

– Uncertainty in VSL estimates (calculated from an individual’s WTP to reduce the 

risk of mortality by a given percent) are associated with the stated and revealed 

preference methods used to calculate VSL, the data, and the assumptions used to 

make the calculation. VSL is also believed to be characterized by variability due 

to preferences, baseline age and health, and other personal and social 

characteristics. 

– The range from $1,000,000 to $10,000,000 with a midpoint of $5,500,000 covers 

the range from most of the current studies and meta-analyses and was 

recommended by the experts who participated in the workshop. The midpoint 

represents the VSL that EPA Office of Air and Radiation used for benefits 

analysis between 2004 and 2009. 

2. Elasticity of WTP with respect to income: 

– Uncertainty in the income elasticity of WTP come from the methods and data 

used to estimate the values. In addition, elasticity estimates usually come from 

cross-sectional data, but they are being used to represent change in elasticity over 

time. 

– The range comes from the BenMAP manual and was supported by the experts 

who participated in the workshop. 
3-5 

Table 3-2. Range Values for Uncertain Parameters 

Low Mid High 

Value from 

BenMAP 

Run 

1. VSL $1,000,000 $5,500,000 $10,000,000 $5,500,000 

2. Elasticity of WTP to income for mortality 0.2 0.40 1 0.40 

3. Discount rate 1% 3% 7% 3%, 7% 

4. Value of an avoided AMI, 3% discount 

rate 

$11,000 $85,309 $372,000 Varies by 

age, years of 

treatment 

5. Value of an avoided case of CB, 3% 

discount on COI value 

$11,000 $340,481 $1,749,539 $340,481 

6. Uncertainty in Census 2000, percent of 

baseline from BenMAP run 

98% 100% 102% 100% 

7. Uncertainty in Census 2020 population 

forecast 

90% 100% 110% 100% 

8. Threshold in mortality CR function 0 0 10ug/m3

 BenMAP 

runs used 0 

and 10ug/m3

9. Beta for mortality CR function with a 

threshold of 10ug/m3 Low Mid High Different for 

each study 

and expert 

10. Beta for mortality CR function with no 

threshold 

Low Mid High Different for 

each study 

and expert 

11. Beta for CB CR function with no 

threshold 

0.004993999 0.0137 0.02240599 0.0137 

12. Beta for AMI CR function with a 

threshold of 10ug/m3 0.02130709 0.033201 0.04509491 0.033201 

13. Beta for AMI CR function with no 

threshold 0.012227397 0.024121 0.036015217 0.024121307 

14. Base mortality rate, percent of value in 

BenMAP 

95% 100% 105% 100% 

15. Incidence rate for CB 0.067872271 0.378 0.798218895 0.378 

16. Incidence rate for AMI 0.1391 0.7747 1.6359 

17. GDP growth rate 1.72% 2.34% 3.04% 

18. Base PM2.5 concentration, percent 

difference from BenMAP run 

85% 100% 115% 100% 

19. Change in PM2.5concentration, percent 

difference from BenMAP run 

70% 100% 130% 100% 
3-6 

3. Discount rate: 3% and 7% are required for use in Federal regulatory analysis. The 

lower end of the range, 1%, was suggested because of the long time frame for the 

model forecasts. 

4. Value of an avoided AMI, 3% discount rate: 

– Uncertainty in the distribution of values for an avoided AMI is associated with 

factors such as the lack of WTP values (BenMAP includes only COI values), the 

age of the studies, the difficulty defining the costs associated with a single AMI 

separate from other conditions, the impact of co-morbidities on cost, the cost of 

potential complications after the AMI, and similar issues. 

– The expert panel recommended a lower bound of one-half the COI and an upper 

bound of WTP plus COI. BenMAP currently uses only COI estimates for AMI’s. 

The COI measures include estimates of direct medical costs and opportunity cost 

(lost earnings). In the absence of readily available estimates of WTP, we used the 

range from BenMAP studies for 5 year and 10 year length of costs discounted at 

3% (Table H-5, USEPA, 2008). 

i. Low: one-half the lowest COI in Table H-5 

ii. Mid: the average COI in Table H-5 across all ages (not including 0-24 age 

group) 

iii. High: because there is not WTP value, the high end of the range was set at 

two times the highest COI in Table H-5 

5. Value of an avoided case of CB, 3% discount rate: 

– The uncertainty associated with the value of an avoided case of CB comes from 

factors such as the age and sample of the existing WTP studies and other factors 

similar to AMI’s. 

– The range of values was derived from the COI estimates from Table H-3 in the 

BenMAP User’s Guide and the WTP formula in Section H.2.1.1 (USEPA, 2008). 

i. Low: the lowest COI value in Table H-3 (which is similar in magnitude to 

one-half the lowest COI for an AMI)

ii. Mid: the median value of the WTP function in BenMAP

iii. High: WTP value plus the highest COI value from Table H-3 (USEPA, 

2008) for the 3% discount rate. WTP was calculated as follows:

1. The formula for WTP in the BenMAP manual is WTP(x) = 

WTP(13)*exp(-beta(13-x)) (H.2.1.1 USEPA, 2008) 

2. x denotes severity and ranges from 1 to 12, where 1 is the least severe; 

beta is assumed to ~ N(0.18,0.0669). The WTP(13), the highest severity, 

come from Viscusi et al. (1991); estimates of the beta coefficient and 

standard error come from Krupnick and Cropper (1992). 
3-7 

3. Using beta=0.18-0.0669 and x=11 and the WTP(13) for the 90th

percentile, the formula above is used to calculate the WTP(x) for the 

high value. The value of WTP thus obtained is $1,595,117. 

6. Census estimate for 2000: 

– Uncertainty about national population estimate in 2000 

– A range of minus and plus 2% was specified based data from Census and 

population forecasts for future years. Population projections based on the 1990 

Census include low, middle and high series estimates (U.S. Department of 

Commerce 1996). The difference between the high and low series in the 

projections for 1995 is less than .0001%. Comparing the 10-year projection of 

2000 population to the population counted by the 2000 Census (U.S. Department 

of Commerce 2002), the difference between the actual 2000 Census and the 

middle series forecast for 2000 is about 2.4%. The calculations suggest that the 

uncertainty in the population estimate for 2000 is low and within a range of 2% of 

the actual estimate. 

7. Population forecast for 2020 and distribution of population within the U.S.: 

– Uncertainty in the population forecast for 2020 comes from variety of sources 

including immigration trends, economic growth, the birth rate and the death rate. 

The share of the U.S. population in a particular county will depend on similar 

factors. 

– BenMAP uses the 2000 Census population estimate at the block group level that 

is aggregated and allocated to grid squares that match the air pollution data. The 

population forecast is based on county forecast data from Woods and Poole 

(2007) and then allocated to grid squares that match the air pollution data in 

BenMAP. The Woods and Poole forecast data uses a model that includes 

economic growth projections. The share of national population in each county 

based on the Woods and Poole data is used to allocate the 2000 Census estimate 

across counties. Confidence intervals reported by Woods and Poole are +/- 6.3% 

for metropolitan statistical areas and +/- 4.7% for states based on comparing 10-

year forecasts with actual values. Uncertainty about whether people will move to 

areas of the country that are more or less polluted will potentially affect the 

benefit estimates. 

– To account for both the uncertainty in the 2020 forecast and the uncertainty in 

how the population will be spread out across counties, the 2020 population by 

grid square from BenMAP was adjusted +/- 10%. 

8. Threshold in mortality CR function: 

– Guided by the expert elicitation and advice from EPA staff, the threshold for the 

PM/mortality CR function ranges from 0 (no threshold) to 10ug/m3

. 

9. Beta for mortality CR function with a threshold of 10ug/m3

: 

– See Section 5.4 for details on calculating mid, low, and high values for beta for 

the different studies and experts. 
3-8 

10. Beta for mortality CR function with no threshold: 

– See Section 5.4 for details on calculating mid, low, and high values for beta for 

the different studies and experts. 

11. Beta for CB CR function with no threshold: 

– Uncertainty in the beta arises from a lack of studies, possible regional variation, 

and other sources. 

– See Section 5.4 for details on calculating mid, low, and high values for beta for 

the different studies and experts. 

– Beta based on Abbey et al. (1993) using the mean of 0.0137 and a standard 

deviation of 0.00679645, the low and high are the 10th and 90th percentiles of the 

distribution (1.281 standard deviations from the man) and the mid value is the 

mean. 

12. Beta for AMI CR function with a threshold of 10ug/m3

: 

– Uncertainty in the beta arises from a lack of studies, possible regional variation, 

and other sources. 

– Beta based on Peters et al. (2001). Using the mean of 0.033201 and a standard 

deviation of 0.009284863, the low and high are the 10th and 90th percentiles of the 

distribution (1.281 standard deviations from the man) and the mid value is the 

mean. 

13. Beta for AMI CR function with no threshold: 

– Uncertainty in the beta arises from a lack of studies, possible regional variation, 

and other sources. 

– Beta based on Peters et al. (2001). Using the mean of 0.024121307 and a standard 

deviation of 0.009284863, the low and high are the 10th and 90th percentiles of the 

distribution (1.281 standard deviations from the man) and the mid value is the 

mean. 

14. Base mortality rate: 

– Uncertainty arises from possible errors in data collection and compilation, 

estimating rates for specific population sub-samples and averaging rates across 

sub-samples. 

– Assumed +/- 5% of base rate for mortality 

15. Incidence rate for CB: 

– Uncertainty arises from lack of studies that estimate incidence rates. 

– BenMAP cites Abbey et al. (1993) and Abbey et al. (1995) as the source for the 

incidence rate values discussed in E.6.2 of the BenMAP manual (USEPA, 2008). 

We calculated the percentage change from low to mid to high using the range of 

AMI incidence rates and applied the percent difference to the mid estimate for 

CB. 

i. Low: 82% of the mid value
3-9 

ii. Mid: consistent with the value from BenMAP run

iii. High: 111% of the mid value

16. Incidence rate for AMI 

– Uncertainty arises from lack of studies that estimate incidence rates. 

– Values come from Exhibit E-6 (which cites data from the 1999 National Hospital 

Discharge Survey and Rosamond et al. [1999]) in BenMAP manual (USEPA, 

2008), we used highest and lowest values across regions and age groups 

(excluding 0-18) 

17. GDP growth rate 

– Used forecasts from Table 20 (Macroeconomic Indicators) of the Annual Energy 

Outlook 2009 (DOE, 2009). The mid, low and high values were calculated based 

on the forecasts from the reference case, low and high economic growth cases, 

respectively. The compound annual average growth rate (CAGR) in GDP (real 

dollars) was computed over the period 2008 to 2020 for these three scenarios. 

These GDP numbers were used to adjust real income growth from the income 

growth projections in BenMAP up or down for the two ranges. 

18. Base PM2.5 concentration 

– Emissions and air quality forecasts were not part of the scope of this project. We 

assumed +/- 15% of base concentrations from BenMAP run to provide 

information on how uncertainty in the base concentrations might affect the range 

of outcomes.

19. Change in PM2.5 concentration 

– Emissions and air quality forecasts were not part of the scope of this project. We 

assumed +/- 30% of base concentrations from BenMAP run to provide 

information on how uncertainty in the base concentrations might affect the range 

of outcomes.

3.4 Alternative Functional Forms 

We used alternative functional form assumptions for: 

 Lag structure for mortality valuation 

 CR functions 
4-1 

SECTION 4 

INFLUENCE DIAGRAMS 

4.1 Introduction to Influence Diagrams 

Influence diagrams trace how different sources of uncertainty affect different elements of 

the model and the final results. Figure 4-1 provides an illustrative influence diagram of a benefitcost analysis with the variable node types used in Analytica influence diagrams labeled. (This 

example is purely to illustrate the notation, and not used in this analysis.) In this influence 

analysis, decisions (green rectangles) identify user options, such as choosing the base year for 

dollars or forecast year. Uncertain parameters (light blue ovals) identify variables on which we 

perform sensitivity analysis. The model is organized as a hierarchy of modules, each with its 

own diagram. Some variables affect variables in other modules. A node in italics (like Year in 

the submodule), is an alias, that is, a copy of a node whose original appears in another diagram, 

shown to clarify dependency relations. 

Figure 4-1. Illustrative Influence Diagram with Notation Defined 
4-2 

4.2 Influence Diagrams for Sensitivity Analysis 

The top influence diagram, Figure 4-2, shows the key components in BenMAP as 

represented in Analytica. Each node with a thick outline represents a module containing an 

influence diagram shown in more detail below. The influence arrows show how one module or 

node affects another, and ultimately affects the final result, “Monetized benefits,” shown as a red 

hexagon representing the final objective variable. 

Figure 4-2. Top-Level Influence Diagram 

The influence diagram for the PM2.5 concentration, Figure 4-3, generates the base and 

delta concentration (change in concentration) in PM2.5 for each concentration bin, using the base 

and delta for each bin computed from BenMAP inputs for each grid square, multiplied by an 

uncertain parameter with mid value 100% (the light blue oval nodes) to include uncertainty in 

these projections. 

Figure 4-4 shows how the incidence rates are calculated. It is the submodule for 

“Incidence rates,” the first module in the top row of Figure 4-2. This figure includes the key 

variables determining mortality by bin—that is, the mortality rate for each level of base 

concentration of PM2.5. It also includes the incidence rate for morbidity endpoints CB and AMI. 

Figure 4-5 shows the influence diagram for the CR functions. The green node “CR 

function data from epidemiological studies” contains the data defining those CR functions. We 

extract the threshold and beta (along with uncertainty expressed as standard deviations) from 

Table 5-1. This is combined with thresholds and betas from the expert elicitation distributions, 
4-3 

Base PM25

concentration

Delta PM25

concentration

Base PM25

conc % unc

Delta PM25

conc % unc

Base PM25

concentration

by bin

Delta

concentration

Figure 4-3. Influence Diagram for PM2.5 Concentration 

Figure 4-4. Influence Diagrams for Incidence Rates 
4-4 

Figure 4-5. Influence Diagram for Health Impacts CR Functions 

and low, mid and high betas for the CR function (the three ovals representing the three uncertain 

parameters used for sensitivity analysis: beta’s for the mortality, CB, and AMI CR functions). 

Together the threshold and betas for all the CR functions feed into the incidence reduction node, 

also shown on the main Influence Diagram, Figure 4-2.

Figure 4-6 shows the influence diagram for the expert elicitation distributions, which feed 

into the “Incidence reduction” module in Figure 4-2. The EPA conducted an expert elicitation on 

the topic of the CR function for PM2.5 and mortality (IEc, 2006). The data elicited from each 

expert was incorporated individually into the analysis. The turquoise node “Expert CR 

functions” contains the parameters defining the probability distributions obtained from experts 

for mortality CR functions for PM2.5. The remaining nodes process these to generate probability 

distributions, to extract mid beta and threshold, combine segments for piecewise functions and 

compute low, mid, and high values for range sensitivity analysis as described below. 

Figure 4-7 contains the influence diagram for the “Population” module in Figure 4-2. 

This module show how U.S. population in the forecast year (2020) is projected from population 

from the U.S. Census in 2000 and the projected increase to 2020. For this simplified analysis, it 

assumes the same percent growth in population for each concentration bin. 
4-5 

Figure 4-6. Influence Diagram for Expert Elicitation Distributions 

Select forecast

year

Population in

forecast year

by bins

Population in

2000

Total US

population in

forecast year

Population in

2020

Census

years

US pop by

census

2010-50

Pop CAGR to

forecast year Population

change factor

from 2020

Population by

conc bin

Figure 4-7. Influence Diagram for Population Projections 

The influence diagram of valuation modules is presented in Figure 4-8, which feeds into 

the “Valuation functions” module in Figure 4-2. The diagram includes the key variables and 

modules used to compute the VSL, the value of avoiding a case of CB, and the value of avoiding 

an AMI for the forecast year adjusted for lags as necessary. The node “Value per case avoided by 

end point” combines the valuation of avoided mortality, CB and AMI. 
4-6 

Discount rate

Value of a

statistical life

saved

VSL for

forecast

year

Elasticity of

WTP to

income

GDP growth

rate

Income

growth

factor

Discount

due to

mortality lag

VSL

adjusted for

lag

Select

forecast year

Valuation

of

morbidity

Inflation and

income

growth

Inflation factor

to base year

Mortality lag

Value per

case avoided

by end point

Calibrated

value per

case

Figure 4-8. Influence Diagram for Valuation Modules 

The diagram in Figure 4-8 includes three sub-modules for inflation and income growth, 

valuation of morbidity, and the mortality valuation lag. Figure 4-9 is the influence diagram for 

inflation and income growth, which calculates income growth to forecast year for mortality and 

CB, and the inflation factor to base currency year from standard EPA tables of income growth 

and inflation index. The income growth factor applies to the WTP functions for mortality and 

CB, but not to COI functions. It provides two income growth methods, one the same as BenMAP 

and the other on projections based on the ratio of GDP to population growth that allows us to use 

corresponding projections from the Annual Energy Outlook. The numbers below assume the 

BenMAP method, unless mentioned otherwise. 

Figure 4-10 is the influence diagram for computing the value per case avoided of CB and 

AMI based on income effect on morbidity (if used) and an inflation factor to base year. 
4-7 

Currency

year

EPA inflation

index from 2000

EPA

inflation

years

EPA standard

income growth

by endpoint

group

EPA

income

years

EPA

income end

point group

Inflation factor

to base year

Income growth

factor 2000 to

baseline year

Income

growth for

mortality

EPA Income

growth 2000 to

forecast year

Income

growth for

CB

Select forecast

year

Select income

growth method

Model type

Figure 4-9. Influence Diagram for Inflation and Income Growth 

Figure 4-10. Influence Diagram for Value of Avoided Morbidity Endpoint 
4-8 

Figure 4-11, the influence diagram for mortality valuation lags, shows how the model 

calculates the effect of different assumptions about the mortality valuation lag structure. The 

mortality valuation lag structure defines the distribution of mortality across time based on 

different assumptions about the delay between exposure and its effect on mortality. The discount 

rate over time means that future mortality valuation is discounted when computing the present 

value at the time of exposure (or reduction in exposure due to improved air quality). 

Lag

structures

Mortality

percentage

by lag year

Lag year

Exponent

for lag

Exponential

decline by

lag year

Select lag

structure

Discount for

exponential

Discount rate

Discount

due to

mortality lag

Figure 4-11. Influence Diagram for Mortality Valuation Lag 

The different lag structures are presented in Figure 4-12. This chart shows the percentage 

of mortality that occurs in each of the 20 years following exposure for six lag structures. None or 

no lag means all mortality occurs in the year of exposure. 15-year means all deaths occur exactly 

15 years after exposure. 20-year distributed means 20% of deaths occur in the year of exposure, 

50% evenly distributed over years two to five, and 30% evenly distributed over the remaining 15 

years. 

BenMAP takes as an input the estimates or measurements of the current air quality and 

projections of improved air quality for a set of points or grid squares over the area of interest, in 

this case the contiguous United States. The air quality estimates and projections are produced by 

a series of models that project emissions, atmospheric transport, and air quality, which are 

depicted in the influence diagram in Figure 4-13. Most commonly, CMAQ provides the air 

quality projections to BenMAP. We provide this parent diagram, Figure 4-13, to show the 

context of the analysis. 
4-9 

Figure 4-12. Mortality Lag 

Figure 4-13. Influence Diagram for Air Quality 
4-10 

Analysis of the uncertainty in the air quality projections and the sources of uncertainty in 

those projections are outside the scope of this project. Accordingly, we selected an arbitrary 

range of uncertainties on the air quality—PM2.5 concentrations—and improvements in air 

quality—delta reductions in PM2.5 concentrations. 
5-1 

SECTION 5 

RESULTS 

5.1 Assumptions for Analysis 

The sensitivity analysis results in this report are generated from the Analytica simplified 

version of the BenMAP model. They are calibrated to a specific run of BenMAP, so that the 

overall mortality and morbidity cases avoided, and the monetary benefits of those cases avoided 

exactly match the BenMAP run. See Section 5.4 for assumptions on the low, mid, and high 

values for each CR functions. Section 3 describes how the ranges for the other parameters were 

selected. 

For simplicity, the primary results assume no mortality valuation lag—i.e., any mortality 

reduction is assumed to occur in the same year as the change in PM2.5 concentration—except 

where we explicitly present the effects of other mortality valuation lag structures. The effects of 

discounting and mortality valuation lag change the benefits of mortality reduction by the given 

mortality valuation lag factor (shown in Figure 5-17). Accordingly, changing mortality valuation 

lag would not affect the relative width of range sensitivities that affect mortality. 

5.2 Measures of Sensitivity Used in Analysis 

We use two measures of sensitivity to compare the effects of each uncertain input X on 

the output variable Y: 

 Elasticity: The percentage change in Y for a one percent change in X. 

 Range sensitivity: The lower and upper values of the output corresponding to lower 

and upper value of input range, holding all other inputs at their nominal values. The 

standard tornado chart displays output ranges. 

Most of our sensitivity results are based on range sensitivity, since, as its name implies, it 

is sensitive to the range of uncertainty (low to high) for each uncertain variable. However, 

elasticity does provide some interesting insights—it depends only on the mathematical structure 

of the model and the mid values of the inputs, but ignores their range of uncertainty. For a model 

y = F(x), 

the elasticity of y with respect to x is 

Elasticity(y, x) = [F(x + dx) / (F(x)] / [(x + dx)/dx] 
5-2 

To provide some insight into this measure, consider these two examples: 

Elasticity additive example 

Total_benefits = Mortality_benefits + Morbidity_benefits 

With values: $100M = $90M + $10M 

Elasticity(Total_benefits, Mortality_benefits ) = 90% 

Elasticity(Total_benefits, Morbidity_benefits) = 10% 

Thus, for a simple additive model, the elasticity of the result is equal to the value of each 

variable added as a percentage of the total. 

Elasticity multiplicative example 

This multiplicative example is a stylized model of the computation of mortality benefits, 

assuming a linear, no-threshold mortality function, with change in concentration (Delta_conc), 

slope (Beta_CR), and value of a statistical life (VSL):1

 Mortality_benefits = Population × Mortality_rate × Delta_conc × Beta_CR × VSL 

Elasticity(Mortality_benefits, Population) = 100% 

Elasticity(Mortality_benefits, Mortality_rate ) = 100% 

Elasticity(Mortality_benefits, Delta_conc ) = 100% 

Elasticity(Mortality_benefits, Beta_CR ) = 100% 

Elasticity(Mortality_benefits, VSL) = 100% 

The interesting point here is that for a purely multiplicative model, the elasticity of the 

product to each variable is 100%. 

5.3 Elasticity Results 

Figure 5-1 shows the elasticity of mortality to uncertain inputs, for the Pope et al. zero 

threshold CR function. The elasticity values were computed using the Analytica function 

Elasticity(Y, X). The function changes the value of X from x1 to x2 by multiplying it by 

 

1

 BenMAP and the Analytica model actually use a log-linear mortality CR-function, not this simplified 

multiplicative form. 
5-3 

Figure 5-1. Elasticity of Mortality Parameters 

(1 + 10−8) and computes the resulting value of Y, changing from y1 to y2. It returns the ratio 

(y2 – y1)/(x2 – y1). Thus, it computes elasticity based on the mid values, ignoring the range on 

the parameters. The elasticity is 100% with respect to Delta PM2.5 (reduction in PM 

concentration), population in 2000, population in 2020, beta for the mortality CR function, base 

mortality rate, and value of a statistical life suggests that the model is almost exactly purely 

multiplicative for these variables. (Note that the population in 2020 is actually a multiplier for 

growth from 2000 to 2020.) It is interesting also to see that the elasticity is zero with respect to 

the base PM2.5 concentration and threshold. 

Figure 5-2 shows the elasticity of total monetized benefits (including mortality and 

morbidity reduction) to each uncertain parameter, using the exponential declining lag structure. 

Again, elasticity with respect to Delta PM2.5, population in 2000 and 2020 is still 100%, 

indicating that total benefits are directly proportional to these values. The elasticity to Beta on 

mortality function, base mortality, and value of a statistical life are 93%. These parameters are 

multiplicative factors for the benefits of mortality reduction, which is about 93% of the total 

benefits. The elasticity of the parameters for morbidity—betas, incidence rates, and WTP to 

avoid—are each about 2% for AMI and 5% for CB, adding up to 7% to account for the 

remaining total benefit. The GDP growth rate affects the income and hence VSL via the income 

elasticity. The discount rate has a negative elasticity because a higher discount rate reduces the 

benefits of reducing mortality given the exponential declining lag structure. 
5-4 

Figure 5-2. Elasticity of Total Monetized Benefits to Each Uncertain Parameter 

5.4 Results for Different Concentration-Response Functions for Mortality and 

Morbidity 

CR functions give the response to changes in mortality or morbidity as a function of 

changes in the atmospheric concentration of air pollution. In this project, we focus on particulate 

matter finer than 2.5 microns in diameter (PM2.5). BenMAP uses a wide variety of CR functions. 

All use a log-linear function, some with and some without a threshold concentration below 

which PM2.5 is assumed to have no health effect. For some cases, the choice of a threshold 

affects the beta slope, because the regression requires a higher slope to account for observed 

mortality when assuming a threshold below which there are no effects. These studies include 

several based on epidemiological studies, notably Pope et al. (2002) and Laden et al. (2006) for 

mortality CR functions, Abbey et al. (1993) for CB and Peters et al. (2001) for AMI. They also 

include probability distributions assessed by twelve experts, who remain anonymous and are 

identified by letters A through L. In this section, we describe the low, mid, and high values for 

each CR function for use in the range sensitivity analysis. 

5.4.1 Ranges for Concentration-Response Functions from Published Epidemiological 

Studies for Mortality and Morbidity 

Table 5-1 provides information on the range values from the published studies, including 

the end point, minimum age of population affected, threshold in μg/m3

, mean and standard 

deviation (SD) for beta parameter of the log-linear function of the CR functions from the 

epidemiological studies. 
5-5 

Table 5-1. Ranges for Epidemiological Studies

Note: The suffix "m" after a number means it is multiplied by 10-3

We set the mid value for each beta equal to its mean. The low and high values are 

selected as the 10th percentile and 90th percentile of a normal distribution with specified mean 

and standard deviation—which are plus or minus 1.281 standard deviations from the mean. 

Figure 5-3 shows the range (low, mid, and high) values for the CR functions for mortality 

and morbidity from selected epidemiological studies. The low (left of red bar) and high (right 

edge of green bar) values correspond to the 10th and 90th percentiles of the normal distributions 

estimated for the beta values, used for range sensitivity analysis. 

5.4.2 Ranges from Expert Elicitations of Probability Distributions on CR Functions 

In 2006, EPA commissioned a study to estimate CR functions for PM2.5 from 12 experts. 

These estimates were elicited in the form of probability distributions (IEc, 2006). Table 5-2 from 

the Analytica model reproduces the value from Exhibit 3-9 “Summary of expert subjective 

uncertainty distributions for C-R coefficients” (page 3-30 IEc, 2006). Some experts (B, F, K, and 

L) assessed a piecewise log-linear function with two segments (pieces) and a different 

distribution on beta on each segment. The cutoff value is the PM2.5 concentration (in ug/m3

) 

where the segments change from one to the other distribution. Where the expert specified a 

custom distribution, we fitted a continuous distribution to the specified minimum, 5th, 25th, 50th, 

75th, 95th, and maximum points on each distribution using a piecewise cubic on the cumulative 

distribution (using Analytica’s CumDist() function). The normal distributions were fitted to the 

5th and 95th percentiles and the triangular fitted to min, median and max for distributions 

specified in those forms. 
5-6 

Pope et al 0

Pope et al 7.5

Pope et al 10

Pope et al 12

Pope et al 15

Laden 0

Laden 10

CB 0

CB 7.5

AMI 0

AMI 7.5

0.00 0.01 0.02 0.03 0.04 0.05

Ranges for CR betas

Low mid and high

Low High

Figure 5-3. Ranges of Beta for CR Functions 

In Table 5-2, “Prob causality” is the likelihood that each expert assigns to the hypothesis 

that human exposure to PM2.5 in ranges up to 30 μg/m3 causes significant mortality. “Includes 

causality” is “Y” or yes for experts that included this probability in their probability 

distributions. 

For the other experts that did not include causality in the distributions they assessed, we 

treated the complement of the probability of causality as a probability mass that beta is zero, 

which we combined with the full distribution in the simulation. We used a Latin Hypercube 

sampling with 10,000 sample points to represent the combined distributions. We used these 

samples to estimate the 10th, 50th (median), and 90th percentile from these distributions, 

combining the continuous distribution with a probability of causality when not included in the 

original distribution assessment. 

Only expert K expressed a belief that there might be a threshold, with a 50% chance that 

there is a threshold and 10% chance that it is greater than 5 μg/m3. Accordingly for the range 

sensitivity, we set the low and mid value to 0 (which is the 10th and 50th percentile of the 

threshold distribution) and the high value (90th percentile) to 5 μg/m3. 
5-7 

Table 5-2. Probability Distributions Elicited from Experts A through L Source: IEc, 2006. 
5-8 

Figure 5-4 shows the range (low, mid, high) values for the CR functions for mortality 

assessed by experts. For experts B, F, K, and L, there are two ranges since the experts assessed 

piecewise log-linear functions with two segments. The low (left of red bar) and high (right edge 

of green bar) values correspond to the 10th and 90th percentiles of the distributions assessed by 

each expert for the beta values, used for range sensitivity analysis. Expert K has a mid and low 

value of zero reflecting his or her likelihood of causality of only 0.35. 

Figure 5-4. Ranges of Beta for CR functions from Experts A to L 

Figure 5-5 provides a chart showing the range of cases of mortality or morbidity avoided 

for each CR function from the published epidemiological studies and from the expert elicitation. 

The chart includes epidemiological studies (Pope et al. and Laden et al.) and the twelve experts 

A to L. These ranges reflect the ranges in the beta of the CR functions shown previously. The 

numbers after Pope et al. and Laden et al. specify the threshold of PM2.5 in μg/m3. Low (left edge 

of red bar), mid (edge between red and green bar) and high (right edge of green bar) are the 10th, 

50th, and 90th percentiles of probability distribution. For Pope et al. and Laden et al., these are 

distributions from the reported standard deviation of the estimates of beta. For the Experts A to 

L, these distributions are those estimated by the expert, as described above. 
5-9 

Figure 5-5. Range of Avoided Premature Mortalities and Morbidities (CB and AMI) for 

Low, Mid, and High Values for Each CR Function 

Pope et al. 15 (with a threshold of 15 μg/m3) has zero mortality and so zero reduction in 

mortality. BenMAP input data did not contain grid squares with current PM2.5 concentration of 

greater than the threshold of 15 μg/m3. Expert K assessed the likelihood of causality as only 0.35. 

So, the 10th and 50th percentiles of his or her distributions on beta of CR function are zero. Only 

the high value, the 90th percentile (right edge of green bar) is nonzero 

Figure 5-6 shows the range of monetized benefits from reduced mortality or morbidity 

for each CR function. Each range (low, mid, and high value) corresponds with the low, mid, and 

high value for each CR function estimated as described above, holding all other parameters at 

their mid value. The mid values match the value from the BenMAP calibration run. 
5-10 

All CR functions

Figure 5-6. Range of Monetized Benefits for Low, Mid, and High Values for Each CR 

Function 

Finally, Table 5-3 provides the number of cases of mortality or morbidity avoided with 

the low, mid, and high value for the beta from the CR function for each epidemiological study or 

expert, where the number after the name of the study or expert specifies the threshold. 
5-11 

Table 5-3. Number of Cases of Mortality or Morbidity Avoided Using the Low, Mid, and 

High Values for the Beta from the CR Function in Each Source 

Source Low Mid High 

"Pope et al 0" 1,957 3,727 5,500

"Pope et al 7.5" 1,935 3,685 5,438

"Pope et al 10" 1,391 2,652 3,917 

"Pope et al 12" 973 1,856 2,743 

"Pope et al 15" 0 0 0 

"Laden 0" 6,124 9,521 12,930 

"Laden 10" 3,872 6,023 8,186 

"CB 0" 961 2,631 4,290 

"CB 7.5" 952 2,605 4,248 

"AMI 0" 4,461 8,768 13,040 

"AMI 7.5" 7,675 11,912 16,116 

"Expert A" 5,837 10,268 16,791 

"Expert B" 820 7,426 15,934 

"Expert C" 5,065 7,691 11,541 

"Expert D" 2,308 5,661 8,303 

"Expert E" 9,212 12,820 17,786 

"Expert F" 4,374 5,955 8,282 

"Expert G" 0 5,992 7,582 

"Expert H" 1,737 4,479 11,368 

"Expert I" 2,503 8,173 12,434 

"Expert J" 2,680 5,763 11,182 

"Expert K" 0 0 3,191 

"Expert L" 0 5,159 9,623 

5.5 Range Sensitivity of Mortality Avoided Outcome 

The following charts show the results of the range sensitivity analysis on avoided 

premature mortality. Each chart lists the uncertain parameters down the left axis. The width of 

each bar shows the effect of changing each parameter from its low value (left edge of red bar) to 

mid value and high value (right edge of blue bar). A wider bar means higher range sensitivity. 

The range sensitivity results in Figure 5-7 use the Pope et al. zero threshold as the CR 

function for mortality. Delta PM2.5 and beta have the largest range sensitivities. There is no 

sensitivity to changes in the base concentration or threshold, since this CR function assumes a 

fixed zero threshold. Figures 5-8a and 5-8b shows the range sensitivity for Pope et al. with a 10 

and 12 ug/m3

 thresholds, respectively. Because of the threshold, the uncertainty in the base PM2.5

concentration has a significant effect, larger even than the Delta PM2.5 in this case, because it 

shifts population in some grid squares below the threshold for effects. Compared to the zero 

threshold case, the numbers of avoided premature mortalities is much lower. 
5-12 

Figure 5-7. Range Sensitivity of Avoided Mortality for Pope et al. with Zero Threshold Uncertain parameters for mortality

Figure 5-8a. Range Sensitivity of Avoided Mortality for Pope et al. with a 10 ug/m3

Threshold 
5-13 

Figure 5-8b. Range Sensitivity of Avoided Mortality for Pope et al. with a 12 ug/m3

Threshold 

Figure 5-9 shows the range sensitivity of avoided mortality for Laden et al. with zero 

threshold. The sensitivity is similar to that for Pope et al. for zero threshold in terms of relative 

size of range sensitivities. Again there is no effect of Base PM2.5. 

Figure 5-9. Range Sensitivity of Avoided Mortality for Laden et al. with Zero Threshold 

Figure 5-10 shows the range sensitivity if we select a different Pope et al. CR function 

assuming a mid threshold of 7.5 μg/m3

. Thus, there is range sensitivity to threshold, with a high 

threshold (blue) of 10 μg/m3 leading to lower cases avoided, and a low threshold (red) 0 

increasing the cases avoided. 
5-14 

Uncertain parameters for 

mortality

Figure 5-10. Range Sensitivity of Avoided Mortality for Pope et al. with Mid Threshold of 

7.5 μg/m3

Range sensitivity of avoided mortality for Laden et al. with 10 ug/m3 threshold, 

Figure 5-11, is similar to that for Pope et al. for high thresholds. Range sensitivity of Base PM2.5 

is close to that for Beta. Uncertain parameters for mortality

Figure 5-11. Range Sensitivity of Avoided Mortality for Laden et al. with 10 ug/m3

Threshold 
5-15 

5.6 Range Sensitivity for Total Monetized Benefits Outcome 

Figure 5-12 shows the range sensitivity of total monetized benefits to the full range of 

uncertain parameters for Pope et al. with zero threshold as the mid value. The zero threshold 

means no range sensitivity to Base PM2.5 concentration. In this case, a higher threshold (based on 

the Pope et al. CR function adjusted for the higher threshold) leads to lower monetized benefits. 

The largest range sensitivity is the statistical value of a life, followed by Beta for mortality CR 

function, and then Delta PM2.5 concentration. All the other range sensitivities are significantly 

smaller, with the value of an avoided case of CB being the largest. The discount rate has no 

effect because we are assuming no mortality valuation lag. 

Figure 5-12. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Pope et al. with Zero Threshold as the Mid Value 

Figure 5-13, like the previous one, assumes the Pope et al. zero threshold CR function. 

The only difference is that we selected an exponential declining mortality valuation lag structure. 

This means that the results are slightly sensitive to the discount rate. 
5-16 

Figure 5-13. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Pope et al. with Zero Threshold as the Mid Value with an Exponential 

Declining Mortality Valuation lag Structure 

Figure 5-14 uses the Laden et al. zero threshold as the mid mortality function. It changes 

to the Laden et al. 10 threshold with high threshold. The relative sizes of the range sensitivities is 

identical to the previous chart using Pope et al. mortality function, but the absolute values are 

more than twice as large—$70 billion instead of $29 billion for mid values. 

The final two range sensitivity charts are for experts E and K, who assessed the largest 

and smallest, respectively, of the mid values for the beta of the mortality CR functions. The chart 

in Figure 5-15 shows range sensitivities using Expert E. There is no sensitivity to threshold or 

Base PM2.5, since Expert E has no threshold. The mid total benefits are $96 billion. Otherwise 

the pattern of the relative sizes of the range sensitivities is similar to Pope and Laden et al. CR 

functions. Figure 5-16 shows range sensitivities using Expert K. The mid value is only $2 billion 

per year, which is the benefits of reducing morbidity only, since Expert K’s mid value (median) 

for the CR beta is zero. The zero mortality reduction leads to zero range sensitivity for VSL. For 

the high value of beta, the total benefits exceed $26 billion. 
5-17 

Figure 5-14. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Laden et al. Zero Threshold as the Mid Value 

Figure 5-15. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Expert E 
5-18 

Figure 5-16. Range Sensitivity of Total Monetized Benefits to the Full Range of Uncertain 

Parameters for Expert K 

Finally, uncertainty about the valuation lag structure for mortality has an impact on the 

outcomes. Figure 5-17 presents the impact on the total outcome of different assumptions about 

the valuation lag structure and about the discount rate. Figure 5-18 shows the range sensitivity of 

total monetized benefits to different discount rate assumptions for each lag structure. This graph 

is for Pope et al. with no threshold. The high discount rate (7%) at left of blue bar gives lower 

benefits, and low discount rate (1%) at right of red bar gives higher benefits. 
5-19 

None

8-year

15-year

20-year distributed

5-year distributed

Exponential declining

0% 100% 20% 40% 60% 80%

Discount due to mortality lag

Discount rate (%)

1 3 7

Figure 5-17. Discount by Lag Structure 

Figure 5-18. Range Sensitivity of Total Monetized Benefits to Discount Rate Assumptions 

for Different Mortality Valuation lag Structures 
6-1 

SECTION 6 

DISCUSSION AND POSSIBLE NEXT STEPS 

6.1 Discussion of Model Results 

This report presents the results from our initial sensitivity analysis of the benefits from air 

quality regulations as estimated by BenMAP. The study examined the sources of uncertainty, the 

possible range of uncertainty for selected sources, the role and influence of the parameters or 

functions, and the sensitivity of the final benefit estimates to the different sources of uncertainty 

all in the simplified Analytica version of the BenMAP model. Here we discuss the results of the 

sensitivity analysis and suggestions for potentially useful next steps to explore the role of 

uncertainty on benefit estimation. 

 Comparing the results from the different mortality CR functions, the differences 

between the functions are larger than the uncertainty of individual CR-functions. In 

other words, the uncertainty inherent in choosing a CR-function has more impact on 

results (mortality reduction or total benefits) than the uncertainty within each CRfunction captured by its range sensitivity. 

 The elasticity analysis finds that the uncertain parameters delta PM2.5 concentration, 

the beta for the mortality CR function, population in 2000 and 2020, base mortality 

rates, and VSL all affect the total mortality reduction in an almost purely 

multiplicative fashion (Figure 5-1). Hence, the relative sizes of the range sensitivities 

(ratios of their widths) of these parameters are similar for each expert. Hence, most of 

the key results below on the relative range sensitivity are the same for all mortality 

CR-functions even if the mid points—mortality avoided and total net benefits—differ 

significantly from one expert to another (excepting Expert K because his mid estimate 

has zero mortality due to low probability of causation). 

 In all cases (again, except Expert K), the parameter with largest range sensitivity is 

the VSL. 

 For most of the results from the different mortality CR functions, the beta for 

mortality has the second largest range sensitivity analysis. 

 The range sensitivity to delta PM2.5, the reduction in concentration of PM2.5, is the 

third largest, and in some cases (experts E and F) comparable to the range sensitivity 

for the morality CR function beta. Examination of the uncertainty inherent in 

projections of air quality and improvements in air quality was beyond the scope of 

this project, so we assumed a plus or minus 30% change in this variable to illustrate 

the possible effects of uncertainty on the results. Again it is not surprising that this 

quantity has a high sensitivity. 
6-2 

 Among the quantities with lower range sensitivity, the most sensitive include 

– Population in 2020—i.e., uncertainty in population projections and the 

distribution of the population across the country. Migration and immigration 

ranked high as sources of uncertainty in the population forecasts and population 

distribution because of the difficulty in predicting movements of people and the 

difficulty in forecasting economic activity many years into the future according to 

experts consulted for the project. 

– The value of an avoided case of CB, which is highly uncertain at present. The 

value of avoiding an AMI is also highly uncertain and if WTP studies existed for 

AMI’s, AMI’s might exhibit larger and more influential range sensitivity. 

– Elasticity of WTP to income. 

 Discount rate is sensitive when using mortality valuation lag structures, especially 

those with longer average lag—and irrelevant with no mortality lag. It is a policy 

variable usually set by the government for purposes of public policy analysis 

(typically at 3% or 7% per year) rather than a quantity subject to empirical study. So, 

it is not clear that further study of this issue would be of value. 

6.1.1 Comments on the Uncertainty Analysis Methods 

The development of the influence diagrams in Section 4 provided an intuitive depiction 

of how the uncertain parameters combine and affect the results of interest. These diagrams 

reflect the actual Analytica model used for the numerical sensitivity analyses. In some cases, 

these diagrams include variables that are needed for the sensitivity analysis, but perhaps do not 

add clarity about the influence from a purely qualitative perspective. In support of transparency 

of our methods, we chose to display diagrams that reflect all the variables in the underlying 

quantitative model. 

Creating a simplified version of a complex model, such as BenMAP, provides a depiction 

of its essentials, but we did not attempt to reproduce all the computations in BenMAP, which we 

felt would detract from clarity and add unhelpful complexity. 

The diagrams in Section 4 also do not include the model elements involved in processing 

and aggregating the 36-km grid square data to the concentration bins, calibrating to BenMAP 

results, and performing the sensitivity analysis since they do not add clarity to the representation 

of the essentials of the benefits analysis. All these are available in the Analytica model for those 

that are interested. 

We developed the extremely simple model with 16 concentration bins to make sure that 

the model would run fast, enabling us to do a wide variety of sensitivity runs. At the end of the 

project, we also did some tests using the full 36 km grid square data in Analytica. It turns out that 
6-3 

it takes about 1 to 5 seconds for the full analysis, which is faster than BenMAP by 2 or 3 orders 

of magnitude. We suspect this may be partly due to Analytica's high-speed algorithms for 

dealing with large arrays, but we would need to do more extensive analysis of both models to 

discover the full reasons. This opens the possibility of doing sensitivity or Monte Carlo analysis 

of BenMAP at the full 36-km level of detail using the Analytica model in a future study if 

needed. 

6.2 Potential Next Steps

Below we discuss potential next steps based on the results from this uncertainty analysis. 

Based on research conducted by EPA and others, some of the largest sources of uncertainty are 

not good candidates for additional work. EPA has already devoted considerable resources to 

characterizing the uncertainty in the CR function for mortality, one of the most influential 

sources of uncertainty. Additional expert elicitation does not seem warranted without more and 

more conclusive empirical evidence. A similar conclusion applies to VSL. VSL is also much 

studied, and it is not clear that further study would lead to any significant reduction in its 

uncertainty. 

6.2.1 Update Studies to Support Morbidity Valuation 

Additional work on morbidity outcomes may offer important data for BenMAP. Without 

additional epidemiology studies, WTP values and updated studies, it is difficult to know whether 

the sensitivity analysis captures the full range of values. The range sensitivity to WTP to avoid 

CB is large ($77K to $1.6 million). The magnitude of the COI component is very small 

compared to the WTP. This highlights the importance of incorporating WTP into morbidity 

valuation. The CB WTP studies are both old and should be updated. Other morbidity endpoints, 

such as AMIs, have not been the subject of WTP studies. It may be worthwhile reviewing recent 

literature and investing in new studies to obtain updated estimates of COI and/or WTP for all 

health effects. The income elasticity of WTP is another parameter that could use additional 

research. 

In general, much of the valuation data in the current version of BenMAP is potentially 

out-of-date, especially with changes over the last decade or two in healthcare and in methods 

used to estimate WTP and COI. Thus, there are both (i) uncertainties in the data itself and (ii) in 

applying them. An example of the former in context of COI would be that newer estimates may 

reflect the advent of newer technologies that result in shorter hospital stays and lower severity of 

illnesses. Also, individuals may be more educated and thus may undertake better preventive and 

follow-up care. On the other hand, overall increased costs of health care may lead to increased 
6-4 

estimates of COI. True COI estimates for different illnesses in recent years may be smaller than 

those in the studies found in BenMAP or they may be larger. 

Uncertainties that stem from applying older estimates can also arise since current 

circumstances may be completely different from those of the studies. For example, the age 

composition (both current and projected) of highly polluted areas may be different from the ones 

that were used in the studies underlying BenMAP. This might create uncertainties associated 

with applying estimates to demographic groups that are different from the ones included in the 

study. For example, an older population will be at a higher risk to certain diseases and also 

experience more severe symptoms. 

The current version of the influence model creates “bins” based on pollution 

concentrations. One way to assess the sensitivity of results to different demographic groups 

might be to “bin” by demographic characteristics. 

6.2.2 More Detailed Mortality Valuation Estimates 

Ideally, WTP and COI estimates would be more tailored to specific populations. For 

example costs should reflect more regional differences in healthcare utilization. Patients with comorbidities may have different costs than otherwise healthy individuals who develop CB. 

Trajectories of costs for different outcomes should reflect common complications or sequela, 

health conditions that arise after an injury or disease. 

6.2.3 Framework to Jointly Consider Multiple Health Endpoints 

BenMAP incorporates information on incidence rates, COI and WTP from studies 

considering each health endpoint separately rather than in conjunction with other co-morbidities. 

This has two implications. First of all, there may be biased estimates due to confounding factors. 

Also, the general framework and associated assumptions differ across these studies. A review of 

recent literature to consider studies exploring multiple endpoints (for example, work being 

conducted by Cameron and DeShazo on WTP that incorporates multiple outcomes) might 

provide ideas. 

6.2.4 Add Additional Outcomes and Expand Application to Other Pollutants 

This study looked at only mortality, CB and AMI. Incorporating other outcomes, both 

health and non-health (such as visibility) would complete the sensitivity analysis. Expansion of 

the uncertainty analysis to cover some of these additional benefits would require extensions of 

the benefits model beyond BenMAP. 
6-5 

The results presented in this report arise from a particular air quality scenario (the 2006 

PM RIA 15/35 option). Application of the model to alternative pollutants, such as ozone, 

additional regulatory scenarios for PM or to different species of PM were all suggested as 

possible topics for exploration that would provide some evidence on how far the results from one 

specific scenario can be generalized to other pollutants or air quality regulations. 

6.2.5 Expanding Scope of the Uncertainty Analysis to Include Air Quality 

The project scope imposed limitations on the study of uncertainties by excluding 

uncertainty in the base concentration and delta concentration in PM2.5. These concentrations are 

estimated and projected by a variety of emissions and air quality models, including CMAQ, as 

we briefly showed in Figure 4-13. An explicit uncertainty analysis of the air quality models 

would be a significant effort, probably larger than the uncertainty analysis of BenMAP 

undertaken here, due to the greater complexity of the emissions and atmospheric transport 

models. In the absence of careful analysis of the uncertainty in the air quality estimates, we chose 

ranges for these quantities, plus or minus 15% for the base concentration, and plus or minus 30% 

for the deltas. The range sensitivity showed low sensitivity to the base concentration for nonthreshold CR functions and range sensitivity to the delta was usually third the largest (beta on the 

mortality CR function and VSL). 

A more thorough uncertainty analysis of benefits would require a more careful 

assessment of the ranges for the base and delta concentrations—based on analysis of uncertainty 

in projections using a simplified model of the air quality models, using an approach similar to 

that conducted here for BenMAP. It should also include uncertainty analysis based on 

comparison of air quality projections with actual observations, something that is not directly 

possible for benefits calculations. 

6.2.6 Investigate Cost Uncertainty 

The study does not address the costs of improving air quality. Explicit treatment of costs 

would require uncertainty analysis of the emissions models used to generate emissions 

inventories used by CMAQ. The addition of economic costs would require uncertainty analysis 

of the costs of reduced emissions. 

6.2.7 A More Comprehensive Treatment of Uncertainties 

This project focused on range sensitivity analysis, varying each of the uncertain 

parameters one at a time while leaving the other parameters fixed at their mid values. This 

approach does not assess the overall uncertainty in the results from the combined effects of all 

the uncertainties. One approach would be to examine the results when you set all parameters to 
6-6 

their low values—or all to their high values. This "Murphy's law" approach assumes that 

"everything that can go wrong does go wrong"—i.e., all parameters are at their "worst" level 

simultaneously. Conversely, everything goes right—all parameters are at their best level 

simultaneously. But the chance of all parameters being at one extreme simultaneously is 

vanishingly small: Given the 17 parameters (in the analysis here), if we assumed that the low 

value is the 10th percentile, the chance of all 17 being at or less than their low value 

simultaneously (or all at their high value) assuming independence would be 10-17. Hence, the 

resulting range would be implausibly wide. It is for this reason we did not perform and do not 

recommend this approach. 

A second limitation of the project scope was the exclusion of a probabilistic analysis. A 

full probabilistic analysis using Monte Carlo simulation would be the best approach to examine 

the combined effect of the multiple uncertainties on results. This approach would require an 

explicit probability distribution be assessed for each parameter—as they already were for the 

CR-mortality functions by experts A through L. It would also require interdependencies between 

parameters to be modeled. If EPA desires to obtain a more complete uncertainty analysis, we 

would strongly recommend this approach. 

6.2.8 Large Changes in Population, Migration and Immigration 

Looking back on past immigration or internal migration patterns might provide bounds 

on the possible size of large immigration or migration events. One could develop scenarios that 

resulted in large movements of population or increased/decreased immigration in response to 

very low probability events that have a large impact (hurricanes, droughts, changes in climate, 

political instability). 

6.2.9 Additions to Analytica Model 

As discussed in this report, the Analytica model is a simplified version of BenMAP. 

Further work to add more of the complexity from BenMAP into the Analytica model and using 

the full 16,000 points rather than bins removes some assumptions and increases the range of 

problems that can be assessed with the simplified model. 
7-1 

SECTION 7 

REFERENCES 

Abbey, D. E., F. Petersen, P. K. Mills and W. L. Beeson. 1993. Long-Term Ambient 

Concentrations of Total Suspended Particulates, Ozone, and Sulfur Dioxide and 

Respiratory Symptoms in a Nonsmoking Population. Archives of Environmental Health. 

Vol. 48 (1): 33-46. 

Abbey, D. E., B. L. Hwang, R. J. Burchette, T. Vancuren and P. K. Mills. 1995. Estimated longterm ambient concentrations of PM10 and development of respiratory symptoms in a 

nonsmoking population. Arch Environ Health. Vol. 50 (2): 139-52. 

Industrial Economics, Inc. (IEc), 2006. Expanded Expert Judgment Assessment of the 

Concentration-Response Relationship Between PM2.5 Exposure and Mortality. Prepared 

for the U.S. EPA, Office of Air Quality Planning and Standards, September. 

Krupnick, A. J. and M. L. Cropper. 1992. The Effect of Information On Health Risk Valuations. 

Journal of Risk and Uncertainty. Vol. 5 (1): 29-48. 

Laden, F., J. Schwartz, F. E. Speizer and D. W. Dockery. 2006. Reduction in Fine Particulate Air 

Pollution and Mortality: Extended follow-up of the Harvard Six Cities Study. Am J 

Respir Crit Care Med. Vol. 173 (6): 667-72. 

Peters, A., D. W. Dockery, J. E. Muller and M. A. Mittleman. 2001. Increased particulate air 

pollution and the triggering of myocardial infarction. Circulation. Vol. 103 (23): 2810-5. 

http://www.circulationaha.org/cgi/content/full/103/23/2810.

Pope, C. A., 3rd, R. T. Burnett, M. J. Thun, E. E. Calle, D. Krewski, K. Ito and G. D. Thurston. 

2002. Lung cancer, cardiopulmonary mortality, and long-term exposure to fine 

particulate air pollution. Jama. Vol. 287 (9): 1132-41. 

Rosamond, W., G. Broda, E. Kawalec, S. Rywik, A. Pajak, L. Cooper and L. Chambless. 1999. 

Comparison of medical care and survival of hospitalized patients with acute myocardial 

infarction in Poland and the United States. Am J Cardiol. Vol. 83 (8): 1180-5. 

U.S. Department of Commerce, Economics and Statistics Administration, Bureau of the Census. 

February 1996. CURRENT POPULATION REPORTS Population Projections of the 

United States by Age, Sex, Race, and Hispanic Origin: 1995 to 2050, P25-1130. 

U.S. Department of Commerce, Economics and Statistics Administration, Bureau of the Census. 

July 2002. U.S. Summary: 2000, Census 2000 Profile, C2KPROF/00-US. 

U.S. Department of Energy, Department of Energy (DOE). March 2009. Macroeconomic 

Indicators, Projection Tables (Table 20), Updated Annual Energy Outlook 2009 

Reference Case Service Report,. Report #:DOE/EIA-0383(2009). 

http://www.eia.doe.gov/oiaf/aeo/index.html. 
7-2 

U.S. Environmental Protection Agency (EPA). September 2008. BenMAP User’s Manual. 

Office of Air Quality Planning and Standards Research Triangle Park, NC. 

Viscusi, W. K., W. A. Magat and J. Huber. 1991. Pricing Environmental Health Risks—Survey 

Assessments of Risk—Risk and Risk—Dollar Trade-Offs For Chronic Bronchitis. 

Journal of Environmental Economics and Management. Vol. 21 (1): 32-51. 

Woods & Poole Economics Inc. 2007. Complete Demographic Database. Washington, DC. 

http://www.woodsandpoole.com/index.php. 
A-1 

APPENDIX A 

EXPERT WORKSHOP 
RTI International is a trade name of Research Triangle Institute

TO: Amy Lamson (Office of Air Quality Planning and Standards, U.S. Environmental 

Protection Agency) 

FROM: Carol Mansfield (RTI International) 

Amir Moktari (RTI International) 

Paramita Sinha (RTI International) 

Max Henrion (Lumina Decisions Systems)

DATE: February 13, 2009

SUBJECT: Influence and Uncertainty Analysis of Human Health Benefits Estimates: 

Capturing Uncertainty in Input Data and Uncertainty Introduced in BenMAP 

Modeling Workshop

On December 15 and 16, 2008, the Office of Air Quality Planning and Standards (OAQPS) of 

the U.S. Environmental Protection Agency (EPA) held an expert workshop to solicit input on their 

planned influence analysis of the human health benefits of reductions in particulate matter (PM). The 

meeting was held at the OAQPS office in Research Triangle Park, North Carolina. 

OAQPS is developing a model to conduct an influence analysis on the uncertain parameters, functions, 

and input data used to calculate the benefits of reductions in PM using BenMAP, the software program 

OAQPS uses to estimate economic benefits for many regulatory analyses. The model will be used to help 

OAQPS identify the degree to which uncertainty about key parameters and functional forms affects the 

benefits analysis. Long term, this analysis will help prioritize future research and provide a template for 

conducting influence analyses on other benefit assessments. The influence analysis will use Analytica 

software to create a simplified representation of BenMAP’s calculations. 

The workshop focused on uncertainty in population forecasts, valuation of mortality, and 

valuation of acute myocardial infarctions (AMIs) and chronic bronchitis in BenMAP. Outside 

experts were asked for input on the sources of uncertainty, structure of the influence analysis 

model, and the range of values that OAQPS should consider. 

Specifically, the goals of the workshop were to 

• Ensure important sources of uncertainty were not omitted, focusing first on the 

population data and forecasts and valuation techniques and estimates; 

• evaluate the simplified model OAQPS is creating to conduct the influence analysis; 

• solicit appropriate range data and distributions for uncertainty parameters and 

alternative functional form assumptions for model uncertainty to use in the sensitivity 

analysis; and 
Amy Lamson 

May 21, 2009 

Page 2 

• solicit input on the methodology for the sensitivity analysis. 

Before the workshop, OAQPS and RTI International identified a set of experts on population 

forecasting, morbidity and mortality valuation, and uncertainty analysis to participate in the 

workshop. OAQPS also indentified EPA staff to be included. Appendix A provides a list of the 

participants. The participants were e-mailed a guide that provided the goals of the workshop, an 

agenda, background information, and several tables that listed the sources of uncertainty related 

to population forecasting, mortality valuation, and morbidity valuation identified before the 

workshop. The Participants Guide is included in Appendix B. 

1. Overview 

The 1.5 day meeting was divided into seven sessions. The first three sessions included an 

introduction by Amy Lamson (EPA), an overview of BenMAP by Neal Fann (EPA), and an 

overview of the simplified Analytica model proposed for the influence analysis by Max Henrion 

(Lumina Decision Systems). The next three sessions each focused on a different topic, including 

population forecasting, valuation of mortality, and valuation of morbidity. For each discussion, 

Max Henrion presented the sources of uncertainty identified by the planning team before the 

workshop and how they fit into the Analytica model. After the presentation, the workshop 

participants spent 5 minutes thinking about the topic and additional sources of uncertainty. After 

5 minutes, the participants at each table discussed their thoughts on sources of uncertainty with 

each other. This discussion was followed by a group discussion where each table reported 

sources of uncertainty they identified, and Ken Elstein (EPA) summarized the discussion on a 

computer linked to an overhead projector. Next, each participant spent time thinking about 

whether each source of uncertainty could be quantified, the possible range of values or functions 

that could be used in the influence analysis, and any comments on the uncertainty. Again, after 

the individual brainstorming, the participants at each table discussed their ideas and then each 

table reported back to the whole group. The group also discussed areas of agreement and 

disagreement, which were captured by Ken Elstein. The final session started with a review of the 

Analytica model by Max Henrion followed by a group discussion on the approach to the current 

influence analysis and on ways to model the uncertainty in BenMAP. 

2. Session 2: Overview of BenMAP 

Neal Fann presented an overview of BenMAP (slides in Appendix C). The main steps in 

BenMAP can be summarized as follows: 
Amy Lamson 

May 21, 2009 

Page 3 

• Step 1: Use studies to find relative risk estimates that show the relationship between 

pollutant concentration and the health effect. Find the incremental change in the 

health outcome based on a unit change in concentration (β) with standard error. 

– Note: BenMAP does not model population exposure, assuming all exposure 

occurs where the population resides. 

– Note: For some functions, EPA includes threshold-adjusted functions in BenMAP 

that refit the data for the threshold model from the original nonthreshold model 

(in the original epidemiology study). 

• Step 2: Apply the health impact function from Step 1 to the incremental air quality 

improvement for the population and background incidence rate in each air quality 

grid square to estimate the change in health outputs. 

– Note: BenMAP includes preloaded national-level health data and applies it to a 

smaller-scale area such as the grid size associated with the air quality data 

(typically 12 km square or 36 km square). If local-scale data is available, it can be 

loaded into BenMAP. 

– Note: County-level population is mapped to the air quality grid squares before 

being imported into BenMAP using a program called Popgrid. 

• Step 3: Value health outcome using willingness to pay (WTP) or cost of illness (COI) 

estimates by multiplying the change in health incidence by WTP or COI. In 

BenMAP, a distribution of health outcome incidence and a distribution of WTP or 

COI are created, and random draws from both distributions create the distribution of 

total benefits. 

3. Session 3: Overview of Simple Analytica Model of 

BenMAP 

Max Henrion presented an overview of the simple model of BenMAP he has created using 

Analytica software (slides in Appendix C). The objectives for the model are to (1) identify 

sources of uncertainty in BenMAP, (2) visualize how the sources of uncertainty affect the results 

using an influence diagram, (3) assess the range of values of key uncertainties, and (4) compare 

their relative contributions to the results using sensitivity analysis. The simplified Analytica 

model will be calibrated to BenMAP by comparing the output of the simplified model at 

different steps with runs of BenMAP. 

BenMAP is a complex program with a lengthy running time for large-scale analyses, and 

conducting an extensive influence or sensitivity analysis with BenMAP would be time 

consuming. With a simplified model of BenMAP, we can conduct initial influence analysis to 

identify parameters or functional form assumptions that have the biggest impact on the outcomes 

of the benefits analysis, including both health incidence and monetary benefits. The simplified 

model allows us to test parameters over a wide range of values without requiring excessive 
Amy Lamson 

May 21, 2009 

Page 4 

resources. Analytica provides a flexible and transparent format for the analysis. Max Henrion 

presented an example using tornado graphs to show sensitivity of the simplified model to 

different sources of uncertainty based on fabricated data. 

To reduce the size of the data set and the complexity of the Analytica model, the air quality grid 

squares were aggregated into 16 bins based on initial ambient air quality concentration. We also 

evaluated sensitivity to the level of aggregation (e.g., instead of 16 bins we can use a smaller 

number of bins). More concentration bins means more detail and decreasing aggregation. A 

variety of aggregation issues can be addressed, including spatial and temporal, over age groups 

and across endpoints. 

The participants discussed some of the assumptions used to create the Analytica model and 

pointed out some differences between the way BenMAP treats certain aspects of the model and 

the way the preliminary Analytica model does. For example, uncertainty varies depending on the 

control strategy. The control strategy that generated data used for this example is a more urbanfocused control strategy. A different control strategy might have a different level of uncertainty 

if it is a more regional approach. With the binning approach, one may lose the ability to consider 

the geographic impact of applying policy changes. To address these concerns, the Analytica 

model will need to be tested using several control strategies and different assumptions about bin 

size and then calibrated to BenMAP. 

The following changes in terminology were requested: 

• Use “avoided premature mortalities” instead of “lives saved” 

• Use “value of a statistical life” instead of “value of statistical life saved” 

• Use “concentration-response” instead of “dose-response” 

• Use “health impact function” instead of “health effects function” 

• Use “monetized benefits” instead of “benefits” (unless referring generically to 

incidence reduction and monetized benefits) 

• Use “total monetized benefits” instead of “net present value of benefits” (because we 

generally do not evaluate streams of benefits) 

• Use “cessation lag” instead of “VSL adjusted for lag” (because we adjust the total 

mortality benefits, not the VSL specifically) 

4. Session 4: Population 

Max Henrion started the session with a presentation on how population enters the Analytica 

model (Appendix C). The experts were asked to think about additional sources of uncertainty not 

captured in the current model and for their judgment on the ranges for uncertain parameters. For 
Amy Lamson 

May 21, 2009 

Page 5 

the influence analysis, we want plausible low, mid, and high values or perhaps the 10th

percentile, median, and 90th percentile. 

Much of the discussion centered on the factors that affect the accuracy of the Census population 

forecasts. The Census forecasts population at the national level and does not use an economic 

model to support the forecasts. Migration and immigration emerged as two factors that the 

Census projections may not capture well. BenMAP uses Woods and Poole to project the 

population data at the county level, and this dataset does include an economic component to its 

forecasts. 

The group also discussed uncertainty introduced through the assignment of the population 

forecasts to the air quality grid squares. The air quality grid squares are at a finer resolution than 

the underlying health data. The Census data capture where people live, but people do not 

typically remain in one location throughout the day. The finer the resolution, the more likely that 

exposure will be misclassified if the population travels between grid squares during the day. A 

higher level of aggregation might be preferable. Participants suggested a case study (e.g., assign 

a population based on land use categories or check the sensitivity of one grid cell by assigning all 

of the population to a single grid square, such as the grid square with the highest PM2.5 

concentration in the area). 

The participants also discussed other measures of sensitivity. Experts were concerned that by 

defining ranges we would actually introduce another source of uncertainty. If the range is 

unknown, it may be more appropriate to use elasticity rather than range since a range can be 

misleading if one end is a low-probability event. 

Discrete events that result in large-scale population and migration shifts within the United States 

(e.g., Hurricane Katrina) were discussed. Scenario analysis could be used to address the impact 

of discrete events. The importance of uncertainty in population forecasts after 2030 was thought 

to be most important for climate change and other policies where the benefits may not occur in 

the near future. Scenario analysis may be the way to address the uncertainty of population 

forecasts over a longer time period. 

Recommendations 

Overall, the population expert and others in the group felt that, although uncertainties related to 

population forecasting were certainly worth exploring, population uncertainties are perhaps not 

as large as other uncertainties in the model. For the influence analysis, the Census low, medium, 

and high forecasts were thought to provide a good range. For allocating population to counties, 

the Woods and Poole confidence intervals were thought to provide a good range. 
Amy Lamson 

May 21, 2009 

Page 6 

5. Session 5: Premature Mortality Valuation 

The session started with an overview from Max Henrion on how the estimates of premature 

mortality and the value of reducing the risk of premature mortality (mortality valuation) are 

generated in the simplified Analytica model of BenMAP (slides in Appendix C). Mortality 

valuation is often referred to in the economics literature as the value of a statistical life or the 

VSL, but it is really the value of reducing the risk of premature mortality. EPA uses a single VSL 

function (a normal distribution with a mean of $5.5 million) that does not vary across people 

(e.g., demographics, health status, or cause of death). Because exposure in a particular year will 

cause premature deaths in the future, the monetized benefits are discounted to account for the 

cessation lag, typically over 5 years to 20 years. 

There was a lively debate on whether VSL should be treated as a policy decision, rather than an 

uncertain empirical value. VSL measures the value of a reduction in mortality risk to an 

individual. VSL is internally consistent within the utilitarian framework, which is the framework 

for benefit-cost analysis. Some argued that policy makers should use alternative VSL 

frameworks (e.g., social choice, altruism, revealed preference for government programs). The 

current VSL estimates do not value the distribution of risk within the population (e.g., 

environmental justice issues). 

The discussion identified a number of factors that influence both the degree of uncertainty (in the 

strict sense of the word) associated with VSL estimates and the variability in VSL across 

individuals or over time as individuals age or wealth increases. Personal characteristics that may 

cause VSL to vary across individuals include age, health status, income or wealth, number of 

children, differing preferences over the characteristics of the risk (voluntary, unknown, dread 

factor), cultural factors, and wealth. 

VSL values have been estimated using both revealed preference (RP) studies and stated 

preference (SP) studies. The RP studies mostly look at the trade-off between wages and job risk. 

The population in RP studies (healthy, working-age adults) and the type of risks differs (mostly 

sudden death from a workplace accident or homicide) from populations susceptible to premature 

mortality from PM exposure, which, on average, will be older and less healthy). SP studies are 

more likely to value risks similar to those associated with PM exposure such as the risk of 

chronic illnesses like cancer or other chronic diseases. Most SP studies survey the general 

population and include older adults and individuals with compromised health status. One 

suggestion from the participants was to use VSL studies that match more closely the type of risk 

posed by exposure to PM and to compare the range of SP and RP studies. Finally, WTP can be 

elicited for risk of death in the future. WTP for risk of death in the future could replace the 
Amy Lamson 

May 21, 2009 

Page 7 

assumed lag structure currently used for benefits analysis, which could be described as somewhat 

ad hoc based on several interpretations of the epidemiology data. 

Participants discussed the income elasticity of VSL and thought, in general, that it was not likely 

to be a large source of uncertainty. However, there were questions about using cross-sectional 

estimates of income elasticity to estimate elasticity over time. 

Recommendations 

Everyone recognizes the importance of the VSL in calculating the total benefits of a reduction in 

PM. For the influence analysis, the range of $1 million to $10 million was supported by most 

participants, although a lower bound of $700,000 could also be considered. If we use the range 

from SP studies, it may be more like $1 million to $3 million. 

6. Session 6: Morbidity Valuation 

The session on morbidity valuation focused on chronic bronchitis and AMI. Max Henrion 

discussed how the incidence and value of the diseases enter the Analytica model (slides in 

Appendix C). 

The participants discussed the general issue of WTP estimates and COI estimates. Theoretical 

differences in the costs captured by each measure and how to avoid double-counting were 

discussed. We are unaware of WTP estimates for AMIs, and the WTP estimates for chronic 

bronchitis are from SP studies with known problems, including the age of the studies, the small 

and unrepresentative sample, and advances in SP methodology since the studies were conducted. 

In addition, the studies compared risk/risk trade-offs with risk/dollar trade-offs and concluded 

the risk/risk trade-offs were more stable. The COI estimates were also thought to be dated. 

Medical costs, technology, and public awareness of diseases and treatments have all changed 

significantly over time. 

Some questions raised by the participants related to the health and epidemiology literature. The 

severity of PM-related AMIs and chronic bronchitis are not known, which complicates the task 

of deciding which WTP or COI estimates are appropriate. Participants suggested that the model 

consider several disease trajectories with different cost implications. Another source of 

uncertainty relates to sequelae resulting from PM-related AMIs and chronic bronchitis, costs 

beyond the hospital stay, costs to unpaid caregivers, and long-term impacts on life expectancy 

and quality of life. Assumptions about the duration of the illnesses and the proper discount rate 

were raised. A number of sources for uncertainty and variability came up in the discussion, 

including regional and urban/rural variation in medical costs, gender, age, and health status. 
Amy Lamson 

May 21, 2009 

Page 8 

Finally, the group also discussed the best methods for pooling the results from several studies for 

morbidity endpoints. 

Recommendations 

For AMIs, the participants thought that we should search for more recent COI estimates to get a 

better sense of the range of possible costs and use all the available studies to create the range of 

values. Several studies have shown that WTP tends to be a factor of 2 larger than COI for the 

same health endpoint, and those studies together with WTP studies for related diseases could be 

compared to COI. Wage losses for individuals under age 27 and over 65 should be included in 

the range. 

For chronic bronchitis, participants suggested we use the WTP values for most severe and least 

severe in the range. They also suggested that COI estimates be incorporated. 

For both morbidity endpoints, the suggested range was 1⁄2 COI in existing BenMAP studies (if 

newer estimates cannot be found) to the top end of COI + WTP + adjustments for sequelae costs 

and other excluded categories. 

7. Session 7: Discussion and Recommendations on 

Approach to Influence and Uncertainty Analysis 

In the final session, Max Henrion reviewed the simplified BenMAP model being developed 

using Analytica software for the influence analysis (slides in Appendix C, revised influence 

diagrams in Appendix D). The discussion focused on the approach being proposed for the 

influence analysis and solicited comments and suggestions from the workshop participants. 

The influence analysis will start by creating a Tornado diagram. For each parameter or functional 

form assumption, the model is run using the range of possible values for the parameter or 

functional form holding all the other parameters at their mean value. The Tornado diagram 

stacks the range of total benefits associated with the range of the parameter value (or functional 

form assumptions) from the parameters and functional form assumptions that generate the largest 

range in total benefit estimates down to the smallest. 

The Tornado diagram does not account for interactions between parameters or between 

parameters and functional form assumptions. There was a discussion about how to handle 

interactions. One suggestion by several participants was to use probabilistic sensitivity analysis 

measures such as rank correlations and regression measures. However, one participant felt 

strongly that we should not make any assumptions about probability distributions where we did 
Amy Lamson 

May 21, 2009 

Page 9 

not have information about the distribution. Instead, it was suggested that a response surface 

using Latin hyper-cube points could be created. 

Recommendations 

The list of recommendations summarizes a variety of perspectives and does not represent a 

consensus on the part of the participants. 

• Conduct analysis both using Tornado diagram and allowing for interactions. 

• Conduct comparisons between the results of the simplified Analytica model of the 

BenMAP model and runs from BenMAP itself to make sure the Analytica model will 

provide results that mimic the results that would have been achieved using BenMAP 

itself. 

• Identify parameters and functional form assumptions that have the largest impact on 

the outcomes of interest (monetary benefits and health incidence). Explore these 

sources in more detail. 

• Explore the sensitivity of both the monetary value of benefits and the incidence of the 

health endpoints, since both are important to people. 

• After influence analysis identifies potentially influential parameters and functional 

form assumptions, EPA will assess the level of resources that might be needed to 

reduce the uncertainty. 

• Use two or three different control scenarios to generate reductions in concentrations. 

Whether the reductions are local or regional, or just an overall percentage reduction, 

may affect the relative importance of the parameters. 

The discussion also included ideas on how to refine the simplified model: 

• Initially, the model assumes that the entire exposed population in each base 

concentration bin experiences the same reduction in PM2.5 concentration levels 

(averaged over the reductions for all grid squares in that bin). Later, if time permits, 

we could expand the bins into a bin array with a second dimension of concentration 

reductions for each of the concentration bins and see if this reduces the aggregation 

uncertainty. Therefore, each bin would include grid squares with the same baseline 

concentration and the same reduction in concentration. 

• The model does not differentiate among population segments (i.e., race, ethnicity, or 

gender). Initially, the model uses an average across age groups, but this will be 

refined for the final analysis. 
Amy Lamson 

May 21, 2009 

Page 10 

Appendix A : Participant List 

Soad Benromdhane, OAQPS, HEID 

Lillian Bradley, OAQPS, HEID 

Linda Chappell, OAQPS, AQAD 

Ken Davidson, OTAQ 

Chris Dockins, OPEI, NCEE 

Ken Elstein, ORD 

Ron Evans, OAQPS, HEID 

Neal Fann, OAQPS, HEID 

Chris Frey, NCSU 

Charlie Fulcher, OAQPS, HEID 

Beth Hassett-Sipple, OAQPS, HEID 

Brian Heninger, OPEI, NCEE 

Max Henrion, Lumina 

Bob Hetes, ORD, NHEERL 

Bryan Hubbell, OAQPS, HEID 

Alan Krupnick, RFF, econ 

Amy Lamson, OAQPS, HEID 

John Langstaff, OAQPS, HEID 

Carol Mansfield, RTI 

Amir Mokhtari, RTI 

Ellen Post, Abt 

Pradeep Rajan, OAQPS, HEID 

Harvey Richmond, OAQPS, HEID 

Paramita Sinha, RTI 

Laura Taylor, NCSU, econ 

Scott Voorhees, OAQPS, HEID 

Paul Voss, UNC 

Darryl Weatherhead, OAQPS, HEID 
Amy Lamson 

May 21, 2009 

Page 11 

Appendix B : Participant Guide 

Influence and Uncertainty Analysis of Human Health Benefits Estimates: 

Capturing Uncertainty in Input Data and Uncertainty Introduced in BenMAP 

Modeling 

Participants Guide 

U.S. Environmental Protection Agency 

Research Triangle Park, NC 

December 15–16, 2008 
Amy Lamson 

May 21, 2009 

Page 12 

Dear Participant, 

On behalf of the U.S. EPA, I would like to welcome you to the workshop. EPA’s Office of Air Quality 

Planning and Standards organized this meeting to solicit your input on our planned influence analysis of 

the human health benefits of reductions in PM. A 2002 National Academy of Sciences report 

recommended that EPA conduct an Influence Analysis to improve uncertainty characterization in our 

benefits analyses and to identify the modeling elements and assumptions that have the largest influence 

on the human health benefits results. We will present a preliminary model for influence analysis focusing 

on the valuation of human health benefits associated with mortality and AMIs estimated using BenMAP. 

As we develop our approach, we want outside advice from experts like you on the sources of uncertainty, 

structure of the influence analysis model, and the range of values we should consider. The results of this 

analysis will help to prioritize future efforts to reduce uncertainty. We also plan to take the results of this 

report to EPA’s Science Advisory Board sometime next year. 

We have set three main goals for this workshop: 

Desired Goals 

1. Ensure we have not left out any important sources of uncertainty focusing first on the population 

data and forecasts and valuation techniques and estimates. 

2. Evaluate the simplified model we are creating to conduct the influence analysis 

3. Solicit appropriate range data and distributions for uncertain parameters and alternative functional 

form assumptions for model uncertainty to use in the sensitivity analysis 

4. Solicit input on the methodology for the sensitivity analysis 

Desired Outcomes 

1. Comprehensive list of uncertainties in benefits analysis arising from population data/forecasts and 

from valuation approach. 

2. Confirmation that simplified influence analysis model of BenMAP is accurate and complete 

3. Appropriate range data and/or distributions to use in the sensitivity analysis 

4. Enhanced understanding of the benefits analysis process and assumptions by participants 

5. Methods for incorporating uncertainties into the sensitivity analysis 

Your active participation is crucial to achieving these goals. 

Please take a few minutes to familiarize yourself with this Participants’ Guide, paying particular note to 

the “Meeting Principles” and “Lists of Uncertainties.” If there is anything we can do to improve the 

workshop, do not hesitate to let me or the facilitator know. Thank you for your interest and participation 

in this workshop. 

Sincerely, 

Amy Lamson 

Air Benefits and Cost Group 

Office of Air Quality Planning and Standards 

U.S. Environmental Protection Agency 
Amy Lamson 

May 21, 2009 

Page 13 

Agenda for

Influence and Uncertainty Analysis of Human Health Benefits Estimates: Capturing uncertainty in 

input data and uncertainty introduced in BenMAP modeling 

Classroom C112 in EPA Main Building, EPA RTP Campus 

(109 T.W. Alexander Drive, RTP N.C. 27711) 

Day 1: Monday, December 15th

Registration (8:30am to 9:00am) 

Session 1 (9:00am to 9:30am): Welcome and Introductions—Amy Lamson, OAQPS 

Session 2 (9:30am to 10:15am): Introduction to benefits and BenMAP—Neal Fann, OAQPS 

Break (10:15am to 10:30am) 

Session 3 (10:30am to 12.00pm): Overview of the simplified influence analysis model and project 

strategy—Max Henrion, Lumina Decision Systems 

Lunch (12:00pm to 12:45pm): EPA cafeteria (on your own) 

Session 4 (12:45pm to 2:45pm): Population estimates 

12:45pm to 1:00pm: Identified sources of uncertainty in population estimates and uncertainty introduced 

by BenMAP modeling assumptions—Max Henrion, Lumina Decision Systems 

1:00pm to 2:45pm: Discussion on current population estimates in BenMAP 

o 5min Quiet brainstorming on the following questions: 

 Are there additional sources of uncertainty in population forecasts or introduced 

by BenMAP’s use of the population data? 

 Do you think this will have big impact on benefits results? 

o 15min Report out and discussion 

o 10min Quiet brainstorming on the following questions: 

 Can/should this source of uncertainty be modeled quantitatively in the influence 

analysis? 

 What range of values should we use for these parameters or where can we get 

the range (i.e., from a study or existing data)? 

 Are there other functional forms or modeling approaches that make sense for 

this component of the model or what would be a good source for this 

information? 

o 30min Small group discussion 

 Where do we agree and where do we disagree? 
Amy Lamson 

May 21, 2009 

Page 14 

o 45min Report out and discussion 

Break (2:45pm to 3:00pm) 

Session 5 (3:00pm to 5:15pm): Valuation of mortality reduction 

3:00pm to 3:30pm: Identified sources of uncertainty in valuation of mortality and uncertainty introduced 

by BenMAP modeling assumptions—Max Henrion, Lumina Decision Systems 

• 3:30pm to 5:15pm: Discussion on valuation of mortality reduction 

o 5min Quiet brainstorming on the following questions: 

 Are there additional sources of uncertainty in valuation of mortality reductions or 

introduced by BenMAP’s use of valuation studies? 

 Do you think this will have big impact on benefits results? 

o 15min Report out and discussion 

o 10min Quiet brainstorming on the following questions: 

 Can/should this source of uncertainty be modeled quantitatively in the influence 

analysis? 

 What range of values should we use for these parameters or where can we get the 

range (i.e., from a study or existing data)? 

 Are there other functional forms or modeling approaches that make sense for this 

component of the model or what would be a good source for this information? 

o 30min Small group discussion 

 Where do we agree and where do we disagree? 

o 45min Report out and discussion 

Wrap-up (5:15 to 5:30) 

Dinner with group 6:30pm (optional, otherwise on your own) Please let Carol Mansfield know if 

you are interested in the group dinner
Amy Lamson 

May 21, 2009 

Page 15 

Day 2: Tuesday, December 16th

Welcome (8:30am to 8:45am): Introduction, recap from Day 1—Amy Lamson, OAQPS 

Session 6 (8:45am to 10:45am): Valuation of morbidity reduction 

8:45am to 9:00am: Identified sources of uncertainty in valuation of mortality and morbidity reduction and 

uncertainty introduced by BenMAP modeling assumptions—Max Henrion, Lumina Decision 

Systems 

• 9:00am to 10:00am: Discussion on valuation of morbidity reduction 

o 5min Quiet brainstorming on the following questions: 

 Are there additional sources of uncertainty in morbidity valuation or introduced 

by BenMAP’s use of the morbidity valuation models? 

 Do you think this will have big impact on benefits results? 

o 15min Report out and discussion 

o 10min Quiet brainstorming on the following questions: 

 Can/should this source of uncertainty be modeled quantitatively in the influence 

analysis? 

 What range of values should we use for these parameters or where can we get the 

range (i.e., from a study or existing data)? 

 Are there other functional forms or modeling approaches that make sense for this 

component of the model or what would be a good source for this information? 

o 30min Small group discussion 

 Where do we agree and where do we disagree? 

Break (10:00am to 10:15am) 

• 10:15am to 10:45am: Discussion on valuation of morbidity reduction, con’t 

o 30min Report out and discussion 

Session 7 (10:45am to 12:20pm): Discussion and Recommendations on Approach to influence and 

uncertainty analysis—Max Henrion, Lumina Decision Systems 

Closing (12:20 to 12:30) Wrap-up and Next Steps 
Amy Lamson 

May 21, 2009 

Page 16 

Meeting Principles 

1. Everyone shares the responsibility for making the meeting a success: This meeting is a true 

collaboration between all attendees, including the organizers and facilitators. Your active 

participation is critical. If you have an idea for a discussion topic or process suggestion, please 

share it orally and/or in writing through a posted note. If you have a criticism, please balance it 

with a solution. 

2. Stay on topic: Start from the “big picture” before moving into details. Jumping to details 

prematurely can consume a lot of time on a topic that the group may later decide is unnecessary. 

A “Parking Lot” is available to post ideas/comments to ensure that they are addressed at the 

appropriate time. 

3. Listen and understand: All participants bring to this meeting diverse experiences, ideas, 

knowledge, and perspectives. Inquire of others to understand their views before advocating your 

own. 

4. Be transparent: Our assumption is that all participants are coming to this meeting with the intent 

of making the workshop more effective. Those with individual needs/concerns should make them 

known to the group so we can develop innovative approaches to meet all needs. 

5. First brainstorm, then critique: The most creative ideas emerge when participants can build 

upon each other’s suggestions. Often the seemingly wildest ideas stimulate true innovation. 

Avoid premature critiquing that can unintentionally shut down the creative process. 

6. Provide everyone an equal opportunity to speak: Part of our diversity includes variations in 

how we prefer to express ourselves. Freely offer your perspectives in whatever way is most 

comfortable for you, and allow others the opportunity to express theirs. 

7. Commit to being fully present: Please turn off cell phones, or put them on vibrate; put away the 

PDAs and Blackberrys. You can always check them during breaks. 
Amy Lamson 

May 21, 2009 

Page 17 

Background Information

For more information regarding the model and type of analysis to be discussed during the 

workshop, please consult the following websites. 

• Benefits analysis 

o OAQPS Economic Guidelines (1999): 

http://www.epa.gov/ttn/ecas/econdata/Rmanual2/0.0.html

o EPA Guidelines for Preparing Economic Analyses: 

http://yosemite.epa.gov/ee/epa/eed.nsf/webpages/Guidelines.html#download 

• EPA benefits analysis for PM 

o Regulatory Impact Analysis for Particle Pollution NAAQS (2006): 

http://www.epa.gov/ttn/ecas/ria.html

o PM Benefits chapter: 

http://www.epa.gov/ttn/ecas/regdata/RIAs/Chapter%205—Benefits.pdf

o PM Sensitivity analyses: 

http://www.epa.gov/ttn/ecas/regdata/RIAs/Appendix%20J—

Additional%20Benefits%20Sensitivity%20Analyses.pdf

• BenMAP model 

o General Introduction: 

http://www.epa.gov/air/benmap/

o Download version 3.0: 

http://www.epa.gov/air/benmap/download.html 

o BenMAP Manual: 

http://www.epa.gov/air/benmap/models/BenMapManualSept08.pdf

o BenMAP Manual Appendices: 

http://www.epa.gov/air/benmap/models/BenMAPappendicesSept08.pdf 
Table 1. List of Major Sources of Uncertainty (Parametric and Model) for Mortality in Different Simu

of the BenMAP Model BenMAP choice: Selected by user from choices in BenMAP BenMAP default: Defined in BenMAP, user can sometimes import different data to change assumption 

Input: Generated outside BenMAP 

Modeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMA

Population Data BenMAP choice Selection of forecast year Parameter Uncertainty, 

Forecast 

 BenMAP allows the user to select the analysis year. M

analyses use 2020. 

BenMAP default Estimating population 

levels in non-census years—Population levels between 2000 and 2030 

Parameter Uncertainty, 

Forecast 

 Using county-level population scale factors based on, 

age4-9,g,t=age4-9,g,2000×age4-9,county,t/age4-9,county,2000

For the gth population grid cell and for 2000<t<2030. 

forecasts for future years through 2030 based on a stuand Poole (2007). 

Input Estimating population 

levels in non-census years—Population levels after 2030 

Model Uncertainty, 

Forecast 

 Even if user imports a new population dataset, it must 

through Popgrid, which has this assumption built in 

 Linear extrapolation after 2030,e.g., 

age4-9,g,2035=age4-9,g,2030+5×(age4-9,2030-age4-9,2029) 

Input Population distribution 

within an age group 

Model Uncertainty, 

Functional form 

 Uniformly distributed within each age group, e.g., 

age3-12 = 1⁄2 × age1-4 + age5-9 + 3/5 × age10-14

Input Forecasts of population 

from Woods and Poole 

Parameter Uncertainty, 

Forecast 

 Woods and Poole forecasts population based on proje

conditions (county economies linked), natural increasdue to economy at the county level (confidence intervWoods and Poole ±6.3%for metropolitan statistical ar±4.7%for states) 

Input Assigning population data 

to grid squares 

Model Uncertainty, 

Functional form 

 Popgrid (a separate program) assigns 2000 Census blo

squares for 304 race/ethnicity/gender/age groups, inclpopulation growth weights. 
Table 1. List of Major Sources of Uncertainty (Parametric and Model) for Mortality in Different Simu

of the BenMAP Model (continued Modeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMA

ValuationBenMAP choice VSL estimate Parameter Uncertainty, 

Measured 

 Uses mean for Value of Statistical Life (VSL) of $5.5 

(2000$), with range from $1 million to $10 million. 

 Alternative distributions for the mean and range can b

BenMAP choice Income elasticity of WTP Parameter Uncertainty, 

Measured 

 Income elasticity estimates for VSL from literature, m

lower bound 0.08 and upper bound 1. 

Input Income growth per person Parameter Uncertainty, 

Forecast 

 Standard & Poor’s projections of future changes in Gr

Product (GDP) occurring after the year 2010 

 Divide the projected change in GDP by the Woods & 

change in total US population to produce an estimate GDP per capita 

Input Lag structure in mortality Model uncertainty, 

Functional form 

 EPA considers alternative lag structures for PM-relate

mortality (a) no lag effect 

(b) 8-year: incidences all occur in the 8th year follow

change in exposure 

(c) 15-year: incidences all occur in the 15th year follo

change in exposure 

(d) Alternative segments: 20% of incidence occur in 

50% in years 2 to 5, and 30% in years 6 to 20 

(e) 5-year distributed: 50% of incidences occur in ye

50% in years 2 to 5 

 Exponential: incidences occur at an exponentially dec

following year of change in exposure 
Table 1. List of Major Sources of Uncertainty (Parametric and Model) for Mortality in Different Simu

of the BenMAP Model (continued Modeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMA

Valuation (continuedBenMAP default Income elasticity and 

income growth adjustment to WTP 

Model Uncertainty, 

Functional form 



2 1 2 1

1 2 2 1

2 1

2 2 1 2 1 1 1 2 

2 1 1 1

2 2 2 1 1 2 1 1 2 

2 1 2 1

2 1 2 1

I I I II I I I

WTP WTP *

WTP *( I I I I ) WTP *( I I I 

I WTP I WTP

I WTP I WTP I WTP I WTP I W

(I I )*(WTP WTP )

(WTP WTP )*(I I )

I

I

WTP

WTP

ε − ε −

−

ε − ε −

−

=

ε − ε −

−

= ε − ε −

−

ε + ε − ε − ε =

−

+

−

+

=

Δ

Δ

ε =

BenMAP default Choice of baseline 

currency year for GDP and WTP 

Parameter Uncertainty, 

Measured 

 BenMAP uses year 2000 baseline 

BenMAP default Conversion of WTP and 

GDP to real dollars 

Parameter Uncertainty, 

Measured 

 BenMAP has inflation indices for all goods, medical c

using year 2000 baseline 

BenMAP choice Discount rate Parameter Uncertainty, 

Measured 

 EPA discounts benefits that occur after the analysis ye

7% 

 Applies when there is a lag between exposure and mo

C-R Function and Issues in the Estimation of Adverse Health Effects BenMAP choice Functional form of the 

C-R function 

Model Uncertainty, 

Functional form 

 Alternative functional forms are sometimes available 

C-R relationship including linear, log-linear, logistic, proportional hazard models 
Table 1. List of Major Sources of Uncertainty (Parametric and Model) for Mortality in Different Simu

of the BenMAP Model (continued Modeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMA

C-R Function and Issues in the Estimation of Adverse Health Effects (continued) BenMAP choice PM Coefficient (

β) in the 

C-R function 

Parameter Uncertainty, 

Measured 

 Each mortality study including expert opinion assume

representing uncertainty in 

β value and a mean value. 

distributions include normal, weibull, and triangular 

BenMAP choice Threshold in the C-R 

function 

Parameter Uncertainty, 

Measured 

 EPA estimates a point estimate threshold below which 

change in incidence rate associated with change in air concentration, e.g., for a log-linear C-R model: 

Δy=y0×[exp(

β×(max(PM1,T)-max(PM0,T)))-1] 

Where, T is the threshold 

BenMAP choice Select air quality metric in 

BenMAP to match metric in epi study 

Parameter Uncertainty, 

Measured 

For PM2.5, EPA currently uses C-R functions that use a qair quality concentration. BenMAP creates an annual averaveraging the 4 quarterly averages; quarterly averages are 8 hour average. 

BenMAP choice Conversion of air quality 

metrics reported in epi studies to common metric 

Parameter Uncertainty, 

Measured 

Currently only done for ozone, not PM 

BenMAP default Extrapolation beyond PM 

concentration in study 

Parameter Uncertainty, 

Extrapolate beyond 

range 

BenMAP makes no adjustment if PM levels resulting from outside the range of PM levels from the study 

BenMAP choice Current exposure and 

cumulative exposure 

Parameter Uncertainty, 

Measured 

Studies do not account for effect current exposure versus cexposure for current and lagged mortality 

BenMAP default Differential toxicity of PM 

components 

Parameter Uncertainty, 

Measured 

Assumes equal toxicity, no mechanism in BenMAP for adassumption 
Table 1. List of Major Sources of Uncertainty (Parametric and Model) for Mortality in Different Simu

of the BenMAP Model (continued Modeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMA

Incidence Rates BenMAP choice Mortality Baseline 

Incidence Rates 

Model Uncertainty, 

Functional form and 

aggregation 

 Age, cause, and county-specific mortality rates were o

CDC for years 1996 through 1998 by county 

 CDC’s age groups include <1, 1–4, 5–9, 10–14, 15–1

34, 35–44, 45–54, 55–64, 65–74, 75–84, and 85+ 

 Mortality rates are averaged across three years  It is assumed that rates are uniformly distributed acros

reported age group 

BenMAP choice Choice of incidence 

baseline year 

Parameter uncertainty, 

Measured 

 Age, cause, and county-specific mortality rates were o

CDC for years 1996 through 1998 by county 

 Mortality rates are averaged across three years 

BenMAP default Forecast baseline 

incidence rates 

Parameter Uncertainty, 

Forecast 

 Project CDC Wonder county and age-specific mortali

1997 to future years using Census projections of natiomortality rates from 1999 through 2050. Adjust countspecific 1997 CDC Wonder rate by ratio of Census pryear Y to the estimated Census age-specific mortality (1997 Census mortality rate estimated by regressing mfrom 1999–2008 on year). 

 All segments of the population assumed to grow at sa

Input Assigning baseline 

incidence to grid cells 

Model Uncertainty, 

Aggregation 

 Baseline incidence rates assigned to grid cells in BenM

how 

Air Pollution and Exposure Estimation AlgorithmBenMAP choice Assigning air pollution 

data to grid-cell population data, monitoring values versus actual exposure 

Model Uncertainty, 

Aggregation 

 People living within a particular air pollution model g

experience the same air pollution levels 
Table 1. List of Major Sources of Uncertainty (Parametric and Model) for Mortality in Different Simu

of the BenMAP Model (continued Modeling Step Uncertainty Source Uncertainty Type Current Mortality Formulation in BenMA

Air Pollution and Exposure Estimation Algorithm (continued)BenMAP choice Size of air quality grid 

cell/CMAQ grid cell size 

Parameter Uncertainty, 

Measured 

 Select grid cell size based on air quality inputs, typica

12 km, or smaller grid cell size 

BenMAP choice Monitor Rollbacks in 

BenMAP (in analysis of regulation, all monitors are assumed to meet the previous standard even if models forecast that they will not) 

Model Uncertainty, 

Functional form 

 Percentage rollback: monitor observations rolled back 

percentage for values over defined background level 

 Incremental rollback: monitors observations rolled ba

increment for values over defined background level 

 Rollback to standard: set values to determine if monit

monitors not in attainment rolled back to attainment leof several methods 

Input Monitor rollbacks to 

standard prior to BenMAP 

Model Uncertainty, 

Functional form 

 Monitor rollbacks conducted using air quality modelin

importing the data into BenMAP 

Input Air Quality Modeling 

for baseline and control strategy 

Parameter Uncertainty, 

Forecast 

 Various changes in emissions assumptions, inventorie

spatial allocation, control strategy, baselines, spatial adistributions, etc. 

Unquantified and non-monetized benefits

 WTP to avoid pain and 

suffering from the illness leading to death (not just the death itself) 

 Pain and suffering by 

family/friends (WTP by family and friends to prevent an individual’s illness) 
Values for Sensitivity Analysis: Population Estimates Reviewer Name:_____________________1 Estimating population levels in non-census years—Population levels between 2000 and 2030 

BenMAP default, Parameter Uncertainty, Forecast 

Analyze 

Quantitatively? 

Minimum & Maximum Values or Source for Values; 

Functional Forms 

Comments 

2 Estimating population levels in non-census years—Population levels after 2030 

Input, Model Uncertainty, Forecast 

Analyze 

Quantitatively? 

Alternative Functional Forms or Sources for Alternative 

Functions 

Comments 

3 Population distribution within an age group 

Input, Model Uncertainty, Functional form 

Analyze 

Quantitatively? 

Alternative Functional Forms or Sources for Alternative 

Functions 

Comments 
4 Forecasts of population from Woods and Poole Input, Parameter Uncertainty, Forecast Analyze Quantitatively? Minimum & Maximum Values or Source for Values Comments 5 Assigning population data to grid squares 

Input, Model Uncertainty, Functional form 

Analyze 

Quantitatively? 

Alternative Functional Forms or Sources for Alternative 

Functions 

Comments 

6 Selection of forecast year 

BenMAP choice, Parameter Uncertainty, Forecast 

Analyze 

Quantitatively? 

Minimum & Maximum Values or Source for Values Comments 
7 Analyze Quantitatively? Minimum & Maximum Values or Source for Values Comments 8 

Analyze 

Quantitatively? 

Minimum & Maximum Values or Source for Values Comments 

9 

Analyze 

Quantitatively? 

Minimum & Maximum Values or Source for Values Comments 
Values for Sensitivity Analysis: Mortality Valuation Reviewer Name:__________________

1 VSL estimate 

BenMAP choice, Parameter Uncertainty, Measured 

Analyze 

Quantitatively? 

Minimum & Maximum Values or Source for Values; 

Functional Forms Comments 

2 Income elasticity of WTP 

BenMAP choice, Parameter Uncertainty, Measured 

Analyze 

Quantitatively? 

Alternative Functional Forms or Sources for Alternative 

Functions Comments 

3 Income growth per person 

Input, Parameter Uncertainty, Forecast 

Analyze 

Quantitatively? 

Alternative Functional Forms or Sources for Alternative 

Functions Comments 
4 Lag structure in mortality Input, Model Uncertainty, Functional form Analyze Quantitatively? Minimum & Maximum Values or Source for Values Comments 5 Income elasticity and income growth adjustment to WTP 

BenMAP default, Model Uncertainty, Functional form 

Analyze 

Quantitatively? 

Alternative Functional Forms or Sources for Alternative 

Functions Comments 

6 Choice of baseline currency year for GDP and WTP 

BenMAP default, Parameter Uncertainty, Measured 

Analyze 

Quantitatively? Minimum & Maximum Values or Source for Values Comments 
7 Conversion of WTP and GDP to real dollars BenMAP default, Parameter Uncertainty, Measured Analyze Quantitatively? Minimum & Maximum Values or Source for Values Comments 8 Discount rate 

BenMAP choice, Parameter Uncertainty, Measured

Analyze 

Quantitatively? Minimum & Maximum Values or Source for Values Comments 

9 

Analyze 

Quantitatively? Minimum & Maximum Values or Source for Values Comments 

10 

Analyze 

Quantitatively? 

Alternative Functional Forms or Sources for Alternative 

Functions Comments 
11 Analyze Quantitatively? Minimum & Maximum Values or Source for Values Comments 12 

Analyze 

Quantitatively? Minimum & Maximum Values or Source for Values Comments 

13 

Analyze 

Quantitatively? Minimum & Maximum Values or Source for Values Comments 
Amy Lamson 

May 21, 2009 

Page 31 

Appendix C: Slides for Sessions 2 through 7 
Approaches to influence diagra

and sensitivity analysis for Ben

Max Henrion

Lumina Decision Systems

Los Gatos, CA

C l M fi ld d A i M kht i Carol Mansfi

eld and A

mir Mokhtari

RTI International

Research Triangle Park, NC

USA EPA, 

Research Triangle Park, NC

15th December 2008
Session 3:

Overview of the simplified 

influence analysis model an

project strategy

Max Henrion Max Henrion

Lumina Decision Systems

Los Gatos, CA
Obj pj ectives of the projec

• To identify the sources of uncertainty i

assessment of benefits from improvemein air quality using BenMAP

• To visualize how the sources of uncerta

affect the results using influence diagra

• To assess the range of values of key 

uncertainties 

• To compare their relative contributions 

the results using sensitivity analysis

• To identify which uncertainties are mo

need of more careful characterization
Overview of strawman propo

for the approach

• Identify and classify the uncertainties and the

effects

Develop an inventory of possible sources of uncertaClassify these sources in terms of source and type ouncertaintyDevelop influence diagrams to visualize their effecb fi ene ts

• Build model and conduct sensitivity analysis

Develop a simplified or reduced-form modelA f l ibl l f i Assess ranges

of 

plausibl

e values for parametric

uncertainties, and ranged of plausible alternative fmodel uncertaintiesConduct sensitivit

yy y anal

ysis to identify their relativ

importance on results using various measures of se

• Compare the simplified model with BenMAP f

selected scenarios to validate and calibrate it

• Make recommendations on which uncertaintie

most important to characterize in more detaihow
Example tornado diagram fo

sensitivity analysis

Any numbers and charts are purely illustrati
Classifyg 

in sources of uncertai

• Uncertainties or assumptions within BenMA

Choice: user can select value or option – e.g. forecfor analysis discount rate for analysis, discount rateInput: from input files – e.g. population projectionDefault: method not easily modifiableOmitted: issue not included in model – e g WTP t

e.g. WTP t

pain and suffering, separate from mortality

• Uncertainties that precede BenMAP – such a

Inputs and assumptions in emissions and air transpomodels, such as CMAQ, air quality measurements popul i j i h f W d & P l 

lation projections, such as from Woods & Pool

and US Census, by country, aggregated to 30km griPopGridWe examine the uncertainty p in these inputs to BenBut analysis of their sources of uncertainty is out oof this project. 
Classify g yp 

ing types of uncerta

• Parameter uncertainties

Measured () p estimated) current or past value.g. population in recent census, current ilevels, or incidence ratesForecast or projection of future values – efuture population or incomeUser choices - e.g. year for analysisP li i bl Policy variables – e.g. di t t VSL(?

 discount rate, VSL(?

• Model uncertainties

Functional form – e g for concentration e.g. for concentrationresponse function, or mortality lag from exAggregation – e.g. population from counti30km grid
What is an influence diagram

Arrows show influ

Decision between variables

Chancevariables

Module
What is Analytica?

• Transparency: Visual influence diagrams Analytica is a visual tool for building and deployianalytic applications. It offers:

make models easier to create, understand, and audit.

• Flexibilit

y: Intelli

g y ent Arra

ysTM make it 

easy to build and extend multidimensional models (data cubes)

• Scalability: Hierarchical modules, 

I t llig t A d lt In

t

elligent Arrays, and 

ultra-compact d co

d

e

let you manage and run models much larger than is practical with spreadsheets. 

• Risk analysis: Integrated Monte Carlo 

simulation enables fast evaluation of risk and uncertainty

“Everything that’s wrong with the 

spreadsheet is fixed in Analyti
Overview of BenMAP eleme

f B MAP U G id from BenMAP User G

uid

e
Influence diagram of BenMA

structure

Air quality

policy

change

Current air

qualit

y from

Future air

qualit

y from

Air quality

modelin

g

Census

q y

model

q y

model

g

data

Estimated

i li

Incidence and

prevalence 

population

data

air quality

Adverse

prevalence

rates 

Population

estimates

Population

exposure

Adverse

healtheffects

Populationprojections Health functions
Top level influence diagram

From strawman model
Influence diagrams for 

selected submodules
Why create a simplified mode

BenMAP?

• BenMAP is too large and takes too long 

perform a large number of runs for sensitivity analysis or Monte Carlo.

• For some kinds of uncertaint

y y ,

 anal

ysis, 

need to modify model structure to handependencies appopriately.

• To explore the effects of alternative 

functional forms or aggregation, it’s a easier to do on a simplified model

• N t Th i lifi d d l i t N

o

te: Th

e

simplifi

ed mo

d

el i

s not 

a

substitute for BenMAP: It could only be because BenMAP exists and is available calibrate it. It does not do mapping or analysis by grid square.
Binning grid square data 

to simplify the analysis

• We import dat

36k id 36k

m grid squa

PopulationCurrent PM2.5 concentration Delta concentcontrol

• We aggregate 

square data inbase concentra

• This sim

plifies 

16,576 grid squabout 16 bins - ~ 1000 times s
Results of binning

Population by base concentration binsDelta conc. by base conc. bins
Sensitivity to the level of 

aggregation: By number of bi

More concentration b d

ins means more 

d

and decreasing aggregation.
Sensitivity to the level of 

aggregation: By number of bi

Th l ld d h he sensitivity to aggregation level depen

ds on t

he 

threshold of C-R function. For zero threshold, nbins has no effect on results. (Illustrative numb
Where are we in this process? And 

are we hoping to get from this work

• We have an initial inventory of sources of 

uncert i t W t h l t id tif t

ain

ty: We request your h

elp t

o identify 

we’ve missed, and to review how they’re clasand characterized

• W h d l d t i fl di We have developed 

strawman i

nfluence diagr

for your comments.

• We have an initial simplified model capable o

illustrative sensitivity analysis We seek advicillustrative sensitivity analysis. We seek advichow to refine it.

• Ranges of parameter values: We request your 

in providing plausible ranges or suggesting hoin providing plausible ranges, or suggesting hoobtain them.

• Model form and aggregation uncertainties• W ld lik d i th t We would lik

e your

a

d

vice on th

e

strawman

proposed process for sensitivity analysis
Session 4:

P l ti E ti t : Population Estimates:

Identified sources of uncertainty 

population estimates and uncertai

introduced by BenMAP modelin

assumptions

Max Henrion

Lumina Decision Systems

L G t CA Los Gatos, CA
Draft influence diagram for 

population projections
Retrospective accuracy of 

population projections

Standard deviation error percent for Woods & Poole projec

population compared against census data. 

From 2008 Technical Description of Woods & Poole projections, table on page 20.
How should we estimate uncert

i US l ti j ti in US population projection

(age group 30 to 99)
Rang p es for uncertain paramet

• We want plausible low, mid, and high valu

not theoretically possible extreme min and max

• Think of them as 10th percentile, median a

90th percentile

alth h ’ t l ki f t b bil

lthough, we’re not looking for expert probabil

distributions for now.

• We’d like your estimates for the ranges

Or suggestions abh b h out how to obtain them

• For functional form, what set of functions 

should be considered?

• How can we estimate aggregation 

uncertainties? 

e.g. population from county to grid sqares
Session 4: Discussion on popula

esti t ma

tes

5min: Quiet brainstorming on these questions: • Are there additional sources of uncertainty in population forec

introduced by BenMAP’s use of the population data? 

• Do you think this will have big impact on benefits results? 15min: Report out and discussion 10min: Quiet brainstorming on these questions: • Can/should this source of uncertainty be modeled quantitative

i fl l i ? influence analysis? 

• What range of values should we use for these parameters or w

we get the range (i.e., from a study or existing data)? 

• Are there other functional forms or modeling approaches that 

sense f thi t f th d l h t ld b 

for thi

s component 

of th

e mo

d

el or

wh

at would b

e

a goo

for this information? 

30min Small group discussion • Where do we agree and where do we disagree? 45min Report out and discussion 
Session 5:

Valuation of mortality 

reduction

Identified sources of uncertainty i

valuation of mortality introduced by Be

Max Henrion

Lumina Decision Systems

Los Gatos, CA
Valuation influence diagra

Draft influence diagram 

for Valuation of 

statistical lives saved, includes

• Value of a statistical 

life

• Elasticity of WTP to 

income

• Income index

GDP growth ratePopulation growth rate

• Select forecast year• Inflation index and 

baseline currency year

• Discount due to 

mortality lag and lag structure
Mortality lag structures

Six lag types representing model 

uncertainty about lag between exposure and mortality:

• None (no lagg in mortality)

• 15-year: All deaths 

years after exposu

• 20-year distributed

• None (no lagg in mortality)

5 di t ib t d

• 8-year: All deaths happen 8 years 

after exposure

•

5-year di

st

rib

u

t

e

d:

• Exponential declin

years
Mortality g la structure

• Using an annual 

discount rate, we convert h l eac agstructure into a corresponding total di scount rate onbenefits
Effect of lag yp t e on benef
Rang p es for uncertain paramet

• We want plausible low, mid, and high valu

not theoretically possible extreme min and max

• Think of them as 10th percentile, median a

90th percentile

alth h ’ t l ki f t b bil

lthough, we’re not looking for expert probabil

distributions for now.

• We’d like your estimates for the ranges

Or suggestions abh b h out how to obtain them

• For functional form, what set of functions 

should be considered?

• How can we estimate aggregation 

uncertainties? 

e.g. population from county to grid sqares
Session 5: Discussion on valuat

of mortality reduction

5min Quiet brainstorming on: • Are there additional sources of uncertainty in valuation of mortal

reductions or introduced by BenMAP’s use of valuation studies? 

• Do you think these will have big impact on benefits results? 15min Report out and discussion

10min Quiet brainstorming on: • Can/should this source of uncertainty be modeled quantitatively 

influence analysis? 

• Wh f l h ld f h h What range

of values

should we use for

t

hese parameters or

wher

we get the range (i.e., from a study or existing data)? 

• Are there other functional forms or modeling approaches that ma

for this component of the model or what would be a good source information? 

30min Small group discussion• Where do we agree and where do we disagree? 45min Report out and discussion5:15 to 5:30: Wrap-up
S i 6 ession 6:

Valuation of morbidity

reduction

Identified sources of uncertainty i

valuation of morbidity introduced by Be
Valuation of morbidit

y
Session 6: Discussion on valuat

of morbidity reduction

5min Quiet brainstorming on: • Are there additional sources of uncertainty in valuation of mortal

reductions or introduced by BenMAP’s use of valuation studies? 

• Do you think these will have big impact on benefits results? 15min Report out and discussion

10min Quiet brainstorming on: • Can/should this source of uncertainty be modeled quantitatively 

influence analysis? 

• Wh f l h ld f h h What range

of values

should we use for

t

hese parameters or

wher

we get the range (i.e., from a study or existing data)? 

• Are there other functional forms or modeling approaches that ma

for this component of the model or what would be a good source information? 

30min Small group discussion• Where do we agree and where do we disagree? 15 mins Break ( ) 10:00am to 10:15am)30min Report out and discussion
Session 7: 

Di i d R d tDiscussion and Recommendat

on Approach to influence 

and uncertainty analysis

Max Henrion

Lumina Decision Systems
Objectives of the project:

Recap

• To identify the sources of uncertainty i

assessment of benefits from improvemein air quality using BenMAP

• To visualize how the sources of uncerta

affect the results using influence diagra

• To assess the range of values of key 

uncertainties 

• To compare their relative contributions 

the results using sensitivity analysis

• To identify which uncertainties are mo

need of more careful characterization
Overview of proposed approa

Recap

• Identify and classify the uncertainties and th

effects

Develop an inventory of possible sources of uncertClassify these sources in terms of source and type uncertaintyDevelop influence diagrams to visualize their effecb fi ene ts

• Build model and conduct sensitivity analysis

Develop a simplified or reduced-form modelA f l ibl l f i Assess ranges

of 

plausibl

e values for parametric

uncertainties, and ranged of plausible alternative for model uncertaintiesConduct sensitivit

yy y anal

ysis to identify their relativ

importance on results using various measures of se

• Compare the simplified model with BenMAP f

selected scenarios to validate and calibrate i

• Make recommendations on which uncertainti

most important to characterize in more detahow
Refinin

g p the sim

plified mod

• Concentration reductions

Initially, we assume that each all exposed population base concentration bin experiences the same reductbase concentration bin experiences the same reduct

PM2.5 concentration levels (averaged over the reducfor all grid squares in that bin)

Later, if time we could expand it to a bin array with a dimension of concentration reductions for each concentration bin, and see if this reduces the aggreguncertainty.

• Population segments Population segments

Initially, the model ignores population segments (sum age groups, ethnic groups, and gender). This is only relevant for CR-functions that are disaggb ( l l bidi li f by segment 

(currently, only morbidity not mortality 

fu

Later, if time, we might add a dimension of segments bins, with separate population growth rates and incidrates b

y g se

gment. 
Sensitivit

y y anal

ysis

• Tornado diagrams

We w ll co duct dete st c se s t v ty a alys s ill co

nduct deterministic se

n

sitivity a

nalysis (Tornado diagram) over all parameter and model 

uncertainties, varying each from low to high valuleaving all others at their mid value.

• For selected pairs or groups of uncertain va

where knowledge of the functional form leads us suspect that interactions may be important and aone parameter is individually important, we will cond l i i i i h i ( 

duct multiway sensitivity,

c

hanging two (or ra

more) parameters at a time.

• We may also use probabilistic sensitivity an

measures such as rank correlations and measures, such as rank correlations and regression measures 

assuming a simple uniform or beta distributions op g arameter ran

ges 

and uniform discrete distributions over alternativfunctional forms
Examp g le tornado diagram
Example “importance analysis

compare input uncertaintie

The relative importance of each uncertain input to the be

computed as the rank correlation of the Monte Carlo safor the benefits with respect to each input sample
Iterative refinement of model 

sensiti it l iti

vit

y analyses

• Based on the initial anal

y , sis

we will identify a subset of parameas the most important contributors develop more careful characterizatof their uncertainty, by reviewing relevant literature and/or more extended conversation with one or experts on those quantities to refinlow, mid, and high values.

• We will repeat the sensitivity 

anal i th fi d ti 

lyses using th

e refined estima
Proposed comparison of simplif

model with BenMAP scenario

• Identify the most important set of parameters 

model forms (p p ), 

erha

ps 5 to 8), from initial sen

analysis with the simplified model

• Define 5 to 10 scenarios -- i.e. combinations o

values of the 

parameters 

– to 

p 

erform sensitiv

of BenMAP. 

The number of scenarios will depend on how many runs are practical given human and computational re

• C th lt f B MAP ith th i 

Compare the results 

from 

BenMAP with the sim

model 

Direct values, including incidence, mortality reductiobenefitsSensitivities: Relative size and ranking of sensitivitieDiagnose the source of any differences

• If time we may refine the simplified model to to If time, we may refine the simplified model to to

improve correspondence and rerun the sensiti
Results and recommendations f

the project

• Summarize the results, identifying th

which sources of uncertainty are moscritical to uncertainty in the results (which are less so

)

• Recommend how those uncertainties 

be refined, e.g.

By cond l f b 

ducting expert elicitations o

f pro

b

distributions on uncertain quantities 

Ex

periments on effects of alternative 

aggregationsObtaining other sources of data or runs opreceding models
Discussion
Related models that provide in

to BenMAP

• Sensitivity analysis for CMAQ or other models are outside the 

thi j hi h f l B MAP hi

s project,

whi

ch focuses only on BenMAP.

• Although most models use population projections as shown in 

diagram, they currently do not necessarily use the same proj
Appendix D: Influence Diagrams 
Influence diagrams to suppor

sensitivity and uncertainty ana

of BenMAP

Max Henrion

Lumina Decision Systems

Los Gatos, CA
Top level influence diagram

to represent BenMap structu
Influence diagrams for 

selected submodules
Draft influence diagram for 

population projections
Valuation of mortality 

influence diagram

Draft influence diagram 

for Valuation of 

statistical lives saved, includes

• Value of a statistical 

life

• Elasticity of WTP to 

income

• Income index

GDP growth ratePopulation growth rate

• Select forecast year• Inflation index and 

baseline currency year

• Discount due to 

mortality lag and lag structure
Valuation of morbidit

y
Related models that provide in

to BenMAP

• Sensitivity analysis for CMAQ or other models are outside the 

thi j hi h f l B MAP hi

s project,

whi

ch focuses only on BenMAP.

• Although most models use population projections as shown in 

diagram, they currently do not necessarily use the same proj