UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

WASHINGTON D.C. 20460 

OFFICE OF THE ADMINISTRATOR 

SCIENCE ADVISORY BOARD 

June 8, 2007 

EPA-COUNCIL-07-002 

Honorable Stephen L. Johnson 

Administrator 

U.S. Environmental Protection Agency 

1200 Pennsylvania Avenue, N.W. 

Washington, DC 20460 

Subject: Benefits and Costs of Clean Air Act – Direct Costs and Uncertainty Analysis 

Dear Administrator Johnson: 

EPA’s Office of Air and Radiation (OAR) is developing the Agency’s “Benefits and 

Costs of the Clean Air Act 1990 - 2020: EPA's Second Prospective Analysis.” This study is 

designed to estimate the net health, welfare, ecological and economic benefits of all 1990 

Clean Air Act Amendment programs. As part of this effort, OAR has recently submitted to 

the Advisory Council on Clean Air Compliance Analysis (Council) its draft Direct Cost 

Report as well as its draft Uncertainty Analysis Plan. In February 2007, OAR requested input 

from the Council on these draft products. The Council met in a face-to-face meeting on 

March 15-16, 2007, to discuss the OAR’s charge questions related to these reports. The 

Council generally endorses OAR’s approach and results to date. 

With respect to OAR’s Direct Cost Report, the Council notes that two types of cost 

estimates are required – partial equilibrium social costs that are frequently reported as the cost 

of regulation, and private costs that are input to the computable general equilibrium (CGE) 

model OAR will use to estimate general equilibrium costs and other effects of the Clean Air 

Act Amendment programs. The Council encourages OAR to distinguish and report separately 

the two types of costs (when they differ significantly). 

With respect to OAR’s Uncertainty Analysis Plan, the Council commends OAR for its 

intention to use several types of graphical display to communicate the uncertainty of its 

estimates. The Council recommends that OAR be vigilant in reporting what uncertainties are 

incorporated in a probability distribution and on what other parameter values these 

distributions are conditioned (e.g., uncertainties in health effects and valuation may be 

represented as probability distributions conditional on an assumed economic growth 
scenario). The Council cautions OAR against presenting scenarios or sensitivity analysis 

without some statement of the probabilities associated with the alternative scenarios or 

parameter values. Readers will understandably conclude that the values presented encompass 

a relevant range of possibilities, and communication should be improved if OAR provides 

some guidance concerning where the presented values lie in an overall probability 

distribution. 

Detailed recommendations are included in the enclosed Advisory. On behalf of the 

entire Council, we appreciate this opportunity to provide timely advice to the Agency. We 

hope these comments are helpful to the Office of Air and Radiation as it proceeds with this 

important work. 

 Sincerely, 

/Signed/ 

 James K. Hammitt, Chair 

Advisory Council on Clean Air Compliance Analysis 

Enclosure 

ii 
NOTICE 

This report has been written as part of the activities of the U.S. Environmental 

Protection Agency’s Advisory Council on Clean Air Compliance Analysis (Council), a 

federal advisory committee administratively located under the EPA Science Advisory Board 

(SAB) Staff Office. The Council is chartered to provide extramural scientific information and 

advice to the Administrator and other officials of the EPA. The Council is structured to 

provide balanced, expert assessment of scientific matters related to issue and problems facing 

the Agency. This report has not been reviewed for approval by the Agency and, hence, the 

contents of this report do not necessarily represent the views and policies of the EPA, nor of 

other agencies in the Executive Branch of the Federal government, nor does mention of trade 

names or commercial products constitute a recommendation for use. Council reports are 

posted on the SAB Web site at: http://www.epa.gov/sab. 

iii 
U.S. Environmental Protection Agency 

Advisory Council on Clean Air Compliance Analysis 

(COUNCIL) 

CHAIR 

Dr. James K. Hammitt, Professor of Economics and Decision Sciences, Harvard Center for 

Risk Analysis, Harvard University, Boston, MA 

COUNCIL MEMBERS 

Dr. David T. Allen, Gertz Regents Professor in Chemical Engineering, Department of 

Chemical Engineering, University of Texas, Austin, TX 

Dr. Dallas Burtraw, Senior Fellow, Resources for the Future, Washington, DC 

Dr. Trudy Ann Cameron, Raymond F. Mikesell Professor of Environmental and Resource 

Economics, Department of Economics, University of Oregon, Eugene, OR 

Dr. Charles T. Driscoll, Jr., Professor, Department of Civil and Environmental Engineering, 

College of Engineering and Computer Science, Syracuse University, Syracuse, NY 

Dr. Shelby Gerking, Professor of Economics, Department of Economics, College of 

Business Administration, University of Central Florida, Orlando, FL 

Dr. Wayne Gray, Professor, Department of Economics, Clark University, Worcester, MA 

Dr. F. Reed Johnson, Senior Fellow and Principal Economist, RTI Health Solutions, 

Research Triangle Institute, Research Triangle Park, NC 

Dr. Katherine Kiel, Associate Professor, Department of Economics, College of the Holy 

Cross, Worcester, MA 

Dr. Nino Kuenzli, Associate Professor, Department of Preventive Medicine, Keck School of 

Medicine, University of Southern California, Los Angeles, CA 

Dr. Virginia McConnell, Senior Fellow and Professor of Economics, Resources for the 

Future, Washington, DC 

Dr. Bart Ostro, Chief, Air Pollution Epidemiology Unit, Office of Environmental Health 

Hazard Assessment, California Environmental Protection Agency, Oakland, CA 

iv 
Dr. David Popp, Associate Professor of Public Administration, Center for Policy Research, 

The Maxwell School, Syracuse University, Syracuse, NY 

Dr. Chris Walcek, Senior Research Scientist, Atmospheric Sciences Research Center, State 

University of New York, Albany, NY 

SCIENCE ADVISORY BOARD STAFF 

Dr. Holly Stallworth, Designated Federal Officer, Environmental Protection Agency, 

Science Advisory Board Staff Office, Washington, DC 

v 
Charge Questions from the Office of Air and Radiation 

for the Advisory Council on Clean Air Compliance Analysis 

Meeting of March 15-16, 2007 

1. Direct Cost Report – General Review. EPA requests that the Council review the draft 

Section 812 Second Prospective Study direct cost report. Consistent with the statutory 

language defining the role of the Council in reviewing the 812 studies, EPA respectfully 

submits the following general charge questions: 

a. Does the Council support the data choices made by the 812 Project Team for the 

development of the draft direct cost report? If not, are there alternative data sets the 

Council recommends should be applied instead? 

b. Does the Council support the methodological choices made for analyzing those data 

and developing the direct cost estimates for the relevant scenarios? If not, are there 

alternative methodologies the Council recommends should be applied instead? 

c. What advice does the Council have for the EPA Administrator with respect to the 

validity, reliability, and utility of the draft direct cost report and the analytical results 

presented therein? If the validity, reliability, and/or utility of the draft direct cost 

report could be improved, what specific steps does the Council recommend the 812 

Project Team pursue to accomplish these improvements? 

2. Direct Cost Report – Specific Issues. The general charge question #1 covers any and all 

aspects of the draft direct cost report which the Council might consider appropriate to 

address in its review. In conducting this review, EPA also respectfully requests that the 

Council consider the following particular issues during its review. These particular topic 

areas represent specific analytical choices or outcomes for which the 812 Project Team is 

especially interested in Council advice. 

a. Unidentified Measures. Currently available air pollution control technologies do not 

appear to be sufficient to achieve full attainment with the current PM and/or ozone 

NAAQS. The first prospective 812 analysis adopted an approach where additional, 

“unidentified measures” were built into the with-CAAA scenario to achieve the 

future emissions reductions needed to progress toward full attainment. The costs of 

those unidentified measures were capped at $10,000 per ton. The current draft cost 

report again applies a fixed $10,000 per ton cap on precursor emissions. However, 

other options have been identified by the 812 Project Team for estimating the costs 

of unidentified measures. Three basic options (plus some variants) are described in a 

white paper accompanying the draft direct cost report. The Agency requests that the 

Council provide advice regarding the relative merits of these alternative approaches. 

In addition, EPA respectfully requests that the Council consider the issue of reliance 

on unidentified measures more broadly, and provide advice regarding the 

appropriateness of providing qualitative discussion, caveats in the characterization of 

results, and/or uncertainty analysis associated with the projection of future control 

technologies and costs. 

1
b. Transferability of Learning Effects. The draft cost report applies learning effects 

adjustments to a limited number of source categories and control technologies where 

empirical data are available to specify a learning effect rate. This limited application 

is consistent with prior Council advice to replace a generic 80% learning effect rate 

with more source-specific rates based on available empirical data. Two of the key 

applications of learning rates in the current draft cost report are for Selective 

Catalytic Reduction (SCR) and Selective Non-Catalytic Reduction (SNCR) at 

Electric Generating Units (EGUs). However, SCR and SNCR technologies are also 

applied to some non-EGU sources, particularly industrial boilers. The current draft 

cost report, however, does not apply learning effect adjustments to non-EGU 

applications of SCR and SNCR. The Agency therefore requests that the Council 

consider providing advice regarding the appropriateness of transferring learning 

effects adjustments for a given control technology from one source category to 

another. 

c. Discount Rate Selection and Implementation. The current draft cost report uses a 5 

percent social discount rate for the primary analysis, consistent with prior Council 

advice to adopt a central discount rate value for the primary analysis, supplemented 

by a sensitivity analysis using the 3 percent and 7 percent rates prescribed by 

prevailing guidelines. The Agency requests that the Council consider providing 

advice pertaining to the reasonableness of the choice of a 5 percent rate for the 

primary analysis, as well as the appropriateness of the 812 Project Team’s 

application of this rate in the derivation of sector-specific direct cost estimates. 

d. Treatment of Manufacturer Profit in Social Costing. Some of the direct compliance 

cost estimates incorporated in the draft direct cost report are based on projected 

changes in manufacturer cost (including production costs, distribution costs, 

overhead, etc) and some estimates are based on projected changes in retail prices for 

consumers. The essential difference between these two costing approaches is the 

inclusion or exclusion of markups for producer profit. The Agency requests that the 

Council review the white paper on the cost markup issue and consider providing 

advice pertaining to the appropriate treatment of producer profit in estimates of the 

social cost of compliance. 

3. Uncertainty Analysis Plan. Consistent with advice from several 812 Council advisories 

and the 2002 NAS report, the 812 Project Team has developed an expanded uncertainty 

analysis plan, which the team has described in a white paper. This updated plan 

represents a significant expansion of the original uncertainty analysis plans contained in 

the May 2003 analytical blueprint. Consistent with the statutory language defining the 

role of the Council in reviewing the 812 studies, EPA respectfully submits the following 

charge questions: 

a. Does the Council support the data choices made by the 812 Project Team for the 

development of the 812 study uncertainty analysis, as described in the uncertainty 

analysis plan white paper? If not, are there alternative data sets the Council 

recommends should be applied instead? 

2
b. Does the Council support the methodological choices made for analyzing uncertainty 

in the 812 analysis, as described in the uncertainty analysis plan white paper? If not, 

are there alternative methodologies the Council recommends should be applied 

instead? 

# # # 

3
Council Advisory 

on 

OAR’s Direct Cost Report and Uncertainty Analysis Plan 

The following comments constitute the Council’s response to the charge questions submitted 

by the Office of Air and Radiation in February of 2007 concerning its draft “Direct Cost Estimates 

for the Clean Air Act Second Section 812 Prospective Analysis” (Direct Cost Report) and its draft 

paper entitled “Estimating Uncertainty in the Second Prospective Analysis of the Benefits and Costs 

of the Clean Air Act” (Uncertainty Plan). Dated February 2007, both of these draft papers are 

interim products developed for the “Benefits and Costs of the Clean Air Act 1990 - 2020: EPA's 

Second Prospective Analysis” (Second Prospective) and are posted at: 

http://www.epa.gov/oar/sect812/prospective2.html#mar07. 

Direct Cost Report – General Comments 

Overall, the Council endorses the methods and data incorporated in the Direct Cost report. 

We note one significant conceptual issue – the difference between conventional partial-equilibrium 

social costs of regulation and the private costs needed as input to the computable general 

equilibrium (CGE) model. This issue is discussed in response 1.c., which is addressed first. 

Afterward, the Council offers some comments on data and methodological choices in responses 1.a. 

and 1.b. 

Validity, reliability and utility of the report: Partial- and general-equilibrium cost estimates (See 

charge question 1.c.) 

The Council notes that two types of costs are relevant to the analysis for the Second 

Prospective Report. First, private costs are needed as input to the computable general equilibrium 

(CGE) model that will be used to estimate general-equilibrium effects of Clean Air Act Amendment 

(CAAA) regulations. Second, conventional partial-equilibrium estimates of social costs of 

regulations are of interest for comparison with the First Prospective Report and other estimates of 

the costs of environmental regulation. The two types of costs differ in how transfer payments, such 

as taxes and firm profits, are incorporated. When there are significant differences between them, the 

Direct Cost report should report both. For social welfare analysis, the social costs are of primary 

interest and so, to minimize reader confusion, it may be appropriate to highlight the social costs and 

report the private costs used as input to the CGE in an appendix. 

In addition, the Council identifies a potential inconsistency in the analysis if the generalequilibrium effects captured by the CGE model are important. The Direct Cost Report starts with 

output predictions from the Annual Energy Outlook (AEO) model, calculates the magnitude of 

emission reductions needed to meet a variety of regulatory constraints, and concludes with the 

increases in production cost resulting from emission reductions. These cost increases are calculated 

by industry and region and are used as input to the CGE model. The CGE model simulates the 

effects of these costs on industry and regional output, which influence regional emissions and may 

affect regional control costs. 

4
To illustrate, suppose that the Direct Cost Report finds that paper production in the South 

becomes much more costly due to local regulatory constraints. The CGE model will predict 

reductions in demand for and production of Southern paper, offset in part by increased production of 

Northern paper and non-paper goods. If these general-equilibrium shifts in production are important, 

the resulting distribution of industry production will diverge from that assumed in the AEO, which 

should in principle feed back into the emissions constraints: less Southern paper production means 

fewer emissions in the South, reducing the cost of meeting regulatory constraints there (for paper 

and other industries); increased Northern paper production raises abatement costs in the North, again 

for paper and non-paper industries. Note that the full impact of increased costs on production levels 

may not be captured for several years in the CGE model, given lags in capital investment. Although 

resource constraints may preclude a complete recalculation of the direct cost model using the CGE 

output, the Council recommends that the Project Team compare the CGE and AEO outputs to 

identify any industry-region cases that are especially sensitive to general-equilibrium effects. 

Data choices (See charge question 1.a.) 

The Council generally supports the data and methodological choices made by the 812 

Project Team. However, some of the assumptions about vehicle fleets and vehicle costs might be 

looked at more closely. 

The assumed failure rate for inspection and maintenance (I&M) programs appears to be a 

constant 14 percent. There is evidence that failure rates are lower for vehicles with on-board 

diagnostic (OBD) systems, particularly those meeting recent standards (OBD-II), and remain lower 

over the life of the vehicle (see National Research Council 2001, Evaluating Vehicle Emissions 

Inspection and Maintenance Programs, http://books.nap.edu/catalog.php?record_id=10133). Other 

I&M cost assumptions (e.g., time in line, value of time) also appear to be on the high end. On the 

other hand, the emissions reductions may also be overestimated. There is evidence that the 

emissions reductions assumed for I&M programs in the Mobile model may be too large (NRC, 

2001). Cost offsets due to fuel savings are estimated using cost per gallon of $1.11 (p. 3-5) and 

$1.14 (p. 3-16), which seem low. Given high excise taxes, the difference between private and social 

fuel costs may be significant. 

With respect to estimating the cost of meeting future motor-vehicle emission standards, the 

Project Team may consider evidence about the ex ante and ex post cost estimates for meeting new 

car standards in the recent past. The California Air Resources Board estimates for meeting the low 

emissions vehicle (LEV) standard showed that the ex post costs appear to be slightly lower than ex 

ante predicted costs for smaller vehicles; for larger vehicles, the ex post costs were slightly higher 

(Presentation by Tom Cackette, 1998, cited in National Research Council 2006, State and Federal 

Standards for Mobile Source Emissions, http://books.nap.edu/catalog.php?record_id=11586). 

Similarly, it appears that California estimates of the costs of achieving zero emissions vehicles were 

too optimistic (National Research Council, 2006). 

Not all costs of compliance with CAAA regulations are incorporated in markets. For 

example, the shift from two-cycle to four-cycle and fuel-injected gasoline engines for recreational 

equipment (e.g., marine outboards, snowmobiles) may produce non-market costs in the form of 

higher weight for these components, reducing ease of use and transport. 

5
Methodological choices (See charge question 1.b.) 

The application of the CGE modeling should reflect recent developments in the literature 

that have moved beyond the general finding of a tax-interaction effect (and associated hidden costs) 

associated with regulatory programs to the recognition that the magnitude of the effect differs by 

sector and context. The recent finding that the tax-interaction effect is much reduced when there are 

quasi-fixed factors of production (Bento and Jacobsen, “Ricardian rents, environmental policy and 

the ‘double-dividend’ hypothesis,” Journal of Environmental Economics and Management 53(1): 

17-31, 2007) appears to be especially relevant to the electricity sector. There is even the possibility 

of a double dividend in this case. 

In comparing current and previous cost estimates (p. 1-8 of Direct Cost Report and 

elsewhere) it is important to distinguish between regulations or regulatory goals considered in the 

previous estimates and additional regulations not considered in the earlier estimates, such as new 

regulations on hazardous air pollutants (HAPs). 

To augment the Integrated Planning Model (IPM) estimates to account for sunk investments 

(p. 2-6), the Project Team might consider comparisons with previous runs of IPM and its 

predecessor. There have been numerous regulatory impact analyses (RIAs) and other reports that 

include information about the cost of investments going forward. If such an estimate appears in a 

1995 RIA, then when looking back from the next vantage point one could take the cost forecast of 

the 1995 report as one measure of the capital costs incurred after 1995. 

Specific Comments on Direct Cost Report 

Unidentified Measures (See charge question 2.a.) 

The Project Team has been unable to identify measures that yield sufficient emission 

reductions to comply with the National Ambient Air Quality Standards (NAAQS) and relies on 

unidentified pollution control measures to make up the difference. Emission reductions attributed to 

unidentified measures appear to account for a large share of emission reductions required for a few 

large metropolitan areas but a relatively small share of emission reductions in other locations and 

nationwide. 

The Council agrees with the Project Team that there is little credibility and hence limited 

value to assigning costs to these unidentified measures. It suggests taking great care in reporting 

cost estimates in cases where unidentified measures account for a significant share of emission 

reductions. At a minimum, the components of the total cost associated with identified and 

unidentified measures should be clearly distinguished. In some cases, it may be preferable to not 

quantify the costs of unidentified measures and to simply report the quantity and share of emissions 

reductions attributed to these measures. 

When assigning costs to unidentified measures, the Council suggests that a simple, 

transparent method that is sensitive to the degree of uncertainty about these costs is best. Of the 

three approaches outlined, assuming a fixed cost/ton appears to be the simplest and most 

6
straightforward. Uncertainty might be represented using alternative fixed costs per ton of emissions 

avoided. 

Transferability of Learning Effects (See charge question 2.b.) 

In previous guidance, the Council suggested that the Project Team use sector-specific 

empirical data on learning effects whenever possible in its cost analysis. The Project Team has 

followed this advice and the draft Direct Cost Report describes a number of sector-specific estimates 

of learning effects. For a number of sectors, however, no empirical data are available. These data 

gaps include sectors, such as non-EGU point source control and costs of producing Clean Air Actmandated fuels, where costs are substantial. 

The Council commends the Project Team for its work in collecting and evaluating sectorspecific learning-curve data and suggests that the Project Team continue to apply empirical data in 

sectors for which such data are available. In sectors for which no data are available, several options 

are available, as summarized below: 

1. Where data gaps exist, attempt to identify sectors, for which data are available, that have 

similar anticipated learning patterns 

2. Assume that a default learning effects apply to sectors for which no empirical data are 

available 

3. Assume no learning effects occur in sector for which no data are available. 

The Council suggests that the Project Team assume that some learning occurs in all sectors, 

even when direct estimates for a specific sector are not available (e.g. non EGU SCR, or Clean Air 

Act mandated fuels). Based on the range of empirical data reported by the Project Team, the Council 

suggests using a consistent rate of 90% or 95% (option 2 above). Assuming a single default learning 

effect for sectors for which empirical data do not exist, rather than attempting to estimate rates for 

each sector, improves the transparency of the methodology. Moreover, several of the cited learning 

curve studies are old (and may not reflect current rates), and some of the observed variation in 

empirical estimates by sector may reflect random variation rather than systematic sectoral 

differences, and so using a common rate across sectors may be more accurate. 

Because learning effects are uncertain, the Council suggests that the Project Team conduct 

one or more sensitivity analyses assuming different default learning rates. One logical sensitivity 

analysis to perform would be to assume that no learning effects occur (option 3 above). 

The Council also suggests that the Project Team consider whether there are learning costs 

that should be incorporated. “Learning” results not only from experience, but also from factors such 

as economies of scale and research and development. If learning costs are not explicitly included in 

the analysis, the Project Team may wish to assume conservative values for learning rates so as to 

avoid overstating the net benefits of learning. The Council also suggests that the Project Team 

consider how cost and quality are related in estimating learning effects. Learning curve studies 

typically hold quality constant. More stringent CAAA regulations will require better technologies 

(e.g. higher removal efficiencies). The Project Team may consider how costs in the model are 

related to the quality of emissions abatement technology. 

7
Discount Rate Selection and Implementation (See charge question 2.c.) 

As discussed above, two types of costs are relevant to the study, private costs that are used as 

input to the CGE model and partial-equilibrium estimates of social costs. Private costs should be 

calculated using private discount rates and social costs should be calculated using social discount 

rates. 

For a social discount rate, the Council agrees that 5 percent is reasonable given the many 

uncertainties involved in choice of discount rate and the modest time horizon (a few decades) of 

concern in this analysis. In a very simple economic model, there would be only a single interest rate 

that would reflect the equilibrium rate of substitution between consumption in different periods. In 

practice, rates of return to private investment and savings differ because of taxes and other factors. 

The analytically preferred method for handling this difference is to determine the extent to which 

regulatory costs displace investment and consumption, to convert the displaced investment to 

forgone future consumption increments (i.e., by adjusting for the shadow price of investment), and 

to discount displaced and forgone consumption at the consumption discount rate. However, this 

analytically preferred method appears to be rarely practical because of difficulties in determining the 

extent to which regulations displace investment and consumption and uncertainty about the shadow 

price of capital. Moreover, a 5 percent rate is intermediate to typical estimates of the rate of return to 

private investment and the rate of return to savings (e.g., OMB Circulars A-94 and A-4 suggest 

values of 7 and 3 percent, respectively). 

For private costs used as input to the CGE model, the relevant discount rate should be the 

private cost of investment funds, which may be somewhat higher than 5 percent. This suggests that 

there is no need to adjust IPM’s annual capital costs to reflect the 5 percent social discount rate (see 

p. 2-9 of the Direct Cost Report). The IPM interest rate and useful life assumptions in Exhibit 2-2 

appear reasonable and are appropriate for the purpose of obtaining cost estimates to use as inputs to 

the computable general equilibrium analysis. 

Treatment of Manufacturer Profit in Social Costing (See charge question 2.d.)

 Appropriate treatment of markups in the analysis is related to the distinction between social 

and private/consumer costs. Markups reflecting pure economic profit are a transfer from consumers 

to producers, and as such should not count towards social costs. However, for an industry to 

generate sizable economic profits in the long run, the firms in that industry need considerable 

market power, which does not seem to be true of the industries in question (e.g., motor vehicle 

producers). Any observed markups probably reflect unobserved opportunity costs of production, not 

economic profit, and should be included in social cost. 

In some cases, other costs that could be considered a transfer (e.g., property taxes) may 

alternatively reflect real resource costs to society (e.g., higher taxes may be incurred when a firm 

adds capital equipment and requires more local government resources for police or fire protection). 

Note that, if there are measurable differences between social and private costs (e.g., because 

of taxes), the Direct Cost Report should calculate both types of costs. The social costs are useful for 

8
comparison to other partial equilibrium numbers, and the private costs are needed as inputs for the 

CGE model. Taxes and other transfers may distort behavior and hence create welfare losses that 

should be captured in a CGE analysis. 

Comments on OAR’s Uncertainty Analysis Plan 

The current Uncertainty Plan includes analyses beyond those proposed in the May 2003 

analytical blueprint. In addition to the primary forecasts of costs and benefits based on central 

economic and population growth assumptions, scenarios corresponding to low and high economic 

and population growth will also be developed. The other major addition to the Uncertainty Plan is 

the inclusion of the results of the expert elicitation study on particulate matter (PM) mortality 

concentration-response functions that can provide an alternative to current uncertainty analysis 

through results from epidemiology studies. Several offline uncertainty analyses are also described. 

The Council concludes that scenario analyses with high and low growth forecasts are a 

reasonable set of results to display. It recommends clearly acknowledging that there are no 

behavioral responses in the estimates to changes in these growth estimates. For example, if incomes 

are lower, the size distribution of the vehicle fleet could shift (toward older and smaller cars) with 

implications for overall emissions. This type of change would not be included in the analysis as 

proposed. 

There is a sense in the plan that uncertainty about benefits is much greater than uncertainty 

about costs. While the Council concurs with this judgment, it fears that uncertainty about costs may 

be underemphasized; as the Project Team recognizes, factors such as uncertainty about unidentified 

control measures, learning effects, and technological change may contribute substantially to cost 

uncertainty. There is direct evidence about historical expenditures and other costs, from which one 

might be able to derive a range of costs for at least some current technologies. Future costs are 

sensitive to learning, technology change, input substitution, and other factors. This uncertainty 

might be best examined through some type of alternative assumptions in a scenario analysis. 

The Council commends the Project Team for proposing to use a variety of visual displays 

that can help to communicate uncertainty to readers (e.g., box-and-whisker plots, cumulative 

distribution functions). As these displays make clear, characterizing uncertainty about study results 

requires estimates of both the possible outcomes (e.g., benefits, costs) and their probabilities. 

Similarly, the proposed break-even analysis requires estimates of the probability of scenarios 

occurring under which net benefits of CAAA regulations are negative. Ideally, the Project Team 

could develop probability distributions to characterize uncertainty about each of the important input 

variables and choices (including, inter alia, model selection) and propagate these uncertainties 

through the suite of simulation models to derive a multivariate probability distribution over output 

variables. 

Such an effort is unrealistic because resource constraints preclude a sufficient number of 

model runs. A more fundamental limitation, however, may be the lack of widely-accepted 

procedures for constructing probability distributions to characterize uncertainty about inputs (note 

these must be multivariate distributions to account for dependence among input variables). In 

response to these limitations, the Project Team reasonably proposes to conduct a mixture of scenario 

9
and sensitivity analyses (estimating outcomes conditional on specified input values with no 

quantification of the likelihood that those input values are “correct”) and uncertainty analyses in 

which some uncertainties about input values are characterized as probability distributions, 

propagated through appropriate simulation models, and reported as a probability distribution on 

outputs. 

A mixed approach to uncertainty analysis appears to be the best that can be achieved. 

Implementation requires many choices about which uncertainties to represent as probabilities and 

which to address through scenario or sensitivity analysis; these should be made carefully, with 

attention to how important the uncertainty is to overall uncertainty about the outputs and to the 

theoretical and evidentiary basis for estimating probabilities of alternative scenarios or input values. 

Similarly, many choices are required about which uncertainties are sufficiently important to justify 

including displays showing their effects on outcomes. 

In reporting uncertainty, the Project Team should recognize that probabilities are always 

conditional on some maintained assumptions (e.g., about the appropriate fate and transport model or 

economic growth scenario) and be careful to make clear what uncertainties are incorporated in a 

distribution and on what other parameter values or other assumptions these distributions are 

conditioned. While it is easy to report multiple results based on alternative input assumptions and to 

declare that one cannot say anything about the relative likelihoods of the results, this approach can 

be misleading. The simple reporting of a possible outcome may lead readers to infer that the 

outcome is likely enough that it merits attention (if not, why report it?). Readers may reasonably 

infer that the range of outcomes that is presented spans some probabilistically relevant subset of all 

possible outcomes. Moreover, a range gives no information about which parts are more or less likely 

– implicitly it may suggest that all values between the ends of the range are equally likely, even 

when values near the center are much more likely than values near the endpoints. 

In choosing multiple scenarios and other parameter values, analysts must be wary of the 

problem of compounding conservatism. This arises when constructing extreme outcomes by 

combining uncertainties about multiple inputs. For example, if outcome z is the product of three 

probabilistically independent factors w, x, and y, then multiplying the 0.01 fractiles for each of the 

three inputs yields the one-in-a-million fractile for z. 

Analyzing uncertainty and communicating the results to decision makers is inherently 

complex. The Council commends the Project Team for its plan to rely on more informative 

graphical displays and cautions the team to avoid simply showing decision makers a range of 

values based on sensitivity analysis. Communication may be improved if the Project Team provides 

guidance about where the outcomes that are presented are likely to lie in an overall probability 

distribution. 

10