a

GAO

United States Government Accountability Office

Report to Congressional Requesters

July 2006 PARTICULATE 

MATTER

EPA Has Started to 

Address the National 

Academies’ 

Recommendations on 

Estimating Health 

Benefits, but More 

Progress Is Needed

GAO-06-780
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

July 2006

PARTICULATE MATTER 

EPA Has Started to Address the National 

Academies’ Recommendations on 

Estimating Health Benefits, but More 

Progress Is Needed 

Highlights of GAO-06-780, a report to 

congressional requesters 

A large body of scientific evidence

links exposure to particulate 

matter—a widespread form of air

pollution—to serious health

problems, including asthma and 

premature death. Under the Clean

Air Act, the Environmental 

Protection Agency (EPA) 

periodically reviews the 

appropriate air quality level at

which to set national standards to

protect the public against the 

health effects of particulate matter. 

EPA proposed revisions to these 

standards in January 2006 and 

issued a draft regulatory impact

analysis of the revisions’ expected

costs and benefits. 

The estimated benefits of air 

pollution regulations have been 

controversial in the past. A 2002

National Academies report

generally supported EPA’s 

approach but made 34

recommendations to improve how 

EPA implements its approach. GAO 

was asked to determine whether

and how EPA applied the 

Academies’ recommendations in its

estimates of the health benefits

expected from the January 2006

proposed revisions to the 

particulate matter standards. GAO

examined the draft analysis, met 

with EPA officials, and interviewed 

members of the National 

Academies’ committee. In 

providing technical comments on 

the report, EPA officials said it was

fair and balanced and noted the 

agency’s progress in addressing 

recommendations via research and 

development and other analyses. 

EPA has begun to change the way it conducts and presents its analyses of 

health benefits in response to recommendations from the National 

Academies. Specifically, EPA applied, at least in part, 22—or about twothirds—of the Academies’ recommendations to its health benefit analysis of 

proposed revisions to particulate matter standards. For example, in 

response to some of the recommendations, EPA took steps toward 

conducting a more rigorous assessment of uncertainty by, for instance, 

evaluating how benefits could change under different assumptions and 

discussing sources of uncertainty not included in the benefit estimates. In 

one case, EPA applied an alternative technique, called expert elicitation, for 

evaluating uncertainty by systematically gathering expert opinion about the 

uncertainty underlying the causal link between exposure to particulate 

matter and premature death. Consistent with the National Academies’ 

recommendation to assess uncertainty by developing ranges of estimates 

and specifying the likelihood of attaining them, EPA used expert elicitation 

to develop ranges of reductions in premature death expected from the 

proposed revisions. EPA officials said that ongoing research and 

development efforts will allow the agency to gradually achieve more 

progress in applying the recommendations. We note that robust uncertainty 

analysis is important because estimates of health benefits can be highly

uncertain, as the draft regulatory impact analysis for particulate matter 

illustrates. EPA viewed the estimates in this analysis as so uncertain that it 

chose not to present them in the executive summary. 

For various reasons, EPA has not applied the remaining 12 

recommendations to the analysis, such as the recommendation to evaluate

the impact of using the simplifying assumption that each component of 

particulate matter is equally toxic. EPA officials viewed most of these 

recommendations as relevant to its health benefit analyses and, citing the 

need for additional research and development, emphasized the agency’s 

commitment to continue to respond to the recommendations. For example, 

EPA did not believe that the state of scientific knowledge on the relative 

toxicity of particulate matter components was sufficiently developed to 

include in the January 2006 regulatory impact analysis, and the agency is

currently sponsoring research on this issue. In addition, a senior EPA 

official said that insufficient resources impeded the agency’s progress in 

applying the recommendations, citing, in particular, the limited availability 

of skilled staff, time, and other resources to conduct the required analyses 

and research and development. EPA officials also said that some of the 

recommendations the agency did not apply to the draft analysis, such as one 

calling for a summary table describing key analytical information to enhance 

transparency, will be applied to the analysis supporting the final rule. To the 

extent that EPA continues to make progress addressing the Academies’ 

recommendations, decision makers and the public will be better able to 

evaluate the basis for EPA’s air regulations. 

www.gao.gov/cgi-bin/getrpt?GAO-06-780.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact John B. 

Stephenson at (202) 512-3841 or

stephensonj@gao.gov. 
Page i GAO-06-780 Particulate Matter

Contents

Letter 1

Results in Brief 3

Background 5

EPA Is in the Process of Addressing Many of the Academies’ 

Recommendations 7

Concluding Observations 15

Agency Comments 16

Appendixes

Appendix I: Scope and Methodology 17

Appendix II: Recommendations Applied or Partially Applied to the Draft 

Particulate Matter Regulatory Impact Analysis 19

Appendix III: Recommendations Not Applied to the Draft Particulate 

Matter Regulatory Impact Analysis 29

Appendix IV: GAO Contact and Staff Acknowledgments 39

Tables Table 1: Recommendations Applied or Partially Applied to the 

Draft Particulate Matter Regulatory Impact Analysis 20

Table 2: Recommendations Not Applied to the Draft Particulate 

Matter Regulatory Impact Analysis 30
This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately.

Contents

Page ii GAO-06-780 Particulate Matter

Abbreviations

Academies National Academies

ACS American Cancer Society

Council Advisory Council on Clean Air Compliance 

EPA Environmental Protection Agency

NAAQS National Ambient Air Quality Standards

OMB Office of Management and Budget

PM particulate matter

RIA regulatory impact analysis

SAB Science Advisory Board
Page 1 GAO-06-780 Particulate Matter

United States Government Accountability Office

Washington, D.C. 20548

A

July 14, 2006 Letr

The Honorable James M. Inhofe

Chairman 

Committee on Environment and Public Works 

United States Senate

The Honorable George Voinovich

Chairman 

Subcommittee on Clean Air, Climate Change,

and Nuclear Safety

Committee on Environment and Public Works 

United States Senate

A large body of scientific evidence links exposure to particulate matter—a 

ubiquitous form of air pollution commonly referred to as soot—to serious 

health problems, including asthma, chronic bronchitis, heart attack, and 

premature death. The many sources releasing particulate matter into the 

air include cars, trucks, power plants, industrial processes, forest fires, and 

waste incinerators. In 1971, the Environmental Protection Agency (EPA) 

first established national air quality standards to protect the public against 

the health effects of particulate matter, one of the six widespread criteria 

pollutants considered harmful to public health. Under the Clean Air Act, 

EPA determines the appropriate level at which to set national air quality 

standards, and the states must develop programs to achieve and maintain 

compliance with them. Further, EPA must review the standards every 5 

years to determine whether they adequately protect human health and 

welfare, given the latest scientific information available, and revise them if 

they do not.

In January 2006, after its most recent review of the national air quality 

particulate matter standards, EPA proposed revisions to the standards and 

issued a draft regulatory impact analysis. Overall, the draft regulatory 

impact analysis discussed the scope and magnitude of the particulate 

matter problem, the likely benefits of the proposed revisions for public 

health and the environment, and the expected costs of implementing the 

standards. Regarding public health, the analysis presented estimates of 

expected health benefits for the particulate matter revisions in five major 

urban areas, including reductions in the number of premature deaths and 

emergency room visits for asthma. Among the changes EPA said it plans to 

make in its final regulatory impact analysis is providing national estimates 
Page 2 GAO-06-780 Particulate Matter

of expected health benefits. EPA is required under a court order to issue 

the final rule on particulate matter standards by September 2006.1 

In 2000, at the direction of the Senate Appropriations Committee, EPA 

asked the National Academies (Academies) to evaluate EPA’s methodology 

for estimating the health benefits of proposed air pollution regulations.2

 

According to the National Academies, these estimates have often been 

controversial, and the methods EPA has used to prepare them have been 

questioned. For example, some observers, such as researchers and 

industry groups, have expressed concerns that EPA does not adequately 

factor uncertainty into its estimates of health benefits. Some level of 

uncertainty is unavoidable, in part because the scientific information used 

to develop estimates, such as the inventory of particulate matter emissions, 

will never be perfect or complete. However, according to some observers, 

EPA’s estimates of benefits appear more definitive than they really are 

because the agency does not adequately account for uncertainty in its 

analyses or in its reporting of health benefit estimates. 

The National Academies’ 2002 report on this subject generally supported 

EPA’s approach to estimating health benefits but, nevertheless, made 34 

detailed recommendations to improve how EPA implements its approach.3

 

Overall, these recommendations focus on conducting more rigorous 

assessments of uncertainty, increasing the transparency of how EPA 

estimates benefits, conducting more detailed analyses of exposure, and 

estimating the benefits of each regulatory option under consideration. 

Many of the recommendations include qualifying language indicating that it 

is reasonable to expect that they can be applied in stages, over time; and a 

number of the recommendations are interrelated and, in some cases, 

overlapping. 

You asked us to determine whether and how EPA applied the National 

Academies’ recommendations in its estimates of the health benefits 

expected from the January 2006 proposed revisions to the particulate 

matter standards. To respond to this objective, we reviewed EPA’s draft 

1

American Lung Ass’n v. Whitman, No. 1:03CV00778 (D. D.C. 2003).

2

The National Academies comprises four organizations: the National Academy of Sciences, 

National Academy of Engineering, Institute of Medicine, and National Research Council. 

3

National Research Council, Estimating the Public Health Benefits of Proposed Air 

Pollution Regulations (Washington, D.C., 2002).
Page 3 GAO-06-780 Particulate Matter

regulatory impact analysis presenting the costs and benefits of the 

proposed rule and met with senior officials from EPA’s Office of Air and 

Radiation, which was responsible for developing the proposed rule and 

analyzing its costs and benefits, and with officials from EPA’s Office of 

Policy, Economics, and Innovation. We also reviewed EPA’s and the Office 

of Management and Budget’s (OMB) guidance on conducting economic 

analyses, prior GAO reports on EPA’s regulatory impact analyses, and other 

relevant reports. As requested, our work addressed the application of the 

National Academies’ recommendations to EPA’s draft regulatory impact 

analysis supporting the 2006 proposed particulate matter rule; thus, we did 

not examine how EPA applied the recommendations to other recent air 

rules. Our work focused on broadly characterizing EPA’s progress toward 

applying the recommendations; we did not evaluate the effectiveness and 

quality of the scientific and technical actions the agency has taken to apply 

the recommendations. See appendix I for a more detailed description of the 

scope and methodology of our review. We performed our work from 

January 2006 to July 2006 in accordance with generally accepted 

government auditing standards. 

Results in Brief EPA has begun to change the way it conducts and presents its analyses of 

health benefits in response to the National Academies’ recommendations, 

which focused on conducting more rigorous assessments of uncertainty, 

increasing the transparency of how EPA estimates benefits, conducting 

more detailed analyses of exposure, and estimating the benefits of each 

regulatory option under consideration. EPA applied, at least in part, about 

two-thirds of the recommendations to its health benefit analysis. 

Specifically, of the 34 recommendations, EPA applied 8 and partially 

applied 14. For example, EPA responded to some of the recommendations 

by taking steps to conduct a more rigorous assessment of uncertainty by, 

for instance, evaluating how benefits might change given alternative 

assumptions and discussing sources of uncertainty not included in the 

benefit estimates. More specifically, EPA applied an alternative technique 

for evaluating one important source of uncertainty in its analysis—the 

uncertainty underlying the causal link between exposure to particulate 

matter and premature death. Consistent with the National Academies’ 

recommendation to assess uncertainty by developing ranges of estimates 

of benefits and specifying the likelihood of attaining that level of benefits, 

EPA systematically gathered expert opinions about this link—through a 

process called expert elicitation—and developed ranges reflecting the 

experts’ confidence in attaining reductions in premature death expected 

from the proposed revisions. However, the health benefit analysis does not 
Page 4 GAO-06-780 Particulate Matter

similarly assess how the benefit estimates would vary in light of other key 

uncertainties as the Academies had recommended. Consequently, this 

represents a partial application of the recommendation. Agency officials 

told us that ongoing research and development efforts will allow EPA to 

gradually achieve more progress in applying this and other 

recommendations to future analyses.

For various reasons, EPA did not apply the remaining 12 recommendations 

to the analysis, such as the recommendation to evaluate the impact of using 

the assumption that the components of particulate matter are equally toxic. 

EPA officials viewed most of these recommendations as relevant to its 

health benefit analyses, but noted that the agency was not ready to apply 

specific recommendations because of, among other things, the need to 

overcome technical challenges stemming from limitations in the state of 

available science. These officials emphasized the agency’s commitment to 

continue to respond to the recommendations. For example, EPA did not 

believe that the state of scientific knowledge on the relative toxicity of 

particulate matter components was sufficiently developed to include it in 

the January 2006 regulatory impact analysis, and the agency is sponsoring 

research on this issue. In addition, according to a senior EPA official, 

insufficient resources have impeded the agency’s progress in applying the 

recommendations, including the limited availability of skilled staff, time, 

and other resources to conduct and oversee the required analyses and 

research and development. Finally, EPA officials stated that some of the 

recommendations the agency did not apply to the draft analysis, such as 

one calling for a summary table describing key analytical information to 

enhance transparency, will be applied to the regulatory impact analysis 

supporting the final rule. To the extent that EPA continues to make 

progress addressing the Academies’ recommendations, decision makers 

and the public will be able to better evaluate the basis for EPA’s air 

regulations.

We provided a draft of this report to EPA for review. EPA provided 

technical comments that we incorporated, as appropriate. Officials from 

EPA’s Office of Air and Radiation noted in their technical comments that 

the report provides a fair and balanced representation of the agency’s 

application of the recommendations to the particulate matter regulatory 

impact analysis and cited EPA’s progress in meeting the National 

Academies’ recommendations through other analyses of air programs and 

through research and development efforts.
Page 5 GAO-06-780 Particulate Matter

Background EPA is required by the Clean Air Act to conduct reviews of the National 

Ambient Air Quality Standards (NAAQS) for the six criteria pollutants, 

including particulate matter, every 5 years. The overarching purpose of 

such reviews is to determine whether the current standards are sufficient 

to protect public health and welfare at large, with an adequate margin of 

safety, given the latest scientific information available at the time of the 

review. Major steps in the NAAQS process include the following: 

• developing a criteria document that synthesizes new research on health 

and environmental effects;

• preparing a staff paper that assesses the policy implications of the 

scientific information in the criteria document, which also discusses 

possible ranges for air quality standards; and

• determining whether and how EPA should revise the NAAQS.

If EPA decides to revise the NAAQS, the agency proposes the changes in 

the Federal Register. As part of the federal rule-making process, EPA is to 

comply with Executive Order 12866, which directs federal agencies to 

analyze the costs and benefits of proposed and final rules expected to 

affect the economy by $100 million or more per year.4

 In September 2003, 

the Office of Management and Budget (OMB) issued its Circular A-4, which 

presents guidance and best practices and states that agencies should 

analyze the costs and benefits in accordance with the principles of full 

disclosure and transparency. Further, in cases such as the particulate 

matter rule, where expected economic impacts exceed $1 billion annually, 

Circular A-4 also states that agencies should conduct a comprehensive 

assessment of key uncertainties in their analyses of costs and benefits, 

which EPA also refers to as regulatory impact analyses.5

 EPA’s January 

2006 regulatory impact analysis presents estimates of the costs and 

benefits for the proposed particulate matter rule. 

4

The Clean Air Act prohibits EPA from basing revisions to the national air quality standards 

on costs. Because most national air quality revisions qualify as significant actions under 

Executive Order 12866, EPA usually assesses the expected benefits and costs of the 

standards under the Executive Order.

5

Specifically, Circular A-4 states that agencies should conduct a formal probabilistic 

assessment of key uncertainties underlying its cost-and-benefit estimates. OMB Circular No. 

A-4, Regulatory Analysis (Sept. 17, 2003).
Page 6 GAO-06-780 Particulate Matter

The focus of the National Academies’ 2002 report was on how EPA 

estimates the health benefits of its proposed air regulations. To develop 

such estimates, EPA conducts analyses to quantify the expected changes in 

the number of deaths and illnesses that are likely to result from proposed 

regulations. The regulatory impact analyses also estimate the costs 

associated with implementing proposed air regulations, although, under 

the Clean Air Act, EPA is not permitted to consider costs in setting healthbased standards for the criteria air pollutants, such as particulate matter. 

Soon after the National Academies issued its report in 2002, EPA staff 

identified key recommendations and developed a strategy, in consultation 

with OMB, to apply some of the recommendations to benefit analyses for 

air pollution regulations under consideration at the time. EPA roughly 

approximated the time and resource requirements to respond to the 

recommendations, identifying those the agency could address within 2 or 3 

years and those that would take longer. According to EPA officials, the 

agency focused primarily on the numerous recommendations related to 

analyzing uncertainty. 

Both the National Academies’ report and the OMB guidance emphasize the 

need for agencies to account for uncertainties and to maintain 

transparency in the course of conducting benefit analyses. Identifying and 

accounting for uncertainties in these analyses can help decision makers 

evaluate the likelihood that certain regulatory decisions will achieve the 

estimated benefits. Transparency is important because it enables the public 

and relevant decision makers to see clearly how EPA arrived at its 

estimates and conclusions. In prior work on regulatory impact analyses, 

we have found shortcomings in EPA’s analyses of uncertainty and the 

information the agency provides with its estimates of costs and benefits.6

6

GAO, Air Pollution: Information Contained in EPA’s Regulatory Impact Analyses Can Be 

Made Clearer, GAO/RCED-97-38 (Washington, D.C.: Apr. 14, 1997); and Cost-Benefit 

Analysis Can Be Useful in Assessing Environmental Regulations, Despite Limitations, 

GAO/RCED-84-62 (Washington, D.C.: Apr. 6, 1984).
Page 7 GAO-06-780 Particulate Matter

EPA Is in the Process 

of Addressing Many of 

the Academies’ 

Recommendations 

EPA applied—either wholly or in part—approximately two-thirds of the 

Academies’ recommendations to its January 2006 regulatory impact 

analysis and continues to address the recommendations through ongoing 

research and development. The January 2006 regulatory impact analysis 

demonstrated progress toward an expanded analysis of uncertainty and 

consideration of different assumptions. EPA officials cited time and 

resource constraints, as well as the need to mitigate complex technical 

challenges, as the basis for not applying other recommendations. 

According to EPA officials, the agency did not apply some of the more 

complex recommendations because it had not achieved sufficient progress 

in the research and development projects under way.

EPA Applied, at Least in 

Part, about Two-thirds of 

the Recommendations to Its 

Particulate Matter Health 

Benefit Analysis in the 

Proposed Rule

The January 2006 regulatory impact analysis on particulate matter 

represents a snapshot of an ongoing EPA effort to respond to the National 

Academies’ recommendations on developing estimates of health benefits 

for air pollution regulations. Specifically, the agency applied, at least in 

part, approximately two-thirds of the recommendations—8 were applied 

and 14 were partially applied—by taking steps toward conducting a more 

rigorous assessment of uncertainty for proposed air pollution regulations 

by, for example, evaluating the different assumptions about the link 

between human exposure to particulate matter and health effects and 

discussing sources of uncertainty not included in the benefit estimates. 

According to EPA officials, the agency focused much of its time and 

resources on the recommendations related to uncertainty. In particular, 

one overarching recommendation suggests that EPA take steps toward 

conducting a formal, comprehensive uncertainty analysis—the systematic 

application of mathematical techniques, such as Monte Carlo simulation—

and include the uncertainty analysis in the regulatory impact analysis to 

provide a “more realistic depiction of the overall uncertainty” in EPA’s 

estimates of the benefits.7

 A number of the other recommendations 

regarding uncertainty are aimed at EPA’s developing the information and 

methodologies needed to carry out a comprehensive uncertainty analysis. 

7

Monte Carlo simulation refers to a computer-based analysis that uses probability 

distributions for key variables, selects random values from each of the distributions 

simultaneously, and repeats the random selection over and over. Rather than presenting a 

single outcome—such as the mostly likely or average scenario—Monte Carlo simulations 

produce a distribution of outcomes that reflect the probability distributions of modeled 

uncertain variables.
Page 8 GAO-06-780 Particulate Matter

Overall, the uncertainty recommendations suggest that EPA should 

determine (1) which sources of uncertainties have the greatest effect on 

benefit estimates and (2) the degree to which the uncertainties affect the 

estimates by specifying a range of estimates and the likelihood of attaining 

them. In response, EPA devoted significant resources to applying an 

alternative technique called expert elicitation in a multiphased pilot 

project. The pilot project was designed to systematically obtain expert 

advice to begin to better incorporate in its health benefit analysis the 

uncertainty underlying the causal link between exposure to particulate 

matter and premature death. EPA used the expert elicitation process to 

help it more definitively evaluate the uncertainty associated with estimated 

reductions in premature death—estimates that composed 85 percent to 95 

percent of EPA’s total health benefit estimates for air pollution regulations 

in the past 5 years, according to the agency.8

 EPA developed a range of 

expected reductions in death rates based on expert opinion systematically 

gathered in its pilot expert elicitation project and provided the results of 

this supplemental analysis in an appendix to the regulatory impact 

analysis.9

 However, the National Academies had recommended that EPA 

merge such supplemental analyses into the main benefit analysis. 

Moreover, the Academies recommended that EPA’s main benefit analysis 

reflect how the benefit estimates would vary in light of uncertainties. In 

addition to the uncertainty underlying the causal link between exposure 

and premature death that EPA analyzed, other key uncertainties can 

influence the estimates. For example, there is uncertainty about the effects 

of the age and health status of people exposed to particulate matter, the 

varying composition of particulate matter, and the measurements of actual 

exposure to particulate matter. EPA’s health benefit analysis, however, does 

not account for these key uncertainties by specifying a range of estimates 

and the likelihood of attaining them, similar to estimates derived from the 

8

It is important to note, according to EPA, that quantified benefit estimates do not include 

other potential benefits, such as reduction in certain illnesses or environmental impacts, 

because of limited data. The fraction of total benefits attributable to reductions in mortality 

may therefore vary as other benefits are incorporated in the numerical estimates. 

9

The pilot expert elicitation, based on methods that were peer-reviewed, involved 

structured, daylong interviews with five experts about particulate matter exposure and 

death. EPA then analyzed the experts’ responses to develop ranges reflecting the experts’ 

confidence in estimated reductions in premature death associated with the proposed 

revisions. Pending completion of a peer review, EPA plans to include the analysis of a fullscale expert elicitation panel in the final regulatory impact analysis. 
Page 9 GAO-06-780 Particulate Matter

expert elicitation addressing causal uncertainty. For these reasons, EPA’s 

responses reflect a partial application of the Academies’ recommendation. 

In addition, the Academies recommended that EPA both continue to 

conduct sensitivity analyses on sources of uncertainty and expand these 

analyses. In the particulate matter regulatory impact analysis, EPA 

included a new sensitivity analysis regarding assumptions about 

thresholds, or levels below which those exposed to particulate matter are 

not at risk of experiencing harmful effects. EPA has assumed no threshold 

level exists—that is, any exposure poses potential health risks.10 Some 

experts have suggested that different thresholds may exist and the National 

Academies recommended that EPA determine how changing its 

assumption—that no threshold exists—would influence the estimates. The 

sensitivity analysis EPA provided in the regulatory impact analysis 

examined how its estimates of expected health benefits would change 

assuming varying thresholds. 

Another recommendation that EPA is researching and partially applied to 

the draft regulatory impact analysis concerns alternative assumptions 

about cessation lags—the time between reductions in exposure to 

particulate matter and the health response. The National Academies made 

several recommendations on this topic, including one that EPA incorporate 

alternative assumptions about lags into a formal uncertainty analysis to 

estimate benefits that account for the likelihood of different lag durations. 

In response, EPA has sought advice from its Advisory Council on Clean Air 

Compliance Analysis on how to address this recommendation and has 

conducted a series of sensitivity analyses related to cessation lags. EPA is 

also funding research to explore ways to address lag effects in its 

uncertainty analysis. According to an EPA official, specifying the 

probability of different lag effects is computationally complex, and the 

agency is working to resolve this challenge. 

In response to another recommendation by the National Academies, EPA 

identified some of the sources of uncertainty that are not reflected in its 

benefit estimates. For example, EPA’s regulatory impact analysis disclosed 

that its benefit estimates do not reflect the uncertainty associated with 

10Recent EPA analyses used the natural background concentrations of particulate matter, 

rather than zero, for its assumption of no threshold level. The National Academies 

supported the assumption of no threshold level, but it recommended that EPA conduct a 

consistent and transparent sensitivity analysis to consider various threshold levels.
Page 10 GAO-06-780 Particulate Matter

future year projections of particulate matter emissions. EPA presented a 

qualitative description about emissions uncertainty, elaborating on 

technical reasons—such as the limited information about the effectiveness 

of particulate matter control programs—why the analysis likely 

underestimates future emissions levels. EPA also applied the Academies’ 

recommendation on the presentation of uncertainty, which encouraged the 

agency to present the results of its health benefit analyses in ways that 

convey the estimated benefits more realistically by, for example, placing 

less emphasis on single estimates and rounding the numbers. EPA’s 

regulatory impact analysis presented ranges for some of the benefit 

estimates. Also, EPA sought to convey the overall uncertainty of its benefit 

estimates in a qualitative manner by clearly stating that decision makers 

and the public should not place significant weight on the quantified benefit 

estimates in the regulatory impact analysis because of data limitations and 

uncertainties. 

Another example of EPA’s response to the National Academies’ 

recommendations involves exploring the various regulatory choices 

available to decision makers. The Academies recommended that EPA 

estimate the health benefits representing the full range of regulatory 

choices available to decision makers. In the particulate matter analysis, 

EPA presented health benefits expected under several regulatory options 

targeting fine particulate matter. Citing a lack of data and tools needed to 

conduct an accurate analysis, EPA did not estimate the benefits expected 

under the proposed regulatory options for coarse particulate matter but, 

consistent with the National Academies’ recommendation, presented its 

rationale for not doing so. Overall, we considered this a partial application 

of the recommendation. (See app. II for more detail on the 

recommendations that EPA has applied or partially applied to the draft 

particulate matter regulatory impact analysis.)

EPA Plans to Address Some 

of the Remaining 

Recommendations in the 

Final Rule and Has 

Research and Development 

Under Way to Address 

Others

EPA did not apply the remaining 12 recommendations to the analysis for 

various reasons. While EPA applied some recommendations—either 

wholly or in part—that require additional studies, methodologies, or data to 

its particulate matter analysis, the agency had not made sufficient progress 

in addressing others and therefore did not apply them to the analysis. EPA 

officials viewed most of these recommendations as relevant to its health 

benefit analyses and, citing the need for additional research and 

development, emphasized the agency’s commitment to continue to respond 

to the recommendations. According to a senior EPA official, insufficient 

resources impeded the agency’s progress in applying the 
Page 11 GAO-06-780 Particulate Matter

recommendations. This official cited limited availability of skilled staff, 

time, and other resources to conduct the required analyses and research 

and development. According to EPA, some of the more complex, long-term 

recommendations include the following: relying less on simplifying 

assumptions, such as the assumption that the various components of 

particulate matter have equal toxicity;11 conducting a formal assessment of 

the uncertainty of particulate matter emissions; and assessing the expected 

reduction of any harmful effects other than air pollution or human health 

problems. 

For example, EPA is in the process of responding to a recommendation 

involving the relative toxicity of components of particulate matter, an 

emerging area of research that has the potential to influence EPA’s 

regulatory decisions in the future.12 Specifically, the agency could, 

hypothetically, refine national air quality standards to address the 

potentially varying health consequences associated with different 

components of particulate matter. The National Academies recommended 

that EPA strengthen its benefit analyses by evaluating a range of alternative 

assumptions regarding relative toxicity and incorporate these assumptions 

into sensitivity or uncertainty analyses as more data become available.13 

EPA did not believe the state of scientific knowledge on relative toxicity 

was sufficiently developed at the time it prepared the draft regulatory

11Particulate matter is a highly complex mixture comprising particles emitted directly from 

sources and particles formed through atmospheric chemical reactions. Particles span many 

sizes and shapes and consist of hundreds of different chemicals. EPA identifies the major 

components of fine particulate matter as carbon, sulfate and nitrate compounds, and 

crustal/metallic materials such as soil and ash.

12Relative toxicity refers to the premise that different components of particulate matter have 

different levels of potency affecting premature mortality and illness. In the draft particulate 

matter regulatory impact analysis, EPA assumed equivalent toxicity, stating that “while it is 

reasonable to expect that the potency of components may vary across the numerous effect 

categories associated with particulate matter, EPA’s interpretation of scientific information 

considered to date is that such information does not yet provide a basis for quantification 

beyond using fine particle mass.” EPA, Draft Regulatory Impact Analysis for the PM-2.5 

National Ambient Air Quality Standards (Washington, D.C., 2006), 3-21.

13In the context of the National Academies recommendations, a sensitivity analysis would 

assess how changes in one or more variables affect the outcome, whereas a comprehensive 

or formal uncertainty analysis would evaluate the probability distributions of multiple 

variables. 
Page 12 GAO-06-780 Particulate Matter

impact analysis to include this kind of analysis.14 However, EPA is 

sponsoring research on this issue. For example, EPA is supporting longterm research on the relative toxicity of particulate matter components 

being conducted by EPA’s intramural research program, its five Particulate 

Matter Research Centers, and the Health Effects Institute, an organization 

funded in part by EPA. In addition, an EPA contractor has begun to 

investigate methods for conducting a formal analysis that would consider 

sources of uncertainty, including relative toxicity and lag effects. To date, 

the contractor has created a model to assess whether and how much these 

sources of uncertainty may affect benefit estimates in one urban area. 

The National Academies also recommended that EPA incorporate an 

assessment of uncertainty into the early stages of its benefit analyses by 

characterizing the uncertainty of its emissions estimates on which the 

agency is going to base its benefit estimates.15 While the agency is 

investigating ways to assess or characterize this uncertainty, EPA did not 

conduct a formal uncertainty analysis for particulate matter emissions for 

the draft regulatory impact analysis because of data limitations. These 

limitations stem largely from the source of emissions data, the National 

Emissions Inventory,16 an amalgamation of data from a variety of entities, 

including state and local air agencies, tribes, and industry. According to 

EPA, these entities use different methods to collect data, which have 

different implications for how to characterize the uncertainty. 

Furthermore, the uncertainty associated with emissions varies by the 

source of emissions. For example, the analytical methods for evaluating the 

uncertainty of estimates of emissions from utilities would differ from those 

for car and truck emissions because the nature of these emissions and the 

data collection methods differ. In sum, to apply this recommendation, EPA 

14In a separate report issued in 2004, the National Academies identified relative toxicity as a 

priority research topic, noting that technical challenges have impeded research progress. 

The Clean Air Scientific Advisory Committee also noted the need for more research and 

concluded in 2005 that not enough data are available to base the particulate matter 

standards on composition. OMB, however, encouraged EPA in 2006 to conduct a sensitivity 

analysis on relative toxicity and referred the agency to a sensitivity analysis on relative 

toxicity funded by the European Commission. 

15Because the precise levels of total emissions are not knowable but rather approximations 

based on a sample of measurements, there is uncertainty about the true quantity of 

emissions. 

16EPA compiles the National Emissions Inventory, a national database of air emissions data 

that includes estimates of annual emissions, by source, of air pollutants in each area of the 

country on an annual basis.
Page 13 GAO-06-780 Particulate Matter

must determine how to characterize the uncertainty of the estimates for 

each source of emissions before aggregating the uncertainty to a national 

level and then factoring that aggregation into its benefit estimates. 

According to EPA officials, the agency needs much more time to resolve 

the complex technical challenges of such an analysis. EPA officials also 

noted that the final particulate matter analysis will demonstrate steps 

toward this recommendation by presenting emissions data according to the 

level emitted by the different kinds of sources, such as utilities, cars, and 

trucks.

Another recommendation that EPA is researching but did not apply to the 

draft regulatory impact analysis concerns whether the proposed revisions 

to the particulate matter standards would have important indirect impacts 

on human health and the environment. According to an EPA official, the 

agency could not rule out the possibility that the revisions could have 

indirect impacts on the environment, such as whether reductions to 

particulate matter emissions would reduce the amount of particulate 

matter deposited in water bodies, thereby decreasing water pollution. EPA 

has considered indirect impacts of air pollution regulations on sensitive 

water bodies in the past and plans to include a similar analysis in the final 

particulate matter rule. An agency official further noted that ongoing 

research about environmental impacts could reveal additional indirect 

impacts for future analyses. 

Other recommendations that EPA did not apply to its benefit estimates in 

the regulatory impact analysis concern issues such as transparency and 

external review of EPA’s benefit estimation process. For example, the 

National Academies recommended that EPA clearly summarize the key 

elements of the benefit analysis in an executive summary that includes a 

table that lists and briefly describes the regulatory options for which EPA 

estimated the benefits, the assumptions that had a substantial impact on 

the benefit estimates, and the health benefits evaluated. EPA did not, 

however, present a summary table as called for by the recommendation or 

summarize the benefits in the executive summary. As EPA stated in the 

particulate matter analysis, the agency decided not to present the benefit 

estimates in the executive summary because they were too uncertain. 

Specifically, officials said the agency was not able to resolve some 

significant data limitations before issuing the draft regulatory impact 

analysis in January 2006—a deadline driven by the need to meet the courtordered issue date for the final rule in September 2006. According to EPA 

officials, EPA has resolved some of these data challenges by, for example, 

obtaining more robust data on anticipated strategies for reducing 
Page 14 GAO-06-780 Particulate Matter

emissions, which will affect the estimates of benefits. The officials also said 

that EPA intends to include in the executive summary of the regulatory 

impact analysis supporting the final rule a summary table that describes 

key analytical information. EPA officials also acknowledged other 

presentation shortcomings, including references to key analytical elements 

that were insufficiently specific, that officials attributed to tight time 

frames and the demands of working on other regulatory analyses 

concurrently. They said they plan to address these shortcomings in the final 

regulatory impact analysis.

Regarding external review, the National Academies recommended that EPA 

establish an independent review panel, supported by permanent technical 

staff, to bolster EPA’s quality control measures for its regulatory impact 

analyses, such as the one for particulate matter. The National Academies 

noted that peer review of EPA’s regulatory impact analyses would be 

advantageous when the agency designs and conducts its economic 

analysis.17 EPA has not directly addressed this recommendation. According 

to the Director of the Office of Policy Analysis and Review in EPA’s Office 

of Air and Radiation, establishing and supporting independent committees 

is costly, making it important for EPA to take advantage of existing panels 

rather than set up new ones. Further, an official in the Office of Air and 

Radiation who oversees the development of regulatory impact analyses 

said that the cost of reviewing all regulatory impact analyses would be 

substantial. In this regard, EPA officials identified peer reviews the agency 

received from its existing independent committees, such as the Clean Air 

Scientific Advisory Committee and the Advisory Council on Clean Air 

Compliance.18 For example, to respond to the Academies’ 

recommendations about lag effects, EPA sought independent advice on the 

assumptions it was developing regarding the time between reduced 

exposure to particulate matter and reductions in incidences of health 

17In prior work on regulatory economic analyses, we recommended that OMB direct 

agencies to obtain peer review of these analyses. See GAO, Regulatory Reform: Agencies 

Could Improve Development, Documentation, and Clarity of Regulatory Economic 

Analyses, GAO/RCED-98-142 (Washington, D.C.: May 26, 1998).

18The Advisory Council on Clean Air Compliance, composed of rotating membership, serves 

to advise the agency on choices relating to data, modeling, and methodology associated with 

air programs and does not review regulatory impact analyses. Established under the Clean 

Air Act and operated through the Science Advisory Board, the advisory council advises EPA, 

under Section 812 of the act, on developing the “statutorily mandated comprehensive 

analyses of the total costs and total benefits of programs implemented pursuant to the Clean 

Air Act.” These analyses are commonly referred to as the 812 studies.
Page 15 GAO-06-780 Particulate Matter

effects.19 Finally, EPA officials noted that although the agency does not 

have each regulatory impact analysis peer reviewed, EPA typically does 

have the methodologies that will be applied to regulatory impact analyses 

peer reviewed. (See app. III for more detail on these recommendations and 

others that EPA did not apply to the draft particulate matter regulatory 

impact analysis.) 

Concluding 

Observations

While EPA has taken a number of steps to respond to the Academies’ 

recommendations on estimating health benefits, continued commitment 

and dedication of resources will be needed if EPA is to fully implement the 

improvements endorsed by the National Academies. In particular, the 

agency will need to ensure that it allocates resources to needed research on 

emerging issues, such as the relative toxicity of particulate matter 

components; assessing which sources of uncertainty have the greatest 

influence on benefit estimates; and estimating other benefits, such as 

environmental improvements. In addition, it is important for EPA to 

continue to improve its uncertainty analysis in accordance with the 

Academies’ recommendations. The agency’s draft regulatory impact 

analysis illustrates that estimates of health benefits can be highly 

uncertain. In fact, EPA officials viewed these estimates as so uncertain that 

they chose to not present them in the executive summary of the regulatory 

impact analysis. While EPA officials said they expect to reduce the 

uncertainties associated with the health benefit estimates in the final 

particulate matter analysis, robust uncertainty analysis will nonetheless be 

important for decision makers and the public to understand the likelihood 

of attaining the estimated health benefits. According to EPA officials, the 

final regulatory impact analysis on particulate matter will reflect further 

responsiveness to the Academies’ recommendations by, for example, 

providing additional sensitivity analysis and improving the transparency of 

the regulatory impact analysis by highlighting key data and assumptions in 

the executive summary. Moreover, these officials emphasized the agency’s 

commitment to further enhancing the transparency of the analysis by 

presenting clear and accurate references to the supporting technical 

documents, which detail the analytical assumptions and describe the data 

supporting the estimates. To the extent EPA continues to make progress 

addressing the Academies’ recommendations, decision makers and the 

public will be able to better evaluate the basis for EPA’s air regulations.

19In 2004, EPA asked for and received guidance from the Advisory Council on Clean Air 

Compliance on the lag estimates the agency should use in the particulate matter analyses. 
Page 16 GAO-06-780 Particulate Matter

Agency Comments We provided a draft of this report to EPA for review. EPA provided 

technical comments that we incorporated, as appropriate. Officials from 

the Office of Policy Analysis and Review within EPA’s Office of Air and 

Radiation noted in their technical comments that the report provides a fair 

and balanced representation of EPA’s efforts to apply the National 

Academies’ recommendations to the draft particulate matter regulatory 

impact analysis. However, these officials also cited progress made in 

applying the National Academies’ recommendations through analyses of 

other air programs and through research and development efforts. We note 

that this report does identify, as appropriate, EPA’s research and 

development efforts for recommendations EPA did not apply to the draft 

particulate matter analysis, its plans to apply some additional 

recommendations to the final particulate matter regulatory impact 

analysis, and the agency’s responses to recommendations in prior rulemaking analyses of air programs.

As agreed with your offices, unless you publicly announce the contents of 

this report earlier, we plan no further distribution until 30 days from the 

report date. At that time, we will send copies to the EPA Administrator and 

other interested parties. We will also make copies available to others upon 

request. In addition, the report will be available at no charge on the GAO 

Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me 

at (202) 512-3841 or stephensonj@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this report. Key contributors to this report are listed in appendix IV.

John B. Stephenson

Director, Natural Resources and Environment 
Page 17 GAO-06-780 Particulate Matter

Appendix I

Scope and Methodology AAppenednxidexiIs

We were asked to determine whether and how the Environmental 

Protection Agency (EPA) applied the National Academies (Academies) 

recommendations in its estimates of the health benefits expected from the 

January 2006 proposed revisions to the particulate matter national ambient 

air quality standards. In response to this objective, we assessed EPA’s 

response to the Academies’ recommendations and present an overview of 

the agency’s completed, ongoing, and planned actions addressing the 

recommendations. To develop this overview, we reviewed EPA’s particulate 

matter regulatory impact analysis, EPA’s economic analysis guidelines, and 

Office of Management and Budget (OMB) guidance on regulatory impact 

analysis. We also analyzed documentation addressing current and future 

agency efforts to address the recommendations, such as project planning 

memorandums and technical support documents discussing the 

application of economic techniques. In addition, we met with senior 

officials from EPA’s Office of Air and Radiation, which was responsible for 

developing the proposed rule and analyzing its economic effects, and with 

officials from EPA’s Office of Policy, Economics, and Innovation to discuss 

the agency’s responses to the recommendations. We interviewed several 

experts outside EPA, including (1) the Chair and other members of the 

National Academies’ Committee on Estimating the Health-Risk-Reduction 

Benefits of Proposed Air Pollution Regulations, to clarify the basis for their 

recommendations; and (2) economists at Resources for the Future, to 

discuss the technical issues underlying the recommendations on 

uncertainty analysis. 

While the 2002 National Academies’ report is generally applicable to EPA 

air pollution regulations, our review focused on the application of the 

recommendations to the proposed revisions to the particulate matter 

standards, as requested. Our work focused on broadly characterizing EPA’s 

progress toward applying the recommendations; we did not evaluate the 

effectiveness or quality of the scientific and technical actions the agency 

has taken to apply them. To assess whether and how EPA has made 

progress in responding to the recommendations, we developed the 

following recommendation classification continuum: applied, partially 

applied, and not applied. The applied and partially applied categories refer 

to completed and initiated actions in EPA’s health benefit analysis of 

particulate matter that corresponds to components of the National 

Academies’ recommendations. The not applied category includes 

recommendations that EPA did not apply when conducting the analysis for 

the January 2006 particulate matter regulatory impact analysis and 

identifies those for which ongoing research and development efforts were 

not far enough along to apply to the particulate matter analysis. We 
Appendix I

Scope and Methodology

Page 18 GAO-06-780 Particulate Matter

performed our work from January 2006 to July 2006 in accordance with 

generally accepted government auditing standards. 
Page 19 GAO-06-780 Particulate Matter

Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis ApendxiI

Table 1 provides a summary of the National Academies’ recommendations 

that EPA has applied or partially applied to its draft regulatory impact 

analysis (RIA) for particulate matter (PM). This table also provides GAO’s 

assessment of EPA’s progress in applying each recommendation, in terms 

of steps EPA has taken thus far to address issues highlighted in the 

National Academies’ report. The final column characterizes EPA’s 

comments regarding each recommendation, including, as pertinent, 

contextual information, potential impediments to application, and intended 

next steps.
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 20 GAO-06-780 Particulate Matter

Table 1: Recommendations Applied or Partially Applied to the Draft Particulate Matter Regulatory Impact Analysis

National Academies’ recommendation Status GAO assessment EPA’s response

1 To the extent possible, EPA should estimate the 

benefits for several regulatory options that 

represent the full range of choices available to 

the decision maker. The regulatory options 

should include graded levels of stringency 

requirements and the time schedule for 

achieving reductions in emissions or 

exposures. If options are eliminated at an 

earlier stage, the rationale for doing so should 

be provided.

Partially 

applied

EPA estimated the health 

benefits for the base case and 

some of the proposed standards 

for fine particulate matter, but did 

not estimate benefits that 

represented the full range of 

choices available to the decision 

maker. In addition, EPA did not 

estimate the benefits for coarse 

particulate matter. While EPA 

provided an explanation as to 

why it did not estimate benefits 

for the regulatory options for 

coarse particulate matter, it did 

not present its rationale for not 

estimating the benefits of the full 

range of options for fine 

particulate matter.

EPA did not estimate benefits for 

coarse particulate matter 

primarily because of a lack of 

data and tools needed to conduct 

an accurate analysis. In addition, 

EPA said that time and resources 

limited the number of regulatory 

options to be modeled. EPA said 

that given these constraints, the 

agency selected regulatory 

options that represent reasonable 

bounds on alternatives. 

2 EPA should modify the air quality models used 

in translating predicted emissions into 

predicted levels of ambient air quality to reduce 

resources required for air quality modeling. 

This change is necessary if EPA is to evaluate 

multiple regulatory alternatives and to evaluate 

each alternative at reasonable time intervals, 

such as every 5 years. Evaluation of the 

ambient air quality associated with more 

emissions scenarios is also essential if the 

uncertainty inherent in emissions estimates is 

to be carried through to the estimation of 

avoided cases of mortality and morbidity.

Partially 

applied

EPA has modified air quality 

models but has not yet 

demonstrated that these 

changes reduce resources. EPA 

reports faster modeling runs 

once it has designated a model 

but has also needed to conduct 

more complex model runs, which 

require time. 

EPA characterized its response to 

this recommendation as a work in 

progress. EPA said that the 

agency has achieved greater 

efficiency in the models, but that 

the overall time and resources 

devoted to modeling have not 

decreased because of the 

increased complexity and 

increased volume of air quality 

model runs. Citing emissions data 

limitations as an ongoing 

challenge, EPA reported notable 

improvements to the emissions 

inventory that contributed to 

efficiency gains in air quality 

modeling runs. Overall, EPA is 

working to balance efforts to 

streamline models with the 

demands for more sophisticated 

analyses. In addition, EPA cited 

additional analytical 

requirements, such as new peer 

review policies and OMB’s 

expanded data quality guidelines, 

that add to the complexity of, and 

therefore time and resources 

needed for, its models. 
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 21 GAO-06-780 Particulate Matter

3 EPA should incorporate estimates of future 

trends in background mortality and morbidity 

for the major health outcomes, such as those 

that make up two-thirds of total deaths or lost 

life years that are under consideration.

Partially 

applied

EPA incorporated future trends in 

background mortality rates but 

did not do so for future morbidity 

trends. 

EPA did not incorporate future 

morbidity trends because of time 

constraints. As time permits, EPA 

plans to incorporate such 

projections in the final analysis. 

4 As in all other stages of the benefits analysis, 

EPA should justify and clearly describe the 

assumptions and methods used to assess 

exposure, choose health outcomes, and select 

studies and concentration-response functions, 

paying careful attention to assessing and 

communicating key sources of uncertainty.

Partially 

applied

The draft PM RIA presented 

EPA’s justifications and included 

clear descriptions of a number of 

assumptions and methods used 

in the benefit analysis. As 

discussed elsewhere, EPA’s 

assessment of key sources of 

uncertainty generally relied on 

qualitative discussion and 

sensitivity analysis (see 

recommendation 14). However, 

the extent to which EPA provided 

justifications and clear 

descriptions varied. For example, 

EPA included detailed 

information about why it chose 

studies related to the 

concentration-response function, 

but the agency did not present its 

justification for an assumption 

used to assess exposure. 

EPA intends to include references 

in the final RIA, detailing the 

assumptions and methods 

underlying its health benefits 

analyses. EPA stated that rather 

than restating information from 

prior RIAs, it provides references 

to these discussions in order to 

manage the length of the RIA and 

save time. Furthermore, EPA 

stated that it is not necessary to 

document assumptions for tools 

that have already been peer 

reviewed, such as BenMAP, the 

model used to estimate health 

benefits. 

5 Because pollution modeling rarely addresses 

the smaller-scale issue of how local 

concentrations from specific source categories 

interact with human time-activity patterns, EPA 

should examine how different major source 

categories—for example, mobile versus large 

stationary sources—affect total exposures per 

unit emissions. 

Partially 

applied

EPA assessed local 

concentrations in terms of how 

different source categories, such 

as stationary and mobile sources 

of PM, affect total exposures. 

However, EPA has not yet 

assessed how human-time 

activity patterns, such as 

lifestyles, affect exposure to PM. 

EPA focused on assessing the 

local concentrations from specific 

source activities but assumed the 

same time and activity patterns 

for each scenario. EPA stated that 

it does not intend to conduct a 

detailed analysis of microenvironmental issues or humantime activity patterns.

6 EPA should consider data from U.S. and nonU.S. studies to extrapolate beyond the age 

groups evaluated and incorporate other 

relevant outcomes not evaluated in its current 

benefit analyses.

Applied EPA considered data from 

additional U.S. studies as part of 

its effort to expand the age 

groups in its estimates of the 

health outcomes (premature 

mortality and illness). EPA 

extrapolated beyond the age 

groups in some cases, but not 

all.

EPA determined, on the basis of 

advice from the Science Advisory 

Board, that it would only 

extrapolate data to other ages 

when it found a reasonable 

physiological basis for doing so. 

For example, EPA used data from 

a study on asthma in children 

ages 7 to 11 to estimate the 

reductions in asthma for the 

entire child age group—ages 6 to 

18.

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 22 GAO-06-780 Particulate Matter

7 EPA provided little information in the benefit 

analyses reviewed by the committee on causal 

association between particular types of air 

pollution and adverse health outcomes. EPA 

should summarize the evidence for causality to 

justify the inclusion or exclusion of the health 

outcomes and to assess the uncertainty 

associated with the assumption of causality.

Applied EPA referred readers to a prior 

RIA for information on the causal 

association between particulate 

matter and adverse health 

outcomes. 

8 EPA should investigate and, if necessary, 

develop methods of evaluating causal 

uncertainty relating to key outcomes so that 

this uncertainty can be represented in the final 

benefit estimates.

Applied EPA investigated one method—

expert elicitation—to evaluate 

how causal uncertainty affects 

final benefit estimates. See 

recommendation 14 for more 

details. 

EPA intends to continue its effort 

to better characterize the 

uncertainty in key health 

outcomes. 

9 Although the committee believes the use of the 

American Cancer Society (ACS) study to 

derive premature mortality estimates was 

reasonable, EPA should thoroughly review its 

selection of the best estimate for long-term 

effects of air pollution on mortality. Several new 

studies have been published since the ACS 

study, including an extended analysis of the 

ACS study, a new U.S. cohort study, and other 

non-U.S. studies. EPA should also consider 

whether the derivation of a weighted mean 

estimate from the cohort studies is appropriate 

following review of the database.

Applied EPA reviewed its selection of the 

best estimate for long-term 

effects of air pollution on 

mortality and concluded that 

data from the extended analysis 

of the ACS study provided the 

best estimates of premature 

mortality. EPA consulted the 

Science Advisory Board to reach 

its decision to emphasize the 

ACS data. EPA also determined 

that it was not appropriate to 

derive a weighted mean estimate 

from cohort studies. 

EPA incorporated new studies 

regarding estimates of premature 

mortality and justified the 

concentration-response functions 

derived from these studies. 

These justifications were included 

in technical appendices to the 

draft PM RIA. EPA also consulted 

with the Science Advisory Board 

regarding the best estimates for 

premature mortality. EPA decided 

not to derive a weighted mean 

estimate because the use of this 

estimate would risk losing the 

variability across cohort studies. 

10 To evaluate short-term effects of air pollution, 

EPA should use concentration-response 

functions from studies that integrate over 

several days or weeks the exposure period and 

the time period to the event (cumulative or 

distributed lag models), rather than those that 

restrict these time periods to 1 or 2 days.

Applied EPA used studies that integrate 

distributed lag models that 

account for the onset of health 

effects occurring more than 

several days after exposure. 

11 Although the assumption of no thresholds in 

the most recent EPA benefit analyses was 

appropriate, EPA should evaluate threshold 

assumptions in a consistent and transparent 

framework using several alternative 

assumptions in the formal uncertainty analysis.

Partially 

applied

EPA evaluated several different 

threshold assumptions in a 

sensitivity analysis but has not 

yet considered these 

assumptions in a formal 

uncertainty analysis. The 

sensitivity analysis was 

transparent—EPA clearly 

explained the basis for the 

different threshold assumptions.

EPA reported some progress 

toward improving its approach to 

characterize uncertainties and to 

conduct a formal uncertainty 

analysis. (See recommendation 

14 for more information.) 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 23 GAO-06-780 Particulate Matter

12 The committee found little justification for the 5-

year time course of exposure and outcome 

assumed in more recent EPA analyses and 

recommends that EPA more fully account for 

the uncertainty regarding lags in health effects 

by incorporating a range of assumptions and 

probabilities about the temporal relationship.

Partially 

applied

EPA sought advice from the 

Advisory Council on Clean Air 

Compliance (Council) and 

incorporated the advice by 

adjusting the agency’s 

assumptions regarding lag times. 

As for the uncertainty 

surrounding lags in health 

effects, EPA conducted a series 

of sensitivity analyses on the 

temporal relationship but did not 

assess underlying probabilities. 

Consistent with the 

recommendation made by the 

Council, EPA now uses a 20-year 

lag model. The revised lag model 

involves a higher percentage of 

mortality reductions occurring in 

the first year (30 percent) than the 

previous EPA lag model (20 

percent). EPA told us that it will 

not include probability-based 

distributions in the final RIA 

because of computational 

complexities. 

13 EPA is encouraged to estimate and report 

benefits by age, sex, and other demographic 

factors. The committee recognizes, however, 

that evaluating the differences for various 

subgroups adds complexity and uncertainty to 

the analysis and that caution must be 

exercised in the interpretation of such results.

Partially 

applied

EPA included some health 

benefit estimates for specific age 

groups in an appendix of the 

draft PM RIA. However, the 

agency did not report the primary 

benefit estimates according to 

age groups or other demographic 

factors. 

EPA intends to present in the final 

RIA a detailed summary table 

with the age breakdown of its 

sample population. EPA will not, 

however, estimate and report 

benefits by other demographic 

factors because of time and 

resource constraints. EPA also 

cited data limitations, noting that it 

lacks data accounting for local 

variations in demographic factors. 

The agency intends to overcome 

this limitation using tract-level 

data, which captures details 

about local level conditions, from 

the U.S. Census data in future 

analyses. 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 24 GAO-06-780 Particulate Matter

14 EPA should begin to move the assessment of 

uncertainties from its ancillary analyses to its 

primary analyses. This shift will require the 

specification of a probability distribution for 

each uncertainty source that is added to the 

primary analysis and, as necessary, the 

specification of joint distributions for the 

uncertainty sources that are not independent of 

each other. Expert judgment, as well as data, 

will be required to specify these distributions. 

Although the effect on the mean of the resulting 

probability distribution might increase, 

decrease, or remain the same, the effect on the 

spread of the distribution will be a predictable 

widening and, therefore, a more realistic 

depiction of the overall uncertainty in the 

analysis.

Partially 

applied

EPA has taken some steps 

toward the formal uncertainty 

analysis called for by this 

recommendation, but the primary 

analysis in the draft PM RIA 

generally addresses uncertainty 

in a qualitative manner. Overall, 

the numerical benefit estimates 

do not capture the key sources of 

uncertainty. The agency 

generally relied on sensitivity 

analysis to assess some 

uncertain factors one at a time 

rather than using more 

comprehensive techniques for 

assessing probability 

distributions of multiple variables. 

In addition, EPA used expert 

elicitation to help assess 

uncertainty relating to the 

concentrations of PM linked to 

premature death and the dollar 

value of risk reductions 

associated with reductions in 

PM. However, these results were 

presented in an appendix and 

not in the primary analysis in the 

draft RIA. In addition, the health 

benefit analysis did not present a 

quantitative assessment of how 

the benefit estimates would vary 

in light of other key uncertainties.

EPA reported some progress 

toward improving its approach to 

characterize uncertainties with 

particular emphasis on one 

source of uncertainty—premature 

death linked to PM exposure. 

Specifically, EPA used expert 

elicitation to begin to specify a 

distribution for the uncertainty in 

concentration levels of PM linked 

to premature death. The agency 

cited technical challenges, such 

as lack of data or reliable 

methods, and resource 

constraints, including limitations 

to its progress to fully 

characterize uncertainty. 

Moreover, EPA stated that its 

focus on the expert elicitation 

technique limited the time and 

resources necessary to address 

other aspects of uncertainty in 

premature death as well as 

illnesses linked to PM exposure. 

EPA applied another formal 

method for assessing 

uncertainty—Monte Carlo 

analysis—to a previous 

regulatory impact analysis but, 

according to EPA, did not have 

time to incorporate this work in 

the PM RIA. Time permitting, the 

agency plans to present the 

results of a Monte Carlo analysis 

in the final PM RIA.

15 Because the incorporation of expert judgment 

when data are unavailable will influence the 

estimates of health benefits as well as the 

uncertainty analyses, the committee also 

recommends that EPA clearly distinguish 

between data-derived estimates of some 

components, such as the concentrationresponse function, and expert opinions about 

other components that are lacking in scientific 

data, such as the degree of compliance with a 

particular regulation 30 years into the future. In 

this way, policymakers will better understand 

how existing data and expert judgment 

combine to produce estimates and where new 

data would be most valuable.

Applied EPA distinguished between dataderived estimates and those 

from expert judgment. For 

example, in the appendix, EPA 

clearly distinguished between 

data derived from experts and 

the data based on an empirical 

study. In addition, EPA discusses 

the basis for assumptions, which 

require analytical judgment, 

either directly in the draft PM RIA 

or by reference to supporting 

documents.

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 25 GAO-06-780 Particulate Matter

16 As EPA begins the transition to incorporate 

additional sources of uncertainty into its 

primary health benefits analyses, it should 

continue the sensitivity analyses it has 

traditionally conducted. These analyses should 

be expanded, however, to consider sources of 

uncertainty jointly rather than singly.

Partially 

applied

EPA expanded the sensitivity 

analysis by, for example, 

considering how benefit 

estimates change according to 

different threshold assumptions, 

such as cut points—

concentrations of particulate 

matter below which there would 

be no benefit to further 

reductions. This sensitivity 

analysis examined how different 

assumed cut points would 

change the estimates of avoided 

cases of death. EPA did not, 

however, conduct a sensitivity 

analysis to consider the sources 

of uncertainty jointly rather than 

singly. 

EPA has begun to explore how to 

expand sensitivity analysis and 

consider sources of uncertainty 

jointly. EPA noted that in recent 

air pollution analyses, the agency 

considered sources of uncertainty 

jointly for the last stage of the 

benefit estimation process—

valuing the benefits in dollar 

terms—by doing a probabilistic 

assessment of the value of a 

statistical life. 

In addition, EPA’s policy and 

economics division has begun a 

long-term project to expand 

uncertainty analysis. EPA reports 

that one technique that 

considered how much different 

sources of uncertainty affect the 

estimates is not ready to be 

applied to a rule-making analysis. 

17 In presenting the probability distribution for 

each health benefit produced by a primary 

analysis, EPA should emphasize even more 

than it has in the past the sources of 

uncertainty that remain unaccounted for in the 

primary analysis. These uncertainties should 

continue to be described as completely and 

realistically as possible.

Partially 

applied

EPA did not present a probability 

distribution for the primary 

benefit analysis but included 

some discussion of sources of 

uncertainty not incorporated into 

the benefit estimates. EPA also 

referred the reader to a previous 

analysis involving particulate 

matter for detailed discussions 

on sources of uncertainty. The 

citations are incomplete, 

however, leaving readers to 

search within the voluminous 

document for the relevant 

information. 

EPA attributed incomplete or 

lacking references in the draft PM 

RIA to time constraints, noting 

that it would present clear 

references in the final version. 

EPA also clarified that it relies on 

references to other documents, 

rather than repeating information 

that has not changed, in order to 

keep the presentation to a 

manageable size. 

(See also recommendation 14 for 

EPA’s basis for not including 

probability distributions.)

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 26 GAO-06-780 Particulate Matter

18 EPA should consider providing a preliminary 

analysis that estimates, in current populations, 

the health benefits resulting from hypothetical 

changes to current levels of emissions. These 

preliminary analyses would help EPA develop 

an idea of the lower bound on the uncertainty 

of future consequences and would have fewer 

uncertainties than analyses of the impacts of 

proposed regulatory actions on future 

exposures and health outcomes.

Applied According to EPA, the agency 

considered providing the 

preliminary analysis described in 

this recommendation. 

EPA determined that this 

information would not provide 

meaningful information to the 

draft PM benefit analysis. Aside 

from its questions about the 

technical feasibility of responding 

to this recommendation, EPA 

expressed doubts that doing this 

analysis for current populations 

would establish a lower bound on 

the uncertainty of future 

consequences. One EPA official 

concluded that the analysis could 

introduce more uncertainty into 

its benefit estimates because 

benefit projections will depend on 

controls implemented in the 

future. Finally, EPA clarified that, 

as provided by Circular A-4, it 

estimates benefits for a future 

year when the regulatory 

revisions become effective, not 

the current year, which would not 

account for changes in variables 

other than emissions.

19 EPA should continue to strive to present the 

results of its health benefit analyses in ways 

that avoid conveying an unwarranted degree of 

certainty. Such ways include rounding to fewer 

significant digits, increasing the use of graphs, 

presenting projected baselines along with 

projected health benefits, and placing less 

emphasis on single numbers (for example, the 

mean of the probability distribution for a health 

benefit) and greater emphasis on ranges (for 

example, the range between 5th and 95th 

percentiles of the distribution).

Applied EPA followed some of the 

National Academies’ 

suggestions to present the data 

in a way that avoids conveying an 

unwarranted level of certainty. 

EPA rounded the estimates to 

fewer significant digits. EPA 

increased the emphasis on 

ranges by presenting some data 

in ranges of benefit and cost 

estimates. EPA summarized the 

primary benefit estimates in a 

table. 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 27 GAO-06-780 Particulate Matter

20 EPA should perform similar detailed analyses 

of uncertainty in the valuation of health benefits 

and in the regulatory cost analyses that the 

committee recommends for the health benefit 

analyses.

Partially 

applied

The draft PM RIA demonstrated 

some steps toward analysis of 

cost uncertainty but did not 

present an uncertainty analysis 

of the benefit valuation—the 

stage when it assigns a dollar 

value to the benefit estimates. In 

terms of cost uncertainty, EPA 

presented a qualitative 

discussion of the uncertainties 

about costs and the expected 

impact on the cost estimates but 

did not perform a formal 

uncertainty analysis for the 

costs. 

Overall, EPA cited technical 

challenges, including data 

limitations, as the primary reason 

for not applying a formal 

uncertainty analysis to the cost 

estimates in the PM RIA. 

According to EPA, the number 

and breadth of PM sources 

considered under the National 

Ambient Air Quality Standards 

rule poses a challenge to 

development of a characterization 

of all costs and their 

uncertainties. Moreover, EPA 

noted that the collection process 

for cost information is not 

systematic, and there is a limited 

amount of information about the 

cost of implementing some types 

of controls. Even when costs are 

known, it may be difficult to 

specify the underlying probability 

distributions. 

EPA told us that it will incorporate 

more refined information, such as 

emissions control analysis, 

affecting cost estimates in the 

final RIA. EPA expects the final 

RIA to include more reasonable 

cutoffs such that cost estimates 

will not be based on ineffective 

controls and excessive costs (i.e., 

controls that are not likely to be 

used). 

In terms of health valuation 

uncertainty, EPA has made 

progress toward a formal 

uncertainty analysis by doing a 

probabilistic assessment of the 

value of a statistical life. EPA 

conducted this analysis in prior 

RIAs but did not have enough 

time to complete model runs in 

time for the draft RIA. EPA plans 

to incorporate an uncertainty 

analysis related to the

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix II

Recommendations Applied or Partially 

Applied to the Draft Particulate Matter 

Regulatory Impact Analysis 

Page 28 GAO-06-780 Particulate Matter

Source: GAO analysis of National Academies and EPA information.

mortality valuation in the final PM 

RIA.

EPA has focused its expansion of 

uncertainty analysis more on 

benefits than on costs, in part 

because the benefit estimates for 

air rules tend to be much larger. 

EPA identified efforts to analyze 

cost uncertainty factors, including 

sensitivity analyses in previous 

RIAs and plans for accountability 

analysis comparing estimated 

costs to actual costs. Finally, EPA 

pointed out that the work it 

conducts to understand the 

benefits also carries over to the 

cost estimates. 

21 EPA should provide health benefit estimates in 

ways that will support multiple kinds of 

analysis, including various approaches to 

mortality valuation and aggregation of benefits 

using quality-adjusted life years.

Partially 

applied

EPA’s detailed breakdown of the 

benefit estimates and the 

reference to its benefit model 

allows others to apply various 

approaches to mortality 

valuation, such as alternative 

estimates of the value of a 

statistical life. EPA did not, 

however, aggregate benefits 

using quality-adjusted life years.

Although EPA used the qualityadjusted life years approach in a 

prior RIA to aggregate benefits, 

EPA did not use this approach in 

the draft PM RIA because of time 

constraints. EPA told us that it 

plans to aggregate the benefits 

using quality-adjusted life years in 

the final PM RIA.

22 Each analysis should provide results according 

to demographic or other subgroups when the 

expected changes in pollution and, thus, the 

health benefits are not distributed uniformly 

across the population. This information would 

aid decision makers in situations in which 

equity issues might be involved.

Partially 

applied

EPA included some health 

benefit estimates for specific age 

groups in an appendix, but the 

agency did not present the 

primary benefit estimates 

according to age groups or other 

demographic factors. (See also 

recommendation 13.)

EPA intends to present in the final 

RIA a detailed summary table 

with the age breakdown of its 

sample population. The agency 

noted that it will not present the 

benefits according to other 

demographic factors, such as 

race and income, because of 

political sensitivities and data 

limitations. EPA clarified that it 

lacks the data on local conditions 

but that it intends to overcome 

this limitation using tract-level 

data—which captures details 

about local conditions—from the 

U.S. Census data in future 

analyses. 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Page 29 GAO-06-780 Particulate Matter

Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact Analysis 

ApendxiI

Table 2 provides a summary of the National Academies’ recommendations 

that EPA has not applied to its draft regulatory impact analysis (RIA) for 

particulate matter (PM). This table provides GAO’s assessment of EPA's 

progress to date regarding recommendations that required additional 

research and development, were deemed as not relevant to the PM 

National Ambient Air Quality Standards (NAAQS) by the agency, or were 

not included in the draft PM RIA due to time and resource constraints. The 

final column characterizes EPA’s comments regarding each 

recommendation, including contextual information, potential impediments 

to application, justification for not addressing the recommendation, and 

intended next steps, if applicable.
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 30 GAO-06-780 Particulate Matter

Table 2: Recommendations Not Applied to the Draft Particulate Matter Regulatory Impact Analysis

National Academies’ recommendation Status GAO assessment EPA’s response

1 The uncertainty in emissions estimates should 

be quantified and carried through the health 

benefit analysis to the calculation of avoided 

cases of mortality and morbidity.

Not applied—

research and 

development 

under way

EPA has not quantified the 

uncertainty related to emissions 

because of limited data and 

computational complexities and has 

therefore not yet carried such 

uncertainty through in the health 

benefit analysis.

EPA stated that the 

application of this 

recommendation—

conducting a formal analysis 

of emissions uncertainty—

requires long-term research 

and development. EPA 

reports that it discussed the 

possibility of conducting a 

quantitative uncertainty 

analysis for emissions in its 

particulate matter analysis, 

but the fixed timeline 

prevented EPA from doing 

this work. The primary 

challenge stems from the 

nature of the emissions 

inventory—data are collected 

from a plethora of entities, 

complicating the agency’s 

ability to evaluate 

uncertainty. 

EPA told us that, currently, 

the only way to assess 

emissions uncertainty is 

through qualitative means. 

EPA also stated that its final 

particulate matter analysis 

will demonstrate steps 

toward this recommendation 

because it will present a 

sensitivity analysis of the 

emissions data and will 

present emissions data 

according to the level emitted 

by the different kinds of 

sources, such as utilities, 

cars, and trucks.
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 31 GAO-06-780 Particulate Matter

2 Because a regulation to improve air quality 

may affect pathways other than air, EPA should 

determine whether there are likely to be any 

important indirect impacts of a regulation on 

human health and the environment. If any such 

impacts are identified, EPA should include in 

the analysis a plan to assess them more 

completely.

Not applied—

research and 

development 

under way

While EPA did not provide 

information in the draft PM RIA to 

show that it considered indirect 

impacts involving pathways other 

than air, EPA’s ongoing research on 

the environmental impacts may 

identify important indirect impacts. 

EPA stated that in past rules, 

the agency has looked at 

indirect impacts in terms of 

deposition of nitrogen and 

sulfates to sensitive water 

bodies. EPA plans to 

incorporate this analysis in 

the final PM RIA. EPA said it 

has not yet identified any 

other indirect impacts. While 

an EPA official suggested 

that this recommendation did 

not seem relevant to the 

NAAQS analysis in terms of 

human health impacts—EPA 

could not determine how 

human health would be 

affected by exposure to PM 

from pathways other than 

air—the agency is 

conducting research to 

identify important indirect 

impacts on the environment. 

EPA characterized 

environmental impacts as an 

area of research. 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 32 GAO-06-780 Particulate Matter

3 EPA has typically made the assumption of 

equivalent potency across particle types 

because of insufficient scientific information. 

As more data become available, EPA should 

strengthen its benefit analyses by evaluating a 

range of alternative assumptions regarding 

relative particle toxicity and incorporate these 

assumptions in sensitivity or uncertainty 

analyses.

Not applied—

research and 

development 

under way

Although EPA assumed equivalent 

toxicity in the PM RIA and did not 

include related sensitivity or 

uncertainty analyses, EPA is 

sponsoring research directed at 

incorporating its findings on relative 

toxicity into future analyses.

EPA stated that it does not 

have sufficient information to 

distinguish between particle 

components for the final rule. 

EPA is funding long-term 

research on relative toxicity, 

including technical studies to 

understand any differential 

toxicities as well as economic 

analyses to explore ways to 

characterize the uncertainty 

in benefit estimates. For 

example, an EPA contractor 

conducted a sensitivity 

analysis of relative toxicity of 

particle components, 

including carbons, nitrates, 

crustal material, and sulfates. 

To date, the contractor has 

created a model to assess 

whether and how much these 

sources of uncertainty may 

affect benefit estimates in 

one urban area. EPA is also 

supporting research to 

explore relative toxicity 

through its intramural 

research program, its five 

Particulate Matter Research 

Centers, and the Health 

Effects Institute, an 

organization funded in part 

by EPA. EPA’s science grant 

program recently awarded 

$40 million to the five 

Particulate Matter Research 

Centers.

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 33 GAO-06-780 Particulate Matter

4 As it incorporates additional sources of 

uncertainty into its primary health benefit 

analyses, EPA should consider conducting 

analyses to determine which uncertainty 

sources have the greatest influence on the 

mean and spread of the probability distribution. 

The need for these sensitivity analyses will be 

particularly great for distributions that are 

based on expert judgment. The uncertainty 

sources that have the greatest consequences 

for decision making, including those that have 

the greatest impact on the spread of the 

distribution, should be given high priority for 

additional research.

Not applied—

research and 

development 

under way 

In the draft PM RIA, although EPA 

presented a qualitative discussion 

about the importance of its 

assumptions that impact uncertainty, 

it did not consider which sources of 

uncertainty have the greatest 

influence on the mean and spread of 

the probability distribution. EPA is 

sponsoring research to incorporate 

influence analysis in future analyses. 

(See recommendation 14; EPA did 

not specify probability distributions 

for uncertainty in its primary 

analysis). 

 

An EPA contractor is 

researching techniques for 

influence analysis, an 

expanded form of uncertainty 

analysis that would 

determine which uncertainty 

sources have the greatest 

influence on the benefit 

estimates. The influence 

analysis work targeted three 

sources of uncertainty, 

including the concentrationresponse function (which 

involves threshold and 

slope); lag effects; and 

relative toxicity. The 

contractor compiled a draft 

report discussing techniques 

to conduct this analysis and 

incorporate the uncertainty 

analysis in benefit estimates.

EPA and the contractor 

stated that because these 

techniques are in the 

exploratory stage, it is 

premature to apply this work 

to a specific rule-making 

analysis. For example, EPA 

cited the need to determine 

how much uncertainty is 

explained by differential 

toxicity or by different 

thresholds. Finally, EPA 

stated that because it 

focused its resources on the 

expert elicitation work (see 

recommendation 14), the 

influence analysis received 

fewer resources and has not 

advanced as quickly. 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 34 GAO-06-780 Particulate Matter

5 EPA should estimate the benefits over the 

regulatory time period, including both the 

implementation period and the expression 

period of all important health effects. Because 

calculating benefits for every future year is 

resource intensive and unlikely to show true 

increases in precision, calculations can be 

made, for example, every fifth year with simple 

interpolation techniques applied to estimate 

benefits for intervening years.

Not applied EPA did not estimate benefits 

covering the implementation period 

and expression period of all 

important health effects. EPA 

estimated the benefits expected in 

one year only—the base case year, 

2015. The year 2015 is the first 

attainment date when states should 

be in compliance with the new 

standards. The Clean Air Act allows 

for up to a 5-year extension for 

states that cannot meet the 

standards by the attainment year. 

Therefore, the implementation 

period for some states may extend to 

2020. 

EPA cited limited time and 

resources to estimate 

benefits for years other than 

2015, but plans to also 

include estimates for 2020, in 

the final RIA. In addition, EPA 

did not think that this 

recommendation is 

particularly meaningful to the 

NAAQS analysis because 

the only variable change over 

the course of the PM 

implementation period is 

population. EPA concluded 

that there would not be much 

difference between the 

benefit estimates given in 5-

year increments (i.e., 2010 

and 2015). 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 35 GAO-06-780 Particulate Matter

6 The components of emissions estimates (such 

as number of vehicles in a class, average miles 

traveled per vehicle, and emissions per mile) 

should be presented with and without 

implementation of the regulation at the national 

level. This will help readers judge how 

reasonable these predictions are and will 

suggest which components of emissions 

estimates drive the emissions reductions 

associated with the regulation. Historical 

trends in these components should also be 

presented.

Not applied EPA presented a qualitative 

discussion about emissions, but did 

not address components of 

emissions estimates or provide 

information to allow readers to 

understand which components of 

emissions estimates drive the 

reductions associated with 

regulation, such as activity level or 

emissions intensity. EPA did not 

present historical trends. 

EPA stated that the kind of 

information called for by this 

recommendation—key input 

data, assumptions, and 

intermediate modeling 

outcomes—would be useful 

in future rule-making 

analyses, but that the agency 

needs to review a large 

amount of data to determine 

which elements would be 

most helpful. EPA questioned 

the value of incorporating all 

of the recommended 

information in regulatory 

impact analyses, noting that 

it works to present enough 

information to readers while 

maintaining a document of 

manageable length. 

EPA stated that it needs time 

to design data reporting 

strategies that would be

appropriate for the different 

scales and scopes of the 

regulatory impact analyses. 

EPA plans to explore this 

recommendation more 

thoroughly as part of its 

comprehensive economic 

analysis of the Clean Air Act. 

Finally, EPA officials 

suggested that the final PM 

RIA will respond to this 

recommendation in part by 

providing information about 

how emissions reductions 

might vary across PM 

sources in order to show the 

primary drivers of emissions 

reductions in the final RIA. 

EPA also plans to compare 

current data to the historical 

trends predicted in the past 

and show that EPA’s current 

predictions are for the future. 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 36 GAO-06-780 Particulate Matter

7 EPA should quantify uncertainties with regard 

to future population distributions and 

background disease rates. EPA should also 

summarize what is known about the potential 

importance of disease interactions and 

competing risks affecting the health outcomes 

of primary interest and discuss the possible 

biases that might be introduced in the final 

analysis by changes in those factors.

Not applied EPA did not quantify the 

uncertainties related to future 

population distributions and 

background disease rates. EPA did 

not summarize what is known about 

the potential importance of disease 

interactions and competing risks 

affecting health outcomes of primary 

interest. 

EPA said its models are not 

configured to quantify these 

sources of uncertainty. EPA 

would need to modify the 

model in order to do this 

analysis. Agency officials 

noted that the agency tries to 

be selective when 

determining which sources of 

uncertainty to assess 

because the cost of doing 

this work might outweigh the 

value added from the 

information.

In addition, EPA officials said 

the agency does not have all 

of the data necessary to 

reconfigure the models to 

quantify key sources of 

uncertainty. EPA disagrees 

with the National Academies’ 

comment that lack of 

information should not 

preclude the quantification of 

uncertainty. EPA believes 

that using the techniques to 

quantify uncertainty without 

empirical data would 

generate results that could 

be more misleading than the 

results that do not account 

for uncertainty. EPA plans to 

add relevant uncertainty 

characterizations as it 

obtains data.

8 The lack of clear categorization of severity of 

certain health outcomes in benefits analyses 

has implications for the quantification and the 

valuation of these outcomes. Although EPA 

has made some attempt to recognize this 

issue, it should continue to develop and 

improve methods used to reconcile differences 

between the severity of disease described in 

air pollution epidemiology and that commonly 

used to develop estimates of background 

disease prevalence and incidence.

Not applied The model EPA used to estimate 

benefits, BenMAP, did not account 

for the potential variations in severity 

of illnesses and prevalence and 

expected incidence of health effects. 

EPA has begun work on 

categorizing severity of some 

health outcomes, including 

chronic bronchitis and 

asthma incidence. EPA is 

continuing to refine its 

BenMAP model to better 

quantify and monetize health 

outcomes.

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 37 GAO-06-780 Particulate Matter

9 EPA should give more emphasis to the 

assessment, presentation, and communication 

of changes in morbidity. Although often difficult 

to quantify, these factors may begin to play a 

more dominant role in benefit analysis if the 

value assigned to mortality decreases.

Not applied EPA did not place more emphasis on 

the assessment, presentation, and 

communication of changes in 

morbidity. Prior RIAs have quantified 

expected changes in morbidity, such 

as reductions in asthma and chronic 

bronchitis. The draft PM RIA did not 

include additional morbidity 

information. Moreover, the main 

benefit estimates in the draft PM RIA 

did not include morbidity estimates. 

EPA acknowledged the 

importance of morbidity—

i.e., illness—benefits and is 

working to expand that 

analysis. For example, EPA 

told us that it plans to include 

sensitivity analysis of 

changes in illnesses in the 

final PM RIA. EPA also 

stated that the final PM RIA 

will reflect updates to its 

model—EPA is working to 

include projections that will 

allow the agency to evaluate 

expected changes in 

illnesses such as asthma and 

chronic bronchitis. 

10 There is a common misperception that a high 

degree of certainty is required for regulatory 

actions to take place to protect public health. 

As a result, primary health benefit analyses 

that more fully and accurately portray the 

uncertainties might not be considered useful. It 

is unrealistic for EPA to defer decisions until it 

can make them on the basis of perfect science. 

A careful and deliberate balancing of the 

benefits and costs is required, and this 

balancing must be informed by a fair 

assessment of the current levels of uncertainty 

and a realistic evaluation of the likely 

reductions in uncertainty attainable through 

further research.

Not applied EPA did not balance the costs and 

benefits—the Clean Air Act prohibits 

EPA from basing revisions to the 

NAAQS on costs. We note that in 

response to other recommendations, 

EPA has taken steps toward 

assessing current levels of 

uncertainty—see appendix II, 

recommendations 14 and 20. 

EPA characterized this 

recommendation as not 

germane to the particulate 

matter regulatory impact 

analysis because the Clean 

Air Act prohibits the agency 

from considering the costs 

when revising NAAQS. 

11 EPA should provide a summary of the analysis 

containing information as outlined in the 

National Academies’ report (table 6-1). This 

information would allow the reader to evaluate 

the study design and verify estimates obtained 

in the analysis.

Not applied EPA did not provide the summary 

table in the executive summary as 

outlined in the National Academies’ 

report, including a description of 

regulatory options, boundaries of 

analysis, regulatory baseline, and 

assumptions that have a significant 

impact on results of analysis.

Due to data and time 

constraints, EPA did not 

summarize its conclusions in 

the executive summary of the 

PM RIA. 

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Appendix III

Recommendations Not Applied to the Draft 

Particulate Matter Regulatory Impact 

Analysis 

Page 38 GAO-06-780 Particulate Matter

Source: GAO analysis of National Academies and EPA information.

12 To enhance the quality of future regulatory 

benefit analyses, a standing, independent 

technical review panel should advise EPA in 

the initial stages of its benefit analysis. This 

panel should have expertise in regulatory 

options analysis, emissions and exposure 

assessment, toxicology, epidemiology, risk 

analysis, biostatistics, and economics and 

should be appointed with strict attention to 

avoiding conflict of interest, balancing biases 

and ensuring broad representation. This panel 

should be supported by permanent technical 

staff to ensure consistency of reviews over 

time. EPA should follow the panel’s guidance 

on the need for peer review.

Not applied EPA did not convene a standing 

group of experts to guide the 

agency’s initial work on the draft PM 

RIA. In the course of developing 

economic methodologies, EPA 

sought and considered information 

from existing advisory committees 

on the soundness of certain 

assumptions, such as cessation 

lags. 

EPA stated that in addition to 

the advice it sought from the 

National Academies, the 

agency continues to seek 

input from independent 

committees, including the 

Clean Air Scientific Advisory 

Committee, the Advisory 

Council on Clean Air 

Compliance Analysis, and 

subcommittees chartered by 

the Science Advisory Board’s 

Environmental Economics 

Advisory Committee. The 

committees advised EPA on 

data, methods, and modeling 

choices applicable to various 

economic analyses, including 

the draft PM RIA. 

EPA stated that the costs 

involved in convening an 

entirely new panel could be 

prohibitively expensive, due 

to organizing costs and travel 

expenses and the scope of 

the Academies’ 

recommendation. As a result, 

EPA said that it has 

attempted to take advantage 

of existing groups, such as 

the Advisory Council on 

Clean Air Compliance 

Analysis, rather than arrange 

for a new panel.

Furthermore, EPA noted that 

new methodologies and 

assumptions used in the PM 

RIA were peer-reviewed and 

that previously used methods 

used in the RIA had already 

been reviewed and validated 

in a prior context.

(Continued From Previous Page)

National Academies’ recommendation Status GAO assessment EPA’s response
Page 39 GAO-06-780 Particulate Matter

Appendix IV

GAO Contact and Staff Acknowledgments ApendxiIV

GAO Contact John B. Stephenson, (202) 512-3841 or stephensonj@gao.gov

Staff 

Acknowledgments

In addition to the contact named above, Christine Fishkin, Assistant 

Director; Kate Cardamone; Nancy Crothers; Cindy Gilbert; Tim Guinane; 

Jessica Lemke; and Meaghan K. Marshall made key contributions to this 

report. Timothy Bober, Marcia Crosse, and Karen Keegan also made 

important contributions.

(360660)
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To 

have GAO e-mail you a list of newly posted products every afternoon, go to 

www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional 

Relations

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548

Public Affairs Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548