United States Government Accountability Office

GAO Testimony 

Before the Committee on Environment 

and Public Works, U.S. Senate 

PARTICULATE MATTER

EPA Needs to Make More 

Progress in Addressing the 

National Academies’ 

Recommendations on 

Estimating Health Benefits 

Statement of John B. Stephenson, Director 

Natural Resources and Environment 

For Release on Delivery

Expected at 9:00 a.m. EDT

Wednesday, July 19, 2006

 

GAO-06-992T 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

July 19, 2006

PARTICULATE MATTER 

EPA Needs to Make More Progress in 

Addressing the National Academies’ 

Recommendations on Estimating Health 

Benefits 

Highlights of GAO-06-992T, a testimony 

before the Committee on Environment and 

Public Works, U.S. Senate 

Scientific evidence links exposure 

to particulate matter—a 

widespread form of air pollution—

to serious health problems, 

including asthma and premature 

death. Under the Clean Air Act, the 

Environmental Protection Agency

(EPA) periodically reviews the

appropriate air quality level at

which to set national standards to

protect the public against the 

health effects of six pollutants,

including particulate matter. EPA 

proposed revisions to the 

particulate matter standards in

January 2006 and issued a 

regulatory impact analysis of the 

revisions’ expected costs and 

benefits. The estimated benefits of

air pollution regulations have been 

controversial in the past, and a 

2002 National Academies report to

EPA made recommendations 

aimed at improving the estimates 

for particulate matter and other air 

pollution regulations. 

This testimony is based on GAO’s 

July 2006 report Particulate 

Matter: EPA Has Started to 

Address the National Academies’ 

Recommendations on Estimating 

Health Benefits, but More Progress 

Is Needed (GAO-06-780). GAO 

determined whether and how EPA 

applied the National Academies’ 

recommendations in its estimates 

of the health benefits expected

from the January 2006 proposed

revisions to the particulate matter

standards. 

While the National Academies’ report generally supported EPA’s approach to 

estimating the health benefits of its proposed air pollution regulations, it 

included 34 recommendations for improvements. EPA has begun to change 

the way it conducts and presents its analyses of health benefits in response 

to the National Academies’ recommendations. For its particulate matter 

health benefit analysis, EPA applied, at least in part, about two-thirds of the 

Academies’ recommendations. Specifically, EPA applied 8 and partially 

applied 14. For example, in response to the Academies’ recommendations, 

EPA evaluated how benefits might change given alternative assumptions and 

discussed sources of uncertainty not included in the benefit estimates. 

Although EPA applied an alternative technique for evaluating one key 

uncertainty—the causal link between exposure to particulate matter and 

premature death—the health benefit analysis did not assess how the benefit

estimates would vary in light of other key uncertainties, as the Academies 

had recommended. Consequently, EPA’s response represents a partial 

application of the recommendation. Agency officials said that ongoing 

research and development efforts will allow EPA to gradually make more 

progress in applying this and other recommendations to future analyses. 

EPA did not apply the remaining 12 recommendations to the analysis, such 

as the recommendation to evaluate the impact of using the assumption that 

the components of particulate matter are equally toxic. EPA officials viewed 

most of these 12 recommendations as relevant to the health benefit analyses 

but noted that the agency was not ready to apply specific recommendations 

because of, among other things, the need to overcome technical challenges 

stemming from limitations in the state of available science. For example, 

EPA did not believe that the state of scientific knowledge on the relative 

toxicity of particulate matter components was sufficiently developed to 

include it in the January 2006 regulatory impact analysis. The agency is

sponsoring research on this issue. 

We note that continued commitment and dedication of resources will be 

needed if EPA is to fully implement the improvements recommended by the 

National Academies. In particular, the agency will need to ensure that it 

allocates resources to needed research on emerging issues, such as the 

relative toxicity of particulate matter components, and to assessing which 

sources of uncertainty have the greatest influence on benefit estimates. 

While EPA officials said they expect to reduce the uncertainties associated 

with the health benefit estimates in the final particulate matter analysis, a

robust uncertainty analysis of the remaining uncertainties will nonetheless 

be important for decision makers and the public to understand the likelihood 

of attaining the estimated health benefits. 

www.gao.gov/cgi-bin/getrpt?GAO-06-992T.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact John B. 

Stephenson at (202) 512-3841 or

stephensonj@gao.gov. 
Mr. Chairman and Members of the Committee: 

I am pleased to be here today as the committee considers the science and

risk assessment supporting the Environmental Protection Agency’s (EPA) 

proposed revisions to the national air quality standards for particulate 

matter. A large body of scientific evidence links exposure to particulate 

matter—a ubiquitous form of air pollution commonly referred to as soot—

to serious health problems, including asthma, chronic bronchitis, heart 

attack, and premature death. Under the Clean Air Act, EPA periodically 

reviews the appropriate air quality level at which to set national standards 

to protect the public against the health effects of particulate matter. As 

you are aware, EPA proposed revisions to the particulate matter standards 

in January 2006 and issued a draft regulatory impact analysis of the 

revisions’ expected costs and benefits. 

EPA’s estimates of the expected benefits from its air pollution regulations 

have often been controversial, and the methods the agency has used to 

prepare these estimates have been questioned. In 2000, at the direction of

the Senate Appropriations Committee, EPA asked the National Academies 

(Academies) to evaluate EPA’s overall methodology for estimating the 

health benefits of proposed air regulations. In 2002, the Academies issued 

a report that made recommendations focusing on conducting more 

rigorous assessments of uncertainty, increasing the transparency of how 

EPA estimates benefits, conducting more detailed analyses of exposure, 

and estimating the benefits of each regulatory option under consideration. 

My testimony summarizes the highlights of our report being released today 

on the extent to which EPA applied the recommendations made by the 

Academies to its January 2006 proposed revisions to the particulate matter 

standards.1

 Our report provides a more detailed discussion of each 

recommendation, including whether and how EPA applied it to the 

regulatory impact analysis on particulate matter.

While the National Academies’ report generally supported EPA’s overall 

approach to estimating benefits, it included 34 recommendations for 

improvements. EPA has begun to change the way it conducts and presents 

its analyses of health benefits in response to the National Academies’ 

Summary 

 1

See GAO, Particulate Matter: EPA Has Started to Address the National Academies’

Recommendations on Estimating Health Benefits, but More Progress Is Needed, 

GAO-06-780 (Washington, D.C.: July 14, 2006). 

Page 1 GAO-06-992T 
recommendations. In the case of the January 2006 proposed rule on 

particulate matter standards, EPA applied, at least in part, about twothirds of the recommendations to its particulate matter health benefit 

analysis; it applied 8 and partially applied 14 more. For example, in 

applying the recommendations, EPA evaluated how benefits might change

given alternative assumptions and discussed sources of uncertainty not 

included in the benefit estimates. In addition, EPA applied an alternative 

technique for evaluating one important source of uncertainty in its 

analysis—the uncertainty underlying the causal link between exposure to 

particulate matter and premature death. Consistent with the National 

Academies’ recommendation to assess uncertainty by developing ranges of 

estimates of benefits and specifying the likelihood of attaining those levels

of benefits, EPA systematically gathered expert opinions about this link 

and developed ranges reflecting the experts’ confidence in attaining 

reductions in premature death expected from the proposed revisions. 

However, the health benefit analysis did not assess how the benefit 

estimates would vary in light of other key uncertainties as the Academies

recommended. Consequently, EPA’s response represents a partial 

application of the recommendation. Agency officials told us that ongoing

research and development efforts will allow EPA to gradually make more 

progress in applying this and other recommendations to future analyses.

EPA did not apply the remaining 12 recommendations to the analysis, such

as the recommendation to evaluate the impact of using the assumption 

that the components of particulate matter are equally toxic. EPA officials 

viewed most of these 12 recommendations as relevant to its health benefit 

analyses but noted that the agency was not ready to apply specific 

recommendations because of, among other things, the need to overcome 

technical challenges stemming from limitations in the state of available 

science. For example, EPA did not believe that the state of scientific 

knowledge on the relative toxicity of particulate matter components was 

sufficiently developed to include it in the January 2006 regulatory impact

analysis, but the agency is sponsoring research on this issue. 

EPA is required by the Clean Air Act to conduct reviews of the National 

Ambient Air Quality Standards (NAAQS) for the six criteria pollutants, 

including particulate matter, every 5 years to determine whether the 

current standards are sufficient to protect public health, with an adequate

margin of safety. If EPA decides to revise the NAAQS, the agency proposes 

changes to the standards and estimates the costs and benefits expected 

from the revisions in an assessment called a regulatory impact analysis. In

January 2006, EPA prepared a regulatory impact analysis for one such 

Background 

Page 2 GAO-06-992T 
rule—particulate matter—that presented limited estimates of the costs 

and benefits expected to result from the proposed particulate matter rule. 

EPA developed the estimates by, for example, quantifying the changes in

the number of deaths and illnesses in five urban areas that are likely to 

result from the proposed rule. 

The National Academies’ 2002 report examined how EPA estimates the 

health benefits of its proposed air regulations and emphasized the need for 

EPA to account for uncertainties and maintain transparency in the course

of conducting benefit analyses. Identifying and accounting for

uncertainties in these analyses can help decision makers evaluate the 

likelihood that certain regulatory decisions will achieve the estimated 

benefits. Transparency is important because it enables the public and 

relevant decision makers to see clearly how EPA arrived at its estimates 

and conclusions. Many of the recommendations include qualifying 

language indicating that it is reasonable to expect that they can be applied

in stages, over time; moreover, a number of the recommendations are 

interrelated and, in some cases, overlapping. Soon after the National 

Academies issued its report, EPA roughly approximated the time and 

resource requirements to respond to the recommendations, identifying 

those the agency could address within 2 or 3 years and those that would 

take longer. According to EPA officials, the agency focused primarily on 

the numerous recommendations related to analyzing uncertainty. As is 

discussed below, EPA applied some of these recommendations to the 

particulate matter analysis. 

EPA applied—either wholly or in part—approximately two-thirds of the 

Academies’ recommendations in preparing its January 2006 particulate 

matter regulatory impact analysis and continues to address the 

recommendations through ongoing research and development. According

to EPA, the agency intends to address some of the remaining 

recommendations in the final rule and has undertaken research and 

development to address others. 

EPA Applied Some, 

but Not All, of the 

National Academies’ 

Recommendations to 

the Particulate Matter 

Regulatory Impact 

Analysis 

Page 3 GAO-06-992T 
The January 2006 regulatory impact analysis on particulate matter 

represents a snapshot of an ongoing EPA effort to respond to the National 

Academies’ recommendations on developing estimates of health benefits 

for air pollution regulations. Specifically, the agency applied, at least in 

part, approximately two-thirds of the recommendations—8 were applied

and 14 were partially applied—by taking steps toward conducting a more 

rigorous assessment of uncertainty by, for example, evaluating the 

different assumptions about the link between human exposure to 

particulate matter and health effects and discussing sources of uncertainty 

not included in the benefit estimates. According to EPA officials, the 

agency focused much of its time and resources on the recommendations 

related to uncertainty. In particular, one overarching recommendation 

suggests that EPA take steps toward conducting a formal, comprehensive 

uncertainty analysis—the systematic application of mathematical 

techniques, such as Monte Carlo simulation—and include the uncertainty

analysis in the regulatory impact analysis to provide a “more realistic 

depiction of the overall uncertainty” in EPA’s estimates of the benefits.2

Recommendations EPA 

Applied or Partially 

Applied to Its Particulate 

Matter Health Benefit 

Analysis 

Overall, the uncertainty recommendations call for EPA to determine (1) 

which sources of uncertainties have the greatest effect on benefit 

estimates and (2) the degree to which the uncertainties affect the 

estimates by specifying a range of estimates and the likelihood of attaining 

them. In response, EPA examined a key source of uncertainty—its 

assumption about the causal link between exposure to particulate matter 

and premature death—and presented a range of expected reductions in 

death rates. EPA based these ranges on expert opinion systematically 

gathered in a multiphased pilot project. The agency did not, however, 

incorporate these ranges into its benefit estimates as the National 

Academies had recommended. 

Moreover, the Academies recommended that EPA’s benefit analysis reflect 

how the benefit estimates would vary in light of multiple uncertainties. In 

addition to the uncertainty underlying the causal link between exposure 

and premature death, other key uncertainties can influence the estimates. 

For example, there is uncertainty about the effects of the age and health

 2

Monte Carlo simulation refers to a computer-based analysis that uses probability 

distributions for key variables, selects random values from each of the distributions

simultaneously, and repeats the random selection over and over. Rather than presenting a

single outcome—such as the mostly likely or average scenario—Monte Carlo simulations

produce a distribution of outcomes that reflect the probability distributions of modeled 

uncertain variables. 

Page 4 GAO-06-992T 
status of people exposed to particulate matter, the varying composition of 

particulate matter, and the measurements of actual exposure to particulate 

matter. EPA’s health benefit analysis, however, does not account for these 

key uncertainties by specifying a range of estimates and the likelihood of 

attaining them. For these reasons, EPA’s responses reflect a partial 

application of the Academies’ recommendation. 

In addition, the Academies recommended that EPA both continue to 

conduct sensitivity analyses on sources of uncertainty and expand these 

analyses. In the particulate matter regulatory impact analysis, EPA 

included a new sensitivity analysis regarding assumptions about 

thresholds, or levels below which those exposed to particulate matter are 

not at risk of experiencing harmful effects. EPA has assumed no threshold 

level exists—that is, any exposure poses potential health risks.3

 Some 

experts have suggested that different thresholds may exist, and the 

National Academies recommended that EPA determine how changing its 

assumption—that no threshold exists—would influence the estimates. The 

sensitivity analysis EPA provided in the regulatory impact analysis 

examined how its estimates of expected health benefits would change 

assuming varying thresholds. 

In response to another recommendation by the National Academies, EPA 

identified some of the sources of uncertainty that are not reflected in its 

benefit estimates. For example, EPA’s regulatory impact analysis disclosed 

that its benefit estimates do not reflect the uncertainty associated with 

future year projections of particulate matter emissions. EPA presented a 

qualitative description about emissions uncertainty, elaborating on 

technical reasons—such as the limited information about the effectiveness 

of particulate matter control programs—why the analysis likely 

underestimates future emissions levels. 

Recommendations EPA 

Did Not Apply to the 

Particulate Matter Analysis 

EPA did not apply the remaining 12 recommendations to the analysis for 

various reasons. Agency officials viewed most of these recommendations 

as relevant to its health benefit analyses and, citing the need for additional 

research and development, emphasized the agency’s commitment to 

continue to respond to the recommendations. EPA has undertaken 

 3

Recent EPA analyses used the natural background concentrations of particulate matter, 

rather than zero, for its assumption of no threshold level. The National Academies

supported the assumption of no threshold level, but it recommended that EPA conduct a

consistent and transparent sensitivity analysis to consider various threshold levels. 

Page 5 GAO-06-992T 
research and development to respond to some of these recommendations 

but, according to agency officials, did not apply them to the analysis 

because the agency had not made sufficient progress. 

For example, EPA is in the process of responding to a recommendation 

involving the relative toxicity4

 of components of particulate matter, an 

emerging area of research that has the potential to influence EPA’s 

regulatory decisions in the future.5

 Hypothetically, the agency could refine

national air quality standards to address the potentially varying health

consequences associated with different components of particulate matter. 

The National Academies recommended that EPA strengthen its benefit 

analyses by evaluating a range of alternative assumptions regarding 

relative toxicity and incorporate these assumptions into sensitivity or 

uncertainty analyses as more data become available.6

 EPA did not believe 

the state of scientific knowledge on relative toxicity was sufficiently 

developed at the time it prepared the draft regulatory impact analysis to 

include this kind of analysis. In a separate report issued in 2004, the

National Academies noted that technical challenges have impeded 

research progress on relative toxicity but nonetheless identified this issue 

as a priority research topic. The Clean Air Scientific Advisory Committee 

also noted the need for more research and concluded in 2005 that not 

enough data are available to base the particulate matter standards on 

composition. The Office of Management and Budget, however, encouraged

EPA in 2006 to conduct a sensitivity analysis on relative toxicity and 

referred the agency to a sensitivity analysis on relative toxicity funded by

the European Commission. 

 4

Particulate matter is a highly complex mixture comprising particles emitted directly from 

sources and particles formed through atmospheric chemical reactions. Particles span many

sizes and shapes and consist of hundreds of different chemicals. EPA identifies the major

components of fine particulate matter as carbon, sulfate and nitrate compounds, and 

crustal/metallic materials such as soil and ash. 

5

Relative toxicity refers to the premise that different components of particulate matter have 

different levels of potency affecting premature mortality and illness. In the draft particulate 

matter regulatory impact analysis, EPA assumed equivalent toxicity, stating that “while it is

reasonable to expect that the potency of components may vary across the numerous effect

categories associated with particulate matter, EPA’s interpretation of scientific information 

considered to date is that such information does not yet provide a basis for quantification 

beyond using fine particle mass.” EPA, Draft Regulatory Impact Analysis for the PM-2.5 

National Ambient Air Quality Standards (Washington, D.C., 2006), 3-21. 

6

In the context of the National Academies’ recommendations, a sensitivity analysis would 

assess how changes in one or more variables affect the outcome, whereas a comprehensive 

or formal uncertainty analysis evaluates the probability distributions of multiple variables. 

Page 6 GAO-06-992T 
We found that EPA is sponsoring research on the relative toxicity of 

particulate matter components. For example, EPA is supporting long-term 

research on this issue through its intramural research program and is also 

funding research through its five Particulate Matter Research Centers and 

the Health Effects Institute. In addition, an EPA contractor has begun to 

investigate methods for conducting a formal analysis that would consider

sources of uncertainty, including relative toxicity. To date, the contractor

has created a model to assess whether and how much these sources of 

uncertainty may affect benefit estimates in one urban area. Agency

officials told us, however, that this work was not sufficiently developed to

include in the final particulate matter analysis, which it says will present 

benefits on a national scale. 

Another recommendation that EPA did not apply to the particulate matter 

analysis focused on assessing the uncertainty of particulate matter 

emissions. The National Academies recommended that EPA conduct a 

formal analysis to characterize the uncertainty of its emissions estimates, 

which serve as the basis for its benefit estimates.7

 While the agency is 

investigating ways to assess or characterize this uncertainty, EPA did not 

conduct a formal uncertainty analysis for particulate matter emissions for 

the draft regulatory impact analysis because of data limitations. These 

limitations stem largely from the source of emissions data, the National 

Emissions Inventory8

—an amalgamation of data from a variety of entities,

including state and local air agencies, tribes, and industry. According to 

EPA, these entities use different methods to collect data, which have 

different implications for how to characterize the uncertainty. EPA 

officials stated that the agency needs much more time to address this data 

limitation and to resolve other technical challenges of such an analysis. 

While the final particulate matter analysis will not include a formal 

assessment of uncertainty about emissions levels, EPA officials noted that

the final analysis will demonstrate steps toward this recommendation by 

presenting emissions data according to the level emitted by the different 

kinds of sources, such as utilities, cars, and trucks. 

 7

Because the precise levels of total emissions are not knowable but rather are 

approximations based on a sample of measurements, there is uncertainty about the true

quantity of emissions. 

8

EPA compiles the National Emissions Inventory, a national database of air emissions data 

that includes estimates of annual emissions, by source, of air pollutants in each area of the 

country on an annual basis. 

Page 7 GAO-06-992T 
Finally, EPA did not apply a recommendation concerning the transparency 

of its benefit estimation process to the particulate matter analysis. 

Specifically, the National Academies recommended that EPA clearly 

summarize the key elements of the benefit analysis in an executive 

summary that includes a table that lists and briefly describes the 

regulatory options for which EPA estimated the benefits, the assumptions 

that had a substantial impact on the benefit estimates, and the health

benefits evaluated. EPA did not, however, present a summary table as 

called for by the recommendation or summarize the benefits in the 

executive summary. EPA stated in the regulatory impact analysis that the 

agency decided not to present the benefit estimates in the executive

summary because they were too uncertain. Agency officials told us that 

the agency could not resolve some significant data limitations before 

issuing the draft regulatory impact analysis in January 2006 but that EPA 

has resolved some of these data challenges. For example, EPA officials 

said they have obtained more robust data on anticipated strategies for 

reducing emissions, which will affect the estimates of benefits. The 

officials also said that EPA intends to include in the executive summary of 

the regulatory impact analysis supporting the final rule a summary table 

that describes key analytical information. 

While EPA officials said that the final regulatory impact analysis on 

particulate matter will reflect further responsiveness to the Academies’ 

recommendations, continued commitment and dedication of resources 

will be needed if EPA is to fully implement the improvements

recommended by the National Academies. In particular, the agency will 

need to ensure that it allocates resources to needed research on emerging

issues, such as the relative toxicity of particulate matter components, and 

to assessing which sources of uncertainty have the greatest influence on

benefit estimates. The uncertainty of the agency’s estimates of health 

benefits in the draft regulatory impact analysis for particulate matter 

underscores the importance of uncertainty analysis that can enable 

decision makers and the public to better evaluate the basis for EPA’s air 

regulations. While EPA officials said they expect to reduce the

uncertainties associated with the health benefit estimates in the final 

particulate matter analysis, a robust uncertainty analysis of the remaining 

uncertainties will nonetheless be important for decision makers and the 

public to understand the likelihood of attaining the estimated health 

benefits. 

Concluding 

Observations 

Page 8 GAO-06-992T 
Mr. Chairman, this concludes my prepared statement. I would be happy to

respond to any questions that you or other Members of the Committee 

may have. 

For further information about this testimony, please contact me at (202) 

512-3841 or stephensonj@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this statement. Individuals who made key contributions to this 

statement include Christine Fishkin, Assistant Director; Kate Cardamone; 

Nancy Crothers; Cindy Gilbert; Tim Guinane; Karen Keegan; Jessica 

Lemke; and Meaghan K. Marshall. 

Page 9 GAO-06-992T 

GAO Contact and 

Staff 

Acknowledgments 

(360741) 
This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “Subscribe to Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Mail or Phone 

To Report Fraud,

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER