Particulate Matter: EPA Needs to Make More Progress in Addressing the National Academies' Recommendations on Estimating Health Benefits
Published: 2006-07-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Scientific evidence links exposure to particulate matter--a widespread form of air pollution--to serious health problems, including asthma and premature death. Under the Clean Air Act, the Environmental Protection Agency (EPA) periodically reviews the appropriate air quality level at which to set national standards to protect the public against the health effects of six pollutants, including particulate matter. EPA proposed revisions to the particulate matter standards in January 2006 and issued a regulatory impact analysis of the revisions' expected costs and benefits. The estimated benefits of air pollution regulations have been controversial in the past, and a 2002 National Academies report to EPA made recommendations aimed at improving the estimates for particulate matter and other air pollution regulations. This testimony is based on GAO's July 2006 report Particulate Matter: EPA Has Started to Address the National Academies' Recommendations on Estimating Health Benefits, but More Progress Is Needed (GAO-06-780). GAO determined whether and how EPA applied the National Academies' recommendations in its estimates of the health benefits expected from the January 2006 proposed revisions to the particulate matter standards.
section_title: What GAO Found
paragraphs: While the National Academies' report generally supported EPA's approach to estimating the health benefits of its proposed air pollution regulations, it included 34 recommendations for improvements. EPA has begun to change the way it conducts and presents its analyses of health benefits in response to the National Academies' recommendations. For its particulate matter health benefit analysis, EPA applied, at least in part, about two-thirds of the Academies' recommendations. Specifically, EPA applied 8 and partially applied 14. For example, in response to the Academies' recommendations, EPA evaluated how benefits might change given alternative assumptions and discussed sources of uncertainty not included in the benefit estimates. Although EPA applied an alternative technique for evaluating one key uncertainty--the causal link between exposure to particulate matter and premature death--the health benefit analysis did not assess how the benefit estimates would vary in light of other key uncertainties, as the Academies had recommended. Consequently, EPA's response represents a partial application of the recommendation. Agency officials said that ongoing research and development efforts will allow EPA to gradually make more progress in applying this and other recommendations to future analyses. EPA did not apply the remaining 12 recommendations to the analysis, such as the recommendation to evaluate the impact of using the assumption that the components of particulate matter are equally toxic. EPA officials viewed most of these 12 recommendations as relevant to the health benefit analyses but noted that the agency was not ready to apply specific recommendations because of, among other things, the need to overcome technical challenges stemming from limitations in the state of available science. For example, EPA did not believe that the state of scientific knowledge on the relative toxicity of particulate matter components was sufficiently developed to include it in the January 2006 regulatory impact analysis. The agency is sponsoring research on this issue. We note that continued commitment and dedication of resources will be needed if EPA is to fully implement the improvements recommended by the National Academies. In particular, the agency will need to ensure that it allocates resources to needed research on emerging issues, such as the relative toxicity of particulate matter components, and to assessing which sources of uncertainty have the greatest influence on benefit estimates. While EPA officials said they expect to reduce the uncertainties associated with the health benefit estimates in the final particulate matter analysis, a robust uncertainty analysis of the remaining uncertainties will nonetheless be important for decision makers and the public to understand the likelihood of attaining the estimated health benefits.

section_title: Letter
paragraphs: I am pleased to be here today as the committee considers the science and  risk assessment supporting the Environmental Protection Agency’s (EPA)  proposed revisions to the national air quality standards for particulate  matter. A large body of scientific evidence links exposure to particulate  matter—a ubiquitous form of air pollution commonly referred to as soot— to serious health problems, including asthma, chronic bronchitis, heart  attack, and premature death. Under the Clean Air Act, EPA periodically  reviews the appropriate air quality level at which to set national standards  to protect the public against the health effects of particulate matter. As  you are aware, EPA proposed revisions to the particulate matter standards  in January 2006 and issued a draft regulatory impact analysis of the  revisions’ expected costs and benefits.
EPA’s estimates of the expected benefits from its air pollution regulations  have often been controversial, and the methods the agency has used to  prepare these estimates have been questioned. In 2000, at the direction of  the Senate Appropriations Committee, EPA asked the National Academies  (Academies) to evaluate EPA’s overall methodology for estimating the  health benefits of proposed air regulations. In 2002, the Academies issued  a report that made recommendations focusing on conducting more  rigorous assessments of uncertainty, increasing the transparency of how  EPA estimates benefits, conducting more detailed analyses of exposure,  and estimating the benefits of each regulatory option under consideration.  My testimony summarizes the highlights of our report being released today  on the extent to which EPA applied the recommendations made by the  Academies to its January 2006 proposed revisions to the particulate matter  standards. Our report provides a more detailed discussion of each  recommendation, including whether and how EPA applied it to the  regulatory impact analysis on particulate matter.
subsections: section_title: Background
paragraphs: EPA is required by the Clean Air Act to conduct reviews of the National  Ambient Air Quality Standards (NAAQS) for the six criteria pollutants,  including particulate matter, every 5 years to determine whether the  current standards are sufficient to protect public health, with an adequate  margin of safety. If EPA decides to revise the NAAQS, the agency proposes  changes to the standards and estimates the costs and benefits expected  from the revisions in an assessment called a regulatory impact analysis. In  January 2006, EPA prepared a regulatory impact analysis for one such  rule—particulate matter—that presented limited estimates of the costs  and benefits expected to result from the proposed particulate matter rule.  EPA developed the estimates by, for example, quantifying the changes in  the number of deaths and illnesses in five urban areas that are likely to  result from the proposed rule.
The National Academies’ 2002 report examined how EPA estimates the  health benefits of its proposed air regulations and emphasized the need for  EPA to account for uncertainties and maintain transparency in the course  of conducting benefit analyses. Identifying and accounting for  uncertainties in these analyses can help decision makers evaluate the  likelihood that certain regulatory decisions will achieve the estimated  benefits. Transparency is important because it enables the public and  relevant decision makers to see clearly how EPA arrived at its estimates  and conclusions. Many of the recommendations include qualifying  language indicating that it is reasonable to expect that they can be applied  in stages, over time; moreover, a number of the recommendations are  interrelated and, in some cases, overlapping. Soon after the National  Academies issued its report, EPA roughly approximated the time and  resource requirements to respond to the recommendations, identifying  those the agency could address within 2 or 3 years and those that would  take longer. According to EPA officials, the agency focused primarily on  the numerous recommendations related to analyzing uncertainty. As is  discussed below, EPA applied some of these recommendations to the  particulate matter analysis.
section_title: EPA Applied Some, but Not All, of the National Academies’ Recommendations to the Particulate Matter Regulatory Impact Analysis
paragraphs: EPA applied—either wholly or in part—approximately two-thirds of the  Academies’ recommendations in preparing its January 2006 particulate  matter regulatory impact analysis and continues to address the  recommendations through ongoing research and development. According  to EPA, the agency intends to address some of the remaining  recommendations in the final rule and has undertaken research and  development to address others.
subsections: section_title: Recommendations EPA Applied or Partially Applied to Its Particulate Matter Health Benefit Analysis
paragraphs: The January 2006 regulatory impact analysis on particulate matter  represents a snapshot of an ongoing EPA effort to respond to the National  Academies’ recommendations on developing estimates of health benefits  for air pollution regulations. Specifically, the agency applied, at least in  part, approximately two-thirds of the recommendations—8 were applied  and 14 were partially applied—by taking steps toward conducting a more  rigorous assessment of uncertainty by, for example, evaluating the  different assumptions about the link between human exposure to  particulate matter and health effects and discussing sources of uncertainty  not included in the benefit estimates. According to EPA officials, the  agency focused much of its time and resources on the recommendations  related to uncertainty. In particular, one overarching recommendation  suggests that EPA take steps toward conducting a formal, comprehensive  uncertainty analysis—the systematic application of mathematical  techniques, such as Monte Carlo simulation—and include the uncertainty  analysis in the regulatory impact analysis to provide a “more realistic  depiction of the overall uncertainty” in EPA’s estimates of the benefits.
Overall, the uncertainty recommendations call for EPA to determine (1)  which sources of uncertainties have the greatest effect on benefit  estimates and (2) the degree to which the uncertainties affect the  estimates by specifying a range of estimates and the likelihood of attaining  them. In response, EPA examined a key source of uncertainty—its  assumption about the causal link between exposure to particulate matter  and premature death—and presented a range of expected reductions in  death rates. EPA based these ranges on expert opinion systematically  gathered in a multiphased pilot project. The agency did not, however,  incorporate these ranges into its benefit estimates as the National  Academies had recommended.
Moreover, the Academies recommended that EPA’s benefit analysis reflect  how the benefit estimates would vary in light of multiple uncertainties. In  addition to the uncertainty underlying the causal link between exposure  and premature death, other key uncertainties can influence the estimates.  For example, there is uncertainty about the effects of the age and health  status of people exposed to particulate matter, the varying composition of  particulate matter, and the measurements of actual exposure to particulate  matter. EPA’s health benefit analysis, however, does not account for these  key uncertainties by specifying a range of estimates and the likelihood of  attaining them. For these reasons, EPA’s responses reflect a partial  application of the Academies’ recommendation.
In addition, the Academies recommended that EPA both continue to  conduct sensitivity analyses on sources of uncertainty and expand these  analyses. In the particulate matter regulatory impact analysis, EPA  included a new sensitivity analysis regarding assumptions about  thresholds, or levels below which those exposed to particulate matter are  not at risk of experiencing harmful effects. EPA has assumed no threshold  level exists—that is, any exposure poses potential health risks. Some  experts have suggested that different thresholds may exist, and the  National Academies recommended that EPA determine how changing its  assumption—that no threshold exists—would influence the estimates. The  sensitivity analysis EPA provided in the regulatory impact analysis  examined how its estimates of expected health benefits would change  assuming varying thresholds.
In response to another recommendation by the National Academies, EPA  identified some of the sources of uncertainty that are not reflected in its  benefit estimates. For example, EPA’s regulatory impact analysis disclosed  that its benefit estimates do not reflect the uncertainty associated with  future year projections of particulate matter emissions. EPA presented a  qualitative description about emissions uncertainty, elaborating on  technical reasons—such as the limited information about the effectiveness  of particulate matter control programs—why the analysis likely  underestimates future emissions levels.
section_title: Recommendations EPA Did Not Apply to the Particulate Matter Analysis
paragraphs: EPA did not apply the remaining 12 recommendations to the analysis for  various reasons. Agency officials viewed most of these recommendations  as relevant to its health benefit analyses and, citing the need for additional  research and development, emphasized the agency’s commitment to  continue to respond to the recommendations. EPA has undertaken  research and development to respond to some of these recommendations  but, according to agency officials, did not apply them to the analysis  because the agency had not made sufficient progress.
For example, EPA is in the process of responding to a recommendation  involving the relative toxicity of components of particulate matter, an  emerging area of research that has the potential to influence EPA’s  regulatory decisions in the future. Hypothetically, the agency could refine  national air quality standards to address the potentially varying health  consequences associated with different components of particulate matter.  The National Academies recommended that EPA strengthen its benefit  analyses by evaluating a range of alternative assumptions regarding  relative toxicity and incorporate these assumptions into sensitivity or  uncertainty analyses as more data become available. EPA did not believe  the state of scientific knowledge on relative toxicity was sufficiently  developed at the time it prepared the draft regulatory impact analysis to  include this kind of analysis. In a separate report issued in 2004, the  National Academies noted that technical challenges have impeded  research progress on relative toxicity but nonetheless identified this issue  as a priority research topic. The Clean Air Scientific Advisory Committee  also noted the need for more research and concluded in 2005 that not  enough data are available to base the particulate matter standards on  composition. The Office of Management and Budget, however, encouraged  EPA in 2006 to conduct a sensitivity analysis on relative toxicity and  referred the agency to a sensitivity analysis on relative toxicity funded by  the European Commission.
We found that EPA is sponsoring research on the relative toxicity of  particulate matter components. For example, EPA is supporting long-term  research on this issue through its intramural research program and is also  funding research through its five Particulate Matter Research Centers and  the Health Effects Institute. In addition, an EPA contractor has begun to  investigate methods for conducting a formal analysis that would consider  sources of uncertainty, including relative toxicity. To date, the contractor  has created a model to assess whether and how much these sources of  uncertainty may affect benefit estimates in one urban area. Agency  officials told us, however, that this work was not sufficiently developed to  include in the final particulate matter analysis, which it says will present  benefits on a national scale.
Another recommendation that EPA did not apply to the particulate matter  analysis focused on assessing the uncertainty of particulate matter  emissions. The National Academies recommended that EPA conduct a  formal analysis to characterize the uncertainty of its emissions estimates,  which serve as the basis for its benefit estimates. While the agency is  investigating ways to assess or characterize this uncertainty, EPA did not  conduct a formal uncertainty analysis for particulate matter emissions for  the draft regulatory impact analysis because of data limitations. These  limitations stem largely from the source of emissions data, the National  Emissions Inventory—an amalgamation of data from a variety of entities,  including state and local air agencies, tribes, and industry. According to  EPA, these entities use different methods to collect data, which have  different implications for how to characterize the uncertainty. EPA  officials stated that the agency needs much more time to address this data  limitation and to resolve other technical challenges of such an analysis.  While the final particulate matter analysis will not include a formal  assessment of uncertainty about emissions levels, EPA officials noted that  the final analysis will demonstrate steps toward this recommendation by  presenting emissions data according to the level emitted by the different  kinds of sources, such as utilities, cars, and trucks.
Finally, EPA did not apply a recommendation concerning the transparency  of its benefit estimation process to the particulate matter analysis.  Specifically, the National Academies recommended that EPA clearly  summarize the key elements of the benefit analysis in an executive  summary that includes a table that lists and briefly describes the  regulatory options for which EPA estimated the benefits, the assumptions  that had a substantial impact on the benefit estimates, and the health  benefits evaluated. EPA did not, however, present a summary table as  called for by the recommendation or summarize the benefits in the  executive summary. EPA stated in the regulatory impact analysis that the  agency decided not to present the benefit estimates in the executive  summary because they were too uncertain. Agency officials told us that  the agency could not resolve some significant data limitations before  issuing the draft regulatory impact analysis in January 2006 but that EPA  has resolved some of these data challenges. For example, EPA officials  said they have obtained more robust data on anticipated strategies for  reducing emissions, which will affect the estimates of benefits. The  officials also said that EPA intends to include in the executive summary of  the regulatory impact analysis supporting the final rule a summary table  that describes key analytical information.
section_title: Concluding Observations
paragraphs: While EPA officials said that the final regulatory impact analysis on  particulate matter will reflect further responsiveness to the Academies’  recommendations, continued commitment and dedication of resources  will be needed if EPA is to fully implement the improvements  recommended by the National Academies. In particular, the agency will  need to ensure that it allocates resources to needed research on emerging  issues, such as the relative toxicity of particulate matter components, and  to assessing which sources of uncertainty have the greatest influence on  benefit estimates. The uncertainty of the agency’s estimates of health  benefits in the draft regulatory impact analysis for particulate matter  underscores the importance of uncertainty analysis that can enable  decision makers and the public to better evaluate the basis for EPA’s air  regulations. While EPA officials said they expect to reduce the  uncertainties associated with the health benefit estimates in the final  particulate matter analysis, a robust uncertainty analysis of the remaining  uncertainties will nonetheless be important for decision makers and the  public to understand the likelihood of attaining the estimated health  benefits.
Mr. Chairman, this concludes my prepared statement. I would be happy to  respond to any questions that you or other Members of the Committee  may have.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For further information about this testimony, please contact me at (202)  512-3841 or stephensonj@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals who made key contributions to this  statement include Christine Fishkin, Assistant Director; Kate Cardamone;  Nancy Crothers; Cindy Gilbert; Tim Guinane; Karen Keegan; Jessica  Lemke; and Meaghan K. Marshall.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.