Particulate Matter: EPA Needs to Make More Progress in Addressing the National Academies' Recommendations on Estimating Health Benefits
Released: 2006-07-19T00:00:00

Letter

Background

EPA is required by the Clean Air Act to conduct reviews of the National Ambient Air Quality Standards (NAAQS) for the six criteria pollutants, including particulate matter, every 5 years to determine whether the current standards are sufficient to protect public health, with an adequate margin of safety. If EPA decides to revise the NAAQS, the agency proposes changes to the standards and estimates the costs and benefits expected from the revisions in an assessment called a regulatory impact analysis. In January 2006, EPA prepared a regulatory impact analysis for one such rule—particulate matter—that presented limited estimates of the costs and benefits expected to result from the proposed particulate matter rule. EPA developed the estimates by, for example, quantifying the changes in the number of deaths and illnesses in five urban areas that are likely to result from the proposed rule.

The National Academies’ 2002 report examined how EPA estimates the health benefits of its proposed air regulations and emphasized the need for EPA to account for uncertainties and maintain transparency in the course of conducting benefit analyses. Identifying and accounting for uncertainties in these analyses can help decision makers evaluate the likelihood that certain regulatory decisions will achieve the estimated benefits. Transparency is important because it enables the public and relevant decision makers to see clearly how EPA arrived at its estimates and conclusions. Many of the recommendations include qualifying language indicating that it is reasonable to expect that they can be applied in stages, over time; moreover, a number of the recommendations are interrelated and, in some cases, overlapping. Soon after the National Academies issued its report, EPA roughly approximated the time and resource requirements to respond to the recommendations, identifying those the agency could address within 2 or 3 years and those that would take longer. According to EPA officials, the agency focused primarily on the numerous recommendations related to analyzing uncertainty. As is discussed below, EPA applied some of these recommendations to the particulate matter analysis.

EPA Applied Some, but Not All, of the National Academies’ Recommendations to the Particulate Matter Regulatory Impact Analysis

EPA applied—either wholly or in part—approximately two-thirds of the Academies’ recommendations in preparing its January 2006 particulate matter regulatory impact analysis and continues to address the recommendations through ongoing research and development. According to EPA, the agency intends to address some of the remaining recommendations in the final rule and has undertaken research and development to address others.

Recommendations EPA Applied or Partially Applied to Its Particulate Matter Health Benefit Analysis

The January 2006 regulatory impact analysis on particulate matter represents a snapshot of an ongoing EPA effort to respond to the National Academies’ recommendations on developing estimates of health benefits for air pollution regulations. Specifically, the agency applied, at least in part, approximately two-thirds of the recommendations—8 were applied and 14 were partially applied—by taking steps toward conducting a more rigorous assessment of uncertainty by, for example, evaluating the different assumptions about the link between human exposure to particulate matter and health effects and discussing sources of uncertainty not included in the benefit estimates. According to EPA officials, the agency focused much of its time and resources on the recommendations related to uncertainty. In particular, one overarching recommendation suggests that EPA take steps toward conducting a formal, comprehensive uncertainty analysis—the systematic application of mathematical techniques, such as Monte Carlo simulation—and include the uncertainty analysis in the regulatory impact analysis to provide a “more realistic depiction of the overall uncertainty” in EPA’s estimates of the benefits.

Overall, the uncertainty recommendations call for EPA to determine (1) which sources of uncertainties have the greatest effect on benefit estimates and (2) the degree to which the uncertainties affect the estimates by specifying a range of estimates and the likelihood of attaining them. In response, EPA examined a key source of uncertainty—its assumption about the causal link between exposure to particulate matter and premature death—and presented a range of expected reductions in death rates. EPA based these ranges on expert opinion systematically gathered in a multiphased pilot project. The agency did not, however, incorporate these ranges into its benefit estimates as the National Academies had recommended.

Moreover, the Academies recommended that EPA’s benefit analysis reflect how the benefit estimates would vary in light of multiple uncertainties. In addition to the uncertainty underlying the causal link between exposure and premature death, other key uncertainties can influence the estimates. For example, there is uncertainty about the effects of the age and health status of people exposed to particulate matter, the varying composition of particulate matter, and the measurements of actual exposure to particulate matter. EPA’s health benefit analysis, however, does not account for these key uncertainties by specifying a range of estimates and the likelihood of attaining them. For these reasons, EPA’s responses reflect a partial application of the Academies’ recommendation.

In addition, the Academies recommended that EPA both continue to conduct sensitivity analyses on sources of uncertainty and expand these analyses. In the particulate matter regulatory impact analysis, EPA included a new sensitivity analysis regarding assumptions about thresholds, or levels below which those exposed to particulate matter are not at risk of experiencing harmful effects. EPA has assumed no threshold level exists—that is, any exposure poses potential health risks. Some experts have suggested that different thresholds may exist, and the National Academies recommended that EPA determine how changing its assumption—that no threshold exists—would influence the estimates. The sensitivity analysis EPA provided in the regulatory impact analysis examined how its estimates of expected health benefits would change assuming varying thresholds.

In response to another recommendation by the National Academies, EPA identified some of the sources of uncertainty that are not reflected in its benefit estimates. For example, EPA’s regulatory impact analysis disclosed that its benefit estimates do not reflect the uncertainty associated with future year projections of particulate matter emissions. EPA presented a qualitative description about emissions uncertainty, elaborating on technical reasons—such as the limited information about the effectiveness of particulate matter control programs—why the analysis likely underestimates future emissions levels.

Recommendations EPA Did Not Apply to the Particulate Matter Analysis

EPA did not apply the remaining 12 recommendations to the analysis for various reasons. Agency officials viewed most of these recommendations as relevant to its health benefit analyses and, citing the need for additional research and development, emphasized the agency’s commitment to continue to respond to the recommendations. EPA has undertaken research and development to respond to some of these recommendations but, according to agency officials, did not apply them to the analysis because the agency had not made sufficient progress.

For example, EPA is in the process of responding to a recommendation involving the relative toxicity of components of particulate matter, an emerging area of research that has the potential to influence EPA’s regulatory decisions in the future. Hypothetically, the agency could refine national air quality standards to address the potentially varying health consequences associated with different components of particulate matter. The National Academies recommended that EPA strengthen its benefit analyses by evaluating a range of alternative assumptions regarding relative toxicity and incorporate these assumptions into sensitivity or uncertainty analyses as more data become available. EPA did not believe the state of scientific knowledge on relative toxicity was sufficiently developed at the time it prepared the draft regulatory impact analysis to include this kind of analysis. In a separate report issued in 2004, the National Academies noted that technical challenges have impeded research progress on relative toxicity but nonetheless identified this issue as a priority research topic. The Clean Air Scientific Advisory Committee also noted the need for more research and concluded in 2005 that not enough data are available to base the particulate matter standards on composition. The Office of Management and Budget, however, encouraged EPA in 2006 to conduct a sensitivity analysis on relative toxicity and referred the agency to a sensitivity analysis on relative toxicity funded by the European Commission.

We found that EPA is sponsoring research on the relative toxicity of particulate matter components. For example, EPA is supporting long-term research on this issue through its intramural research program and is also funding research through its five Particulate Matter Research Centers and the Health Effects Institute. In addition, an EPA contractor has begun to investigate methods for conducting a formal analysis that would consider sources of uncertainty, including relative toxicity. To date, the contractor has created a model to assess whether and how much these sources of uncertainty may affect benefit estimates in one urban area. Agency officials told us, however, that this work was not sufficiently developed to include in the final particulate matter analysis, which it says will present benefits on a national scale.

Another recommendation that EPA did not apply to the particulate matter analysis focused on assessing the uncertainty of particulate matter emissions. The National Academies recommended that EPA conduct a formal analysis to characterize the uncertainty of its emissions estimates, which serve as the basis for its benefit estimates. While the agency is investigating ways to assess or characterize this uncertainty, EPA did not conduct a formal uncertainty analysis for particulate matter emissions for the draft regulatory impact analysis because of data limitations. These limitations stem largely from the source of emissions data, the National Emissions Inventory—an amalgamation of data from a variety of entities, including state and local air agencies, tribes, and industry. According to EPA, these entities use different methods to collect data, which have different implications for how to characterize the uncertainty. EPA officials stated that the agency needs much more time to address this data limitation and to resolve other technical challenges of such an analysis. While the final particulate matter analysis will not include a formal assessment of uncertainty about emissions levels, EPA officials noted that the final analysis will demonstrate steps toward this recommendation by presenting emissions data according to the level emitted by the different kinds of sources, such as utilities, cars, and trucks.

Finally, EPA did not apply a recommendation concerning the transparency of its benefit estimation process to the particulate matter analysis. Specifically, the National Academies recommended that EPA clearly summarize the key elements of the benefit analysis in an executive summary that includes a table that lists and briefly describes the regulatory options for which EPA estimated the benefits, the assumptions that had a substantial impact on the benefit estimates, and the health benefits evaluated. EPA did not, however, present a summary table as called for by the recommendation or summarize the benefits in the executive summary. EPA stated in the regulatory impact analysis that the agency decided not to present the benefit estimates in the executive summary because they were too uncertain. Agency officials told us that the agency could not resolve some significant data limitations before issuing the draft regulatory impact analysis in January 2006 but that EPA has resolved some of these data challenges. For example, EPA officials said they have obtained more robust data on anticipated strategies for reducing emissions, which will affect the estimates of benefits. The officials also said that EPA intends to include in the executive summary of the regulatory impact analysis supporting the final rule a summary table that describes key analytical information.

Concluding Observations

While EPA officials said that the final regulatory impact analysis on particulate matter will reflect further responsiveness to the Academies’ recommendations, continued commitment and dedication of resources will be needed if EPA is to fully implement the improvements recommended by the National Academies. In particular, the agency will need to ensure that it allocates resources to needed research on emerging issues, such as the relative toxicity of particulate matter components, and to assessing which sources of uncertainty have the greatest influence on benefit estimates. The uncertainty of the agency’s estimates of health benefits in the draft regulatory impact analysis for particulate matter underscores the importance of uncertainty analysis that can enable decision makers and the public to better evaluate the basis for EPA’s air regulations. While EPA officials said they expect to reduce the uncertainties associated with the health benefit estimates in the final particulate matter analysis, a robust uncertainty analysis of the remaining uncertainties will nonetheless be important for decision makers and the public to understand the likelihood of attaining the estimated health benefits.

Mr. Chairman, this concludes my prepared statement. I would be happy to respond to any questions that you or other Members of the Committee may have.

GAO Contact and Staff Acknowledgments

For further information about this testimony, please contact me at (202) 512-3841 or stephensonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this statement include Christine Fishkin, Assistant Director; Kate Cardamone; Nancy Crothers; Cindy Gilbert; Tim Guinane; Karen Keegan; Jessica Lemke; and Meaghan K. Marshall.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.