Particulate Matter: EPA Has Started to Address the National Academies' Recommendations on Estimating Health Benefits, but More Progress Is Needed
Released: 2006-07-19T00:00:00

Letter

Background

EPA is required by the Clean Air Act to conduct reviews of the National Ambient Air Quality Standards (NAAQS) for the six criteria pollutants, including particulate matter, every 5 years. The overarching purpose of such reviews is to determine whether the current standards are sufficient to protect public health and welfare at large, with an adequate margin of safety, given the latest scientific information available at the time of the review. Major steps in the NAAQS process include the following: developing a criteria document that synthesizes new research on health preparing a staff paper that assesses the policy implications of the scientific information in the criteria document, which also discusses possible ranges for air quality standards; and determining whether and how EPA should revise the NAAQS.

If EPA decides to revise the NAAQS, the agency proposes the changes in the Federal Register. As part of the federal rule-making process, EPA is to comply with Executive Order 12866, which directs federal agencies to analyze the costs and benefits of proposed and final rules expected to affect the economy by $100 million or more per year. In September 2003, the Office of Management and Budget (OMB) issued its Circular A-4, which presents guidance and best practices and states that agencies should analyze the costs and benefits in accordance with the principles of full disclosure and transparency. Further, in cases such as the particulate matter rule, where expected economic impacts exceed $1 billion annually, Circular A-4 also states that agencies should conduct a comprehensive assessment of key uncertainties in their analyses of costs and benefits, which EPA also refers to as regulatory impact analyses. EPA’s January 2006 regulatory impact analysis presents estimates of the costs and benefits for the proposed particulate matter rule.

The focus of the National Academies’ 2002 report was on how EPA estimates the health benefits of its proposed air regulations. To develop such estimates, EPA conducts analyses to quantify the expected changes in the number of deaths and illnesses that are likely to result from proposed regulations. The regulatory impact analyses also estimate the costs associated with implementing proposed air regulations, although, under the Clean Air Act, EPA is not permitted to consider costs in setting health- based standards for the criteria air pollutants, such as particulate matter.

Soon after the National Academies issued its report in 2002, EPA staff identified key recommendations and developed a strategy, in consultation with OMB, to apply some of the recommendations to benefit analyses for air pollution regulations under consideration at the time. EPA roughly approximated the time and resource requirements to respond to the recommendations, identifying those the agency could address within 2 or 3 years and those that would take longer. According to EPA officials, the agency focused primarily on the numerous recommendations related to analyzing uncertainty.

Both the National Academies’ report and the OMB guidance emphasize the need for agencies to account for uncertainties and to maintain transparency in the course of conducting benefit analyses. Identifying and accounting for uncertainties in these analyses can help decision makers evaluate the likelihood that certain regulatory decisions will achieve the estimated benefits. Transparency is important because it enables the public and relevant decision makers to see clearly how EPA arrived at its estimates and conclusions. In prior work on regulatory impact analyses, we have found shortcomings in EPA’s analyses of uncertainty and the information the agency provides with its estimates of costs and benefits.

EPA Is in the Process of Addressing Many of the Academies’ Recommendations

EPA applied—either wholly or in part—approximately two-thirds of the Academies’ recommendations to its January 2006 regulatory impact analysis and continues to address the recommendations through ongoing research and development. The January 2006 regulatory impact analysis demonstrated progress toward an expanded analysis of uncertainty and consideration of different assumptions. EPA officials cited time and resource constraints, as well as the need to mitigate complex technical challenges, as the basis for not applying other recommendations. According to EPA officials, the agency did not apply some of the more complex recommendations because it had not achieved sufficient progress in the research and development projects under way.

EPA Applied, at Least in Part, about Two-thirds of the Recommendations to Its Particulate Matter Health Benefit Analysis in the Proposed Rule

The January 2006 regulatory impact analysis on particulate matter represents a snapshot of an ongoing EPA effort to respond to the National Academies’ recommendations on developing estimates of health benefits for air pollution regulations. Specifically, the agency applied, at least in part, approximately two-thirds of the recommendations—8 were applied and 14 were partially applied—by taking steps toward conducting a more rigorous assessment of uncertainty for proposed air pollution regulations by, for example, evaluating the different assumptions about the link between human exposure to particulate matter and health effects and discussing sources of uncertainty not included in the benefit estimates. According to EPA officials, the agency focused much of its time and resources on the recommendations related to uncertainty. In particular, one overarching recommendation suggests that EPA take steps toward conducting a formal, comprehensive uncertainty analysis—the systematic application of mathematical techniques, such as Monte Carlo simulation— and include the uncertainty analysis in the regulatory impact analysis to provide a “more realistic depiction of the overall uncertainty” in EPA’s estimates of the benefits. A number of the other recommendations regarding uncertainty are aimed at EPA’s developing the information and methodologies needed to carry out a comprehensive uncertainty analysis.

Overall, the uncertainty recommendations suggest that EPA should determine (1) which sources of uncertainties have the greatest effect on benefit estimates and (2) the degree to which the uncertainties affect the estimates by specifying a range of estimates and the likelihood of attaining them. In response, EPA devoted significant resources to applying an alternative technique called expert elicitation in a multiphased pilot project. The pilot project was designed to systematically obtain expert advice to begin to better incorporate in its health benefit analysis the uncertainty underlying the causal link between exposure to particulate matter and premature death. EPA used the expert elicitation process to help it more definitively evaluate the uncertainty associated with estimated reductions in premature death—estimates that composed 85 percent to 95 percent of EPA’s total health benefit estimates for air pollution regulations in the past 5 years, according to the agency. EPA developed a range of expected reductions in death rates based on expert opinion systematically gathered in its pilot expert elicitation project and provided the results of this supplemental analysis in an appendix to the regulatory impact analysis. However, the National Academies had recommended that EPA merge such supplemental analyses into the main benefit analysis.

Moreover, the Academies recommended that EPA’s main benefit analysis reflect how the benefit estimates would vary in light of uncertainties. In addition to the uncertainty underlying the causal link between exposure and premature death that EPA analyzed, other key uncertainties can influence the estimates. For example, there is uncertainty about the effects of the age and health status of people exposed to particulate matter, the varying composition of particulate matter, and the measurements of actual exposure to particulate matter. EPA’s health benefit analysis, however, does not account for these key uncertainties by specifying a range of estimates and the likelihood of attaining them, similar to estimates derived from the expert elicitation addressing causal uncertainty. For these reasons, EPA’s responses reflect a partial application of the Academies’ recommendation.

In addition, the Academies recommended that EPA both continue to conduct sensitivity analyses on sources of uncertainty and expand these analyses. In the particulate matter regulatory impact analysis, EPA included a new sensitivity analysis regarding assumptions about thresholds, or levels below which those exposed to particulate matter are not at risk of experiencing harmful effects. EPA has assumed no threshold level exists—that is, any exposure poses potential health risks. Some experts have suggested that different thresholds may exist and the National Academies recommended that EPA determine how changing its assumption—that no threshold exists—would influence the estimates. The sensitivity analysis EPA provided in the regulatory impact analysis examined how its estimates of expected health benefits would change assuming varying thresholds.

Another recommendation that EPA is researching and partially applied to the draft regulatory impact analysis concerns alternative assumptions about cessation lags—the time between reductions in exposure to particulate matter and the health response. The National Academies made several recommendations on this topic, including one that EPA incorporate alternative assumptions about lags into a formal uncertainty analysis to estimate benefits that account for the likelihood of different lag durations. In response, EPA has sought advice from its Advisory Council on Clean Air Compliance Analysis on how to address this recommendation and has conducted a series of sensitivity analyses related to cessation lags. EPA is also funding research to explore ways to address lag effects in its uncertainty analysis. According to an EPA official, specifying the probability of different lag effects is computationally complex, and the agency is working to resolve this challenge.

In response to another recommendation by the National Academies, EPA identified some of the sources of uncertainty that are not reflected in its benefit estimates. For example, EPA’s regulatory impact analysis disclosed that its benefit estimates do not reflect the uncertainty associated with future year projections of particulate matter emissions. EPA presented a qualitative description about emissions uncertainty, elaborating on technical reasons—such as the limited information about the effectiveness of particulate matter control programs—why the analysis likely underestimates future emissions levels. EPA also applied the Academies’ recommendation on the presentation of uncertainty, which encouraged the agency to present the results of its health benefit analyses in ways that convey the estimated benefits more realistically by, for example, placing less emphasis on single estimates and rounding the numbers. EPA’s regulatory impact analysis presented ranges for some of the benefit estimates. Also, EPA sought to convey the overall uncertainty of its benefit estimates in a qualitative manner by clearly stating that decision makers and the public should not place significant weight on the quantified benefit estimates in the regulatory impact analysis because of data limitations and uncertainties.

Another example of EPA’s response to the National Academies’ recommendations involves exploring the various regulatory choices available to decision makers. The Academies recommended that EPA estimate the health benefits representing the full range of regulatory choices available to decision makers. In the particulate matter analysis, EPA presented health benefits expected under several regulatory options targeting fine particulate matter. Citing a lack of data and tools needed to conduct an accurate analysis, EPA did not estimate the benefits expected under the proposed regulatory options for coarse particulate matter but, consistent with the National Academies’ recommendation, presented its rationale for not doing so. Overall, we considered this a partial application of the recommendation. (See app. II for more detail on the recommendations that EPA has applied or partially applied to the draft particulate matter regulatory impact analysis.)

EPA Plans to Address Some of the Remaining Recommendations in the Final Rule and Has Research and Development Under Way to Address Others

EPA did not apply the remaining 12 recommendations to the analysis for various reasons. While EPA applied some recommendations—either wholly or in part—that require additional studies, methodologies, or data to its particulate matter analysis, the agency had not made sufficient progress in addressing others and therefore did not apply them to the analysis. EPA officials viewed most of these recommendations as relevant to its health benefit analyses and, citing the need for additional research and development, emphasized the agency’s commitment to continue to respond to the recommendations. According to a senior EPA official, insufficient resources impeded the agency’s progress in applying the recommendations. This official cited limited availability of skilled staff, time, and other resources to conduct the required analyses and research and development. According to EPA, some of the more complex, long-term recommendations include the following: relying less on simplifying assumptions, such as the assumption that the various components of particulate matter have equal toxicity; conducting a formal assessment of the uncertainty of particulate matter emissions; and assessing the expected reduction of any harmful effects other than air pollution or human health problems.

For example, EPA is in the process of responding to a recommendation involving the relative toxicity of components of particulate matter, an emerging area of research that has the potential to influence EPA’s regulatory decisions in the future. Specifically, the agency could, hypothetically, refine national air quality standards to address the potentially varying health consequences associated with different components of particulate matter. The National Academies recommended that EPA strengthen its benefit analyses by evaluating a range of alternative assumptions regarding relative toxicity and incorporate these assumptions into sensitivity or uncertainty analyses as more data become available. EPA did not believe the state of scientific knowledge on relative toxicity was sufficiently developed at the time it prepared the draft regulatory impact analysis to include this kind of analysis. However, EPA is sponsoring research on this issue. For example, EPA is supporting long- term research on the relative toxicity of particulate matter components being conducted by EPA’s intramural research program, its five Particulate Matter Research Centers, and the Health Effects Institute, an organization funded in part by EPA. In addition, an EPA contractor has begun to investigate methods for conducting a formal analysis that would consider sources of uncertainty, including relative toxicity and lag effects. To date, the contractor has created a model to assess whether and how much these sources of uncertainty may affect benefit estimates in one urban area.

The National Academies also recommended that EPA incorporate an assessment of uncertainty into the early stages of its benefit analyses by characterizing the uncertainty of its emissions estimates on which the agency is going to base its benefit estimates. While the agency is investigating ways to assess or characterize this uncertainty, EPA did not conduct a formal uncertainty analysis for particulate matter emissions for the draft regulatory impact analysis because of data limitations. These limitations stem largely from the source of emissions data, the National Emissions Inventory, an amalgamation of data from a variety of entities, including state and local air agencies, tribes, and industry. According to EPA, these entities use different methods to collect data, which have different implications for how to characterize the uncertainty. Furthermore, the uncertainty associated with emissions varies by the source of emissions. For example, the analytical methods for evaluating the uncertainty of estimates of emissions from utilities would differ from those for car and truck emissions because the nature of these emissions and the data collection methods differ. In sum, to apply this recommendation, EPA must determine how to characterize the uncertainty of the estimates for each source of emissions before aggregating the uncertainty to a national level and then factoring that aggregation into its benefit estimates. According to EPA officials, the agency needs much more time to resolve the complex technical challenges of such an analysis. EPA officials also noted that the final particulate matter analysis will demonstrate steps toward this recommendation by presenting emissions data according to the level emitted by the different kinds of sources, such as utilities, cars, and trucks.

Another recommendation that EPA is researching but did not apply to the draft regulatory impact analysis concerns whether the proposed revisions to the particulate matter standards would have important indirect impacts on human health and the environment. According to an EPA official, the agency could not rule out the possibility that the revisions could have indirect impacts on the environment, such as whether reductions to particulate matter emissions would reduce the amount of particulate matter deposited in water bodies, thereby decreasing water pollution. EPA has considered indirect impacts of air pollution regulations on sensitive water bodies in the past and plans to include a similar analysis in the final particulate matter rule. An agency official further noted that ongoing research about environmental impacts could reveal additional indirect impacts for future analyses.

Other recommendations that EPA did not apply to its benefit estimates in the regulatory impact analysis concern issues such as transparency and external review of EPA’s benefit estimation process. For example, the National Academies recommended that EPA clearly summarize the key elements of the benefit analysis in an executive summary that includes a table that lists and briefly describes the regulatory options for which EPA estimated the benefits, the assumptions that had a substantial impact on the benefit estimates, and the health benefits evaluated. EPA did not, however, present a summary table as called for by the recommendation or summarize the benefits in the executive summary. As EPA stated in the particulate matter analysis, the agency decided not to present the benefit estimates in the executive summary because they were too uncertain. Specifically, officials said the agency was not able to resolve some significant data limitations before issuing the draft regulatory impact analysis in January 2006—a deadline driven by the need to meet the court- ordered issue date for the final rule in September 2006. According to EPA officials, EPA has resolved some of these data challenges by, for example, obtaining more robust data on anticipated strategies for reducing emissions, which will affect the estimates of benefits. The officials also said that EPA intends to include in the executive summary of the regulatory impact analysis supporting the final rule a summary table that describes key analytical information. EPA officials also acknowledged other presentation shortcomings, including references to key analytical elements that were insufficiently specific, that officials attributed to tight time frames and the demands of working on other regulatory analyses concurrently. They said they plan to address these shortcomings in the final regulatory impact analysis.

Regarding external review, the National Academies recommended that EPA establish an independent review panel, supported by permanent technical staff, to bolster EPA’s quality control measures for its regulatory impact analyses, such as the one for particulate matter. The National Academies noted that peer review of EPA’s regulatory impact analyses would be advantageous when the agency designs and conducts its economic analysis. EPA has not directly addressed this recommendation. According to the Director of the Office of Policy Analysis and Review in EPA’s Office of Air and Radiation, establishing and supporting independent committees is costly, making it important for EPA to take advantage of existing panels rather than set up new ones. Further, an official in the Office of Air and Radiation who oversees the development of regulatory impact analyses said that the cost of reviewing all regulatory impact analyses would be substantial. In this regard, EPA officials identified peer reviews the agency received from its existing independent committees, such as the Clean Air Scientific Advisory Committee and the Advisory Council on Clean Air Compliance. For example, to respond to the Academies’ recommendations about lag effects, EPA sought independent advice on the assumptions it was developing regarding the time between reduced exposure to particulate matter and reductions in incidences of health effects. Finally, EPA officials noted that although the agency does not have each regulatory impact analysis peer reviewed, EPA typically does have the methodologies that will be applied to regulatory impact analyses peer reviewed. (See app. III for more detail on these recommendations and others that EPA did not apply to the draft particulate matter regulatory impact analysis.)

Concluding Observations

While EPA has taken a number of steps to respond to the Academies’ recommendations on estimating health benefits, continued commitment and dedication of resources will be needed if EPA is to fully implement the improvements endorsed by the National Academies. In particular, the agency will need to ensure that it allocates resources to needed research on emerging issues, such as the relative toxicity of particulate matter components; assessing which sources of uncertainty have the greatest influence on benefit estimates; and estimating other benefits, such as environmental improvements. In addition, it is important for EPA to continue to improve its uncertainty analysis in accordance with the Academies’ recommendations. The agency’s draft regulatory impact analysis illustrates that estimates of health benefits can be highly uncertain. In fact, EPA officials viewed these estimates as so uncertain that they chose to not present them in the executive summary of the regulatory impact analysis. While EPA officials said they expect to reduce the uncertainties associated with the health benefit estimates in the final particulate matter analysis, robust uncertainty analysis will nonetheless be important for decision makers and the public to understand the likelihood of attaining the estimated health benefits. According to EPA officials, the final regulatory impact analysis on particulate matter will reflect further responsiveness to the Academies’ recommendations by, for example, providing additional sensitivity analysis and improving the transparency of the regulatory impact analysis by highlighting key data and assumptions in the executive summary. Moreover, these officials emphasized the agency’s commitment to further enhancing the transparency of the analysis by presenting clear and accurate references to the supporting technical documents, which detail the analytical assumptions and describe the data supporting the estimates. To the extent EPA continues to make progress addressing the Academies’ recommendations, decision makers and the public will be able to better evaluate the basis for EPA’s air regulations.

Agency Comments

We provided a draft of this report to EPA for review. EPA provided technical comments that we incorporated, as appropriate. Officials from the Office of Policy Analysis and Review within EPA’s Office of Air and Radiation noted in their technical comments that the report provides a fair and balanced representation of EPA’s efforts to apply the National Academies’ recommendations to the draft particulate matter regulatory impact analysis. However, these officials also cited progress made in applying the National Academies’ recommendations through analyses of other air programs and through research and development efforts. We note that this report does identify, as appropriate, EPA’s research and development efforts for recommendations EPA did not apply to the draft particulate matter analysis, its plans to apply some additional recommendations to the final particulate matter regulatory impact analysis, and the agency’s responses to recommendations in prior rule- making analyses of air programs.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the EPA Administrator and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3841 or stephensonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Scope and Methodology

We were asked to determine whether and how the Environmental Protection Agency (EPA) applied the National Academies (Academies) recommendations in its estimates of the health benefits expected from the January 2006 proposed revisions to the particulate matter national ambient air quality standards. In response to this objective, we assessed EPA’s response to the Academies’ recommendations and present an overview of the agency’s completed, ongoing, and planned actions addressing the recommendations. To develop this overview, we reviewed EPA’s particulate matter regulatory impact analysis, EPA’s economic analysis guidelines, and Office of Management and Budget (OMB) guidance on regulatory impact analysis. We also analyzed documentation addressing current and future agency efforts to address the recommendations, such as project planning memorandums and technical support documents discussing the application of economic techniques. In addition, we met with senior officials from EPA’s Office of Air and Radiation, which was responsible for developing the proposed rule and analyzing its economic effects, and with officials from EPA’s Office of Policy, Economics, and Innovation to discuss the agency’s responses to the recommendations. We interviewed several experts outside EPA, including (1) the Chair and other members of the National Academies’ Committee on Estimating the Health-Risk-Reduction Benefits of Proposed Air Pollution Regulations, to clarify the basis for their recommendations; and (2) economists at Resources for the Future, to discuss the technical issues underlying the recommendations on uncertainty analysis.

While the 2002 National Academies’ report is generally applicable to EPA air pollution regulations, our review focused on the application of the recommendations to the proposed revisions to the particulate matter standards, as requested. Our work focused on broadly characterizing EPA’s progress toward applying the recommendations; we did not evaluate the effectiveness or quality of the scientific and technical actions the agency has taken to apply them. To assess whether and how EPA has made progress in responding to the recommendations, we developed the following recommendation classification continuum: applied, partially applied, and not applied. The applied and partially applied categories refer to completed and initiated actions in EPA’s health benefit analysis of particulate matter that corresponds to components of the National Academies’ recommendations. The not applied category includes recommendations that EPA did not apply when conducting the analysis for the January 2006 particulate matter regulatory impact analysis and identifies those for which ongoing research and development efforts were not far enough along to apply to the particulate matter analysis. We performed our work from January 2006 to July 2006 in accordance with generally accepted government auditing standards.

Recommendations Applied or Partially Applied to the Draft Particulate Matter Regulatory Impact Analysis

Table 1 provides a summary of the National Academies’ recommendations that EPA has applied or partially applied to its draft regulatory impact analysis (RIA) for particulate matter (PM). This table also provides GAO’s assessment of EPA’s progress in applying each recommendation, in terms of steps EPA has taken thus far to address issues highlighted in the National Academies’ report. The final column characterizes EPA’s comments regarding each recommendation, including, as pertinent, contextual information, potential impediments to application, and intended next steps.

Recommendations Not Applied to the Draft Particulate Matter Regulatory Impact Analysis

Table 2 provides a summary of the National Academies’ recommendations that EPA has not applied to its draft regulatory impact analysis (RIA) for particulate matter (PM). This table provides GAO’s assessment of EPA's progress to date regarding recommendations that required additional research and development, were deemed as not relevant to the PM National Ambient Air Quality Standards (NAAQS) by the agency, or were not included in the draft PM RIA due to time and resource constraints. The final column characterizes EPA’s comments regarding each recommendation, including contextual information, potential impediments to application, justification for not addressing the recommendation, and intended next steps, if applicable.

GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Christine Fishkin, Assistant Director; Kate Cardamone; Nancy Crothers; Cindy Gilbert; Tim Guinane; Jessica Lemke; and Meaghan K. Marshall made key contributions to this report. Timothy Bober, Marcia Crosse, and Karen Keegan also made important contributions.