Wildland Fire Management: Agencies' Efforts to Assess Program Effectiveness and Modernize the Firefighting Aviation Fleet
Released: 2015-11-17T00:00:00

Letter

Agencies Assess the Effectiveness of Their Programs in Several Ways but Have Not Consistently Conducted Reviews That Could Improve Responses to Fires

In our September 2015 report, we found that the agencies assess the effectiveness of their wildland fire management programs in several ways, including through performance measures, efforts to assess specific activities, and reviews of specific wildland fire incidents. We found that both the Forest Service and Interior were developing new performance measures, in part to help better assess the results of their current emphasis on risk-based management, according to agency officials. In addition, the agencies have undertaken multiple efforts to assess the effectiveness of activities such as fuel reduction treatments and aerial firefighting. We also found that the agencies had conducted reviews of their responses to wildland fires, but that they did not consistently follow agency policy in doing so or use specific criteria for selecting the fires they reviewed, limiting their ability to help ensure that their fire reviews provided useful information and meaningful results.

Agencies Use Various Performance Measures to Assess Wildland Fire Management

As we found in our September 2015 report, both the Forest Service and Interior use various performance measures, such as the number of acres treated to reduce fuels and the percentage of wildland fires contained during initial attack, to assess their wildland fire management effectiveness. These measures are reported in, among other things, the agencies’ annual congressional budget justifications. Officials from both the Forest Service and Interior, however, told us their performance measures need improvement to more appropriately reflect their emphasis on a risk-based approach to wildland fire management and, in June 2015, officials from both agencies told us that they were working to improve them. For example, in fiscal year 2014, the Forest Service began developing a performance measure intended to reflect that, in some cases, allowing naturally-ignited fires to burn can provide natural resource benefits at a lower cost and lower risk to personnel than fully suppressing the fire as quickly as possible. Forest Service officials told us they are working with field units to evaluate whether this measure will effectively assess their efforts to implement a risk-based approach to fire management, and said they will adjust it as needed. The officials told us they plan to finalize the measure and use it in 2017. Similarly, in fiscal year 2014 Interior began using a new performance measure intended to better reflect the variety of strategies, in addition to all-out suppression efforts, available to respond to wildland fires.

Agencies Have Undertaken Multiple Efforts to Assess Effectiveness of Specific Activities

The agencies have also undertaken multiple efforts to assess the effectiveness of particular wildland fire management activities, such as fuel reduction and aerial firefighting. Regarding fuel reduction activities, in prior work we found that demonstrating the effectiveness of fuel reduction treatments is inherently complex and that the agencies did not have sufficient information to evaluate fuel treatment effectiveness, such as the extent to which treatments changed fire behavior. Without such information, we concluded that the agencies could not ensure that fuel reduction funds were directed to the areas where they can best minimize risk to communities and natural and cultural resources. Accordingly, we recommended in 2007 that the agencies take actions to develop additional information on fuel treatment effectiveness. The agencies agreed with this recommendation and have taken steps to address it. In our September 2015 report, we found that the agencies are continuing efforts to improve their understanding of fuel treatment effectiveness. For example, the Forest Service and Interior agencies use a system called Fuel Treatment Effectiveness Monitoring to assess fuel reduction treatment effectiveness. The Forest Service began requiring such assessments in 2011, and Interior requested such assessments be completed starting in 2012. Under this approach, the agencies are to complete a monitoring report whenever a wildfire interacts with an area where a fuel reduction treatment was previously conducted.

Regarding aerial firefighting, in our August 2013 report, we found that Forest Service and Interior had not collected information on the performance and effectiveness of firefighting aircraft as part of their efforts to identify their firefighting aircraft needs. Specifically, we found that the agencies had not established data collection mechanisms to track the specific tactical uses of firefighting aircraft—for example, where retardant or water is dropped in relation to a fire as well as the objective of a drop, such as protecting a structure—or measure their performance and effectiveness in those uses. Since the 1960s, multiple reviews of federal fire aviation programs have called for the Forest Service and Interior to collect information on the performance of firefighting aircraft. At the time of our 2013 report, the Forest Service had recently begun an effort known as the Aerial Firefighting Use and Effectiveness Study to address this concern. We noted, however, that this study focused on large airtankers (which, as noted, are managed by the Forest Service) and that Interior did not have plans to collect performance information on the types of firefighting aircraft it manages. Accordingly, in that report we recommended that the agencies expand efforts to collect information on aircraft performance and effectiveness to include all types of firefighting aircraft in the federal fleet. The agencies generally agreed with our recommendation.

In our September 2015 report, we found that the Forest Service and Interior were jointly implementing the Aerial Firefighting Use and Effectiveness Study begun in 2012. According to the study website and agency officials, the agencies are collecting information on how aerial retardant and suppressant delivery affects fire behavior, and they plan to use this and other collected information to track the performance of specific aircraft types. As of November 2015, according to agency officials, the agencies had collected data on aircraft use in more than 100 fires as part of the study. In addition, according to these officials, as part of the study the agencies are developing specific ways to assess firefighting aircraft performance effectiveness to be used during the 2016 fire season. Agency officials told us the study is not a one-time activity but is an ongoing effort to continually provide information to help improve their use of firefighting resources.

The Forest Service and Interior Agencies Have Not Consistently Conducted Reviews of Wildland Fire Incidents to Assess their Effectiveness

As detailed in our September 2015 report, the Forest Service and the Interior agencies have conducted reviews to assess their effectiveness in responding to wildland fires, but have not consistently followed agency policy in doing so and did not always use specific criteria for selecting the fires they reviewed. Congressional committee reports and agency policy have generally called for the agencies to review their responses to wildland fires involving federal expenditures of $10 million or more, in part to help understand how to better contain suppression costs. The agencies, in turn, have each developed their own policies that generally direct them to review each fire that exceeds the $10 million threshold and, in some cases, those policies note that fire reviews may be conducted for other purposes, such as where the fire raised significant political, social, natural resource, or policy concerns.

The agencies have not consistently conducted reviews of fire incidents meeting the $10 million threshold, in part because, according to officials, doing so does not reflect the agencies’ focus on assessing the effectiveness of their response to fire. However, the agencies have not developed specific criteria for selecting fire incidents for review. Forest Service officials told us that rather than selecting all fires with federal expenditures of $10 million or more to review, they changed their selection approach. These officials told us that focusing exclusively on suppression costs when selecting fires may keep the agency from choosing those fires where it can obtain important information and best assess management actions and ensure they are appropriate, risk-based, and effective. Instead, Forest Service officials told us the agency judgmentally selects incidents to review based on a range of broad criteria, such as complexity and national significance, taking into account political, social, natural resource, or policy concerns. Using these broad selection criteria, the Forest Service reviewed 5 wildland fires that occurred in 2012 and 10 that occurred in 2013. However, with these broad criteria it is not clear why the Forest Service selected those particular fires and not others. For example, the 2013 Rim Fire, which burned more than 250,000 acres and cost more than $100 million to suppress—by far the costliest fire to suppress that year—was not among the 2013 fires selected for review. Moreover, the reviews completed for each of those years did not use consistent or specific criteria for conducting the reviews. As of July 2015, the agency had not selected fires to review from the 2014 wildland fire season and, when asked, agency officials did not indicate a time frame for doing so.

Forest Service officials told us they believe it is appropriate to judgmentally select fires to provide them flexibility in identifying which fires to review and which elements of the fire response to analyze. Nevertheless, Forest Service officials also acknowledged the need to develop more specific criteria for selecting fires to review and conducting the reviews. In July 2015, officials told us they were working to update their criteria for doing so. They provided us a draft update of the Forest Service policy manual, but this draft did not contain specific criteria for selecting fires for review or conducting the reviews. Moreover, officials did not provide a time frame for completing their update.

Within Interior, Bureau of Land Management officials told us that the agency completed its last fire review based on significant cost (i.e., federal expenditures of $10 million or more) in 2013. These officials told us that the Bureau of Land Management, similar to the Forest Service, plans to shift the emphasis of its fire reviews to evaluate management actions rather than focusing on cost, and that officials are working to determine criteria for selecting fires for review. Interior headquarters officials told us that the Fish and Wildlife Service and National Park Service have continued to follow the direction provided through their policies regarding reviews of fires that met the $10 million threshold. Interior headquarters officials, however, acknowledged the need to improve Interior’s approach to selecting fires for review to focus more on information about decision making rather than fire costs. In July 2015, the officials told us they planned to develop criteria other than cost for use by all Interior agencies in selecting fires to review, and that they planned to develop standard criteria for implementing the reviews, but they did not provide information about how they planned to develop such criteria or the factors they would consider.

Agency reports have likewise cited the need to improve both the processes for selecting fires for review and the implementation of the reviews. A 2010 report, for example, noted the importance of improving the selection of fires to review and stated that the agencies would benefit from a more productive review strategy. The report said the agencies’ existing approach to conducting reviews tended to produce isolated efforts and unrelated recommendations rather than establishing a consistent foundation for continuous improvement. A 2013 report assessing the usefulness of the Forest Service’s five reviews of 2012 fires noted shortcomings in consistency across the reviews, including unclear criteria for selecting fires and conducting reviews, as well as limitations in the specificity of the resulting reports and recommendations. Our previous body of work on performance management has shown that it is important for agencies to collect performance information to inform key management decisions, such as how to identify problems and take corrective actions and how to identify and share effective approaches. We concluded that, by developing specific criteria for selecting fires for review and conducting the reviews, the agencies may enhance their ability to obtain useful, comparable information about their effectiveness in responding to wildland fires, which, in turn, may help them identify needed improvements in their wildland fire approach. As a result, we recommended in our September 2015 report that the Secretaries of Agriculture and the Interior direct the Chief of the Forest Service and the Director of Interior’s Office of Wildland Fire to (1) develop specific criteria for selecting wildland fires for review and for conducting the reviews as part of their efforts to improve their approach to reviewing fires and (2) once such criteria are established, revise agency policies to align with the specific criteria developed by the agencies. In their written comments on our report, the agencies generally agreed with our recommendations and stated that they were developing criteria for selecting fires to review and conducting reviews.

The Forest Service Has Faced Challenges in Its Efforts to Modernize the Large Airtanker Fleet, but Has Increased the Availability of Large Airtankers Since Our 2013 Report

In our August 2013 report, we found that the Forest Service faced challenges in modernizing the government’s fleet of large airtankers— which had declined from 44 in 2002 to 8 in 2013—but since that report the agency has increased the availability of such aircraft, with some challenges remaining. Specifically, we found that the Forest Service planned to modernize the large airtanker fleet by obtaining large airtankers from various sources over the near, medium, and long terms, but that each component of this approach faced challenges that made the continued availability of such aircraft to meet national fire suppression needs uncertain. Since that report, some of these challenges remain, while others are no longer relevant. In addition, the Forest Service has increased the availability of large airtankers, in part by increasing the number of airtankers under contract.

We found in our 2013 report that, in the near term, the agency planned to rely on a mix of contracted “legacy” airtankers, including several P-2V Neptune aircraft—Korean War-era maritime patrol aircraft—as well as supplemental aircraft available through additional contracts and agreements with the military and with other governments. However, agency concerns existed regarding the availability, capability, and costs of these resources. For example, the agency had seven P-2V Neptune aircraft under contract, the ages of which made their availability throughout the entire 5-year contract period uncertain. Specifically, aircraft vendors told us they might need to retire some aircraft prior to the end of the contract period because of the cost of maintaining the aging aircraft. As of November 2015, six P-2V Neptune aircraft remained under contract to the Forest Service.

For the medium term, the Forest Service had awarded contracts for seven “next-generation” large airtankers that were expected to be faster and more up-to-date than the legacy aircraft. However, at the time of our 2013 report, it was uncertain when those aircraft would begin supporting fire suppression activities, in part because bid protests had delayed contract issuance and, at the time of that report, only one had completed necessary federal approval and certification processes to support fire suppression activities. Since then, according to Forest Service officials, six of these seven aircraft have completed the needed approval and certification processes. The Forest Service has issued additional contracts for next-generation aircraft and, as of November 2015, the agency had a total of 20 privately-owned large airtankers under contract, according to Forest Service officials. Seven of these airtankers were added under contract in September 2015.

For the long term, the Forest Service’s plan included a shift from the agency’s long-standing practice of contracting for, rather than owning, aircraft. Specifically, the Forest Service had indicated its long-term intention to obtain up to 14 Alenia C-27J Spartan transport aircraft through intergovernmental transfer at no initial cost if they were declared surplus by the military and to purchase other airtankers. In our 2013 report, however, we found that challenges existed regarding the retardant capacity and operating cost of the C-27J transport aircraft the Forest Service was to obtain through intergovernmental transfer, and that the Forest Service had been unable to justify previous plans for purchasing large airtankers to the Office of Management and Budget. Regarding intergovernmental transfer, the Forest Service had expressed interest in obtaining up to 14 Alenia C-27J Spartan transport aircraft from the Department of Defense if they were declared surplus equipment. Since our report, the National Defense Authorization Act of 2014 required a different intergovernmental transfer than that anticipated by the Forest Service—thereby making the challenge we identified with the C-27J transport aircraft no longer relevant. Rather than transferring the C-27J transport aircraft to the Forest Service, section 1098 of the act directs the Coast Guard to transfer seven Lockheed Martin HC-130H Hercules aircraft to the Forest Service for use in wildfire suppression. According to Forest Service officials, none of these aircraft have been transferred as of November 2015; these officials told us they expect the aircraft will be transferred to the Forest Service between 2017 and 2019. One aircraft, however, was used by the Forest Service during the 2015 wildland fire season under an agreement with the Coast Guard, according to Forest Service officials. The Forest Service equipped that aircraft with a Modular Airborne Firefighting System (MAFFS) unit—a portable, pressurized retardant delivery system. However, as we found in our 2013 report, Forest Service and Interior officials expressed concern that MAFFS performance can be inadequate in some circumstances. Forest Service officials told us the agency intends to operate one HC-130H aircraft with the MAFFS unit through the 2016 fire season, but the agency plans to equip all seven HC-130H aircraft with traditional gravity-fed retardant delivery systems. Forest Service officials told us that the expected service life of these aircraft is 6 to 12 years, after which time the Forest Service would likely need to invest in significant maintenance given the demands placed on the aircraft by the firefighting mission.

In another development since our 2013 report, the Consolidated and Further Continuing Appropriations Act, 2015, included $65 million “for the purpose of acquiring aircraft for the next-generation airtanker fleet to enhance firefighting mobility, effectiveness, efficiency, and safety.” In August 2015, the Forest Service issued draft specifications for aircraft it would consider purchasing in accordance with the act. In November 2015, Forest Service officials told us the Department of Agriculture is preparing a business case to demonstrate the feasibility of such a purchase as required by Office of Management and Budget guidance, but did not provide a time frame for its submission.

Chairman Murkowski, Ranking Member Cantwell, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time.

GAO Contact and Staff Acknowledgments

If you or your staff members have any questions about this testimony, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals who made key contributions to this testimony include Steve Gaty (Assistant Director), Ulana M. Bihun, Mark Braza, Richard P. Johnson, Kyle M. Stetler, and Kiki Theodoropoulos.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.