Federal Aviation Administration: Improved Tracking of Results Needed to Provide Confidence in Benefits of Streamlining Activities
Released: 2015-02-26T00:00:00

Letter

Background

FAA is the largest operating administration in DOT with almost 46,000 employees and a budget of $15.9 billion in fiscal year 2014. FAA carries out its mission—to provide the safest, most efficient airspace system in the world—through four lines of business and 10 staff offices (offices). Headquartered in Washington, D.C., with offices across the United States and an extensive global reach, FAA operates and maintains this system, known as the National Airspace System, and oversees the safety of aircraft and operators. Concurrent with the day-to-day operation of this system, FAA is also working to transform the nation’s ground-based radar air-traffic control system to an air-traffic management system using satellite-based navigation and other advanced technology. This transformation is referred to as the Next Generation Air Transportation System (NextGen). Among other duties, FAA regulates and encourages the U.S. commercial space transportation industry by licensing all commercial launches and reentries by U.S. citizens or companies that take place in the United States and overseas. FAA also administers programs related to airport safety and inspections, and standards for airport design, construction, and operation.

As mentioned above, Section 812 of the FAA Modernization and Reform Act, enacted in February 2012, mandated that FAA identify and undertake actions necessary to streamline and reform the agency. The mandate set out timelines for FAA to conduct this work and report to Congress. Specifically, FAA was to undertake its review no later than 60 days after enactment of the Act and undertake actions to address its findings no later than 120 days after enactment. Finally, FAA was to submit a report to Congress on the actions undertaken no later than 150 days after enactment.

FAA’s Assistant Administrator for Finance and Management serves as the agency official for process change management and provided leadership for FAA’s Section 812 effort through the Office of Finance and Management (AFN). In April 2012, AFN held an FAA-wide kickoff meeting to discuss the Section 812 requirements. Individual offices within FAA designated a representative to work with AFN to identify initiatives. According to AFN, FAA had a number of major efficiency improvements already under way in early 2012, so AFN asked offices to identify initiatives that reflected ongoing work as well as any additional opportunities for improvement and reform. Through this process, FAA identified 36 initiatives across its offices. Figure 1 shows the number of initiatives that each office is leading. After AFN and other FAA offices identified these 36 initiatives, a point of contact was identified for each initiative. Each point of contact was either a program manager for an initiative or a selected person within an office with access to information on the initiative. AFN collected information from the points of contact on the (1) problem statement, (2) proposed solution, (3) expected benefits, and (4) status of each initiative to create FAA’s January 2013 report to Congress. The heads of each responsible office validated the initiative information that the points of contact supplied to AFN.

Beyond the Section 812 mandate, Executive Orders and efforts by the Office of Management and Budget (OMB), DOT, and FAA have encouraged process improvements and efficiency initiatives within FAA. For example: In February 2011, according to FAA officials, FAA officially launched its Foundation for Success initiative, which aims to transform certain governance, shared services, human capital, and NextGen activities to improve the management of FAA functions. According to FAA, this initiative provides a more efficient organizational and management structure for ensuring the timely, cost-effective delivery of NextGen. In November 2011, Executive Order 13589 on promoting efficient spending in the federal government required each executive-branch agency to establish a plan for reducing the combined costs associated with certain functions, such as travel, information technology, printing, and agency fleets. In December 2012, AFN created the Community of Practice for Process Improvement to support FAA process improvements. This community of practice is designed to be a collaborative environment for subject matter experts to discuss ideas and share best practices related to agency improvements and efficiencies.

Using a Decentralized Process, FAA Reports That Most Initiatives Are Complete, Though Many Involve Continuous Action

The benchmarks—that is, the amount of work or milestones—that FAA used to determine when each initiative was considered “implemented” differ. The process to determine the status of the initiatives was a decentralized one in which the offices responsible for leading the individual initiatives determined the status (i.e., either in-progress or implemented) when AFN requested the status of the initiatives from points of contact. The heads of each contributing office subsequently validated the information on status, and then AFN accepted and reported that status.

Since each office independently determined the status of its initiative with limited direction from AFN, internal stakeholders and Congress do not have a clear, overarching picture of the status of the initiatives, including the work that FAA undertook to carry them out and how the actions addressed the mandate. As of January 2015, FAA considered 33 of the 36 initiatives implemented. However, the benchmarks FAA offices used to determine status varied, for example:

One initiative was to have the Office of Human Resources audit the leadership training offered throughout the agency and identify and work to reduce redundancies to obtain efficiencies and cost savings. FAA officials said that the team leading this work considered the initiative “implemented” after a Learning Professional Guiding Coalition was created, reviewed existing courses, and completed a road map to develop a centralized series of leadership courses. According to FAA officials, one leadership course has been deployed and work continues to develop 32 leadership courses.

Another initiative was to have the Office of Airports develop standard operating procedures for field operations to gain efficiencies from adopting best practices and ensuring regulations are followed. This office will consider the initiative “implemented” after it creates the 24 planned procedures and these procedures are in use by staff to issue grants, review documents, and complete other processes.

See appendix I for information on the benchmarks FAA reported using to determine the status of each initiative.

In addition to this variation in how offices determined the status of an initiative, many of the 36 initiatives started prior to the February 2012 passage of Section 812. Specifically, FAA started 33 of the 36 initiatives before passage of the mandate, according to FAA officials and documents. For example, 17 of the 36 initiatives were driven by or related to Foundation for Success, the Administrator’s examination of the agency’s organizational structure that officially began in 2011 to improve delivery of agency-wide services, such as information technology and budgeting, through a single, shared-services organization. Also, the Office of Aviation Safety initiative to close the London international field office began in February 2011. By contrast, FAA officials said that the Joint Resource Council initiative to review FAA’s acquisitions and investment strategy to optimize funding capital investments began in May 2012 after enactment of the FAA Modernization and Reform Act.

As stated, FAA was to begin its review no later than 60 days after enactment of the Act and undertake actions from its findings no later than 120 days after enactment. To meet these deadlines, AFN sought to identify improvements that FAA had under way or had already completed, as well as additional opportunities for improvement that aligned with Section 812, according to FAA documents. AFN officials explained that the FAA had already embraced a culture of continuous improvement and that the agency had ongoing efforts that were directly in line with the objectives Congress outlined in Section 812. By the time FAA reported to Congress in January 2013, the agency categorized 15 of the 36 initiatives as complete.

In addition, FAA officials leading 30 of the 36 initiatives told us that further or continuous action would be needed to realize benefits. We categorized FAA officials’ descriptions of the continuous action being taken for each initiative to realize expected benefits, even after an initiative is considered “implemented.” Table 1 describes the types of continuous actions completed or planned for the initiatives, and appendix I provides further information on the type of continuous action for each initiative.

The types of continuous action ranged from being directly related to realizing an initiative to helping ensure an initiative remained in place and achieved expected benefits. The following are examples of these types:

Primary actions—For an Office of Human Resources initiative to improve customer service, the office considered the initiative as “implemented” after it developed a draft agreement to establish the range of Human Resources’ services to be offered and performance targets for these services. After this determination of implementation, the office took further actions, including getting senior leadership’s approval of the agreement and using the agreement with the three offices Human Resources considered to be their major customers.

Secondary/related actions—The FAA Academy led an initiative to conduct a pilot program for the use of iPads for technician and pilot training. After a year, the pilot program was completed, and FAA officials considered the initiative “implemented.” FAA officials said that the success of the initiative led the Academy to expand the use of tablet devices in classrooms and other areas where the devices could expand quality or reduce cost.

Monitoring actions—One Air Traffic Organization initiative sought to align safety and technical training into a single office. Officials leading this initiative said that the Air Traffic Organization created the new, single office but would continue to take steps to improve the new office’s efficiency as needed, such as eliminating any duplicative positions.

While FAA has made some progress in implementing its streamlining and reform initiatives, our past work has highlighted issues FAA has had addressing a set of recommendations and fully executing changes related to a few of these 36 streamlining and reform initiatives. In July 2013, we reported on DOT’s progress in addressing 10 recommendations made to DOT and FAA by the Future of Aviation Advisory Committee to promote future success of the aviation industry. We found that DOT and FAA officials said they had addressed 7 of the 10 recommendations, but that a majority of advisory committee members believed only 1 recommendation had been addressed. Advisory committee members noted that some recommendations may not have been fully addressed since they were linked to ongoing efforts that DOT had identified. We have also previously reviewed specific initiatives. For example, one Office of Aviation Safety initiative was to establish an Unmanned Aircraft Systems Integration Office, which FAA created in January 2013. In February 2014, we testified that though the office had been officially created and had over 30 full time employees, it lacked an operations budget and had not finalized agreements related to the creation of the office. As of November 2014, FAA officials told us that the office had increased to 43 full time employees and had been allocated operations and facilities and equipment funding.

FAA’s Initiatives Were Generally Consistent with Selected Key Practices, but Limited Focus on Results Hinders Ability to Determine Overall Benefits

Selected Key Practices for Organizational Transformations

There are a number of key practices that can help agencies successfully carry out organizational transformations and improve the efficiency, effectiveness, and accountability of such efforts. The four selected key practices we used to evaluate FAA’s efforts for each initiative are consistently found at the center of successful transformations. These key practices are described in table 2. We identified these key practices based on our previous work on organizational transformations—both in the public and private sectors—and our work on implementing a results- oriented approach to agency management. We assessed FAA’s efforts for all 36 initiatives against two key practices—establish a communication strategy and adopt leading practices for results-oriented strategic planning and reporting. For the five initiatives that FAA classified as “in- progress,” we assessed FAA’s efforts against two additional key practices—dedicate an implementation team and set implementation goals and a timeline.

FAA’s Actions to Carry Out Initiatives Were Generally Consistent with Selected Key Practices but Were Less Consistent with Results-Oriented Reporting

FAA’s actions to carry out its initiatives were generally consistent with our selected key practices for organizational transformations; however, FAA’s actions were less consistent with the key practice to adopt leading practices for results-oriented reporting, as shown in figure 2. Appendix II provides more detail on the methodology we used to assess FAA’s actions, and appendix III shows the extent to which each initiative was consistent with the selected key practices we identified.

FAA’s actions were consistent with the key practice of dedicating an implementation team to manage the transformation process for the five in-progress initiatives we evaluated. Dedicating a strong and stable implementation team that will be responsible for the transformation’s day- to-day management is important for ensuring that it receives the focused, full-time attention needed to be sustained and successful. Initiatives that were consistent with this key practice identified an implementation team or contact, selected experienced team members, and established networks to support the implementation. For example, the Office of Airports’ initiative to standardize its field office structure and balance its field workload was consistent with this key practice. Specifically, two senior Airports officials are leading this initiative, one located in Washington, D.C., and one located in a regional office, and all five of the regions in which FAA is changing the structure and workload formed working groups to develop region-specific implementation plans and schedules.

Set Implementation Goals and a Timeline

FAA’s actions were consistent with the key practice of setting implementation goals and a timeline for 4 of the 5 in-progress initiatives but were inconsistent with the key practice for the remaining initiative. A transformation is a substantial commitment that could take years before it is completed and therefore must be carefully and closely managed, as we stated in our previous work on organizational transformations. Initiatives that were consistent with this key practice established implementation goals and timelines and developed plans for assessing and mitigating risk. For example, AFN’s initiative to modernize FAA’s records management system was consistent with this key practice. The team managing the initiative used short- and long-term timelines, weekly status reports, and a work schedule to set goals and a timeline for activities. However, Commercial Space Transportation’s initiative to move some inspectors and engineering staff to field office locations was inconsistent with the key practice. Officials leading this initiative did not provide an implementation plan, schedule, or other supporting documentation to demonstrate that they developed implementation goals, timelines, or plans to address risks.

Establish a Communication Strategy

FAA’s actions were consistent with the key practice of establishing a communication strategy for 30 of the 36 initiatives and partially consistent for 6 of the 36 initiatives. Creating an effective, on-going communication strategy is essential for executing a transformation, and the organization must develop a comprehensive communication strategy that reaches out to employees and seeks to engage them in the transformation. Initiatives that were consistent with this key practice had officials leading the effort who communicated early and often to build trust, encouraged two-way communication, and provided information to meet the specific needs of employees. For example, AFN’s initiative to consolidate strategic sourcing and related strategic programs into a new office was consistent with this key practice. AFN officials communicated to employees through briefings at the onset of the initiative and used regular newsletters to share information with affected employees during the transition. In addition, employees were able to provide feedback to their leadership through email, a hotline, and survey, as well as on “IdeaHub”—a DOT-administered internal website where employees can propose solutions or ideas regarding existing challenges. For initiatives that were partially consistent with this key practice, FAA demonstrated some but not all implementation steps for the key practice. For example, the Joint Resource Council’s initiative to review FAA’s acquisitions and investment strategy to optimize funding capital investments was only partially consistent with the key practice. Although the Joint Resource Council communicated information on the initiative to employees through internal websites and informational meetings, based on documents we reviewed, there was limited two-way communication to elicit feedback from employees on implementing this initiative.

Adopt Leading Practices for Results-Oriented Strategic Planning and Reporting

FAA’s actions were consistent with the key practice of adopting leading practices for results-oriented strategic planning and reporting for 21 of the 36 initiatives, partially consistent for 12 of the 36 initiatives, and inconsistent for 3 of the 36 initiatives. Initiatives that were consistent with this key practice established a basis for comparing results and used or were planning to use performance measures to assess results. Performance measures should show progress toward achieving an intended level of performance or results. Additionally, meaningful performance measures should be limited to a vital few and cover multiple government priorities such as quality, timeliness, cost of service, and other results.

For many initiatives, FAA’s actions were consistent with this key practice. For instance, for the Office of Aviation Safety’s initiative to close its London international field office, FAA reported a cost savings of $2.5 million through fiscal year 2015. FAA calculated cost savings due to the office closure and transfer of responsibilities to the Frankfurt international field office from, among other things, reduced staffing, savings in office rent, and savings in rent payments for personnel in Frankfurt. Another initiative for which FAA’s actions were consistent with this key practice was AFN’s initiative to centralize FAA’s acquisition functions and identify areas for process improvements to more efficiently distribute work and standardize processes. For this initiative, AFN officials had tracked a number of acquisition-related metrics prior to the consolidation of acquisition functions, which allowed AFN officials to examine trends in these metrics following the consolidation. For example, one metric FAA tracked was the number of certified contracting staff, which aligns with the initiative’s expected benefits to standardize processes and to offer expanded career paths for contracting professionals. Between September 2012 and September 2013, during which AFN said it completed this consolidation, the number of certified contracting staff increased from 143 to 191. In addition to quantifiable metrics such as this, AFN also tracked qualitative measures for expected benefits, such as clarifying authorities and responsibilities and sharing best practices and lessons learned.

However, not all initiatives were fully consistent with the implementation steps for the key practice. For example, the Office of Airports’ initiative to develop standard operating procedures to standardize its regional processes, such as grant reviews, was partially consistent with the key practice. Airports officials stated that performance measures had not been developed to assess the expected benefit of this initiative—to gain necessary efficiencies. Officials further stated that no baseline information exists, which would allow for a valid comparison of any change in overall efficiency. Officials stated that they intend to develop performance measures of efficiency for the initiative once they have developed and implemented all standard operating procedures. Until those performance measures are developed, officials said they will only measure the degree to which Airports employees use the new standard operating procedures.

In addition, three initiatives were inconsistent with the implementation steps for the key practice. For example, the Office of NextGen’s initiative to incorporate process improvements—termed “Ideas 2 In-Service” (I2I)— into its Acquisitions Management System was inconsistent with the key practice. FAA stated in its Section 812 report that this initiative would increase accountability and enable FAA to streamline the management of NextGen programs and activities through a single entry point for ideas to change the National Airspace System. When discussing the initiative to incorporate I2I into the Acquisitions Management System, NextGen officials stated that no performance measures currently exist to assess whether the Office of NextGen has achieved increased accountability or streamlining, nor is there a plan to develop measures to assess the performance of the initiative now or in the future.

Limited Focus on Measuring Results Hinders FAA’s Ability to Assess Whether Its Initiatives Achieved Expected Benefits

AFN has not effectively encouraged or coordinated performance measurement across the offices leading the streamlining and reform initiatives. As a result, FAA and Congress may have limited information on the extent to which FAA achieved the intended benefits outlined in the Section 812 mandate. As stated previously, FAA used a decentralized approach to respond to the Section 812 mandate. According to FAA officials, offices leading the initiatives were responsible for identifying initiatives and associated expected benefits. Although AFN encouraged offices to describe expected benefits and specific metrics when initially collecting information in April 2012 for FAA’s report to Congress, AFN did not explicitly communicate that offices should measure and track performance as initiatives were carried out and completed. Further, AFN provided limited guidance and oversight to offices on how to determine expected benefits, establish performance measures, and then track whether they were achieved.

As a result of this limited coordination on measuring results, FAA offices reported varied types of expected benefits across the 36 initiatives. Specifically, offices identified a range of quantifiable and qualitative expected benefits and reported the same types of benefits for few initiatives, even when initiatives had similar goals. For example, AFN reported a quantifiable benefit—cost savings—for an initiative that consolidated strategic sourcing and other strategic initiatives into a new organization. However, for a similar Air Traffic Organization initiative that consolidated oversight for major system acquisitions into a new office, FAA reported qualitative benefits, including a stronger acquisitions community and defined program-management career paths.

Given the range of expected benefits, performance measures for FAA’s offices also vary across the 36 initiatives. Varied performance measures may allow FAA to better capture the unique benefits for individual initiatives, such as fleet petroleum reduction. However, a limited focus by AFN on communicating the importance of measuring specific results, including those such as cost savings that may be applicable to multiple efficiency initiatives, hinders FAA from tracking and reporting on the overall benefit of the Section 812 effort. Offices are using or plan to use a variety of performance measures, including quantifiable and qualitative measures, to measure the different expected benefits, according to FAA officials. For example, the Policy, International Affairs, and Environment office’s initiative to facilitate an agency-wide sustainability program is tracking nine quantifiable performance measures, including water efficiency and alternative fuel use, on a quarterly basis against baseline information. In another instance, AFN’s initiative to streamline and improve its executive-level committees used qualitative measures to assess the effectiveness of changes to executive committees. Specifically, officials surveyed executives before and after changes were made to committees to determine the extent to which the initiative achieved its expected benefits, such as improved cross-organizational decision-making. Further, according to one of the current points of contact for the initiative to create a project management office to consolidate oversight of major air traffic organization acquisitions, one measure of the benefits from this initiative is that the organization has been able to support an increasing number of programs and stakeholders without increasing its workforce.

We have found in past work that FAA could improve its efforts to measure the performance of large-scale program implementation efforts, improvement initiatives, and certain oversight programs, and FAA is taking actions to address our related recommendations. For example, we found in September 2012 that FAA did not have performance measures to assess whether its new safety management system approach was improving safety, and we recommended that FAA identify and collect data on performance measures to assess whether the new approach meets its goals and objectives. FAA expects to have tools and processes in place to evaluate the safety management system’s performance by April 2015. In addition, in July 2014, we found that FAA did not develop performance metrics to measure the individual or collective outcomes of a number of its aviation certification and approval process-improvement initiatives, and we recommended that it develop and track measurable performance goals. We initially identified the need for performance measures in this area in 2010 and recommended that FAA develop and track measurable performance goals. FAA officials responded that they plan to develop these measures over time in three phases and will specifically develop measures to evaluate each initiative’s outcomes.

Performance information is needed for federal programs and activities to help inform decisions about how to address fragmentation, overlap, or duplication and is critical for achieving results and maximizing the return on federal funds, as we found in April 2014. In our previous work, we have found that federal agencies engaging in large projects, such as those FAA is currently undertaking, should establish activities to monitor performance measures and compare actual performance to expected benefits throughout the organization. Moreover, for federal agency consolidation efforts, we have found that agencies should have implementation plans that include measures that show an organization’s progress toward achieving an intended level of performance, such as quality, timeliness, cost of service, or customer service that the consolidation was intended to achieve.

By not further coordinating with FAA offices on the use of objective and balanced measures of efficiency and other improvements across initiatives, AFN and FAA overall are missing an opportunity to more consistently assess and aggregate the benefits from FAA’s streamlining and reform initiatives. To help produce an objective assessment of benefits, performance measures should typically include a quantifiable, measurable value to the greatest extent possible. Quantifiable measures can allow for a more useful assessment of benefits as these measures apply numerical targets or other measurable values to such benefits, providing a more objective comparison of benefits across initiatives and time periods. For example, for an initiative to centralize acquisition functions into a new office, FAA officials measured the percentage of employees with contract specialist certifications, a metric that demonstrated the organization’s progress towards developing a qualified workforce. Several measures FAA officials identified, such as improved communication, do not explicitly allow FAA to measure efficiency or other outcomes. Further, overemphasizing certain aspects of performance, such as improving timeliness, could result in deterioration in other aspects of performance, such as quality. By developing a balanced suite of measures, agencies can better ensure that they cover their various priorities while maintaining quality.

Without a more coordinated effort to encourage offices to track performance measures that can be aggregated across multiple initiatives, FAA and Congress as well as other stakeholders cannot have confidence that the agency’s efforts met or will meet the intent of Section 812 to streamline and reform the agency. AFN recognized this need in its description of one initiative to develop agreements to define the services to be offered by AFN to other FAA offices, such as information technology and acquisition functions; specifically, AFN stated that the absence of common performance metrics for these functions makes it difficult to determine the success or failure of efforts undertaken to consolidate these services and thus increase operational efficiency. FAA, through its Community of Practice for Process Improvement, is creating a database to track information on its process improvement efforts. FAA officials said this database will initially contain information only for the Section 812 initiatives but will eventually become a broader database on process improvement activities across the agency. According to FAA officials, they have not yet decided on the full range of information that the database will capture. FAA officials said that the database will initially contain, for each of the initiatives, only the four items in the 812 report—problem statement, proposed solution, expected benefits, and status—though the content could be expanded beyond these items. Further, FAA has not determined whether information in the database will ultimately provide a basis for measuring of the overall or net benefit of the Section 812 response.

Lastly, Section 812 required FAA to submit a report to Congress on the actions taken to streamline and reform the agency but did not require that FAA track or report to Congress on the results of these actions. As noted above, performance information for federal programs and activities is critical for achieving results and maximizing the return on federal funds. If Congress directs FAA to undertake a similar review to streamline and reform the agency in the next authorization of FAA, Congress could help ensure that FAA provides information on any realized efficiencies and improvements by requiring tracking and reporting. Without such information on results, Congress may have difficulty fully monitoring FAA’s efforts to make the agency more efficient and effective. Moreover, such a requirement would better position FAA to take steps to assess the overall results of its efforts. For example, FAA and its offices, if asked to undertake and report on the results of a streamlining and reform review in the next authorization for FAA, could take necessary steps, such as collecting baseline information, to establish performance measures and a basis for comparing the results in line with key practices for organizational transformations.

Conclusions

FAA responded to Section 812 of the FAA Modernization and Reform Act by working with each of its offices to identify and carry out initiatives to streamline and reform the agency. AFN, which led the agency’s response to the Section 812 mandate, collected and reported information on the 36 streamlining and reform initiatives but provided limited guidance on measuring performance and expectations to offices leading the initiatives. As a result, the offices leading the initiatives determined the status of the initiatives in different ways. Moreover, offices identified a wide variety of expected benefits across the initiatives and, where in place, used a wide variety of performance measures to gauge whether benefits were achieved for an initiative. Given the diverse nature of the 36 initiatives, some variation in how offices determined status and measured benefits is expected. However, FAA’s decentralized process to identify and track the initiatives intensified the variation and makes it difficult to discern FAA’s progress in making reforms and measure the overall impact of the initiatives. Without better performance measures, FAA lacks information to help it improve the performance of the initiatives and make decisions on issues targeted by the mandate, such as duplication and overlap. In addition, FAA has a limited ability to hold initiative leaders accountable for fully implementing the initiatives and for achieving planned benefits. Further, Congress and FAA will not know the extent to which the agency’s efforts met the aims of Section 812—including making the agency more efficient—without the use of some common performance measures that FAA can use to more easily aggregate benefits and assess results across multiple initiatives. While AFN provided limited guidance that likely contributed to the lack of common, consistent performance measures across initiatives, FAA has already responded to the Section 812 mandate. However, as many of the initiatives involve continuous action to realize benefits, collecting information on the results of these initiatives through the planned database can help FAA aggregate and report the results of the Section 812 initiatives. Moreover, by creating a mechanism to collect and manage such information in its planned database, FAA will be better positioned to measure the results of any future improvement and efficiency initiatives.

Further, key practices for organizational transformations and GAO’s work on streamlining government highlight the importance of using performance measures to show progress toward achieving desired results and outcomes. If Congress were to require FAA to report on actual results of a future streamlining and reform mandate, it would obtain information to judge whether FAA’s efforts met Congress’s intent and produced actual benefits and assist Congress with its oversight of the agency. By setting such a requirement and expectation, coupled with action by FAA to collect performance information in its planned database for ongoing and future improvement initiatives, Congress would enable FAA to better focus on measuring the results of any future mandated streamlining and reform efforts.

Matter for Congressional Consideration

If, in the next authorization for FAA, Congress chooses to mandate that FAA take actions to streamline and reform the agency, Congress may wish to consider requiring FAA to (1) track measures of and (2) report to Congress on the actual results of such efforts.

Recommendation for Executive Action

To better enable FAA to track, aggregate, and report on the results of its streamlining and reform initiatives, we recommend that the Secretary of Transportation direct FAA to develop a mechanism to capture the results of its efficiency initiatives in its planned database for process improvements. Measures of results might include, for example, cost savings, timeliness, or customer service metrics, which may be common to several types of process improvement efforts and therefore facilitate aggregation across improvements.

Agency Comments

We provided a draft of this report to DOT for review and comment. In its written comments, reproduced in appendix IV, DOT concurred with the recommendation. DOT also provided technical comments, which were incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Transportation and the appropriate congressional committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Information Related to the Status of the FAA Streamlining and Reform Initiatives

Table 3 provides a list of the Federal Aviation Administration’s (FAA) 36 streamlining and reform initiatives that the agency identified in response to Section 812 of the FAA Modernization and Reform Act of 2012. As of January 2015, FAA considered 3 initiatives in-progress (shown in italics in table 3) and 33 initiatives implemented. The table provides information on the benchmark FAA used to determine that an initiative was “implemented” for the purposes of Section 812. The table also summarizes FAA officials’ descriptions of the continuous action needed, if any, to realize expected benefits for each initiative, even after it is considered “implemented.” We categorized the continuous actions into the four following groups: primary actions that are directly related to realizing the initiative; secondary or related actions that enhance or expand the initiative, monitoring actions that are conducted in order to maintain an initiative, and none.

Appendix II: Objectives, Scope, and Methodology

This report examines FAA’s actions to respond to the Section 812 mandate and efforts to implement 36 initiatives to streamline and reform the agency. FAA identified these 36 initiatives in response to Section 812 of the FAA Modernization and Reform Act of 2012. In particular, this report provides information on (1) how FAA determined the status of the streamlining and reform initiatives that the agency reported on in response to Section 812 and (2) the extent to which FAA’s efforts to carry out these initiatives were consistent with selected key practices for organizational transformations.

To describe how FAA determined the status of the 36 recommendations, we examined FAA’s Section 812 report to Congress and other agency documents related to the streamlining and reform initiatives. We reviewed FAA officials’ descriptions of the continuous action being taken for each initiative to realize expected benefits, even after an initiative is considered “implemented,” and we categorized the action in the following four groups: primary actions that are directly related to realizing the initiative, secondary or related actions that enhance or expand the initiative, monitoring actions that are conducted in order to maintain an initiative, and none/no planned continuous action. According to FAA, the agency identified the 36 initiatives by documenting both ongoing and newly identified improvements that were in line with the Section 812 language. We did not assess the appropriateness of the initiatives FAA identified. In addition, we reviewed prior GAO reports on the Department of Transportation’s (DOT) and FAA’s actions to implement a set of recommendations and reports specific to topics covered by the 36 initiatives. We also interviewed FAA officials responsible for individual initiatives as well as for coordinating the agency’s Section 812 efforts to discuss the status of each initiative and any future plans or remaining actions.

To examine the extent to which FAA’s efforts to carry out the 36 initiatives were consistent with selected key practices, we identified key practices applicable to the FAA initiatives cited in prior GAO work on organizational transformations. We searched past GAO publications for reports on project management or implementation and discussed possible sources for criteria with internal stakeholders. We identified several relevant reports and sources that examined organizational transformations and streamlining government, in particular, efficiency initiatives and proposals to consolidate infrastructure and management functions. Most of these reports drew on or included the key practices for mergers and organizational transformations outlined in our 2003 report. We selected the key practices for organizational transformations as criteria against which to assess FAA’s efforts since Section 812 directs FAA to review the agency and take necessary actions to reform the agency and since the key practices were used in past work to examine government streamlining efforts like efficiency initiatives and consolidations. However, since we reviewed 36 initiatives led by offices within FAA rather than a single, agency-wide initiative, we found that not all the key practices for organizational transformations were relevant. Given the scope and status of the initiatives, we identified four key practices that were applicable. We determined that all four of these key practices were applicable for evaluating the initiatives that FAA reported as “in-progress,” and two of them—establish a communication strategy and adopt leading practices for results-oriented strategic planning and reporting—were applicable for evaluating the initiatives that FAA reported as “implemented.” Table 4 lists the selected key practices, with the relevant implementation steps for each key practice.

We systematically assessed the extent to which FAA’s efforts to carry out an initiative were consistent with the key practices. For each initiative, we (1) reviewed FAA documents—schedules, communications, and other planning documents—and (2) conducted semi-structured interviews with the FAA point(s) of contact. We developed a template to help us consistently analyze the collected information related to the implementation steps for each key practice (see table 4). For one key practice—adopt leading practices for results-oriented strategic planning and reporting—we supplemented the implementation step with requirements for performance plans outlined in An Evaluator’s Guide to Assessing Agency Annual Performance Plans. This guide, which was based in part on requirements for agency performance plans from the Government Performance and Results Act of 1993 (GPRA), identifies key issues and criteria to assess performance plans. Specifically, we used criteria from the guide on defining expected performance that aligned with the key practice and our past work on government streamlining. We determined whether FAA’s efforts to carry out each initiative were consistent, partially consistent, or inconsistent with each applicable key practice, or whether there was not enough information to make an assessment. We used the following general decision rules to make our assessment: partially consistent, if FAA had shown some progress toward consistent, if FAA had instituted the practice; instituting, or started but not completed the practice; inconsistent, if FAA had made minimal or no progress toward instituting the practice; and not enough information to tell, if, for example, the initiative was implemented several years ago, and/or documentation or testimonial evidence does not exist.

We used the FAA documentation and interviews for two initiatives to do trial assessments using a draft version of the template; we then made revisions to the template to clarify the information to collect and decision rules before carrying out assessments for all initiatives. Two analysts reviewed the documentation to make an assessment for each initiative. After we completed our assessments for all 36 initiatives, we identified themes or trends in FAA’s implementation across all initiatives, including the extent to which FAA’s efforts were consistent with the key practices. While we reviewed FAA’s measures or plans to measure the benefits of individual initiatives as part of our effort to assess FAA’s efforts to adopt leading practices for results-oriented strategic planning and reporting, we did not validate FAA’s estimates of benefits against independent measures. Thus, we did not report on the actual or achieved benefits of the initiatives.

In addition, we interviewed FAA officials responsible for coordinating the agency’s Section 812 efforts to discuss the guidance given to offices and points of contact implementing individual initiatives and the information collected and tracked for all the streamlining and reforming initiatives.

Appendix III: Evaluation of FAA Streamlining and Reform Initiatives against Selected Key Practices

Table 5 provides a list of FAA’s 36 streamlining and reform initiatives, by FAA office, which the agency identified in response to Section 812 of the FAA Modernization and Reform Act of 2012. The table also provides our assessment of the extent to which FAA’s efforts to implement each initiative were consistent with four selected key practices for organizational transformations. Appendix II contains information on the scope and methodology of this analysis.

Appendix IV: Comments from the Department of Transportation

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Gerald L. Dillingham, Ph.D., (202) 512-2834 or dillinghamg@gao.gov.

Staff Acknowledgments

In addition to the contact person named above, Catherine Colwell, Assistant Director; Melissa Bodeau; Elizabeth Curda; Kevin Egan; Aracely Galvan; Dave Hinchman; Bert Japikse; Heather Krause; Brandon Kruse; Joanie Lofgren; SaraAnn Moessbauer; Josh Ormond; Sarah E. Veale; and William T. Woods made key contributions to this report.

Related GAO Products

Aviation Manufacturing: Status of FAA’s Efforts to Improve Certification and Regulatory Consistency. GAO-14-829T. Washington, D.C.: July 31, 2014.

FAA Reauthorization Act: Progress and Challenges Implementing Various Provisions of the 2012 Act. GAO-14-285T. Washington, D.C.: February 5, 2014.

Managing For Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.

Strategic Sourcing: Leading Commercial Practices Can Help Federal Agencies Increase Savings When Acquiring Services. GAO-13-417. Washington, D.C.: April 15, 2013.

NextGen Air Transportation System: FAA Has Made Some Progress in Midterm Implementation, but Ongoing Challenges Limit Expected Benefits. GAO-13-264. Washington, D.C.: April 8, 2013.

Acquisition Workforce: DOT Lacks Data, Oversight, and Strategic Focus Needed to Address Significant Workforce Challenges. GAO-13-117. Washington, D.C.: January 23, 2013.

Unmanned Aircraft Systems: Measuring Progress and Addressing Potential Privacy Concerns Would Facilitate Integration into the National Airspace System. GAO-12-981. Washington, D.C.: September 14, 2012.

Streamlining Government: Questions to Consider When Evaluating Proposals to Consolidate Physical Infrastructure and Management Functions. GAO-12-542. Washington, D.C.: May 23, 2012.

Streamlining Government: Key Practices from Select Efficiency Initiatives Should Be Shared Governmentwide. GAO-11-908. Washington, D.C.: September 30, 2011.

NextGen Air Transportation System: FAA’s Metrics Can Be Used to Report on Status of Individual Programs, but Not of Overall NextGen Implementation or Outcomes. GAO-10-629. Washington, D.C.: July 27, 2010.

Air Traffic Control: FAA Reports Progress in System Acquisitions, but Changes in Performance Measurement Could Improve Usefulness of Information. GAO-08-42. Washington, D.C.: December 18, 2007.

Results-Oriented Cultures: Implementation Steps to Assist Mergers and Organizational Transformations. GAO-03-669. Washington, D.C.: July 2, 2003.