Federal Aviation Administration: Improved Tracking of Results Needed to Provide Confidence in Benefits of Streamlining Activities
Published: 2015-01-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: As fiscal pressures facing the federal government continue, so too does the need for federal agencies to improve the efficiency and effectiveness of programs and activities. Section 812 of the FAA Modernization and Reform Act of 2012 mandated that FAA review its programs, offices, and organizations to, among other things, identify and address inefficient processes, wasteful practices, and duplication. In response, FAA identified 36 initiatives, including centralizing administrative functions and modernizing records management.
GAO was asked to examine FAA's progress to streamline and reform the agency as Congress considers reauthorizing FAA in fiscal year 2015. GAO examined how FAA determined the status of initiatives and the extent to which its efforts to implement initiatives were consistent with selected key practices for organizational transformations. Since each initiative sought to streamline or reform FAA, GAO identified four key practices for organizational transformations as applicable to these initiatives. GAO assessed FAA's efforts by comparing FAA documents to the selected key practices and interviewing agency officials leading each initiative.
section_title: What GAO Found
paragraphs: The Federal Aviation Administration (FAA) used a decentralized process to track the status of streamlining and reform initiatives identified in response to the Section 812 mandate in the FAA Modernization and Reform Act of 2012. FAA's actions to implement the initiatives were mostly consistent with three key practices for organizational transformations but were less consistent with the key practice of adopting leading practices for results-oriented reporting, which includes using performance measures to show progress toward achieving results. Without information on the results of the initiatives, FAA and Congress cannot have confidence that FAA's efforts streamlined and reformed the agency.
Decentralized process: The Office of Finance and Management (AFN)—which led FAA's response to the Section 812 mandate—used a decentralized process to track initiatives. Individual offices responsible for the initiatives determined their status using varied definitions for “implemented.” For example, FAA considered an initiative to centralize leadership training “implemented” after officials created a plan for developing a series of courses, while FAA will consider an ongoing initiative to create standard procedures for the Office of Airports “implemented” after officials develop and deploy 24 new, standard procedures. As of January 2015, FAA considered 33 of the 36 initiatives implemented.
FAA's actions generally consistent with three key practices: GAO found that FAA's actions to implement the initiatives were mostly consistent for three key practices for organizational transformations—dedicate an implementation team, set implementation goals and a timeline, and establish a communication strategy. For example, FAA's actions were consistent with establishing a communication strategy for 30 of 36 initiatives and partially consistent for 6 of 36 initiatives.
FAA's actions less consistent with key practice regarding results-oriented reporting: GAO found that FAA's actions were inconsistent with this key practice for 3 of 36 initiatives, partially consistent for 12 of the 36, and consistent for 21 of 36. For example, for an initiative that was partially consistent, officials said that until they develop performance measures for the effect of the initiative, they would measure only whether staff use the new procedures. FAA's limited efforts to measure performance or outcomes of the initiatives hinder its ability to assess the initiatives' results. AFN has neither required offices to track performance measures nor made a specific effort to track any common measures across initiatives. As a result, offices used a range of performance measures to report results. GAO has previously found that information on results is critical for improving program performance and that agencies should have measures for the intended results of streamlining efforts—like cost savings and customer service—to help decision makers improve program performance. Actions to implement most of the 36 initiatives are continuing, and FAA plans to create a database to track these initiatives. Moving forward, FAA also plans to use the database to track other process improvement activities. To date, FAA has not decided what information to capture in the database but initially plans to include only descriptive information on each initiative. Lastly, Section 812 did not require FAA to track or report to Congress on the initiatives' results. By requiring such tracking and reporting, Congress could help ensure that FAA provides information on the results of a reform mandate, if required of FAA in the next authorization.
section_title: What GAO Recommends
paragraphs: As Congress considers FAA reauthorization, GAO suggests that Congress consider requiring FAA to track and report on the actual results of future agency-reform efforts. GAO recommends that FAA take steps to capture the results of improvement initiatives in its planned database for process improvements. The Department of Transportation agreed with the recommendation.

section_title: Letter
paragraphs: As the fiscal pressures facing the government continue, so too does the  need for executive branch agencies, like the Federal Aviation  Administration (FAA), to improve the efficiency and effectiveness of  government programs and activities. Opportunities to take action exist in  areas where federal programs or activities are fragmented, overlapping,  or duplicative. In February 2012, Section 812 of the FAA Modernization  and Reform Act of 2012 directed FAA to take various actions to  streamline and reform the agency. This mandate required the FAA to  thoroughly review its programs, offices, and organization to identify: duplicative positions, programs, roles, or offices; wasteful practices; redundant, obsolete, or unnecessary functions;   inefficient processes; and   ineffectual or outdated policies.
FAA responded with a January 2013 report to Congress that identified 36  initiatives that reflected both ongoing efforts in these areas and new  opportunities to improve processes and drive efficiencies across the  agency.  These initiatives varied from furthering the use of shared  information technology services and reforming records management to  relocating staff to field offices and closing an international field office.   See appendix I for a list of the 36 initiatives.
You requested that we review and analyze FAA’s actions to respond to  the Section 812 mandate and progress in implementing the identified  initiatives to streamline and reform the agency as you begin considering  reauthorization of FAA.  This report examines (1) how FAA determined  the status of the streamlining and reform initiatives that the agency  reported in response to Section 812 and (2) the extent to which FAA’s  efforts to carry out these initiatives were consistent with selected key  practices for organizational transformations.
To describe how FAA determined the status of the 36 initiatives, we  examined FAA’s Section 812 report to Congress and other agency  documents such as schedules and examples of communication related to  the streamlining and reform initiatives. In addition, we reviewed prior  GAO reports on the Department of Transportation’s (DOT) and FAA’s  actions to implement a set of recommendations as well as reports specific  to topics covered by the 36 initiatives.  We also interviewed FAA officials  and reviewed and categorized FAA officials’ descriptions of any actions  that are needed for an initiative to be considered implemented or will  continue after an initiative’s implementation.  To examine the extent to  which FAA’s efforts to carry out the 36 initiatives were consistent with  selected key practices for organizational transformations, we first  identified key practices cited in prior GAO work on organizational  transformations that were applicable to the FAA initiatives.  We identified  the selected key practices for organizational transformations as criteria  against which to assess FAA’s efforts since Section 812 directed FAA to  review the agency and undertake necessary actions to reform the  agency. In addition, key practices for organizational transformations have  been used in past GAO work to examine government-streamlining efforts,  including efficiency initiatives and consolidations.  For each initiative, we  (1) reviewed FAA documents—schedules, communications, and other  planning documents—and (2) conducted semi-structured interviews with  the FAA point(s) of contact for the initiative.  We then systematically  assessed the extent to which FAA’s efforts for an initiative were  consistent with the selected key practices for organizational  transformations.  While we assessed FAA’s measures or plans to  measure benefits against the key practice for organizational  transformation, we did not validate FAA’s estimates of benefits.  See  appendix II for a more detailed description of our scope and methodology.
We conducted this performance audit from February 2014 to January  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: FAA is the largest operating administration in DOT with almost 46,000  employees and a budget of $15.9 billion in fiscal year 2014. FAA carries  out its mission—to provide the safest, most efficient airspace system in  the world—through four lines of business and 10 staff offices (offices).   Headquartered in Washington, D.C., with offices across the United States  and an extensive global reach, FAA operates and maintains this system,  known as the National Airspace System, and oversees the safety of  aircraft and operators. Concurrent with the day-to-day operation of this  system, FAA is also working to transform the nation’s ground-based radar  air-traffic control system to an air-traffic management system using  satellite-based navigation and other advanced technology. This  transformation is referred to as the Next Generation Air Transportation  System (NextGen).  Among other duties, FAA regulates and encourages  the U.S. commercial space transportation industry by licensing all  commercial launches and reentries by U.S. citizens or companies that  take place in the United States and overseas. FAA also administers  programs related to airport safety and inspections, and standards for  airport design, construction, and operation.
As mentioned above, Section 812 of the FAA Modernization and Reform  Act, enacted in February 2012, mandated that FAA identify and undertake  actions necessary to streamline and reform the agency. The mandate set  out timelines for FAA to conduct this work and report to Congress.   Specifically, FAA was to undertake its review no later than 60 days after  enactment of the Act and undertake actions to address its findings no  later than 120 days after enactment. Finally, FAA was to submit a report  to Congress on the actions undertaken no later than 150 days after  enactment.
FAA’s Assistant Administrator for Finance and Management serves as  the agency official for process change management and provided  leadership for FAA’s Section 812 effort through the Office of Finance and  Management (AFN). In April 2012, AFN held an FAA-wide kickoff meeting  to discuss the Section 812 requirements. Individual offices within FAA  designated a representative to work with AFN to identify initiatives.  According to AFN, FAA had a number of major efficiency improvements  already under way in early 2012, so AFN asked offices to identify  initiatives that reflected ongoing work as well as any additional  opportunities for improvement and reform. Through this process, FAA  identified 36 initiatives across its offices.  Figure 1 shows the number of  initiatives that each office is leading.  After AFN and other FAA offices  identified these 36 initiatives, a point of contact was identified for each  initiative. Each point of contact was either a program manager for an  initiative or a selected person within an office with access to information  on the initiative.  AFN collected information from the points of contact on  the (1) problem statement, (2) proposed solution, (3) expected benefits,  and (4) status of each initiative to create FAA’s January 2013 report to  Congress. The heads of each responsible office validated the initiative  information that the points of contact supplied to AFN.
Beyond the Section 812 mandate, Executive Orders and efforts by the  Office of Management and Budget (OMB), DOT, and FAA have  encouraged process improvements and efficiency initiatives within FAA.  For example:  In February 2011, according to FAA officials, FAA officially launched  its Foundation for Success initiative, which aims to transform certain  governance, shared services, human capital, and NextGen activities  to improve the management of FAA functions. According to FAA, this  initiative provides a more efficient organizational and management  structure for ensuring the timely, cost-effective delivery of NextGen.  In November 2011, Executive Order 13589 on promoting efficient  spending in the federal government required each executive-branch  agency to establish a plan for reducing the combined costs associated  with certain functions, such as travel, information technology, printing,  and agency fleets.   In December 2012, AFN created the Community of Practice for  Process Improvement to support FAA process improvements. This  community of practice is designed to be a collaborative environment  for subject matter experts to discuss ideas and share best practices  related to agency improvements and efficiencies.
section_title: Using a Decentralized Process, FAA Reports That Most Initiatives Are Complete, Though Many Involve Continuous Action
paragraphs: The benchmarks—that is, the amount of work or milestones—that FAA  used to determine when each initiative was considered “implemented”  differ.  The process to determine the status of the initiatives was a  decentralized one in which the offices responsible for leading the  individual initiatives determined the status (i.e., either in-progress or  implemented) when AFN requested the status of the initiatives from  points of contact. The heads of each contributing office subsequently  validated the information on status, and then AFN accepted and reported  that status.
Since each office independently determined the status of its initiative with  limited direction from AFN, internal stakeholders and Congress do not  have a clear, overarching picture of the status of the initiatives, including  the work that FAA undertook to carry them out and how the actions  addressed the mandate. As of January 2015, FAA considered 33 of the  36 initiatives implemented. However, the benchmarks FAA offices used to  determine status varied, for example:
One initiative was to have the Office of Human Resources audit the  leadership training offered throughout the agency and identify and  work to reduce redundancies to obtain efficiencies and cost savings.  FAA officials said that the team leading this work considered the  initiative “implemented” after a Learning Professional Guiding  Coalition was created, reviewed existing courses, and completed a  road map to develop a centralized series of leadership courses.   According to FAA officials, one leadership course has been deployed  and work continues to develop 32 leadership courses.
Another initiative was to have the Office of Airports develop standard  operating procedures for field operations to gain efficiencies from  adopting best practices and ensuring regulations are followed.  This  office will consider the initiative “implemented” after it creates the 24  planned procedures and these procedures are in use by staff to issue  grants, review documents, and complete other processes.
See appendix I for information on the benchmarks FAA reported using to  determine the status of each initiative.
In addition to this variation in how offices determined the status of an  initiative, many of the 36 initiatives started prior to the February 2012  passage of Section 812.  Specifically, FAA started 33 of the 36 initiatives  before passage of the mandate, according to FAA officials and  documents.  For example, 17 of the 36 initiatives were driven by or  related to Foundation for Success, the Administrator’s examination of the  agency’s organizational structure that officially began in 2011 to improve  delivery of agency-wide services, such as information technology and  budgeting, through a single, shared-services organization.  Also, the  Office of Aviation Safety initiative to close the London international field  office began in February 2011.  By contrast, FAA officials said that the  Joint Resource Council initiative to review FAA’s acquisitions and  investment strategy to optimize funding capital investments began in May  2012 after enactment of the FAA Modernization and Reform Act.
As stated, FAA was to begin its review no later than 60 days after  enactment of the Act and undertake actions from its findings no later than  120 days after enactment.  To meet these deadlines, AFN sought to  identify improvements that FAA had under way or had already completed,  as well as additional opportunities for improvement that aligned with  Section 812, according to FAA documents.  AFN officials explained that  the FAA had already embraced a culture of continuous improvement and  that the agency had ongoing efforts that were directly in line with the  objectives Congress outlined in Section 812. By the time FAA reported to  Congress in January 2013, the agency categorized 15 of the 36 initiatives  as complete.
In addition, FAA officials leading 30 of the 36 initiatives told us that further  or continuous action would be needed to realize benefits.  We  categorized FAA officials’ descriptions of the continuous action being  taken for each initiative to realize expected benefits, even after an  initiative is considered “implemented.”  Table 1 describes the types of  continuous actions completed or planned for the initiatives, and appendix  I provides further information on the type of continuous action for each  initiative.
The types of continuous action ranged from being directly related to  realizing an initiative to helping ensure an initiative remained in place and  achieved expected benefits. The following are examples of these types:
Primary actions—For an Office of Human Resources initiative to  improve customer service, the office considered the initiative as  “implemented” after it developed a draft agreement to establish the  range of Human Resources’ services to be offered and performance  targets for these services.  After this determination of implementation,  the office took further actions, including getting senior leadership’s  approval of the agreement and using the agreement with the three  offices Human Resources considered to be their major customers.
Secondary/related actions—The FAA Academy led an initiative to  conduct a pilot program for the use of iPads for technician and pilot  training.  After a year, the pilot program was completed, and FAA  officials considered the initiative “implemented.” FAA officials said that  the success of the initiative led the Academy to expand the use of  tablet devices in classrooms and other areas where the devices could  expand quality or reduce cost.
Monitoring actions—One Air Traffic Organization initiative sought to  align safety and technical training into a single office. Officials leading  this initiative said that the Air Traffic Organization created the new,  single office but would continue to take steps to improve the new  office’s efficiency as needed, such as eliminating any duplicative  positions.
While FAA has made some progress in implementing its streamlining and  reform initiatives, our past work has highlighted issues FAA has had  addressing a set of recommendations and fully executing changes related  to a few of these 36 streamlining and reform initiatives. In July 2013, we  reported on DOT’s progress in addressing 10 recommendations made to  DOT and FAA by the Future of Aviation Advisory Committee to promote  future success of the aviation industry.  We found that DOT and FAA  officials said they had addressed 7 of the 10 recommendations, but that a  majority of advisory committee members believed only 1 recommendation  had been addressed. Advisory committee members noted that some  recommendations may not have been fully addressed since they were  linked to ongoing efforts that DOT had identified. We have also previously  reviewed specific initiatives.  For example, one Office of Aviation Safety  initiative was to establish an Unmanned Aircraft Systems Integration  Office, which FAA created in January 2013.  In February 2014, we  testified that though the office had been officially created and had over 30  full time employees, it lacked an operations budget and had not finalized  agreements related to the creation of the office. As of November 2014,  FAA officials told us that the office had increased to 43 full time  employees and had been allocated operations and facilities and  equipment funding.
section_title: FAA’s Initiatives Were Generally Consistent with Selected Key Practices, but Limited Focus on Results Hinders Ability to Determine Overall Benefits
subsections: section_title: Selected Key Practices for Organizational Transformations
paragraphs: There are a number of key practices that can help agencies successfully  carry out organizational transformations and improve the efficiency,  effectiveness, and accountability of such efforts. The four selected key  practices we used to evaluate FAA’s efforts for each initiative are  consistently found at the center of successful transformations. These key  practices are described in table 2. We identified these key practices  based on our previous work on organizational transformations—both in  the public and private sectors—and our work on implementing a results- oriented approach to agency management. We assessed FAA’s efforts  for all 36 initiatives against two key practices—establish a communication  strategy and adopt leading practices for results-oriented strategic  planning and reporting. For the five initiatives that FAA classified as “in- progress,” we assessed FAA’s efforts against two additional key  practices—dedicate an implementation team and set implementation  goals and a timeline.
section_title: FAA’s Actions to Carry Out Initiatives Were Generally Consistent with Selected Key Practices but Were Less Consistent with Results-Oriented Reporting
paragraphs: FAA’s actions to carry out its initiatives were generally consistent with our  selected key practices for organizational transformations; however, FAA’s  actions were less consistent with the key practice to adopt leading  practices for results-oriented reporting, as shown in figure 2. Appendix II  provides more detail on the methodology we used to assess FAA’s  actions, and appendix III shows the extent to which each initiative was  consistent with the selected key practices we identified.
FAA’s actions were consistent with the key practice of dedicating an  implementation team to manage the transformation process for the five  in-progress initiatives we evaluated. Dedicating a strong and stable  implementation team that will be responsible for the transformation’s day- to-day management is important for ensuring that it receives the focused,  full-time attention needed to be sustained and successful. Initiatives that  were consistent with this key practice identified an implementation team  or contact, selected experienced team members, and established  networks to support the implementation. For example, the Office of  Airports’ initiative to standardize its field office structure and balance its  field workload was consistent with this key practice. Specifically, two  senior Airports officials are leading this initiative, one located in  Washington, D.C., and one located in a regional office, and all five of the  regions in which FAA is changing the structure and workload formed  working groups to develop region-specific implementation plans and  schedules.
subsections: section_title: Set Implementation Goals and a Timeline
paragraphs: FAA’s actions were consistent with the key practice of setting  implementation goals and a timeline for 4 of the 5 in-progress initiatives  but were inconsistent with the key practice for the remaining initiative. A  transformation is a substantial commitment that could take years before it  is completed and therefore must be carefully and closely managed, as we  stated in our previous work on organizational transformations. Initiatives  that were consistent with this key practice established implementation  goals and timelines and developed plans for assessing and mitigating  risk. For example, AFN’s initiative to modernize FAA’s records  management system was consistent with this key practice. The team  managing the initiative used short- and long-term timelines, weekly status  reports, and a work schedule to set goals and a timeline for activities.  However, Commercial Space Transportation’s initiative to move some  inspectors and engineering staff to field office locations was inconsistent  with the key practice. Officials leading this initiative did not provide an  implementation plan, schedule, or other supporting documentation to  demonstrate that they developed implementation goals, timelines, or  plans to address risks.
section_title: Establish a Communication Strategy
paragraphs: FAA’s actions were consistent with the key practice of establishing a  communication strategy for 30 of the 36 initiatives and partially consistent  for 6 of the 36 initiatives. Creating an effective, on-going communication  strategy is essential for executing a transformation, and the organization  must develop a comprehensive communication strategy that reaches out  to employees and seeks to engage them in the transformation.  Initiatives that were consistent with this key practice had officials leading  the effort who communicated early and often to build trust, encouraged  two-way communication, and provided information to meet the specific  needs of employees. For example, AFN’s initiative to consolidate  strategic sourcing and related strategic programs into a new office was  consistent with this key practice. AFN officials communicated to  employees through briefings at the onset of the initiative and used regular  newsletters to share information with affected employees during the  transition. In addition, employees were able to provide feedback to their  leadership through email, a hotline, and survey, as well as on  “IdeaHub”—a DOT-administered internal website where employees can  propose solutions or ideas regarding existing challenges. For initiatives  that were partially consistent with this key practice, FAA demonstrated  some but not all implementation steps for the key practice. For example,  the Joint Resource Council’s initiative to review FAA’s acquisitions and  investment strategy to optimize funding capital investments was only  partially consistent with the key practice. Although the Joint Resource  Council communicated information on the initiative to employees through  internal websites and informational meetings, based on documents we  reviewed, there was limited two-way communication to elicit feedback  from employees on implementing this initiative.
section_title: Adopt Leading Practices for Results-Oriented Strategic Planning and Reporting
paragraphs: FAA’s actions were consistent with the key practice of adopting leading  practices for results-oriented strategic planning and reporting for 21 of the  36 initiatives, partially consistent for 12 of the 36 initiatives, and  inconsistent for 3 of the 36 initiatives. Initiatives that were consistent with  this key practice established a basis for comparing results and used or  were planning to use performance measures to assess results.  Performance measures should show progress toward achieving an  intended level of performance or results. Additionally, meaningful  performance measures should be limited to a vital few and cover multiple  government priorities such as quality, timeliness, cost of service, and  other results.
For many initiatives, FAA’s actions were consistent with this key practice.  For instance, for the Office of Aviation Safety’s initiative to close its  London international field office, FAA reported a cost savings of $2.5  million through fiscal year 2015.  FAA calculated cost savings due to the  office closure and transfer of responsibilities to the Frankfurt international  field office from, among other things, reduced staffing, savings in office  rent, and savings in rent payments for personnel in Frankfurt.  Another  initiative for which FAA’s actions were consistent with this key practice  was AFN’s initiative to centralize FAA’s acquisition functions and identify  areas for process improvements to more efficiently distribute work and  standardize processes.  For this initiative, AFN officials had tracked a  number of acquisition-related metrics prior to the consolidation of  acquisition functions, which allowed AFN officials to examine trends in  these metrics following the consolidation. For example, one metric FAA  tracked was the number of certified contracting staff, which aligns with the  initiative’s expected benefits to standardize processes and to offer  expanded career paths for contracting professionals. Between September  2012 and September 2013, during which AFN said it completed this  consolidation, the number of certified contracting staff increased from 143  to 191. In addition to quantifiable metrics such as this, AFN also tracked  qualitative measures for expected benefits, such as clarifying authorities  and responsibilities and sharing best practices and lessons learned.
However, not all initiatives were fully consistent with the implementation  steps for the key practice. For example, the Office of Airports’ initiative to  develop standard operating procedures to standardize its regional  processes, such as grant reviews, was partially consistent with the key  practice. Airports officials stated that performance measures had not  been developed to assess the expected benefit of this initiative—to gain  necessary efficiencies. Officials further stated that no baseline information  exists, which would allow for a valid comparison of any change in overall  efficiency. Officials stated that they intend to develop performance  measures of efficiency for the initiative once they have developed and  implemented all standard operating procedures.  Until those performance  measures are developed, officials said they will only measure the degree  to which Airports employees use the new standard operating procedures.
In addition, three initiatives were inconsistent with the implementation  steps for the key practice. For example, the Office of NextGen’s initiative  to incorporate process improvements—termed “Ideas 2 In-Service” (I2I)— into its Acquisitions Management System was inconsistent with the key  practice. FAA stated in its Section 812 report that this initiative would  increase accountability and enable FAA to streamline the management of  NextGen programs and activities through a single entry point for ideas to  change the National Airspace System.  When discussing the initiative to  incorporate I2I into the Acquisitions Management System, NextGen  officials stated that no performance measures currently exist to assess  whether the Office of NextGen has achieved increased accountability or  streamlining, nor is there a plan to develop measures to assess the  performance of the initiative now or in the future.
section_title: Limited Focus on Measuring Results Hinders FAA’s Ability to Assess Whether Its Initiatives Achieved Expected Benefits
paragraphs: AFN has not effectively encouraged or coordinated performance  measurement across the offices leading the streamlining and reform  initiatives.  As a result, FAA and Congress may have limited information  on the extent to which FAA achieved the intended benefits outlined in the  Section 812 mandate. As stated previously, FAA used a decentralized  approach to respond to the Section 812 mandate. According to FAA  officials, offices leading the initiatives were responsible for identifying  initiatives and associated expected benefits. Although AFN encouraged  offices to describe expected benefits and specific metrics when initially  collecting information in April 2012 for FAA’s report to Congress, AFN did  not explicitly communicate that offices should measure and track  performance as initiatives were carried out and completed. Further, AFN  provided limited guidance and oversight to offices on how to determine  expected benefits, establish performance measures, and then track  whether they were achieved.
As a result of this limited coordination on measuring results, FAA offices  reported varied types of expected benefits across the 36 initiatives.  Specifically, offices identified a range of quantifiable and qualitative  expected benefits and reported the same types of benefits for few  initiatives, even when initiatives had similar goals. For example, AFN  reported a quantifiable benefit—cost savings—for an initiative that  consolidated strategic sourcing and other strategic initiatives into a new  organization. However, for a similar Air Traffic Organization initiative that  consolidated oversight for major system acquisitions into a new office,  FAA reported qualitative benefits, including a stronger acquisitions  community and defined program-management career paths.
Given the range of expected benefits, performance measures for FAA’s  offices also vary across the 36 initiatives. Varied performance measures  may allow FAA to better capture the unique benefits for individual  initiatives, such as fleet petroleum reduction. However, a limited focus by  AFN on communicating the importance of measuring specific results,  including those such as cost savings that may be applicable to multiple  efficiency initiatives, hinders FAA from tracking and reporting on the  overall benefit of the Section 812 effort. Offices are using or plan to use a  variety of performance measures, including quantifiable and qualitative  measures, to measure the different expected benefits, according to FAA  officials. For example, the Policy, International Affairs, and Environment  office’s initiative to facilitate an agency-wide sustainability program is  tracking nine quantifiable performance measures, including water  efficiency and alternative fuel use, on a quarterly basis against baseline  information. In another instance, AFN’s initiative to streamline and  improve its executive-level committees used qualitative measures to  assess the effectiveness of changes to executive committees.  Specifically, officials surveyed executives before and after changes were  made to committees to determine the extent to which the initiative  achieved its expected benefits, such as improved cross-organizational  decision-making. Further, according to one of the current points of contact  for the initiative to create a project management office to consolidate  oversight of major air traffic organization acquisitions, one measure of the  benefits from this initiative is that the organization has been able to  support an increasing number of programs and stakeholders without  increasing its workforce.
We have found in past work that FAA could improve its efforts to measure  the performance of large-scale program implementation efforts,  improvement initiatives, and certain oversight programs, and FAA is  taking actions to address our related recommendations. For example, we  found in September 2012 that FAA did not have performance measures  to assess whether its new safety management system approach was  improving safety, and we recommended that FAA identify and collect data  on performance measures to assess whether the new approach meets its  goals and objectives. FAA expects to have tools and processes in place  to evaluate the safety management system’s performance by April  2015. In addition, in July 2014, we found that FAA did not develop  performance metrics to measure the individual or collective outcomes of a  number of its aviation certification and approval process-improvement  initiatives, and we recommended that it develop and track measurable  performance goals. We initially identified the need for performance  measures in this area in 2010 and recommended that FAA develop and  track measurable performance goals. FAA officials responded that they  plan to develop these measures over time in three phases and will  specifically develop measures to evaluate each initiative’s outcomes.
Performance information is needed for federal programs and activities to  help inform decisions about how to address fragmentation, overlap, or  duplication and is critical for achieving results and maximizing the return  on federal funds, as we found in April 2014. In our previous work, we  have found that federal agencies engaging in large projects, such as  those FAA is currently undertaking, should establish activities to monitor  performance measures and compare actual performance to expected  benefits throughout the organization. Moreover, for federal agency  consolidation efforts, we have found that agencies should have  implementation plans that include measures that show an organization’s  progress toward achieving an intended level of performance, such as  quality, timeliness, cost of service, or customer service that the  consolidation was intended to achieve.
By not further coordinating with FAA offices on the use of objective and  balanced measures of efficiency and other improvements across  initiatives, AFN and FAA overall are missing an opportunity to more  consistently assess and aggregate the benefits from FAA’s streamlining  and reform initiatives. To help produce an objective assessment of  benefits, performance measures should typically include a quantifiable,  measurable value to the greatest extent possible. Quantifiable  measures can allow for a more useful assessment of benefits as these  measures apply numerical targets or other measurable values to such  benefits, providing a more objective comparison of benefits across  initiatives and time periods. For example, for an initiative to centralize  acquisition functions into a new office, FAA officials measured the  percentage of employees with contract specialist certifications, a metric  that demonstrated the organization’s progress towards developing a  qualified workforce. Several measures FAA officials identified, such as  improved communication, do not explicitly allow FAA to measure  efficiency or other outcomes. Further, overemphasizing certain aspects of  performance, such as improving timeliness, could result in deterioration in  other aspects of performance, such as quality. By developing a balanced  suite of measures, agencies can better ensure that they cover their  various priorities while maintaining quality.
Without a more coordinated effort to encourage offices to track  performance measures that can be aggregated across multiple initiatives,  FAA and Congress as well as other stakeholders cannot have confidence  that the agency’s efforts met or will meet the intent of Section 812 to  streamline and reform the agency. AFN recognized this need in its  description of one initiative to develop agreements to define the services  to be offered by AFN to other FAA offices, such as information technology  and acquisition functions; specifically, AFN stated that the absence of  common performance metrics for these functions makes it difficult to  determine the success or failure of efforts undertaken to consolidate  these services and thus increase operational efficiency. FAA, through its  Community of Practice for Process Improvement, is creating a database  to track information on its process improvement efforts. FAA officials said  this database will initially contain information only for the Section 812  initiatives but will eventually become a broader database on process  improvement activities across the agency. According to FAA officials, they  have not yet decided on the full range of information that the database will  capture.  FAA officials said that the database will initially contain, for each  of the initiatives, only the four items in the 812 report—problem  statement, proposed solution, expected benefits, and status—though the  content could be expanded beyond these items. Further, FAA has not  determined whether information in the database will ultimately provide a  basis for measuring of the overall or net benefit of the Section 812  response.
Lastly, Section 812 required FAA to submit a report to Congress on the  actions taken to streamline and reform the agency but did not require that  FAA track or report to Congress on the results of these actions.  As noted  above, performance information for federal programs and activities is  critical for achieving results and maximizing the return on federal funds.  If  Congress directs FAA to undertake a similar review to streamline and  reform the agency in the next authorization of FAA, Congress could help  ensure that FAA provides information on any realized efficiencies and  improvements by requiring tracking and reporting.  Without such  information on results, Congress may have difficulty fully monitoring  FAA’s efforts to make the agency more efficient and effective.  Moreover,  such a requirement would better position FAA to take steps to assess the  overall results of its efforts.  For example, FAA and its offices, if asked to  undertake and report on the results of a streamlining and reform review in  the next authorization for FAA, could take necessary steps, such as  collecting baseline information, to establish performance measures and a  basis for comparing the results in line with key practices for organizational  transformations.
section_title: Conclusions
paragraphs: FAA responded to Section 812 of the FAA Modernization and Reform Act  by working with each of its offices to identify and carry out initiatives to  streamline and reform the agency.  AFN, which led the agency’s  response to the Section 812 mandate, collected and reported information  on the 36 streamlining and reform initiatives but provided limited guidance  on measuring performance and expectations to offices leading the  initiatives.  As a result, the offices leading the initiatives determined the  status of the initiatives in different ways.  Moreover, offices identified a  wide variety of expected benefits across the initiatives and, where in  place, used a wide variety of performance measures to gauge whether  benefits were achieved for an initiative. Given the diverse nature of the 36  initiatives, some variation in how offices determined status and measured  benefits is expected.  However, FAA’s decentralized process to identify  and track the initiatives intensified the variation and makes it difficult to  discern FAA’s progress in making reforms and measure the overall  impact of the initiatives.  Without better performance measures, FAA  lacks information to help it improve the performance of the initiatives and  make decisions on issues targeted by the mandate, such as duplication  and overlap.  In addition, FAA has a limited ability to hold initiative leaders  accountable for fully implementing the initiatives and for achieving  planned benefits. Further, Congress and FAA will not know the extent to  which the agency’s efforts met the aims of Section 812—including making  the agency more efficient—without the use of some common  performance measures that FAA can use to more easily aggregate  benefits and assess results across multiple initiatives.  While AFN  provided limited guidance that likely contributed to the lack of common,  consistent performance measures across initiatives, FAA has already  responded to the Section 812 mandate.  However, as many of the  initiatives involve continuous action to realize benefits, collecting  information on the results of these initiatives through the planned  database can help FAA aggregate and report the results of the Section  812 initiatives. Moreover, by creating a mechanism to collect and manage  such information in its planned database, FAA will be better positioned to  measure the results of any future improvement and efficiency initiatives.
Further, key practices for organizational transformations and GAO’s work  on streamlining government highlight the importance of using  performance measures to show progress toward achieving desired  results and outcomes.  If Congress were to require FAA to report on  actual results of a future streamlining and reform mandate, it would obtain  information to judge whether FAA’s efforts met Congress’s intent and  produced actual benefits and assist Congress with its oversight of the  agency.  By setting such a requirement and expectation, coupled with  action by FAA to collect performance information in its planned database  for ongoing and future improvement initiatives, Congress would enable  FAA to better focus on measuring the results of any future mandated  streamlining and reform efforts.
section_title: Matter for Congressional Consideration
paragraphs: If, in the next authorization for FAA, Congress chooses to mandate that  FAA take actions to streamline and reform the agency, Congress may  wish to consider requiring FAA to (1) track measures of and (2) report to  Congress on the actual results of such efforts.
section_title: Recommendation for Executive Action
paragraphs: To better enable FAA to track, aggregate, and report on the results of its  streamlining and reform initiatives, we recommend that the Secretary of  Transportation direct FAA to develop a mechanism to capture the results  of its efficiency initiatives in its planned database for process  improvements. Measures of results might include, for example, cost  savings, timeliness, or customer service metrics, which may be common  to several types of process improvement efforts and therefore facilitate  aggregation across improvements.
section_title: Agency Comments
paragraphs: We provided a draft of this report to DOT for review and comment. In its  written comments, reproduced in appendix IV, DOT concurred with the  recommendation. DOT also provided technical comments, which were  incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of  Transportation and the appropriate congressional committees. In addition,  the report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or dillinghamg@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix V.
section_title: Appendix I: Information Related to the Status of the FAA Streamlining and Reform Initiatives
paragraphs: Table 3 provides a list of the Federal Aviation Administration’s (FAA) 36  streamlining and reform initiatives that the agency identified in response  to Section 812 of the FAA Modernization and Reform Act of 2012. As of  January 2015, FAA considered 3 initiatives in-progress (shown in italics in  table 3) and 33 initiatives implemented.  The table provides information  on the benchmark FAA used to determine that an initiative was  “implemented” for the purposes of Section 812.  The table also  summarizes FAA officials’ descriptions of the continuous action needed, if  any, to realize expected benefits for each initiative, even after it is  considered “implemented.” We categorized the continuous actions into  the four following groups: primary actions that are directly related to  realizing the initiative; secondary or related actions that enhance or  expand the initiative, monitoring actions that are conducted in order to  maintain an initiative, and none.
section_title: Appendix II: Objectives, Scope, and Methodology
paragraphs: This report examines FAA’s actions to respond to the Section 812  mandate and efforts to implement 36 initiatives to streamline and reform  the agency.  FAA identified these 36 initiatives in response to Section 812  of the FAA Modernization and Reform Act of 2012.  In particular, this  report provides information on (1) how FAA determined the status of the  streamlining and reform initiatives that the agency reported on in  response to Section 812 and (2) the extent to which FAA’s efforts to carry  out these initiatives were consistent with selected key practices for  organizational transformations.
To describe how FAA determined the status of the 36 recommendations,  we examined FAA’s Section 812 report to Congress and other agency  documents related to the streamlining and reform initiatives. We reviewed  FAA officials’ descriptions of the continuous action being taken for each  initiative to realize expected benefits, even after an initiative is considered  “implemented,” and we categorized the action in the following four  groups: primary actions that are directly related to realizing the initiative,  secondary or related actions that enhance or expand the initiative,  monitoring actions that are conducted in order to maintain an initiative,  and none/no planned continuous action. According to FAA, the agency  identified the 36 initiatives by documenting both ongoing and newly  identified improvements that were in line with the Section 812 language.   We did not assess the appropriateness of the initiatives FAA identified.  In  addition, we reviewed prior GAO reports on the Department of  Transportation’s (DOT) and FAA’s actions to implement a set of  recommendations and reports specific to topics covered by the 36  initiatives. We also interviewed FAA officials responsible for individual  initiatives as well as for coordinating the agency’s Section 812 efforts to  discuss the status of each initiative and any future plans or remaining  actions.
To examine the extent to which FAA’s efforts to carry out the 36 initiatives  were consistent with selected key practices, we identified key practices  applicable to the FAA initiatives cited in prior GAO work on organizational  transformations.  We searched past GAO publications for reports on  project management or implementation and discussed possible sources  for criteria with internal stakeholders.  We identified several relevant  reports and sources that examined organizational transformations and  streamlining government, in particular, efficiency initiatives and proposals  to consolidate infrastructure and management functions.  Most of these  reports drew on or included the key practices for mergers and  organizational transformations outlined in our 2003 report.  We selected  the key practices for organizational transformations as criteria against  which to assess FAA’s efforts since Section 812 directs FAA to review the  agency and take necessary actions to reform the agency and since the  key practices were used in past work to examine government  streamlining efforts like efficiency initiatives and consolidations.  However,  since we reviewed 36 initiatives led by offices within FAA rather than a  single, agency-wide initiative, we found that not all the key practices for  organizational transformations were relevant.  Given the scope and status  of the initiatives, we identified four key practices that were applicable.  We  determined that all four of these key practices were applicable for  evaluating the initiatives that FAA reported as “in-progress,” and two of  them—establish a communication strategy and adopt leading practices  for results-oriented strategic planning and reporting—were applicable for  evaluating the initiatives that FAA reported as “implemented.”  Table 4  lists the selected key practices, with the relevant implementation steps for  each key practice.
We systematically assessed the extent to which FAA’s efforts to carry out  an initiative were consistent with the key practices.  For each initiative, we  (1) reviewed FAA documents—schedules, communications, and other  planning documents—and (2) conducted semi-structured interviews with  the FAA point(s) of contact.  We developed a template to help us  consistently analyze the collected information related to the  implementation steps for each key practice (see table 4).  For one key  practice—adopt leading practices for results-oriented strategic planning  and reporting—we supplemented the implementation step with  requirements for performance plans outlined in An Evaluator’s Guide to  Assessing Agency Annual Performance Plans.  This guide, which was  based in part on requirements for agency performance plans from the  Government Performance and Results Act of 1993 (GPRA), identifies key  issues and criteria to assess performance plans.  Specifically, we used  criteria from the guide on defining expected performance that aligned with  the key practice and our past work on government streamlining. We  determined whether FAA’s efforts to carry out each initiative were  consistent, partially consistent, or inconsistent with each applicable key  practice, or whether there was not enough information to make an  assessment.  We used the following general decision rules to make our  assessment: partially consistent, if FAA had shown some progress toward  consistent, if FAA had instituted the practice;  instituting, or started but not completed the practice;  inconsistent, if FAA had made minimal or no progress toward  instituting the practice; and not enough information to tell, if, for example, the initiative was  implemented several years ago, and/or documentation or testimonial  evidence does not exist.
We used the FAA documentation and interviews for two initiatives to do  trial assessments using a draft version of the template; we then made  revisions to the template to clarify the information to collect and decision  rules before carrying out assessments for all initiatives.  Two analysts  reviewed the documentation to make an assessment for each initiative.   After we completed our assessments for all 36 initiatives, we identified  themes or trends in FAA’s implementation across all initiatives, including  the extent to which FAA’s efforts were consistent with the key practices.    While we reviewed FAA’s measures or plans to measure the benefits of  individual initiatives as part of our effort to assess FAA’s efforts to adopt  leading practices for results-oriented strategic planning and reporting, we  did not validate FAA’s estimates of benefits against independent  measures. Thus, we did not report on the actual or achieved benefits of  the initiatives.
In addition, we interviewed FAA officials responsible for coordinating the  agency’s Section 812 efforts to discuss the guidance given to offices and  points of contact implementing individual initiatives and the information  collected and tracked for all the streamlining and reforming initiatives.
section_title: Appendix III: Evaluation of FAA Streamlining and Reform Initiatives against Selected Key Practices
paragraphs: Table 5 provides a list of FAA’s 36 streamlining and reform initiatives, by  FAA office, which the agency identified in response to Section 812 of the  FAA Modernization and Reform Act of 2012.  The table also provides our  assessment of the extent to which FAA’s efforts to implement each  initiative were consistent with four selected key practices for  organizational transformations. Appendix II contains information on the  scope and methodology of this analysis.
section_title: Appendix IV: Comments from the Department of Transportation
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Gerald L. Dillingham, Ph.D., (202) 512-2834 or dillinghamg@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact person named above, Catherine Colwell,  Assistant Director; Melissa Bodeau; Elizabeth Curda; Kevin Egan;  Aracely Galvan; Dave Hinchman; Bert Japikse; Heather Krause; Brandon  Kruse; Joanie Lofgren; SaraAnn Moessbauer; Josh Ormond; Sarah E.  Veale; and William T. Woods made key contributions to this report.
section_title: Related GAO Products
paragraphs: Aviation Manufacturing: Status of FAA’s Efforts to Improve Certification  and Regulatory Consistency. GAO-14-829T. Washington, D.C.: July 31,  2014.
FAA Reauthorization Act: Progress and Challenges Implementing Various  Provisions of the 2012 Act. GAO-14-285T. Washington, D.C.: February 5,  2014.
Managing For Results: Executive Branch Should More Fully Implement  the GPRA Modernization Act to Address Pressing Governance  Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.
Strategic Sourcing: Leading Commercial Practices Can Help Federal  Agencies Increase Savings When Acquiring Services. GAO-13-417.  Washington, D.C.: April 15, 2013.
NextGen Air Transportation System: FAA Has Made Some Progress in  Midterm Implementation, but Ongoing Challenges Limit Expected  Benefits. GAO-13-264. Washington, D.C.: April 8, 2013.
Acquisition Workforce: DOT Lacks Data, Oversight, and Strategic Focus  Needed to Address Significant Workforce Challenges. GAO-13-117.  Washington, D.C.: January 23, 2013.
Unmanned Aircraft Systems: Measuring Progress and Addressing  Potential Privacy Concerns Would Facilitate Integration into the National  Airspace System. GAO-12-981. Washington, D.C.: September 14, 2012.
Streamlining Government: Questions to Consider When Evaluating  Proposals to Consolidate Physical Infrastructure and Management  Functions. GAO-12-542. Washington, D.C.: May 23, 2012.
Streamlining Government: Key Practices from Select Efficiency Initiatives  Should Be Shared Governmentwide. GAO-11-908. Washington, D.C.:  September 30, 2011.
NextGen Air Transportation System: FAA’s Metrics Can Be Used to  Report on Status of Individual Programs, but Not of Overall NextGen  Implementation or Outcomes. GAO-10-629. Washington, D.C.: July 27,  2010.
Air Traffic Control: FAA Reports Progress in System Acquisitions, but  Changes in Performance Measurement Could Improve Usefulness of  Information. GAO-08-42. Washington, D.C.: December 18, 2007.
Results-Oriented Cultures: Implementation Steps to Assist Mergers and  Organizational Transformations. GAO-03-669. Washington, D.C.: July 2,  2003.