Information Technology: FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed
Published: 2004-08-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Federal Aviation Administration's (FAA) mission is to promote the safe, orderly, and expeditious flow of air traffic in the United States airspace system, commonly referred to as the National Airspace System (NAS). To maintain its ability to effectively carry out this mission FAA embarked, in 1981,on a multi-billion dollar effort to modernize its aging air traffic control (ATC) system, the principle technology component of the NAS. Yet the NAS modernization has continued to be plagued by cost increases, schedule delays, and performance shortfalls. To gain insight into how FAA is meeting its management challenges, congressional requesters asked GAO to evaluate FAA's processes for making IT investment management decisions. The objectives of this review included (1) evaluating FAA's capabilities for managing its IT investments and (2) determining what plans, if any, the agency might have for improving these capabilities.
section_title: What GAO Found
paragraphs: Judged against the criteria of GAO's framework for information technology investment management (ITIM), which measures the maturity of an organization's investment management processes, FAA has established about 80 percent of the basic selection and control practices that it needs to manage its mission-critical investments. For example, business lines actively monitor projects throughout their life cycles. However, the agency's senior IT investment board does not regularly review investments that are in the "in-service management," or operational, phase of their life cycles, and this creates a weakness in FAA's ability to oversee more than $1 billion of its IT investments. In addition, the agency has not yet established the key practices that would allow it to manage all of its investments as one portfolio--an integrated set of competing options. Until FAA has established the practices that would enable it to effectively manage its annual IT budget of about $2.5 billion, agency executives lack assurance that they are selecting and managing the mix of investments that best meets the agency's needs and priorities. The agency has initiated efforts to improve its investment management processes, but it has not yet developed and implemented a comprehensive plan--supported by management--to guide all of its improvement efforts. Such a plan is crucial in helping FAA to coordinate and prioritize its improvement efforts and sustain its commitment to the efforts it already has under way. Without such a plan--and controls for implementing it--FAA will be unlikely to develop a mature investment management capability.

section_title: Letter
paragraphs: The Federal Aviation Administration’s (FAA) mission is to promote the  safe, orderly, and expeditious flow of air traffic in the United States  airspace system, commonly referred to as the National Airspace System  (NAS). To maintain its ability to effectively carry out this mission FAA  embarked, in 1981, on a multibillion dollar effort to modernize its aging air  traffic control (ATC) system, the principle technology component of the  NAS. Over the past 2 decades, individual FAA modernization projects have  experienced cost overruns, schedule delays, and performance shortfalls of  large proportions. Because of the size, complexity, cost, and problem- plagued past of FAA’s modernization program, we have designated it a high- risk information technology investment since 1995.
This report is one in a series of reports responding to your request to  evaluate FAA’s efforts to address the information technology (IT)  management challenges it faces as it continues to modernize the ATC  system. It focuses on FAA’s processes for making IT investment  management decisions and uses our Information Technology Investment  Management (ITIM) framework, which was released at a hearing of the  subcommittee on March 3, 2004. The framework provides a method for  evaluating and assessing how well an agency is selecting and managing its  IT resources. As agreed, our objectives were to (1) evaluate FAA’s  capabilities for managing its IT investments, (2) determine what plans the  agency might have for improving these capabilities, and (3) describe the  Department of Transportation’s (DOT) oversight of FAA’s investments and  investment management process. To address these objectives we analyzed  documents and interviewed agency officials to (1) validate and update  FAA’s self-assessments of the key practices in the framework, (2) evaluate  FAA’s plans for improving its capabilities, and (3) describe the department’s  oversight role. We performed our work from October 2003, through July  2004, in accordance with generally accepted government auditing  standards. Appendix I contains further details on our objectives, scope, and  methodology.
subsections: section_title: Background
subsections: section_title: FAA’s Mission and Organizational Structure
paragraphs: As an agency of the Department of Transportation, FAA’s mission is to  promote the safe, orderly, and expeditious flow of air traffic in the national  airspace. To fulfill its mission requires the extensive use of technology. The  achievement of the agency’s mission is also dependent in large part on the  skills and expertise of its workforce. Its workforce of nearly 50,000 people  provides aviation services that include air traffic control; maintenance of  air traffic control equipment; and certification of aircraft, airline  operations, and pilots.
FAA is organized into several staff support offices (examples include the  Office of Information Services and the Office of Human Resource  Management) and five lines of business, which include Airports, Regulation  and Certification, Commercial Space Transportation, the Office of Security  and Hazardous Materials, and the newly formed Air Traffic Organization  (ATO). The ATO was formed on February 8, 2004, to better provide safe,  secure, and cost-effective air traffic services now and into the future. The  Air Traffic Services and the Research and Acquisitions units, which had  been primarily responsible for managing air traffic services within FAA,  were combined into one performance-based organization to create ATO.  ATO is led by FAA’s Chief Operating Officer and consists of 10 service  units.
section_title: FAA’s Use of IT
paragraphs: FAA relies extensively on information technology to carry out its NAS  operations. It constantly depends on the adequacy and reliability of the  nation’s ATC system, which comprises a vast network of radars; automated  data processing, navigation, and communications equipment; and ATC  facilities. Through this system, FAA provides services such as controlling  takeoffs and landings and managing the flow of traffic between airports.  For example, the Integrated Terminal Weather System is employed to allow  maximum use of airport runways in all kinds of weather through a variety  of weather sensors. The Wide Area Augmentation System is used to provide  vertically guided landing to aircraft at thousands of airports and airstrips  where there is currently no vertically guided landing capability.
FAA also relies on IT to carry out its mission-support and administrative  operations (non-NAS operations). For example, FAA uses IT to support  accident and incident investigations, security inspections, and personnel  and payroll functions.
With an IT budget of about $2.5 billion for fiscal year 2004, FAA accounts  for over 90 percent of the Department of Transportation’s IT budget. The  amount of investments in both NAS and non-NAS IT is shown in the table 1  below.
section_title: Prior Reviews Identified Weaknesses in the Agency’s IT Investment Management Process
paragraphs: In 1995, we designated FAA’s modernization of its air traffic control system,  the principle technology component of the NAS, as a high-risk area because  of the size and complexity of the program and FAA’s many failures in  meeting projects’ cost, schedule, and performance goals. In our latest High- Risk Series, issued in January 2003, we addressed the critical need for FAA  to continue to improve its investment management practices—the  management processes the agency uses to select, control and evaluate the  benefits realized from its IT spending—because the agency would be  spending nearly $16 billion more through FY 2007, after having already  spent $35 billion since 1981. Other reports have also noted weaknesses in  FAA’s IT investment management processes and have made a number of  recommendations to address this area. For instance, last year we reported  that while FAA had improved its processes, several issues remained  unresolved. We noted, for example, that the agency had not yet  implemented processes for evaluating projects after implementing them, in  order to identify lessons learned and improve the investment management  process.
section_title: FAA’s Current Approach to Investment Management
paragraphs: FAA’s process for managing an IT investment varies depending on the type  of investment—NAS systems in development through the second year of  operation (F&E), NAS systems in operation after the second year (OPS),  and non-NAS systems each follow different processes. NAS investments  are managed through a standardized process, the FAA Acquisition  Management System (AMS), and non-NAS investments are managed  through a number of different processes.
subsections: section_title: Process for Managing NAS Investments
paragraphs: In April 1996, FAA implemented its AMS in response to legislation that  directed the agency to develop a new acquisition management system.  Because of FAA’s contention that some of its modernization problems were  caused by federal acquisition regulations, the Congress enacted legislation  in November 1995 that exempted the agency from most federal  procurement laws and regulations and directed FAA to develop and  implement a new acquisition management system that would address the  unique needs of the agency. AMS was intended to reduce the time and cost  for fielding new products and services by introducing (1) a new investment  management system that spans the entire life cycle of an acquisition, (2) a  new procurement system that provides flexibility in selecting and  managing contractors, and (3) organizational and human capital reforms  that support the new investment and procurement systems.
AMS provides high-level acquisition policy and guidance for selecting and  controlling FAA’s NAS investments through all phases of the acquisition life  cycle, which is organized into a series of phases and decision points that  include (1) mission analysis, (2) investment analysis, (3) solution  implementation, and (4) in-service management. To select investments,  FAA has established two processes—mission analysis and investment  analysis—which together constitute a set of policies, procedures, and  guidance that enhance the agency’s ability to screen projects that are  submitted for funding. Also, through these two processes FAA is to assess  and rank each project based on its relative costs, benefits, risks, and  contribution to FAA’s mission, and a senior, corporate-level decision- making group selects projects for funding. After a project has been  selected, FAA officials are required to formally establish the life cycle cost,  schedule, benefits, and performance baselines that are used to monitor the  project’s status throughout the remaining phases of the acquisition  management life cycle. See figure 1 for a graphic depiction of FAA’s life  cycle management process.
Several groups are involved in managing FAA’s NAS investments; they  perform functions from analysis of mission needs and alternative  investments through system development, implementation, operation, and,  ultimately, disposal. The roles and responsibilities of each group are  described below: Joint Resources Council (JRC)—This board makes corporate-level  resource and investment decisions and establishes investment programs.  Members include Associate Administrators representing FAA’s lines of  business, the FAA Acquisition Executive, the Chief Financial Officer, the  Chief Information Officer (CIO), and the Assistant Administrators for  System Safety, for Policy, Planning and International Aviation, and for  Region and Center Operations. The board is supported by the JRC  Secretariat Team, a group that facilitates the board’s processes by  maintaining the meeting calendar and guidance documents, developing  records of decisions, and providing advisory and liaison support to  programs.
Systems Engineering/Operational Analysis Team—This team performs  affordability assessments for newly proposed investments and prepares  recommendations for the reprogramming of funds from lower priority  programs. It also prepares annual budget submissions for approval by the  JRC. This team is composed of representatives from each line of business  and from other functional disciplines and is chaired by the Director, System  Architecture and Investment Analysis.
Investment Analysis Team (IAT)—This team is assembled for a relatively  short period for each specific investment being considered, to conduct the  detailed analysis of alternatives that will lead to selecting and  recommending a preferred acquisition solution. The team draws experts  from the integrated product teams, the organizational unit with the need,  the investment analysis staff, and other organizations.
Corporate Mission Analysis Organization—Performs agency-level mission  analysis and coordinates service area analysis, an activity that is conducted  during mission analysis to (1) identify capability shortfalls for or in  conjunction with service organizations, (2) ensure alignment with agency  strategic goals, and (3) eliminate redundant activity, duplicate benefits,  service gaps, and service overlaps. It also develops and maintains  standards and tools for conducting service area analysis, and it assists  service organizations in establishing a service area analysis capability.
In addition to identifying the roles and responsibilities of the groups  involved in the management process, AMS provides guidance on the  documents and decisions that result from each of the life cycle phases. For  example, through the mission analysis phase, FAA identifies critical needs  that the agency must meet for improving the safety, capacity, efficiency,  and effectiveness of the NAS. Approval of a mission need statement by the  JRC signifies that the agency agrees that the need is critical enough to  proceed to the next phase—investment analysis. During the investment  analysis phase, the IAT is to analyze and recommend a solution that best  satisfies FAA’s performance goals and customer service needs. This team is  then to rank each proposed project based on a number of factors, including  how well it meets mission needs compared to other projects and whether it  has a favorable cost-benefit ratio. As part of the JRC selection process, the  life cycle cost, schedule, benefits, and performance baselines are  established in a formal document called the acquisition program baseline  (APB), which is designed to be used by program offices to monitor a  project’s status in achieving those baselines throughout the remaining  phases of the acquisition management life cycle.
The solution implementation phase begins when the JRC approves and  funds a project, establishes its acquisition program baseline, and authorizes  the service organizations to implement and manage the project over its life  cycle. After the project has been implemented and is in operation (FAA’s in- service management phase), the service organizations monitor and assess  operational performance. Also during this phase, the project is monitored  to determine whether the current capability satisfies the demand for  services or whether another solution offers the potential for improving  safety or effectiveness or for significantly lowering costs. If the current  capability is lacking, FAA initiates a process whereby the mission need  would be revalidated and the investment analysis process begun again,  possibly leading to a new investment decision. Figure 2 provides detail on  the phases of FAA’s IT investment management process and decision  points. The highlighted decision points represent those for which the JRC  must make an approval decision before a project can move forward.
Senior executives have stated that with the reorganization of the ATO in  February 2004, discussions have been held about realigning the investment  management process to make the heads of the service units responsible  and accountable for managing programs’ capital investments and operating  costs from inception to retirement. In the past, the business units have  been organized to manage either capital investments or operating costs,  but not both. These discussions have not yet led to specific changes in  FAA’s investment management processes and responsibilities.
While the AMS was intended to apply to all FAA investment programs, it  has not been implemented for non-NAS investments. Each of the agency’s  business line and staff offices that manage non-NAS investments has  implemented its own processes for managing these investments. Examples  of these various non-NAS investment processes include the following:  Regarding an investment management board structure, the Financial  Services staff office has an informal board consisting of the Chief  Financial Officer, Deputy Chief Financial Officer, and heads of offices  within Financial Services. The Financial Services life cycle process  guide directs the board’s operations. In the Regulation and Certification  unit, the senior management team makes investment management  decisions with input from the Chief Information management team. This  unit is developing an IT investment management processes guide, which  is expected to be completed by the end of the fiscal year.
When selecting investments, the Human Resource Management unit  uses its established annual budget formulation process, while the  Region and Center Operations unit is moving toward a new process  whereby in order to be selected investments need to demonstrate, at a  minimum, that they (1) are compliant with FAA’s architecture, (2) have a  business sponsor, (3) have a solid business case, and (4) can be funded.
In controlling investments, Information Services has developed  processes to monitor contract expenditures, and unit managers  regularly perform financial management reviews of the programs under  their purview, but there is no structured process for oversight of  projects’ performance against expectations. In the Human Resource  Management unit, division managers hold quarterly reviews to assess  projects’ progress in meeting cost and schedule expectations and  aligning with strategic goals.
Descriptions of the processes used by each of the units responsible for  managing non-NAS investments can be found in appendix II.
In January 2004, the FAA Administrator established the Information  Technology Executive Board (ITEB) to “strengthen FAA’s ability to use IT  as an agencywide strategic asset” and “guide fundamental changes in the  governance of IT assets.” Its charter calls for the ITEB to assume  responsibility for making investment decisions about non-NAS IT  investments. However, the ITEB has not yet implemented this aspect of its  charter. Therefore, at the current time there is no single board or  investment management process for non-NAS investments that would be  analogous to the JRC board and AMS process that are used for NAS  investments.
section_title: ITIM Maturity Framework
paragraphs: The ITIM framework is a maturity model composed of five progressive  stages of maturity that an agency can achieve in its investment  management capabilities. It was developed on the basis of our research  into the IT investment management practices of leading private- and public- sector organizations. The framework identifies critical processes for  making successful IT investments, organized into the five increasingly  mature stages. These maturity stages are cumulative; that is, in order to  attain a higher stage of maturity, the agency must have institutionalized all  of the requirements for all of the lower stages, in addition to those for the  higher stage.
The ITIM can be used both to assess the maturity of an agency’s investment  management processes and as a tool for organizational improvement. The  overriding purpose of the framework is to encourage investment processes  that increase business value and mission performance, reduce risk, and  increase accountability and transparency in the decision process. We have  used the framework in several of our evaluations, and a number of  agencies have adopted it. These agencies have used ITIM for purposes  ranging from self-assessment to redesign of their IT investment  management processes.
ITIM’s five maturity stages represent steps toward achieving stable and  mature processes for managing IT investments. Each stage builds on the  lower stages; the successful attainment of each stage leads to improvement  in the organization’s ability to manage its investments. With the exception  of the first stage, each maturity stage is composed of “critical processes”  that must be implemented and institutionalized in order for the  organization to achieve that stage. These critical processes are further  broken down into key practices that describe the types of activities that an  organization should be performing to successfully implement each critical  process. An organization may be performing key practices from more than  one maturity stage at the same time. This is not unusual, but efforts to  improve investment management capabilities should focus on becoming  compliant with lower-stage practices before addressing higher-stage  practices.
Stage 2 of the ITIM framework encompasses building a sound investment  management process by establishing basic capabilities for selecting new IT  projects. It also involves developing the capability to control projects so  that they finish predictably within established cost and schedule  expectations and the capability to identify potential exposures to risk and  put in place strategies to mitigate that risk. The basic selection processes  established in Stage 2 lays the foundation for more mature selection  capabilities in Stage 3.
Stage 3 requires that an organization continually assess both proposed and  ongoing projects as parts of a complete investment portfolio—an  integrated and competing set of investment options. It focuses on  establishing a consistent, well-defined perspective on the IT investment  portfolio and maintaining mature, integrated selection (and reselection),  control, and evaluation processes, which are to be evaluated during  postimplementation reviews (PIR). This portfolio perspective allows  decision makers to consider the interaction among investments and the  contributions to organizational mission goals and strategies that could be  made by alternative portfolio selections, rather than relying exclusively on  the balance between the costs and benefits of individual investments.
Stages 4 and 5 require the use of evaluation techniques to continuously  improve both the investment portfolio and investment processes in order  to better achieve strategic outcomes. At Stage 4 maturity an organization  has the capacity to conduct IT succession activities and therefore can plan  and implement the deselection of obsolete, high-risk, or low-value IT  investments. An organization with Stage 5 maturity conducts proactive  monitoring for breakthrough information technologies that will enable it to  change and improve its business performance. Organizations implementing  Stages 2 and 3 have in place the selection, control, and evaluation  processes that are required by the Clinger-Cohen Act. Stages 4 and 5 define  key attributes that are associated with the most capable organizations.
Figure 3 shows the five maturity stages and the critical processes  associated with each.
As defined by the model, each critical process consists of “key practices”  that must be executed to implement the critical process.
section_title: FAA Has Established Many Key Practices for Managing NAS Investments but Lacks Oversight of Operational Systems
paragraphs: In order to have the capabilities to effectively manage IT investments, an  agency should, at a minimum, (1) build an investment foundation by  putting basic, project-level control and selection practices in place (Stage 2  capabilities) and (2) manage its projects as a portfolio of investments,  treating them as an integrated package of competing investment options  and pursuing those that best meet the strategic goals, objectives, and  mission of the agency; and it should also conduct PIRs to maintain mature,  integrated selection, control, and evaluation processes (Stage 3  capabilities). In addition, an agency would be well served by implementing  capabilities for improving its investment process through performance  evaluations of its portfolio and succession management of current  investments (Stage 4 capabilities). In order to develop the capabilities to  effectively manage its investments, FAA would, at minimum, need to  implement Stage 2 capabilities for both its NAS and non-NAS investments  and Stage 3 capabilities for its portfolio of investments.
FAA’s investment management capabilities vary depending on whether an  investment is considered to be NAS or non-NAS. Specifically:  For NAS investments, FAA has executed 30 of the 38 Stage 2 key  practices that are required to establish a foundation for investment  management maturity. For these investments, the agency has in place a  strong set of processes to support investment management, although  the JRC does not regularly review investments that have passed into the  in-service management phase (i.e., operational systems).
For its non-NAS investments, the agency has not yet adequately  implemented a single management line of responsibility and the  standard processes needed to manage in a consistent manner. Although  some structured processes exist within individual business units, this  lack of consistency undermines the agency’s maturity.
In Stage 3, the lack of regular JRC oversight of operational systems and  the absence of a structured approach to managing non-NAS investments  prevent FAA from managing its investments as a portfolio that includes  all major NAS and non-NAS investments. In addition, the agency is not  conducting PIRs on its major investments.
FAA has not executed any of the Stage 4 key practices for managing the  succession of its information systems, although the agency has begun to  address this weakness by defining procedures for retiring investments  in the AMS.
When FAA implements all of the key practices associated with building the  investment foundation and managing its investments as a portfolio, the  agency will have greater assurance that it has selected the mix of  investments that best supports its strategic goals and that it will be able to  manage the investments to successful completion.
section_title: FAA Has Established Much of the Foundation Needed to Manage Its NAS Investments
paragraphs: At the ITIM Stage 2 level of maturity, an organization has attained  repeatable, successful IT project-level investment control processes and  basic selection processes. Through these processes, the organization can  identify expectation gaps early and take appropriate steps to address them.  According to ITIM, critical processes at Stage 2 include (1) defining IT  investment board operations, (2) identifying the business needs for each  IT investment, (3) developing a basic process for selecting new IT  proposals and reselecting ongoing investments, (4) developing project- level investment control processes, and (5) collecting information about  existing investments. Table 2 describes the purpose of each of the Stage 2  critical processes.
To its credit, FAA has put in place about 80 percent of the key practices  associated with managing its NAS investments through the Stage 2 critical  processes. The agency has satisfied all of the key practices associated with  capturing investment information and most of those associated with  instituting the investment board, meeting business needs, selecting an  investment, and providing investment oversight. Most of the weaknesses in  these critical processes relate to NAS investments in the in-service  management phase. Table 3 summarizes the status of FAA’s critical  processes for Stage 2, showing how many key practices FAA has executed  in managing its NAS investments.
section_title: FAA Has Established an IT Management Structure to Manage Its NAS Investments
paragraphs: The establishment of decision-making bodies or boards is a key component  of the IT investment management process. At the Stage 2 level of maturity,  organizations define one or more boards, provide resources to support  their operations, and appoint members who have expertise in both  operational and technical aspects of proposed investments. The boards  operate according to a written IT investment process guide that is tailored  to the organization’s unique characteristics, thus ensuring that consistent  and effective management practices are implemented across the  organization. Once board members are selected, the organization ensures  that they are knowledgeable about policies and procedures for managing  investments. Organizations at the Stage 2 level of maturity also take steps  to ensure that executives and line managers support and carry out the  decisions of the IT investment board. According to ITIM, an IT investment  management process guide should be a key authoritative document that  the organization uses to initiate and manage IT investment processes and  should provide a comprehensive foundation for the policies and  procedures that are developed for all of the other related processes. (The  complete list of key practices is provided in table 4.)
FAA has executed 7 of the 8 key practices for this critical process. For  example, in 1996, Congress directed FAA to develop a new acquisition  management system as part of a broad mandate for acquisition reform at  the agency. In response, FAA implemented AMS in April 1996. AMS  establishes policy and guidance for all aspects of the agency’s acquisition  life cycle and documents the investment management process used for  NAS investments. The agency established the JRC as its corporate-level  investment board for the NAS investments. The JRC makes select and  control decisions, including corporate decisions on mission needs,  acquisition investments, and acquisition program baseline changes; it also  reviews and recommends approval of the agency’s F&E budget submission.
The board is adequately resourced to support its operations. The JRC  Secretariat Team supports the board in such ways as developing and  updating guidance, scheduling meetings, and preparing and executing the  JRC readiness process. In addition, the Mission Analysis Steering Group is  responsible for assisting the board in prioritizing mission needs, while the  Systems Engineering/Operational Analysis Team is to assist in addressing  budget issues among investments. The JRC consists of senior officials from  both business and IT areas, including the Chief Information Officer and the  associate administrators representing FAA lines of business. These  members are to exhibit the core competencies required by FAA in selecting  executives and in assessing executive training needs. In addition, the  agency offers a 3-day AMS overview course for all employees, including  JRC members. Although the board as an entity does not oversee the  development and maintenance of AMS, it is involved through FAA’s  Acquisition System Advisory Group, which evaluates all proposed changes  to AMS. To ensure that the board’s decisions are carried out, an acquisition  program baseline document is approved at the JRC final investment  decision point; this document identifies the capabilities, benefits, costs,  and schedule for the approved investment, which are monitored by FAA  through its variance reporting process.
Despite these strengths, FAA has not yet clearly defined the relationship  between the JRC and the newly formed ITEB. Although the ITEB was  established by the Administrator to function as the central authority  responsible for assuring that FAA IT investments are based on sound  business practices, FAA has not yet clearly delineated the specific roles the  ITEB is to play and the relationship it will have with the JRC. This task has  been assigned to the ITEB as a longer-range initiative.
Table 4 shows the rating for each key practice required to implement the  critical process for instituting the investment board at the Stage 2 level of  maturity. Each of the “Executed” ratings shown below represents instances  where, based on the evidence provided by FAA officials, we concluded that  the specific key practices were executed by the organization.
subsections: section_title: FAA Has a Process for Ensuring That its Investments Support Business Needs and Meet Users’ Needs
paragraphs: Defining business needs for each IT project helps to ensure that projects  and systems support the organization’s business needs and meet users’  needs. This critical process ensures that a link exists between the  organization’s business objectives and its IT management strategy.  According to ITIM, effectively meeting business needs requires, among  other things, (1) documenting business needs with stated goals and  objectives, (2) identifying specific users and other beneficiaries of IT  projects and systems, (3) providing adequate resources to ensure that  projects and systems support the organization’s business needs and meet  users’ needs, and (4) periodically evaluating the alignment of IT projects  and systems with the organization’s strategic goals and objectives. (The  complete list of key practices is provided in table 5.)
FAA has in place 6 of the 7 key practices for meeting business needs. The  agency’s AMS and mission analysis guidance calls for business needs for  both proposed and ongoing IT projects and systems to be identified in the  mission need statement developed during the mission analysis phase. FAA  also has detailed procedures for developing this document that call for  identifying business needs. Resources for ensuring that IT projects and  systems support the organization’s business needs and meet users’ needs  include service organizations, the Corporate Mission Analysis  Organization, the Mission Analysis Steering Group, and detailed  procedures and associated templates for developing mission need  statements. FAA’s specific business mission, with stated goals and  objectives, is defined in the Federal Aviation Administration Flight Plan for  fiscal years 2004 through 2008.
Further, FAA defines and documents business needs for both proposed and  ongoing IT projects and identifies users and other beneficiaries during its  mission analysis activities. In addition, the AMS policy calls for users to  participate in project management throughout the FAA life cycle  management process. For the three projects we reviewed, we verified that  business needs and specific users and other beneficiaries were identified  and documented in mission needs statements as well as in other  documents. In addition, users are involved in project management  throughout the life cycle of the projects. For example, according to project  officials, En Route Communications Gateway (ECG) users participate in  project meetings, weekly integrated product team status meetings, and  monthly En Route domain national deployment teleconferences. FAA  Telecommunications Infrastructure’s (FTI) end users are heavily involved  in the “operational test” period, which determines whether the equipment  can be safely implemented in NAS. VSCS Control Subsystem Upgrade  users are involved in the project’s life cycle via a Web site through which  they review and comment on project documentation.
Despite these strengths, the JRC has no process for evaluating the  organizational alignment of NAS systems through most of their in-service  management phase (and non-NAS investments, which are described  separately in this report). While the JRC does evaluate the alignment of  projects and systems with organizational goals throughout the systems’  development and 2 years into their operations as part of the annual budget  formulation process, it does not use any consistent process to review  projects and systems after that point in their life cycles. For NAS systems in  the in-service management phase, these activities are carried out within the  business unit that owns the system, but the JRC does not regularly oversee  these processes and may go for several years without reviewing a system’s  alignment with organizational goals. In-service NAS systems only return to  the JRC if they are judged to require additional funds for correction. Until  FAA establishes a process for periodic evaluation of systems throughout  the in-service management phase and takes corrective actions when  misalignment occurs, the agency will not be able to ensure that these  projects, totaling about $1.3 billion per year, are still continuing to maintain  alignment with the FAA’s strategic plans and its business goals and  objectives.
Table 5 shows the rating for each key practice required to implement the  critical process for meeting business needs at the Stage 2 level of maturity  and summarizes the evidence that supports these ratings.
section_title: FAA Has a Disciplined Process for Selecting New IT Proposals but Lacks a Similar Process for Reselecting Ongoing Investments
paragraphs: Selecting new IT proposals and reselecting ongoing investments requires a  well-defined and disciplined process to provide the agency’s investment  board, business units, and developers with a common understanding of the  process and the cost, benefit, schedule, and risk criteria that will be used  both to select new projects and to reselect ongoing projects for continued  funding. According to ITIM, this critical process requires, among other  things, (1) making funding decisions for new proposals according to an  established process; (2) providing adequate resources for investment  selection activities; (3) using a defined selection process to select new  investments and reselect ongoing investments; (4) establishing criteria for  analyzing, prioritizing, and selecting new IT investments and for reselecting  ongoing investments; and (5) creating a process for ensuring that the  criteria change as organizational objectives change. (The complete list of  key practices is provided in table 6.)
FAA has executed 7 of the 10 key practices associated with selecting an  investment. For example, the AMS establishes two processes—mission  analysis and investment analysis—that together constitute a set of policies  and procedures, as well as guidance that is designed to enhance the  agency’s ability to select investments. In addition, FAA has policies and  procedures for its annual F&E budget formulation process to reselect  ongoing IT projects. Also, FAA’s AMS sets forth policies and procedures for  reselecting ongoing IT investments by identifying their capability shortfalls  and addressing them as new investments.
The AMS also integrates funding with the process of selecting an  investment by requiring the Systems Engineering/Operational Analysis  Team to perform affordability assessments for new proposed investment  programs; it may recommend funding reallocations from lower priority  programs when an alternative solution cannot be funded within FAA  planning and budgeting baselines. This team also supports the JRC to  ensure that the executives’ funding decisions are aligned with selection  decisions during the investment analysis activities.
Resources for proposal selection activities include the program director,  the Integrated Product Team, and the Investment Analysis Team, as well as  detailed procedures and a template that have been defined for developing  investment analysis reports. The investment analysis reports identify the  evaluation criteria used, the alternatives analyzed, and the ranking of each  alternative so that the JRC can select the best overall solution identified in  the mission need statement. The criteria that were established during the  initial investment analysis phase are used by the Investment Analysis Team  to rank each proposed project on the basis of how well it meets the  agency’s mission needs compared with other projects.
FAA uses the processes defined in the AMS for selecting new IT  investments. In addition, it uses two processes to reselect ongoing IT  investments. Specifically, the FAA uses its annual budget formulation  process for projects in development or in the first 2 years of operations. It  also uses the AMS process when a system’s capability shortfall is identified,  and it treats the correction of the shortfall as a new investment. The  managers of the three projects we reviewed confirmed that their projects  were selected using the AMS process. One project’s officials stated that this  included market, alternatives, investment, and affordability analyses. The  program managers also stated that the annual F&E budget formulation  process is used to reselect their projects. These project officials also noted  that if a project is scheduled for a hardware replacement, a reselection is  done. The AMS process is followed to explore new alternatives and make  sure the replacement is in the best interest of the government.
Despite these strengths, FAA has not developed similarly strong processes  for NAS investments more than 2 years into their operations—those NAS  systems that are in the in-service management phase. For example, while  FAA’s F&E budget formulation process establishes criteria for analyzing,  prioritizing, and reselecting IT investments for systems in development or  up until 2 years into operations, neither of the two processes used to  reselect IT investments has established criteria for investments beyond 2  years into operations. In addition, while FAA uses its annual budget  formulation process to reselect projects that are part of the F&E budget,  the agency does not have an analogous reselection process as part of its  operations budget formulation. Until FAA establishes consistent criteria for  reselecting all of its IT investments, it will not be adequately assured that it  is consistently and objectively continuing to fund ongoing projects that still  meet the needs and priorities of the agency in a cost-effective and risk- insured manner.
Table 6 shows the rating for each key practice required to implement the  critical process for selecting an investment at the Stage 2 level of maturity  and summarizes the evidence that supports these ratings.
section_title: FAA Does Not Have a Process for Effectively Overseeing Investments in All Phases of Their Life Cycles
paragraphs: An organization should provide effective oversight for its IT projects  throughout all phases of their life cycles. Its investment board should  maintain adequate oversight and observe each project’s performance and  progress toward predefined cost and schedule expectations as well as each  project’s anticipated benefits and risk exposure. The investment board  should also employ early warning systems that enable it to take corrective  action at the first sign of cost, schedule, or performance slippages. This  board has ultimate responsibility for the activities within this critical  process. According to ITIM, effective project oversight requires, among  other things, (1) having written policies and procedures for management  oversight; (2) developing and maintaining an approved management plan  for each IT project; (3) making up-to-date cost and schedule data for each  project available to the oversight boards; (4) having regular reviews by  each investment board of each project’s performance against stated  expectations; and (5) ensuring that corrective actions for each  underperforming project are documented, agreed to, implemented, and  tracked until the desired outcome is achieved. (The complete list of key  practices is provided in table 7.)
FAA has in place 4 of the 7 key practices associated with effective project  oversight. The agency has developed written policies and procedures for  management oversight of its investments. These include (1) AMS; (2) the  integrated program plan, which is the detailed planning document for all  aspects of a program’s implementation, including program control; and  (3) the Integrated Baseline Establishment and Management Process  document for reporting variances from the performance expectations  approved by the JRC in the acquisition program baseline.
We verified that cost, schedule, benefit, and risk expectations were  documented in the acquisition program baseline and that the integrated  program plan contained details for project execution for En Route  Communications Gateway and FAA Telecommunications Infrastructure.  For the VSCS Control Subsystem Upgrade, performance expectations and  details on project execution were both captured in the integrated program  plan. In addition, the JRC Secretariat Team maintains a tracking system  for action items that are assigned during a project’s acquisition reviews,  including the action to be taken, the responsible FAA organization, and  whether the underlying problem has been resolved.
FAA has not established processes that bring investments before the JRC  for oversight on a regular basis. There is a process for reporting variances  from the performance expectations that were approved by the JRC in the  investment’s acquisition program baseline. However, although this process  is carried out as part of the F&E budget formulation for IT investments in  development or less than 2 years into operations, it is not being carried out  for investments that are part of the operations budget. Investments that are  meeting performance expectations may not return to the JRC for several  years. FAA also conducts acquisition reviews as a means for program  offices to report to agency executives on the status of investments  compared to program baselines. However, since program offices may  select which investments they wish to bring forward for review, many  investments may never come forward. Until FAA develops (1) procedures  for reporting on an investment throughout its entire acquisition life cycle  and (2) mechanisms for ensuring that all investments are reviewed  regularly, the agency is placing itself at risk that underperforming  investments will not be reported to the JRC in order for it to take  appropriate actions.
Table 7 shows the rating for each key practice that is required to implement  the critical process for project oversight at the Stage 2 level of maturity and  summarizes the evidence that supports these ratings.
section_title: FAA Has a Structured Process for Capturing Investment Information and Using It to Support Investment Management
paragraphs: To make good IT investment decisions, an organization must be able to  acquire pertinent information about each investment and store that  information in a retrievable format. During this critical process an  organization identifies its IT assets and creates a comprehensive repository  of investment information. This repository provides information to  investment decision makers to help them evaluate the impacts and  opportunities that would be created by proposed or continuing  investments. It can provide insights and trends about major IT cost and  management drivers. The repository can take many forms and does not  have to be centrally located, but the collection method should identify each  IT investment and its associated components. This critical process may be  satisfied by the information contained in the organization’s current  enterprise architecture, augmented by additional information—such as  financial information and information on risk and benefits—that the  investment board may require to ensure that informed decisions are being  made. According to ITIM, effectively managing this repository requires,  among other things, (1) developing written policies and procedures for  identifying and collecting the information, (2) assigning responsibility for  ensuring that the information being collected meets the needs of the  investment management process, (3) identifying IT projects and systems  and collecting relevant information to support decisions about them, and  (4) making the information easily accessible to decision makers and others.  (The complete list of key practices is provided in table 8.)
FAA’s AMS guidance identifies specific information that is needed in the  investment management process, including information for its investment  analysis phase. FAA maintains a number of repositories of relevant  information, including its Simplified Program Information Reporting &  Evaluation database, which reports variances in cost, schedule,  performance, or benefits from an investment’s approved acquisition  program baseline. The information that is collected is made available to the  JRC in several documents, including program plans and the acquisition  program baseline document. The JRC Secretariat Team ensures that the  investment board has all the relevant information it needs for its decision- making process.
Table 8 shows the rating for each key practice required to implement the  critical process for capturing investment information at the Stage 2 level of  maturity and summarizes the evidence that supports these ratings.
section_title: FAA Does Not Have Structured Processes to Manage Its Non-NAS Investments
paragraphs: FAA does not have a single set of processes for making consistent basic  selection and control decisions for its non-NAS investments (Stage 2  capabilities). As previously discussed in the background section of this  report, several business units within FAA make decisions about non-NAS  investments. We reviewed the investment management processes of seven  of these units—Information Services, Region and Center Operations,  Regulation and Certification, Financial Services, Research and Acquisition,  Air Traffic Services, and Human Resource Management. Appendix II  describes the investment management processes we found in these units.  The extent to which these processes comply with the ITIM framework for  Stage 2 varies considerably by business unit, and FAA currently does not  specify non-NAS investment management processes in a coordinated  manner. Since the ITIM framework calls for a consistent investment  management process, we assessed FAA’s non-NAS investment management  capability at an aggregate level. That is, we assessed FAA’s capability to  manage its non-NAS investments, not the capability of each individual  business unit. Even though individual business units may have some of  these processes in place, FAA as a whole has not yet defined  an investment management structure that allows the agency to  consistently manage its non-NAS investments,  a uniform process for ensuring that non-NAS investments are linked to  business needs and meet users’ needs,  a process for selecting new IT proposals and reselecting ongoing   a single process for reviewing the progress of investments and taking  corrective action when performance expectations are not being met, or  a comprehensive inventory of project and system information to  support investment decisions.
According to FAA officials, the agency has not defined a coherent  investment management structure and a set of processes for non-NAS  investments in the past because many of these investments have not had  the agencywide impact of the NAS investments. However, because there is  now recognition that a disciplined approach to managing non-NAS  investments could help control FAA’s IT assets and costs in general, efforts  are currently under way to address this weakness. As previously discussed,  an IT Executive Board (ITEB) has been chartered with responsibility for,  among other things, making decisions about non-NAS IT investments, but it  has not yet taken action on developing a standard process. Until FAA fully  establishes the consistent practices it needs to make basic project selection  and control decisions, executives will be hampered in their ability to  effectively manage non-NAS investments and ultimately to find the  opportunities to achieve the cost savings they are seeking.
section_title: FAA Lacks Key Capabilities Needed to Manage All IT Investments as a Portfolio and Does Not Conduct Postimplementation Reviews
paragraphs: During Stage 3, the investment board enhances the IT investment  management process by developing a complete investment portfolio and  carrying out PIRs. An IT investment portfolio is an integrated, agencywide  collection of investments that are assessed and managed collectively on the  basis of common criteria. Managing investments within the context of such  a portfolio is a conscious, continuous, and proactive approach to  expending limited resources on an organization’s competing initiatives in  light of the relative benefits expected from these investments. Taking an  agencywide perspective enables an organization to consider its  investments comprehensively, so that collectively the investments  optimally address the organization’s missions, strategic goals, and  objectives. Managing IT investments with a portfolio approach also allows  an organization to determine priorities and make decisions about which  projects to fund, and continue to fund, based on analyses of the relative  organizational value and risks of all projects, including projects that are  proposed, under development, and in operation. For an organization to  reap the full benefits of the portfolio process, it should collect all of its  investments into an enterprise-level portfolio that is overseen by its senior  investment board. Although investments may initially be selected into  subordinate portfolios—based on, for example, lines of business or life  cycle stages—and managed by subordinate investment boards, they should  ultimately be aggregated into this enterprise-level portfolio.
The purpose of a PIR is to evaluate an investment after its development has  been completed (i.e., after its transition from the implementation phase to  the in-service management phase) in order to validate actual investment  results. This review is conducted to (1) examine differences between  estimated and actual investment costs and benefits and their possible  ramifications for unplanned funding needs in the future and (2) extract  “lessons learned” about the investment selection and control processes  that can be used as the basis for management improvements. Similarly,  PIRs should be conducted for investment projects that were terminated  before completion, to help to readily identify potential management and  process improvements.
According to ITIM, critical processes performed by Stage 3 organizations  include (1) defining the portfolio criteria, (2) creating the portfolio,  (3) evaluating the portfolio, and (4) conducting PIRs. Table 9 shows the  purpose of each critical process in Stage 3.
FAA has executed only 1 of the 27 key practices associated with Stage 3  critical processes: it has a process for distributing portfolio criteria to  project management personnel and other stakeholders. The remaining 26  key practices were not executed—primarily because FAA does not involve  the JRC in the regular oversight of non-NAS investments or in NAS  investments during their in-service management phase, weaknesses that  we noted in our assessment of Stage 2 requirements. Since Stage 3 requires  an enterprisewide perspective, the lack of oversight of these classes of  investments precludes the successful completion of most Stage 3 critical  processes. In addition, Stage 3 requires an enterprisewide perspective that  FAA has not adopted, which would enable the JRC to oversee all major IT  investments, regardless of life cycle phase or business unit. Although it can  be appropriate for FAA to manage its NAS, in-service NAS, and non-NAS  investments as separate subordinate portfolios—depending on the  successful execution of all Stage 2 key practices—its enterprise-level  portfolio should contain all major IT investments regardless of life cycle  stage or business line. In building this enterprise-level portfolio, the JRC  can choose whether to include specific investments based on  predetermined criteria, as described by the ITIM framework. Until FAA  fully implements the critical processes associated with managing its  investments as a complete portfolio, it will not have the data or  enterprisewide perspective it needs to make informed decisions about all  of its major IT investments.
In addition, FAA has not executed the six key practices for conducting  PIRs. In June 2004, in response to a recommendation contained in our 1999  report that FAA initiate PIRs for projects or programs within 3 to 12  months of deployment or termination, the NAS Configuration Management  and Evaluation Staff developed a proposed approach to PIRs, but this  approach was not implemented. In November 2003, the life cycle  management policy team proposed a change to the AMS that would require  conducting these reviews, but there has been no action on the proposal.  Although the JRC has recently reaffirmed its commitment to implement  PIRs, there is no policy and no established process to carry them out. If  PIRs are not conducted on a routine basis, then FAA will not be able to  effectively evaluate the results of its IT investments; this will affect the  agency’s ability to determine whether to continue, modify, or terminate an  IT investment in order to meet its stated mission objectives.
Table 10 summarizes the status of FAA’s critical processes for Stage 3,  showing how many associated key practices it has executed.
section_title: FAA Has Not Established a Process for Managing the Succession of Its Information Systems
paragraphs: Once an agency has attained Stage 3 maturity, it evaluates its IT investment  processes and portfolios to identify opportunities for improvement  (Stage 4 capabilities). This entails (1) improving the portfolio’s  performance and (2) managing systems and technology succession. We did  not assess FAA’s capability for improving the portfolio’s performance,  because it did not claim to be executing any of the relevant key practices in  its self-assessment.
According to ITIM, regarding system and technology succession  management includes (1) defining policies and procedures for managing  the IT succession process, (2) assigning responsibility for the IT succession  process, (3) developing criteria for identifying IT investments that may  meet succession status, and (4) periodically analyzing IT investments to  determine whether they are ready for succession. This critical process  enables an organization to recognize low-value or high-cost IT investments  and augments the routine replacement of systems at the end of their useful  lives. It also promotes the development of a forward-looking, solution- oriented view of IT investments that anticipates future resource  requirements and allows the organization to plan appropriately. This  process differs from the reselection activity in Stages 2 and 3 in that it  focuses on anticipating and planning for the retirement of legacy systems  and on meeting remaining requirements with other, perhaps new, systems.  In addition, succession management takes place at the end of a system’s  life cycle.
FAA has not executed any of the nine key practices required to implement  this critical process. Although the agency has defined procedures in AMS  for retiring investments, it still needs to describe how to regularly review  systems that are in operations in order to identify candidates for  retirement. According to FAA, decisions on succession are made by the  service organizations. However, no individual or group has been assigned  responsibility for managing the succession process from an enterprise  perspective, which would allow the FAA to better anticipate and plan for  future resource requirements. Without an institutionalized process for  succession management, the FAA may not be able to identify those IT  investments that are eligible for succession in enough time to minimize the  effect of the transition on their successors. In addition, by establishing an  effective succession management process, the agency can identify systems  for retirement, freeing resources for other, superior, investments.
section_title: FAA Has Initiated Efforts to Improve Its Investment Management Process
paragraphs: We have previously reported that to effectively implement IT investment  management processes, organizations need to be guided by a plan that  (1) is based on an assessment of strengths and weaknesses; (2) specifies  measurable goals, objectives, and milestones; (3) specifies needed  resources; (4) assigns clear responsibility and accountability for  accomplishing tasks; and (5) is approved by senior management.
FAA has begun to take steps to resolve some of the weaknesses identified  in this report. For example, at a June 10, 2004, meeting, the JRC decided to  incorporate budget justification documents (Exhibit 300s), which are  currently prepared for the Office of Management and Budget (OMB) as part  of the President’s Budget formulation process, into the AMS process for  managing NAS investments. The Exhibit 300 will become the board’s  decision-making document, and essential information from existing AMS- required documents—the investment management report, the acquisition  strategy paper, the integrated program plan, and the requirements  documents—will be incorporated into the Exhibit 300. The JRC also  recently decided to implement PIRs in order to track metrics during  program implementation. Finally, at that same meeting, the board decided  to collectively determine, at the meeting where the F&E budget is  approved, which F&E and OPS programs should be brought forward for  review the following year. This decision serves to bring certain investments  in the in-service management phase under the JRC’s direct purview,  although it does not specify that consistent criteria be established, as the  ITIM framework requires.
FAA has also begun to initiate steps to bring more clarity to the ITEB’s  responsibilities, although the specifics have yet to be defined. In its charter,  the ITEB is charged with making investment decisions about non-NAS IT  investments. This action would begin to bring all of the non-NAS  investments under a single authority. The charter suggests that the ITEB  choose among three options: (1) to send major non-NAS investment  decisions to the JRC, (2) to make the decision itself, given an acceptable  review process similar to the JRC processes, or (3) have the CIO, Chief  Financial Officer, and owning assistant/associate administrator make the  decision jointly. This description of the ITEB’s roles and responsibilities  further alludes to the senior board’s evolving responsibility toward major  non-NAS IT investments, although it falls short of laying out specific  criteria for selecting which investments should be sent forward to the JRC.  The ITEB has been given responsibility for four short-term initiatives as  well, including establishing an agencywide cost control program for non- NAS expenditures and ensuring that all OMB Exhibit 300s receive a passing  grade for the 2006 budget year. The ITEB has been charged with the long- term initiative of clearly delineating the roles it plays and its relationship  with the more senior board. The successful completion of this initiative is  likely to satisfy the single key practice that FAA has not yet executed in the  Instituting the Investment Board critical process of the ITIM.
The Chief Operating Officer’s recent reorganization of the ATO is intended  to make the heads of the service units responsible for IT projects from their  inception through the in-service management phase. This new organization  is designed to support his expressed intentions to increase accountability  for systems in operation in order to manage costs more effectively.  According to the Chief Operating Officer, FAA recognizes that good  processes are needed for both NAS and non-NAS to improve the way the  agency manages its investments.
While FAA has initiated these improvement efforts, it has not linked them  together in a plan with the characteristics listed above that would help  coordinate and guide the efforts. Until FAA develops a plan that would  allow for the systematic prioritization, sequencing, and evaluation of  improvement efforts, the agency risks not being able to effectively  establish mature investment management processes.
section_title: DOT is Taking Steps to Integrate Oversight of FAA’s IT Investments
paragraphs: DOT has recently initiated several efforts that can serve to provide better  departmental oversight of FAA investments. This fiscal year DOT and FAA  reached an agreement by which DOT reviews FAA’s Exhibit 300s as part of  the department’s annual budget process, in which all departmental  components participate. Under this agreement, DOT conducts a review of  all FAA Exhibit 300s starting in June of each budget year and culminating in  the review of all Exhibit 300s by the Department Investment Review Board  in late August, prior to the submission of the budget to OMB in September.  As part of this agreement, DOT has outlined a process and schedule for  reviewing the fiscal year 2006 budget justifications for major FAA programs  and is monitoring FAA’s progress in meeting this schedule. In addition, the  department has identified about a dozen programs that it plans to monitor  regularly and has begun reviewing these programs through its senior  investment management decision-making board, on which the FAA  Administrator is a voting member. DOT has also requested that FAA set  reasonable expectations for cost, schedule, and performance for its major  projects and that it then report quarterly on variances to those  expectations. FAA submitted its first quarterly report as of June 2004.
These regular reports are intended to help DOT maintain oversight of FAA’s  processes and ensure that they are appropriate and consistent with OMB’s  requirements. Furthermore, the department is currently planning to issue  an investment management guide that specifies minimum expectations that  its operating administrations (including FAA) are to follow in managing  their investments. According to DOT officials, FAA has been complying  with the department’s requests for information to facilitate its oversight  process.
Department officials are attributing their increased oversight—and  cooperation from FAA—to the fact that the department has recently  reinstituted its own investment management processes. In addition, DOT  officials said that FAA now understands the role the department can play in  helping it to obtain the funding it needs for its programs.
section_title: Conclusions
paragraphs: FAA has established most of the project selection and control capabilities  needed to manage its NAS investments. This should help provide the  executive-level decision-making and oversight capabilities required to  establish accountability and guide major IT investments through most of  their life cycles. However, weaknesses remain. For example, although  business units are involved in the regular review of investments throughout  their life cycles, the JRC may not review the performance of operations  systems for several years unless they require significant additional funds.  Also, FAA has yet to define and implement the practices it needs to select  and control its non-NAS investments. Ultimately, because the JRC does not  regularly review NAS systems during the in-service management phase and  does not regularly review the non-NAS systems in general, significant  portions of FAA’s approximately $2.5 billion investment in IT go without  top-level executive oversight and are not viewed as part of an  enterprisewide portfolio. FAA has taken some initial steps to implement  PIRs, but it has not yet established a process to carry them out.
The agency has begun to take some steps to develop improvements to  address some of these weaknesses, such as establishing an Information  Technology Executive Board with relevant responsibilities. In addition, the  JRC has begun integrating some budgeting and oversight processes, and  the Chief Operating Officer has begun to articulate a vision that includes  additional accountability for investments in operations. But FAA has not  developed a comprehensive plan to guide all improvement efforts. Such a  plan would help coordinate and prioritize improvement efforts and help  sustain commitment to the efforts under way. The increasing collaboration  between FAA and DOT further contributes to the likelihood that the  management of FAA’s investments will improve as FAA’s Exhibit 300s have  the benefit of department-level review and the departmental investment  review board conducts periodic reviews of selected projects.
section_title: Recommendations for Executive Action
paragraphs: To strengthen FAA’s investment management capability and address the  weaknesses discussed in this report, we recommend that the Secretary of  the Department of Transportation direct the FAA Administrator to develop  and implement a plan for improving FAA’s IT investment management  processes. The plan should address the weaknesses described in this  report, beginning with those we identified in our Stage 2 analysis and  continuing with those we identified in our Stage 3. The plan should also  draw together ongoing efforts as well as instituting new initiatives where  called for. The plan should, at a minimum, provide for accomplishing the  following:  Define procedures for aligning the JRC and the newly established ITEB.
Establish a process for the JRC to periodically reevaluate the alignment  of projects in the in-service management phase with strategic goals and  objectives.
Establish a process for the JRC to regularly review the performance of  IT systems throughout their life cycles and take corrective actions when  expected performance is not being met.
Define and implement an IT investment management structure,  including an investment management board and a disciplined process  for managing all non-NAS investments.
Define and implement processes for managing major investments as  part of an enterprise-level portfolio, including NAS F&E investments,  NAS investments in the in-service management phase, and non-NAS  investments.
Define and implement processes for carrying out PIRs on investments  as they enter the in-service management stage.
In developing the plan, the FAA Administrator should ensure that it  (1) specifies measurable goals, objectives, and milestones; (2) specifies  needed resources; (3) assigns clear responsibility and accountability for  accomplishing tasks; and (4) is approved by senior management. In  implementing the plan, the FAA Administrator should ensure that the  needed resources are provided to carry out the plan and that progress is  measured and reported periodically to the Secretary of Transportation.
section_title: Agency Comments
paragraphs: In commenting on a draft of this report, DOT’s Director of Audit Relations  stated via e-mail that DOT appreciated the opportunity to review and offer  comment on our report and that GAO had done a good job keeping the  report balanced and fair, showing where FAA has many capabilities in  place and identifying areas that need improvement. The Director also  provided a technical comment, which we have incorporated into the report.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from the  date of this letter. At that time, we will send copies to other interested  congressional committees, the Director of the Office of Management and  Budget, the Secretary of Transportation, FAA’s Administrator and CIO, and  other interested parties. We also will make copies available to others upon  request. In addition, the report will be available at no charge on the GAO  Web site at www.gao.gov.
Should you or your offices have questions on matters discussed in this  report, please contact me at (202) 512-9286 or Lester P. Diamond, Assistant  Director, at (202) 512-7957. We can also be reached by e-mail at  pownerd@gao.gov, or diamondl@gao.gov, respectively. Key contributors to  this report were William G. Barrick, Niti Bery, Joanne Fiorino, Michael  Giannone, Sabine R. Paul, and Nik Rapelje.
section_title: Objectives, Scope, and Methodology
paragraphs: The objectives of our review were to (1) evaluate FAA’s capabilities for  managing its IT investments, (2) determine what plans the agency might  have for improving these capabilities, and (3) describe how DOT oversees  FAA’s investments and investment process. Because FAA told us that it  managed its NAS and non-NAS investments differently, we performed  separate assessments for the practices to evaluate FAA’s capabilities for  managing IT investments.
To address the first objective, for the NAS investments we reviewed the  results of the agency’s self-assessment of Stages 2, 3, and 4 practices using  GAO’s ITIM framework and validated and updated the results of the self- assessment through document reviews and interviews with officials. We  reviewed written policies, procedures, and guidance and other  documentation providing evidence of executed practices, including FAA’s  Acquisition Management System guidance, mission analysis and  investment analysis guidance, and memorandums. We also reviewed JRC  guidance and records of decision, acquisition review guidance and meeting  minutes, and variance reporting procedures and reports. We did not assess  FAA’s progress in establishing the capabilities found in one of the two Stage  4 critical processes, entitled Improving the Portfolio’s Performance, or in  any of the Stage 5 critical processes, because FAA acknowledged that it  had not executed any of the key practices in these critical processes. For  the non-NAS investments, we reviewed the results of FAA’s self- assessments of Stage 2 practices using GAO’s ITIM framework and  conducted interviews to clarify and update the results. We did not perform  a detailed assessment of these practices because they most likely will be  superseded by a new process (when it is defined) for managing non-NAS  investments, and non-NAS investments are of lower cost and impact to  FAA.
As part of our analysis, we selected three IT projects as case studies to  verify that the critical processes and key practices were being applied. We  selected projects that (1) supported different FAA functional areas,  (2) were in different life cycle phases, and (3) required different levels of  funding. The three projects are described below:  FAA Telecommunications Infrastructure (FTI)—FTI is a performance- based telecommunications services contract for voice, video, and data  point-to-point support for telecommunications for the National Airspace  System and its support system. It contributes to both the separation of  aircraft (the mission-support network) and other FAA uses (the  operational network, e.g., e-mail and phone). FTI will replace the  current telecom system. FTI will eliminate the need for other  subnetworks, of which there are currently eight or nine, and therefore  eliminate the management overhead associated with operating so many  networks. The integration of multiple networks and subnetworks will  provide a single source and single vehicle for telecom. FTI is in the  Technical Operations unit and has estimated life cycle costs of  $2 billion. The contract for FTI was awarded in June 2002.
En Route Communications Gateway (ECG)—ECG is a mission critical  gateway, or interface, for data from radar sites to Air Route Traffic  Control Centers. ECG will serve as a single domain communications  gateway and will provide the path for exchanging flight plan data from  outside sources and transfer data among systems. ECG provides a  commercial-off-the-shelf nondevelopmental item digital gateway using a  modern, open and extensible platform consisting of modular scalable  hardware components. ECG will incorporate interface capability to  support legacy and future systems and will provide the capability to  transition to modern network communications and access more  surveillance sources. The flexibility provided by the ECG system  architecture will facilitate the evolution of the En Route domain  modernization. ECG will replace the Peripheral Adapter Module  Replacement Item system and provide a modern domain gateway that  will support the current and future En Route infrastructure. ECG is in  the En Route & Oceanic Service group and has estimated life cycle costs  of $442.5 million through September 2015.
Voice Switching and Control System (VSCS)—In our review of the VSCS  program, we focused our review on one of VSCS’s subcomponents, the  VSCS Control Subsystem Upgrade (VCSU). The VCSU program, part of  the Technical Operations Communications service group, is designed to  maintain overall supportability of VSCS by replacing the hardware for  the existing control subsystem, associated VSCS operational and  application software, required software licenses, and supporting  software and hardware documentation. Deliverables for the VCSU  program include all hardware, spare parts, software, software licenses,  system baseline documentation, training, and other technical  documentation necessary to support the product at 21 locations.  According to FAA, the VCSU program has a funding baseline of over $59  million and is in the operations and maintenance phase.
For these projects, we reviewed project management documentation, such  as mission needs statements, acquisition program baselines, and integrated  program plans. We also interviewed the project managers for these  projects.
We compared the evidence collected from our document reviews and  interviews to the key practices in ITIM. We rated the key practices as  “executed” on the basis of whether the agency demonstrated (by providing  evidence of performance) that it had met the criteria of the key practice. A  key practice was rated as “not executed” when we found insufficient  evidence of a practice during the review or when we determined that there  were significant weaknesses in FAA’s execution of the key practice.
To address our second objective, we obtained and evaluated documents  showing what management actions had been taken and what initiatives had  been planned by the agency. This documentation included JRC records of  decisions, the agency’s capital investment guidance, and the recently  formed ITEB charter and meeting minutes. We also interviewed the Chief  Information Officer, other members of the JRC, and the Chief Operating  Officer to determine what efforts FAA had undertaken to improve IT  investment management processes.
To address our third objective, we reviewed documentation on DOT’s  process for reviewing FAA’s budget proposals and capital planning and  investment control reviews. We also conducted interviews with both FAA  and DOT officials, including DOT’s CIO and Director for Capital Planning  and Investment Control to determine DOT’s oversight role in FAA’s  investments and investment management processes.
We conducted our work at FAA Headquarters in Washington, D.C., from  October 2003 through July 2004, in accordance with generally accepted  government auditing standards.
section_title: Investment Management Process Used by Some Organizational Units to Manage Non-NAS Investments
paragraphs: ABA has an investment board that conducts periodic and monthly program reviews for all IT  programs to determine whether a program will be approved as an IT investment. A life cycle  process guide is now in place to direct the activities of the investment board along with  providing oversight of IT projects within ABA.
The business needs of a project within ABA, along with the dates for achieving them, need  to be aligned with the strategic goals established in the FAA Flight Plan. Projects or systems  that are no longer aligned with the Flight Plan will be decommissioned.
A project management plan identifies, among other things, the system’s users, customers,  and types of services to be provided.
Selecting and reselecting an IT investment within ABA involves both the executive  management team and ABA’s CIO team. The executive management team reviews the  business needs of the investment and compares them against the ABA’s IT budget, while  ABA’s CIO team is involved with the selecting and reselecting processes by analyzing the  technical costs associated with the IT investment and comparing those technical costs  against the ABA’s IT budget.
ABA uses its life cycle process guide to help manage its $25 million IT budget, which  consists of 22 or 23 financial systems, 5 or 6 of them considered major programs under  OMB’s definition of a major IT investment. A requirement of the life cycle process guide is  for every critical system in ABA to have a detailed project management plan that addresses  performance measures such as cost, schedule, benefits, and risks.
The day-to-day progress of IT projects is tracked against critical milestones that have been  already established through weekly summary reviews with IT staff. For major IT projects,  biweekly meetings are conducted that address any concerns with meeting the performance  measures.
ABA captures its IT asset information using its Information Technology Investment Portfolio  System (ITIPS), which is available to all ABA management and system support personnel.  The information in ITIPS is used to manage projects that are in production as well as  ensuring that the life cycle activities are in alignment with FAA’s mission statements.
ARA uses its Operations Resource Management Team guide to select, control, and  evaluate ARA IT investments. The team composed of representatives from ARA service  units. ARA investments are controlled and tracked through quarterly reviews. These reviews  look at the cost, schedule, and overall performance of the investment.
The business needs for ARA investments need to be mapped back to the Flight Plan. A  monthly status review report is prepared in order to ensure that the business needs are  tracking back to the Flight Plan.
ARA does not have any well-defined selection criteria since each program uses its own  configuration management plan. ARA Ops build process guides the establishment of new  projects.
A project plan does exist, along with established expenditures, which the program  managers submit to the ARA CIO on a monthly basis. These monthly status reports occur  between the CIO and the program managers to decide if an investment’s resources, such  as funding, need to be reallocated. Once the CIO and program manager decide that it is  necessary for an investment’s resources to be reallocated, the CIO will discuss the need  further with the Deputy Associate Administrator for ARA, who ultimately will determine  whether a program will receive additional resources, such as funding.
With respect to the level of interaction that ARA has had with the JRC in the past, only one  program from ARA, NextGen, has gone before the JRC. According to the ARA CIO, in order  for a program to go to the JRC, there must be justification made to the council that the  program is fully operational and is considered to be a benefit and a priority to FAA. The ARA  Deputy Associate Administrator will determine if a program should go before the JRC for  approval and funding.
The configuration control board uses a database to capture asset inventory data about the  systems that are owned by the ARA CIO. According to the ARA CIO, in order for IT assets to  be effectively managed in ARA, there needs to be vision from AIO about what programs to  invest in over the next 5 years.
The Information Resource Management Executive Board is responsible for selecting,  controlling, and evaluating ATS IT investments.
Not all services within ATS have defined their business needs. Even though ATS has the  NAS Support Integration Process (NSIP) data repository available for capturing IT asset  information, including business needs, and for defining system users, there is no  consistency in terms of the records being complete because there are systems within ATS  that have not registered with NSIP.
The ATS CIO manages the selection process, which begins with the NSIP registration  criteria.
Each business unit within ATS has its own project management plan and procedures. The  day-to-day tracking of projects as well as the monitoring of whether corrective actions are  being executed is also the responsibility of the individual business units. Even though the  individual business units are tasked with this level of responsibility, the ATS CIO does play  an oversight role by setting the criteria and policies for the investments to be made for the  projects.
ATS uses the NSIP meta data repository to collect any changes to the IT projects and  systems by providing a full declaration of the project or system. This includes providing  information to help ATS avoid unwanted costs due to systems having redundant  functionality and determining whether a system’s or a project’s functions match the stated  mission goals for ATS. NSIP also handles the technical rollover for ATS systems or projects.
AIO’s investment management process can be characterized as iterative and well  managed, but undocumented. The AIO Business Plan and IT Strategy are used to ensure  that when funds are appropriated and allocated that they map back to the Flight Plan.  Investments are controlled or tracked by the Deputy CIO on a monthly basis to get an  indication as to where the program is in the process against the expenditures that have  been already established. Weekly meetings are held with the unit’s CIO to discuss any  issues regarding AIO’s investment management process.
AIO does not have any written policies or procedures for identifying business needs for its IT  projects. Only one of its major projects, NAS Adaptation Service and Environment, has  documented its requirements, which includes specific users.
AIO uses an undocumented process for reviewing new IT proposals to reach an agreement  on selection.
There are no AIO-wide policies or procedures for managing projects or investment  oversight. The Information Technology Executive Board (ITEB) has been formed to provide  a governing structure for non-NAS programs. One of the targets for ITEB is to look at cost  control and cross-cutting IT initiatives by involving the heads of the lines of business. The  ITEB is also going to be involved with improving the scores on the Exhibit 300 business  cases for OMB.
AIO uses ITIPS to track its asset inventory and IT investments. The Deputy CIO of AIO is  responsible for ensuring that the inventory located in ITIPS meets the needs of AIO’s  investment management process. According to AIO, the information within ITIPS is updated  at least twice a year.
AHR does not have an investment board. Instead, AHR’s senior management is  responsible for selecting, controlling, and evaluating all IT investments by using established  agency acquisition policies and procedures to conduct investment management decisions.
Business needs and specific users for each project are identified within the project plan and  are aligned with the AHR Strategic Plan, the FAA Flight Plan, and the AIO Plan. AHR is also  aligning its business needs to the ITEB plans. Business needs are re-evaluated on a  quarterly basis to ensure that a project is aligned with FAA’s strategic goals and objectives.
AHR senior management uses its prioritization process to evaluate and select investments  for funding. The office and center directors determine their requirements and then a budget  request is submitted for proposal funding. AHR receives an allowance amount from the  budget office. The first priority is to handle personnel payments. The remaining balance is  then redistributed to the business divisions. The “building blocks” process starts at this  point. This is when base funding is reviewed to decide if a current investment needs  continued funding by asking questions about the importance of continuing the funding of a  particular project by looking at the project activity and what the impact will be if this project is  no longer funded. Each division will submit a list of prioritized projects with costs to the  directorate. This list may exceed the budget level. The directorate will reprioritize the original  list.
AHR has a Human Resource Management Automation Plan that contains procedures for  approving IT projects, and describes the policies and procedures that AHR uses for project  management. Despite having project management policies and procedures, not all projects  within AHR have a formal project plan. The size and scope of the project are two factors that  help determine whether a project has a formal project plan. AHR Division Managers ensure  that projects are on time by performing quarterly reviews that assess a project’s cost and  schedule. AHR uses a color scheme (red, green, and yellow) to indicate the schedule status  of major milestones.
AHR uses the ITIPS as its inventory for making investment management decisions. AHR  projects are listed in ITIPS, along with business cases.
The IT Configuration Management Board is ARC’s investment review board. The board’s  charter has recently been redone to provide more traceability back to the Flight Plan. The  board functions include evaluating potential IT investment options for ARC, making  recommendations on IT investment, establishing ARC-wide IT standards, and developing  and maintaining investment policies and procedures. The board is led by the unit’s CIO and  includes four IT managers from the regional offices and aeronautical center and two  members from the ARC Management Team. The ARC Management Team makes the final-  selection decisions. The IT investment management decisions are then incorporated into  the ARC Business Plan. The ARC unit is also involved with cross-organizational investment  decisions for FAA through its membership on the FAA CIO Council.
Business needs are identified through entries made in ITIPS, along with documentation  from Exhibit 300s and Exhibit 53s.
ARC does not have its selection criteria documented. To evaluate and select IT investments,  the ARC IT Configuration Management Board considers such things as benefits to ARC  across the regions, expected return on investment, technical feasibility, and risk. The ARC  business plan and the Flight Plan are the documents that address these priorities.
ARC does not have policies or procedures for project management. Instead, ARC uses a  weekly teleconference to address expectations and progress of ARC-wide IT initiatives at  the IT manager level across ARC. According to the ARC CIO, a second teleconference has  been added to discuss portfolio management—schedule, budget, training, and deployment  along with whether the project will be integrated with other lines of business.
ARC uses ITIPS as its standardized repository for collecting asset information that will be  useful for ARC’s IT investment management decisions by providing information about what  types of systems and functions are available and how they are supporting a specific  business issue.
Similar to an IT investment board, AVR has a two-tiered management structure that is  composed of the AVR management team and the CIO management team. The AVR  management team includes the Associate Administrator and the Service Directors who  make the final decisions based upon recommendations and input from the CIO  management team and its business partners from each of the service units. According to  AVR, its IT investment process guide is still under development and will be completed at the  end of Fiscal Year 2004.
Each line of business within AVR identifies and documents its business needs including  project requirements and specific users. Once the business needs have been identified, the  IT Management and Resources section prioritizes them for funding.
Programs in AVR are reviewed quarterly. For major projects, meetings are designed to look  at project milestones to see if they are being met. These meetings are carried out biweekly  and presented to the AVR management team.
The AVR CIO management team is responsible for monitoring projects and reporting to the  AVR Management team. Biweekly meetings are held for major projects within AVR.
AVR’s system inventory is a part of its enterprise architecture. The system inventory is  being used primarily in developing the Exhibit 300s. The performance of IT projects in AVR  is monitored daily, based upon each project’s individual plan, using project management  tools such as MS Project. According to AVR, not all projects have a project plan in place, but  AVR is trying to make it a requirement.
subsections: section_title: GAO’s Mission
paragraphs: The Government Accountability Office, the audit, evaluation and  investigative arm of Congress, exists to support Congress in meeting its  constitutional responsibilities and to help improve the performance and  accountability of the federal government for the American people. GAO  examines the use of public funds; evaluates federal programs and policies;  and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost  is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts  newly released reports, testimony, and correspondence on its Web site. To  have GAO e-mail you a list of newly posted products every afternoon, go to  www.gao.gov and select “Subscribe to Updates.”
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Congressional Relations
section_title: Public Affairs