Information Technology: FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed
Released: 2004-09-20T00:00:00

Letter

Background

FAA’s Mission and Organizational Structure

As an agency of the Department of Transportation, FAA’s mission is to promote the safe, orderly, and expeditious flow of air traffic in the national airspace. To fulfill its mission requires the extensive use of technology. The achievement of the agency’s mission is also dependent in large part on the skills and expertise of its workforce. Its workforce of nearly 50,000 people provides aviation services that include air traffic control; maintenance of air traffic control equipment; and certification of aircraft, airline operations, and pilots.

FAA is organized into several staff support offices (examples include the Office of Information Services and the Office of Human Resource Management) and five lines of business, which include Airports, Regulation and Certification, Commercial Space Transportation, the Office of Security and Hazardous Materials, and the newly formed Air Traffic Organization (ATO). The ATO was formed on February 8, 2004, to better provide safe, secure, and cost-effective air traffic services now and into the future. The Air Traffic Services and the Research and Acquisitions units, which had been primarily responsible for managing air traffic services within FAA, were combined into one performance-based organization to create ATO. ATO is led by FAA’s Chief Operating Officer and consists of 10 service units.

FAA’s Use of IT

FAA relies extensively on information technology to carry out its NAS operations. It constantly depends on the adequacy and reliability of the nation’s ATC system, which comprises a vast network of radars; automated data processing, navigation, and communications equipment; and ATC facilities. Through this system, FAA provides services such as controlling takeoffs and landings and managing the flow of traffic between airports. For example, the Integrated Terminal Weather System is employed to allow maximum use of airport runways in all kinds of weather through a variety of weather sensors. The Wide Area Augmentation System is used to provide vertically guided landing to aircraft at thousands of airports and airstrips where there is currently no vertically guided landing capability.

FAA also relies on IT to carry out its mission-support and administrative operations (non-NAS operations). For example, FAA uses IT to support accident and incident investigations, security inspections, and personnel and payroll functions.

With an IT budget of about $2.5 billion for fiscal year 2004, FAA accounts for over 90 percent of the Department of Transportation’s IT budget. The amount of investments in both NAS and non-NAS IT is shown in the table 1 below.

Prior Reviews Identified Weaknesses in the Agency’s IT Investment Management Process

In 1995, we designated FAA’s modernization of its air traffic control system, the principle technology component of the NAS, as a high-risk area because of the size and complexity of the program and FAA’s many failures in meeting projects’ cost, schedule, and performance goals. In our latest High- Risk Series, issued in January 2003, we addressed the critical need for FAA to continue to improve its investment management practices—the management processes the agency uses to select, control and evaluate the benefits realized from its IT spending—because the agency would be spending nearly $16 billion more through FY 2007, after having already spent $35 billion since 1981. Other reports have also noted weaknesses in FAA’s IT investment management processes and have made a number of recommendations to address this area. For instance, last year we reported that while FAA had improved its processes, several issues remained unresolved. We noted, for example, that the agency had not yet implemented processes for evaluating projects after implementing them, in order to identify lessons learned and improve the investment management process.

FAA’s Current Approach to Investment Management

FAA’s process for managing an IT investment varies depending on the type of investment—NAS systems in development through the second year of operation (F&E), NAS systems in operation after the second year (OPS), and non-NAS systems each follow different processes. NAS investments are managed through a standardized process, the FAA Acquisition Management System (AMS), and non-NAS investments are managed through a number of different processes.

Process for Managing NAS Investments

In April 1996, FAA implemented its AMS in response to legislation that directed the agency to develop a new acquisition management system. Because of FAA’s contention that some of its modernization problems were caused by federal acquisition regulations, the Congress enacted legislation in November 1995 that exempted the agency from most federal procurement laws and regulations and directed FAA to develop and implement a new acquisition management system that would address the unique needs of the agency. AMS was intended to reduce the time and cost for fielding new products and services by introducing (1) a new investment management system that spans the entire life cycle of an acquisition, (2) a new procurement system that provides flexibility in selecting and managing contractors, and (3) organizational and human capital reforms that support the new investment and procurement systems.

AMS provides high-level acquisition policy and guidance for selecting and controlling FAA’s NAS investments through all phases of the acquisition life cycle, which is organized into a series of phases and decision points that include (1) mission analysis, (2) investment analysis, (3) solution implementation, and (4) in-service management. To select investments, FAA has established two processes—mission analysis and investment analysis—which together constitute a set of policies, procedures, and guidance that enhance the agency’s ability to screen projects that are submitted for funding. Also, through these two processes FAA is to assess and rank each project based on its relative costs, benefits, risks, and contribution to FAA’s mission, and a senior, corporate-level decision- making group selects projects for funding. After a project has been selected, FAA officials are required to formally establish the life cycle cost, schedule, benefits, and performance baselines that are used to monitor the project’s status throughout the remaining phases of the acquisition management life cycle. See figure 1 for a graphic depiction of FAA’s life cycle management process.

Several groups are involved in managing FAA’s NAS investments; they perform functions from analysis of mission needs and alternative investments through system development, implementation, operation, and, ultimately, disposal. The roles and responsibilities of each group are described below: Joint Resources Council (JRC)—This board makes corporate-level resource and investment decisions and establishes investment programs. Members include Associate Administrators representing FAA’s lines of business, the FAA Acquisition Executive, the Chief Financial Officer, the Chief Information Officer (CIO), and the Assistant Administrators for System Safety, for Policy, Planning and International Aviation, and for Region and Center Operations. The board is supported by the JRC Secretariat Team, a group that facilitates the board’s processes by maintaining the meeting calendar and guidance documents, developing records of decisions, and providing advisory and liaison support to programs.

Systems Engineering/Operational Analysis Team—This team performs affordability assessments for newly proposed investments and prepares recommendations for the reprogramming of funds from lower priority programs. It also prepares annual budget submissions for approval by the JRC. This team is composed of representatives from each line of business and from other functional disciplines and is chaired by the Director, System Architecture and Investment Analysis.

Investment Analysis Team (IAT)—This team is assembled for a relatively short period for each specific investment being considered, to conduct the detailed analysis of alternatives that will lead to selecting and recommending a preferred acquisition solution. The team draws experts from the integrated product teams, the organizational unit with the need, the investment analysis staff, and other organizations.

Corporate Mission Analysis Organization—Performs agency-level mission analysis and coordinates service area analysis, an activity that is conducted during mission analysis to (1) identify capability shortfalls for or in conjunction with service organizations, (2) ensure alignment with agency strategic goals, and (3) eliminate redundant activity, duplicate benefits, service gaps, and service overlaps. It also develops and maintains standards and tools for conducting service area analysis, and it assists service organizations in establishing a service area analysis capability.

In addition to identifying the roles and responsibilities of the groups involved in the management process, AMS provides guidance on the documents and decisions that result from each of the life cycle phases. For example, through the mission analysis phase, FAA identifies critical needs that the agency must meet for improving the safety, capacity, efficiency, and effectiveness of the NAS. Approval of a mission need statement by the JRC signifies that the agency agrees that the need is critical enough to proceed to the next phase—investment analysis. During the investment analysis phase, the IAT is to analyze and recommend a solution that best satisfies FAA’s performance goals and customer service needs. This team is then to rank each proposed project based on a number of factors, including how well it meets mission needs compared to other projects and whether it has a favorable cost-benefit ratio. As part of the JRC selection process, the life cycle cost, schedule, benefits, and performance baselines are established in a formal document called the acquisition program baseline (APB), which is designed to be used by program offices to monitor a project’s status in achieving those baselines throughout the remaining phases of the acquisition management life cycle.

The solution implementation phase begins when the JRC approves and funds a project, establishes its acquisition program baseline, and authorizes the service organizations to implement and manage the project over its life cycle. After the project has been implemented and is in operation (FAA’s in- service management phase), the service organizations monitor and assess operational performance. Also during this phase, the project is monitored to determine whether the current capability satisfies the demand for services or whether another solution offers the potential for improving safety or effectiveness or for significantly lowering costs. If the current capability is lacking, FAA initiates a process whereby the mission need would be revalidated and the investment analysis process begun again, possibly leading to a new investment decision. Figure 2 provides detail on the phases of FAA’s IT investment management process and decision points. The highlighted decision points represent those for which the JRC must make an approval decision before a project can move forward.

Senior executives have stated that with the reorganization of the ATO in February 2004, discussions have been held about realigning the investment management process to make the heads of the service units responsible and accountable for managing programs’ capital investments and operating costs from inception to retirement. In the past, the business units have been organized to manage either capital investments or operating costs, but not both. These discussions have not yet led to specific changes in FAA’s investment management processes and responsibilities.

While the AMS was intended to apply to all FAA investment programs, it has not been implemented for non-NAS investments. Each of the agency’s business line and staff offices that manage non-NAS investments has implemented its own processes for managing these investments. Examples of these various non-NAS investment processes include the following: Regarding an investment management board structure, the Financial Services staff office has an informal board consisting of the Chief Financial Officer, Deputy Chief Financial Officer, and heads of offices within Financial Services. The Financial Services life cycle process guide directs the board’s operations. In the Regulation and Certification unit, the senior management team makes investment management decisions with input from the Chief Information management team. This unit is developing an IT investment management processes guide, which is expected to be completed by the end of the fiscal year.

When selecting investments, the Human Resource Management unit uses its established annual budget formulation process, while the Region and Center Operations unit is moving toward a new process whereby in order to be selected investments need to demonstrate, at a minimum, that they (1) are compliant with FAA’s architecture, (2) have a business sponsor, (3) have a solid business case, and (4) can be funded.

In controlling investments, Information Services has developed processes to monitor contract expenditures, and unit managers regularly perform financial management reviews of the programs under their purview, but there is no structured process for oversight of projects’ performance against expectations. In the Human Resource Management unit, division managers hold quarterly reviews to assess projects’ progress in meeting cost and schedule expectations and aligning with strategic goals.

Descriptions of the processes used by each of the units responsible for managing non-NAS investments can be found in appendix II.

In January 2004, the FAA Administrator established the Information Technology Executive Board (ITEB) to “strengthen FAA’s ability to use IT as an agencywide strategic asset” and “guide fundamental changes in the governance of IT assets.” Its charter calls for the ITEB to assume responsibility for making investment decisions about non-NAS IT investments. However, the ITEB has not yet implemented this aspect of its charter. Therefore, at the current time there is no single board or investment management process for non-NAS investments that would be analogous to the JRC board and AMS process that are used for NAS investments.

ITIM Maturity Framework

The ITIM framework is a maturity model composed of five progressive stages of maturity that an agency can achieve in its investment management capabilities. It was developed on the basis of our research into the IT investment management practices of leading private- and public- sector organizations. The framework identifies critical processes for making successful IT investments, organized into the five increasingly mature stages. These maturity stages are cumulative; that is, in order to attain a higher stage of maturity, the agency must have institutionalized all of the requirements for all of the lower stages, in addition to those for the higher stage.

The ITIM can be used both to assess the maturity of an agency’s investment management processes and as a tool for organizational improvement. The overriding purpose of the framework is to encourage investment processes that increase business value and mission performance, reduce risk, and increase accountability and transparency in the decision process. We have used the framework in several of our evaluations, and a number of agencies have adopted it. These agencies have used ITIM for purposes ranging from self-assessment to redesign of their IT investment management processes.

ITIM’s five maturity stages represent steps toward achieving stable and mature processes for managing IT investments. Each stage builds on the lower stages; the successful attainment of each stage leads to improvement in the organization’s ability to manage its investments. With the exception of the first stage, each maturity stage is composed of “critical processes” that must be implemented and institutionalized in order for the organization to achieve that stage. These critical processes are further broken down into key practices that describe the types of activities that an organization should be performing to successfully implement each critical process. An organization may be performing key practices from more than one maturity stage at the same time. This is not unusual, but efforts to improve investment management capabilities should focus on becoming compliant with lower-stage practices before addressing higher-stage practices.

Stage 2 of the ITIM framework encompasses building a sound investment management process by establishing basic capabilities for selecting new IT projects. It also involves developing the capability to control projects so that they finish predictably within established cost and schedule expectations and the capability to identify potential exposures to risk and put in place strategies to mitigate that risk. The basic selection processes established in Stage 2 lays the foundation for more mature selection capabilities in Stage 3.

Stage 3 requires that an organization continually assess both proposed and ongoing projects as parts of a complete investment portfolio—an integrated and competing set of investment options. It focuses on establishing a consistent, well-defined perspective on the IT investment portfolio and maintaining mature, integrated selection (and reselection), control, and evaluation processes, which are to be evaluated during postimplementation reviews (PIR). This portfolio perspective allows decision makers to consider the interaction among investments and the contributions to organizational mission goals and strategies that could be made by alternative portfolio selections, rather than relying exclusively on the balance between the costs and benefits of individual investments.

Stages 4 and 5 require the use of evaluation techniques to continuously improve both the investment portfolio and investment processes in order to better achieve strategic outcomes. At Stage 4 maturity an organization has the capacity to conduct IT succession activities and therefore can plan and implement the deselection of obsolete, high-risk, or low-value IT investments. An organization with Stage 5 maturity conducts proactive monitoring for breakthrough information technologies that will enable it to change and improve its business performance. Organizations implementing Stages 2 and 3 have in place the selection, control, and evaluation processes that are required by the Clinger-Cohen Act. Stages 4 and 5 define key attributes that are associated with the most capable organizations.

Figure 3 shows the five maturity stages and the critical processes associated with each.

As defined by the model, each critical process consists of “key practices” that must be executed to implement the critical process.

FAA Has Established Many Key Practices for Managing NAS Investments but Lacks Oversight of Operational Systems

In order to have the capabilities to effectively manage IT investments, an agency should, at a minimum, (1) build an investment foundation by putting basic, project-level control and selection practices in place (Stage 2 capabilities) and (2) manage its projects as a portfolio of investments, treating them as an integrated package of competing investment options and pursuing those that best meet the strategic goals, objectives, and mission of the agency; and it should also conduct PIRs to maintain mature, integrated selection, control, and evaluation processes (Stage 3 capabilities). In addition, an agency would be well served by implementing capabilities for improving its investment process through performance evaluations of its portfolio and succession management of current investments (Stage 4 capabilities). In order to develop the capabilities to effectively manage its investments, FAA would, at minimum, need to implement Stage 2 capabilities for both its NAS and non-NAS investments and Stage 3 capabilities for its portfolio of investments.

FAA’s investment management capabilities vary depending on whether an investment is considered to be NAS or non-NAS. Specifically: For NAS investments, FAA has executed 30 of the 38 Stage 2 key practices that are required to establish a foundation for investment management maturity. For these investments, the agency has in place a strong set of processes to support investment management, although the JRC does not regularly review investments that have passed into the in-service management phase (i.e., operational systems).

For its non-NAS investments, the agency has not yet adequately implemented a single management line of responsibility and the standard processes needed to manage in a consistent manner. Although some structured processes exist within individual business units, this lack of consistency undermines the agency’s maturity.

In Stage 3, the lack of regular JRC oversight of operational systems and the absence of a structured approach to managing non-NAS investments prevent FAA from managing its investments as a portfolio that includes all major NAS and non-NAS investments. In addition, the agency is not conducting PIRs on its major investments.

FAA has not executed any of the Stage 4 key practices for managing the succession of its information systems, although the agency has begun to address this weakness by defining procedures for retiring investments in the AMS.

When FAA implements all of the key practices associated with building the investment foundation and managing its investments as a portfolio, the agency will have greater assurance that it has selected the mix of investments that best supports its strategic goals and that it will be able to manage the investments to successful completion.

FAA Has Established Much of the Foundation Needed to Manage Its NAS Investments

At the ITIM Stage 2 level of maturity, an organization has attained repeatable, successful IT project-level investment control processes and basic selection processes. Through these processes, the organization can identify expectation gaps early and take appropriate steps to address them. According to ITIM, critical processes at Stage 2 include (1) defining IT investment board operations, (2) identifying the business needs for each IT investment, (3) developing a basic process for selecting new IT proposals and reselecting ongoing investments, (4) developing project- level investment control processes, and (5) collecting information about existing investments. Table 2 describes the purpose of each of the Stage 2 critical processes.

To its credit, FAA has put in place about 80 percent of the key practices associated with managing its NAS investments through the Stage 2 critical processes. The agency has satisfied all of the key practices associated with capturing investment information and most of those associated with instituting the investment board, meeting business needs, selecting an investment, and providing investment oversight. Most of the weaknesses in these critical processes relate to NAS investments in the in-service management phase. Table 3 summarizes the status of FAA’s critical processes for Stage 2, showing how many key practices FAA has executed in managing its NAS investments.

FAA Has Established an IT Management Structure to Manage Its NAS Investments

The establishment of decision-making bodies or boards is a key component of the IT investment management process. At the Stage 2 level of maturity, organizations define one or more boards, provide resources to support their operations, and appoint members who have expertise in both operational and technical aspects of proposed investments. The boards operate according to a written IT investment process guide that is tailored to the organization’s unique characteristics, thus ensuring that consistent and effective management practices are implemented across the organization. Once board members are selected, the organization ensures that they are knowledgeable about policies and procedures for managing investments. Organizations at the Stage 2 level of maturity also take steps to ensure that executives and line managers support and carry out the decisions of the IT investment board. According to ITIM, an IT investment management process guide should be a key authoritative document that the organization uses to initiate and manage IT investment processes and should provide a comprehensive foundation for the policies and procedures that are developed for all of the other related processes. (The complete list of key practices is provided in table 4.)

FAA has executed 7 of the 8 key practices for this critical process. For example, in 1996, Congress directed FAA to develop a new acquisition management system as part of a broad mandate for acquisition reform at the agency. In response, FAA implemented AMS in April 1996. AMS establishes policy and guidance for all aspects of the agency’s acquisition life cycle and documents the investment management process used for NAS investments. The agency established the JRC as its corporate-level investment board for the NAS investments. The JRC makes select and control decisions, including corporate decisions on mission needs, acquisition investments, and acquisition program baseline changes; it also reviews and recommends approval of the agency’s F&E budget submission.

The board is adequately resourced to support its operations. The JRC Secretariat Team supports the board in such ways as developing and updating guidance, scheduling meetings, and preparing and executing the JRC readiness process. In addition, the Mission Analysis Steering Group is responsible for assisting the board in prioritizing mission needs, while the Systems Engineering/Operational Analysis Team is to assist in addressing budget issues among investments. The JRC consists of senior officials from both business and IT areas, including the Chief Information Officer and the associate administrators representing FAA lines of business. These members are to exhibit the core competencies required by FAA in selecting executives and in assessing executive training needs. In addition, the agency offers a 3-day AMS overview course for all employees, including JRC members. Although the board as an entity does not oversee the development and maintenance of AMS, it is involved through FAA’s Acquisition System Advisory Group, which evaluates all proposed changes to AMS. To ensure that the board’s decisions are carried out, an acquisition program baseline document is approved at the JRC final investment decision point; this document identifies the capabilities, benefits, costs, and schedule for the approved investment, which are monitored by FAA through its variance reporting process.

Despite these strengths, FAA has not yet clearly defined the relationship between the JRC and the newly formed ITEB. Although the ITEB was established by the Administrator to function as the central authority responsible for assuring that FAA IT investments are based on sound business practices, FAA has not yet clearly delineated the specific roles the ITEB is to play and the relationship it will have with the JRC. This task has been assigned to the ITEB as a longer-range initiative.

Table 4 shows the rating for each key practice required to implement the critical process for instituting the investment board at the Stage 2 level of maturity. Each of the “Executed” ratings shown below represents instances where, based on the evidence provided by FAA officials, we concluded that the specific key practices were executed by the organization.

FAA Has a Process for Ensuring That its Investments Support Business Needs and Meet Users’ Needs

Defining business needs for each IT project helps to ensure that projects and systems support the organization’s business needs and meet users’ needs. This critical process ensures that a link exists between the organization’s business objectives and its IT management strategy. According to ITIM, effectively meeting business needs requires, among other things, (1) documenting business needs with stated goals and objectives, (2) identifying specific users and other beneficiaries of IT projects and systems, (3) providing adequate resources to ensure that projects and systems support the organization’s business needs and meet users’ needs, and (4) periodically evaluating the alignment of IT projects and systems with the organization’s strategic goals and objectives. (The complete list of key practices is provided in table 5.)

FAA has in place 6 of the 7 key practices for meeting business needs. The agency’s AMS and mission analysis guidance calls for business needs for both proposed and ongoing IT projects and systems to be identified in the mission need statement developed during the mission analysis phase. FAA also has detailed procedures for developing this document that call for identifying business needs. Resources for ensuring that IT projects and systems support the organization’s business needs and meet users’ needs include service organizations, the Corporate Mission Analysis Organization, the Mission Analysis Steering Group, and detailed procedures and associated templates for developing mission need statements. FAA’s specific business mission, with stated goals and objectives, is defined in the Federal Aviation Administration Flight Plan for fiscal years 2004 through 2008.

Further, FAA defines and documents business needs for both proposed and ongoing IT projects and identifies users and other beneficiaries during its mission analysis activities. In addition, the AMS policy calls for users to participate in project management throughout the FAA life cycle management process. For the three projects we reviewed, we verified that business needs and specific users and other beneficiaries were identified and documented in mission needs statements as well as in other documents. In addition, users are involved in project management throughout the life cycle of the projects. For example, according to project officials, En Route Communications Gateway (ECG) users participate in project meetings, weekly integrated product team status meetings, and monthly En Route domain national deployment teleconferences. FAA Telecommunications Infrastructure’s (FTI) end users are heavily involved in the “operational test” period, which determines whether the equipment can be safely implemented in NAS. VSCS Control Subsystem Upgrade users are involved in the project’s life cycle via a Web site through which they review and comment on project documentation.

Despite these strengths, the JRC has no process for evaluating the organizational alignment of NAS systems through most of their in-service management phase (and non-NAS investments, which are described separately in this report). While the JRC does evaluate the alignment of projects and systems with organizational goals throughout the systems’ development and 2 years into their operations as part of the annual budget formulation process, it does not use any consistent process to review projects and systems after that point in their life cycles. For NAS systems in the in-service management phase, these activities are carried out within the business unit that owns the system, but the JRC does not regularly oversee these processes and may go for several years without reviewing a system’s alignment with organizational goals. In-service NAS systems only return to the JRC if they are judged to require additional funds for correction. Until FAA establishes a process for periodic evaluation of systems throughout the in-service management phase and takes corrective actions when misalignment occurs, the agency will not be able to ensure that these projects, totaling about $1.3 billion per year, are still continuing to maintain alignment with the FAA’s strategic plans and its business goals and objectives.

Table 5 shows the rating for each key practice required to implement the critical process for meeting business needs at the Stage 2 level of maturity and summarizes the evidence that supports these ratings.

FAA Has a Disciplined Process for Selecting New IT Proposals but Lacks a Similar Process for Reselecting Ongoing Investments

Selecting new IT proposals and reselecting ongoing investments requires a well-defined and disciplined process to provide the agency’s investment board, business units, and developers with a common understanding of the process and the cost, benefit, schedule, and risk criteria that will be used both to select new projects and to reselect ongoing projects for continued funding. According to ITIM, this critical process requires, among other things, (1) making funding decisions for new proposals according to an established process; (2) providing adequate resources for investment selection activities; (3) using a defined selection process to select new investments and reselect ongoing investments; (4) establishing criteria for analyzing, prioritizing, and selecting new IT investments and for reselecting ongoing investments; and (5) creating a process for ensuring that the criteria change as organizational objectives change. (The complete list of key practices is provided in table 6.)

FAA has executed 7 of the 10 key practices associated with selecting an investment. For example, the AMS establishes two processes—mission analysis and investment analysis—that together constitute a set of policies and procedures, as well as guidance that is designed to enhance the agency’s ability to select investments. In addition, FAA has policies and procedures for its annual F&E budget formulation process to reselect ongoing IT projects. Also, FAA’s AMS sets forth policies and procedures for reselecting ongoing IT investments by identifying their capability shortfalls and addressing them as new investments.

The AMS also integrates funding with the process of selecting an investment by requiring the Systems Engineering/Operational Analysis Team to perform affordability assessments for new proposed investment programs; it may recommend funding reallocations from lower priority programs when an alternative solution cannot be funded within FAA planning and budgeting baselines. This team also supports the JRC to ensure that the executives’ funding decisions are aligned with selection decisions during the investment analysis activities.

Resources for proposal selection activities include the program director, the Integrated Product Team, and the Investment Analysis Team, as well as detailed procedures and a template that have been defined for developing investment analysis reports. The investment analysis reports identify the evaluation criteria used, the alternatives analyzed, and the ranking of each alternative so that the JRC can select the best overall solution identified in the mission need statement. The criteria that were established during the initial investment analysis phase are used by the Investment Analysis Team to rank each proposed project on the basis of how well it meets the agency’s mission needs compared with other projects.

FAA uses the processes defined in the AMS for selecting new IT investments. In addition, it uses two processes to reselect ongoing IT investments. Specifically, the FAA uses its annual budget formulation process for projects in development or in the first 2 years of operations. It also uses the AMS process when a system’s capability shortfall is identified, and it treats the correction of the shortfall as a new investment. The managers of the three projects we reviewed confirmed that their projects were selected using the AMS process. One project’s officials stated that this included market, alternatives, investment, and affordability analyses. The program managers also stated that the annual F&E budget formulation process is used to reselect their projects. These project officials also noted that if a project is scheduled for a hardware replacement, a reselection is done. The AMS process is followed to explore new alternatives and make sure the replacement is in the best interest of the government.

Despite these strengths, FAA has not developed similarly strong processes for NAS investments more than 2 years into their operations—those NAS systems that are in the in-service management phase. For example, while FAA’s F&E budget formulation process establishes criteria for analyzing, prioritizing, and reselecting IT investments for systems in development or up until 2 years into operations, neither of the two processes used to reselect IT investments has established criteria for investments beyond 2 years into operations. In addition, while FAA uses its annual budget formulation process to reselect projects that are part of the F&E budget, the agency does not have an analogous reselection process as part of its operations budget formulation. Until FAA establishes consistent criteria for reselecting all of its IT investments, it will not be adequately assured that it is consistently and objectively continuing to fund ongoing projects that still meet the needs and priorities of the agency in a cost-effective and risk- insured manner.

Table 6 shows the rating for each key practice required to implement the critical process for selecting an investment at the Stage 2 level of maturity and summarizes the evidence that supports these ratings.

FAA Does Not Have a Process for Effectively Overseeing Investments in All Phases of Their Life Cycles

An organization should provide effective oversight for its IT projects throughout all phases of their life cycles. Its investment board should maintain adequate oversight and observe each project’s performance and progress toward predefined cost and schedule expectations as well as each project’s anticipated benefits and risk exposure. The investment board should also employ early warning systems that enable it to take corrective action at the first sign of cost, schedule, or performance slippages. This board has ultimate responsibility for the activities within this critical process. According to ITIM, effective project oversight requires, among other things, (1) having written policies and procedures for management oversight; (2) developing and maintaining an approved management plan for each IT project; (3) making up-to-date cost and schedule data for each project available to the oversight boards; (4) having regular reviews by each investment board of each project’s performance against stated expectations; and (5) ensuring that corrective actions for each underperforming project are documented, agreed to, implemented, and tracked until the desired outcome is achieved. (The complete list of key practices is provided in table 7.)

FAA has in place 4 of the 7 key practices associated with effective project oversight. The agency has developed written policies and procedures for management oversight of its investments. These include (1) AMS; (2) the integrated program plan, which is the detailed planning document for all aspects of a program’s implementation, including program control; and (3) the Integrated Baseline Establishment and Management Process document for reporting variances from the performance expectations approved by the JRC in the acquisition program baseline.

We verified that cost, schedule, benefit, and risk expectations were documented in the acquisition program baseline and that the integrated program plan contained details for project execution for En Route Communications Gateway and FAA Telecommunications Infrastructure. For the VSCS Control Subsystem Upgrade, performance expectations and details on project execution were both captured in the integrated program plan. In addition, the JRC Secretariat Team maintains a tracking system for action items that are assigned during a project’s acquisition reviews, including the action to be taken, the responsible FAA organization, and whether the underlying problem has been resolved.

FAA has not established processes that bring investments before the JRC for oversight on a regular basis. There is a process for reporting variances from the performance expectations that were approved by the JRC in the investment’s acquisition program baseline. However, although this process is carried out as part of the F&E budget formulation for IT investments in development or less than 2 years into operations, it is not being carried out for investments that are part of the operations budget. Investments that are meeting performance expectations may not return to the JRC for several years. FAA also conducts acquisition reviews as a means for program offices to report to agency executives on the status of investments compared to program baselines. However, since program offices may select which investments they wish to bring forward for review, many investments may never come forward. Until FAA develops (1) procedures for reporting on an investment throughout its entire acquisition life cycle and (2) mechanisms for ensuring that all investments are reviewed regularly, the agency is placing itself at risk that underperforming investments will not be reported to the JRC in order for it to take appropriate actions.

Table 7 shows the rating for each key practice that is required to implement the critical process for project oversight at the Stage 2 level of maturity and summarizes the evidence that supports these ratings.

FAA Has a Structured Process for Capturing Investment Information and Using It to Support Investment Management

To make good IT investment decisions, an organization must be able to acquire pertinent information about each investment and store that information in a retrievable format. During this critical process an organization identifies its IT assets and creates a comprehensive repository of investment information. This repository provides information to investment decision makers to help them evaluate the impacts and opportunities that would be created by proposed or continuing investments. It can provide insights and trends about major IT cost and management drivers. The repository can take many forms and does not have to be centrally located, but the collection method should identify each IT investment and its associated components. This critical process may be satisfied by the information contained in the organization’s current enterprise architecture, augmented by additional information—such as financial information and information on risk and benefits—that the investment board may require to ensure that informed decisions are being made. According to ITIM, effectively managing this repository requires, among other things, (1) developing written policies and procedures for identifying and collecting the information, (2) assigning responsibility for ensuring that the information being collected meets the needs of the investment management process, (3) identifying IT projects and systems and collecting relevant information to support decisions about them, and (4) making the information easily accessible to decision makers and others. (The complete list of key practices is provided in table 8.)

FAA’s AMS guidance identifies specific information that is needed in the investment management process, including information for its investment analysis phase. FAA maintains a number of repositories of relevant information, including its Simplified Program Information Reporting & Evaluation database, which reports variances in cost, schedule, performance, or benefits from an investment’s approved acquisition program baseline. The information that is collected is made available to the JRC in several documents, including program plans and the acquisition program baseline document. The JRC Secretariat Team ensures that the investment board has all the relevant information it needs for its decision- making process.

Table 8 shows the rating for each key practice required to implement the critical process for capturing investment information at the Stage 2 level of maturity and summarizes the evidence that supports these ratings.

FAA Does Not Have Structured Processes to Manage Its Non-NAS Investments

FAA does not have a single set of processes for making consistent basic selection and control decisions for its non-NAS investments (Stage 2 capabilities). As previously discussed in the background section of this report, several business units within FAA make decisions about non-NAS investments. We reviewed the investment management processes of seven of these units—Information Services, Region and Center Operations, Regulation and Certification, Financial Services, Research and Acquisition, Air Traffic Services, and Human Resource Management. Appendix II describes the investment management processes we found in these units. The extent to which these processes comply with the ITIM framework for Stage 2 varies considerably by business unit, and FAA currently does not specify non-NAS investment management processes in a coordinated manner. Since the ITIM framework calls for a consistent investment management process, we assessed FAA’s non-NAS investment management capability at an aggregate level. That is, we assessed FAA’s capability to manage its non-NAS investments, not the capability of each individual business unit. Even though individual business units may have some of these processes in place, FAA as a whole has not yet defined an investment management structure that allows the agency to consistently manage its non-NAS investments, a uniform process for ensuring that non-NAS investments are linked to business needs and meet users’ needs, a process for selecting new IT proposals and reselecting ongoing a single process for reviewing the progress of investments and taking corrective action when performance expectations are not being met, or a comprehensive inventory of project and system information to support investment decisions.

According to FAA officials, the agency has not defined a coherent investment management structure and a set of processes for non-NAS investments in the past because many of these investments have not had the agencywide impact of the NAS investments. However, because there is now recognition that a disciplined approach to managing non-NAS investments could help control FAA’s IT assets and costs in general, efforts are currently under way to address this weakness. As previously discussed, an IT Executive Board (ITEB) has been chartered with responsibility for, among other things, making decisions about non-NAS IT investments, but it has not yet taken action on developing a standard process. Until FAA fully establishes the consistent practices it needs to make basic project selection and control decisions, executives will be hampered in their ability to effectively manage non-NAS investments and ultimately to find the opportunities to achieve the cost savings they are seeking.

FAA Lacks Key Capabilities Needed to Manage All IT Investments as a Portfolio and Does Not Conduct Postimplementation Reviews

During Stage 3, the investment board enhances the IT investment management process by developing a complete investment portfolio and carrying out PIRs. An IT investment portfolio is an integrated, agencywide collection of investments that are assessed and managed collectively on the basis of common criteria. Managing investments within the context of such a portfolio is a conscious, continuous, and proactive approach to expending limited resources on an organization’s competing initiatives in light of the relative benefits expected from these investments. Taking an agencywide perspective enables an organization to consider its investments comprehensively, so that collectively the investments optimally address the organization’s missions, strategic goals, and objectives. Managing IT investments with a portfolio approach also allows an organization to determine priorities and make decisions about which projects to fund, and continue to fund, based on analyses of the relative organizational value and risks of all projects, including projects that are proposed, under development, and in operation. For an organization to reap the full benefits of the portfolio process, it should collect all of its investments into an enterprise-level portfolio that is overseen by its senior investment board. Although investments may initially be selected into subordinate portfolios—based on, for example, lines of business or life cycle stages—and managed by subordinate investment boards, they should ultimately be aggregated into this enterprise-level portfolio.

The purpose of a PIR is to evaluate an investment after its development has been completed (i.e., after its transition from the implementation phase to the in-service management phase) in order to validate actual investment results. This review is conducted to (1) examine differences between estimated and actual investment costs and benefits and their possible ramifications for unplanned funding needs in the future and (2) extract “lessons learned” about the investment selection and control processes that can be used as the basis for management improvements. Similarly, PIRs should be conducted for investment projects that were terminated before completion, to help to readily identify potential management and process improvements.

According to ITIM, critical processes performed by Stage 3 organizations include (1) defining the portfolio criteria, (2) creating the portfolio, (3) evaluating the portfolio, and (4) conducting PIRs. Table 9 shows the purpose of each critical process in Stage 3.

FAA has executed only 1 of the 27 key practices associated with Stage 3 critical processes: it has a process for distributing portfolio criteria to project management personnel and other stakeholders. The remaining 26 key practices were not executed—primarily because FAA does not involve the JRC in the regular oversight of non-NAS investments or in NAS investments during their in-service management phase, weaknesses that we noted in our assessment of Stage 2 requirements. Since Stage 3 requires an enterprisewide perspective, the lack of oversight of these classes of investments precludes the successful completion of most Stage 3 critical processes. In addition, Stage 3 requires an enterprisewide perspective that FAA has not adopted, which would enable the JRC to oversee all major IT investments, regardless of life cycle phase or business unit. Although it can be appropriate for FAA to manage its NAS, in-service NAS, and non-NAS investments as separate subordinate portfolios—depending on the successful execution of all Stage 2 key practices—its enterprise-level portfolio should contain all major IT investments regardless of life cycle stage or business line. In building this enterprise-level portfolio, the JRC can choose whether to include specific investments based on predetermined criteria, as described by the ITIM framework. Until FAA fully implements the critical processes associated with managing its investments as a complete portfolio, it will not have the data or enterprisewide perspective it needs to make informed decisions about all of its major IT investments.

In addition, FAA has not executed the six key practices for conducting PIRs. In June 2004, in response to a recommendation contained in our 1999 report that FAA initiate PIRs for projects or programs within 3 to 12 months of deployment or termination, the NAS Configuration Management and Evaluation Staff developed a proposed approach to PIRs, but this approach was not implemented. In November 2003, the life cycle management policy team proposed a change to the AMS that would require conducting these reviews, but there has been no action on the proposal. Although the JRC has recently reaffirmed its commitment to implement PIRs, there is no policy and no established process to carry them out. If PIRs are not conducted on a routine basis, then FAA will not be able to effectively evaluate the results of its IT investments; this will affect the agency’s ability to determine whether to continue, modify, or terminate an IT investment in order to meet its stated mission objectives.

Table 10 summarizes the status of FAA’s critical processes for Stage 3, showing how many associated key practices it has executed.

FAA Has Not Established a Process for Managing the Succession of Its Information Systems

Once an agency has attained Stage 3 maturity, it evaluates its IT investment processes and portfolios to identify opportunities for improvement (Stage 4 capabilities). This entails (1) improving the portfolio’s performance and (2) managing systems and technology succession. We did not assess FAA’s capability for improving the portfolio’s performance, because it did not claim to be executing any of the relevant key practices in its self-assessment.

According to ITIM, regarding system and technology succession management includes (1) defining policies and procedures for managing the IT succession process, (2) assigning responsibility for the IT succession process, (3) developing criteria for identifying IT investments that may meet succession status, and (4) periodically analyzing IT investments to determine whether they are ready for succession. This critical process enables an organization to recognize low-value or high-cost IT investments and augments the routine replacement of systems at the end of their useful lives. It also promotes the development of a forward-looking, solution- oriented view of IT investments that anticipates future resource requirements and allows the organization to plan appropriately. This process differs from the reselection activity in Stages 2 and 3 in that it focuses on anticipating and planning for the retirement of legacy systems and on meeting remaining requirements with other, perhaps new, systems. In addition, succession management takes place at the end of a system’s life cycle.

FAA has not executed any of the nine key practices required to implement this critical process. Although the agency has defined procedures in AMS for retiring investments, it still needs to describe how to regularly review systems that are in operations in order to identify candidates for retirement. According to FAA, decisions on succession are made by the service organizations. However, no individual or group has been assigned responsibility for managing the succession process from an enterprise perspective, which would allow the FAA to better anticipate and plan for future resource requirements. Without an institutionalized process for succession management, the FAA may not be able to identify those IT investments that are eligible for succession in enough time to minimize the effect of the transition on their successors. In addition, by establishing an effective succession management process, the agency can identify systems for retirement, freeing resources for other, superior, investments.

FAA Has Initiated Efforts to Improve Its Investment Management Process

We have previously reported that to effectively implement IT investment management processes, organizations need to be guided by a plan that (1) is based on an assessment of strengths and weaknesses; (2) specifies measurable goals, objectives, and milestones; (3) specifies needed resources; (4) assigns clear responsibility and accountability for accomplishing tasks; and (5) is approved by senior management.

FAA has begun to take steps to resolve some of the weaknesses identified in this report. For example, at a June 10, 2004, meeting, the JRC decided to incorporate budget justification documents (Exhibit 300s), which are currently prepared for the Office of Management and Budget (OMB) as part of the President’s Budget formulation process, into the AMS process for managing NAS investments. The Exhibit 300 will become the board’s decision-making document, and essential information from existing AMS- required documents—the investment management report, the acquisition strategy paper, the integrated program plan, and the requirements documents—will be incorporated into the Exhibit 300. The JRC also recently decided to implement PIRs in order to track metrics during program implementation. Finally, at that same meeting, the board decided to collectively determine, at the meeting where the F&E budget is approved, which F&E and OPS programs should be brought forward for review the following year. This decision serves to bring certain investments in the in-service management phase under the JRC’s direct purview, although it does not specify that consistent criteria be established, as the ITIM framework requires.

FAA has also begun to initiate steps to bring more clarity to the ITEB’s responsibilities, although the specifics have yet to be defined. In its charter, the ITEB is charged with making investment decisions about non-NAS IT investments. This action would begin to bring all of the non-NAS investments under a single authority. The charter suggests that the ITEB choose among three options: (1) to send major non-NAS investment decisions to the JRC, (2) to make the decision itself, given an acceptable review process similar to the JRC processes, or (3) have the CIO, Chief Financial Officer, and owning assistant/associate administrator make the decision jointly. This description of the ITEB’s roles and responsibilities further alludes to the senior board’s evolving responsibility toward major non-NAS IT investments, although it falls short of laying out specific criteria for selecting which investments should be sent forward to the JRC. The ITEB has been given responsibility for four short-term initiatives as well, including establishing an agencywide cost control program for non- NAS expenditures and ensuring that all OMB Exhibit 300s receive a passing grade for the 2006 budget year. The ITEB has been charged with the long- term initiative of clearly delineating the roles it plays and its relationship with the more senior board. The successful completion of this initiative is likely to satisfy the single key practice that FAA has not yet executed in the Instituting the Investment Board critical process of the ITIM.

The Chief Operating Officer’s recent reorganization of the ATO is intended to make the heads of the service units responsible for IT projects from their inception through the in-service management phase. This new organization is designed to support his expressed intentions to increase accountability for systems in operation in order to manage costs more effectively. According to the Chief Operating Officer, FAA recognizes that good processes are needed for both NAS and non-NAS to improve the way the agency manages its investments.

While FAA has initiated these improvement efforts, it has not linked them together in a plan with the characteristics listed above that would help coordinate and guide the efforts. Until FAA develops a plan that would allow for the systematic prioritization, sequencing, and evaluation of improvement efforts, the agency risks not being able to effectively establish mature investment management processes.

DOT is Taking Steps to Integrate Oversight of FAA’s IT Investments

DOT has recently initiated several efforts that can serve to provide better departmental oversight of FAA investments. This fiscal year DOT and FAA reached an agreement by which DOT reviews FAA’s Exhibit 300s as part of the department’s annual budget process, in which all departmental components participate. Under this agreement, DOT conducts a review of all FAA Exhibit 300s starting in June of each budget year and culminating in the review of all Exhibit 300s by the Department Investment Review Board in late August, prior to the submission of the budget to OMB in September. As part of this agreement, DOT has outlined a process and schedule for reviewing the fiscal year 2006 budget justifications for major FAA programs and is monitoring FAA’s progress in meeting this schedule. In addition, the department has identified about a dozen programs that it plans to monitor regularly and has begun reviewing these programs through its senior investment management decision-making board, on which the FAA Administrator is a voting member. DOT has also requested that FAA set reasonable expectations for cost, schedule, and performance for its major projects and that it then report quarterly on variances to those expectations. FAA submitted its first quarterly report as of June 2004.

These regular reports are intended to help DOT maintain oversight of FAA’s processes and ensure that they are appropriate and consistent with OMB’s requirements. Furthermore, the department is currently planning to issue an investment management guide that specifies minimum expectations that its operating administrations (including FAA) are to follow in managing their investments. According to DOT officials, FAA has been complying with the department’s requests for information to facilitate its oversight process.

Department officials are attributing their increased oversight—and cooperation from FAA—to the fact that the department has recently reinstituted its own investment management processes. In addition, DOT officials said that FAA now understands the role the department can play in helping it to obtain the funding it needs for its programs.

Conclusions

FAA has established most of the project selection and control capabilities needed to manage its NAS investments. This should help provide the executive-level decision-making and oversight capabilities required to establish accountability and guide major IT investments through most of their life cycles. However, weaknesses remain. For example, although business units are involved in the regular review of investments throughout their life cycles, the JRC may not review the performance of operations systems for several years unless they require significant additional funds. Also, FAA has yet to define and implement the practices it needs to select and control its non-NAS investments. Ultimately, because the JRC does not regularly review NAS systems during the in-service management phase and does not regularly review the non-NAS systems in general, significant portions of FAA’s approximately $2.5 billion investment in IT go without top-level executive oversight and are not viewed as part of an enterprisewide portfolio. FAA has taken some initial steps to implement PIRs, but it has not yet established a process to carry them out.

The agency has begun to take some steps to develop improvements to address some of these weaknesses, such as establishing an Information Technology Executive Board with relevant responsibilities. In addition, the JRC has begun integrating some budgeting and oversight processes, and the Chief Operating Officer has begun to articulate a vision that includes additional accountability for investments in operations. But FAA has not developed a comprehensive plan to guide all improvement efforts. Such a plan would help coordinate and prioritize improvement efforts and help sustain commitment to the efforts under way. The increasing collaboration between FAA and DOT further contributes to the likelihood that the management of FAA’s investments will improve as FAA’s Exhibit 300s have the benefit of department-level review and the departmental investment review board conducts periodic reviews of selected projects.

Recommendations for Executive Action

To strengthen FAA’s investment management capability and address the weaknesses discussed in this report, we recommend that the Secretary of the Department of Transportation direct the FAA Administrator to develop and implement a plan for improving FAA’s IT investment management processes. The plan should address the weaknesses described in this report, beginning with those we identified in our Stage 2 analysis and continuing with those we identified in our Stage 3. The plan should also draw together ongoing efforts as well as instituting new initiatives where called for. The plan should, at a minimum, provide for accomplishing the following: Define procedures for aligning the JRC and the newly established ITEB.

Establish a process for the JRC to periodically reevaluate the alignment of projects in the in-service management phase with strategic goals and objectives.

Establish a process for the JRC to regularly review the performance of IT systems throughout their life cycles and take corrective actions when expected performance is not being met.

Define and implement an IT investment management structure, including an investment management board and a disciplined process for managing all non-NAS investments.

Define and implement processes for managing major investments as part of an enterprise-level portfolio, including NAS F&E investments, NAS investments in the in-service management phase, and non-NAS investments.

Define and implement processes for carrying out PIRs on investments as they enter the in-service management stage.

In developing the plan, the FAA Administrator should ensure that it (1) specifies measurable goals, objectives, and milestones; (2) specifies needed resources; (3) assigns clear responsibility and accountability for accomplishing tasks; and (4) is approved by senior management. In implementing the plan, the FAA Administrator should ensure that the needed resources are provided to carry out the plan and that progress is measured and reported periodically to the Secretary of Transportation.

Agency Comments

In commenting on a draft of this report, DOT’s Director of Audit Relations stated via e-mail that DOT appreciated the opportunity to review and offer comment on our report and that GAO had done a good job keeping the report balanced and fair, showing where FAA has many capabilities in place and identifying areas that need improvement. The Director also provided a technical comment, which we have incorporated into the report.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies to other interested congressional committees, the Director of the Office of Management and Budget, the Secretary of Transportation, FAA’s Administrator and CIO, and other interested parties. We also will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at www.gao.gov.

Should you or your offices have questions on matters discussed in this report, please contact me at (202) 512-9286 or Lester P. Diamond, Assistant Director, at (202) 512-7957. We can also be reached by e-mail at pownerd@gao.gov, or diamondl@gao.gov, respectively. Key contributors to this report were William G. Barrick, Niti Bery, Joanne Fiorino, Michael Giannone, Sabine R. Paul, and Nik Rapelje.

Objectives, Scope, and Methodology

The objectives of our review were to (1) evaluate FAA’s capabilities for managing its IT investments, (2) determine what plans the agency might have for improving these capabilities, and (3) describe how DOT oversees FAA’s investments and investment process. Because FAA told us that it managed its NAS and non-NAS investments differently, we performed separate assessments for the practices to evaluate FAA’s capabilities for managing IT investments.

To address the first objective, for the NAS investments we reviewed the results of the agency’s self-assessment of Stages 2, 3, and 4 practices using GAO’s ITIM framework and validated and updated the results of the self- assessment through document reviews and interviews with officials. We reviewed written policies, procedures, and guidance and other documentation providing evidence of executed practices, including FAA’s Acquisition Management System guidance, mission analysis and investment analysis guidance, and memorandums. We also reviewed JRC guidance and records of decision, acquisition review guidance and meeting minutes, and variance reporting procedures and reports. We did not assess FAA’s progress in establishing the capabilities found in one of the two Stage 4 critical processes, entitled Improving the Portfolio’s Performance, or in any of the Stage 5 critical processes, because FAA acknowledged that it had not executed any of the key practices in these critical processes. For the non-NAS investments, we reviewed the results of FAA’s self- assessments of Stage 2 practices using GAO’s ITIM framework and conducted interviews to clarify and update the results. We did not perform a detailed assessment of these practices because they most likely will be superseded by a new process (when it is defined) for managing non-NAS investments, and non-NAS investments are of lower cost and impact to FAA.

As part of our analysis, we selected three IT projects as case studies to verify that the critical processes and key practices were being applied. We selected projects that (1) supported different FAA functional areas, (2) were in different life cycle phases, and (3) required different levels of funding. The three projects are described below: FAA Telecommunications Infrastructure (FTI)—FTI is a performance- based telecommunications services contract for voice, video, and data point-to-point support for telecommunications for the National Airspace System and its support system. It contributes to both the separation of aircraft (the mission-support network) and other FAA uses (the operational network, e.g., e-mail and phone). FTI will replace the current telecom system. FTI will eliminate the need for other subnetworks, of which there are currently eight or nine, and therefore eliminate the management overhead associated with operating so many networks. The integration of multiple networks and subnetworks will provide a single source and single vehicle for telecom. FTI is in the Technical Operations unit and has estimated life cycle costs of $2 billion. The contract for FTI was awarded in June 2002.

En Route Communications Gateway (ECG)—ECG is a mission critical gateway, or interface, for data from radar sites to Air Route Traffic Control Centers. ECG will serve as a single domain communications gateway and will provide the path for exchanging flight plan data from outside sources and transfer data among systems. ECG provides a commercial-off-the-shelf nondevelopmental item digital gateway using a modern, open and extensible platform consisting of modular scalable hardware components. ECG will incorporate interface capability to support legacy and future systems and will provide the capability to transition to modern network communications and access more surveillance sources. The flexibility provided by the ECG system architecture will facilitate the evolution of the En Route domain modernization. ECG will replace the Peripheral Adapter Module Replacement Item system and provide a modern domain gateway that will support the current and future En Route infrastructure. ECG is in the En Route & Oceanic Service group and has estimated life cycle costs of $442.5 million through September 2015.

Voice Switching and Control System (VSCS)—In our review of the VSCS program, we focused our review on one of VSCS’s subcomponents, the VSCS Control Subsystem Upgrade (VCSU). The VCSU program, part of the Technical Operations Communications service group, is designed to maintain overall supportability of VSCS by replacing the hardware for the existing control subsystem, associated VSCS operational and application software, required software licenses, and supporting software and hardware documentation. Deliverables for the VCSU program include all hardware, spare parts, software, software licenses, system baseline documentation, training, and other technical documentation necessary to support the product at 21 locations. According to FAA, the VCSU program has a funding baseline of over $59 million and is in the operations and maintenance phase.

For these projects, we reviewed project management documentation, such as mission needs statements, acquisition program baselines, and integrated program plans. We also interviewed the project managers for these projects.

We compared the evidence collected from our document reviews and interviews to the key practices in ITIM. We rated the key practices as “executed” on the basis of whether the agency demonstrated (by providing evidence of performance) that it had met the criteria of the key practice. A key practice was rated as “not executed” when we found insufficient evidence of a practice during the review or when we determined that there were significant weaknesses in FAA’s execution of the key practice.

To address our second objective, we obtained and evaluated documents showing what management actions had been taken and what initiatives had been planned by the agency. This documentation included JRC records of decisions, the agency’s capital investment guidance, and the recently formed ITEB charter and meeting minutes. We also interviewed the Chief Information Officer, other members of the JRC, and the Chief Operating Officer to determine what efforts FAA had undertaken to improve IT investment management processes.

To address our third objective, we reviewed documentation on DOT’s process for reviewing FAA’s budget proposals and capital planning and investment control reviews. We also conducted interviews with both FAA and DOT officials, including DOT’s CIO and Director for Capital Planning and Investment Control to determine DOT’s oversight role in FAA’s investments and investment management processes.

We conducted our work at FAA Headquarters in Washington, D.C., from October 2003 through July 2004, in accordance with generally accepted government auditing standards.

Investment Management Process Used by Some Organizational Units to Manage Non-NAS Investments

ABA has an investment board that conducts periodic and monthly program reviews for all IT programs to determine whether a program will be approved as an IT investment. A life cycle process guide is now in place to direct the activities of the investment board along with providing oversight of IT projects within ABA.

The business needs of a project within ABA, along with the dates for achieving them, need to be aligned with the strategic goals established in the FAA Flight Plan. Projects or systems that are no longer aligned with the Flight Plan will be decommissioned.

A project management plan identifies, among other things, the system’s users, customers, and types of services to be provided.

Selecting and reselecting an IT investment within ABA involves both the executive management team and ABA’s CIO team. The executive management team reviews the business needs of the investment and compares them against the ABA’s IT budget, while ABA’s CIO team is involved with the selecting and reselecting processes by analyzing the technical costs associated with the IT investment and comparing those technical costs against the ABA’s IT budget.

ABA uses its life cycle process guide to help manage its $25 million IT budget, which consists of 22 or 23 financial systems, 5 or 6 of them considered major programs under OMB’s definition of a major IT investment. A requirement of the life cycle process guide is for every critical system in ABA to have a detailed project management plan that addresses performance measures such as cost, schedule, benefits, and risks.

The day-to-day progress of IT projects is tracked against critical milestones that have been already established through weekly summary reviews with IT staff. For major IT projects, biweekly meetings are conducted that address any concerns with meeting the performance measures.

ABA captures its IT asset information using its Information Technology Investment Portfolio System (ITIPS), which is available to all ABA management and system support personnel. The information in ITIPS is used to manage projects that are in production as well as ensuring that the life cycle activities are in alignment with FAA’s mission statements.

ARA uses its Operations Resource Management Team guide to select, control, and evaluate ARA IT investments. The team composed of representatives from ARA service units. ARA investments are controlled and tracked through quarterly reviews. These reviews look at the cost, schedule, and overall performance of the investment.

The business needs for ARA investments need to be mapped back to the Flight Plan. A monthly status review report is prepared in order to ensure that the business needs are tracking back to the Flight Plan.

ARA does not have any well-defined selection criteria since each program uses its own configuration management plan. ARA Ops build process guides the establishment of new projects.

A project plan does exist, along with established expenditures, which the program managers submit to the ARA CIO on a monthly basis. These monthly status reports occur between the CIO and the program managers to decide if an investment’s resources, such as funding, need to be reallocated. Once the CIO and program manager decide that it is necessary for an investment’s resources to be reallocated, the CIO will discuss the need further with the Deputy Associate Administrator for ARA, who ultimately will determine whether a program will receive additional resources, such as funding.

With respect to the level of interaction that ARA has had with the JRC in the past, only one program from ARA, NextGen, has gone before the JRC. According to the ARA CIO, in order for a program to go to the JRC, there must be justification made to the council that the program is fully operational and is considered to be a benefit and a priority to FAA. The ARA Deputy Associate Administrator will determine if a program should go before the JRC for approval and funding.

The configuration control board uses a database to capture asset inventory data about the systems that are owned by the ARA CIO. According to the ARA CIO, in order for IT assets to be effectively managed in ARA, there needs to be vision from AIO about what programs to invest in over the next 5 years.

The Information Resource Management Executive Board is responsible for selecting, controlling, and evaluating ATS IT investments.

Not all services within ATS have defined their business needs. Even though ATS has the NAS Support Integration Process (NSIP) data repository available for capturing IT asset information, including business needs, and for defining system users, there is no consistency in terms of the records being complete because there are systems within ATS that have not registered with NSIP.

The ATS CIO manages the selection process, which begins with the NSIP registration criteria.

Each business unit within ATS has its own project management plan and procedures. The day-to-day tracking of projects as well as the monitoring of whether corrective actions are being executed is also the responsibility of the individual business units. Even though the individual business units are tasked with this level of responsibility, the ATS CIO does play an oversight role by setting the criteria and policies for the investments to be made for the projects.

ATS uses the NSIP meta data repository to collect any changes to the IT projects and systems by providing a full declaration of the project or system. This includes providing information to help ATS avoid unwanted costs due to systems having redundant functionality and determining whether a system’s or a project’s functions match the stated mission goals for ATS. NSIP also handles the technical rollover for ATS systems or projects.

AIO’s investment management process can be characterized as iterative and well managed, but undocumented. The AIO Business Plan and IT Strategy are used to ensure that when funds are appropriated and allocated that they map back to the Flight Plan. Investments are controlled or tracked by the Deputy CIO on a monthly basis to get an indication as to where the program is in the process against the expenditures that have been already established. Weekly meetings are held with the unit’s CIO to discuss any issues regarding AIO’s investment management process.

AIO does not have any written policies or procedures for identifying business needs for its IT projects. Only one of its major projects, NAS Adaptation Service and Environment, has documented its requirements, which includes specific users.

AIO uses an undocumented process for reviewing new IT proposals to reach an agreement on selection.

There are no AIO-wide policies or procedures for managing projects or investment oversight. The Information Technology Executive Board (ITEB) has been formed to provide a governing structure for non-NAS programs. One of the targets for ITEB is to look at cost control and cross-cutting IT initiatives by involving the heads of the lines of business. The ITEB is also going to be involved with improving the scores on the Exhibit 300 business cases for OMB.

AIO uses ITIPS to track its asset inventory and IT investments. The Deputy CIO of AIO is responsible for ensuring that the inventory located in ITIPS meets the needs of AIO’s investment management process. According to AIO, the information within ITIPS is updated at least twice a year.

AHR does not have an investment board. Instead, AHR’s senior management is responsible for selecting, controlling, and evaluating all IT investments by using established agency acquisition policies and procedures to conduct investment management decisions.

Business needs and specific users for each project are identified within the project plan and are aligned with the AHR Strategic Plan, the FAA Flight Plan, and the AIO Plan. AHR is also aligning its business needs to the ITEB plans. Business needs are re-evaluated on a quarterly basis to ensure that a project is aligned with FAA’s strategic goals and objectives.

AHR senior management uses its prioritization process to evaluate and select investments for funding. The office and center directors determine their requirements and then a budget request is submitted for proposal funding. AHR receives an allowance amount from the budget office. The first priority is to handle personnel payments. The remaining balance is then redistributed to the business divisions. The “building blocks” process starts at this point. This is when base funding is reviewed to decide if a current investment needs continued funding by asking questions about the importance of continuing the funding of a particular project by looking at the project activity and what the impact will be if this project is no longer funded. Each division will submit a list of prioritized projects with costs to the directorate. This list may exceed the budget level. The directorate will reprioritize the original list.

AHR has a Human Resource Management Automation Plan that contains procedures for approving IT projects, and describes the policies and procedures that AHR uses for project management. Despite having project management policies and procedures, not all projects within AHR have a formal project plan. The size and scope of the project are two factors that help determine whether a project has a formal project plan. AHR Division Managers ensure that projects are on time by performing quarterly reviews that assess a project’s cost and schedule. AHR uses a color scheme (red, green, and yellow) to indicate the schedule status of major milestones.

AHR uses the ITIPS as its inventory for making investment management decisions. AHR projects are listed in ITIPS, along with business cases.

The IT Configuration Management Board is ARC’s investment review board. The board’s charter has recently been redone to provide more traceability back to the Flight Plan. The board functions include evaluating potential IT investment options for ARC, making recommendations on IT investment, establishing ARC-wide IT standards, and developing and maintaining investment policies and procedures. The board is led by the unit’s CIO and includes four IT managers from the regional offices and aeronautical center and two members from the ARC Management Team. The ARC Management Team makes the final- selection decisions. The IT investment management decisions are then incorporated into the ARC Business Plan. The ARC unit is also involved with cross-organizational investment decisions for FAA through its membership on the FAA CIO Council.

Business needs are identified through entries made in ITIPS, along with documentation from Exhibit 300s and Exhibit 53s.

ARC does not have its selection criteria documented. To evaluate and select IT investments, the ARC IT Configuration Management Board considers such things as benefits to ARC across the regions, expected return on investment, technical feasibility, and risk. The ARC business plan and the Flight Plan are the documents that address these priorities.

ARC does not have policies or procedures for project management. Instead, ARC uses a weekly teleconference to address expectations and progress of ARC-wide IT initiatives at the IT manager level across ARC. According to the ARC CIO, a second teleconference has been added to discuss portfolio management—schedule, budget, training, and deployment along with whether the project will be integrated with other lines of business.

ARC uses ITIPS as its standardized repository for collecting asset information that will be useful for ARC’s IT investment management decisions by providing information about what types of systems and functions are available and how they are supporting a specific business issue.

Similar to an IT investment board, AVR has a two-tiered management structure that is composed of the AVR management team and the CIO management team. The AVR management team includes the Associate Administrator and the Service Directors who make the final decisions based upon recommendations and input from the CIO management team and its business partners from each of the service units. According to AVR, its IT investment process guide is still under development and will be completed at the end of Fiscal Year 2004.

Each line of business within AVR identifies and documents its business needs including project requirements and specific users. Once the business needs have been identified, the IT Management and Resources section prioritizes them for funding.

Programs in AVR are reviewed quarterly. For major projects, meetings are designed to look at project milestones to see if they are being met. These meetings are carried out biweekly and presented to the AVR management team.

The AVR CIO management team is responsible for monitoring projects and reporting to the AVR Management team. Biweekly meetings are held for major projects within AVR.

AVR’s system inventory is a part of its enterprise architecture. The system inventory is being used primarily in developing the Exhibit 300s. The performance of IT projects in AVR is monitored daily, based upon each project’s individual plan, using project management tools such as MS Project. According to AVR, not all projects have a project plan in place, but AVR is trying to make it a requirement.

GAO’s Mission

The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs