Information Technology Investment: Agencies Can Improve Performance, Reduce Costs, and Minimize Risks
Released: 1996-09-30T00:00:00

Letter

Introduction

There is an increasing demand, coming from the Congress and the public, for a smaller government that works better and costs less. Having valuable, accurate, and accessible financial and programmatic information is a critical element for any improvement effort to succeed. Furthermore, increasing the quality and speed of service delivery while reducing costs will require the government to make significant investments in three fundamental assets—personnel, knowledge, and capital property/fixed assets.

Investments in information technology (IT) projects can dramatically affect all three of these assets. Indeed, the government’s ability to improve performance and reduce costs in the information age will depend, to a large degree, on how well it selects and uses information systems investments to modernize its often outdated operations. However, the impact of information technology is not necessarily dependent on the amount of money spent, but rather on how the investments are selected and managed. This, in essence, is the challenge facing federal executives: Increasing the return on money spent on IT projects by spending money wiser, not faster.

IT projects, however, are often poorly managed. For example, one market research group estimates that about a third of all U.S. IT projects are canceled, at a estimated cost in 1995 of over $81 billion. In the last 12 years, the federal government has obligated at least $200 billion for information management with mixed results at best. Yet despite this huge investment, government operations continue to be hampered by inaccurate data and inadequate systems. Too often, IT projects cost much more and produce much less than what was originally envisioned. Even worse, often these systems do not significantly improve mission performance or they provide only a fraction of the expected benefits. Of 18 major federal agencies, 7 have an IT effort that has been identified as high risk by either the Office of Management and Budget (OMB) or us.

Some private and public sector organizations, on the other hand, have designed and managed IT to improve their organizational performance. In a 1994 report, we analyzed the information management practices of several leading private and state organizations. These leading organizations were identified as such by their peers and independent researchers because of their progress in managing information to improve service quality, reduce costs, and increase work force productivity and effectiveness. From this analysis, we derived 11 fundamental IT management practices that, when taken together, provide the basis for the successful outcomes that we found in leading organizations. (See figure 1.1.)

One of the best practices exhibited by leading organizations was that they manage information systems projects as investments. This particular practice offers organizations great potential for gaining better control over their IT expenditures. In the short term (within 2 years), this practice serves as a powerful tool for carefully managing and controlling IT expenditures and better understanding the explicit costs and projected returns for each IT project. In the long term (from 3 to 5 years), this practice serves as an effective process for linking IT projects to organizational goals and objectives. However, managing IT projects as investments works most effectively when implemented as part of an integrated set of management practices. For example, project management systems must also be in place, reengineering improvements analyzed, and planning processes linked to mission goals.

While the specific processes used to implement an investment approach may vary depending upon the structure of the organization (e.g., centralized versus decentralized operations), we nonetheless found that the leading organizations we studied shared several common management practices related to the strategic use of information and information technologies. Specifically, they maintained a decision-making process consisting of three phases—selection, control, and evaluation—designed to minimize risks and maximize return on investment. (See figure 1.2.)

The Congress has passed several pieces of legislation that lay the groundwork for agencies to establish a investment approach for managing IT. For instance, revisions to the Paperwork Reduction Act (PRA) (Public Law 104-13) have put more emphasis on evaluating the operational merits of information technology projects. The Chief Financial Officers (CFO) Act (Public Law 101-576) focuses on the need to significantly improve financial management and reporting practices of the federal government. Having accurate financial data is critical to establishing performance measures and assessing the returns on IT investments. Finally, the Government Performance and Results Act (GPRA) (Public Law 103-62) requires agencies to set results-oriented goals, measure performance, and report on their accomplishments.

In addition, the recently passed Information Technology Management Reform Act (ITMRA) (Division E of Public Law 104-106) requires federal agencies to focus more on the results achieved through IT investments while streamlining the federal IT procurement process. Specifically this act, which became effective August 8 of this year, introduces much more rigor and structure into how agencies approach the selection and management of IT projects. Among other things, the head of each agency is required to implement a process for maximizing the value and assessing and managing the risks of the agency’s IT acquisitions. Appendix V summarizes the primary IT investment provisions contained in ITMRA.

ITMRA also heightens the role of OMB in supporting and overseeing agencies’ IT management activities. The Director of OMB is now responsible for promoting and directing that federal agencies establish capital planning processes for IT investment decisions. The Director is also responsible for evaluating the results of agency IT investments and enforcing accountability. The results of these decisions will be used to develop recommendations for the President’s budget.

OMB has begun to take action in these areas. In November 1995, OMB, with substantial input from GAO, published a guide designed to help federal agencies systematically manage and evaluate their IT-related investments.This guide was based on the investment processes found at the leading organizations. Recent revisions to OMB Circular A-130 on federal information resources management have also placed greater emphasis on managing information system projects as investments. And the recently issued Part 3 of OMB Circular A-11, which replaced OMB Bulletin 95-03, “Planning and Budgeting for the Acquisition of Fixed Assets,” provides additional guidance and information requirements for major fixed asset acquisitions.

Objectives, Scope, and Methodology

The Chairman, Senate Subcommittee on Oversight of Government Management and the District of Columbia, Committee on Governmental Affairs and the Chairman and Ranking Minority Member, House Committee on Government Reform and Oversight, requested that we compare and contrast the management practices and decision processes used by leading organizations with a small sample of federal agencies. The process used by leading organizations is embodied in OMB’s Evaluating Information Technology Investments: A Practical Guide and specific provisions contained in the Information Technology Management Reform Act of 1996. The agencies we examined are the National Aeronautics and Space Administration (NASA) ($1.6 billion spent on IT in FY 1994), National Oceanic and Atmospheric Administration (NOAA) ($296 million spent on IT in FY 1994), Environmental Protection Agency (EPA) ($302 million spent on IT in FY 1994), Coast Guard ($157 million spent on IT in FY 1994), and the Internal Revenue Service (IRS) ($1.3 billion spent on IT in FY 1994).

We selected the federal agencies for our sample based on one or more of the following characteristics: (1) large IT budgets, (2) expected IT expenditure growth rates, and (3) programmatic risk as assessed by GAO and OMB. In addition, the Coast Guard was selected because of its progress in implementing an investment process. Collectively, these agencies spent about $3.7 billion on IT in FY 1994—16 percent of the total spent on IT. Our review focused exclusively on how well these five agencies manage information technology as investments, one of the 11 practices used by leading organizations to improve mission performance, as described in our best practices report. As such, our evaluation only focused on policies and practices used at the agencywide level; we did not evaluate the agencies’ performance in the 10 other practices. In addition, we did not systematically examine the overall IT track records of each agency.

During our review of agency IT investment decision-making processes, we did the following: reviewed agencies’ policies, practices, and procedures for managing IT investments; interviewed senior executives, program managers, and IRM professionals; and determined whether agencies followed practices similar to those used by leading organizations to manage information systems projects as investments.

We developed the attributes needed to manage information systems projects as investments from the Paperwork Reduction Act, the Federal Acquisition Streamlining Act, OMB Circular A-130, GAO’s “best practices” report on strategic information management, GAO’s strategic information management toolkit, and OMB’s guide Evaluating Information Technology Investments: A Practical Guide. Many of the characteristics of this investment approach are contained in the Information Technology Management Reform Act of 1996 (as summarized in appendix V). However, this law was not in effect at the time of our review.

To identify effects associated with the presence or absence of investment controls, we reviewed agencies’ reports and documents, related GAO and Inspector General reports, and other external reports. We also discussed the impact of the agencies’ investment controls with senior executives, program managers, and IRM professionals to get an agencywide perspective on the controls used to manage IT investments. Additionally, we reviewed agency documentation dealing with IT selection, budgetary development, and IT project reviews.

To determine how much each agency spent on information technology, we asked each agency for information on spending, staffing, and their 10 largest IT systems and projects. The agencies used a variety of sources for the same data elements, which may make comparisons among agencies unreliable. While data submitted by the agencies were validated by agency officials, we did not independently verify the accuracy of the data.

Most of our work was conducted at agencies’ headquarters in Washington, D.C. Similarly, we visited NOAA offices in Rockville, Maryland, and the National Weather Service in Silver Spring, Maryland. We also visited NASA program, financial, and IRM officials at Johnson Space Center in Houston, Texas, and Ames Research Center in San Francisco, California, to learn how they implement NASA policy on IT management. We performed the majority of our work from April 1995 through September 1995, with selected updates through July 1996, in accordance with generally accepted government auditing standards. We updated our analyses of IRS and NASA in conjunction with other related audit work. In addition, several of the agencies provided us with updated information as part of their comments on a draft version of the report. Many of these changes have only recently occurred and we have not fully evaluated them to determine their effect on the agency’s IT investment process.

We provided and discussed a draft of this report with officials from OMB, EPA, NASA, NOAA, IRS, and the Coast Guard, and have incorporated their comments where appropriate. OMB’s written comments, as well as our evaluation, are provided in appendix I.

Appendix II profiles each agency’s IT spending, personnel, and major projects. Appendix III provides a brief description of an IT investment process approach based on work by GAO and OMB. Appendix IV provides a brief overview of each agency’s IT management processes. Because of its relevance to this report, the investment provisions of the Information Technology Management Reform Act of 1996 are summarized in appendix V. Major contributors to this report are listed in appendix VI.

IT Investment Approaches in Five Case Study Agencies: Progress Has Been Made, but Challenges Remain

All of the agencies we studied—NASA, IRS, the Coast Guard, NOAA, and EPA—had at least elements or portions of an IT investment process in place. For instance, the Coast Guard had a selection process with decision criteria that included an analysis of cost, risk, and return data;

EPA had created a executive management group to address cross-agency IT

NASA and NOAA utilized program control meetings to ensure senior management involvement in monitoring the progress of important ongoing IT projects; and IRS had developed a systems investment evaluation review methodology and used it to conduct postimplementation reviews of some Tax Systems Modernization projects.

However, none of these five agencies had implemented a complete, institutionalized investment approach that would fulfill requirements of PRA and ITMRA. Consequently, IT decision-making at these agencies was often inconsistent or based on the priorities of individual units rather than the organization as a whole. Additionally, cost-benefit and risk analyses were rarely updated as projects proceeded and were not used for managing project results. Also, the mission-related benefits of implemented systems were often difficult to determine since agencies rarely collected or compared data on anticipated versus actual costs and benefits.

In general, we found that the IT investment control processes used at the case study agencies at the time of our review contained four main weaknesses. While all four weaknesses may not have been present at each agency, in comparison to leading organizations, the case study agencies lacked a consistent process (used at all levels of the agency) for uniformly selecting and managing systems investments; focused their selection processes on selected efforts, such as justifying new project funding or focusing on projects already under development, rather than managing all IT projects—new, under development, and operational—as a portfolio of competing investments; made funding decisions without giving adequate attention to management control or evaluation processes, and made funding decisions based on negotiations or undefined decision criteria and did not have the up-to-date, accurate data needed to support IT investment decisions.

Appendix IV provides a brief overview of how each agency’s current processes for selecting, controlling, and evaluating IT projects worked.

Agencies Need Consistent Processes to Select and Manage IT Investments

Leading organizations use the selection, control, and evaluation decision-making processes in a consistent manner throughout different units. This enables the organization, even one that is highly decentralized, to make trade-offs between projects, both within and across business units.

Figure 2.1 illustrates how this process can be applied to the federal government where major cabinet departments may have several agencies under their purview. IT portfolio investment processes can exist at both the departmental and agency levels. As with leading organizations, the key factor is being able to determine which IT projects and resources are shared (and should be reviewed at the departmental level) and which are unique to each agency. Three common criteria used by leading organizations are applicable in the federal setting. These threshold criteria include (1) high-dollar, high-risk IT projects (risk and dollar amounts having been already defined), (2) cross-functional projects (two or more organizational units will benefit from the project), and (3) common infrastructure support (hardware and telecommunications). Projects that meet these particular threshold criteria are discussed, reviewed, and decided upon at a departmentwide level. The key to making this work is having clearly defined roles, responsibilities, and criteria for determining the types of projects that will be reviewed at the different organizational levels.

As described in ITMRA, agency heads are to implement a process for maximizing the value and assessing and managing the risks of IT investments. Further, this process should be integrated with the agency’s budget, financial, and program management process(es). Whether highly centralized or decentralized, matrixed or hierarchial, agencies can most effectively reap the benefits of an investment process by developing and maintaining consistent processes within and across their organizations.

One of the agencies we reviewed—the Coast Guard—used common investment criteria for making cross-agency IT decisions. IRS had defined some criteria, but was not yet using these criteria to make decisions. The three other agencies—NASA, EPA, and NOAA—chose IT projects based on inconsistent or nonexistent investment processes. There was little or no uniformity in how risks, benefits, and costs of various IT projects across offices and divisions within these three agencies were evaluated. Thus, cross-comparisons between systems of similar size, function, or organizational impact were difficult at best. More important, management had no assurance that the most important mission objectives of the agency were being met by the suite of system investments that was selected.

NASA, for instance, allowed its centers and programs to make their own IT funding decisions for mission-critical systems. These decisions were made without an agencywide mechanism in place to identify low-value IT projects or costs that could be avoided by capitalizing on opportunities for data sharing and system consolidation across NASA units. As a result, identifying cross-functional system opportunities was problematic at best.

The scope of this problem became apparent as a result of a special NASA IT review. In response to budget pressures, NASA conducted an agencywide internal information systems review to identify cost savings. The resulting March 1995 report described numerous instances of duplicate IT resources, such as large-scale computing and wide area network services, that were providing similar functions. A subsequent NASA Inspector General’s (IG) report, also issued in March 1995, substantiated this special review, finding that at one center NASA managers had expended resources to purchase or develop information systems that were already available elsewhere, either within NASA or, in some cases, within that center itself.

While this special review prompted NASA to plan several consolidation efforts, such as consolidating its separate wide area networks (for a NASA projected savings of $236 million over 5 years), the risk of purchasing duplicate IT resources remained because of weaknesses in its current decentralized decision-making process. For example, NASA created chief information officer (CIO) positions for NASA headquarters and for each of its 23 centers. These CIOs have a key role in improving agencywide IT cooperation and coordination. However, the CIOs have limited formal authority and to date have only exercised control over NASA’s administrative systems—which account for about 10 percent of NASA’s total IT budget. With more defined CIO roles, responsibility, and authority, it is likely that additional opportunities for efficiencies will be identified.

NASA recently established a CIO council to establish high-level policies and standards, approve information resources management plans, and address issues and initiatives. The council will also serve as the IT capital investment advisory group to the proposed NASA Capital Investment Council. NASA plans for this Capital Investment Council to have responsibility for looking at all capital investments across NASA, including those for IT. NASA’s proposed Capital Investment Council may fill this need for identifying cross-functional opportunities; however, it is too early to evaluate its impact.

By having consistent, quantitative, and analytical processes across NASA that address both mission-critical and administrative systems, NASA could more easily identify cross-functional opportunities. NASA has already demonstrated that savings can be achieved by looking within mission-critical systems for cross-functional opportunities. For instance, NASA estimated that $74 million was saved by developing a combined Space Station and Space Shuttle control center using largely commercial off-the-shelf software and a modular development approach, rather than the original plan of having two separate control centers that used mainframe technology and custom software.

EPA, like NASA, followed a decentralized approach for making IT investment decisions. Program offices have had control and discretion over their specific IT budgets, regardless of project size or possible cross-office impact. As we have previously reported, this has led to stovepiped systems that do not have standard data definitions or common interfaces, making it difficult to share environmental data across the agency. This is important because sharing environmental data across the agency is crucial to implementing EPA’s strategic goals. In 1994, EPA began to address this problem by creating a senior management Executive Steering Committee (ESC) charged with ensuring that investments in agencywide information resources are managed efficiently and effectively. This committee, comprised of senior EPA executives, has the responsibility to (1) recommend funding on major system development efforts and (2) allocate the IT budget reserved for agencywide IRM initiatives, such as geographical information systems (GIS) support and data standards.

At the time of our review, the ESC had not reviewed or made recommendations on any major information system development efforts. Instead, the ESC focused its activity on spending funds allocated to it for agencywide IRM policy initiatives, such as intra-agency data standards. The ESC met on June 26, 1996, to assess the impact of ITMRA upon EPA’s IT management process.

Agencies Need to Manage Their IT Projects as a Portfolio

In conducting their selection processes, leading organizations assess and manage the different types of IT projects, such as mission-critical or infrastructure, at all different phases of their life cycle, in order to create a complete strategic investment portfolio. (See figure 2.2.) By scrutinizing and analyzing their entire IT portfolio, managers can examine the costs of maintaining existing systems versus investing in new ones. By continually and rigorously reevaluating the entire project portfolio based on mission priorities, organizations can reach decisions on systems based on overall contribution to organizational goals. Under ITMRA, agencies will need to compare and prioritize projects using explicit quantitative and qualitative decision criteria.

At the federal agencies we studied, some prioritization of projects was conducted, but none made managerial trade-offs across all types of projects. IRS, NOAA, and the Coast Guard each conducted some type of portfolio analyses; EPA and NASA did not. Additionally, the portfolio analyses that were performed generally covered projects that were either high dollar, new, or under development. For example, in 1995 we reported that IRS executives were consistently maintaining that all 36 TSM projects, estimated to cost up to $10 billion through the year 2001, were equally important and must all be completed for the modernization to succeed.This approach, as well as the accompanying initial failure to rank the TSM projects according to their prioritized needs and mission performance improvements, has meant that IRS could not be sure that the most important projects were being developed first.

Since our 1995 report, IRS has begun to rank and prioritize all of the proposed TSM projects using cost, risk, and return decision criteria. However, these decision criteria are largely qualitative, the data used for decisions were not validated or reliable, and analyses were not based on calculations of expected return on investment.

In addition, according to IRS, its investment review board uses a separate process with different criteria for analyzing operational systems. IRS also said that the board does not review research and development (R&D) systems or field office systems. Using separate processes for some system types and not including all systems prevents IRS from making comparisons and trade-offs as part of a complete IT portfolio.

Of all the agencies we reviewed, the Coast Guard had the most experience using a comprehensive selection phase. In 1991, the Coast Guard started a strategic information resources management process and shortly thereafter initiated an IT investment process. Under this investment process, a Coast Guard working group from the IRM office ranks and prioritizes new IT projects and those under development based on explicit risk and return decision criteria. A senior management board meets annually to rank the projects and decide on priorities.

The Coast Guard has derived benefits from its project selection process. During the implementation of its IT investment process, the Coast Guard identified opportunities for systems consolidation. For example, the Coast Guard reported that five separate personnel systems are being incorporated into the Personnel Management Information System/Joint Military Pay System II for a cost avoidance of $10.2 million. The Coast Guard also identified other systems consolidation opportunities that, if implemented, could result in a total cost savings of $77.4 million.

However, at the time of our review, the Coast Guard’s selection process was still incomplete. For example, R&D projects and operational systems were not included in the prioritization process. As a result, the Coast Guard could not make trade-offs between all types of proposed systems investments, creating a risk that new systems would be implemented that duplicate existing systems. Additionally, the Coast Guard was at risk of overemphasizing investments in one area, such as maintenance and enhancements for existing systems, at the expense of higher value investments in other areas, such as software applications development supporting multiple unit needs.

Management Control and Evaluation Processes Were Often Absent

Leading organizations continue to manage their investments once selection has occurred, maintaining a cycle of continual control and evaluation. Senior managers review the project at specific milestones as the project moves through its life cycle and as the dollar amounts spent on the project increase. (See figure 2.3.) At these milestones, the executives compare the expected costs, risks, and benefits of earlier phases with the actual costs incurred, risks encountered, and benefits realized to date. This enables senior executives to (1) identify and focus on managing high-potential or high-risk projects, (2) reevaluate investment decisions early in a project’s life cycle if problems arise, (3) be responsive to changing external and internal conditions in mission priorities and budgets, and (4) learn from past success and mistakes in order to make better decisions in the future. The level of management attention focused on each of the three investment phases is proportional based on such factors as the relative importance of each project in the portfolio, the relative project risks, and the relative number of projects in different phases of the system development process.

The control phase focuses senior executive attention on ongoing projects to regularly monitor their interim progress against projected risks, cost, schedule, and performance. The control phase requires projects to be modified, continued, accelerated, or terminated based on the results of those assessments. In the evaluation phase, the attention is focused on implemented systems to give a final assessment of risks, costs, and returns. This assessment is then used to improve the selection of future projects.

Similarly in the federal government, GPRA forces a shift in the focus of federal agencies—away from such traditional concerns as staffing and activity levels and towards one overriding issue: results. GPRA requires agencies to set goals, measure performance, and report on their accomplishments. Just as in leading organizations, GPRA, in concert with the CFO Act, is intended to bring a more disciplined, businesslike approach to the management of federal programs.

The agencies we reviewed focused most of their resources and attention on selecting projects and gave less attention to controlling or evaluating those projects. While IRS, NASA, and NOAA had implemented control mechanisms, and IRS had developed a postimplementation review methodology, none of the agencies had complete and comprehensive control and evaluation processes in place. Specifically, in the five case study agencies we evaluated, we found that control mechanisms were driven primarily by cost and schedule concerns without any focus on quantitative performance measures, evaluations of actual versus projected returns were rarely conducted, and information and lessons learned in either the control or evaluation phases were not systematically fed back to the selection phase to improve the project selection process.

Management Control Processes Were Focused Primarily on Cost and Schedule

Leading organizations maintain control of a project throughout its life cycle by regularly measuring its progress against not only projected cost and schedule estimates, but also quantitative performance measures, such as benefits realized or demonstrated in pilot projects to date. To do this, senior executives from the program, IRM, and financial units continually monitor projects and systems for progress and identify problems. When problems are identified, they take immediate action to resolve them, minimize their impact, or alter project expectations.

Legislation now requires federal executives to conduct this type of rigorous project monitoring. With the passage of ITMRA, agencies are required to demonstrate, through performance measures, how well IT projects are improving agency operations and mission effectiveness. Senior managers are also to receive independently verifiable information on cost, technical and capability requirements, timeliness, and mission benefit data at project milestones. Furthermore, pursuant to the Federal Acquisition Streamlining Act of 1994 (Public Law 103-355), if a project deviates from cost, schedule, and performance goals, the agency head is required to conduct a timely review of the project and identify appropriate corrective action—to include project termination.

Two of the agencies we reviewed—the Coast Guard and EPA—did not use management control processes that focused on IT systems projects. The other three agencies—IRS, NOAA, and NASA—had management control processes that focused primarily on schedule and cost concerns, but not interim evaluations of performance and results. Rarely did we find examples in which anticipated benefits were compared to results at critical project milestones. We also found few examples of lessons that were learned during the control phase being cycled back to improve the selection phase.

To illustrate, both IRS and NASA used program control meetings (PCMs) to keep senior executives informed of the status of their major systems by requiring reports, in the form of self-assessments, from the project managers. However, these meetings did not focus on how projects were achieving interim, measurable improvement targets for quality, speed, and service that could form the basis for project decisions about major modifications or termination. IRS, for instance, used an implementation schedule to track different components of each of its major IT projects under TSM. Based on our discussions with IRS officials, the PCMs focused on factors bearing on real or potential changes in project costs or schedule. Actual, verified data on interim application or system testing results—compared to projected improvements in operational, mission improvements—were not evaluated.

At NASA, senior program executives attended quarterly Program Management Council (PMC) meetings to be kept informed of major programs and projects and to take action when problems arose. While not focused exclusively on IT issues, the PCMs were part of a review process that looked at implementation issues of programs and projects that (1) were critical to fulfilling NASA’s mission, particularly those that were assigned to two or more field installations, (2) involved the allocation of significant resources, defined as projects whose life-cycle costs were over $200 million, or (3) warranted special management attention, including those that required external agency reporting on a regular basis. During the PMC meetings, senior executives reviewed self-assessments (grades of green, yellow, and red), done by the responsible project manager, on the cost, schedule, and technical progress of the project.

Using this color-coded grading scheme, NASA’s control process focused largely on cost, schedule, and technical concerns, but not on assessing improvements to mission performance. Additionally, the grading scheme was not based on quantitative criteria, but instead was largely qualitative and subjective in nature. For instance, projects were given a “green” rating if they were “in good shape and on track consistent with the baseline.” A “yellow” rating was defined as a “concern that is expected to be resolved within the schedule and budget margins,” and a “red” rating was defined as “a serious problem that is likely to require a change in the baseline determined at the beginning of the project.” However, the lack of quantitative criteria, benefit analysis, and performance data invited the possibility for widely divergent interpretations and a misunderstanding of the true value of the projects under review.

As of 1995, three IT systems had met NASA’s review criteria and had been reviewed by the PMC. These three systems constituted about 7 percent of NASA’s total fiscal year 1994 IT spending. No similar centralized review process existed for lower dollar projects, which could have resulted in problem projects and systems that collectively added up to significant costs being overlooked. For instance, in 1995 NASA terminated an automated accounting system project that had been under development for about 6 years, had cost about $45 million to date, and had an expected life-cycle cost of over $107 million.

In responding to a draft of this report, the NASA CIO said that the current cost threshold of $200 million is being reduced to a lower level to ensure that most, if not all, agency IT projects will be subject to PMC reviews. In addition, the CIO noted that NASA’s internal policy directive on program/project management is being revised to (1) include IT evaluation criteria that are aligned with ITMRA and executive-branch guidance and (2) clearly establish the scope and levels of review (agency, lead center, or center) for IT investment decisions.

Evaluations of Actual Versus Projected Returns Are Rarely Conducted

Once projects have been implemented and become operational, leading organizations evaluate them to determine whether they have achieved the expected benefits, such as lowered cost, reduced cycle time, increased quality, or increased the speed of service delivery. They do this by conducting project postimplementation reviews (PIRs) to compare actual to planned cost, returns, and risks. The PIR results are used to calculate a final return on investment, determine whether any unanticipated modifications may be necessary to the system, and provide “lessons learned” input for changes to the organization’s IT investment processes and strategy. ITMRA now requires agencies to report to OMB on the performance benefits achieved by their IT investments and how those benefits support the accomplishment of agency goals.

Only one of the five federal agencies we reviewed—IRS—systematically evaluated implemented IT projects to determine actual costs, benefits, and risks. Indeed, we found that most of the agencies rarely evaluated implemented IT projects at all. In general, the agency review programs were insufficiently staffed and used poorly defined and inconsistent approaches. In addition, in cases where evaluations were done, the findings were not used to consider improvements or revisions in the IT investment decision-making process.

NOAA, for instance, had no systematic process in place to ensure that it was achieving the planned benefits from its annual $300 million IT expenditure. For example, of the four major IT projects that constitute the $4.5 billion National Weather Service (NWS) modernization effort, only the benefits actually accruing from one of four—the NEXRAD radars—had been analyzed.

While not the only review mechanism used by the agency, NOAA’s central review program was poorly staffed. NOAA headquarters, with half a staff year devoted to this review program, generally conducted reviews in collaboration with other organizational units and had participated in only four IT reviews over the last 3 fiscal years. Additionally, these reviews generally did not address the systems’ projected versus actual cost, performance, and benefits.

IRS had developed a PIR methodology that it used to conduct five systems postimplementation reviews. A standardized methodology is important because it makes the reviews consistent and adds rigor to the analytical steps used in the review process. The IRS used the June 1994 PIR on the Corporate Files On-Line (CFOL) system as the model for this standardized methodology. In December 1995, IRS used the PIR methodology to complete a review of the Service Center Recognition/Image Processing System (SCRIPS). Subsequently, three more PIRs have been completed (TAXLINK, the Enforcement Revenue Information System, and the Integrated Collection System) and five more are scheduled. IRS estimated that the five completed systems have an aggregate cost of about $845 million.

However, the PIR methodology was not integrated into a cohesive investment process. Specifically, there were no mechanisms in place to take the lessons learned from the PIRs and apply them to the decision criteria and other tools and techniques used in their investment process.

As a result, the PIRs that were conducted did not meet one of their primary objectives—to ensure continual improvement based on lessons learned—and IRS ran the risk of repeating past mistakes.

Agency IT Decisions Are Not Based on Adequate Data

To help make continual decisions on IT investments, leading organizations require all projects to have complete and up-to-date project information. This information includes cost and benefit data, risk assessments, implementation plans, and initial performance measures. (See figure 2.4). Maintaining this information allows senior managers to rigorously evaluate the current status of projects. In addition, it allows them to compare IT projects across the organization; consider continuation, delay, or cancellation trade-offs; and take action accordingly.

ITMRA requires agencies to use quantitative and qualitative criteria to evaluate the risks and the returns of IT investments. As such, agencies need to collect and maintain accurate and reliable cost, benefit, risk, and performance data to support project selection and control decisions. The requirement for accurate, reliable, and up-to-date financial and programmatic information is also a primary requirement of the CFO Act and is essential to fulfilling agency requirements for evaluating program results and outcomes under GPRA.

At the five case study agencies we evaluated, we found that, in general agency IT investment decisions were based on undefined or implicit data on the project’s cost, schedule, risks, and returns were not documented, defined, or kept up-to-date, and, in many cases, were not used to make investment decisions.

Explicit Decision Criteria Were Not Defined

To ensure that all projects and operational systems are treated consistently, leading organizations define explicit risk and return decision criteria. These criteria are then used to evaluate every IT project or system. Risk criteria involve managerial, technical, resource, skill, security, and organizational factors, such as the size and scope of the project, the extent of use of new technology, the potential effects on the user organization, the project’s technical complexity, and the project’s level of dependency on other systems or projects. Return criteria are measured in financial and nonfinancial terms. Financial measurements can include return on investment and internal rate of return analyses while nonfinancial assessments can include improvements in operational efficiency, reductions in cycle time, and progress in better meeting customer needs.

Of the five agencies in our sample, only the Coast Guard used a complete set of decision criteria. These decision criteria included (1) risk assessments of schedule, cost, and technical feasibility dimensions, (2) cost-benefit impacts of the investment, (3) mission effectiveness measures, (4) degree of alignment with strategic goals and high-level interest (such as Congress or the President), and (5) organizational impact in the areas of personnel training, quality of work life, and increased scope of service. The Coast Guard used these criteria to prioritize IT projects and justify final selections. The decision criteria were weighted and scored, and projects were evaluated to determine those with the greatest potential to improve mission performance.

Generally, officials in other agencies stated that they determine which projects to fund based on the judgmental expertise of decisionmakers involved in the process. NOAA, for instance, had a board of senior executives that met annually to determine budget decisions across seven strategic goals. Working groups for each strategic goal met and each created a prioritized funding list, which was then submitted to the executive decision-making board. These working groups did not have uniform criteria for selecting projects. The executive board accepted the prioritized lists as submitted and made funding threshold decisions based on these lists. As a result, the executive board could not easily make consistent, accurate trade-offs among the projects that were selected by these individual working groups on a repeatable basis.

In addition, to maximize funding for a specific working group, project rankings may not have been based on true risk or return. According to a NOAA senior manager and the chair of one of the NOAA working groups, one group ranked high-visibility projects near the bottom of the list to encourage the senior decision-making board to draw the budgetary cut-off line below these high visibility projects. Few of these high-visibility projects were at the top of the list, despite being crucial to NOAA and high on the list of the NOAA Administrator’s priorities. Explicit decision criteria would eliminate this type of budgetary gamesmanship.

Data Were Often Not Consistent or Up-To-Date

Leading organizations consider project data the foundation by which they select, control, and evaluate their IT investments. Without it, participants in an investment process cannot determine the value of any one project. Leading organizations use rigorous and up-to-date cost-benefit analyses, risk assessments, sensitivity analyses, and project specific data including current costs, staffing, and performance, to make funding decisions and project modifications based, whenever possible, on quantifiable data.

While the agencies in our sample developed documents in order to get project approvals, little effort was made to ensure that the information was kept accurate and up-to-date, and rarely were the data used to manage the project throughout its life cycle. During our review, we asked each agency to supply us with basic data on its largest dollar IT projects. However, this information was not readily available and gathering it required agency officials to rely on a variety of sometimes incomparable sources for system cost, life-cycle phase, and staffing levels.

In addition, some of the agencies could not comparatively analyze IT projects because they did not keep a comprehensive accounting of data on all of the IT systems. For example, EPA had to conduct a special information collection to identify life-cycle cost estimates on its major systems and projects for this report. While the individual system managers at EPA did have system life-cycle cost estimates, the fact that this information was decentrally maintained made cross-system comparisons unlikely. In a 1995 report, the NASA IG found that neither NASA headquarters nor any of the NASA centers had a complete inventory of all information systems for which they were responsible.

All of the agencies we reviewed conducted cost-benefit analyses for their major IT projects. However, these analyses were generally done to support decisions for project approval and were seldom kept current. In addition, the cost-benefit projections were rarely used to evaluate actual project results.

The NWS modernization, for instance, has a cost-benefit analysis that was done in 1992. This analysis covers the four major systems under the modernization. To be effective, an analysis should include the costs and benefits of each project, alternatives to that project, and finally, a combined cost-benefit analysis for the entire modernization. However, the cost-benefit analysis that was conducted only compares the aggregate costs and benefits of the NWS modernization initiative against the current system. It does not assess or analyze the costs and benefits of each system, nor does it examine alternatives to those systems. As a result, NWS does not know if each of the modernization projects is cost-beneficial, and cannot make trade-offs among them. If using only this analysis, decision-makers are forced to choose either the status quo or all of the projects proposed under the modernization.

Without updated cost-benefit data, informed management decisions become difficult. We reported in April 1995 that NWS was trying to assess user concerns related to the Automated Surface Observing System (ASOS), one of the NWS modernization projects, but that NWS did not have a complete estimate of what it would cost to address these concerns. As we concluded in the report, without reliable estimates of what an enhanced or supplemented ASOS would cost, it would be difficult for NWS to know whether continued investment in ASOS is cost-beneficial.

Agency Comments and Our Evaluation

We provided and discussed a draft of this report with officials from EPA, NASA, NOAA, IRS, and the Coast Guard, and have incorporated their comments where appropriate. Several of the agencies noted that they, in response to the issuance of OMB’s guidance on IT investment decision-making and the passage of ITMRA, have made process changes and organizational modifications affecting IT funding decisions. We have incorporated this information into the report where applicable. However, many of the process changes and modifications have occurred very recently, and we have not fully evaluated these changes or determined their effects.

Officials from NOAA and NASA also had reservations about the applicability of the investment portfolio approach to their organizations because their decentralized operating environments were not conducive to a single agencywide portfolio model with a fixed set of criteria. Because any organization, whether centralized or decentralized, has to operate within the parameters of a finite budget, priorities must still be set, across the organization, about where limited IT dollars will be spent to achieve maximum mission benefits. We agree that many IT spending decisions can be made at the agency or program level. However, there are some decisions—especially those involving projects that are (1) high-risk, high-dollar, (2) cross-functional, (3) or providing a common infrastructure (e.g., telecommunications)—that should be made at a centralized, departmental level. Establishing a common, organizationwide focus, while still maintaining a flexible distribution of departmental and agency/program/site decision-making, can be achieved by implementing standard decision criteria. These criteria help ensure that projects are assessed and evaluated consistently at lower levels, while still maintaining an enterprisewide portfolio of IT investments.

Conclusions and Recommendations

Conclusions

Buying information technology can be a high-risk, high-return undertaking that requires strong management commitment and a systematic process to ensure successful outcomes. By using an investment-driven management approach, leading organizations have significantly increased the realized return on information technology investments, reduced the risk of cost overruns and schedule delays, and made better decisions about how their limited IT dollar should be spent.

Adopting such an investment-driven approach can provide federal agencies with similar opportunities to achieve greater benefits from their IT investments on a more consistent basis. However, the federal case study agencies we examined used decision-making processes that lacked many essential components associated with an investment approach. Critical weaknesses included the absence of reliable, quantitative cost figures, net return on investment calculations, rigorous decision criteria, and postimplementation project reviews. With sustained management attention and substantive improvements to existing processes, these agencies should be able to meet the investment-related provisions of ITMRA.

Implementing and refining an IT investment process, however, is not an easy undertaking and cannot be accomplished overnight. Maximizing the returns and minimizing the risks on the billions of dollars that are spent each year for IT will require continued efforts on two fronts. First, agencies must fundamentally change how they select and manage their IT projects. They must develop and begin using a structured IT investment approach that encompasses all aspects of the investment process—selection, control, and evaluation.

Second, oversight attention far beyond current levels must be given to agencies’ management processes and to actual results that are being produced. Such attention should include the development of policies and guidance as well as selective evaluations of processes and results. These evaluations should have a dual focus: They should identify and address deficiencies that are occurring, but they should also highlight positive results in order to share lessons learned and speed success.

OMB’s established leadership role, as well as the policy development and oversight responsibilities that it was given under ITMRA, place it in a key position to provide such oversight. OMB has already initiated several changes to governmentwide guidance to encourage the investment approach to IT decision-making, and has drawn upon the assistance of several key interagency working groups comprised of senior agency officials. Such efforts should be continued and expanded, to ensure that the federal government gets the most return for its information technology investments.

Recommendations

Given its significant leadership responsibility in supporting agencies’ improvement efforts and responding to requirements of ITMRA, it is imperative that OMB continue to clearly define expectations for agencies and for itself to successfully implement investment decision-making approaches. As such, we are recommending four specific actions for the Director of OMB to take.

OMB’s first challenge is to help agencies improve their investment management processes. With effective processes in place, agencies should be in much stronger positions to make informed decisions about the relative benefits and risks of proposed IT spending. Without them, agencies will continue to be vulnerable to risks associated with excessively costly projects that produce questionable mission-related improvements. Under Sections 5112 and 5113 of the Information Technology Management Reform Act, the Director of OMB has responsibility for promoting and directing that federal agencies establish capital planning processes for information technology investment decisions. In designing governmentwide guidance for this process, we recommend that the Director of the Office of Management and Budget require agencies to: Implement IT investment decision-making processes that use explicitly defined, complete, and consistent criteria applied to all projects, regardless of whether project decisions are made at the departmental, bureau, or program level. With criteria that reflect cost, benefit, and risk considerations, applied consistently, agencies should be able to make more reasonable and better informed trade-offs between competing projects in order to achieve the maximum economic impact for their scarce investment dollars.

Periodically analyze their entire portfolios of IT investments—at a minimum new projects, as well as projects in development and operations and maintenance expenditures—to determine which projects to approve, cancel or delay. With development and maintenance efforts competing directly with one another for funding, agencies will be better able to gauge the best proportion of investment in each category of spending to move away from their legacy bases of systems with excessive maintenance costs.

Design control and evaluation processes that include cost, schedule, and quantitative performance assessments of projected versus actual improvement in mission outcomes. As a result, they should increase their capacity to both assess actual project results and learn from their experience which operational areas produce the highest returns and how well they estimate projects and deliver final results.

Advise agencies in setting minimum quality standards for data used to assess (qualitatively and quantitatively) cost, benefit, and risks decisions on IT investments. Agencies should demonstrate that all IT funding proposals include only data meeting these quality requirements and that projected versus actual results are assessed at critical project milestones. The audited data required by the CFO Act should help produce this accurate, reliable cost information. Higher quality information should result in better and more consistent decisions on complex information systems investments.

OMB’s second challenge is to use the results produced by the improved investment processes to develop recommendations for the President’s budget that reflect an agency’s actual track record in delivering mission performance for IT funds expended. Under Section 5113 of ITMRA, the Director of OMB is charged with evaluating the results of agency IT investments and enforcing accountability—including increases or reductions in agency IT funding proposals—through the annual budget process. In carrying out these responsibilities, we recommend that the Director of the Office of Management and Budget:

Evaluate information system project cost, benefit, and risk data when analyzing the results of agency IT investments. Such analyses should produce agency track records that clearly and definitively show what improvements in mission performance have been achieved for the IT dollars expended.

Ensure that the agency investment control process are in compliance with OMB’s governmentwide guidance, and if not, assess strengths and weaknesses and recommend actions and timetables for improvements. When results are questionable or difficult to determine, monitoring agency investment processes will help OMB diagnose problem causes by determining the degree of agency control and the quality of decisions being made.

Use OMB’s evaluation of each agency’s IT investment control processes and IT performance results as a basis for recommended budget decisions to the President. This direct linkage should give agencies a strong, much needed incentive to maximize the returns and minimize the risks of their scarce IT investments.

To effectively implement improved investment management processes and make the appropriate linkages between agency track records and budget recommendations, OMB also has a third challenge. It will need to marshal the resources and skills to execute the new types of analysis required to make sound investment decisions on agency portfolios. Specifically, we recommend that the Director of the Office of Management and Budget:

Organize an interagency group comprised of budget, program, financial, and IT professionals to develop, refine and transfer guidance and knowledge on best practices in IT investment management. Such a core group can serve as an ongoing source of practical knowledge and experience on the state of the practice for the federal government.

Obtain expertise on an advisory basis to assist these professionals in implementing complete and effective investment management systems. Agency senior IRM management could benefit greatly from a high quality, easily accessible means to solicit advice from capital planning and investment experts outside the federal government. Identify the type and amount of skills required for OMB to execute IT portfolio analyses, determine the degree to which these needs are currently satisfied, specify the gap and both design and implement a plan, with timeframes and goals, to close the gap. Given existing workloads and the resilience of the OMB culture, without a determined effort to build the necessary skills, OMB will have little impact on the quality of IT investment decision-making. If necessary to augment its own staff resources, OMB should consider the option of obtaining outside support to help perform such assessments.

Finally, as part of its internal implementation strategy, the Director of the Office of Management and Budget should consider developing an approach to assessing OMB’s own performance in executing oversight responsibilities under the ITMRA capital planning and investment provisions. Such a process could focus on whether OMB reviews of agency processes and results have an impact on reducing risk or increasing the returns on information technology investments—both within and across federal agencies.

Agency Comments and Our Evaluation

In its written comments on a draft of our report, OMB generally supported our recommendations and said that it is working towards implementing many aspects of the recommendations as part of the fiscal year 1998 budget review process of fixed capital assets. OMB also provided observations or suggestions in two additional areas. First, OMB stated that given ITMRA’s emphasis on agencies being responsible for IT investment results, it did not plan to validate or verify that each agency’s investment control process is in compliance with OMB’s guidance contained in its management circulars. As discussed in our more detailed evaluation of OMB’s comments in appendix I, conducting selective evaluations is an important aspect of an overall oversight and leadership role because it can help identify management deficiencies that are contributing to poor IT investment results.

Second, OMB noted that the relationship of IT investment processes between a Cabinet department and bureaus or agencies within the department was not fully evaluated and that additional attention would be needed as more data on this issue become available. We agree that our focus was on assessing agencywide processes and that continued attention to the relationships between departments, bureaus, and agencies will contribute to increased understanding across the government and will ultimately improve ITMRA’s chances of success. This issue is discussed in more detail in our response to comments provided by the five agencies we reviewed (summarized at the end of chapter 2).

Comments From the Office of Management and Budget and Our Evaluation

The following are GAO’s comments on the Office of Management and Budget’s letter dated July 26, 1996.

GAO Comments

1. As stated in the scope and methodology section of the report, we focused our analysis on agencywide processes. We agree that continued attention to this issue will contribute to increased understanding across the government and will ultimately improve ITMRA’s chances of success. As noted in our response to comments received from the agencies we reviewed (provided at the end of chapter 2), we believe that a flexible distribution of departmental and agency/program/site IT decision-making is possible and can best be achieved by implementing standard decision criteria for all projects. In addition, we note that particular types of IT decisions, such as those with unusually high-risk, cross-functional impact or that provide common infrastructure needs, are more appropriately decided at a centralized, departmental level.

Experience gained during implementation of the Chief Financial Officers (CFO) Act showed that departmental-level CFOs needed time to build effective working relationships with their agency- or bureau-level counterparts. We believe the same will be true for Chief Information Officers (CIOs) established by ITMRA and that establishing and maintaining this bureau-level focus will be integral for ensuring the act’s success. 2. ITMRA does squarely place responsibility and accountability for IT investment results with the head of each agency. Nevertheless, ITMRA clearly requires that OMB provide a key policy leadership and implementation oversight role. While we agree that it may not be feasible to validate and verify every agency’s investment processes, it is still essential that selected evaluations be conducted on a regular basis. These evaluations can effectively support OMB’s performance and results-based approach. They can help to identify and understand problems that are contributing to poor investment outcomes and also help perpetuate success by providing increased learning and sharing about what is and is not working.

Agency Investment Technology Profiles

In order to develop a profile of each agency’s IT environment, we asked the agencies to provide us information on the following: total IT expenditures for fiscal year 1990 through fiscal year 1994; total number of staff devoted to IRM functions and activities for fiscal year 1990 through fiscal year 1994; and costs for the 10 largest IT projects for fiscal year 1994 (as measured by total project life-cycle cost).

To gather this information, we developed a data collection instrument and submitted it to responsible agency officials. Information supplied by the agencies is summarized in the following tables. We did not independently verify the accuracy of this information. Moreover, comparison of figures across the agencies is difficult because agency officials used different sources (such as budget data, IRM strategic plans, etc.) for the same data elements.

U.D.

Provides an organizationwide microcomputer infrastructure and is the primary source for acquiring desktop, server and portable hardware; operating system and office automation system software; utilities and peripherals, training, personnel support, and cabling.

Op.

Provides continued support for the Coast Guard’s existing microcomputer infrastructure.

Op.

Provides a consolidated accounting and pay system.

U.D.

A configuration of sensors, communication links, personnel, and decision support tools that will modernize and expand the systems in three cities by incorporating radar sensor information overlaid on digital nautical charts as well as improved decision support systems.

U.D.

Provides an automated and consolidated communication system.

U.D.

Merges two maintenance systems for tracking and recording scheduled aviation maintenance actions.

U.D.

Reprograms most of the existing Coast Guard developed applications to comply with the National Institute of Standards and Technology’s Application Portability Profile.

Op.

Provides safety performance histories of vessels and involved parties and is used as a decision support tool for the Commercial Vessel Safety program.

Op.

Provides aviation technical publications in electronic format.

U.D.

Consolidated into the Coast Guard Standard Workstation III system.

Op.

Performs funds control from commitments through payment; updates all ledgers and tables as transactions are processed; provides a standard means of data entry, edit, and inquiry; and provides a single set of reference and control files.

Op.

Contains data submitted to EPA under the Emergency Planning and Community Right to Know Act for chemicals and chemical categories listed by the agency. Data include chemical identity, amount of on-site users, release and off-site transfers, on-site treatment, minimization/prevention actions. Public access is provided by the National Library of Medicine.

Op.

Supports management and administration of chemical samples from Superfund sites that are analyzed under agency contracts with chemical laboratories. The system schedules and tracks samples from site collection, through analysis, to delivery to the agency.

Op.

Stores air quality, point source emissions, and area/mobile source data required by federal regulations from the 50 states.

Op.

Superfund’s official source of planning and accomplishment data. Serves as the primary basis for strategic decision-making and site-by-site tracking of cleanup activities.

Op.

Contains a set of computer applications and a major relational database which is used to support regulation development, air quality analysis, compliance audits, investigations, assembly line testing, in-use compliance, legislation development, and environmental initiatives.

Op.

Maintains basic data identifying and describing hazardous waste handlers; detailed information about hazardous waste treatment storage and disposal processes, environmental permitting, information on inspections, violations, and enforcement actions; and tracks specific corrective action information needed to regulate facilities with hazardous waste releases.

Op.

Supports the National Pollutant Discharge Elimination System, a Clean Water Act program that issues permits and tracks facilities that discharge pollutants into our navigable waters. (continued)

U.D.

A replacement for the existing Comprehensive Environmental Response, Compensation, and Liability Information System described above.

Op.

A PC LAN version of the Comprehensive Environmental Response, Compensation, and Liability Information System database used by EPA regional offices for data input and local analysis needs.

U.D.

Acquire and install Tax System Modernization host-tier computers at three computing centers.

U.D.

Integrates five systems that control, assign, prioritize, and track taxpayer inquiries; provides office automation, case folder review and inventories, and display and manipulation of case inquiry folders; automates collection cases; provide access to current tax return information; automates case preparation and closure; and provides standardized hardware and custom software to the criminal investigation function on a nationwide basis.

U.D.

Integrates six systems that will receive and control information being transmitted to or from IRS; automates remittance processing activities; scans paper tax returns and correspondence for processing in an automated database; provides automated telephone assistance to customers; permits individual and business tax returns to be filed by utilizing a touch-tone phone; and provides access to all electronically filed returns that have been scored for potential fraud.

Op.

Provides case tracking, expanded legal research, a document management system for briefs, an integrated office system, time reporting, issue tracking, litigation support, and a decision support system. (continued)

U.D.

Integrates three systems that provide application programs to query, search, update, analyze and extract information from a database; aggregates tax information into electronic case folders and distributes them to field locations; and provides the security infrastructure to support all components of the Tax System Modernization.

U.D.

Provides a variety of workstation models, monitors, printers, operating systems and related equipment; provides for standardization of the small and medium-scale computers used by front line programs in the national and field offices and service centers.

Op.

Provides funding for (1) the mainframe and miscellaneous peripherals at each service center, (2) magnetic media and ADP supplies for all service centers, (3) lease and maintenance for support equipment, and (4) on-line access to taxpayer information and account status.

U.D.

Provides an interim hardware platform at two computing centers to support master file processing and full implementation of the CFOL data retrieval/delivery system.

U.D.

Provides upgradable software development workstations and workbench tools, including automated analysis and design tools; requirements traceability tools; construction kits with smart editors, compilers, animators, and debuggers; and static analyzers.

U.D.

Integrates four systems that provide for ordering and delivery of telecommunication systems and services for Treasury bureaus; serves as a Government Open Systems Interconnection Profile prototype; provides centralized network and operations management and will acquire about 14,000 workstations.

U.D.

Receives, processes, archives, and distributes earth science research data from U.S., European, and Japanese polar platforms, selected Earth probes, the Synthetic Aperture Radar free flyer, selected existing databases, and other sources of related data.

Op.

Provides telecommunications and computation services for Marshall Space Flight Center.

Op.

Supports most data systems, networks, user workstations and telecommunications systems and provides maintenance, operations, software development, engineering, and customer support functions at Johnson Space Center.

Op.

Provides a family of compatible computing systems covering a broad performance range that will provide ground-based mission operations systems support.

Op.

Op.

Provides continuity of base operations, including federal information processing resources of sustaining engineering, computer operations, and communications services for Kennedy Space Center.

Op.

Acquisition of seven classes of scientific and engineering workstations plus supporting equipment.

Op.

Furnishes, installs, and tests the Advanced Computer Generated Image System; provides direct computational analysis and programming support to specific research disciplines and flight projects; provides for the analysis, programming, engineering, and maintenance services for the flight simulation facilities. Also provides support for the Central Scientific and Computing Complex operation and systems maintenance as well as Complex-wide communications systems support and system administration of distributed computing and data reduction systems.

Op.

Provides a wide array of supporting services, including computational, professional, technical, administrative, engineering, and operations at the Lewis Research Center. (continued)

U.D.

U.D.

An information system including workstations, associated data processing, and communications, designed to integrate data from several National Weather Service information systems, as well as from field offices, regional and national centers, and other sources.

Op.

An initiative to acquire supercomputers necessary to run large complex numeric models as a key component of the weather forecast system.

Op.

A distributed-processing system architecture designed to acquire, process, and distribute satellite data and products.

Op.

An effort to replace a variety of obsolete technology in the National Marine Fisheries Service with a common computing infrastructure that supports distributed processing in an open system environment. The system stores, integrates, analyzes, and disseminates large quantities of living marine resource data.

Op.

Procurement of a high-performance computer system to provide support services for climate and weather research activities.

Geostationary Operational Environmental Satellite (GOES I-M)

Op.

Ground system consisting of minicomputers with associated peripherals and satellite-dependent customized applications software to provide the monitoring, supervision, and data acquisition and processing functions for the GOES-Next satellites.

Op.

A system designed to support weather radars and associated display systems. (continued)

Op.

An effort to replace old mainframes as well as the associated channel-connected architecture with an open systems architecture.

Op.

Ground system consisting of minicomputers with associated peripherals and satellite-dependent customized applications software intended to provide the monitoring, supervision, and data acquisition and processing functions for the polar satellites.

Imp.

A system of sensors, computers, display units, and communications equipment to automatically collect and process basic data on surface weather conditions, including temperature, pressure, wind, visibility, clouds, and precipitation.

Description of an Information Technology Investment Process Approach

This appendix is a compilation of work done by OMB and us on how federal agencies should manage information systems using an investment process. It is based upon analysis of the IT management best practices found in leading private and public sector organizations and is explained in greater detail in OMB’s Evaluating Information Technology Investments: A Practical Guide.

Select

How do you know you have selected the best projects?

Based on your evaluation, did the systems deliver what you expected?

Selection Phase: Choosing the Best IT Investments

Key Question: How can you select the right mix of IT projects that best meets mission needs and improvement priorities?

The goal of the selection phase is to assess and prioritize current and proposed IT projects and then create a portfolio of IT projects. In doing so, this phase helps ensure that the organization (1) selects those IT projects that will best support mission needs and (2) identifies and analyzes a project’s risks and returns before spending a significant amount of project funds. A critical element of this phase is that a group of senior executives makes project selection and prioritization decisions based on a consistent set of decision criteria that compares costs, benefits, risks, and potential returns of the various IT projects.

Steps of the Selection Phase

Initially filter and screen IT projects for explicit links to mission needs and program performance improvement targets using a standard set of decision criteria.

Analyze the most accurate and up-to-date cost, benefit, risk, and return information in detail for each project.

Create a ranked list of prioritized projects.

Determine the most appropriate mix of IT projects (new versus operational, strategic versus maintenance, etc.) to serve as the portfolio of IT investments.

Management Tools and Techniques Applicable to This Phase

An executive management team that makes funding decisions based on comparisons and trade-offs between competing project proposals, especially for those projects expected to have organizationwide impact.

A documented and defined set of decision criteria that examines expected return on investment (ROI), technical risks, improvement to program effectiveness, customer impact, and project size and scope.

Predefined dollar thresholds and authority levels that recognize the need to channel project evaluations and decisions to appropriate management levels to accommodate unit-specific versus agency-level needs.

Minimal acceptable ROI hurdle rates that apply to projets across the organization that must be met for projects to be considered for funding.

Risk assessments that expose potential technical and managerial weaknesses.

Control Phase: Manage the Investments by Monitoring for Results

Key Question: What controls are you using to ensure that the selected projects deliver the projected benefits at the right time and the right price?

Once the IT projects have been selected, senior executives periodically assess the progress of the projects against their projected cost, schedule, milestones, and expected mission benefits. The type and frequency of the reviews associated with this monitoring activity are usually based on the analysis of risk, complexity, and cost that went into selecting the project and that are performed at critical project milestones. If a project is late, over cost, or not meeting performance expectations, senior executives decide whether it should be continued, modified, or canceled.

Steps of the Control Phase

Use a set of performance measures to monitor the developmental progress for each IT project to identify problems.

Take action to correct discovered problems.

Management Tools and Techniques During This Phase

Established processes that involve senior managers in ongoing reviews and force decisive action steps to address problems early in the process.

Explicit cost, schedule, and performance measures to monitor expected versus actual project outcomes.

An information system to collect project cost, schedule, and performance data, in order to create a record of progress for each project. Incentives for exposing and solving project problems.

Evaluation Phase: Learn From the Process

Key Question: Based on your evaluation, did the system deliver what was expected?

The evaluation phase provides a mechanism for constantly improving the organization’s IT investment process. The goal of this phase is to measure, analyze, and record results, based on the data collected throughout each phase. Senior executives assess the degree to which each project met its planned cost and schedule goals and fulfilled its projected contribution to the organization’s mission. The primary tool in this phase is the postimplementation review (PIR), which should be conducted once a project has been completed. PIRs help senior managers assess whether a project’s proposed benefits were achieved and refine the IT selection criteria.

Steps of the Evaluation Phase

Compare actual project costs, benefits, risks, and return information against earlier projections. Determine the causes of any differences between planned and actual results.

For each system in operation, decide whether it should continue operating without adjustment, be further modified to improve performance, or be canceled.

Modify the organization’s investment process based on lessons learned.

Management Tools and Techniques During This Phase

Postimplementation reviews to determine actual costs, benefits, risks, and return.

Modification of decision criteria and investment management processes, based on lessons learned, to improve the process.

Maintenance of accountability by measuring actual project performance and creating incentives for even better project management in the future.

Brief Description of Agency IT Management Processes

The following sections briefly describe the information technology management processes at each of the five agencies we reviewed. These descriptions are intended to characterize the general workings of the agency processes at the time of our review. We used the selection/control/evaluation model (as summarized in appendix III and described in detail in OMB’s Evaluating Information Technology Investments: A Practical Guide) as a template for describing each agency’s IT management process.

Agency Selection Processes

Coast Guard

The Coast Guard had an IT investment process used to select IT projects for funding. IT project proposals were screened, evaluated, and ranked by a group of senior IRM managers using explicit decision criteria that took into account project costs, expected benefits, and risk assessments. The ranked list with recommended levels of funding for each project was submitted for review to a board of senior Coast Guard officers and then forwarded to the Coast Guard Chief of Staff for final approval.

Environmental Protection Agency

EPA used a decentralized IT project initiation, selection, and funding process. Under this broad process, program offices independently selected and funded IT projects on a case-by-case basis as the need for the system was identified. EPA had IRM policy and guidance for IT project data and analysis requirements—such as a project-level risk assessment and a cost-benefit study—that the program offices had to identify in order to proceed with system development. EPA did not have a consistent set of decision criteria for selecting IT projects.

Internal Revenue Service

IT selection and funding activities within IRS differed depending on whether the project was part of the Tax System Modernization (TSM) or an operational system. In 1995, IRS created a senior-level board for selecting, controlling, and evaluating information technology investments and began to rank all of the proposed TSM projects using its cost, risk, and return decision criteria. However, these criteria were largely qualitative, data used were not validated or reliable, and the analyses were not based on calculations of expected return on investment. According to IRS, its investment review board used a separate process with different criteria for evaluating operational systems. The board did not review research and development systems or field office systems. IRS did not compare the results of its different evaluation processes.

National Aeronautics and Space Administration

Within NASA, IT project selection and funding decisions were made by domain-specific program managers. NASA had two general types of IT funding—program expenditures and administrative spending. Most of NASA’s IT funding was embedded within program-specific budgets. Managers of these programs had autonomy to make system-level and system support IT selection decisions. Administrative IT systems were generally managed by the cognizant NASA program office or center.

NASA has recently established a CIO council to establish high-level policies and standards, approve information resources management plans, and address issues and initiatives. The council will also serve as the IT capital investment advisory group to the proposed NASA Capital Investment Council. NASA plans for this Capital Investment Council to have responsibility for looking at all capital investments across NASA, including those for IT. While this Capital Investment Council may fill the need for identifying cross-functional opportunities, it is not yet operational.

National Oceanic and Atmospheric Administration

IT project selection and funding decisions at NOAA were made as part of its strategic management and budgeting process. NOAA had seven work teams—each supporting a NOAA strategic goal—that prioritized incoming funding requests. Managers on these work teams negotiated to determine IT project funding priorities within the scope of their respective strategic goals. These prioritization requests were then submitted to NOAA’s Executive Management Board, which had final agency decision authority over all expenditures. A key decision criterion used by the work teams was the project’s contribution to the agency’s strategic goals; however, no standard set of decision criteria was used in the prioritization decisions. Other data, such as cost-benefit analyses, were also sometimes used to evaluate IT project proposals, although use of these data sources was not mandatory.

Agency Control Processes

Coast Guard

The Coast Guard conducted internal system reviews, but these reviews were not used to monitor the progress of IT projects. The review efforts were designed to address ways to improve efficiency, reduce project cost, and reduce project risk. Cost, benefit, and schedule data were also collected annually for some new IT projects, but the Coast Guard did not measure mission benefits derived from each of its projects.

Environmental Protection Agency

EPA had a decentralized managerial review process for monitoring IT projects. EPA’s IRM policy set requirements for the minimum level of review activity that program offices had to conduct, but program offices had primary responsibility for overseeing the progress of their IT projects. In an effort to provide a forum for senior managerial review of IT projects, EPA, in 1994, created the Executive Steering Committee (ESC) for IRM to guide EPA’s agencywide IRM activities. The ESC was chartered to review IRM projects that are large, important, or cross-organizational. The committee’s first major system review was scheduled for some time in 1996. EPA is currently formulating the data submission requirements for the ESC reviews.

Internal Revenue Service

IRS regularly conducted senior management program control meetings (PCM) to review the cost and schedule activity of TSM projects. IRS had two types of PCMs. The four TSM sites—Submission Processing, Computing Center, Customer Service, and District Office—conducted PCMs to monitor the TSM activity under their purview. Also, IRS could hold “combined PCMs” to resolve issues that spanned across the TSM sites. IRS did not conduct PCMs to monitor the performance of operational systems. To date, (1) working procedures, (2) required decision documents, (3) reliable cost, benefit, and return data, (4) and explicit quantitative decision criteria needed for an effective investment control process are not in place for the IRS Investment Review Board.

National Aeronautics and Space Administration

NASA senior executives regularly reviewed the cost and schedule performance of major programs and projects, but they reviewed only the largest IT projects. No central IRM review has been conducted since 1993.

NASA put senior-level CIOs in place for each NASA center, but these CIOs exercised limited control over mission-related systems and had limited authority to enforce IT standards or architecture policies. NASA’s proposed Capital Investment Council, which is intended to supplement the Program Management Council by reviewing major capital investments, may address this concern once the Investment Council is operational.

National Oceanic and Atmospheric Administration

NOAA conducted quarterly senior-level program status meetings to review the progress and performance of major systems and programs, such as those in the NWS modernization. NOAA had defined performance measures to gauge the progress toward its strategic goals, but did not have specific performance measures for individual IT systems. Also, while some offices had made limited comparisons of actual to expected IT project benefits, NOAA did not require the collection or assessment of mission benefit accrual information on IT projects.

Agency Evaluation Processes

Coast Guard

The Coast Guard did not conduct any postimplementation reviews of IT projects. Instead the Coast Guard focused its review activity on systems that were currently under development.

Environmental Protection Agency

EPA did not conduct any centralized postimplementation reviews. EPA did conduct postimplementation reviews as part of the General Services Administration’s (GSA) triennial review requirement, but curtailed this activity in 1992 when the GSA requirement was lifted.

Internal Revenue Service

IRS directives required that postimplementation reviews be conducted 6 months after an IT system is implemented. At the time of our review, IRS had conducted five postimplementation reviews and had developed a standard postimplementation review methodology. However, no mechanisms were in place to ensure that the results of these IRS investment evaluation reviews were used to modify the IRS selection and control decision-making processes or alter funding decisions for individual projects.

National Aeronautics and Space Administration

NASA did not conduct or require any centralized project postimplementation reviews. NASA stopped conducting centralized IRM reviews in 1993 and now instead urges programs to conduct IRM self-assessments.

National Oceanic and Atmospheric Administration

While the agency conducted other reviews, NOAA’s IRM office has participated in only four IRM reviews over the last 3 years. These reviews tended to focus on specific IT problems, such as evaluating the merits of electronic bulletin board systems or difficulties being encountered digitizing nautical navigation maps. No postimplementation reviews had been conducted over the past 3 years.

Summary of Investment-Related Provisions of the Information Technology Management Reform Act of 1996

On February 10, 1996, the Information Technology Management Reform Act of 1996 (Division E of Public Law 104-106) was signed into law. This appendix is a summary of the information technology investment-related provisions from this act, it is not the actual language contained in the law.

Information technology (IT) is defined as any equipment, or interconnected system or subsystem of equipment, that is used in the automatic acquisition, storage, manipulation, management, movement, control, display, switching, interchange, transmission, or reception of data or information. It may include equipment used by contractors.

The OMB Director is to promote and be responsible for improving the acquisition, use, and disposal of IT by federal agencies The OMB Director is to develop a process (as part of the budget process) for analyzing, tracking, and evaluating the risks and results of major capital investments for information systems; the process shall include explicit criteria for analyzing the projected and actual costs, benefits, and risks associated with the investments over the life of each system.

The OMB Director is to report to the Congress (at the same time the budget is submitted) on the net program performance benefits achieved by major capital investments in information systems and how the benefits relate to the accomplishment of agency goals.

The OMB Director shall designate (as appropriate) agency heads as executive agents to acquire IT for governmentwide use.

The OMB Director shall encourage agencies to develop and use “best practices” in acquiring IT.

The OMB Director shall direct that agency heads (1) establish effective and efficient capital planning processes for selecting, managing, and evaluating information systems investments, (2) before investing in new information systems, determine whether a government function should be performed by the private sector, the government, or government contractor, and (3) analyze their agencys’ missions and revise the mission-related and administrative processes (as appropriate) before making significant investments in IT.

Through the budget process, the OMB Director is to review selected agency IRM activities to determine the efficiency and effectiveness of IT investments in improving agency performance.

Agency heads are to design and implement a process for maximizing the value and assessing and managing the risks of IT investments.

The agency process is to (1) provide for the selection, management, and evaluation of IT investments, (2) be integrated with the processes for making budget, financial, and program management decisions, (3) include minimum criteria for selecting IT investments and specific quantitative and qualitative criteria for comparing and prioritizing projects, (4) provide for identifying potential IT investments that would result in shared benefits with other federal, state, or local governments, (5) provide for identifying quantifiable measurements for determining the net benefits and risks of IT investments, and (6) provide the means for senior agency managers to obtain timely development progress information, including a system of milestones for measuring progress, on an independently verifiable basis, in terms of cost, capability of the system to meet specified requirements, timeliness, and quality.

Agency heads are to ensure that performance measurements are prescribed for IT and that the performance measurements measure how well the IT supports agency programs. (continued)

Where comparable processes and organizations exist in either the public or private sectors, agency heads are to quantitatively benchmark agency process performance against such processes in terms of cost, speed, productivity, and quality of outputs and outcomes.

Agency heads may acquire IT as authorized by law (the Brooks Act—40 U. S. C. 759—is repealed by sec. 5101) except that the GSA Administrator will continue to manage the FTS 2000 and follow-on to that program (sec. 5124(b)).

Agency heads are to designate Chief Information Officers (in lieu of designating IRM officials—as a result of amending the Paperwork Reduction Act appointment provision).

Agency Chief Information Officers (CIOs) are responsible for (1) providing advice and assistance to agency heads and senior management to ensure that IT is acquired and information resources are managed in a manner that implements the policies and procedures of the Information Technology Management Reform Act of 1996, is consistent with the Paperwork Reduction Act, and is consistent with the priorities established by the agency head, (2) developing, maintaining, and facilitating the implementation of a sound and integrated agency IT architecture, and (3) promoting effective and efficient design and operation of major IRM processes.

Agency heads (in consultation with the CIO and CFO) are to establish policies and procedures that (1) ensure accounting, financial, and asset management systems and other information systems are designed, developed, maintained, and used effectively to provide financial or program performance data for agency financial statements, (2) ensure that financial and related program performance data are provided to agency financial management systems on a reliable, consistent, and timely basis, and (3) ensure that financial statements support the assessment and revision of agency mission-related and administrative processes and the measurement of performance of agency investments in information systems.

Agency heads are to identify (in their IRM plans required under the Paperwork Reduction Act) major IT acquisition programs that have significantly deviated from the cost, performance, or schedule goals established for the program (the goals are to be established under title V of the Federal Acquisition Streamlining Act of 1994).

This section establishes which provisions of the title apply to “national security systems.” “National security systems” are defined as any telecommunications or information system operated by the United States government that (1) involves intelligence activities, (2) involves cryptologic activities related to national security, (3) involves command and control of military forces, (4) involves equipment that is an integral part of a weapon or weapon system, or (5) is critical to the direct fulfillment of military or intelligence missions.

This section requires the GSA Administrator to provide (through the Federal Acquisition Computer Network established under the Federal Acquisition Streamlining Act of 1994 or another automated system) not later than January 1, 1998, governmentwide on-line computer access to information on products and services available for ordering under the multiple award schedules.

The Information Technology Management Reform Act takes effect 180 days from the date of enactment (February 10, 1996).

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

David McClure, Assistant Director Danny R. Latta, Adviser Alicia Wright, Senior Business Process Analyst Bill Dunahay, Senior Evaluator John Rehberger, Information Systems Analyst Shane Hartzler, Business Process Analyst Eugene Kudla, Staff Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.