Information Technology: Federal Agencies Need to Address Aging Legacy Systems
Released: 2016-05-25T00:00:00

Letter

Background

Over the last three decades, Congress has enacted several laws to assist agencies and the federal government in managing IT investments. For example, to assist agencies in managing their investments, Congress enacted the Clinger-Cohen Act of 1996. More recently, in December 2014, Congress enacted IT acquisition reform legislation (commonly referred to as the Federal Information Technology Acquisition Reform Act or FITARA) that, among other things, requires OMB to develop standardized performance metrics, including cost savings, and to submit quarterly reports to Congress on cost savings.

In carrying out its responsibilities, OMB uses several data collection mechanisms to oversee federal IT spending during the annual budget formulation process. Specifically, OMB requires federal departments and agencies to provide information related to their Major Business Cases (previously known as exhibit 300) and IT Portfolio Summary (previously known as exhibit 53).

OMB directs agencies to break down IT investment costs into two categories: (1) O&M and (2) development, modernization, and enhancement (DME). O&M (also known as steady-state) costs refer to the expenses required to operate and maintain an IT asset in a production environment. DME costs refers to those projects and activities that lead to new IT assets/systems, or change or modify existing IT assets to substantively improve capability or performance.

In addition, OMB has developed guidance that calls for agencies to develop an operational analysis policy for examining the ongoing performance of existing legacy IT investments to measure, among other things, whether the investment is continuing to meet business and customer needs.

Nevertheless, federal IT investments have too frequently failed or incurred cost overruns and schedule slippages while contributing little to mission-related outcomes. The federal government has spent billions of dollars on failed and poorly performing IT investments which often suffered from ineffective management, such as project planning, requirements definition, and program oversight and governance.

Accordingly, in February 2015, we introduced a new government-wide high-risk area, Improving the Management of IT Acquisitions and Operations. This area highlights several critical IT initiatives underway, including reviews of troubled projects, an emphasis on incremental development, a key transparency website, data center consolidation, and the O&M of legacy systems.

To make progress in this area, we identified actions that OMB and the agencies need to take. These include implementing the recently-enacted statutory requirements promoting IT acquisition reform, as well as implementing our previous recommendations. In the last 6 years, we made approximately 800 recommendations to OMB and multiple agencies to improve effective and efficient investment in IT. As of October 2015, about 32 percent of these recommendations had been implemented.

GAO Has Reported on the Need to Improve Oversight of Legacy IT

We have previously reported on legacy IT and the need for the federal government to improve its oversight of such investments. For example, in October 2012, we reported on agencies’ operational analyses policies and practices. In particular, we reported that although OMB guidance called for each agency to develop an operational analysis policy and perform such analyses annually, the extent to which the selected federal agencies we reviewed carried out these tasks varied significantly. The Departments of Defense (Defense), the Treasury (Treasury), and Veterans Affairs (VA) had not developed a policy or conducted operational analyses.

As such, we recommended that the agencies develop operational analysis policies, annually perform operational analyses on all investments, and ensure the assessments include all key factors. Further, we recommended that OMB revise its guidance to include directing agencies to post the results of such analyses on the IT Dashboard. OMB and the five selected agencies agreed with our recommendations and have efforts planned and underway to address them. In particular, OMB issued guidance in August 2012 directing agencies to report operational analysis results along with their fiscal year 2014 budget submission documentation (e.g., exhibit 300) to OMB. Thus far, operational analyses have not yet been posted on the IT Dashboard.

We further reported in November 2013 that agencies were not conducting proper analyses. Specifically, we reported on IT O&M investments and the use of operational analyses at selected agencies and determined that of the top 10 investments with the largest spending in O&M, only a Department of Homeland Security (DHS) investment underwent an operational analysis. DHS’s analysis addressed most, but not all, of the factors that OMB called for (e.g., comparing current cost and schedule against original estimates). The remaining agencies did not assess their investments, which accounted for $7.4 billion in reported O&M spending. Consequently, we recommended that seven agencies perform operational analyses on their IT O&M investments and that DHS ensure that its analysis was complete and addressed all OMB factors. Three of the agencies agreed with our recommendations; two partially agreed; and two agencies had no comments.

Government-wide Spending on IT Operations and Maintenance Is Increasing

As discussed in our report, federal agencies reported spending the majority of their fiscal year 2015 IT funds on operating and maintaining a large number of legacy (i.e., steady-state) investments. Of the more than $80 billion reportedly spent on federal IT in fiscal year 2015, 26 federal agencies spent about $61 billion on O&M, more than three-quarters of the total amount spent. Specifically, data from the IT Dashboard shows that, in 2015, 5,233 of the government’s nearly 7,000 IT investments were spending all of their funds on O&M activities. This is a little more than three times the amount spent on DME activities (see figure 1).

According to agency data reported to OMB’s IT Dashboard, the 10 IT investments spending the most on O&M for fiscal year 2015 total $12.5 billion, 20 percent of the total O&M spending, and range from $4.4 billion on Department of Health and Human Services’ (HHS) Centers for Medicare and Medicaid Services’ Medicaid Management Information System to $666.1 million on HHS’s Centers for Medicare and Medicaid Services IT Infrastructure investment (see table 1).

Spending on O&M Has Increased over 7 Years

Over the past 7 fiscal years, O&M spending has increased, while the amount invested in developing new systems has decreased by about $7.3 billion since fiscal year 2010. (See figure 2.)

Further, agencies have increased the amount of O&M spending relative to their overall IT spending by 9 percent since 2010. Specifically, in fiscal year 2010, O&M spending was 68 percent of the federal IT budget, while in fiscal year 2017, agencies plan to spend 77 percent of their IT funds on O&M. (See figure 3.)

Further, 15 of the 26 agencies have increased their spending on O&M from fiscal year 2010 to fiscal year 2015, with 10 of these agencies having over a $100 million increase. The spending changes per agency range from an approximately $4 billion increase (HHS) to a decrease of $600 million (National Aeronautics and Space Administration).

OMB staff in the Office of E-Government and Information Technology have recognized the upward trend of IT O&M spending and identified several contributing factors, including (1) the support of O&M activities requires maintaining legacy hardware, which costs more over time, and (2) costs are increased in maintaining applications and systems that use older programming languages, since programmers knowledgeable in these older languages are becoming increasingly rare and thus more expensive. Further, OMB officials stated that in several situations where agencies are not sure whether to report costs as O&M or DME, agencies default to reporting as O&M. According to OMB, agencies tend to categorize investments as O&M because they attract less oversight, require reduced documentation, and have a lower risk of losing funding.

Many O&M Investments Were at Risk and Lacked Proper Oversight

According to OMB guidance, the O&M phase is often the longest phase of an investment and can consume more than 80 percent of the total lifecycle costs. As such, agencies must actively manage their investment during this phase. To help them do so, OMB requires that CIOs submit ratings that reflect the level of risk facing an investment.

In addition, in instances where investments experience problems, agencies can perform a TechStat, a face-to-face meeting to terminate or turn around IT investments that are failing or not producing results. In addition, OMB directs agencies to monitor O&M investments through operational analyses, which should be performed annually and assess costs, schedules, whether the investment is still meeting customer and business needs, and investment performance.

Several O&M investments were rated as moderate to high risk in fiscal year 2015. Specifically, CIOs from the 12 selected agencies reported that 23 of their 187 major IT O&M investments were moderate to high risk as of August 2015. They requested $922.9 million in fiscal year 2016 for these investments. Of the 23 investments, agencies had plans to replace or modernize 19 investments. However, the plans for 12 of those were general or tentative in that the agencies did not provide specificity on time frames, activities to be performed, or functions to be replaced or enhanced. Further, agencies did not plan to modernize or replace 4 of the investments (see table 2). The lack of specific plans to modernize or replace these investments could result in wasteful spending on moderate and high-risk investments.

While agencies generally conducted the required operational analyses, they did not consistently perform TechStat reviews on all of the at-risk investments. Although not required, agencies had performed TechStats on only five of the 23 at-risk investments. In addition, operational analyses were not conducted for four of these investments (see table 3).

Agencies provided several reasons for not conducting TechStats and required assessments. For example, according to agency officials, several of the investments’ risk levels were reduced to low or moderately low risk in the months since the IT Dashboard had been publicly updated. Regarding assessments, one official stated that, in place of operational analyses, the responsible bureau reviews the status of the previous month’s activities for the development, integration, modification, and procurement to report issues to management. However, this monthly process does not include all of the key elements of an operational analysis. Until agencies ensure that their O&M investments are fully reviewed, the government’s oversight of old and vulnerable investments will be impaired and the associated spending could be wasteful.

IT Investments Are Becoming Obsolete and Agencies Are Not Required to Identify Investments That Need Attention

Legacy IT investments across the federal government are becoming increasingly obsolete. Specifically, many use outdated languages and old parts. Numerous old investments are using obsolete programming languages. Several agencies, such as the Department of Agriculture (USDA), DHS, HHS, Justice, Treasury, and VA, reported using Common Business Oriented Language (COBOL)—a programming language developed in the late 1950s and early 1960s—to program their legacy systems. It is widely known that agencies need to move to more modern, maintainable languages, as appropriate and feasible. For example, the Gartner Group, a leading IT research and advisory company, has reported that organizations using COBOL should consider replacing the language and in 2010 noted that there should be a shift in focus to using more modern languages for new products.

In addition, some legacy systems may use parts that are obsolete and more difficult to find. For instance, Defense is still using 8-inch floppy disks in a legacy system that coordinates the operational functions of the United States’ nuclear forces. (See figure 4.)

Further, in some cases, the vendors no longer provide support for hardware or software, creating security vulnerabilities and additional costs. For example, each of the 12 selected agencies reported using unsupported operating systems and components in their fiscal year 2014 reports pursuant to the Federal Information Security Management Act of 2002. Commerce, Defense, Treasury, HHS, and VA reported using 1980s and 1990s Microsoft operating systems that stopped being supported by the vendor more than a decade ago.

Lastly, legacy systems may become increasingly more expensive as agencies have to deal with the previously mentioned issues and may pay a premium to hire staff or contractors with the knowledge to maintain outdated systems. For example, one agency (SSA) reported re-hiring retired employees to maintain its COBOL systems.

Selected agencies reported that they continue to maintain old investments in O&M. For example, Treasury reported systems that were about 56 years old.

Table 4 shows the 10 oldest investments and/or systems, as reported by selected agencies. Agencies reported having plans to modernize or replace each of these investments and systems. However, the plans for five of those were general or tentative in that the agencies did not provide specific time frames, activities to be performed, or functions to be replaced or enhanced.

Separately, in our related report, we profiled one system or investment from each of the 12 selected agencies. The selected systems and investments range from 11 to approximately 56 years old, and serve a variety of purposes. Of the 12 investments or systems, agencies had plans to replace or modernize 11 of these. However, the plans for 3 of those were general or tentative in that the agencies did not provide specificity on time frames, activities to be performed, or functions to be replaced or enhanced. Further, there were no plans to replace or modernize 1 investment.

We have previously provided guidance that organizations should periodically identify, evaluate, and prioritize their investments, including those that are in O&M; at, near, or exceeding their planned life cycles; and/or are based on technology that is now obsolete, to determine whether the investment should be kept as-is, modernized, replaced, or retired. This critical process allows the agency to identify and address high-cost or low-value investments in need of update, replacement, or retirement.

Agencies are, in part, maintaining obsolete investments because they are not required to identify, evaluate, and prioritize their O&M investments to determine whether they should be kept as-is, modernized, replaced, or retired. According to OMB staff from the Office of E-Government and Information Technology, OMB has created draft guidance that will require agencies to identify and prioritize legacy information systems that are in need of replacement or modernization. Specifically, the guidance is intended to develop criteria through which agencies can identify the highest priority legacy systems, evaluate and prioritize their portfolio of existing IT systems, and develop modernization plans that will guide agencies’ efforts to streamline and improve their IT systems. The draft guidance includes time frames for the efforts regarding developing criteria, identifying and prioritizing systems, and planning for modernization. However, OMB did not commit to a firm time frame for when the policy would be issued. Until this policy is finalized and carried out, the federal government runs the risk of continuing to maintain investments that have outlived their effectiveness and are consuming resources that outweigh their benefits.

Regarding upgrading obsolete investments, in April 2016, the IT Modernization Act was introduced into the U.S. House of Representatives. If enacted, it would establish a revolving fund of $3 billion that could be used to retire, replace, or upgrade legacy IT systems to transition to new, more secure, efficient, modern IT systems. It also would establish processes to evaluate proposals for modernization submitted by agencies and monitor progress and performance in executing approved projects.

Implementation of Our Recommendations Should Allow Federal Agencies to Better Manage Legacy Systems and Investments

Our report that is being released today contains 2 recommendations to OMB and 14 to selected federal agencies. Among other things, we recommend that the Director of OMB commit to a firm date by which its draft guidance on legacy systems will be issued, and subsequently direct agencies to identify legacy systems and/or investments needing to be modernized or replaced and that the selected agency heads direct their respective agency CIOs to identify and plan to modernize or replace legacy systems as needed and consistent with OMB’s draft guidance. If agencies implement our recommendations, they will be positioned to better manage legacy systems and investments.

In commenting on a draft of the report, eight agencies (USDA, Commerce, HHS, DHS, State, Transportation, VA, and SSA) and OMB agreed with our recommendations. Defense and Energy partially agreed with our recommendation. Defense stated that it planned to continue to identify, prioritize, and manage legacy systems, based on existing department policies and processes, and consistent to the extent practicable with OMB’s draft guidance. Energy stated that while the department continues to take steps to modernize its legacy investments and systems, it could not agree fully with our recommendation because OMB’s guidance is in draft and the department has not had an opportunity to review it. Defense and Energy’s comments are consistent with the intent of our recommendation. Upon finalization of OMB’s guidance, we encourage both agencies to implement OMB’s guidance. In addition, Justice and the Treasury stated that they had no comment on their recommendations.

In summary, O&M spending has steadily increased over the past 7 years and as a result, key agencies are devoting a smaller amount of IT spending to DME activities. Further, legacy federal IT investments are becoming obsolete and several aging investments are using unsupported components, many of which did not have specific plans for modernization or replacement. This O&M spending has steadily increased and as a result, key agencies are devoting a smaller amount of IT spending to DME activities. To its credit, OMB has developed a draft initiative that calls for agencies to analyze and review O&M investments. However, it has not finalized its policy. Until it does so, the federal government runs the risk of continuing to maintain investments that have outlived their effectiveness and are consuming resources that outweigh their benefits.

Chairman Chaffetz, Ranking Member Cummings, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time.

GAO Contact and Staff Acknowledgments

If you have any questions on matters discussed in this testimony, please contact David A. Powner at (202) 512-9286 or at pownerd@gao.gov. Other key contributors include Gary Mountjoy (assistant director), Kevin Walsh (assistant director), Jessica Waselkow (analyst in charge), Scott Borre, Rebecca Eyler, and Tina Torabi.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.