Information Technology: Federal Agencies Need to Address Aging Legacy Systems
Published: 2016-05-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government invests more than $80 billion on IT annually, with much of this amount reportedly spent on operating and maintaining existing (legacy) IT systems. Given the magnitude of these investments, it is important that agencies effectively manage their O&M.
GAO's objectives were to (1) assess federal agencies' IT O&M spending, (2) evaluate the oversight of at-risk legacy investments, and (3) assess the age and obsolescence of federal IT.
To do so, GAO reviewed OMB and 26 agencies' IT O&M spending for fiscal years 2010 through 2017. GAO further reviewed the 12 agencies that reported the highest planned IT spending for fiscal year 2015 to provide specifics on agency spending and individual investments.
section_title: What GAO Found
paragraphs: The federal government spent about 75 percent of the total amount budgeted for information technology (IT) for fiscal year 2015 on operations and maintenance (O&M) investments. Such spending has increased over the past 7 fiscal years, which has resulted in a $7.3 billion decline from fiscal years 2010 to 2017 in development, modernization, and enhancement activities.
Specifically, 5,233 of the government's approximately 7,000 IT investments are spending all of their funds on O&M activities. Moreover, the Office of Management and Budget (OMB) has directed agencies to identify IT O&M expenditures known as non-provisioned services that do not use solutions often viewed as more efficient, such as cloud computing and shared services. Agencies reported planned spending of nearly $55 billion on such non-provisioned IT in fiscal year 2015. OMB has developed a metric for agencies to measure their spending on services such as cloud computing and shared services, but has not identified an associated goal. Thus, agencies may be limited in their ability to evaluate progress.
Many O&M investments in GAO's review were identified as moderate to high risk by agency CIOs, and agencies did not consistently perform required analysis of these at-risk investments. Further, several of the at-risk investments did not have plans to be retired or modernized. Until agencies fully review their at-risk investments, the government's oversight of such investments will be limited and its spending could be wasteful.
Federal legacy IT investments are becoming increasingly obsolete: many use outdated software languages and hardware parts that are unsupported. Agencies reported using several systems that have components that are, in some cases, at least 50 years old. For example, Department of Defense uses 8-inch floppy disks in a legacy system that coordinates the operational functions of the nation's nuclear forces. In addition, Department of the Treasury uses assembly language code—a computer language initially used in the 1950s and typically tied to the hardware for which it was developed. OMB recently began an initiative to modernize, retire, and replace the federal government's legacy IT systems. As part of this, OMB drafted guidance requiring agencies to identify, prioritize, and plan to modernize legacy systems. However, until this policy is finalized and fully executed, the government runs the risk of maintaining systems that have outlived their effectiveness. The following table provides examples of legacy systems across the federal government that agencies report are 30 years or older and use obsolete software or hardware, and identifies those that do not have specific plans with time frames to modernize or replace these investments.
section_title: What GAO Recommends
paragraphs: GAO is making 16 recommendations, one of which is for OMB to develop a goal for its spending measure and finalize draft guidance to identify and prioritize legacy IT needing to be modernized or replaced. GAO is also recommending that selected agencies address at-risk and obsolete legacy O&M investments. Nine agencies agreed with GAO's recommendations, two agencies partially agreed, and two agencies stated they had no comment. The two agencies that partially agreed, Defense and Energy, outlined plans that were consistent with the intent of our recommendations.

section_title: Letter
paragraphs: The federal government spends more than $80 billion annually on  information technology (IT), with about 75 percent reportedly spent on  operating and maintaining existing (legacy) IT systems. Given the size  and magnitude of these investments, it is important that agencies  effectively manage the operations and maintenance (O&M) of existing  investments.
Our objectives were to (1) assess federal agencies’ IT O&M spending, (2)  evaluate the oversight of at-risk legacy investments, and (3) assess the  age and obsolescence of federal IT.
Our review of O&M spending included the Office of Management and  Budget (OMB) and the 26 agencies that report to OMB’s IT Dashboard.  For specific information on individual systems or investments, we focused  on the 12 agencies that reported the highest planned IT spending for  fiscal year 2015, given that these agencies make up over 90 percent of  reported federal IT spending.
To assess federal agencies’ IT O&M spending, we reviewed data  reported to OMB as part of the budget process for fiscal years 2010  through 2017. We analyzed that data to determine whether spending had  changed over those years and compared OMB’s associated performance  measure to federal best practices.
We evaluated the extent to which the 12 selected federal agencies are  performing oversight on their existing legacy investments by reviewing  agency IT Dashboard data to identify investments in O&M that had been  identified as being moderate to high risk. We also reviewed agency  documentation such as TechStat documentation and operational  analyses, as available.
To assess the age and obsolescence of federal IT, we reviewed agency  documentation, such as operational analyses and enterprise architecture  documents, and interviewed agency officials on issues related to legacy  investments. We also requested that the 12 agencies provide a list of  their three oldest systems. In some cases, agencies reported that they do  not track the ages of individual systems. In those cases, we requested  that the agency provide their three oldest IT investments. We also  compared OMB and agencies’ current practices with federal guidance,  such as OMB’s Circular No. A-11: Preparation, Submission, and  Execution of the Budget and its associated supplement on capital assets,  to determine whether OMB and agencies are adequately managing the  age and obsolescence of federal IT. In addition, we profiled selected  systems and investments. To select those, we selected a system or  investment that was identified as one of the agency’s oldest or had been  identified as being at-risk. In particular, we selected one system or  investment per agency using factors such as investment type (major or  non-major), system or investment age, and risk level. We reviewed  agency documentation and interviewed agency officials on the profiled  systems or investments.
To assess the reliability of the OMB budget data and IT Dashboard data,  we reviewed related documentation, such as OMB guidance on budget  preparation, capital planning, and IT Dashboard submissions. In addition,  we corroborated with each agency that the data downloaded were  accurate and reflected the data it had reported to OMB. We determined  that the data were reliable for the purposes of our reporting objectives.  Details of our objectives, scope, and methodology are contained in  appendix I.
We conducted this performance audit from April 2015 to May 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Over the last three decades, Congress has enacted several laws to assist  agencies and the federal government in managing IT investments. For  example, to assist agencies in managing their investments, Congress  enacted the Clinger-Cohen Act of 1996. This act requires OMB to  establish processes to analyze, track, and evaluate the risks and results  of major capital investments in information systems made by federal  agencies and report to Congress on the net program performance  benefits achieved as a result of these investments. Most recently, in  December 2014, Congress enacted IT acquisition reform legislation  (commonly referred to as the Federal Information Technology Acquisition  Reform Act or FITARA) that, among other things, requires OMB to  develop standardized performance metrics, including cost savings, and to  submit quarterly reports to Congress on cost savings.
In carrying out its responsibilities, OMB uses several data collection  mechanisms to oversee federal IT spending during the annual budget  formulation process. Specifically, OMB requires federal departments and  agencies to provide information related to their Major IT Business Cases  (previously known as exhibit 300) and IT Portfolio Summary (previously  known as exhibit 53).
Major IT Business Case. The purpose of this requirement is to  provide a business case for each major IT investment and to allow  OMB to monitor IT investments once they are funded. Agencies are  required to provide information on each major investment’s cost,  schedule, and performance.
IT Portfolio Summary. The purpose of the IT portfolio summary is to  identify all IT investments—both major and non-major —and their  associated costs within a federal organization. This information is  designed, in part, to help OMB better understand what agencies are  spending on IT investments.
OMB directs agencies to break down IT investment costs into two  categories: (1) O&M and (2) development, modernization, and  enhancement (DME). O&M (also known as steady state) costs refer to  the expenses required to operate and maintain an IT asset in a  production environment. DME costs refers to those projects and activities  that lead to new IT assets/systems, or change or modify existing IT  assets to substantively improve capability or performance.
Beginning in 2014, OMB directed agencies to further break down their  O&M and DME costs to identify provisioned IT service costs. A  provisioned IT service is one that is (1) owned, operated, and provided by  an outside vendor or external government organization and (2) consumed  by the agency on an as-needed basis. Examples of provisioned IT service  could include cloud services or shared services from another federal  agency or a private service provider. About 8.5 percent of federal  agencies’ planned spending for fiscal year 2016 has gone toward  provisioned IT services, leaving the vast majority of spending going  toward IT that is non-provisioned. Figure 1 shows the breakdown in  planned spending for fiscal year 2016.
Further, OMB has developed guidance that calls for agencies to develop  an operational analysis policy for examining the ongoing performance of  existing legacy IT investments to measure, among other things, whether  the investment is continuing to meet business and customer needs. This  guidance calls for the policy to provide for an annual operational analysis  of each investment that addresses cost, schedule, customer satisfaction,  strategic and business results, financial goals, and innovation.
Nevertheless, federal IT investments have too frequently failed or  incurred cost overruns and schedule slippages while contributing little to  mission-related outcomes. The federal government has spent billions of  dollars on failed and poorly performing IT investments which often  suffered from ineffective management, such as project planning,  requirements definition, and program oversight and governance.
Accordingly, in February 2015, we introduced a new government-wide  high-risk area, Improving the Management of IT Acquisitions and  Operations. This area highlights several critical IT initiatives underway,  including reviews of troubled projects, an emphasis on incremental  development, a key transparency website, data center consolidation, and  the O&M of legacy systems.
To make progress in this area, we identified actions that OMB and the  agencies need to take. These include implementing the recently-enacted  statutory requirements promoting IT acquisition reform, as well as  implementing our previous recommendations. In the last 6 years, we  made approximately 800 recommendations to OMB and multiple  agencies to improve effective and efficient investment in IT. As of October  2015, about 32 percent of these recommendations had been  implemented.
subsections: section_title: OMB’s Recent Major Initiatives for Overseeing IT Investments
paragraphs: OMB has implemented a series of initiatives to improve the oversight of  underperforming investments and more effectively manage IT. These  efforts include the following:  IT Dashboard. In June 2009, to further improve the transparency into  and oversight of agencies’ IT investments, OMB publicly deployed the  IT Dashboard. As part of this effort, OMB issued guidance directing  federal agencies to report, via the Dashboard, the performance of  their IT investments. Currently, the Dashboard publicly displays  information on the cost, schedule, and performance of over 700 major  federal IT investments at 26 federal agencies. Further, the public  display of these data is intended to allow OMB, other oversight  bodies, and the general public to hold the government agencies  accountable for results and progress. Among other things, agencies  are to submit ratings from their Chief Information Officers (CIO),  which, according to OMB’s instructions, should reflect the level of risk  facing an investment relative to that investment’s ability to accomplish  its goals. To do so, each agency CIO is to assess his or her IT  investments against a set of six pre-established evaluation factors  identified by OMB and then assign a rating of 1 (high risk and red) to 5  (low risk and green) based on the CIO’s best judgement of the level of  risk facing the investment. Over the past several years, we have  made over 20 recommendations to help improve the accuracy and  reliability of the information on the IT Dashboard and to increase its  availability. Most agencies agreed with our recommendations or had  no comment.
TechStat reviews. In January 2010, the Federal CIO began leading  TechStat sessions—face-to-face meetings to terminate or turn around  IT investments that are failing or are not producing results. These  meetings involve OMB and agency leadership and are intended to  increase accountability and improve performance. OMB also  empowered agency CIOs to begin to hold their own TechStat  sessions within their respective agencies by June 2012. In June 2013,  we reported that OMB and selected agencies held multiple TechStats,  but additional OMB oversight was needed to ensure that these  meetings were having the appropriate impact on underperforming  projects and that resulting cost savings were valid. Among other  things, we recommended that OMB require agencies to address high- risk investments. OMB generally agreed with this recommendation.  However, as of October 28, 2015, OMB had only conducted one  TechStat review in the prior 2 years and OMB had not listed any  savings from TechStats in its quarterly reporting to Congress since  June 2012.
Cloud computing strategy. In order to accelerate the adoption of  cloud computing solutions across the government, OMB’s 25-Point IT  Reform Plan included a “Cloud First” policy that required each agency  CIO to, among other things, implement cloud-based solutions  whenever a secure, reliable, and cost-effective cloud option exists.  Building on this requirement, in February 2011, OMB issued the  Federal Cloud Computing Strategy, which provided definitions of  cloud computing services; benefits of cloud services, such as  accelerating data center consolidations; case studies to support  agencies’ migration to cloud computing; and roles and responsibilities  for federal agencies. In April 2016, we reported, among other  things, that we had identified 10 key practices that if included in cloud  service agreements can help agencies ensure services are performed  effectively, efficiently, and securely. OMB’s guidance, released in  February 2012, included most of the key practices, and we  recommended that OMB include all 10 key practices in future  guidance.
PortfolioStat reviews. To better manage existing IT systems, OMB  launched the PortfolioStat initiative in March 2012, which requires  agencies to conduct an annual, agency-wide IT portfolio review to,  among other things, reduce commodity IT spending and  demonstrate how their IT investments align with the agency’s mission  and business functions. In 2013 and 2015 we reported that agencies  had the potential to save at least $3.8 billion through this initiative.  However, we noted that weaknesses existed in agencies’  implementation of the initiative; therefore, we made more than 60  recommendations to OMB and agencies. OMB partially agreed with  our recommendations, and responses from 21 of the agencies varied,  with some agreeing and others not.
IT Shared Services Strategy. In May 2012, OMB released its  Federal IT Shared Services Strategy. The strategy requires  agencies to use shared services—IT functions that are provided for  consumption by multiple organizations within or between federal  agencies—for IT service delivery in order to increase return on  investment, eliminate waste and duplication, and improve the  effectiveness of IT solutions. Examples of commodity IT areas to  consider migrating to a shared environment, as described in the  strategy, include software licenses, e-mail systems, and human  resource systems.
section_title: GAO Has Reported on the Need to Improve Oversight of Legacy IT
paragraphs: We have previously reported on legacy IT and the need for the federal  government to improve its oversight of such investments. For example, in  October 2012, we reported on agencies’ operational analyses policies  and practices. As previously mentioned, operational analysis is a key  performance evaluation and oversight mechanism required by OMB to  ensure O&M investments continue to meet agency needs. In particular,  we reported that although OMB guidance called for agencies to develop  an operational analysis policy and perform such analyses annually, the  extent to which the selected five federal agencies we reviewed carried out  these tasks varied significantly. Specifically, the Departments of  Homeland Security (DHS) and Health and Human Services (HHS)  developed policies and conducted analyses, but excluded key  investments and assessment factors. The Departments of Defense  (Defense), the Treasury (Treasury), and Veterans Affairs (VA) had not  developed a policy or conducted operational analyses. As such, we  recommended that the agencies develop operational analysis policies,  annually perform operational analyses on all investments, and ensure the  assessments include all key factors. Further, we recommended that OMB  revise its guidance to include directing agencies to post the results of  such analyses on the IT Dashboard. OMB and the five selected agencies  agreed with our recommendations and have efforts planned and  underway to address them. In particular, OMB issued guidance in August  2012 directing agencies to report operational analysis results along with  their fiscal year 2014 budget submission documentation (e.g., exhibit 300)  to OMB. Thus far, operational analyses have not yet been posted on the  IT Dashboard.
We further reported in November 2013 that agencies were not conducting  proper analyses. Specifically, we reported on IT O&M investments and  the use of operational analyses at selected agencies and determined that  of the top 10 investments with the largest spending in O&M, only the DHS  investment underwent an operational analysis. DHS’s analysis addressed  most, but not all, of the factors that OMB called for (e.g., comparing  current cost and schedule against original estimates). DHS officials  attributed this to the department still being in the process of implementing  its new operational analysis policy. The remaining agencies did not  assess their investments, which accounted for $7.4 billion in reported  O&M spending. Agency officials cited several reasons for not doing so,  including relying on budget submission and related management reviews  that measure performance; however, OMB has noted that these are not a  substitute for an operational analysis. Consequently, we recommended  that seven agencies perform operational analyses on their IT O&M  investments and that DHS ensure that its analysis was complete and  addressed all OMB factors. Three of the agencies agreed with our  recommendations; two partially agreed; and two agencies had no  comments.
section_title: Government-wide Spending on IT Operations and Maintenance Is Increasing
paragraphs: Federal agencies reported spending the majority of their fiscal year 2015  IT funds on operating and maintaining a large number of legacy (i.e.,  steady-state) investments. Of the more than $80 billion reportedly spent  on federal IT in fiscal year 2015, 26 federal agencies spent about $61  billion on O&M, more than three-quarters of the total amount spent.  Specifically, data from the IT Dashboard shows that, in 2015, 5,233 of the  government’s nearly 7,000 investments were spending all of their funds  on O&M activities. This is a little more than three times the amount spent  on DME activities (See figure 2).
According to agency data reported to OMB’s IT Dashboard, the 10 IT  investments spending the most on O&M for fiscal year 2015 total $12.5  billion, 20 percent of the total O&M spending, and range from $4.4 billion  on the Department of Health and Human Services’ (HHS) Centers for  Medicare and Medicaid Services’ Medicaid Management Information  System to $666.1 million on HHS’s Centers for Medicare and Medicaid  Services IT Infrastructure investment (see table 1).
subsections: section_title: Spending on O&M Has Increased over 7 Years
paragraphs: Over the past 7 fiscal years, O&M spending has increased, while the  amount invested in developing new systems has decreased by about  $7.3 billion since fiscal year 2010. (See figure 3.)
Further, agencies have increased the amount of O&M spending relative  to their overall IT spending by 9 percent since 2010. Specifically, in fiscal  year 2010, O&M spending was 68 percent of the federal IT budget, while  in fiscal year 2017, agencies plan to spend 77 percent of their IT funds on  O&M. (See figure 4.)
Further, 15 of the 26 agencies have increased their spending on O&M  from fiscal year 2010 to fiscal year 2015, with 10 of these agencies  having over a $100 million increase. The spending changes per agency  range from an approximately $4 billion increase (HHS) to a decrease of  $600 million (National Aeronautics and Space Administration). See table  2 for more details on agency spending.
In addition, 20 of the 26 agencies have increased the percentage of total  IT spending on O&M from fiscal year 2010 to fiscal year 2015, with 13  agencies having an increase of over 5 percent. The percentage of total IT  spending on O&M ranges from a 20 percent increase (Department of  Education) to a 16 percent decrease (Office of Personnel Management).  Appendix II provides detailed information on agency spending on  operations and maintenance from fiscal year 2010 to fiscal year 2015.
According to agency officials, reasons for the increase in O&M spending  include the recent shift of major systems from DME to O&M (as the  investment completed development activities and began O&M activities);  and rising costs to maintain legacy IT infrastructure, such as those that  use older programming languages. They also noted that improved  reporting (i.e., ensuring that O&M expenditures were properly reported as  O&M instead of as DME) has made it appear that O&M spending has  increased.
For example, a DHS official in the Office of the CIO stated that one  reason for the increased spending on O&M as a percentage of its total is  because initially DHS had high DME spending to setup the agency, but  now that the major parts of the agency are established, the funding has  shifted to O&M. DHS officials stated that they anticipate future increases  in DME funding as prioritized IT modernization efforts are approved and  funded. Further, an official in Department of State’s (State) Bureau of  Information Resource Management stated that the increase is largely due  to increased costs of maintaining the infrastructure, including meeting  security requirements. Moreover, VA officials stated that updates to its  technology are the primary reason for the increase in spending. In  addition, an official in HHS’s Office of the CIO stated that the increased  spending on O&M was largely due to grants to states and local entities for  new programs, such as the Affordable Care Act.
Conversely, several agencies have decreased spending on O&M. For  example, as we have previously reported, the Department of Energy  (Energy) reduced spending by approximately $300 million, which it  attributed to the reclassification of high performance computers from the  IT portfolio to facilities. According to Energy officials, these investments  were re-categorized because they include both supercomputers and  laboratory facilities. Similarly, the Department of Commerce  (Commerce) reduced spending by approximately $110 million and  attributed it to the reclassification of satellite ground systems from its IT  portfolio. In making this decision, Commerce determined that it needed to  refocus oversight efforts to a more appropriate level and consequently  minimized the role of the CIO and others in the oversight of satellites. We  disagreed with these reclassifications, and reported that they run contrary  to the Clinger-Cohen Act of 1996, which specifies requirements for the  management of IT. Further, we reported that by gathering incomplete  information on IT investments, OMB increases the risk of not fulfilling its  oversight responsibilities, of agencies making inefficient and ineffective  investment decisions, and of Congress and the public being misinformed  as to the performance of federal IT investments. We recommended that  Energy and Commerce appropriately categorize their IT investments, but  both agencies disagreed.
A policy analyst within OMB’s Office of E-Government and Information  Technology expressed concern when agencies, or their bureaus, spend a  low percentage of their IT funds on DME. The analyst further stated that  this could indicate that the agency’s maintenance costs are reducing its  flexibility and the agency or bureau is unable to innovate. For example, 5  of the 26 agencies that report to the IT Dashboard reported spending less  than 10 percent on DME activities in fiscal year 2015 (see table 3).
Further, 34 percent of bureaus (i.e., 51 of the 151) spent less than 10  percent on DME. For more details on the bureaus spending less than 10  percent on DME activities, see appendix III.
According to agency officials, reasons for these bureaus’ low spending on  DME include the size and mission of the bureau (e.g., smaller bureaus do  not perform much DME work), as well as several bureaus having recently  completed major DME work that is now in the O&M phase. Further,  according to Commerce officials, one of their bureaus had no actual IT  systems in its budget, as its IT has been absorbed by headquarters, and  thus any DME spending is part of the Office of IT Services’ budget.
OMB staff in the Office of E-Government and Information Technology  have recognized the upward trend of O&M spending and identified  several contributing factors, including (1) the support of O&M activities  requires maintaining legacy hardware, which costs more over time, and  (2) costs are increased in maintaining applications and systems that use  older programming languages, since programmers knowledgeable in  these older languages are becoming increasingly rare and thus more  expensive. Further, OMB officials stated that in several situations where  agencies are not sure whether to report costs as O&M or DME, agencies  default to reporting as O&M. According to OMB, agencies tend to  categorize investments as O&M because they attract less oversight,  require reduced documentation, and have a lower risk of losing funding.
section_title: Less than a Quarter of Federal IT Spending Is Categorized as Provisioned
paragraphs: OMB encourages agencies to adopt provisioned IT services, such as  cloud computing and shared services, to make IT more efficient and agile,  and enable innovation. Specifically, it provides an approach for  agencies to implement cloud-based solutions whenever a secure,  reliable, and cost-effective cloud option exists and to use shared services  for IT service delivery in order to increase return on investment, eliminate  waste and duplication, and improve the effectiveness of IT solutions.  Further, as part of its guidance on the implementation of recent IT  legislation, OMB identified a series of performance metrics for agencies’  PortfolioStat sessions to measure the federal government’s progress in  driving value in federal IT investments. One measure is the percent of IT  spending on non-provisioned O&M spending. An OMB official stated that  focusing on the O&M spending that has not been provisioned will allow  OMB to identify legacy systems in need of modernization.
Federal agencies reported spending $55 billion—69 percent of total IT  spending—on non-provisioned O&M in fiscal year 2015, with the percent  allocated to non-provisioned O&M varying by agency. For example, State  allocates about 87 percent of its IT spending on non-provisioned O&M,  whereas the Department of Transportation (DOT) allocates 50 percent.  See figure 5 for details on agencies’ planned spending allocations.
Additionally, OMB has not identified an associated goal with its non- provisioned IT measure that is part of PortfolioStat process. An OMB  official within the Office of E-Government and Information Technology  stated that the aim is for the amount of spending on DME and provisioned  IT services to rise, thus reducing the percent of spending on non- provisioned IT. This official also stated that OMB has not identified a  specific goal for the measure because it would be ever changing. While  goals for performance measures may change over time, it is still  important for OMB to set a target by which agencies can measure their  progress in meeting this measure.
In particular, leading practices stress that organizations should measure  performance in order to evaluate the success or failure of their activities  and programs. Performance measurement involves identifying  performance goals and measures, identifying targets for improving  performance, and measuring progress against those targets. Without  links to outcomes and goals, organizations are not able to effectively  measure progress toward those goals. Further, OMB’s own website,  performance.gov states that when measuring performance, a goal is a  simple but powerful way to motivate people and communicate priorities.  In addition, the website states that the federal government operates more  effectively when agency leaders, at all levels of the organization, starting  at the top, set clear measurable goals aligned to achieving better  outcomes.
Until OMB develops a specific goal associated with measuring non- provisioned services, OMB and agencies will be limited in their ability to  evaluate progress that has been made and whether or not they are  achieving their goals to increase the amount spent on development  activities and provisioned IT services.
section_title: Many O&M Investments Were at Risk and Lacked Proper Oversight
paragraphs: According to OMB guidance, the O&M phase is often the longest phase  of an investment and can consume more than 80 percent of the total  lifecycle costs. As such, agencies must actively manage their investment  during this phase. To help them do so, OMB requires that CIOs submit  ratings that reflect the level of risk facing an investment.
Several O&M investments were rated as moderate to high risk in fiscal  year 2015. Specifically, CIOs from the 12 selected agencies reported that  23 of their 187 major IT O&M investments were moderate to high risk as  of August 2015. They requested $922.9 million in fiscal year 2016 for  these investments. Of the 23 investments, agencies had plans to replace  or modernize 19 investments. However, the plans for 12 of those were  general or tentative in that the agencies did not provide specificity on time  frames, activities to be performed, or functions to be replaced or  enhanced. Further, agencies did not plan to modernize or replace 4 of the  investments (see table 4).
The lack of specific plans to modernize or replace these investments  could result in wasteful spending on moderate- and high-risk investments.
subsections: section_title: Many O&M Investments Lacked Reviews and Oversight
paragraphs: In instances where investments experience problems, agencies can  perform a TechStat, a face-to-face meeting to terminate or turn around IT  investments that are failing or not producing results. In addition, OMB  directs agencies to monitor O&M investments through operational  analyses, which should be performed annually and assess costs,  schedules, whether the investment is still meeting customer and business  needs, and investment performance.
While agencies generally conducted the required operational analyses,  they did not consistently perform TechStat reviews on all of the at-risk  investments. Table 5 provides details on the 23 investments and whether  the operational analyses and TechStats were performed.
Although not required, agencies had performed TechStats on only five of  the at-risk investments. Moreover, TechStats were not performed on  three of the four investments rated as high risk:
DHS’s Customs and Border Protection - Integrated Fixed Towers,
HHS’s Trusted Internet Connection Investment, and
U.S. Department of Agriculture’s (USDA) Enterprise  Telecommunications Shared Services.
Agencies provided several reasons for not conducting TechStats. For  example, according to agency officials, several of the investments’ risk  levels were reduced to low or moderately-low risk in the months since the  IT Dashboard had been publicly updated. An Acting Deputy Executive  Director in DHS’s Enterprise Business Management Office stated that the  agency had performed an internal “health assessment” on its Integrated  Fixed Towers investment, understood the issues it was facing, and  decided that a TechStat was not necessary. An official from HHS’s Office  of the CIO stated that, at the time it was evaluated, its Trusted Internet  Connection Investment did not meet its internal TechStat criteria of  having cost variance over 10 percent. An official from USDA’s Office of  the CIO stated that while the office did not hold a formal TechStat, the  program was required to work on a corrective action plan and has since  been upgraded from high to moderate risk.
It should be noted that recent legislation requires agencies to perform a  review of each major IT investment that receives a high-risk rating for 4  consecutive quarters. Further, the associated OMB guidance requires  agencies to hold a TechStat on an investment if it has been rated as high  risk for 3 consecutive months.
In addition, operational analyses were not conducted for four at-risk  investments. These investments were:
Commerce’s Enterprise Cyber Security Monitoring and Operations,
DHS’s Integrated Fixed Towers,
Treasury’s Departmental Offices IT Infrastructure Mainframes and  Servers Services and Support, and
Treasury’s Departmental Offices IT Infrastructure End User Systems  and Support.
An official from Commerce’s Office of the CIO stated that, in place of  operational analyses, National Weather Service (the responsible bureau)  reviews the status of the previous month’s activities for the development,  integration, modification, and procurement to report issues to  management. However, Commerce’s monthly process does not include  all of the key elements of an operational analysis. The Integrated Fixed  Towers Program Manager stated that since the investment had only  become operational in October 2015, an operational analysis was not yet  required. DHS plans to perform the analysis on the investment in August  2017. Performing the analysis once the investment is operational will  enable DHS to determine whether it is meeting the needs of the agency  and delivering the expected value.
The Director of Treasury’s Capital Planning and Investment Control  program stated that the department’s policy does not require  infrastructure investments to have an operational analysis performed.  However, OMB’s guidance on operational analyses does not exclude  infrastructure investments.
Until agencies ensure that their O&M investments are fully reviewed, the  government’s oversight of old and vulnerable investments will be impaired  and the associated spending could be wasteful.
section_title: IT Investments Are Becoming Obsolete and Agencies Are Not Required to Identify Investments That Need Attention
paragraphs: Legacy IT investments across the federal government are becoming  increasingly obsolete. Specifically, many use outdated languages and old  parts. Numerous old investments are using obsolete programming  languages. Several agencies, such as the Department of Justice  (Justice), DHS, HHS, Treasury, USDA, and VA, reported using Common  Business Oriented Language (COBOL)—a programming language  developed in the late 1950s and early 1960s—to program their legacy  systems. It is widely known that agencies need to move to more modern,  maintainable languages, as appropriate and feasible. For example, the  Gartner Group, a leading IT research and advisory company, has  reported that organizations using COBOL should consider replacing the  language and in 2010 noted that there should be a shift in focus to using  more modern languages for new products.
In addition, some legacy systems may use parts that are obsolete and  more difficult to find. For instance, Defense is still using 8-inch floppy  disks in a legacy system that coordinates the operational functions of the  United States’ nuclear forces. (See figure 6).
Further, in some cases, the vendors no longer provide support for  hardware or software, creating security vulnerabilities and additional  costs. For example, each of the 12 selected agencies reported using  unsupported operating systems and components in their fiscal year 2014  reports pursuant to the Federal Information Security Management Act of  2002. Commerce, Defense, DOT, HHS, and VA reported using 1980s  and 1990s Microsoft operating systems that stopped being supported by  the vendor more than a decade ago.
Lastly, legacy systems may become increasingly more expensive as  agencies have to deal with the previously mentioned issues and may pay  a premium to hire staff or contractors with the knowledge to maintain  outdated systems. For example, one agency (SSA) reported re-hiring  retired employees to maintain its COBOL systems.
Selected agencies reported that they continue to maintain old  investments in O&M. For example, Treasury reported systems that were  about 56 years old.
Table 6 shows the 10 oldest investments and/or systems, as reported by  selected agencies. Agencies reported having plans to modernize or  replace each of these investments and systems. However, the plans for  five of those were general or tentative in that the agencies did not provide  specific time frames, activities to be performed, or functions to be  replaced or enhanced. For a full list of the agencies’ reported oldest  systems, see appendix IV.
Separately, we profiled one system or investment from each of the 12  selected agencies. The selected systems and investments range from 11  to approximately 56 years old, and serve a variety of purposes. For  example, Treasury’s Individual Master File was first initiated about 56  years ago and currently is the authoritative data source for individual  taxpayer accounts where accounts are updated, taxes are assessed, and  refunds are generated during the tax filing period. In addition, DOT’s  profiled system was initiated about 46 years ago and allows the agency to  maintain comprehensive information on hazardous materials incidents. Of  the 12 investments or systems, agencies had plans to replace or  modernize 11 of these. However, the plans for 3 of those were general or  tentative in that the agencies did not provide specificity on time frames,  activities to be performed, or functions to be replaced or enhanced.  Further, there were no plans to replace or modernize 1 investment. The  profiles of these systems and investments are summarized in table 7 and  can be found in appendix V.
We have previously provided guidance that organizations should  periodically identify, evaluate, and prioritize their investments, including  those that are in O&M; at, near, or exceeding their planned life cycles;  and/or are based on technology that is now obsolete, to determine  whether the investment should be kept as-is, modernized, replaced or  retired. This critical process allows the agency to identify and address  high-cost or low-value investments in need of update, replacement, or  retirement.
Agencies are, in part, maintaining obsolete investments because they are  not required to identify, evaluate, and prioritize their O&M investments to  determine whether they should be kept as-is, modernized, replaced, or  retired. According to OMB staff from the Office of E-Government and  Information Technology, OMB has created draft guidance that will require  agencies to identify and prioritize legacy information systems that are in  need of replacement or modernization. Specifically, the guidance is  intended to develop criteria through which agencies can identify the  highest priority legacy systems, evaluate and prioritize their portfolio of  existing IT systems, and develop modernization plans that will guide  agencies’ efforts to streamline and improve their IT systems.
The draft guidance includes time frames for the efforts regarding  developing criteria, identifying and prioritizing systems, and planning for  modernization. However, OMB did not commit to a firm time frame for  when the policy would be issued. Until OMB’s policy is finalized and  carried out, the federal government runs the risk of continuing to maintain  investments that have outlived their effectiveness and are consuming  resources that outweigh their benefits.
Regarding upgrading obsolete investments, in April 2016, the IT  Modernization Act was introduced into the U.S. House of  Representatives. If enacted, it would establish a revolving fund of $3  billion that could be used to retire, replace, or upgrade legacy IT systems  to transition to new, more secure, efficient, modern IT systems. It also  would establish processes to evaluate proposals for modernization  submitted by agencies and monitor progress and performance in  executing approved projects.
section_title: Conclusions
paragraphs: Of the more than $80 billion that the 26 agencies reported spending for  federal IT in fiscal year 2015, the agencies spent about $61 billion on  O&M. This O&M spending has steadily increased and as a result, key  agencies are devoting a small amount of IT spending to DME activities.  To its credit, OMB has identified a performance metric to measure the  percent of IT spending on non-provisioned IT spending. However, it has  not identified an associated goal with this measure. Until it does so, OMB  and agencies will be constrained in their ability to evaluate their progress  in adopting cloud and shared services.
Several of the 12 selected agencies’ major O&M investments were rated  as moderate or high risk in fiscal year 2015. While the agencies had  specific plans to retire or modernize some of these investments, most  investments did not have specific plans with time frames, activities to be  performed, or functions to be replaced or enhanced. Further, agencies did  not consistently perform required analysis on at-risk investments. Until  agencies fully review at-risk O&M investments, the government’s  oversight of such investments will be impaired and its spending could be  wasteful.
Finally, legacy federal IT investments are becoming obsolete. Several  aging investments are using unsupported components, many of which did  not have specific plans for modernization or replacement. This is contrary  to OMB’s draft initiative, which calls for agencies to analyze and review  O&M investments. Until this policy is finalized and implemented, the  federal government runs the risk of continuing to maintain investments  that have outlived their effectiveness and are consuming resources that  outweigh their benefits.
section_title: Recommendations for Executive Action
paragraphs: To better manage legacy systems and investments, we are making 2  recommendations to OMB and 14 recommendations to federal agencies.
Specifically, we recommend that the Director of OMB  identify and publish a specific goal associated with its non-provisioned  O&M spending measure, and  commit to a firm date by which its draft guidance on legacy systems  will be issued, and subsequently direct agencies to identify legacy  systems and/or investments needing to be modernized or replaced.
To monitor whether existing investments are meeting the needs of their  agencies, we recommend that the Secretaries of Commerce and the  Treasury direct the respective agency CIO to ensure that required  analyses are performed on investments in the operations and  maintenance phase.
Further, to address obsolete IT investments in need of modernization or  replacement, we recommend that the Secretaries of Agriculture,  Commerce, Defense, Energy, Health and Human Services, Homeland  Security, State, the Treasury, Transportation, and Veterans Affairs; the  Attorney General; and the Commissioner of Social Security direct their  respective agency CIOs to identify and plan to modernize or replace  legacy systems as needed and consistent with OMB’s draft guidance,  including time frames, activities to be performed, and functions to be  replaced or enhanced.
section_title: Agency Comments and Our Evaluation
paragraphs: We received comments on a draft of this report from OMB and the other  12 agencies in our review. Eight agencies (USDA, Commerce, HHS,  DHS, State, Transportation, VA, and SSA) and OMB agreed with our  recommendations, Defense and Energy partially agreed, and Justice and  the Treasury stated they had no comment on the recommendations. Each  agency’s comments are discussed in more detail below.
In comments provided via e-mail on May 12, 2016, an official from  OMB’s Office of E-Government and Technology stated that it  concurred with our recommendations. The agency also provided  technical comments, which we have incorporated in the report as  appropriate.
In comments provided via e-mail on May 3, 2016, an official from  USDA’s Office of the CIO’s Oversight and Compliance Division stated  that the department concurred with our recommendation.
In written comments, Commerce concurred with both of its  recommendations. Regarding the recommendation that the  department ensure that required analyses are performed on  investments in the O&M phase, the department concurred and stated  that it will reiterate and expand the department’s existing policies  requiring such analyses. The department also concurred with the  recommendation to identify and plan to modernize or replace legacy  systems and stated that it is already appropriately replacing and  modernizing systems as needed within budget constraints.  Commerce’s comments are reprinted in appendix VI. The department  also provided technical comments, which we have incorporated in the  report as appropriate.
In written comments, Defense partially concurred with our  recommendation to address obsolete IT investments in need of  modernization or replacement. It stated that the department has  modernized, upgraded, or retired hundreds of systems in the last  several years through an investment review process. The department  stated it plans to continue to identify, prioritize, and manage legacy  systems that should be modernized or replaced, based on existing  department policies and processes, and consistent to the extent  practicable with OMB’s draft guidance. Defense’s plan to be  consistent with OMB’s guidance to the extent practicable is consistent  with the intent of our recommendation. Defense’s comments are  reprinted in appendix VII.
In written comments, Energy partially concurred with our  recommendation to address obsolete IT investments and stated that  the department continues to take steps to modernize its legacy  investments and systems, as needed and as funding is available. It  further stated that all four of the systems listed in appendix IV have  been identified for modernization or replacement and three have been  modernized as recently as fiscal year 2015. However, since OMB’s  draft guidance has not yet been issued, Energy could not concur with  this part of the recommendation, but plans to review and consider  implementation of such guidance. Energy’s plan to consider OMB’s  guidance when it is finalized is consistent with the intent of our  recommendation. Energy’s comments are reprinted in appendix VIII.  The department also provided technical comments, which we have  incorporated in the report as appropriate.
In written comments, HHS stated that it concurred with our  recommendation and is working to identify and plan to modernize or  replace IT systems. HHS’s comments are reprinted in appendix IX.  The department also provided technical comments, which we have  incorporated in the report as appropriate.
In written comments, DHS stated that it concurred with its  recommendation and that the department plans to establish a  framework for identifying and replacing or modernizing legacy  systems after receipt of the finalized guidance. DHS’s comments are  reprinted in appendix X. The department also provided technical  comments, which we have incorporated in the report as appropriate.
In comments provided via e-mail on May 11, 2016, an official from  Justice’s audit liaison group, speaking on behalf of the department,  stated that it had no comment on the recommendation but plans to  follow OMB’s guidance once it is formally issued. The department also  provided technical comments, which we have incorporated in the  report as appropriate.
In written comments, State agreed with the recommendation and  noted that it is currently awaiting final modernization guidance from  OMB. Upon publication of OMB’s guidance, it plans to work with OMB  to develop detailed plans for modernization. State’s comments are  reprinted in appendix XI. The department also provided technical  comments, which we have incorporated in the report as appropriate.
In comments provided via e-mail on May 12, 2016, an official from  Treasury’s Office of the CIO stated that the department had no  comments on the draft report.
In comments provided via e-mail on May 6, 2016, an official from  Transportation’s Office of the Secretary stated that the department  concurred with the draft findings and recommendations and had no  additional comments on the report.
In written comments, VA concurred with our recommendation and  stated that it launched a new office in April 2016 that will provide  lifecycle management oversight for portfolios of systems. In addition, it  stated that the department is planning to retire two high-risk, COBOL- based systems (Personnel and Accounting Integrated Data and  Benefits Delivery Network) in 2017 and 2018, respectively. VA’s  comments are reprinted in appendix XII.
In written comments, SSA stated that it agreed with our  recommendation and that it has already initiated numerous activities  to modernize or replace legacy systems. SSA’s comments are  reprinted in appendix XIII.
We are sending copies of this report to interested congressional  committees; the Secretaries of Agriculture, Commerce, Defense, Energy,  Health and Human Services, Homeland Security, State, the Treasury,  Transportation, and Veterans Affairs; the U.S. Attorney General of the  Department of Justice; the Commissioner of the Social Security  Administration; the Director of the Office of Management and Budget and  other interested parties. This report will also be available at no charge on  our website at http://www.gao.gov.
If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix XIV.
section_title: Appendix I: Objectives, Scope and Methodology
paragraphs: Our objectives were to (1) assess the extent to which federal agencies  have invested in operating and maintaining existing information  technology (IT), (2) evaluate the oversight of at-risk legacy investments,  and (3) assess the age and obsolescence of federal IT.
For our first objective, our review included the Office of Management and  Budget (OMB) and the 26 agencies that report to OMB’s IT Dashboard.  For all three objectives, to identify specific reasons for changes in  spending and specific information on individual systems or investments,  we focused on the 12 agencies with the highest planned IT spending for  fiscal year 2015, given that these agencies make up over 90 percent of  reported federal IT spending:
Department of Agriculture,
Department of Commerce,
Department of Defense,
Department of Energy,
Department of Health and Human Services,
Department of Homeland Security,
Department of Justice,
Department of State,
Department of the Treasury,
Department of Transportation,
Department of Veterans Affairs, and
Social Security Administration.
To assess the extent to which federal agencies have invested in  operating and maintaining existing IT, we reviewed data reported to OMB  as part of the budget process to determine operations and maintenance  (O&M) spending for fiscal years 2010 through 2017. We analyzed that  data to determine the extent to which spending had changed over those  years. We also compared OMB’s associated performance measure on  driving value in federal IT investments (the percent of IT spending that is  on development, modernization, and enhancement (DME) activities or  provisioned O&M services) to federal best practices. To assess the  cause of the changes in spending, we evaluated OMB budget data and  interviewed officials at the 12 selected agencies and OMB.
To evaluate the oversight of at-risk legacy investments, we reviewed  agency IT Dashboard data from the 12 selected agencies to identify  investments in O&M that had been identified as being moderate to high  risk. Specifically, we reviewed IT Dashboard data on O&M investments to  identify those that were rated as moderate to high risk by the agency chief  information officer (CIO). We reviewed agency documentation such as  TechStat documentation and operational analyses that had been  performed on the investments, as available. In addition, we interviewed  agency officials to determine plans for replacing or modernizing the  investments.
To assess the age and obsolescence of federal IT, we reviewed agency  documentation associated with their legacy investments, such as  operational analyses and enterprise architecture documents, and  interviewed agency officials on the issues related to legacy investments.  We also requested that agencies provide a list of their three oldest  systems. In some cases, agencies reported that they do not track the  ages of individual systems. In those cases, we requested that the  agencies provide their three oldest IT investments. Agencies noted that  these systems and investments may have components that are newer  than their operational age. We also compared OMB and agencies’ current  practices with federal guidance, such as OMB’s Circular No. A-11:  Preparation, Submission, and Execution of the Budget and its associated  supplement on capital assets, to determine whether OMB and agencies  are adequately managing the age and obsolescence of federal IT. We  then interviewed agency officials to confirm and obtain additional  information on the systems or investments.
To select systems or investments to profile, we identified agencies’  existing investments in O&M that were rated as medium or high risk by  their agencies’ CIO (from the previous objective on oversight). Since not  all of our selected agencies had identified an at-risk O&M investment (the  Departments of Defense, Justice, State, Transportation, and Veterans  Affairs and the Social Security Administration did not), we also used the  list of agency-identified oldest systems or investments. From the resulting  list of systems and investments, we selected one system or investment  per agency using the following factors: investment type (major or non- major), system or investment age, and risk level as of November 2015. In  particular, we sought to have a mix of systems and investments that  included both major and non-major investment types; a range of ages;  and a range of risk ratings. We also reviewed agency documentation and  interviewed agency officials on those profiled systems or investments.
To assess the reliability of the OMB budget data and IT Dashboard data,  we reviewed related documentation, such as OMB guidance on budget  preparation, capital planning, and IT Dashboard submissions. In addition,  we corroborated with each agency that the data downloaded were  accurate and reflected the data it had reported to OMB. We determined  that the budget and IT Dashboard data were reliable for our purposes of  reporting IT O&M spending and related information on O&M investments.
We conducted this performance audit from April 2015 to May 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Agency-Reported Spending on O&M
paragraphs: Table 8 provides the reported spending by agency on operations and  maintenance (O&M) and the percentage of IT spending on O&M for fiscal  years 2010 and 2015.
section_title: Appendix III: Bureaus Reporting Spending Less than 10 Percent on Development, Modernization, and Enhancement
paragraphs: Table 9 lists the 51 federal bureaus which reported spending less than 10  percent of their IT funds on development, modernization, and  enhancement in fiscal year 2015.
section_title: Appendix IV: Agency-Reported Oldest Systems or Investments
paragraphs: As part of this review, we requested that agencies provide a list of their  three oldest systems. In some cases, agencies reported that they do not  track the ages of individual systems, and as a result, we requested that  the agency provide their 3 oldest IT investments and their approximate  age. Table 10 provides a listing these systems or investments, as  reported by agencies.
section_title: Appendix V: Profiles of 12 Legacy Investments or Systems
paragraphs: We selected one system or investment per agency using a combination of  factors including investment type (major or non-major), system or  investment age, and risk level as of November 2015. In particular, we  sought to have a mix of systems and investments that included both  major and non-major investment types, a range of ages, and a range of  risk ratings.
subsections: section_title: Contractor Business Financial and Administrative Systems
paragraphs: The Contractor Business Financial and Administrative Systems  investment is intended to provide business and administrative systems for  the Department of Energy’s (Energy) Savannah River Site’s  management and operating contractor, liquid waste contractor, and the  site security contractor to manage human resources (including payroll,  benefits, and retirement for 13,000 employees and pensioners),  transparent financial reporting to Energy, supply chain, and project  management.
The investment is a commercial off-the-shelf system that runs on  Windows and Unix servers using Oracle’s PeopleSoft applications.  Specifically, the investment uses the PeopleSoft Supply Chain  Management and PeopleSoft Financials modules. According to an  agency official in Savannah River Operations, the vendors still support all  of the hardware and software used by this investment.
The agency is not currently planning future modernization activity  because the investment has gone through several updates in the past,  with the last allowing the retirement of 16 associated legacy applications  in 2011 and retired two mainframe systems. The officials stated that there  is no projected end of life date, and they plan to continue to maintain and  use the system.
section_title: Core Accounting System Suite
paragraphs: Department of Homeland Security—U.S.  Coast Guard  Number of users: 14,000 Coast Guard users  and also services the Transportation Security  Administration  Investment start date: 1998  Age: 18 years  Investment anticipated end date: 2018  Fiscal year 2016 funding: $29.8 million  Total estimated life-cycle costs: $543.2  million  Development costs: $0  Operations and maintenance costs: $543.2  million  Plans for retirement or modernization:  Agency plans to transition to federal shared  services in fiscal year 2018.
The Core Accounting System (CAS) Suite is the primary financial  management system for the U.S. Coast Guard (USCG) and, as a shared  service, the financial management system for the Transportation Security  Administration (TSA) and the Domestic Nuclear Detection Office (DNDO).  CAS is a set of several applications that assist the agencies in several  areas, including accounts receivable, accounts payable, purchasing,  asset management, procurement, and document imaging and processing.
According to the investment’s operational analysis document, CAS relies  on outdated and heavily customized Oracle software and has become  expensive to support. Specifically, it uses a version of Oracle Federal  Financials software that was first available in 2004 and the extended  vendor support for the software ended in November 2013. Further, it  relies on Windows 2003 servers and any changes would require recoding  of many functions within the CAS suite.
The agency plans to pursue using other shared services to provide its  financial management services and, therefore, began the Financial  Management Service Improvement Initiative to migrate the services from  CAS to the Department of the Interior’s shared service offering for  financial management. In August 2014, the agencies agreed to a  staggered transition of these services, with DNDO transitioning in fiscal  year 2016, TSA in fiscal year 2017, and USCG in fiscal year 2018. Until  the migration is complete and CAS can be decommissioned, USCG plans  to resolve emergent issues and maintain applications. In the meantime,  due to the costs associated with implementing a full fix and the impending  transition to shared services, USCG has accepted the security risks  associated with its legacy software.
section_title: Diversity Visa Information System
paragraphs: The Diversity Visa Information System (DVIS) is an electronic case  management system used by approximately 30 federal employees and  contractor staff working at the Department of State’s (State) Kentucky  Consular Center to track and validate application information submitted  by foreign nationals under the Diversity Visa immigration program.
Department of State  Number of users: Approximately 30 to 40  consular center staff and 55,000 applicants  annually  System start date: early 1990s  Age: Approximately 26 years  System anticipated end date: 2020  Fiscal year 2016 funding: about $164,000  Total estimated life-cycle costs: $2.4 million  Development costs: Not tracked at system  level  Operations and maintenance costs: Not  tracked at system level  Plans for retirement or modernization:  Plans to replace with ConsularOne at an  unknown time.
The DVIS interface software, PowerBuilder, is no longer supported by the  vendor. According to State officials, the main challenges in maintaining  DVIS’s aging technology are related to information security and  infrastructure concerns.
In 2013, State initiated an effort to replace numerous legacy systems,  including DVIS. As a part of this effort, State plans to replace DVIS’s  functionality with a project called ConsularOne. According to State  officials, the replacement effort is to begin in October 2018 and they plan  to retire DVIS when appropriate. In the meantime, the department plans  to upgrade the unsupported software to a new version, which is also not  supported.
section_title: Hazardous Materials Information System
paragraphs: Department of Transportation—Pipeline  and Hazardous Materials Safety  Administration  Number of users: 250 federal users, 547,000  active hazardous materials facilities, 10,000  active pipeline operators, and several million  potential facilities  Investment start date: 1970s  Age: about 41 years  Investment anticipated end date: 2018  Fiscal year 2016 funding : $6.7 million  Total estimated life-cycle costs: $38.0  million  Development costs: $22.7 million  Operations and maintenance costs: $15.3  million  Plans for retirement or modernization:  Developing new system to replace legacy  modules, and plan to retire the legacy  modules by the end of fiscal year 2018.
The Department of Transportation’s (DOT) Hazardous Materials  Information System maintains and provides access to comprehensive  information on hazardous materials incidents, exemptions and approvals,  enforcement actions, and other elements that support the regulatory  program. The system consists of five modules that register carriers and  shippers, document incidents involving hazardous materials, issue special  permits, facilitate approvals and exemptions pertaining to safety  regulations, and document standards.
Officials from Pipeline and Hazardous Material Safety’s Office of the Chief  Information Officer stated that software applications and processes used  by the system have become outdated and costly to maintain. For  example, the system uses Microsoft.NET and Classic Active Server  Pages. Officials stated that costs have increased due to maintaining the  personnel with the knowledge to use these older applications. In  particular, the costly applications include those for scanning, imaging, and  documentation management. Further, these applications are  compartmentalized, so data is duplicated and not integrated. Finally, the  system uses an application that is no longer supported by the  manufacturer, which can cause security risks, among other issues.  Specifically, the system uses Kofax Indicius software to perform optical  character recognition on scanned hazardous materials incident reports;  the software was no longer supported by the vendor, as of December  2014.
DOT is in the process of updating the functions performed by the system.  The new system’s modules are intended to be integrated, automated, and  improve efficiency, effectiveness, and data quality. Further, the  unsupported application is planned to be eliminated. While DOT does not  have dates for when individual legacy modules will be retired, officials  stated that they plan to have all the legacy modules retired by the end of  fiscal year 2018.
section_title: Individual Master File
paragraphs: Department of the Treasury—Internal  Revenue Service  Number of users: Approximately 230 million  tax accounts  Investment start date: 1960s  Age: Approximately 56 years  Investment anticipated end date: 2019  Fiscal year 2016 funding: $13.6 million  Total estimated life-cycle costs: $135.4  million   Development costs: $38.8 million  Operations and maintenance costs: $96.6  million  Plans for retirement or modernization: Will  be replaced by Customer Account Data  Engine 2 at an unknown date.
The Internal Revenue Service’s (IRS), Individual Master File (IMF) is the  authoritative data source for individual taxpayer accounts. Within IMF,  accounts are updated, taxes are assessed, and refunds are generated as  required during each tax filing period. Virtually all IRS information system  applications and processes depend on output, directly or indirectly, from  this data source.
IMF was written in an outdated assembly language code and operates  on a 2010 IBM z196/2817-m32 mainframe. This has resulted in difficulty  delivering technical capabilities addressing identify theft and refund fraud,  among other things. In addition, there is a risk of inaccuracies and system  failures due to complexity of managing dozens of systems synchronizing  individual taxpayer data across multiple data files and databases,  limitations in meeting normal financial requirements and security controls,  and keeping pace with modern financial institutions.
IRS plans to address these issues by replacing IMF with the Customer  Account Data Engine 2 (CADE 2) investment. The CADE 2 investment  includes plans to re-engineer the IMF by: (1) applying modern  programming languages, (2) establishing CADE 2 as the authoritative  data source, and (3) implementing functionality to address the IRS  financial material weakness. However, the replacement date is currently  unknown. In addition, we have previously reported on IRS’s difficulty in   delivering planned capabilities on time and on budget. Further, a key  phase of the replacement project was initially to be completed by March  2015, but IRS is currently planning to complete parts of this phase well  into 2020. As a result, the agency will continue to maintain two separate  systems until the replacement is complete.
section_title: Medicare Appeals System
paragraphs: Department of Health and Human Services  – Centers for Medicare and Medicaid  Services  Number of users: 1,900 system users  Investment start date: 2005  Age: 11 years  Investment anticipated end date: none  Fiscal year 2016 funding: $6.7 million  Total estimated life-cycle costs (through  fiscal year 2017): $99.6 million  Development costs: $20.6 million  Operations and maintenance costs: $79.0  million  Plans for retirement or modernization: The  agency has general plans to continuously  update the system.
The Centers for Medicare and Medicaid Services’ Medicare Appeals  System is a case tracking system that is to facilitate maintenance and  transfer of case specific data with regard to Medicare appeals through  multiple levels of the appeal process. In addition, the system is to provide  the capability to report on appeals data and enable more accurate and  expedient responses to Congressional questions.
The system runs on a Solaris 10 operating system, last updated in  February 2016, and uses commercial off-the-shelf systems for case  management and reporting. According to the agency, the software is still  supported by the vendors. The system has faced challenges due to the  rapid growth in appeals processed each year, expanded use of  settlements, and the increased interest in appeals data. This has resulted  in an increased need for infrastructure changes, such as more storage,  licenses, and processing capacity.
Agency officials stated that they do not have any plans to address these  gaps and that doing so is contingent on funding. They also noted general  plans to continuously update the system, but they too are contingent on  receiving funding.
section_title: National Weather Service Telecommunication Gateway
paragraphs: Department of Commerce—National  Oceanic and Atmospheric Administration  Number of users: Thousands of users  across the federal, state, and local levels, in  addition to those in the international and  academic communities  Investment start date: 1985  Age: 31 years  Investment anticipated end date: 2017  Fiscal year 2016 funding: $12.7 million  Total estimated life-cycle costs: $282.8  million  Development costs: $35.4 million  Operations and maintenance costs: $247.4  million  Plans for retirement or modernization:  Agency plans to retire the system in fiscal  year 2017 and replace it with a new system.
The National Weather Service Telecommunication Gateway (NWSTG)  system is operated by the National Oceanic and Atmospheric  Administration, a component of the Department of Commerce. It is the  nation’s hub for the collection and distribution of weather data and  products and provides national and global real-time exchange services  using automated communications resources to collect and distribute a  wide variety of environmental data such as observations, analysis, and  forecast products. Thousands of customers worldwide use data  distributed by the NWSTG and these data affect a wide range of  economic and emergency management decisions.
Concerns with the system had been increasing because the investment  faced risks and challenges associated with an aging and unsupportable  infrastructure, limited backup capability, and un-scalable architecture to  support future data volume collection and dissemination. In 2013, the  agency upgraded its hardware and software to Power7 IBM servers and  Unix operating systems (as depicted in the figure); however, NWSTG still  lacks full backup capability for 26 percent of its functions.
In fiscal year 2013, a major rearchitecture and redesign effort began  which, according to Department of Commerce officials, will result in an  entirely new dissemination architecture which will replace the NWSTG  with an integrated system that is more capable, more reliable, and have  100 percent backup capability. According to officials, a detailed project  plan to rearchitecture NWSTG is now being carried out and is scheduled  to replace the NWSTG in early fiscal year 2017.
section_title: Personnel and Accounting Integrated Data
paragraphs: The Personnel and Accounting Integrated Data (PAID) system automates  time and attendance for employees, timekeepers, payroll, and  supervisors in the Department of Veterans Affairs (VA). The PAID  software has three major modules: Time and Attendance, Employee  Master Record Downloads, and Education Tracking.
Department of Veterans Affairs  Number of users: 2,900 system users across  200 human resources offices  System start date: 1963  Age: 53 years  System anticipated end date: 2017  Fiscal year 2016 funding: $6.7 million  Total estimated life-cycle costs: n/a, not  tracked by system  Development costs: n/a, not tracked by  system  Operations and maintenance costs: $6.6  million yearly  Plans for retirement or modernization: The  system will mostly be replaced by the Human  Resources Information System Shared  Service Center, which will consolidate several  IT services to provide core human resources- related functions.
According to VA officials, PAID is a 50-year old COBOL-based system  at the end of its life span. The system runs on IBM mainframes and  uses an IBM database. Officials stated the system is not user friendly and  requires extensive training in order to use the system successfully. As a  result, the cost of maintaining the personnel to manage the system is  high.
VA officials stated that PAID is intended to be mostly replaced by Human  Resources Information System Shared Service Center in 2017, which is  to consolidate human resources IT functions and services to provide core  human resources-related functions, such as benefits and compensation.  However, the target solution is experiencing cost overruns of $14.8 million  and VA officials stated that they will not be able to replace all of PAID’s  functions. The agency is currently working on a transition plan and will  determine whether VA should find another solution for the missing  functionality or continue to keep PAID running indefinitely.
section_title: Resource Ordering and Status System
paragraphs: The U.S. Department of Agriculture’s (USDA) U.S. Forest Service’s  Resource Ordering and Status System (ROSS) is used to mobilize and  deploy a multitude of resources, including qualified individuals, teams,  aircraft, equipment, and supplies to fight wildland fires and respond to all  hazard incidents. The system supports the basic needs of the first  responders and support personnel at an incident location by processing  orders and replenishing supplies.
According to the U.S Forest Service, the technology used by ROSS is on  the verge of technical obsolescence. Specifically, one of the applications  ROSS uses is no longer supported by the vendor, creating vulnerability  issues. In addition, in order to use the system, users must download client  software onto their local computers, as opposed to accessing the system  through the web.
In September 2015, the U.S. Forest Service issued a request for  information for services to develop ROSS’s replacement—Interagency  Resources Ordering Capability. Additionally, in January 2016, Forest  Service officials signed a charter to begin this project. Agency officials  estimated that this effort will cost $14 million through fiscal year 2019 and  the solution will go live in 2018.
section_title: Sentry
paragraphs: Department of Justice—Bureau of Prisons  Number of users: 38,000 Bureau of Prisons  staff, private institution staff, and other federal  agencies  Investment start date: 1981  Age: 35 years  Investment anticipated end date: None  Fiscal year 2016 funding: $40.4 million  Total estimated life-cycle costs: $571.5  million  Development costs: $9.8 million  Operations and maintenance costs: $561.7  million  Plans for retirement or modernization:  Plans to update the user interface and  integrate system data through September  2016. Plans for continuous modernization.
The Bureau of Prisons Sentry is a real-time information system comprised  of various modules that are to enable the agency to maintain proper  custody of persons committed to their custody. It provides information  regarding security and custody levels, inmate program and work  assignments, and other pertinent information about the inmate population.  Sentry is used to process inmates at all phases of incarceration, including  release, transfer, and sentence computation.
When Sentry was first deployed over 30 years ago, it was comprised of  approximately 700 program routines written in COBOL and ran on a  mainframe platform with an Integrated Database Management System  database. It became increasingly more difficult and expensive to maintain  complex, highly-customized systems written in older programming  languages. Sentry’s entire platform—its mainframe operating system,  transaction processing software, the system software, and the database  software and system were recently updated in 2012 and uses Java and a  new database. As part of this, the bureau migrated the older database,  merged the legacy data into the newer database platform, and modified  the COBOL programs to ensure compatibility with the new software and  database. In addition, the legacy Sentry programs are now accessible via  a web browser and use a relational database and both COBOL and Java  programming languages.
The bureau has plans for updating the user interface and integrating the  data through September 2016. According to agency officials, there are no  plans to replace Sentry, as the system is the main system used by the  bureau.
section_title: Strategic Automated Command and Control System
paragraphs: Department of Defense—Air Force  Number of users: 175 users across  command functions for intercontinental  ballistic missiles, bombers, tankers, munitions  sites, intelligence, surveillance, and  reconnaissance platforms  Investment start date: 1963  Age: 53 years  Investment anticipated end date: 2030  Fiscal year 2016 funding: $5.6 million  Total estimated life-cycle costs: $135  million through 2032  Development costs: Approximately $60  million  Operations and maintenance costs: $75  million  Plans for retirement or modernization:  Defense is initiating a $60 million full system  replacement which is scheduled to be  completed in fiscal year 2020.
The Strategic Automated Command and Control System is the  Department of Defense’s (Defense) dedicated high-speed data  transmission, processing, and display system. The system coordinates  the operational functions of the United States’ nuclear forces, such as  intercontinental ballistic missiles, nuclear bombers, and tanker support  aircrafts, among others. For those in the nuclear command area, the  system’s primary function is to send and receive emergency action  messages to nuclear forces.
According to Defense officials, the system is made up of technologies and  equipment that are at the end of their useful lives. For example, the  system is still running on an IBM Series/1 Computer, which is a 1970s  computing system, and written in assembly language code. It also uses  8-inch floppy disks, which are a 1970s-era storage device; and assembly  programming code typically used in mainframes. Replacement parts for  the system are difficult to find because they are now obsolete.
As of March 2016, Defense is initiating a $60 million full system  replacement which is scheduled to be completed in fiscal year 2020. In  addition, Defense is also replacing some legacy functions in the near  term—according to officials, there is a plan underway to replace the  floppy disks with secure digital cards. This effort is underway and is  expected to be completed in the fourth quarter of fiscal year 2017.
section_title: Title II Systems
paragraphs: Social Security Administration  Number of users: Services general public,  1,300+ field offices with 50,000 to 60,000  employees, as well as other federal agencies  System start date: 1985  Age: 31 years  Investment anticipated end date: None  Fiscal year 2016 funding: $9.6 million  Total estimated life-cycle costs (through  fiscal year 2017): $519.8 million  Development costs: $235.7 million  Operations and maintenance costs: $284.1  million  Plans for retirement or modernization:  Agency has ongoing modernization efforts.
The Social Security Administration’s (SSA) Title II investment includes the  Title II system which determines retirement benefits eligibility and  amounts, 162 subsystems, as well as several smaller IT improvement  initiatives and projects. According to SSA officials, the Title II investment  accomplished its goal to improve service delivery by eliminating  antiquated Title II programs, reducing compartmentalized systems across  the agency, and reducing maintenance costs through integration.
SSA officials stated that Title II is comprised of 162 subsystems and  some are still written in COBOL. These systems were also built in a  compartmentalized structure by SSA, rather than contractors, because  the agency determined that there were not commercial programs that  could satisfy the agency’s business needs and the volume of data  needed. SSA officials stated that most of the employees who developed  these systems are ready to retire and the agency will lose their collective  knowledge. Officials further stated that training new employees to  maintain the older systems takes a lot of time.
SSA does not have plans to retire the Title II system. Rather, the agency  plans to continue to eliminate and replace Title II’s older and more costly  subsystems. Specifically, SSA currently is planning to retire four Title II  subsystems, including a claims control system, and one that processes  changes in earnings transactions. In addition, SSA has other efforts to  modernize or consolidate Title II systems, such as its database  management systems. To address the issues associated with losing  knowledgeable employees, SSA officials stated that the agency has  rehired retirees to work on the legacy systems.
section_title: Appendix VI: Comments from Department of Commerce
section_title: Appendix VII: Comments from the Department of Defense
section_title: Appendix VIII: Comments from the Department of Energy
section_title: Appendix IX: Comments from the Department of Health and Human Services
section_title: Appendix X: Comments from the Department of Homeland Security
section_title: Appendix XI: Comments from the Department of State
section_title: Appendix XII: Comments from the Department of Veterans Affairs
section_title: Appendix XIII: Comments from the Social Security Administration
section_title: Appendix XIV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, individuals making contributions to  this report included Gary Mountjoy (assistant director), Kevin Walsh  (assistant director), Scott Borre, Rebecca Eyler, Bradley Roach, Tina  Torabi, and Jessica Waselkow.