Information Technology: Agencies Need to Strengthen Oversight of Multibillion Dollar Investments in Operations and Maintenance
Released: 2013-11-06T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: Of the over $82 billion that federal agencies plan to spend on IT in fiscal year 2014, at least $59 billion is to be spent on O&M, which consists of legacy systems (i.e., steady state) and systems that are in both development and O&M (known as mixed life cycle). OMB calls for agencies to perform annual OAs, which are a key method for examining the performance of O&M investments.

GAO was asked to review IT O&M investments and agency use of OAs. The objectives of this report were to among other things (1) identify the federal IT O&M investments with the largest budgets, including their responsible agencies and how each investment supports its agency’s mission; (2) determine the extent to which these investments have undergone OAs; and (3) assess whether the responsible agency’s major IT investments are in development, mixed life cycle, or steady state. To do so, GAO focused on the 10 IT investments with the largest budgets in O&M and their responsible eight agencies, and assessed whether OAs were conducted on the investments. In addition, GAO evaluated what agencies spent on mixed, development, and O&M investments and whether agencies were using O&M funds for development activities.

section_title: What GAO Found
paragraphs: The 10 federal information technology (IT) operations and maintenance (O&M) investments with the largest budgets in fiscal year 2012—and the eight agencies that operate them—are identified by GAO in the table below. They support agencies by providing, for example, global telecommunications infrastructure and information transport services for the Department of Defense.

Of the 10 investments, only the Department of Homeland Security (DHS) investment underwent an operational analysis (OA)—a key performance evaluation and oversight mechanism required by the Office of Management and Budget (OMB) to ensure O&M investments continue to meet agency needs. DHS’s OA addressed most factors that OMB calls for; it did not address three factors (e.g., comparing current cost and schedule against original estimates). DHS officials attributed these factors not being addressed to the department still being in the process of implementing its new OA policy. The remaining agencies did not assess their investments, which accounted for $7.4 billion in reported O&M spending. Agency officials cited several reasons for not doing so, including relying on budget submission and related management reviews that measure performance; however, OMB has noted that these are not a substitute for OAs. Until the agencies ensure their operational investments are assessed, there is a risk that they will not know whether these multibillion dollar investments fully meet intended objectives.

For the eight agencies in this review, the majority of their 401 major IT investments were mixed life cycle (i.e., having activities and systems that are in both development and O&M) with regard to total spending and number of investments. Specifically, 193 (48 percent) of the investments were mixed investments, accounting for about $18 billion (61 percent) of planned spending. As such, successful oversight of such investments should involve a combination of conducting OAs to address operational portions of an investment and establishing IT governance and program management disciplines to manage those portions under development. GAO’s experience at the agencies and this report have identified agency inconsistencies in conducting OAs and establishing the capabilities that are key to effectively managing IT investments; accordingly, GAO has made prior recommendations to strengthen agency efforts in these areas.

section_title: What GAO Recommends
paragraphs: GAO is recommending that the seven agencies that did not perform OAs on their large IT O&M investments do so, and that DHS ensure that its OA is complete and addresses all OMB factors. Of the seven agencies, three agreed with GAO’s recommendations; two partially agreed; and two had no comments. DHS agreed with the GAO recommendation to it.

section_title: Letter
paragraphs: Federal agencies plan to spend over $82 billion on information technology  (IT) in fiscal year 2014. Of this amount, at least $59 billion was reported  by the agencies to be spent on operations and maintenance (O&M). O&M  investments consist of existing legacy systems (i.e., steady state) and  systems that are in both development and O&M (known as mixed life  cycle). Given the size and magnitude of these investments, it is important  that agencies effectively manage the operations and maintenance of  existing investments to ensure they (1) continue to meet agency needs,  (2) deliver value, and (3) do not unnecessarily duplicate or overlap with  other investments.

The Office of Management and Budget (OMB) directs agencies to  periodically examine the performance of these investments against,  among other things, established cost, schedule, and performance goals.  Specifically, OMB calls for agencies to perform annual operational  analyses (OA), which are a key method for examining the performance of  such investments in O&M.

The objectives of our current review were to:  identify the federal IT O&M investments with the largest budgets,  including their responsible agencies and how each investment  supports its agency’s mission; determine the extent to which these investments have undergone assess whether the responsible agency’s major IT investments are in  development, mixed life cycle, or steady state and the extent to which  funding for investments in O&M have been transferred and used to  fund investments in development.

To do so, our work focused on the 10 federal IT investments with the  largest reported budgets in O&M and the eight responsible agencies— namely, the Departments of Defense (DOD), Energy, Homeland Security  (DHS), Health and Human Services (HHS), the Treasury, and Veterans  Affairs (VA), and the National Aeronautics and Space Administration and  Social Security Administration—that operate these investments. In  addition, we determined how these 10 investments supported their  agencies’ mission by analyzing OMB and agency documentation and  conducting interviews with agency officials responsible for these  investments. We also addressed whether the agencies conducted OAs to  manage these 10 investments in accordance with OMB guidance. More  specifically, we assessed whether an analysis had been performed on  each of the 10 investments during fiscal year 2012 (because it was the  last full year for completing OAs) and if so, we compared it to the OMB  guidance criteria to identify any gaps and their causes. Further, to assess  whether each of the eight agency’s major IT investments are in  development, mixed life cycle, or steady state, we analyzed OMB budget  documentation on the agencies’ planned spending on each IT investment  by phase (i.e., development, mixed, or steady state). Moreover, to assess  the extent to which these agencies’ had used their O&M funds on  development activities, we compared what the agencies had planned to  spend on development and O&M (in fiscal year 2012) with what was  reported to have been spent in order to identify any variances that  indicated O&M funds had been reprogrammed and used to fund  development activities. We also conducted interviews with agency  officials and reviewed agency documentation to verify whether any such  reprogramming of funds had occurred, their causes, and the extent to  which such changes were subject to management oversight.

We conducted this performance audit from December 2012 to October  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Details on our objectives,  scope, and methodology can be found in appendix I.
subsections: section_title: Background
paragraphs: The President’s fiscal year 2014 budget request included plans for the  federal government to spend over $82 billion on IT. The stated goal of the  President’s IT budget request is to support making federal agencies more  efficient and effective for the American people; it also states that the  strategic use of IT is critical to success in achieving this goal. Of the $82  billion budgeted for IT, the budget provides that 26 key agencies plan to  spend the bulk of it, approximately $76 billion. Further, of the $76 billion,  over $59 billion is to be spent on O&M investments with the remainder  ($17 billion) being budgeted for development of new capabilities. As  shown in figure 1, the $59 billion represents a significant majority (i.e., 77  percent) of total budgeted spending for these agencies ($76 billion).

Although O&M spending by these agencies is about 77 percent of total IT  spending, the amount spent by each agency varies from a high of 98  percent to a low of 46 percent (as shown in the following table).

Development spending, which is intended for the inclusion of new  capabilities, accounts for approximately 23 percent of the total amount to  be spent on IT in fiscal year 2014 by these agencies. However, the  investments in development vary greatly, from 54 percent by the  Department of Transportation to a low of 2 percent by the National  Aeronautics and Space Administration.

Further, in addition to including amounts to be spent on IT development  and O&M, the budget also further specifies how the total $76 billion  budgeted for IT is to be spent on agency IT investments by the following  three categories:  those solely under development ($6 billion),  those involving activities and systems that are in both development  and O&M—known as mixed life cycle ($40 billion), and  those existing operational systems—commonly referred to by OMB as  steady state investments—that are solely in O&M ($30 billion).
subsections: section_title: OMB’s Roles and Responsibilities for Overseeing IT Investments, Including Operations and Maintenance
paragraphs: To assist agencies in managing their investments, Congress enacted the  Clinger-Cohen Act of 1996, which requires OMB to establish processes to  analyze, track, and evaluate the risks and results of major capital  investments in information systems made by federal agencies and report  to Congress on the net program performance benefits achieved as a   Further, the act places responsibility for  result of these investments.managing investments with the heads of agencies and establishes chief  information officers to advise and assist agency heads in carrying out this  responsibility.

In carrying out its responsibilities, OMB uses several data collection  mechanisms to oversee federal IT spending during the annual budget  formulation process. Specifically, OMB requires federal departments and  agencies to provide information to it related to their IT investments (called  exhibit 53s) and capital asset plans and business cases (called exhibit  300s).

Exhibit 53. The purpose of the exhibit 53 is to identify all IT  investments—both major and nonmajor within a federal organization. Information included on agency exhibit  53s is designed, in part, to help OMB better understand what  agencies are spending on IT investments. The information also  supports cost analyses prescribed by the Clinger-Cohen Act. As part  of the annual budget, OMB publishes a report on IT spending for the  federal government representing a compilation of exhibit 53 data  submitted by agencies.

According to OMB guidance, a major IT investment requires special management  attention because of its importance to the mission or function to the government;  significant program or policy implications; high executive visibility; high development,  operating, or maintenance costs; unusual funding mechanism; or definition as major by  the agency’s capital planning and investment control process.

Exhibit 300. The purpose of the exhibit 300 is to provide a business  case for each major IT investment and to allow OMB to monitor IT  investments once they are funded. Agencies are required to provide  information on each major investment’s cost, schedule, and  performance.

In addition, in June 2009, to further improve the transparency into and  oversight of agencies’ IT investments, OMB publicly deployed a website,  known as the Federal IT Dashboard (Dashboard), which replaced its  Management Watch List and High-Risk List. As part of this effort, OMB  issued guidance directing federal agencies to report, via the Dashboard,  the performance of their IT investments. Currently, the Dashboard  publicly displays information on the cost, schedule, and performance of  major federal IT investments at key federal agencies. In addition, the  Dashboard allows users to download exhibit 53 data, which include  information on both major and nonmajor investments. According to OMB,  these data are intended to provide a near real-time perspective of the  performance of these investments, as well as a historical perspective.  Further, the public display of these data is intended to allow OMB, other  oversight bodies, and the general public to hold the government agencies  accountable for results and progress. Since the Dashboard has been  implemented, we have reported and made recommendations to improve  the data accuracy and reliability. In 2010, 2011, and 2012, we reported on  the progress of the Dashboard and made recommendations to further  improve how it rates investments relative to current performance. OMB  concurred with our recommendations and has actions planned and  underway to address them.

Further, OMB has developed guidance that calls for agencies to develop  an OA policy for examining the ongoing performance of existing legacy IT  investments to measure, among other things, that the investment is  continuing to meet business and customer needs and is contributing to  meeting the agency’s strategic goals.provide for an annual OA of each investment that addresses the  following: cost, schedule, customer satisfaction, strategic and business  results, financial goals, and innovation. To address these areas, the  guidance specifies the following 17 key factors that are to be addressed:   This guidance calls for the policy to assessment of current costs against life-cycle costs; a structured schedule assessment (i.e., measuring the performance of the investment against its established schedule); a structured assessment of performance goals (i.e., measuring the  performance of the investment against established goals);  identification of whether the investment supports customer processes  as designed and is delivering goods and services it was designed to  deliver; a measure of the effect the investment has on the performing a measure of how well the investment contributes to achieving the  organization’s business needs and strategic goals; a comparison of current performance with a pre-established cost areas for innovation in the areas of customer satisfaction, strategic  and business results, and financial performance;  indication if the agency revisited alternative methods for achieving the  same mission needs and strategic goals;  consideration of issues, such as greater utilization of technology or  consolidation of investments to better meet organizational goals; an ongoing review of the status of the risks identified in the  investment’s planning and acquisition phases;  identification of whether there is a need to redesign, modify, or  terminate the investment; an analysis on the need for improved methodology (i.e., better ways  for the investment to meet cost and performance goals);  lessons learned;  cost or schedule variances;  recommendations to redesign or modify an asset in advance of  potential problems; and overlap with other investments.

With regard to overseeing the agencies’ development of policies and  annual performance, OMB officials responsible for governmentwide OA  policy stated that they expect agencies to perform all the steps specified  in the guidance and to be prepared to show documentation as evidence  of compliance with the guidance.

section_title: GAO Has Previously Reported on IT Investments in O&M
paragraphs: In October 2012 we reported on five agencies’ use of OAs (during fiscal  year 2011) and how they varied significantly. Specifically, of the five  agencies, we found that three—namely, DOD, Treasury, and VA—did not  perform analyses on their 23 major steady state investments with annual  budgets totaling $2.1 billion. The other two agencies—DHS and HHS— performed analyses but did not do so for all investments. For example,  DHS analyzed 16 of its 44 steady state investments, meaning 28  investments with annual budgets totaling $1 billion were not analyzed;  HHS analyzed 7 of its 8 steady state investments, thus omitting a single  investment totaling $77 million from being assessed. We also found that  of those OAs performed by these two agencies, none fully addressed all  the key factors. Specifically, our analysis showed that only about half of  the key factors were addressed in these assessments.

Consequently, we recommended, among other things, that the agencies  conduct annual OAs and in doing, ensure they are performed for all  investments and that all factors are fully assessed. To ensure this is done  and to provide transparency into the results of these analyses, we also  recommended that OMB revise its guidance to include directing agencies  to post the results on the Dashboard. OMB and the five agencies agreed  with our recommendations and have efforts planned and underway to  address them. In particular, OMB issued guidance (dated August 2012) to  the agencies directing them to report OA results along with their fiscal  year 2014 budget submission documentation (e.g., exhibit 300) to OMB.  According to OMB officials, they are currently establishing a process on  how agencies are to provide the information to OMB which they plan to  have in place over the next 6 months. As part of this, OMB is defining a  process for what they plan to do with the information once they receive it.

section_title: The 10 O&M IT Investments with the Largest Budgets Support Agencies in a Variety of Ways and Represent a Significant Part of Overall Federal IT O&M Funding
paragraphs: The 10 federal IT O&M investments with the largest budgets, identified  during our review, support agencies in a variety of ways such as providing worldwide telecommunications infrastructure and  information transport for DOD operations; enabling HHS to conduct research, award grants, and disseminate  biomedical research and health information to the public and National  Institutes of Health stakeholders; and providing SSA the capability to maintain demographic, wage, and  benefit information on all American citizens. Including ensuring the  availability, changeability, stability, and security of SSA’s IT operations  for the entire agency.

These investments are operated by eight agencies, such as the  Department of Energy (DOE), the National Aeronautics and Space  Administration (NASA), and Social Security Administration (SSA). In total,  the investments accounted for about $7.9 billion in O&M spending for  fiscal year 2012, which was approximately 14 percent of all such  spending for federal IT O&M. The following table identifies the 10  investments and describes the agency responsible for each investment,  the amount budgeted for O&M and development for fiscal year 2012,  investment type, and how each investment supports the organization’s  mission.

section_title: Most of the Largest O&M IT Investments Did Not Undergo Operational Analyses
paragraphs: Although required to do so, seven of the eight agencies did not conduct  OAs on their largest O&M investments. Specifically, of the 10 O&M IT  investments (with the largest budgets) we reviewed, only one agency— DHS—conducted an analysis on its investment. In doing so, the  department addressed most of the required OMB factors. However, the  other seven agencies—DOD, DOE, HHS, Treasury, VA, NASA, and  SSA—did not conduct OAs on their O&M investments, which have  combined annual O&M budgets of $7.4 billion.

The following table lists the 10 investments and whether an analysis was  completed for fiscal year 2012. Further, it provides the total O&M amount  for the investment that had an OA and for the investments that did not  have one—$529 million and $7.4 billion, respectively.

With regard to the OA DHS performed on its investment (the Customs  and Border Protection Infrastructure), the department addressed 14 of the  17 OMB factors. For example, in addressing the factor on assessing  performance goals, DHS made efforts to consolidate software licenses  and maintenance in order to eliminate redundancy and reduce costs  associated with software licenses and maintenance. Although DHS  addressed these factors, it excluded 3 factors. Specifically, the  department did not (1) assess current costs against life-cycle costs, (2)  perform a structured schedule assessment, and (3) compare current  performance against cost baseline and estimates developed when the  investment was being planned. These factors are important because,  among other things, they provide information to agency decision makers  on whether an investment’s actual annual O&M costs are as they were  planned to be and whether there is a need to examine more cost effective  approaches to meeting agency mission objectives. Table 5 shows our  analysis of DHS’s assessment of its Customs and Border Protection  Infrastructure investment.

With regard to why DHS’s analyses did not address all OMB factors,  officials from the DHS Office of the Chief Information Officer (who are  responsible for overseeing the performance of OAs departmentwide)  attributed this to the department still being in the process of updating their  Management Directive 102-01 and its related guidance, which will provide  additional instructions for completing OAs. As part of this update,  department officials told us they plan to provide additional guidance on  conducting OAs for programs once they have achieved full operational  capability. The department expects the guidance to be completed in  calendar year 2014. Further, according to DHS, once completed, this  guidance will complement existing program review processes—referred  to by DHS as program health assessments—that requires all major IT  investments, in support or mixed lifecycle phases, to complete an OA  every 12 months.

The other seven agencies attributed not performing OAs on these  investments to several factors, including relying on other management  and performance reviews—such as those used as part of developing their  annual exhibit 300 submissions to OMB—although OMB has stated that  these reviews are not to be a substitute for conducting annual analyses.  The specific reasons cited by each agency are as follows:

DOD: Officials from DOD’s Defense Information Systems Agency  stated that they did not conduct an OA for the Defense Information  Systems Network due to the fact that the investment undergoes  constant oversight through weekly meetings to review issues such as  the project status and accomplishments. Further, they said that the  program manager exercises cost, schedule, and performance  oversight using earned value management techniques. In addition,  they stated that monthly reviews of actual versus planned spending  are collected to flag any discrepancies from expected cost and  schedule objectives. While these reviews are important steps to  monitoring performance management, OMB states such ongoing  efforts to manage investment performance are not a substitute for  conducting an annual OA. According to the OMB guidance, OAs are  to be conducted for all existing IT investments to ensure that, among  other things, an investment is continuing to meet business and  customer needs and is contributing to meeting the agency’s strategic  goals.

With regard to the Next Generation Enterprise Network, officials from  the Navy who manage and oversee this investment stated an OA was  not performed due to it going through a transition from a mature  fielded system to a new service delivery model, which will become  operational in 2014. Nonetheless, OMB guidance calls for agencies to  also conduct annual analyses on all existing IT investments as part of  ensuring that such investments continue to deliver value and support  mission needs.

DOE: Officials from the Office of the Chief Information Officer stated  that an OA was not conducted on its Consolidated Infrastructure,  Office Automation, and Telecommunications Program investment  because in the summer of 2012 they began to separate it into smaller,  more manageable pieces—referred to by these agency officials as  deconsolidation— to better provide insight into the departmentwide  infrastructure. In addition, to gain further insight into the infrastructure  spending, the DOE Chief Information Officer led an in-depth analysis  in collaboration with senior IT executives, which included a commodity  IT TechStat review in the fall of 2011, and a commodity IT  PortfolioStat review in the fall of 2012. While these latter reviews are  helpful in monitoring performance, our analysis shows that they do not  fully address all 17 OMB factors. Specifically, the reviews do not  address, among other things, factors in the areas of customer  satisfaction, strategic and business results, and financial performance.  Addressing these factors is important because it provides information  to agency decision makers on whether the investment supports  customer processes and is delivering the goods and services it was  designed to deliver.

HHS: According to officials from the department’s National Institutes  of Health, the National Institutes of Health IT Infrastructure  investment, which had an annual budget of $371 million for fiscal year  2012, did not undergo an OA because this investment is an  aggregation of all the components’ infrastructure and not a particular  system or set of systems suited for this kind of macro analysis. In  addition, they noted that National Institutes of Health does monitor the  operational performance of its IT infrastructure and conducts a more  strategic analysis of services within its IT infrastructure to evaluate the  operational effectiveness at a strategic level. While these types of  performance monitoring efforts are important, OMB guidance  nonetheless calls for agencies to also conduct annual analyses on all  existing IT investments as part of ensuring that such investments  continue to deliver value and support mission needs.

Treasury: Officials from the department’s IT Capital Planning and  Investment Control branch (within the office of Treasury’s Chief  Information Officer) noted that its Internal Revenue Service Main  Frames and Servers Services and Support investment, which had a  budget of $482 million for fiscal year 2012, was deconsolidated in  fiscal year 2011 to allow for greater visibility into the infrastructure and  that it is currently undergoing an OA but were not able to provide  documentation at the time of our work.

VA: Officials from VA’s Office of Information and Technology said an  OA was not conducted on its Medical IT Support or Enterprise IT  Support investments because performance is currently being reported  monthly via the Federal IT Dashboard and internally through monthly  performance reviews. The officials added that the department plans to  develop a policy and begin conducting OAs on investments. However,  VA has not yet determined when these analyses will be completed.

NASA: Officials from NASA’s Office of the Chief Information Officer  stated while they did not conduct a formal OA on the NASA IT  Infrastructure investment, they did review the performance of the  investment using monthly performance status reviews and bimonthly  service delivery transition status updates. The officials noted that  these reviews address financial performance, schedule,  transformation initiatives, risks, customer satisfaction, performance  metrics, and business results. According to officials, the investment  underwent a service delivery transition status update and a  performance status review in May 2012. While these NASA reviews  are essential IT management tools, they do not incorporate all 17  OMB factors. For example, the reviews do not address, among other  things, innovation and whether the investment overlapped with other  systems. Fully addressing the OMB factors is essential to ensuring  investments continue to deliver value and do not unnecessarily  duplicate or overlap with other investments.

SSA: According to officials from SSA’s Office of the Chief Information  Officer, SSA’s Infrastructure Data Center investment did not undergo  an analysis because it has significant development content and  therefore an earned value analysis was conducted, which is called for  by SSA guidance for mixed life-cycle investments. Officials stated  they generally perform either an earned value analysis or OA, as  applicable to the investment. While earned value management  analyses are important to evaluating investment performance, OMB  guidance nonetheless calls for agencies to also conduct annual OAs  on all existing IT investments as part of ensuring that such  investments continue to deliver value and support mission needs.

Until the agencies address these shortcomings and ensure all their O&M  investments are fully assessed, there is increased risk that these  agencies will not know whether these multibillion dollar investments fully  meet intended objectives, including whether there are more efficient ways  to deliver their intended purpose, therefore increasing the potential for  waste and duplication.

section_title: Majority of Agencies’ Investments Were Mixed Life Cycle and Use of O&M Funds for Development Activities Was Limited
paragraphs: For the eight selected agencies, the majority of their 401 major IT  investments—totaling $29 billion— were in the mixed life-cycle phase in  both spending and number of investments. Specifically, of the $29 billion,  our analysis, as shown in figure 2, found that mixed life-cycle investments accounted for approximately $18 billion,  or 61 percent;  steady state investments accounted for approximately $8 billion, or 27  percent; and development investments accounted for approximately $3 billion, or  12 percent.

With regard to the number of investments by phase, our analysis, as  shown in figure 3, found that of the total 401 investments

193, or 48 percent, were in the mixed life-cycle phase,

139, or 35 percent, were in the steady state phase, and

69 or 17 percent, were in the development phase.

On an individual agency basis, table 6 provides the total amount each  agency reportedly spent on IT. It also shows of how each agency  allocates this total by development, mixed life cycle, or steady state  investments. Further for the mixed investments, it shows the amounts for  O&M and development.

Further, the following table provides for each of the eight agencies, their  total number of investments and of that total, the number of investments  in development, mixed life cycle, and steady state.

The implications of the above analyses—especially the results in table 6  that show mixed investments having significant amounts of funding for  both development and O&M activities—are noteworthy, particularly as it  relates to the oversight of such investments. More specifically, overseeing  these investments will involve a set of IT management capabilities for  those portions of the investment that are operational and a different set of  IT management capabilities for those portions that are still under  development. In the case of those portions that are operational, this will  include agencies having the capability to perform thorough OAs, the  importance of which is discussed earlier in this report.

For those portions still under development, OMB guidance and our best  practices research and experience at federal agencies show such  effective oversight will involve agencies having structures and  processes—commonly referred to as IT governance and program  management disciplines—that include  instituting an investment review board to define and establish the  management structure and processes for selecting, controlling, and  evaluating IT investments; ensuring that a well-defined and disciplined process is used to select  new IT proposals; and overseeing the progress of IT investments—using predefined criteria  and checkpoints—in meeting cost, schedule, risk, and benefit  expectations and to take corrective action when these expectations  are not being met.

Having these disciplines are important because they help agencies,  among other things, ensure such investments are supporting strategic  mission needs and meeting cost, schedule, and performance  expectations.

However, our experience at federal agencies has shown that agencies  have not yet fully established effective governance and program  management capabilities essential to managing IT investments. For  example, we reported in April 2011 that many agencies did not have the  mechanisms in place for investment review boards to effectively control  their investments. More specifically, we reported that while these  agencies largely had established IT investment management boards,  these boards did not have key policies and procedures in place for  ensuring that projects are meeting cost, schedule, and performance  expectations.

In addition, our experience at federal agencies, along with the results  from this audit, has found that agencies do not consistently conduct OAs.  Specifically, as noted in the background, we reported in 2012 on five  agencies’ use of them and how they varied significantly.  Of the five  agencies, we found that three—namely, DOD, Treasury, and VA—did not  perform analyses on 23 major steady state investments with annual  budgets totaling $2.1 billion. The other two agencies—DHS and HHS— performed them but did not do so for all investments. Accordingly, we  have made recommendations to these agencies to improve their use of  OAs and fully implement effective governance and program management  capabilities. They have in large part agreed to our recommendations and  have efforts underway and planned to implement them.

GAO-13-87. reprogram IT O&M funds to be used on development activities and we  identified no evidence to the contrary; two agencies—Treasury and VA— reported they did so in two instances. With regard to Treasury, the  department—on its CADE 2 investment which has a total O&M budget of  $40 million—reallocated a total of $10,000 to fund development activities  planned for the investment. According to Treasury documentation, the  cost of the investment’s operations and maintenance came in under  budget by $10,000 so the department reallocated the funds to be used on  new CADE 2 development efforts. Treasury reported this reallocation was  discussed and approved by the Internal Revenue Service’s investment  review board (the Internal Revenue Service is responsible for overseeing  CADE 2) during its monthly executive steering meetings held during fiscal  year 2012.

With regard to VA, it reprogrammed a total of $13.3 million from O&M to  development on investments within an investment category which VA  referred to as a portfolio. Specifically, during fiscal year 2012, the  department reprogrammed $13.3 million from an O&M investment within  its Medical Portfolio to investments under development within the portfolio  requiring additional funding. This reprogramming of funds was approved  by the Secretary of Veterans Affairs in June 2012.

section_title: Conclusions
paragraphs: The 10 largest federal O&M IT investments represent a significant part of  the federal government’s multibillion dollar commitment to operating and  maintaining its IT investments. Although OMB has established that  agencies are to use OAs to evaluate the performance of such  investments, their use by the agencies on these investments was very  limited. DHS was the only agency to perform such an assessment and in  doing so largely addressed the required OMB factors. While Treasury and  VA had planned to perform analyses, they had not done so. Further,  DOD, DOE, HHS, NASA, and SSA had not intended to perform these  analyses on their large O&M investments. This limited use of OAs is due  in part to a number of factors, including agencies relying on other types of  performance oversight reviews that can be helpful but are not intended to  be a substitute for these assessments. Until these agencies address  these shortcomings and ensure all their large O&M investments are fully  assessed, there is increased risk that these agencies will not know  whether these multibillion dollar investments fully meet intended  objectives, including whether there are more efficient ways to deliver their  intended purpose.

section_title: Recommendations for Executive Action
paragraphs: To ensure that the largest IT O&M investments are being adequately  analyzed, we recommend that the

Secretary of Defense direct appropriate officials to perform OAs on  the two investments identified in this report, including ensuring the  analyses include all OMB factors;

Secretary of Energy direct appropriate officials to perform an OA on  the investment identified in this report, including ensuring the analysis  includes all OMB factors;

Secretary of Health and Human Services direct appropriate officials to  perform an OA on the investment identified in this report, including  ensuring the analysis includes all OMB factors;

Secretary of Treasury direct appropriate officials to perform an OA on  the investment identified in this report, including ensuring the analysis  include all OMB factors;

Secretary of Veterans Affairs direct appropriate officials to perform  OAs on the two investments identified in this report, including  ensuring the analyses include all OMB factors;

NASA Administrator direct appropriate officials to perform an OA on  the investment identified in this report, including ensuring the analysis  includes all OMB factors; and

Commissioner of Social Security direct appropriate officials to perform  an OA on the investment identified in this report, including ensuring  the analysis includes all OMB factors.

In addition, we recommend that the Secretary of Homeland Security  direct appropriate officials to ensure the department’s OA for the Customs  and Border Protection Infrastructure is complete and assesses missing  OMB factors identified in this report.

section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, four agencies—DHS, NASA, SSA,  and VA—agreed with our recommendations; two agencies—DOD and  DOE—partially agreed; and two agencies—HHS and Treasury—had no  comments. The specific comments from the four agencies that agreed are  as follows:

DHS in its written comments, which are reprinted in appendix II,  stated that it concurred with our findings and recommendation. It  also commented that DHS’s Office of the Chief Information Officer  and the Office of Information Technology within Customs and  Border Protection (the DHS component agency responsible for the  Customs and Border Protection Infrastructure investment) are to  work closely to ensure future OAs conducted on the investment  fully address the OMB assessment factors.

NASA, in its written comments—which are reprinted in appendix  III—stated it concurred with our recommendation. NASA also  stated that it planned to conduct an OA on its NASA IT  Infrastructure investment in April 2014 that is to include all OMB  factors.

In its written comments, SSA stated it agreed with our  recommendation. It also stated that since 2008, SSA has had a  process to perform OAs on investments that were solely in O&M  and that it recently expanded the process to include mixed life  cycle IT investments that have significant systems in O&M. SSA  further commented that it was in the process of performing OAs  on the SSA mixed life cycle investment identified in our report and  other similar agency investments, with the goal of completing  these analyses by September 30, 2013. SSA’s comments are  reprinted in appendix IV.

VA, in its written comments, stated it agreed with our conclusions  and concurred with our recommendation. It also said that it had  scheduled OAs for the two investments identified in our report to  begin in the second half of fiscal year 2014. VA’s comments are  reprinted in appendix V.

The specific comments of the two agencies that partially agreed are as  follows:

DOD, in its written comments, stated that it partially concurred  with our recommendation. Specifically, DOD said it agreed with  our recommendation that its OAs should address all OMB  assessment factors and said it is establishing an OA policy in  coordination with OMB. The department further agreed with our  recommendation that it perform an OA on its Defense Information  System Network investment.

The department disagreed with our recommendation to perform  an OA on its Next Generation Enterprise Network investment  stating the investment is no longer in O&M and such investments,  per OMB policy, do not require an OA. More specifically, as noted  earlier in this report, DOD is transitioning the investment from a  mature fielded system to a new service delivery model, which will  become operational in 2014, and has moved the entire investment  back into planning and acquisition. Nonetheless, consistent with  our recommendation and as required by OMB policy, DOD plans  to conduct an OA on this investment once the department begins  to make it operational in 2014. DOD’s comments are reprinted in  appendix VI.

In its written comments—which are reprinted in appendix VII— DOE commented that it partially concurred with our  recommendation. DOE stated it was not required to perform an  OA on the Consolidated Infrastructure, Office Automation, and  Telecommunications Program because the investment no longer  exists. Specifically, DOE said it decided in 2012 to separate this  large investment into smaller, more manageable pieces—referred  to by DOE as deconsolidation—to better provide insight into its  departmentwide infrastructure, and that since the investment no  longer exists, there is no reason to perform an OA on it.

Nonetheless, consistent with our recommendation, DOE added  that it will ensure that OAs are conducted on the O&M  components of all current major IT investments in DOE’s IT  portfolio. DOE stated that it had already performed OAs on  applicable operational components that used to comprise the  Consolidated Infrastructure, Office Automation, and  Telecommunications Program. For example, DOE commented  that one of the investments created during deconsolidation— called Consolidated Infrastructure—had already undergone an OA  most recently in August 2013.

While DOE reported this progress in its comments to us, it did not  provide us with documentation to support that this OA had been  performed and whether it addressed all the OMB assessment  factors. Consequently, we are revising our recommendation to  DOE that it ensure OAs are performed on the applicable  operational components that used to comprise the Consolidated  Infrastructure, Office Automation, and Telecommunications  Program, including the newly created Consolidated Infrastructure  investment.

With regard to HHS and Treasury,

HHS, in comments provided via e-mail from its GAO Intake  Coordinator within the Office of the Assistant Secretary for  Legislation, stated that it did not have any general comments on  this report, and

Treasury in its written response said it had no comments on our  report; the department’s comments are reprinted in appendix VIII.

DHS and HHS also provided technical comments, which we incorporated  as appropriate.

We are sending copies of this report to interested congressional  committees; the Secretaries of the Departments of Defense, Energy,  Health and Human Services, Homeland Security, Treasury, and Veterans  Affairs; the Administrator of the National Aeronautics and Space  Administration; and the Commissioner of the Social Security  Administration. This report will also be available at no charge on our  website at http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix IX.

section_title: Appendix I: Objectives, Scope, Methodology
paragraphs: Our objectives were to (1) identify the federal IT O&M investments with  the largest budgets, including their responsible agencies and how each  investment supports its agency’s mission; (2) determine the extent to  which these investments have undergone OAs; and (3) assess whether  the responsible agency’s major IT investments are in development, mixed  life cycle, or steady state, and the extent to which funding for investments  in O&M have been used to finance investments in development.

To identify those federal IT O&M investments with the largest budgets, we  used data reported to the Office of Management and Budget (OMB) as  part of the budget process, and focused on the 10 largest reported  budgets in O&M and the responsible eight agencies (the Departments of  Defense, Energy, Homeland Security, Health and Human Services, the  Treasury, and Veteran Affairs; and the National Aeronautics and Space  Administration and Social Security Administration) that operate these  investments. In addition, to determine how these 10 investments support  their agencies’ missions, we reviewed OMB and agency documentation  (e.g., exhibit 300s, exhibit 53s) and interviewed agency officials.

To determine the extent to which OAs were conducted to manage these  investments in accordance with OMB guidance, we analyzed agency  documentation and interviewed responsible agency officials to determine  whether any operational analyses had been performed on these 10  investments during fiscal year 2012 because it was the last full year for  completing OAs. In those cases where an OA had been performed, we  compared it against OMB guidance on conducting them, including the 17  factors that are to be addressed as part of such assessments, to identify  any variances. Where there were variances, we reviewed agency  documentation and interviewed agency officials responsible for the OA to  identify the cause of their occurrence. In those instances where an  analysis was not performed, we reviewed documentation and interviewed  agency officials to identify why it was not done.

To assess whether each of the eight agency’s major IT investments are in  development, mixed life cycle, or steady state, we analyzed agencies’  reported spending data provided to OMB as part of the budget process to  determine what phase the majority of the investments were in and where  the majority of funds were invested (i.e., development, mixed, or steady  state). To assess the reliability of the data we analyzed, we corroborated  them by interviewing investment and other agency officials to determine  whether the OMB information we used was consistent with that reported  by the agencies; based on this assessment, we determined the data were  reliable for the purposes of this report. Further, to assess the extent to  which these and other agency IT O&M investments involve development  activities, we analyzed agency data and evaluated whether the eight  agencies were using their O&M funds for development activities (i.e.,  through the reprogramming or reallocation of funds). Specifically, we  compared what agencies planned to spend on development and O&M  with what was reported to have been spent to identify any variances that  indicated O&M funds were reprogrammed and used for development  activities. In addition, we reviewed agencies’ documentation to determine  if agencies had any processes in place to manage investments  transitioning from development to O&M. Lastly, we reviewed agency  documentation and interviewed agency IT budget and investment officials  to verify whether any reprogramming occurred, its causes, and the extent  of which any reprogramming was subject to management oversight.

We conducted this performance audit from December 2012 to October  2013 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Comments from the Department of Homeland Security

section_title: Appendix III: Comments from the National Aeronautics and Space Administration

section_title: Appendix IV: Comments from the Social Security Administration

section_title: Appendix V: Comments from the Department of Veterans Affairs

section_title: Appendix VI: Comments from the Department of Defense

section_title: Appendix VII: Comments from the Department of Energy

section_title: Appendix VIII: Comments from the Department of the Treasury

section_title: Appendix IX: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, individuals making contributions to  this report included Gary Mountjoy (Assistant Director), Gerard Aflague,  Camille Chaires, Rebecca Eyler, and Lori Martinez.


Question-summary annotations: questions: question: Which investment underwent an operational analysis?
answer: Of the 10 investments, only the Department of Homeland Security (DHS) investment underwent an operational analysis (OA)—a key performance evaluation and oversight mechanism required by the Office of Management and Budget (OMB) to ensure O&M investments continue to meet agency needs.
child_questions: question: What did DHS's OA address?
answer: DHS’s OA addressed most factors that OMB calls for; it did not address three factors (e.g., comparing current cost and schedule against original estimates).
child_questions: question: What is the reason for not addressing some factors?
answer: DHS officials attributed these factors not being addressed to the department still being in the process of implementing its new OA policy.

question: What did the remaining agencies do?
answer: The remaining agencies did not assess their investments, which accounted for $7.4 billion in reported O&M spending.
child_questions: question: Why did they not assess their investments?
answer: Agency officials cited several reasons for not doing so, including relying on budget submission and related management reviews that measure performance; however, OMB has noted that these are not a substitute for OAs.

question: What is the risk of not assessing operational investments?
answer: Until the agencies ensure their operational investments are assessed, there is a risk that they will not know whether these multibillion dollar investments fully meet intended objectives.

questions: question: What are the major IT investments for the agencies in this review?
answer: For the eight agencies in this review, the majority of their 401 major IT investments were mixed life cycle (i.e., having activities and systems that are in both development and O&M) with regard to total spending and number of investments.
child_questions: question: How many were mixed investments?
answer: Specifically, 193 (48 percent) of the investments were mixed investments, accounting for about $18 billion (61 percent) of planned spending.

question: What is the implication for oversight of these investments?
answer: As such, successful oversight of such investments should involve a combination of conducting OAs to address operational portions of an investment and establishing IT governance and program management disciplines to manage those portions under development.

question: What has GAO identified regarding IT investments?
answer: GAO’s experience at the agencies and this report have identified agency inconsistencies in conducting OAs and establishing the capabilities that are key to effectively managing IT investments; accordingly, GAO has made prior recommendations to strengthen agency efforts in these areas.

questions: question: What was GAO asked to review?
answer: GAO was asked to review IT O&M investments and agency use of OAs.
child_questions: question: What were the objectives of the report?
answer: The objectives of this report were to among other things (1) identify the federal IT O&M investments with the largest budgets, including their responsible agencies and how each investment supports its agency’s mission; (2) determine the extent to which these investments have undergone OAs; and (3) assess whether the responsible agency’s major IT investments are in development, mixed life cycle, or steady state.

question: What else did GAO evaluate?
answer: In addition, GAO evaluated what agencies spent on mixed, development, and O&M investments and whether agencies were using O&M funds for development activities.

question: How did GAO conduct this report?
answer: To do so, GAO focused on the 10 IT investments with the largest budgets in O&M and their responsible eight agencies, and assessed whether OAs were conducted on the investments.