Information Technology: Agencies Need to Strengthen Oversight of Billions of Dollars in Operations and Maintenance Investments
Published: 2012-10-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Of the $79 billion federal agencies budgeted for IT in 2011, $54 billion (about 69 percent) was reported to have been spent on the operations and maintenance of existing legacy IT systems--commonly referred to as steady state investments. Given the size and magnitude of these investments, it is essential that agencies effectively manage them to ensure they continue to meet agency needs. As such, OMB directs agencies to periodically examine the performance of such investments against, among other things, established cost, schedule, and performance goals by performing annual OAs.
GAO was asked to determine the extent to which federal agencies analyze the performance of steady state investments in accordance with OMB guidance. To do so, GAO (1) selected five agencies, DOD, HHS, DHS, Treasury, and VA, which reported spending $4.6 billion annually on major steady state investments; and (2) and compared their fiscal year 2011 OAs to OMB criteria. GAO also analyzed documents and interviewed agency officials regarding any variances as well as their causes.
section_title: What GAO Found
paragraphs: Federal agency assessments of the performance of information technology (IT) investments in operations and maintenance (O&M)--commonly referred to as operational analyses (OAs)--vary significantly. Office of Management and Budget (OMB) guidance calls for agencies to develop an OA policy and perform such analyses annually to ensure steady state investments continue to meet agency needs. The guidance also includes 17 key factors (addressing areas such as cost, schedule, customer satisfaction, and innovation) that are to be assessed. The five agencies GAO reviewed varied in the extent to which they carried out these tasks.
The Departments of Homeland Security (DHS) and Health and Human Services (HHS) developed a policy which included all OMB assessment factors and performed OAs. However, they did not include all investments and key factors. In particular, DHS analyzed 16 of its 44 steady state investments, meaning 28 investments with annual budgets totaling $1 billion were not analyzed; HHS analyzed 7 of its 8 steady state investments. For OAs performed by DHS and HHS, both fully addressed approximately half of the key factors. With regard to the DHS and HHS investments that did not undergo an analysis or were not fully assessed against key factors, agency officials said this was due in part to program officials inconsistently applying OMB and agency guidance in conducting OAs and that OAs were not a priority. DHS and HHS have recently begun to take action to make OAs a priority and improve consistency. For example, DHS's chief information officer recently issued a directive requiring all steady state IT investments to conduct analyses annually and plans to assign staff in the office of the chief information officer to review them to ensure they are complete.
The Departments of Defense (DOD), the Treasury (Treasury), and Veterans Affairs (VA) did not develop a policy and did not perform analyses on their 23 major steady state investments with annual budgets totaling $2.1 billion. DOD and VA officials said that they did not have a policy or perform analyses because they measure the performance of steady state investments via development of plans and business cases submitted to OMB (called exhibit 300s) as part of the budget process. While these can be helpful in managing performance and do address aspects of the 17 key factors, they do not address 11 of the key factors. For example, the exhibit 300 does not address reviewing strategic business results and making recommendations to modify or terminate an investment. Treasury officials stated that they did not to perform OAs in 2011 and instead decided to use the time to develop a policy. However, the officials stated that they did not anticipate the policy to be completed until the end of this calendar year.
Overall, these five agencies have steady state investments with a fiscal year 2011 budget of over $3 billion which have not undergone needed analyses. While OMB requires agencies to perform OAs, its existing guidance does not provide mechanisms that ensure the OAs are completed and allow public transparency into the results of the assessments. Until agencies address these shortcomings, there is increased risk that these agencies will not know whether the multibillion dollar investments fully meet their intended objectives.
section_title: What GAO Recommends
paragraphs: GAO is recommending that DOD, Treasury, and VA develop an OA policy and conduct annual OAs; and that DHS and HHS ensure OAs are being performed for all investments and that all factors are fully assessed. GAO is also recommending that OMB revise its guidance to incorporate mechanisms to ensure OAs are completed and provide for increased transparency. In commenting on a draft of this report, OMB and the five agencies GAO reviewed agreed with its content and recommendations.

section_title: Letter
paragraphs: In fiscal year 2011, 26 key federal agencies reported spending  approximately $79 billion on information technology (IT) systems to the  Office of Management and Budget (OMB). Of the $79 billion, $54 billion  was reported by the agencies to be spent on operations and maintenance  (O&M), which consists of existing legacy systems (i.e., steady state) and  systems that are in both development and O&M (known as mixed life  cycle). Given the size and magnitude of these investments, it is important  that agencies effectively manage the operations and maintenance of  existing investments to ensure they (1) continue to meet agency needs,  (2) deliver value, and (3) do not unnecessarily duplicate or overlap with  other investments.
OMB directs agencies to periodically examine the performance of these  investments against, among other things, established cost, schedule, and  performance goals. Specifically, OMB calls for agencies to perform  annual operational analyses (OA), which is a key method for examining  the performance of such investments in O&M.
As requested, our objective was to determine the extent to which federal  agencies assess the performance of steady state IT investments in  accordance with this OMB guidance. To do so, we selected five agencies,  the Departments of Defense (DOD), Health and Human Services (HHS),  Homeland Security (DHS), the Treasury (Treasury), and Veterans Affairs  (VA), which have the largest budgets for major steady state IT  investments, accounting for approximately $37 billion annually or about  70 percent of all reported O&M spending in fiscal year 2011. In doing this  we focused on these agencies’ 75 major IT investments valued at $4.6  billion annually that were strictly in the O&M phase (i.e., excluded mixed  cycle investments). We determined whether the agencies developed OA  policies in accordance with OMB guidance. We also determined whether  these agencies were conducting OAs to manage these investments. More  specifically, we reviewed all of these agencies’ OAs performed during  fiscal year 2011 and compared them to OMB and related criteria.
We conducted this performance audit from October 2011 to September  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objective. Details on our objective,  scope, and methodology are contained in appendix I.
subsections: section_title: Background
paragraphs: In fiscal year 2011, the 26 key federal agencies that report to OMB on  their IT investments reported spending approximately $79 billion on a  wide variety of IT systems. Of this amount, agencies reported spending  $54 billion on O&M for existing steady state investments; they plan on  spending about $53 billion in fiscal year 2012. As shown in figure 1, these  amounts represent a significant majority (i.e., 69 and 71 percent) of the  overall reported IT spending in 2011 ($79 billion) and that planned for  2012 ($75 billion), respectively.
Although O&M spending governmentwide is about 70 percent of total IT  spending, the amount spent by each agency varies from a high of 98  percent to a low of 45 percent (as shown in the table below).
The National Aeronautics and Space Administration (NASA) reported  spending approximately 90 percent of its total IT spending on O&M with  the remaining 10 percent going to new investments. The reason for this  mix of spending, according to NASA officials, is due to NASA’s mission  (i.e., the space shuttle mission) which relies heavily on legacy systems.  By contrast, the Department of Transportation reported spending  approximately 45 percent of its total IT costs on O&M with the other 55  percent going to new investments. According to department officials, this  mix of spending is largely due to the fact that the department has a  number of IT development investments underway that involve large  financial commitments relative to O&M investments.
subsections: section_title: OMB’s Roles and Responsibilities for Overseeing IT Investments, Including Operations and Maintenance
paragraphs: To assist agencies in managing their investments, Congress enacted the  Clinger-Cohen Act of 1996, which requires OMB to establish processes to  analyze, track, and evaluate the risks and results of major capital  investments in information systems made by federal agencies and report  to Congress on the net program performance benefits achieved as a   Further, the act places responsibility for  result of these investments.managing investments with the heads of agencies and establishes chief  information officers to advise and assist agency heads in carrying out this  responsibility.
In carrying out its responsibilities, OMB uses several data collection  mechanisms to oversee federal IT spending during the annual budget  formulation process. Specifically, OMB requires 26 key federal  departments and agencies to provide information to it related to their IT  investments (called exhibit 53s) and capital asset plans and business  cases (called exhibit 300s).
Exhibit 53. The purpose of the exhibit 53 is to identify all IT investments— both major and nonmajor organization. Information included on agency exhibit 53s is designed, in  part, to help OMB better understand what agencies are spending on IT  investments. The information also supports cost analyses prescribed by  the Clinger-Cohen Act. As part of the annual budget, OMB publishes a  report on IT spending for the federal government representing a  compilation of exhibit 53 data submitted by the 26 agencies.  —and their associated costs within a federal
Exhibit 300. The purpose of the exhibit 300 is to provide a business case  for each major IT investment and to allow OMB to monitor IT investments  once they are funded. Agencies are required to provide information on  each major investment’s cost, schedule, and performance.
According to OMB guidance, a major investment is a system or acquisition requiring  special management attention because of its importance to the mission or function of the  agency, a component of the agency, or another organization; is for financial management  and obligates more than $500,000 annually; has significant program or policy implications;  has high executive visibility; has high development, operating, or maintenance costs; is  funded through other than direct appropriations; or is defined as major by the agency’s  capital planning and investment control process. display of these data is intended to allow OMB, other oversight bodies,  and the general public to hold the government agencies accountable for  results and progress. Since the Dashboard has been implemented, we  have reported and made recommendations to improve the data accuracy  and reliability. In 2010 and 2011, we reported on the progress of the  Dashboard and made recommendations to further improve how it rates  investments relative to current performance.
Capital Programming Guide, Supplement to OMB Circular A-11, Part 7 (July 2012); OMB  Memorandum M-10-27 (June 2010), requires agencies to establish a policy for performing  OAs on steady state investments as a part of managing and monitoring investment  baselines. a comparison of current performance with a pre-established cost areas for innovation in the areas of customer satisfaction, strategic  and business results, and financial performance;  indication if the agency revisited alternative methods for achieving  the same mission needs and strategic goals;  consideration of issues, such as greater utilization of technology  or consolidation of investments to better meet organizational  goals; an ongoing review of the status of the risks identified in the  investment’s planning and acquisition phases;  identification of whether there is a need to redesign, modify, or  terminate the investment; an analysis on the need for improved methodology (i.e., better  ways for the investment to meet cost and performance goals);   lessons learned;  cost or schedule variances;  recommendations to redesign or modify an asset in advance of  potential problems; and overlap with other investments.
With regard to overseeing the agencies’ development of policies and  annual performance, OMB officials responsible for governmentwide OA  policy stated that they expect agencies to perform all the steps specified  in the guidance and to be prepared to show documentation as evidence  of compliance with the guidance should OMB decide to check.
section_title: Federal Agencies’ Assessments of Major IT Steady State Investments Vary Significantly
paragraphs: Although OMB guidance calls for agencies to develop an OA policy and  perform such analyses annually, the extent to which the five federal  agencies we reviewed carried out these tasks varied significantly.  Specifically, DHS and HHS developed a policy and conducted OAs, but in  doing so, they excluded key investments and assessment factors. DOD,  Treasury, and VA did not develop a policy or conduct OAs. The following  table shows the total number of steady state investments for each  agency, and provides the number and budgeted amount for those  investments that underwent an assessment and those that did not.
Until agencies more completely address their policy and performance  shortcomings, there is increased risk that existing multibillion dollar  investments will continue to be funded although it is not fully known  whether they meet their intended objectives.
subsections: section_title: DHS and HHS Developed an OA Policy and Performed OAs, but Did Not Address All Investments and Key Factors
paragraphs: DHS and HHS had developed policies, which contained all performance  factors identified in OMB’s guidance. Specifically,   In 2008, DHS issued its policy called “Operational Analysis  Guidance.” The guidance states that OAs should be performed on an  annual basis to evaluate the operational results of agency steady  state investments. In addition, the guidance provides a report  template which includes sections that should be contained and  reported on in it. DHS’s policy addressed all of the key factors in the  OMB guidance, including, for example, assessing current costs  against life-cycle costs and a detailed schedule assessment.   In 2008, HHS issued its policy called “Practices Guide: Annual  Operational Analysis.” The guide states OAs are required to be  performed on an annual basis. Further, the guide includes a template  and a checklist for conducting them. In addition, agencies within the  department have issued their own policy. For example, in 2011, The  Centers for Disease Control and Prevention issued its “Operational  Analysis Guide” and in 2010, the National Institutes of Health issued   These policies contained all of the  its framework, “A How-to Guide.”key factors identified in the OMB policy, such as measuring the effect  the investment has on the performing organization itself and  identifying any areas for innovation.
Further, DHS and HHS performed OAs on some of their steady state  investments, but not for all. Specifically, of their 52 total steady state  investments, DHS and HHS conducted analyses on 23 with total budgets  of $1.4 billion and did not conduct analyses on 29 investments with total  budgets of $1.1 billion. More specifically,
Of DHS’s 44 steady state investments, the department conducted  OAs on16 of them, which have an annual budget of $1.2 billion; it did  not perform analyses on the other 28, which have an annual budget of  almost $1 billion.
Of HHS’s 8 steady state investments, the department conducted  analyses on 7 of them, which have an annual budget of $207 million;  it did not perform an OA on the remaining investment, which has an  annual budget of $77 million.
Tables 3 and 4 show DHS’s and HHS’s steady state investments by  component agency and whether OAs were performed on these  investments in fiscal year 2011. (Details of our analysis of all the analyses  and a brief description of the investments are included in app. II.)
In addition, although DHS and HHS performed analyses, the agencies did  not address all key factors in conducting them. Specifically, of DHS’s 16 OAs, which were to include a total 272 key factors, DHS: addressed 145 (or 53 percent), partially addressed 20 (or 7 percent), and did not address 107 (or 39 percent); and of HHS’s 7 OAs, which were to include a total of 119 key factors, HHS: addressed 66 (or 55 percent), partially addressed 6 (or 5 percent), and did not address 47 (or 39 percent) factors.
The following provides key examples by component agency to illustrate  how factors were fully addressed, partially addressed, or not addressed at  all.
In its operational analysis of its U.S. Coast Guard Business Intelligence  investment, the U.S. Coast Guard fully addressed five key factors (see  table 16 in app. II). For example, on the factor regarding whether the  investment supports customer processes as designed and is delivering  the goods and services it was designed to deliver, the component  measured (via surveys) customer satisfaction, usage trends, system  trends, and feedback, and used this information to implement system  improvements. U.S. Coast Guard partially addressed three factors. For  example, in assessing performance goals, the component identified two  major goals of the investment, but did not include how or when these  goals were to be achieved. U.S. Coast Guard did not address nine key  factors, including those on identifying lessons learned and reviewing the  status of risk versus cost, schedule, and performance. These factors are  important because they provide management with key information on  why problems occurred and how they can be avoided in the future, as  well as whether the investment is worth pursing given anticipated costs,  benefits, and associated risks.  In assessing the Information Technology Infrastructure Program,  Transportation Security Administration addressed eight key factors (see  table 14 in app. II). For example, on the factor calling for performance of  a structured schedule assessment, the component analyzed a detailed  list of task descriptions, start and end dates, and planned versus actual  costs to ensure the investment is performing against an established  schedule which can minimize costs over the life cycle of an investment.  The component partially addressed one key factor; specifically, the factor  calling for identifying whether the investment supports customer  processes and is delivering the goods and services intended. In  assessing this factor, Transportation Security Administration conducted  surveys to measure customer satisfaction, but in doing so did not include  measures to assess whether the investment was delivering the goods  and services it was designed to deliver. The component did not address  eight key factors. For example, it did not identify any areas for innovation  or whether the investment overlapped with other systems. These latter  steps are essential to identifying investment improvements, increasing  value and reducing costs, and eliminating duplicate systems and the  costs associated with them.
For the U.S. Immigration and Customs Enforcement’s Intelligence Fusion  System, the component fully addressed nine key factors (see table 9 in  app. II). These factors included analyzing current costs against life-cycle  costs and whether the investment supports customer processes as  designed and is delivering the goods and services it was designed to  deliver, through measures such as customer surveys and help desk  metrics. The component partially addressed the factor on identifying  areas (e.g., business results and customer satisfaction, financial  performance) for innovation. Specifically, it identified two areas for  innovation, namely strategic and business results and customer  satisfaction, but did not address financial performance. U.S. Immigration  and Customs Enforcement did not address seven key factors; for  example, it did not identify lessons learned or assess whether to modify  or terminate the investment. Fully addressing these factors is crucial to  agencies in determining whether to continue an investment that is not  performing as required.
For its Infrastructure, Office Automation, and Telecommunications  investment, Indian Health Service fully addressed 14 key factors (see  table 26 in app. II). For example, in addressing the factor on assessing  performance goals, it analyzed the investment’s performance goals  against the results to date for each goal. The component partially  addressed the factor on the status of risks versus cost, schedule, and  performance. Specifically, it analyzed cost and schedule progress, but  did not include an assessment of risks. Indian Health Service did not  address two key factors. For example, it did not identify lessons learned  and whether the investment overlapped with other systems. Addressing  these factors is important because they help agencies to, among other  things, identify where cost-effective improvements can be made.
HHS’s Health Resources and Services Administration fully addressed 15  key factors in its operational analysis of its Electronic Handbooks  Program Management Office (see table 24 in app. II). For example, it  conducted a structured assessment of performance goals, including a  detailed list of goals, and how and when they were addressed. Health  Resources and Services Administration only partially addressed the key  factor on providing a structured schedule assessment. Specifically, the  component identified certain parts of the investment schedule, such as  standard and unscheduled maintenance efforts, but other schedule  elements, such as completion dates and goals, were not identified.  Health Resources and Services Administration did not address one  factor.  For example, it did not assess current costs against life-cycle  costs. This factor is important because it can, among other things,  provide information to agency decision makers with answers to whether  annual operating and maintenance costs are comparable to the  estimated costs developed during the development phase.       In its analysis of the Business Intelligence System, National Institutes of  Health fully assessed six key factors (see table 27 in app. II). For  example, on the factor calling for identifying whether the investment  supports customer processes as designed and is delivering the goods  and services it was designed to deliver, the component analyzed user  and customer assessments that showed improvement in this area. The  component partially addressed the key factor on measuring the effect an  investment has on the performing organization itself. For example,  National Institutes of Health identified that the investment was in line with  the appropriate component enterprise architecture, but did not identify the  effect the investment had on other aspects of the department such as its  mission and business processes. National Institutes of Health did not  address 10 factors, including lessons learned and determining whether to  modify or terminate the investment. These factors are critical to whether  to continue an investment that is not performing as required.
With regard to why analyses were not performed on all investments and  why those that were conducted did not fully address all factors, DHS and  HHS attributed these shortfalls to the following:
Officials from DHS’s Office of the CIO who are responsible for overseeing  the performance of OAs department-wide told us the components only  performed 16 of the 44 analyses and did not address all key factors (in  the 16 OAs that were performed) because they were not consistently  implementing department and OMB policy as they should have because  it was not a priority. To illustrate their point, the officials told us that while  most components strive to perform annual analyses, other components  do not require them to be conducted on an annual basis citing other tasks  as taking precedence. To address these shortfalls, the department  recently took steps to make OAs a priority and to ensure consistent  application of department and OMB policy. Specifically, in May 2012,  DHS’s CIO issued a memorandum stating that all steady state IT  investments are required to have an annual OA completed no later than  June of each fiscal year and that component CIOs are to work with  program managers to implement and ensure compliance with DHS OA  requirements. Further, as part of this initiative, DHS’s CIO plans to assign  resources and responsibility to CIO office staff to review and ensure  compliance with DHS’s policy. These are steps in the right direction;  however, the DHS CIO officials told us that these initiatives will not be  fully implemented until sometime in fiscal year 2013.
HHS officials from the Office of the CIO said their shortfalls (i.e., one  component did not perform an OA and those that were performed did not  address all factors) were due, in part, to inconsistent implementation of  department and OMB policy across the components due to analyses not  being a priority. As an example of this, officials from the office of the HHS  CIO who are responsible for overseeing the department OA program  cited how they had planned to implement an initiative to annually review  all analyses performed by the components to ensure consistency and  quality but have not been able to do so due to limited CIO staff being  assigned to other initiatives.
Although DHS and HHS had 23 investments—with collective annual  budgets of $1.4 billion—that underwent OAs, these investments were not  thoroughly assessed against all key factors. Until these agencies assess  all steady state investments and ensure that they are fully assessed  against factors, there is increased risk that these agencies will not know  whether the multibillion dollar investments fully meet their intended  objectives.
section_title: DOD, Treasury, and VA Did Not Develop Policies or Perform OAs
paragraphs: DOD, Treasury, and VA had not developed a policy for performing OAs  and did not conduct OAs for their 23 steady state investments that have  combined annual budgets of $2.1 billion. Specifically,
DOD did not conduct analyses for its 4 major investments that have  annual budgets totaling $381 million,
Treasury did not conduct such analyses for its16 major investments that  have annual budgets totaling $152 million, and
VA did not conduct OAs for its 3 major investments that have annual  budgets totaling $1.6 billion.
Regarding why DOD and VA had not developed policies and are not  performing analyses, officials from those agencies stated that in lieu of  conducting OAs, they assess the performance of steady state  investments as part of developing their annual exhibit 300 submissions to  OMB. While we have previously reported that using the exhibit 300  process can be a tool to manage investment performance, our analysis  shows that the process does not fully address 11 of the 17 factors. For  example, the exhibit 300 process does not fully provide for addressing the  following factors:  identifying alternative methods for achieving the same mission needs and  strategic goals. Doing this is important because it helps agencies assess  whether they are using the most cost effective solution to achieving  agency goals; addressing greater utilization of technology or consolidation of  investments will better meet organizational goals. It is also critical to  helping agencies ensure that their investments are meeting performance  goals in the most cost-effective manner;  identifying lessons learned, why problems occurred, or how savings were  realized, which is essential to avoid repeating the same mistakes, which  helps saving resources; and  identifying where the agency needs to redesign, modify, or terminate the  investment, which is a means to achieving solutions that return the  greatest benefit for funds invested.
Further, OMB officials told us that the exhibit 300 process is not a  substitute for conducting OAs. Although OMB requires OAs for all steady  state systems, its guidance does not provide a mechanism for ensuring  they are completed and submitted to OMB for review. In particular, it does  not have a reporting mechanism that provides for public transparency into  the results of these assessments, which the IT Dashboard could provide.  Having such a mechanism for the performance of steady state systems is  consistent with Clinger-Cohen Act requirements that call for OMB to  analyze and report on the performance of IT capital investments.  Moreover, such public disclosure promotes increased transparency which  is one of OMB’s goals in establishing the IT Dashboard.
Treasury officials from the department’s office of the CIO told us they  decided not to perform OAs in 2011 and instead decided to use the time  to develop a policy. However, the officials stated that they did not  anticipate the policy to be completed until the end of this calendar year.
Until these agencies establish policies and begin performing OA  assessments for their steady state investments, there is increased risk  that these agencies will not know whether the multibillion dollar  investments fully meet their intended objectives, therefore increasing the  potential for waste and duplication.
section_title: Conclusions
paragraphs: The federal government has made a multibillion dollar commitment to  operating and maintaining its IT investments. OMB has established  guidance for federal agencies to use to evaluate the performance of such  investments, including whether a sound basis exists for agencies to  continue funding them. DHS and HHS had developed policies in  accordance with OMB guidance and performed analyses, but did not do  so for all of their investments and their analyses did not address all key  factors. During the course of this review, DHS reiterated the importance  of performing OAs and issued a memorandum with initiatives to address  the department’s shortcomings. Further, DOD, Treasury, and VA had not  developed a policy nor had they performed OAs. Taken together, these  five agencies continue to invest billions of dollars each year on IT steady  state investments without ensuring that they are continuing to meet  agency needs and are delivering value. These shortcomings are due in  part to a number of factors, including agencies relying on budget  submission processes through their annual exhibit 300 submissions,  which are not intended as a substitute for OAs, and not viewing  performance of these assessments as a priority. Although OMB’s current  guidance does not require agencies to report on OAs to OMB, using  existing oversight and transparency tools like the IT Dashboard could  help ensure that these important performance assessments are  completed and available for public viewing. Nonetheless, until the  agencies address these shortcomings and ensure all their steady state  investments are fully assessed, there is increased potential for these  multibillion dollar investments to result in unnecessary waste and  duplication.
section_title: Recommendations for Executive Action
paragraphs: To ensure that major steady state IT investments are being adequately  analyzed, we recommend that the Secretaries of Defense, Veterans  Affairs, and the Treasury direct appropriate officials to develop an OA  policy, annually perform OAs on all investments, and ensure the  assessments include all key factors.
In addition, we recommend that the Secretaries of Homeland Security  and Health and Human Services direct their Chief Information Officers to  ensure OAs are performed annually on all major steady state investments  and the assessments include all key factors.
Further, to ensure that OA policies are developed and that annual  analyses are conducted and to promote transparency into the results of  these analyses, we recommend that the Director of OMB revise existing  guidance to include directing agencies to report on the IT Dashboard the  results from the OAs of their steady state investments.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, OMB and the five agencies  agreed with our findings and recommendations. Their comments are  discussed in more detail below.
In oral comments, staff from OMB’s Office of E-Government and  Information Technology concurred with our recommendations and  stated that OMB had recently initiated an effort to address the specific  recommendation directed to it. Specifically, the staff stated that OMB’s  fiscal year 2014 budget guidance (dated August 3, 2012) directs  agencies to include OAs as part of their exhibit 300 submissions to  OMB.   In written comments—signed by DOD’s Deputy Chief Information  Officer for Information Enterprise and reprinted in appendix III—DOD  concurred with our recommendation and said it plans to establish an  OA policy in coordination with OMB.
In written comments—signed by the Director of the Departmental  GAO-OIG Liaison Office and reprinted in appendix IV—DHS  concurred with our recommendation. The department, after receiving  our draft report, identified and provided to us OAs that it had  performed for 3 additional investments in fiscal year 2011. DHS also  provided technical comments which we incorporated in the report as  appropriate.   In comments provided via e-mail from its GAO Intake Coordinator  within the Office of the Assistant Secretary for Legislation, HHS stated  that it did not have any general comments on the report. The  department did provide technical comments which we incorporated in  the report as appropriate.   In written comments—signed by the Deputy Assistant Secretary for  Information Systems and Chief Information Officer and reprinted in  appendix V—Treasury agreed with the report’s recommendations. In  addition, after receiving our draft report, the department identified and  provided to us OAs that it had performed on 9 of its 17 investments in  fiscal year 2011.   In written comments—signed by its Chief of Staff and reprinted in  appendix VI—VA generally agreed with our conclusions and  concurred with the recommendation to it.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees; the Secretaries of Defense, Homeland Security, Health and  Human Services, the Treasury, Veterans Affairs; and other interested  parties. In addition, the report will be available at no charge on the GAO  website at http://www.gao.gov.
If you or your staff have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix VII.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to determine the extent to which selected federal  agencies assess the performance of steady state information technology  (IT) investments in accordance with Office of Management and Budget  (OMB) guidance. To accomplish our objective, we selected the five  agencies (Departments of Defense (DOD), Health and Human Services  (HHS), Homeland Security (DHS), the Treasury (Treasury), and Veterans  Affairs (VA)) that have the largest budgets for major steady state IT  investments; collectively, these investments accounted for approximately  $37 billion annually or about 70 percent of all reported IT operations and  maintenance (O&M) spending. In particular, we focused on these  agencies’ 75 major IT investments valued at $4.6 billion annually that  were strictly in the steady state phase as opposed to the agencies’ other  O&M investments—called mixed life-cycle investments by the agencies  and OMB—which are not solely in O&M; these mixed life-cycle  investments have projects under development as well as projects being  placed into O&M.
We analyzed OMB’s guidance and identified a key practice called  operational analysis (OA) that agencies are to use to assess the  performance of existing O&M investments. We also interviewed OMB  officials to corroborate our understanding of the key practice.
We then determined whether the five agencies developed OA policies as  called for by the OMB guidance. Specifically, we compared each  agency’s policy, if they had one, to the OMB criteria to determine the  extent of compliance and where there were variances. We further  determined whether the agencies were conducting analyses. Specifically,  for the 75 major investments, we determined whether the agencies had  performed an OA on each of them. In those cases where one had been  performed, we analyzed the agencies’ efforts to address the OMB criteria  in the analysis and categorized the extent to which the OMB key factors  had been addressed using the following criteria:
Yes: if all aspects of the key factor specified in the OMB criteria were fully  addressed.
No: if none of the aspects of the key factor were addressed.
Partial: if some, but not the entire key factor was addressed.
In cases where agencies did not fully address factors (i.e., partially or not  all), we analyzed documentation and interviewed agency officials,  including staff from the offices of the chief information officers responsible  for overseeing these investments, to assist in identifying causes for  shortfalls and any actions planned or underway to address the causes.
We conducted this performance audit from October 2011 to September  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objective.
section_title: Appendix II: Extent to Which DHS’s and HHS’s OAs Addressed Key Factors
paragraphs: Tables 5-20 show our analysis for DHS’s investments with OAs in fiscal  year 2011.
Tables 21-27 show our analysis for HHS’s investments with OAs in fiscal  year 2011.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: Comments from the Department of the Treasury
section_title: Appendix VI: Comments from the Department of Veterans Affairs
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, individuals making contributions to  this report included Gary Mountjoy (Assistant Director), Gerard Aflague,  Rebecca Eyler, Lori Martinez, and Teresa Smith.