Information Technology: Agencies Need to Strengthen Oversight of Billions of Dollars in Operations and Maintenance Investments
Released: 2012-11-15T00:00:00

Letter

Background

In fiscal year 2011, the 26 key federal agencies that report to OMB on their IT investments reported spending approximately $79 billion on a wide variety of IT systems. Of this amount, agencies reported spending $54 billion on O&M for existing steady state investments; they plan on spending about $53 billion in fiscal year 2012. As shown in figure 1, these amounts represent a significant majority (i.e., 69 and 71 percent) of the overall reported IT spending in 2011 ($79 billion) and that planned for 2012 ($75 billion), respectively.

Although O&M spending governmentwide is about 70 percent of total IT spending, the amount spent by each agency varies from a high of 98 percent to a low of 45 percent (as shown in the table below).

The National Aeronautics and Space Administration (NASA) reported spending approximately 90 percent of its total IT spending on O&M with the remaining 10 percent going to new investments. The reason for this mix of spending, according to NASA officials, is due to NASA’s mission (i.e., the space shuttle mission) which relies heavily on legacy systems. By contrast, the Department of Transportation reported spending approximately 45 percent of its total IT costs on O&M with the other 55 percent going to new investments. According to department officials, this mix of spending is largely due to the fact that the department has a number of IT development investments underway that involve large financial commitments relative to O&M investments.

OMB’s Roles and Responsibilities for Overseeing IT Investments, Including Operations and Maintenance

To assist agencies in managing their investments, Congress enacted the Clinger-Cohen Act of 1996, which requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by federal agencies and report to Congress on the net program performance benefits achieved as a Further, the act places responsibility for result of these investments.managing investments with the heads of agencies and establishes chief information officers to advise and assist agency heads in carrying out this responsibility.

In carrying out its responsibilities, OMB uses several data collection mechanisms to oversee federal IT spending during the annual budget formulation process. Specifically, OMB requires 26 key federal departments and agencies to provide information to it related to their IT investments (called exhibit 53s) and capital asset plans and business cases (called exhibit 300s).

Exhibit 53. The purpose of the exhibit 53 is to identify all IT investments— both major and nonmajor organization. Information included on agency exhibit 53s is designed, in part, to help OMB better understand what agencies are spending on IT investments. The information also supports cost analyses prescribed by the Clinger-Cohen Act. As part of the annual budget, OMB publishes a report on IT spending for the federal government representing a compilation of exhibit 53 data submitted by the 26 agencies. —and their associated costs within a federal

Exhibit 300. The purpose of the exhibit 300 is to provide a business case for each major IT investment and to allow OMB to monitor IT investments once they are funded. Agencies are required to provide information on each major investment’s cost, schedule, and performance.

According to OMB guidance, a major investment is a system or acquisition requiring special management attention because of its importance to the mission or function of the agency, a component of the agency, or another organization; is for financial management and obligates more than $500,000 annually; has significant program or policy implications; has high executive visibility; has high development, operating, or maintenance costs; is funded through other than direct appropriations; or is defined as major by the agency’s capital planning and investment control process. display of these data is intended to allow OMB, other oversight bodies, and the general public to hold the government agencies accountable for results and progress. Since the Dashboard has been implemented, we have reported and made recommendations to improve the data accuracy and reliability. In 2010 and 2011, we reported on the progress of the Dashboard and made recommendations to further improve how it rates investments relative to current performance.

Capital Programming Guide, Supplement to OMB Circular A-11, Part 7 (July 2012); OMB Memorandum M-10-27 (June 2010), requires agencies to establish a policy for performing OAs on steady state investments as a part of managing and monitoring investment baselines. a comparison of current performance with a pre-established cost areas for innovation in the areas of customer satisfaction, strategic and business results, and financial performance; indication if the agency revisited alternative methods for achieving the same mission needs and strategic goals; consideration of issues, such as greater utilization of technology or consolidation of investments to better meet organizational goals; an ongoing review of the status of the risks identified in the investment’s planning and acquisition phases; identification of whether there is a need to redesign, modify, or terminate the investment; an analysis on the need for improved methodology (i.e., better ways for the investment to meet cost and performance goals); lessons learned; cost or schedule variances; recommendations to redesign or modify an asset in advance of potential problems; and overlap with other investments.

With regard to overseeing the agencies’ development of policies and annual performance, OMB officials responsible for governmentwide OA policy stated that they expect agencies to perform all the steps specified in the guidance and to be prepared to show documentation as evidence of compliance with the guidance should OMB decide to check.

Federal Agencies’ Assessments of Major IT Steady State Investments Vary Significantly

Although OMB guidance calls for agencies to develop an OA policy and perform such analyses annually, the extent to which the five federal agencies we reviewed carried out these tasks varied significantly. Specifically, DHS and HHS developed a policy and conducted OAs, but in doing so, they excluded key investments and assessment factors. DOD, Treasury, and VA did not develop a policy or conduct OAs. The following table shows the total number of steady state investments for each agency, and provides the number and budgeted amount for those investments that underwent an assessment and those that did not.

Until agencies more completely address their policy and performance shortcomings, there is increased risk that existing multibillion dollar investments will continue to be funded although it is not fully known whether they meet their intended objectives.

DHS and HHS Developed an OA Policy and Performed OAs, but Did Not Address All Investments and Key Factors

DHS and HHS had developed policies, which contained all performance factors identified in OMB’s guidance. Specifically, In 2008, DHS issued its policy called “Operational Analysis Guidance.” The guidance states that OAs should be performed on an annual basis to evaluate the operational results of agency steady state investments. In addition, the guidance provides a report template which includes sections that should be contained and reported on in it. DHS’s policy addressed all of the key factors in the OMB guidance, including, for example, assessing current costs against life-cycle costs and a detailed schedule assessment. In 2008, HHS issued its policy called “Practices Guide: Annual Operational Analysis.” The guide states OAs are required to be performed on an annual basis. Further, the guide includes a template and a checklist for conducting them. In addition, agencies within the department have issued their own policy. For example, in 2011, The Centers for Disease Control and Prevention issued its “Operational Analysis Guide” and in 2010, the National Institutes of Health issued These policies contained all of the its framework, “A How-to Guide.”key factors identified in the OMB policy, such as measuring the effect the investment has on the performing organization itself and identifying any areas for innovation.

Further, DHS and HHS performed OAs on some of their steady state investments, but not for all. Specifically, of their 52 total steady state investments, DHS and HHS conducted analyses on 23 with total budgets of $1.4 billion and did not conduct analyses on 29 investments with total budgets of $1.1 billion. More specifically,

Of DHS’s 44 steady state investments, the department conducted OAs on16 of them, which have an annual budget of $1.2 billion; it did not perform analyses on the other 28, which have an annual budget of almost $1 billion.

Of HHS’s 8 steady state investments, the department conducted analyses on 7 of them, which have an annual budget of $207 million; it did not perform an OA on the remaining investment, which has an annual budget of $77 million.

Tables 3 and 4 show DHS’s and HHS’s steady state investments by component agency and whether OAs were performed on these investments in fiscal year 2011. (Details of our analysis of all the analyses and a brief description of the investments are included in app. II.)

In addition, although DHS and HHS performed analyses, the agencies did not address all key factors in conducting them. Specifically, of DHS’s 16 OAs, which were to include a total 272 key factors, DHS: addressed 145 (or 53 percent), partially addressed 20 (or 7 percent), and did not address 107 (or 39 percent); and of HHS’s 7 OAs, which were to include a total of 119 key factors, HHS: addressed 66 (or 55 percent), partially addressed 6 (or 5 percent), and did not address 47 (or 39 percent) factors.

The following provides key examples by component agency to illustrate how factors were fully addressed, partially addressed, or not addressed at all.

In its operational analysis of its U.S. Coast Guard Business Intelligence investment, the U.S. Coast Guard fully addressed five key factors (see table 16 in app. II). For example, on the factor regarding whether the investment supports customer processes as designed and is delivering the goods and services it was designed to deliver, the component measured (via surveys) customer satisfaction, usage trends, system trends, and feedback, and used this information to implement system improvements. U.S. Coast Guard partially addressed three factors. For example, in assessing performance goals, the component identified two major goals of the investment, but did not include how or when these goals were to be achieved. U.S. Coast Guard did not address nine key factors, including those on identifying lessons learned and reviewing the status of risk versus cost, schedule, and performance. These factors are important because they provide management with key information on why problems occurred and how they can be avoided in the future, as well as whether the investment is worth pursing given anticipated costs, benefits, and associated risks. In assessing the Information Technology Infrastructure Program, Transportation Security Administration addressed eight key factors (see table 14 in app. II). For example, on the factor calling for performance of a structured schedule assessment, the component analyzed a detailed list of task descriptions, start and end dates, and planned versus actual costs to ensure the investment is performing against an established schedule which can minimize costs over the life cycle of an investment. The component partially addressed one key factor; specifically, the factor calling for identifying whether the investment supports customer processes and is delivering the goods and services intended. In assessing this factor, Transportation Security Administration conducted surveys to measure customer satisfaction, but in doing so did not include measures to assess whether the investment was delivering the goods and services it was designed to deliver. The component did not address eight key factors. For example, it did not identify any areas for innovation or whether the investment overlapped with other systems. These latter steps are essential to identifying investment improvements, increasing value and reducing costs, and eliminating duplicate systems and the costs associated with them.

For the U.S. Immigration and Customs Enforcement’s Intelligence Fusion System, the component fully addressed nine key factors (see table 9 in app. II). These factors included analyzing current costs against life-cycle costs and whether the investment supports customer processes as designed and is delivering the goods and services it was designed to deliver, through measures such as customer surveys and help desk metrics. The component partially addressed the factor on identifying areas (e.g., business results and customer satisfaction, financial performance) for innovation. Specifically, it identified two areas for innovation, namely strategic and business results and customer satisfaction, but did not address financial performance. U.S. Immigration and Customs Enforcement did not address seven key factors; for example, it did not identify lessons learned or assess whether to modify or terminate the investment. Fully addressing these factors is crucial to agencies in determining whether to continue an investment that is not performing as required.

For its Infrastructure, Office Automation, and Telecommunications investment, Indian Health Service fully addressed 14 key factors (see table 26 in app. II). For example, in addressing the factor on assessing performance goals, it analyzed the investment’s performance goals against the results to date for each goal. The component partially addressed the factor on the status of risks versus cost, schedule, and performance. Specifically, it analyzed cost and schedule progress, but did not include an assessment of risks. Indian Health Service did not address two key factors. For example, it did not identify lessons learned and whether the investment overlapped with other systems. Addressing these factors is important because they help agencies to, among other things, identify where cost-effective improvements can be made.

HHS’s Health Resources and Services Administration fully addressed 15 key factors in its operational analysis of its Electronic Handbooks Program Management Office (see table 24 in app. II). For example, it conducted a structured assessment of performance goals, including a detailed list of goals, and how and when they were addressed. Health Resources and Services Administration only partially addressed the key factor on providing a structured schedule assessment. Specifically, the component identified certain parts of the investment schedule, such as standard and unscheduled maintenance efforts, but other schedule elements, such as completion dates and goals, were not identified. Health Resources and Services Administration did not address one factor. For example, it did not assess current costs against life-cycle costs. This factor is important because it can, among other things, provide information to agency decision makers with answers to whether annual operating and maintenance costs are comparable to the estimated costs developed during the development phase. In its analysis of the Business Intelligence System, National Institutes of Health fully assessed six key factors (see table 27 in app. II). For example, on the factor calling for identifying whether the investment supports customer processes as designed and is delivering the goods and services it was designed to deliver, the component analyzed user and customer assessments that showed improvement in this area. The component partially addressed the key factor on measuring the effect an investment has on the performing organization itself. For example, National Institutes of Health identified that the investment was in line with the appropriate component enterprise architecture, but did not identify the effect the investment had on other aspects of the department such as its mission and business processes. National Institutes of Health did not address 10 factors, including lessons learned and determining whether to modify or terminate the investment. These factors are critical to whether to continue an investment that is not performing as required.

With regard to why analyses were not performed on all investments and why those that were conducted did not fully address all factors, DHS and HHS attributed these shortfalls to the following:

Officials from DHS’s Office of the CIO who are responsible for overseeing the performance of OAs department-wide told us the components only performed 16 of the 44 analyses and did not address all key factors (in the 16 OAs that were performed) because they were not consistently implementing department and OMB policy as they should have because it was not a priority. To illustrate their point, the officials told us that while most components strive to perform annual analyses, other components do not require them to be conducted on an annual basis citing other tasks as taking precedence. To address these shortfalls, the department recently took steps to make OAs a priority and to ensure consistent application of department and OMB policy. Specifically, in May 2012, DHS’s CIO issued a memorandum stating that all steady state IT investments are required to have an annual OA completed no later than June of each fiscal year and that component CIOs are to work with program managers to implement and ensure compliance with DHS OA requirements. Further, as part of this initiative, DHS’s CIO plans to assign resources and responsibility to CIO office staff to review and ensure compliance with DHS’s policy. These are steps in the right direction; however, the DHS CIO officials told us that these initiatives will not be fully implemented until sometime in fiscal year 2013.

HHS officials from the Office of the CIO said their shortfalls (i.e., one component did not perform an OA and those that were performed did not address all factors) were due, in part, to inconsistent implementation of department and OMB policy across the components due to analyses not being a priority. As an example of this, officials from the office of the HHS CIO who are responsible for overseeing the department OA program cited how they had planned to implement an initiative to annually review all analyses performed by the components to ensure consistency and quality but have not been able to do so due to limited CIO staff being assigned to other initiatives.

Although DHS and HHS had 23 investments—with collective annual budgets of $1.4 billion—that underwent OAs, these investments were not thoroughly assessed against all key factors. Until these agencies assess all steady state investments and ensure that they are fully assessed against factors, there is increased risk that these agencies will not know whether the multibillion dollar investments fully meet their intended objectives.

DOD, Treasury, and VA Did Not Develop Policies or Perform OAs

DOD, Treasury, and VA had not developed a policy for performing OAs and did not conduct OAs for their 23 steady state investments that have combined annual budgets of $2.1 billion. Specifically,

DOD did not conduct analyses for its 4 major investments that have annual budgets totaling $381 million,

Treasury did not conduct such analyses for its16 major investments that have annual budgets totaling $152 million, and

VA did not conduct OAs for its 3 major investments that have annual budgets totaling $1.6 billion.

Regarding why DOD and VA had not developed policies and are not performing analyses, officials from those agencies stated that in lieu of conducting OAs, they assess the performance of steady state investments as part of developing their annual exhibit 300 submissions to OMB. While we have previously reported that using the exhibit 300 process can be a tool to manage investment performance, our analysis shows that the process does not fully address 11 of the 17 factors. For example, the exhibit 300 process does not fully provide for addressing the following factors: identifying alternative methods for achieving the same mission needs and strategic goals. Doing this is important because it helps agencies assess whether they are using the most cost effective solution to achieving agency goals; addressing greater utilization of technology or consolidation of investments will better meet organizational goals. It is also critical to helping agencies ensure that their investments are meeting performance goals in the most cost-effective manner; identifying lessons learned, why problems occurred, or how savings were realized, which is essential to avoid repeating the same mistakes, which helps saving resources; and identifying where the agency needs to redesign, modify, or terminate the investment, which is a means to achieving solutions that return the greatest benefit for funds invested.

Further, OMB officials told us that the exhibit 300 process is not a substitute for conducting OAs. Although OMB requires OAs for all steady state systems, its guidance does not provide a mechanism for ensuring they are completed and submitted to OMB for review. In particular, it does not have a reporting mechanism that provides for public transparency into the results of these assessments, which the IT Dashboard could provide. Having such a mechanism for the performance of steady state systems is consistent with Clinger-Cohen Act requirements that call for OMB to analyze and report on the performance of IT capital investments. Moreover, such public disclosure promotes increased transparency which is one of OMB’s goals in establishing the IT Dashboard.

Treasury officials from the department’s office of the CIO told us they decided not to perform OAs in 2011 and instead decided to use the time to develop a policy. However, the officials stated that they did not anticipate the policy to be completed until the end of this calendar year.

Until these agencies establish policies and begin performing OA assessments for their steady state investments, there is increased risk that these agencies will not know whether the multibillion dollar investments fully meet their intended objectives, therefore increasing the potential for waste and duplication.

Conclusions

The federal government has made a multibillion dollar commitment to operating and maintaining its IT investments. OMB has established guidance for federal agencies to use to evaluate the performance of such investments, including whether a sound basis exists for agencies to continue funding them. DHS and HHS had developed policies in accordance with OMB guidance and performed analyses, but did not do so for all of their investments and their analyses did not address all key factors. During the course of this review, DHS reiterated the importance of performing OAs and issued a memorandum with initiatives to address the department’s shortcomings. Further, DOD, Treasury, and VA had not developed a policy nor had they performed OAs. Taken together, these five agencies continue to invest billions of dollars each year on IT steady state investments without ensuring that they are continuing to meet agency needs and are delivering value. These shortcomings are due in part to a number of factors, including agencies relying on budget submission processes through their annual exhibit 300 submissions, which are not intended as a substitute for OAs, and not viewing performance of these assessments as a priority. Although OMB’s current guidance does not require agencies to report on OAs to OMB, using existing oversight and transparency tools like the IT Dashboard could help ensure that these important performance assessments are completed and available for public viewing. Nonetheless, until the agencies address these shortcomings and ensure all their steady state investments are fully assessed, there is increased potential for these multibillion dollar investments to result in unnecessary waste and duplication.

Recommendations for Executive Action

To ensure that major steady state IT investments are being adequately analyzed, we recommend that the Secretaries of Defense, Veterans Affairs, and the Treasury direct appropriate officials to develop an OA policy, annually perform OAs on all investments, and ensure the assessments include all key factors.

In addition, we recommend that the Secretaries of Homeland Security and Health and Human Services direct their Chief Information Officers to ensure OAs are performed annually on all major steady state investments and the assessments include all key factors.

Further, to ensure that OA policies are developed and that annual analyses are conducted and to promote transparency into the results of these analyses, we recommend that the Director of OMB revise existing guidance to include directing agencies to report on the IT Dashboard the results from the OAs of their steady state investments.

Agency Comments and Our Evaluation

In commenting on a draft of this report, OMB and the five agencies agreed with our findings and recommendations. Their comments are discussed in more detail below.

In oral comments, staff from OMB’s Office of E-Government and Information Technology concurred with our recommendations and stated that OMB had recently initiated an effort to address the specific recommendation directed to it. Specifically, the staff stated that OMB’s fiscal year 2014 budget guidance (dated August 3, 2012) directs agencies to include OAs as part of their exhibit 300 submissions to OMB. In written comments—signed by DOD’s Deputy Chief Information Officer for Information Enterprise and reprinted in appendix III—DOD concurred with our recommendation and said it plans to establish an OA policy in coordination with OMB.

In written comments—signed by the Director of the Departmental GAO-OIG Liaison Office and reprinted in appendix IV—DHS concurred with our recommendation. The department, after receiving our draft report, identified and provided to us OAs that it had performed for 3 additional investments in fiscal year 2011. DHS also provided technical comments which we incorporated in the report as appropriate. In comments provided via e-mail from its GAO Intake Coordinator within the Office of the Assistant Secretary for Legislation, HHS stated that it did not have any general comments on the report. The department did provide technical comments which we incorporated in the report as appropriate. In written comments—signed by the Deputy Assistant Secretary for Information Systems and Chief Information Officer and reprinted in appendix V—Treasury agreed with the report’s recommendations. In addition, after receiving our draft report, the department identified and provided to us OAs that it had performed on 9 of its 17 investments in fiscal year 2011. In written comments—signed by its Chief of Staff and reprinted in appendix VI—VA generally agreed with our conclusions and concurred with the recommendation to it.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees; the Secretaries of Defense, Homeland Security, Health and Human Services, the Treasury, Veterans Affairs; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII.

Appendix I: Objective, Scope, and Methodology

Our objective was to determine the extent to which selected federal agencies assess the performance of steady state information technology (IT) investments in accordance with Office of Management and Budget (OMB) guidance. To accomplish our objective, we selected the five agencies (Departments of Defense (DOD), Health and Human Services (HHS), Homeland Security (DHS), the Treasury (Treasury), and Veterans Affairs (VA)) that have the largest budgets for major steady state IT investments; collectively, these investments accounted for approximately $37 billion annually or about 70 percent of all reported IT operations and maintenance (O&M) spending. In particular, we focused on these agencies’ 75 major IT investments valued at $4.6 billion annually that were strictly in the steady state phase as opposed to the agencies’ other O&M investments—called mixed life-cycle investments by the agencies and OMB—which are not solely in O&M; these mixed life-cycle investments have projects under development as well as projects being placed into O&M.

We analyzed OMB’s guidance and identified a key practice called operational analysis (OA) that agencies are to use to assess the performance of existing O&M investments. We also interviewed OMB officials to corroborate our understanding of the key practice.

We then determined whether the five agencies developed OA policies as called for by the OMB guidance. Specifically, we compared each agency’s policy, if they had one, to the OMB criteria to determine the extent of compliance and where there were variances. We further determined whether the agencies were conducting analyses. Specifically, for the 75 major investments, we determined whether the agencies had performed an OA on each of them. In those cases where one had been performed, we analyzed the agencies’ efforts to address the OMB criteria in the analysis and categorized the extent to which the OMB key factors had been addressed using the following criteria:

Yes: if all aspects of the key factor specified in the OMB criteria were fully addressed.

No: if none of the aspects of the key factor were addressed.

Partial: if some, but not the entire key factor was addressed.

In cases where agencies did not fully address factors (i.e., partially or not all), we analyzed documentation and interviewed agency officials, including staff from the offices of the chief information officers responsible for overseeing these investments, to assist in identifying causes for shortfalls and any actions planned or underway to address the causes.

We conducted this performance audit from October 2011 to September 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Appendix II: Extent to Which DHS’s and HHS’s OAs Addressed Key Factors

Tables 5-20 show our analysis for DHS’s investments with OAs in fiscal year 2011.

Tables 21-27 show our analysis for HHS’s investments with OAs in fiscal year 2011.

Appendix III: Comments from the Department of Defense

Appendix IV: Comments from the Department of Homeland Security

Appendix V: Comments from the Department of the Treasury

Appendix VI: Comments from the Department of Veterans Affairs

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, individuals making contributions to this report included Gary Mountjoy (Assistant Director), Gerard Aflague, Rebecca Eyler, Lori Martinez, and Teresa Smith.