Information Technology: Agencies Need to Improve the Accuracy and Reliability of Investment Information
Released: 2006-02-13T00:00:00

Letter

Background

The President’s Budget for Fiscal Year 2006 included 1,087 IT projects, totaling about $65 billion. The planned expenditures at the civilian agencies comprised about $35 billion of that total cost. In particular, the five departments in our review made up about one-third of the civilian planned expenditures (see fig. 1).

OMB plays a key role in overseeing these IT investments and how they are managed, stemming from its predominant mission: to assist the President in overseeing the preparation of the federal budget and to supervise budget administration in executive branch agencies. In helping to formulate the President’s spending plans, OMB is responsible for evaluating the effectiveness of agency programs, policies, and procedures; assessing competing funding demands among agencies; and setting funding priorities. To carry out these responsibilities, OMB depends on agencies to collect and report accurate and complete information; these activities depend in turn on agencies having effective IT management practices.

Improvements in IT Management Are Goals of Laws and Guidance

To drive improvement in the implementation and management of IT projects, the Congress enacted the Clinger-Cohen Act in 1996, which expanded the responsibilities of OMB and the agencies that had been set under the Paperwork Reduction Act. The Clinger-Cohen Act requires that agencies engage in capital planning and performance- and results-based management. The act also requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by executive agencies. OMB is also required to report to the Congress on the net program performance benefits achieved as a result of major capital investments in information systems that are made by executive agencies.

With regard to OMB’s responsibilities in this area, we recently issued a report that provided recommendations to improve OMB’s processes for monitoring high-risk IT investments. Since that report was released, OMB has issued additional guidance outlining steps that agencies must take for all high-risk projects to better ensure improved execution and performance as well as promote more effective oversight.

In response to the Clinger-Cohen Act and other statutes, OMB developed policy for planning, budgeting, acquisition, and management of federal capital assets. This policy is set forth in OMB Circular A-11 (section 300) and in OMB’s Capital Programming Guide (supplement to Part 7 of Circular A-11), which directs agencies to develop, implement, and use a capital programming process to build their capital asset portfolios. Among other things, OMB’s Capital Programming Guide directs agencies to evaluate and select capital asset investments that will support core mission functions that must be performed by the federal government and demonstrate projected returns on investment that are clearly equal to or better than alternative uses of available public resources; institute performance measures and management processes that monitor actual performance and compare to planned results; and establish oversight mechanisms that require periodic review of operational capital assets to determine how mission requirements might have changed and whether the asset continues to fulfill mission requirements and deliver intended benefits to the agency and customers.

Among OMB’s goals in requiring the use of a capital programming process is to assist agencies in complying with a number of results-oriented requirements. Key requirements include those set by the Federal Acquisition Streamlining Act of 1994, which (1) requires agencies to establish cost, schedule, and measurable performance goals for all major acquisition programs and (2) establishes that agencies should achieve on average 90 percent of those goals; the Government Performance and Results Act of 1993, which establishes the foundation for budget decision making to achieve strategic goals in order to meet agency mission objectives; and the Federal Information Security Management Act of 2002, which requires agencies to integrate IT security into their strategic and operational planning processes, such as the capital planning and enterprise architecture processes at the agency.

OMB is aided in its responsibilities by the Chief Information Officers (CIO) Council as described by the E-Government Act of 2002. The council is designated the principal interagency forum for improving agency practices related to the design, acquisition, development, modernization, use, operation, sharing, and performance of federal government information resources. Among the specific functions of the CIO Council are the development of recommendations for the Director of OMB on government information resources management policies and requirements and the sharing of experiences, ideas, best practices, and innovative approaches related to information resources management. The CIO Council has issued several guides on capital planning and investment management over the past several years.

To further support the implementation of IT capital planning practices, we have developed an IT investment management (ITIM) framework that agencies can use in developing a stable and effective capital planning process, as required by statute and directed in OMB’s Capital Programming Guide. Consistent with the statutory focus on selecting, controlling, and evaluating investments, this framework focuses on these processes in relation to IT investments specifically. It is a tool that can be used to determine both the status of an agency’s current IT investment management capabilities and the additional steps that are needed to establish more effective processes. Mature and effective management of IT investments can vastly improve government performance and accountability. Without good management, such investments can result in wasteful spending and lost opportunities for improving delivery of services to the public.

The ITIM framework lays out a coherent collection of key practices that, when implemented in a coordinated manner, can lead an agency through a robust set of analyses and decision points that support effective IT investment management. The framework explicitly calls for consideration of cost, schedule, benefit, and risk objectives, including the development of analyses such as return on investment and a risk management plan. The framework also describes the criticality of tracking progress using valid and complete data. The guidance laid out in the ITIM framework is consistent with the requirements of OMB’s Circular A-11 and matches it in many instances. For example, among the requirements on the exhibit 300 is that agencies indicate that the investment has been reviewed and approved by the responsible oversight entity. The agency investment review board is a critical element of the ITIM framework, and the expectation for the board to select and oversee IT investments is explicit.

In previous work using our IT investment management framework, we reported that the use of IT investment management practices by agencies was mixed. For example, a few agencies that have followed the ITIM framework in implementing capital planning processes have made significant improvements. In contrast, however, we and others have continued to identify weaknesses at agencies in many areas, including immature management processes to support both the selection and oversight of major IT investments and the measurement of actual versus expected performance in meeting established IT performance measures. For example: We recently reported that the HHS senior investment board does not regularly review component agencies’ IT investments, leaving close to 90 percent of its discretionary investments without an appropriate level of executive oversight. To remedy this weakness, we recommended that the department (1) establish a process for the investment board to regularly review and track the performance of a defined set of component agency IT systems against expectations and (2) take corrective actions when these expectations are not being met.

At DHS, we determined that the department’s draft information resources management strategic plan did not include fully defined goals and performance measures. To address this weakness, we recommended that the department establish IT goals and performance measures that, at a minimum, address how information and technology management contributes to program productivity, the efficiency and effectiveness of agency operations, and service to the public.

A recent review by the DOD Inspector General determined that over 90 percent of the business cases submitted to OMB in support of the DOD fiscal year 2006 budget request did not completely respond to one or more data elements addressing security funding, certification and accreditation, training and security plans, and enterprise architecture. The DOD Inspector General concluded that, as a result, these submissions continued to have limited value and did not demonstrate that the department was effectively managing its proposed IT investments for fiscal year 2006.

Exhibit 300 Supports OMB and Agency Oversight of IT Management

Besides providing policy for planning, budgeting, acquisition, and management of federal capital assets, section 300 of OMB’s Circular A-11 instructs agencies on budget justification and reporting requirements for major IT investments. Section 300 defines the budget exhibit 300, also called the Capital Asset Plan and Business Case, as a document that agencies submit to OMB to justify resource requests for major IT investments. According to OMB, only priority capital asset investments that comply with the policies for good capital programming, as described in the Capital Programming Guide, will be recommended for funding in the President’s Budget.

The exhibit 300 was established as a source of information on which to base both quantitative decisions about budgetary resources consistent with the Administration’s program priorities and qualitative assessments about whether the agency’s planning, acquisition, management, and use of capital assets (investments) are consistent with OMB policy and guidance. The types of information included in the exhibit 300 are intended, among other things, to help OMB and the agencies identify and correct poorly planned or performing investments (i.e., investments that are behind schedule, over budget, or not delivering expected results) and real or potential systemic weaknesses in federal information resource management (such as a shortage of sufficiently qualified project managers).

According to Circular A-11, the information in the exhibit 300 allows the agency and OMB to review and evaluate each agency’s IT spending and to compare IT spending across the federal government. Further, the information helps the agency and OMB to provide a full and accurate accounting of IT investments for the agency, as required by the Paperwork Reduction Act and the Clinger-Cohen Act.

The exhibit 300 is required for all assets, though certain sections apply only to information technology. Table 1 provides a description of the key sections of the exhibit 300, as well as examples of the types of documentation that provide support for the data summarized in the exhibit 300 (although the supporting documentation may take other forms). This support may be derived from a variety of sources, including financial management systems and management processes that agencies carry out to comply with federal requirements and guidelines (such as the Federal Information Security Management Act of 2002 and the Federal Enterprise Architecture), as well as from analyses carried out specifically in support of the management of the investment.

According to OMB guidance, the life-cycle stage of the asset affects what is reported on the exhibit 300: New investments (i.e., proposed for budget year or later, or in development) must be justified based on the need to fill a gap in the agency’s ability to meet strategic goals and objectives with the lowest life-cycle costs of all possible alternatives and provide risk-adjusted cost and schedule goals and measurable performance benefits.

Mixed life-cycle investments (i.e., investments that are operational but include some developmental effort, such as a technology refresh) must demonstrate satisfactory progress toward achieving baseline cost, schedule, and performance goals using an EVM system.

Operational investments (i.e., steady state) must demonstrate, among other things, how close actual annual operating and maintenance costs are to the original life-cycle cost estimates; whether the technical merits of the investment continue to meet the needs of the agency and customers; and that an analysis of alternatives was performed with a future focus.

OMB requires agencies to transmit exhibit 300s electronically, using a predefined format. To meet this requirement and facilitate the aggregation of elements of the exhibits from various sources throughout the organization, many agencies use software applications to compile their exhibits 300s. Besides aggregating portions of the exhibit 300, these tools are designed to also perform certain calculations, such as return on investment and those required for earned value analysis.

Exhibit 300s Were Generally Not Based on Adequate Support

Although the agencies reported that all 29 exhibit 300s had been approved by their investment review boards (as required), in many instances, support for the information provided was not adequate. (Details on the 29 investment projects described in the exhibit 300s that we reviewed are provided in app. III.) Three types of problems were evident.

First, all exhibit 300s had documentation weaknesses. For example, each investment lacked documentary support for one or more of the following: Analysis of Alternatives, Risk Inventory and Assessment, and Performance Measures and Goals. In other cases, the supporting material that was provided to us did not match the information in the exhibit 300.

Second, agencies did not always demonstrate (for example, in the Security and Privacy and the Project and Funding Plan sections) that they complied with federal requirements or policies with regard to management and reporting processes.

Finally, information in some sections (such as the Summary of Spending table and the Project and Funding Plan) could not be relied upon because the numbers were not derived using repeatable processes or reliable systems.

Agency officials attributed the absence of adequate support for their exhibit 300s to lack of understanding of the requirements or of how to respond to them. Agency officials mentioned in particular insufficient guidance or training, as well as lack of familiarity with particular requirements, such as the EVM process. If underlying support is inadequate in key areas, OMB and agency executives are depending on unreliable information to monitor the management of major IT projects and to make critical decisions on their funding, thus putting at risk millions of dollars in investments.

Underlying Documentation Was Lacking or Did Not Support the Exhibit 300

OMB Circular A-11 states that agencies must justify funding requests for major acquisitions by demonstrating, among other things, measurable performance benefits, comprehensive risk mitigation and management planning, and positive return on investment for the planned investment. Agencies are instructed to establish performance metrics (including baselines from which progress can be measured) to ensure that project managers are accountable in meeting expected performance goals and that projects are aligned with the agencies’ strategic goals. Agencies are also expected to manage investment risk through a robust risk management program; according to OMB’s guidance, agencies need to actively manage risks from initial concept throughout the life cycle of each investment. To demonstrate a positive return on investment for the selected alternative and identify a project’s total lifetime cost, OMB requires agencies to compare alternatives and report summary cost information for investments (including calculations for payback period and net present value).

Documents produced in the performance of these activities provide evidence that they were carried out as required.

Performance measures. The investments did not usually demonstrate the basis for the performance measure information provided in the exhibit 300. Only 6 of the 29 investments had documentation to support how agencies initially measured their baseline levels of performance, from which they measured progress toward the agency’s strategic goals. In most cases, the investments lacked documentation describing the levels of performance that had been achieved or how these results actually helped meet agency strategic needs. The absence of documentation in these cases could indicate a systemic weakness in agency performance management practices, since well-developed practices should provide the expected support. This finding is consistent with our prior work where we determined that agencies were generally not measuring actual versus expected performance in meeting IT performance goals. Weak performance management practices reduce the ability of agency executives to track investment performance in meeting performance objectives and raise the risk that investments will not be well aligned with agency strategic objectives.

Risk management. About 75 percent of the investments were unable to demonstrate that they were actively addressing the risk elements that OMB specifies in Circular A-11, or how they had determined that any of those risks were not applicable. In addition, documentation of risk management that was provided had significant weaknesses. In one case, a risk management plan was approximately 9 years old and had not been updated, and for three investments, the risk documentation addressed only the project development phase, even though the systems had exited that phase and were in full operation.

Analysis of alternatives. All 29 investments reported cost information in the analysis of alternatives section of the exhibit 300. However, in about 72 percent of the exhibit 300s reviewed, either supporting documentation was missing for this cost information, or information in the documentation did not agree with that in the exhibit 300.

In cases where investments lacked documentation to support information reported in the performance and risk areas, project officials frequently told us that they had filled out these sections of the exhibit 300 to satisfy the reporting requirement, relying on their own knowledge of the investment rather than any project documentation. However, such an approach is not consistent with the requirement for providing accurate information in compliance with OMB capital programming and capital planning and investment control policies. In addition, several project officials told us that they believed some of the 19 risk management areas required in the exhibit 300 were not applicable to their investment, but they reported on those categories nonetheless to fulfill the requirement. Although the guidance instructs agencies to indicate whether the risk category was not applicable, officials stated that their impression is that “not applicable” responses might lower the evaluation of their investments and reduce or eliminate their funding.

Further, agency officials generally responded that the training they received for preparing the exhibit 300 was not sufficient. For example, one agency commented that agencies would benefit from targeted OMB training that would address agency-specific questions. Several agencies stressed that OMB training should occur earlier in the budget cycle. In addition, one agency said that it needed OMB training on preparing each section of the exhibit 300.

Overall, the lack of documentation supporting the exhibit 300s raises questions regarding the sufficiency of the business case for the investment and the quality of the projects’ management.

Agencies Did Not Always Comply or Provide Evidence of Compliance with Federal Requirements and Policies

Compliance with OMB and other federal guidance and related federal laws helps ensure that agency investments are managed in a manner consistent with the intent of the Congress and that key information is available to OMB and agency managers on which they can base informed decisions.

The security section of the exhibit 300 requires that agencies demonstrate that they have developed information security plans in accordance with the Federal Information Security Management Act of 2002 (FISMA); according to FISMA, these plans must include rules of behavior for system use, technical security controls, and procedures for incident handling—that is, how to respond to system security breaches. In addition, agencies ensure that employees and contractors receive security awareness training. Guidance from the National Institute of Standards and Technology (NIST) supports FISMA by outlining the necessary components of key security documentation, including security plans, certification and accreditation packages, and security controls testing.

For the analysis of alternatives section, OMB’s instructions for the exhibit 300 cite the Clinger-Cohen Act, which requires agencies to complete a cost-benefit analysis for new IT investments, and OMB Circular A-94, which outlines requirements for completing cost-benefit and cost-effectiveness analyses, including the comparison of at least three alternatives, a discussion of assumptions for each alternative, and an analysis of uncertainty (a sensitivity assessment to raise awareness of the potential for unforeseen impacts on the investment).

For the project and funding plan section, OMB Circular A-11 provides guidance that requires an agency to have in place a process for monitoring the investment’s status in accomplishing baseline cost and schedule goals.

For the 29 investments, agency compliance with the FISMA and NIST requirements described above was mixed. For example, about 86 percent of all investments could demonstrate, based on documentation, that security awareness training had been conducted for employees and contractors and that a mechanism for tracking completion of security awareness training had been established. In addition, 21 of the 22 operational investments (for which information security plans are required) had security plans that addressed areas such as the rules of behavior for system use and technical security controls. In contrast, about 77 percent of these 22 investments did not provide support describing how incident handling activities would be performed at a system level, such as detecting, reporting, and mitigating risks associated with security incidents.

While the compliance of security documentation with federal requirements was mixed, the documented support for the analysis of alternatives and the project and funding plan areas of the exhibit 300 provided little assurance that investments complied with applicable guidance and laws. None of the investments had cost analysis documentation that fully complied with Circulars A-94 and A-11 criteria (lacking, for example, a comparison of at least three alternatives, a discussion of assumptions for each alternative, or an analysis of uncertainty).

Project officials attributed deficiencies in the analysis of alternatives to, among other things, a lack of understanding of what was expected for reporting in the exhibit 300. In a few instances, officials noted that they believed that their investments were excluded from meeting the federal requirements because the investments were near the end of their operational or, in some cases, useful life cycles. OMB guidance on analysis of alternatives does not differentiate between operational and developmental investments; nonetheless, one agency’s internal guidance explicitly states that no analysis of alternatives is necessary for investments in the steady state (that is, operational). However, a forward-looking analysis of alternatives for operational investments can help agencies recognize when an alternative solution may be more efficient or effective than the current investment, thereby freeing scarce resources to be reallocated.

The agencies’ lack of compliance with OMB guidelines for analysis of alternatives, including the cost-benefit analysis, leaves senior executive managers at risk of making poor investment management decisions on incomplete and sometimes inaccurate information.

For the project and funding plan section of the exhibit 300, OMB Circular A-11 provides guidance on the information to be provided, which depends upon the state of the investment (i.e., new, mixed life cycle, or steady state). According to this guidance, information presented in the project and funding plan is to be derived from one of two types of analysis: for steady state investments, an operational analysis, and for new and mixed life-cycle investments, an analysis based on an EVM process that is compliant with ANSI/EIA-748-A. Operational analysis is a method for assessing the technical merits of an existing investment in meeting user needs, while EVM is a method for assessing the value of work performed compared to its actual cost during development of an investment.

Of the eight steady state investments we reviewed, only two had conducted an operational analysis. Furthermore, only one of those had documented procedures that were in accordance with OMB’s Capital Programming Guide criteria, such as addressing user needs and technical performance. In most cases for which no operational analyses were in place, agency officials commented that OMB guidance describing how to perform an operational analysis was at such a high level of generality that they found it difficult to follow. Instead of attempting to devise and perform an operational analysis, therefore, they implemented variations on an EVM process. However, these implementations of EVM did not address topics required for the operational analysis, such as user needs and technical performance. Unless they address these topics, agencies may not have the information they need to determine, among other things, whether investments are performing as intended and meeting user needs.

Similarly, of the 21 new and mixed life-cycle investments required to use EVM, only 6 used an EVM process that generally followed the ANSI standard. Since fiscal year 2002, OMB has required the use of EVM as a project management tool. The ANSI standard is intended to ensure that data produced by an EVM process are reliable so as to allow objective reports of project status, produce early warning signs of impending schedule delays and cost overruns, and ultimately provide unbiased estimates of anticipated costs at completion. If agencies do not implement EVM processes that follow the ANSI standard, they have reduced assurance that the information used for tracking the cost, schedule, and performance of the investment is reliable.

For the remaining 15 investments that did not have EVM processes following the required standard, project officials commented that EVM was relatively new to them and that they did not understand how to implement an ANSI-compliant process at the time of the fiscal year 2006 submission. At the time of our review, all five departments stated that they were working toward implementing compliant processes.

To OMB’s credit, it recognized the need for improvement in the execution of agencies’ IT projects and has issued clarifying guidance on the implementation of EVM. This guidance, issued in August 2005, could be expected to have an impact on the exhibit 300s prepared for fiscal year 2008. Under this guidance, agencies are instructed, among other things, to develop comprehensive agency policies for using EVM to plan and manage development activities for major IT investments no later than December 31, 2005; include a provision and clause in major acquisition contracts or agency in-house project charters directing the use of an EVM system compliant with the required standard; and provide documentation demonstrating that the contractor’s or agency’s in-house EVM system complies with the required standard and conduct periodic surveillance reviews.

Additionally, the Civilian Agency Acquisition Council and the Defense Acquisition Regulations Council published in the Federal Register a proposed amendment to the Federal Acquisition Regulation (FAR Case 2004-019) to standardize EVM contract policy across the government.

In previous work, we have reported that EVM can have a significant impact on the success of an IT acquisition because it heightens visibility into whether a program is on target with respect to cost, schedule, and technical performance. Therefore, it is important that the process is implemented properly to maximize its value as a project management tool. If it is not implemented effectively, agency executives and OMB risk making poor investment decisions based on inaccurate and potentially misleading EVM information.

Cost Data Supporting Business Cases Were Unreliable

Accurate and timely cost management information is critical for federal managers to understand the progress of major projects and vital in developing meaningful links among budget, accounting, and performance. The Federal Financial Management Improvement Act of 1996 emphasizes the need for agencies to have systems that are able to generate reliable, useful, and timely information for decision-making purposes and to ensure accountability on an ongoing basis. In previous work, we have reported on the lack of adherence to federal accounting standards throughout the federal government and have made recommendations that agencies improve cost-accounting systems.

At every agency, cost information reported in the 29 exhibit 300s was derived from ad hoc processes rather than from cost-accounting systems with adequate controls to ensure accountability. This condition had impact in two particular areas of the exhibit 300—the summary of spending table and the project and funding plan section: Figures for dollars expended for the prior year (in this case, fiscal year 2004) were not reliable. In all cases, documentation provided to support prior year cost figures in the summary of spending table showed that the information was derived from ad hoc sources, such as spreadsheet estimates, handwritten figures, or e-mails. Therefore, the cost data reported in the exhibit 300 are not verifiable.

Information in the project and funding plans was also unreliable for the 21 new and mixed life-cycle investments required to use EVM. As discussed earlier, 15 of these investments reported cost figures based on EVM processes that did not follow the ANSI standard; because the standard was not followed, these processes did not have the controls necessary to ensure that the data they produced were reliable. The other 6 investments had ANSI-compliant EVM processes in place for the contractor component of the investment costs, but the government component of the investment costs was derived from ad hoc systems (such as tracking government costs in spreadsheets based on project managers’ own records); thus, that portion of the data was not reliable, lending a degree of unreliability to the overall EVM reports.

The lack of accurate cost figures limits decision makers’ ability to determine the actual resources expended on an investment, and therefore inhibits their ability to make fully informed decisions on whether to proceed. Without reliable systems that meet federal standards, government agencies cannot produce reliable cost-based performance information.

Conclusions

The usefulness of the exhibit 300 business case as a mechanism to support the selection and oversight of federal IT investments is undercut by the kinds of weaknesses displayed in the 29 business cases that we reviewed. Although we cannot directly project these examples to the more than one thousand business cases developed each year across the federal government, our results suggest that the issues raised need attention. The shortcomings in guidance and training are likely to be widespread, and so the weaknesses may extend beyond the specific examples identified here. The kinds of weaknesses displayed and the causes behind them are consistent with the pervasive problems with project and investment management that we have documented in numerous prior reports.

The absence of documentary support in the cases reviewed raises questions regarding the sufficiency of the justification provided for the investment and undermines the management decisions being made based on it. More troubling, it may indicate an underlying weakness in the management of the investment, particularly since several sections of the exhibit are specifically designed to capture information from systems used in project management, such as those that support EVM and financial management. In many cases, inadequate support raises questions regarding the adequacy of an agency’s management processes and internal controls, which strongly affect the reliability of the information presented to decision makers. Further, in view of the inaccuracies in the cases reviewed, it is evident that agencies are not taking sufficient actions to ensure the accuracy of the information in the exhibit 300s. To make reasonable decisions, management needs to be aware of limitations in the data on which they rely and thus be able to take steps to mitigate the risks involved.

Collectively, our findings raise questions on whether fundamental project management processes are in place, whether project managers are adequately trained in these processes, and whether they receive sufficient guidance on these processes and on preparing all areas of the exhibit 300. At a minimum, this situation undermines the usefulness of the exhibit 300 as a mechanism to communicate to OMB and agency executives that the project team has employed the disciplines of good project management. By reporting information that is not supported by documentation, the exhibit 300 can create the misleading appearance that investments are being managed properly, when in fact they are not. In addition, OMB has relied on these exhibits to identify and oversee high-risk projects; thus, our finding that the data being presented to OMB may not be reliable or accurate further complicates its oversight.

While OMB is applying more rigor to its oversight processes through such processes as its tracking of high-risk investments, these advances may be undermined by inaccurate or unreliable data used in decision making. Unless these weaknesses are addressed, OMB, agency executives, and Congress will not have assurance that key decisions to pursue and oversee the $65 billion in IT investments are being made based on accurate and reliable information.

Recommendations for Executive Action

To improve the accuracy and validity of exhibit 300s for major IT investments and to increase the value of using the information they provide in selection, oversight, and resource allocation decisions, we are making three recommendations. 1. Because decision makers should be aware of any weaknesses in the processes used to develop the information in the exhibit 300s, we are recommending that the Director of OMB direct agencies to determine the extent to which the information contained in each exhibit 300 is accurate and reliable. Where weaknesses in accuracy and reliability are identified, the agency should be required to disclose them and explain the agency’s approach to mitigating them.

In addition, to help ensure that agency personnel completing exhibit 300s better understand their responsibilities, we recommend that the Director of OMB take the following additional actions: 2.

In advance of OMB’s next issuance of the Circular A-11 update, develop and promulgate clearer and more explicit guidance for sections of the exhibit 300 business case that cause confusion, including addressing weaknesses identified in this report (as indicated below) and consulting with agency personnel having responsibility for completing exhibit 300s across government to identify other areas of confusion. At a minimum, the guidance should do the following: Provide a more detailed description of the requirements for completing an operational analysis, as referred to in the supplement to Part 7 of Circular A-11, the Capital Programming Guide.

Address or clarify possible flexibilities and alternative approaches available to agencies in completing their exhibit 300s: for example, whether the analysis of alternatives section of the exhibit 300 needs to be updated every year for steady state investments and whether all risk areas are relevant for all investments. 3. Provide for training of agency personnel responsible for completing exhibit 300s. This training should go beyond a description of changes from prior years’ guidance and include working through examples for a variety of investments. In developing the training, OMB should consult with agencies to identify deficiencies that the training should address.

In implementing these recommendations, OMB should work with the CIO Council to develop the necessary guidance and implement an effective training program to ensure governmentwide acceptance of these changes.

Because we have outstanding recommendations aimed at enhancing OMB’s audit guidance related to federal cost-accounting systems, we are not making any new recommendations in this report regarding federal cost accounting.

Agency Comments and Our Evaluation

We provided a draft of this report to OMB and the five agencies whose exhibit 300s we reviewed.

In written comments received on December 23, 2005, the Administrator of OMB’s Office for E-Government and Information Technology accepted the findings of the draft report. OMB described two of our three recommendations and expressed three concerns: first, that our report does not address the need for agencies to ensure the accuracy of their IT investment requests; second, that the report focuses on the way agency employees fill out OMB’s exhibit 300s and not on the underlying management responsibilities; and third, that by directing our recommendations to OMB rather than to the agencies, we could be seen as suggesting that OMB and not the agencies are responsible for data accuracy and employee training.

OMB’s concern regarding data accuracy is addressed by our first recommendation: that the Director of OMB instruct agencies to determine the extent to which the information contained in each exhibit 300 is accurate and reliable, to disclose weaknesses, and to describe the agency’s approach to mitigating these weaknesses. This recommendation clearly places responsibility on the agencies for assessing the quality of their budget information and the processes that produced this information.

With respect to OMB’s concern that the recommendations do not focus on how well agencies fulfill their underlying information resources management responsibilities, our view is that our recommendation on disclosing and mitigating weaknesses does address these underlying responsibilities. The report specifically addresses the exhibit 300s and the reliability of these documents when used as support in the agencies’ and OMB’s decision-making processes. As our report clearly states, the lack of documentation may indicate an underlying weakness in the management of the investment. In many cases inadequate support raises questions about the investments’ program management and internal controls. Requiring agencies to disclose and mitigate associated weaknesses presupposes that agencies examine and address their approach to fulfilling information resources management responsibilities.

Regarding OMB’s third concern, we do not intend to suggest that agencies are not responsible and accountable for the weaknesses we describe. We place significant responsibility on agencies to manage their information assets effectively, as reflected in our first recommendation and in the large number of evaluations that we have previously conducted at individual agencies and the recommendations resulting, some of which are still outstanding. In this report, however, our recommendations are directed to OMB because they address findings relating to OMB-required budget documents, and OMB has statutory responsibility for providing information resources management guidance governmentwide.

Regarding OMB’s comment that agencies be held responsible for employee training in information resources management, we agree that agencies are responsible for such training. However, as agencies indicated during the review, additional training by OMB would be helpful, especially in the understanding of OMB’s requirements for the exhibit 300. This is also consistent with OMB’s responsibility under the E-Government Act of 2002 to identify where current training does not satisfy the personnel needs related to information technology and information resource management.

The Deputy Associate Chief Information Officer for Information Technology Reform of the Department of Energy provided largely technical comments, which we incorporated as appropriate. The Director of Audit Relations of the Department of Transportation also provided technical comments that were incorporated as appropriate. The Departments of Agriculture, Commerce, and the Treasury provided no comments.

The written comments from OMB are reproduced in appendix II.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretaries of the Departments of Agriculture, Commerce, Energy, Transportation, and the Treasury and the Director of Office of Management and Budget. We will also make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions on matters discussed in this report, please contact me at (202) 512-9286. I can also be reached by e-mail at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other contacts and key contributors to this report are listed in appendix IV.

Objective, Scope, and Methodology

Our objective was to ascertain the extent to which selected agencies have underlying support for the information described in their fiscal year 2006 exhibit 300s as submitted to the Office of Management and Budget (OMB) in September 2004. To address our objective, we reviewed the supporting documentation for 29 exhibit 300s from agencies and components from the Departments of Agriculture, Commerce, Energy, Transportation, and the Treasury.

We selected the five departments for our review on the basis of two criteria. First, to ensure that we examined significant investments, we selected departments that expected to spend $1 billion or more on information technology (IT) investments in fiscal year 2006. Second, of those agencies with significant investments, we further narrowed our selection of agencies to those with the first and second largest number of IT investments in each of three categories of the federal government’s Business Reference Model (BRM): Services for Citizens, Support Delivery of Services, and Management of Government Resources. We did this to ensure that the agencies under review reflect the primary business operations performed by the federal government. We excluded the Mode of Delivery Business Area because we found investments in this area to be largely from one agency, the Department of Defense (DOD). (In general, Mode of Delivery describes the mechanisms the government uses to achieve its purposes: Services for Citizens.) (We excluded DOD and the Department of Homeland Security (DHS) from our selection, because the Defense Inspector General recently performed an extensive review of exhibit 300s, and we have both completed and ongoing work on several major IT investments at DHS). This process resulted in the selection of the five departments mentioned above.

To make our selection of IT investments from the five departments, we used OMB capital planning and budget documentation to identify a mix of investments. Specifically, we chose IT investments that (1) supported government operations across each of the three BRM business areas identified above and (2) reflected different stages of investment (e.g., new, mixed life cycle, and steady state). Initially, we selected a total of 30 investments (i.e., 6 investments from each department). However, one IT investment was dropped from our total of 30 selected investments because we determined during our review that OMB and the agency had cancelled its funding.

To determine the extent of each investment’s underlying support, we developed a set of questions regarding the types of analysis and documentation that were associated with the information provided in each of the major sections of OMB’s exhibit 300. Using our set of questions, we met with agency officials for each selected investment to collect and analyze investment documentation associated with each exhibit 300 area in our evaluation. We further compared the documentation against the exhibit 300 to ascertain whether the documentation agreed with what the investment reported in the exhibit 300. Where federal requirements, laws, and other guidelines were cited in Circular A-11, we also used these to assess the extent to which agencies and components had complied with specific documentation requirements as prescribed in these sources (including National Institute of Standards and Technology (NIST) guidance, OMB circulars, and OMB memorandums).

In areas where federal directives were cited in the exhibit 300, we conducted limited reliability testing; these areas included security, analysis of alternatives, and the project and funding plan. In our evaluation of security documentation, we used criteria set forth in NIST guidance to assess whether the major components were present in key documents, which included the security plan and system-level incident handling procedures. For security awareness training, we identified whether training was conducted and tracked but did not assess its content. In our evaluation of the analysis of alternatives, we used criteria from OMB Circular A-94 to assess whether the major components were present in the cost-benefit or cost-effectiveness analysis. In cases where investment managers told us that their earned value management (EVM) processes were in conformance with ANSI/EIA-748-A (for our evaluation of the project and funding plan sections), we used criteria from ANSI/EIA-748-A to assess whether key EVM processes were in place. We did not test the quality of the documentation in these areas of evaluation.

Regarding the reliability of cost data, we did not test the adequacy of agency or contractor cost-accounting systems. Our evaluation was based on what we were told by the agencies and the information they could provide (to the extent to which they had information).

We performed our work at the agencies’ offices in the Washington, D.C., metropolitan area. We conducted our review between March and November 2005 in accordance with generally accepted government auditing standards.

Comments from the Office of Management and Budget

Descriptions of Investments Reviewed

The following provides additional detail on the agencies and investments that we reviewed as part of this audit. We reviewed a total of 29 investments at five departments: Agriculture, Commerce, Energy, Transportation, and the Treasury. The selected departments account for the first and second largest number of IT investments in each of three categories of the federal government’s Business Reference Model: Services for Citizens, Support Delivery of Services, and Management of Government Resources.

According to OMB guidance, the life-cycle stage of the asset affects what is reported on the exhibit 300: New investments (i.e., proposed for budget year or later, or in development) must be justified based on the need to fill a gap in the agency’s ability to meet strategic goals and objectives with the lowest life-cycle costs of all possible alternatives and must provide risk- adjusted cost and schedule goals and measurable performance benefits.

Mixed life-cycle investments (i.e., investments that are operational but include some developmental effort, such as a technology refresh) must demonstrate satisfactory progress toward achieving baseline cost, schedule, and performance goals using an EVM system.

Operational investments (i.e., steady state) must demonstrate, among other things, how close actual annual operating and maintenance costs are to the original life-cycle cost estimates, whether the technical merits of the investment continue to meet the needs of the agency and customers, and that an analysis of alternatives was performed with a future focus.

Department of Agriculture

System: Comprehensive Electronic Permit System (ePermits)

Brief description: This system is expected to automate processes to allow the Department of Agriculture to issue, track, and rapidly verify the validity of a federal permit allowing the importation of plants and animals. It is also expected to assist the public by allowing applicants to apply for permits, check the status of permit applications, and receive permits online.

System: Corporate Financial Management Systems (CFMS)

Brief description: This investment is designed to represent the entire portfolio of current corporate financial management and administrative payment systems for the department. It is a corporatewide solution for financial management reform and systems integration that provides tools for program and financial managers to manage and evaluate federal programs.

System: Integrated Acquisition System (IAS)

Brief description: This system is intended to be a single enterprisewide acquisition management system to support a strategic and more standardized acquisition management process for Agriculture. It is expected to provide a real-time interface to the department’s core financial system, reliable data, and a shortened time for acquiring goods and services.

System: Phytosanitary Certificate Issuance and Tracking System (PCIT)

Brief description: This system is expected to establish a new process to collect and track phytosanitary certificates issued by the department, which attest to compliance with import regulations of importing countries. It is also intended to provide better service to users by reducing the need for repetitive data entry from applicants and enabling certifying officials to deliver certificates in a timelier manner.

System: Processed Commodity Inventory Management System (PCIMS)

Brief description: This system is designed to support the annual acquisition, tracking, and distribution of commodities acquired by Agriculture for domestic and foreign food assistance programs by providing financial and program management, reporting, and control to track commodity requests against purchases and distributions from inventory.

System: Store Tracking and Redemption System Redesign (STARS II)

Brief description: This system is intended to support the department’s Food-Stamp program mission by tracking and monitoring food coupon/electronic benefit redemption activities and regulatory violations by businesses and associated administrative actions related to enforcement of penalties, among other things. This initiative is expected to replace the current legacy system, which has been in place since 1993.

Department of Commerce

System: Advanced Weather Interactive Processing System (AWIPS)

Brief description: This system is designed to be an interactive computer system that integrates all meteorological and hydrological data and all satellite and radar data to enable the forecaster to prepare and issue more accurate and timely forecasts and warnings.

System: Comprehensive Large Array-Data Stewardship System (CLASS)

Brief description: This system is expected to provide an integrated solution to weather and water data archive and access, including an access portal with search, browse, and geospatial capabilities for users to obtain environmental data, contributing to improvements in prediction capabilities.

System: Economic Census and Surveys (ECON)

Brief description: This investment is designed to provide statistical programs that count and profile U.S. businesses and government organizations through the gathering of surveys and principal economic indicators in order to conduct research and technical studies.

System: Improve the Automated Export System (AES)

Brief description: The current system is designed to expedite monthly statistics on international trade, remedy shortcomings in export statistics, and help to control the export of weapons or other hazardous items that could be a threat to U.S. national security or public welfare. The proposed initiative is designed to improve the current system to handle electronic filing of all export transactions, incorporate an electronic manifest system, and provide for verification of export information reported on export transactions.

System: National Weather Service Telecommunication Gateway (NWSTG) System

Brief description: The current system is designed to collect and distribute raw and processed hydrometeorological data and products, disseminating weather observations and guidance to a national and international community of customers. Improvements to current system are expected to provide sufficient performance, capacity, and catastrophic backup capability to meet current and future demands for data.

System: Satellite Operations Control Center Command and Data Acquisition (SOCC/CDA)

Brief description: This system is designed to command and control Commerce’s operational environmental satellites and to acquire and manage the weather and water data the satellites collect, in order to provide support functions that are not available commercially, such as real- time hurricane support.

Department of Energy

System: Energy Sciences Network (ESnet)

Brief description: This project is designed to support scientific research by providing an interoperable, effective, and reliable communications infrastructure and network services to the Department of Energy research facilities.

System: E-content Management System (eCMS)

Brief description: This system is expected to be an enterprisewide, integrated document and records management system that will include portal accessibility and integration with knowledge management tools in order to improve decision and service delivery quality and serve as a resource for operations management.

System: Integrated Planning, Accountability, and Budgeting System Information System (IPABS- IS)

Brief description: This system is designed to support the routine collection and reporting needs of Energy for life-cycle planning, budget formulation, and project and budget execution.

System: Licensing Support Network (LSN)

Brief description: This is a Web-based system that is intended to make relevant documentary material supporting the Nuclear Regulatory License Application available to users, as part of the requirements of the Nuclear Waste Policy Act.

System: Los Alamos National Laboratory Enterprise Project (LANL ERP)

Brief description: This investment is intended to identify, design, and implement the systems, processes, and controls related to financial management, human resources, supply chain management, facilities maintenance, information management, project management, and manufacturing in order to lower costs and provide more efficient operations and improved management.

Department of Transportation

System: Asset Supply Chain Management (ASCM)

Brief description: This investment is intended to provide the Department of Transportation (DOT) with asset management and supply chain management information systems to track and manage over $21 billion in federal government assets. Reducing the number of information systems, optimizing supply chain operations, and streamlining business operations of employees are expected to result in reduced costs to the agency.

System: DOT Financial System Consolidation

Brief description: This program is expected to consolidate several major and nonmajor DOT financial systems to interface or integrate all related systems in order to eliminate redundant data and processes.

System: National Transit Database (NTD)

Brief description: This system is designed to collect performance data from over 640 local transit agencies for the purpose of reporting statistical data on the U.S. transit industry.

System: Next Generation Air/Ground Communications (NEXCOM)

Brief description: This system is intended to provide air pilot/controller voice and data communications by utilizing a digital-based air/ground communication system.

System: System Approach for Safety Oversight (SASO)

Brief description: This system is expected to consolidate the agency’s 28 oversight systems on aviation regulatory compliance into 5 integrated aviation safety risk management systems. Its intended purpose is to allow applicable government agencies and the aviation industry to use common system safety applications and databases for managing and overseeing flight safety.

System: Wide-Area Augmentation System (WAAS)

Brief description: This is a navigation system that is designed to provide navigation across the entire United States for all classes of aircraft in all flight operations, including en-route navigation, airport departures, and airport arrivals including precision landing approaches in all weather conditions.

Department of the Treasury

System: Customer Account Data Engine (CADE)

Brief description: This system is part of a modernization program that is expected to provide the Department of the Treasury with the capability to manage its tax accounts utilizing new technology, applications, and databases. This system is designed to create applications for daily posting, settlement, maintenance, refund processing, and issue detection for taxpayer tax account and return data to improve customer service and compliance.

System: Debt Management Accounting System (DMAS)

Brief description: This system is designed to be a financial accounting system for activities associated with Treasury’s debt collection program to track funds recovered by the agency, post these funds to the proper account in an accurate and timely manner, and transfer moneys due to the appropriate government agencies. The system is also designed to record the general ledger activity and produce operational, management, and standard external reports.

System: Electronic Management System (EMS)

Brief description: This system is designed to be a front-end processing system that receives, validates, stores, forwards to mainframe electronic filing systems, and acknowledges electronic files containing tax documents. The system is intended to receive returns from third parties, acknowledge the receipt of information, format the information for mainframe processing, provide acknowledgements to the third parties, and send state return data to participating states.

System: Governmentwide Accounting and Reporting Modernization (GWA)

Brief description: This system is designed to produce accurate, accessible, and timely governmentwide financial information through the streamlining of reports and the reduction of the reconciliation burden on government agencies in order to minimize the amount of labor necessary to transfer financial information.

System: Service Center Recognition/Images Processing System (SCRIPS)

Brief description: This system is intended to be a data capture, management, and storage system used to process tax documents automatically in order to meet mandated timelines and processing requirements for various tax forms and the Federal Tax Deposits, which directly impacts revenue brought into the federal treasury.

System: Secure Payment System (SPS)

Brief description: This system is designed to be a browser-based Internet version of the current Electronic Certification System, which will allow federal program agencies to submit certified requests for payment disbursement online. It is intended to provide a more secure payment process, increase the ability to protect sensitive financial and privacy data, and improve the financial performance of federal program agencies by providing program agencies a method of providing financial data to Treasury.

GAO Contact and Staff Acknowledgments

GAO Contact

David A. Powner, (202) 512-9286, pownerd@gao.gov.

Acknowledgments

In addition to the contact named above, the following people made key contributions to this report: Carol Cha, Barbara Collier, Joseph Cruz, Lester Diamond, Valerie Hopkins, Sandra Kerr, Linda Lambert, Tammi Nguyen, Chris Owens, Mark Shaw, Kevin Walsh, and Martin Yeung.