Business Systems Modernization: Scope and Content of DOD's Congressional Report and Executive Oversight of Investments Need to Improve
Released: 2010-05-24T00:00:00

Letter

Background

DOD is a massive and complex organization entrusted with more taxpayer dollars than any other federal department or agency. Organizationally, the department includes the Office of the Secretary of Defense, the Joint Chiefs of Staff, the military departments, numerous defense agencies and field activities, and various unified combatant commands that are responsible for either specific geographic regions or specific functions. (See fig. 1 for a simplified depiction of DOD’s organizational structure.)

In support of its military operations, the department performs an assortment of interrelated and interdependent business functions, including logistics management, procurement, health care management, and financial management. As we have previously reported, the DOD systems environment that supports these business functions is overly complex and error prone, and is characterized by (1) little standardization across the department, (2) multiple systems performing the same tasks, (3) the same data stored in multiple systems, and (4) the need for data to be entered manually into multiple systems. Moreover, the department’s nonintegrated and duplicative systems impair its ability to combat fraud, waste, and abuse. The department recently reported that this systems environment is composed of approximately 2,480 separate business systems. For fiscal year 2010, DOD requested about $15.5 billion in funds to operate, maintain, and modernize its business systems and associated information technology (IT) infrastructure.

DOD currently bears responsibility, in whole or in part, for 15 of the 31 programs across the federal government that we have designated as high risk because they are highly susceptible to fraud, waste, abuse, and mismanagement. Eight of these areas are specific to the department, and 7 other high-risk areas are shared with other federal agencies. Collectively, these high-risk areas relate to DOD’s major business operations that are inextricably linked to the department’s ability to perform its overall mission, directly affect the readiness and capabilities of U.S. military forces, and can affect the success of a mission. DOD’s business systems modernization is one of the high-risk areas, and it is an essential enabler to addressing many of the department’s other high-risk areas. For example, modernized business systems are integral to the department’s efforts to address its financial, supply chain, and information security management high-risk areas.

DOD’s Institutional Approach to Business Systems Modernization

The National Defense Authorization Act (NDAA) for Fiscal Year 2008 designates the Deputy Secretary of Defense as the Chief Management Officer (CMO) for DOD and created a Deputy CMO position. The CMO’s responsibilities include developing and maintaining a departmentwide strategic plan for business reform and establishing performance goals and measures for improving and evaluating overall economy, efficiency, and effectiveness and monitoring and measuring the progress of the department. The Deputy CMO’s responsibilities include recommending to the CMO methodologies and measurement criteria to better synchronize, integrate, and coordinate the business operations to ensure alignment in support of the warfighting mission. The Business Transformation Agency (BTA) supports the Deputy CMO in leading and coordinating business transformation efforts across the department. This includes maintaining and updating the department’s enterprise architecture for its business mission area.

The CMO and Deputy CMO are to interact with several entities to guide the direction, oversight, and execution of DOD’s business transformation efforts, which include business systems modernization. These entities include the Defense Business Systems Management Committee (DBSMC), which serves as the highest-ranking investment review and decision- making body for business systems modernization activities and is chaired by the Deputy Secretary of Defense; the principal staff assistants, who serve as the certification authorities for business system modernizations in their respective core business missions; the investment review boards (IRB), which are chaired by the certifying authorities and form the review and decision-making bodies for business system investments in their respective areas of responsibility; and the BTA, which supports IRBs and leads and coordinates business transformation efforts across the department. (Table 1 lists these entities and provides greater detail on their roles, responsibilities, and composition.)

Overview of DOD’s Tiered Accountability for Business Systems Modernization

Since 2005, DOD has employed a “tiered accountability” approach to business systems modernization. Under this approach, responsibility and accountability for business architectures and systems investment management are assigned to different levels in the organization. For example, the BTA is responsible for developing the corporate BEA (i.e., the thin layer of DOD-wide policies, capabilities, standards, and rules) and the associated enterprise transition plan (ETP). Each component is responsible for defining a component-level architecture and transition plan associated with its own tiers of responsibility and for doing so in a manner that is aligned with (i.e., does not violate) the corporate BEA. Similarly, program managers are responsible for developing program-level architectures and plans and for ensuring alignment with the architectures and transition plans above them. This concept is to allow for autonomy while also ensuring linkages and alignment from the program level through the component level to the corporate level. (Table 2 describes the four investment tiers and identifies the associated reviewing and approving entities.)

Consistent with the tiered accountability approach, the NDAA for Fiscal Year 2008 required the secretaries of the military departments to designate the department under secretaries as CMOs with primary responsibility for business operations. Moreover, the Duncan Hunter NDAA for Fiscal Year 2009 required the military departments to establish business transformation offices to assist their CMOs.

Summary of Fiscal Year 2005 NDAA Requirements

Congress included provisions in the NDAA for Fiscal Year 2005 that are aimed at ensuring DOD’s establishment and implementation of effective investment management structures and processes. According to the act, DOD is required to develop a BEA; develop an ETP for implementing the architecture; identify each business system proposed for funding in DOD’s fiscal year budget submissions; delegate the responsibility for business systems to designated approval authorities within the Office of the Secretary of Defense; require each approval authority to establish investment review structures and processes; and, effective October 1, 2005, not obligate appropriated funds for a defense business system modernization with a total cost of more than $1 million unless the approval authority certifies that the business system modernization meets several conditions.

The NDAA for Fiscal Year 2005 also requires that the Secretary of Defense submit to congressional defense committees an annual report on the department’s compliance with the act’s provisions. This report is to 1. describe actions taken and planned for meeting the act’s requirements, a) specific milestones and actual performance against specified performance measures and any revision of such milestones and performance measures and b) specific actions on the defense business system modernizations submitted for certification under such subsection; 2. discuss specific improvements in business operations and cost savings resulting from successful defense business systems modernization efforts; 3. identify the number of defense business system modernizations certified; and 4. identify any defense business system modernization with an obligation in excess of $1 million during the preceding fiscal year that was not certified as required, and the reasons for the waiver.

Summary of Recent GAO Reviews of DOD’s Business Systems Modernization and Business Transformation Efforts

Between 2005 and 2008, we reported that DOD had taken increasing steps to comply with key requirements of the NDAA for Fiscal Year 2005 relative to architecture development, transition plan development, budgetary disclosure, and investment review and to satisfy relevant systems modernization management guidance, but that much remained to be accomplished relative to the act’s requirements and relevant guidance. Nevertheless, we concluded that the department had made important progress in defining and beginning to implement institutional management controls (i.e., processes, structures, and tools).

Notwithstanding this progress, in May 2009, we reported that the pace of DOD’s efforts in defining and implementing key institutional modernization management controls had slowed compared with progress made in each of the last 4 years, leaving much to be accomplished to fully implement the act’s requirements and related guidance. For example: The corporate BEA had yet to be extended (i.e., federated) to the entire family of business mission area architectures, including using an independent verification and validation agent to assess the components’ subsidiary architectures and federation efforts.

The fiscal year 2009 budget submission included some, but omitted other key information about business system investments, in part because of the lack of a reliable, comprehensive inventory of all defense business systems.

IT investment management policies and procedures at the corporate and component levels were not fully defined.

The business system information used to support the development of the transition plan and DOD’s budget requests, as well as certification and annual reviews, was of questionable reliability.

Business system investments costing more than $1 million continue to be certified and approved, but these decisions were not always based on complete information.

Accordingly, we reiterated existing recommendations to address each of these areas and further recommended that DOD, among other things, improve the quality of investment-related information. DOD partially agreed with our recommendations and described actions being planned or under way to address them. DOD is currently in the process of addressing these recommendations.

DOD’s Annual Report Discusses Programs’ Performance Against Milestones, but Does Not Address Performance Against Other Program Commitments

The act requires that DOD’s report describe milestones for business system modernization programs and actual performance against performance measures. In addition, the act requires that the report specify any revisions to milestones and performance measures.

To its credit, DOD’s annual report includes milestones, performance against milestones, and milestone revisions for 76 programs. However, other important performances measures, which typically include measures associated with determining progress against program cost, capability, and benefit commitments, are not included in the report.

BTA officials cited various reasons for the scope and content of the information provided and not provided, but these reasons are at odds with other aspects of its report. Without including information on program performance against, and revisions to, such key measures as cost, capability, and benefit commitments, DOD is not providing Congress with the information needed to inform its oversight of business system modernization programs.

DOD’s Report Includes Program Milestones as Well as Revisions to and Performance Against These Milestones

Consistent with the act’s requirement that DOD report on specific milestones and revisions to the milestones, DOD’s March 2010 report includes a summary of the status of milestones that were to be completed during fiscal 2009, the revisions associated with these milestones (e.g. delayed or deleted), and the reason for the revision. Specifically, the report lists three categories of milestones: Standard acquisition milestones: key events and dates that are provided for under DOD’s system acquisition process.

BEA compliance milestones: time frames for addressing specific IRB certification conditions related to ensuring BEA compliance.

Interim milestones: key events and dates to supplement DOD’s system acquisition process milestones (e.g., implementing specific system capabilities or upgrading infrastructure by a given date).

DOD’s report includes a total of 224 milestones that collectively span 76 programs. Of these 224 milestones, 35 are standard acquisition milestones, 22 are BEA compliance milestones, and 167 are interim milestones. The report also discusses performance against these milestones. Specifically, of the 224 milestones, 56 percent are reported to have been met, while 21 and 23 percent are reported to have been deleted (i.e., determined to be unnecessary) or not met, respectively. (See fig 2.) With respect to acquisition and compliance milestones, the percentage of milestones that were reported as not being met was 66 and 50 percent, respectively.

DOD’s Report Does Not Address Performance Against Other Program Commitments

Beyond milestones, the act requires DOD’s annual report to address actual performance against performance measures and any revision of these performance measures. As we have previously reported, meaningful information about program performance is typically measured in terms of program cost, capability, and benefit commitments, in addition to schedule commitments. Through such a range of performance measures, valuable insight into the health and success of a business system investment can be gained.

As we previously discussed in this report, DOD’s annual report does report schedule commitments (i.e., milestones) for its modernizing programs. While DOD’s annual report also includes examples of business improvements and costs savings, this program performance information is not reported against performance measure baselines. Further, the report remains silent with respect to other important performance measures such as progress made against cost and capability commitments, which would allow congressional decision makers to understand the extent to which programs are meeting cost, capability, and benefit commitments.

BTA officials stated they focused the annual report on programs that had planned milestones during fiscal year 2009. Further, they said they focused on program milestones because most of the investments covered by the report have not progressed far enough in their life cycles to measure cost, capability, and benefit performance. In addition, the annual report states that DOD does not include performance measures in its annual report for any system that has reached initial or full operational capability and is no longer modernizing. Nevertheless, the report includes descriptions of a number of programs that have progressed to the point where DOD reports on actual operational efficiencies and dollar savings that have accrued, which, in turn, means that these programs have progressed to the point that DOD can report on progress against defined performance commitments, such as the costs that have been incurred, the capabilities that have been delivered, and benefits that have been realized. Moreover, the programs that have not yet delivered capabilities or realized benefits have incurred costs, which DOD can report relative to expected costs.

Establishing performance measures and monitoring actual-versus- expected performance using the full range of measures are essential to understanding the health of any IT investment. By not including information on each program’s performance against defined cost, capability, and benefit commitments, DOD is not providing Congress with important information for informing its oversight of business system modernization investments.

DOD’s Report Discusses Business Systems Modernization Programs’ Improvements in Business Operations and Cost Savings

The act requires that DOD’s annual report discuss specific improvements in business operations and cost savings resulting from successful business system modernization programs. DOD met this requirement by including 18 “case in point” examples in the report. Among other things, each narrative generally describes the program and provides high-level information on system capabilities delivered and benefits achieved to date. Specific examples include: The Air Force Recruiting Information Support System is to be the primary Web-based recruiting system for the Air Force. According to the annual report, the Air Force’s legacy recruiting information system has been slow and, at times, unavailable to users. Modernization of the system (e.g., new hardware and software upgrades) is reported to have improved recruiters’ productivity by reducing the wait time for processing recruiter requests. Other reported improvements include allowing recruiters to build applicant files off line and upload them at a later time and reducing the recruiter’s dependence on Internet connectivity. Future plans include merging the Air Force Recruiting Information Support System with the Air Force Reserve Recruiting System to increase functionality and decrease system response time.

The Army Learning Management System is a Web-based system for training, scheduling, and career planning for soldiers. According to the annual report, in 2009 the system contributed to an 88 percent increase in the number of student accounts, 111 percent increase in the number of courses offered, and 157 percent increase in the number of course completions.

Wide Area Work Flow is a Web-based system to centralize and automate DOD’s largely manual business payment process. The annual report states that the system has thus far allowed the cost of processing a payment to decrease from between $22 and $30 to between $6 and $12. Other cited benefits include allowing suppliers to have a single point of interface with DOD for payment invoicing, receipt, and acceptance.

The Defense Agencies Initiative is an enterprise resource planning (ERP) system to standardize and integrate enterprise data to support financial decisions in real time. According to the annual report, the system resulted in a reduction in the time it takes to post financial obligations from 60 days to less than 2 days, and a reduction in the time to close out monthly financial reports from 4 days to less than 1 day. Further, the report states that financial information is now available to BTA on a real- time basis and thereby enables proactive management of agency finances.

DOD’s Annual Report Does Not Describe Certification Actions for All Systems, and Justification of Reported Certification Actions Is Limited

Among other things, the act requires DOD’s annual report to describe specific actions the department has taken on each business system modernization investment submitted for certification. More specifically, the act states that such investments involving more than $1 million in obligations must be certified by a designated approval authority as meeting specific criteria, such as demonstrating compliance with DOD’s BEA. Further, the act requires the DBSMC to approve each of these certifications.

To its credit, DOD’s annual report identifies IRB certification actions associated with 116 business system investments. However, certification actions associated with 40 other investments are not included. Further, the bases for several of the fiscal year 2009 system certification actions and subsequent approvals are limited because program weaknesses and issues that our prior work has raised about, for example, the systems’ economic analyses and BEA compliance determinations, are not reflected in the reported certification actions.

According to BTA officials, only new certifications are included in the report, even though DOD guidance states that recertifications are also certification actions. Moreover, this guidance does not require programs to disclose program weaknesses and issues raised by us or others.

By not fully identifying in its annual report the certification actions taken on all business system modernization investments, DOD is not fully informing congressional oversight. Further, by not ensuring that all certifications reflect known program weaknesses, business system modernization program certification and approval decisions are not being fully informed and thus may not be adequately justified.

DOD Has Established an Approach to Certifying Business System Investments

DOD has established what it describes as a “tiered accountability” approach to meeting the act’s requirements for certifying business system investments. Under this approach, investment review begins within the military departments and defense agencies and advances through a hierarchy of review and decision-making authorities, depending on the size, nature, and significance of the investment. For those investments that meet the act’s dollar threshold, namely those with planned modernizations in excess of $1 million, this sequence of review and decision making includes component precertification, IRB certification, and DBMSC approval. For those investments that do not meet the dollar threshold, investment decision-making authority remains with the responsible component.

According to the department’s approach, reviews for modernization investments of more than $1 million focus on program alignment with the BEA; alignment with the department’s strategic mission, goals, and objectives; and oversight commensurate with the program’s cost, scope, and complexity. The approach further requires that these reviews be completed before a component can obligate modernization funds.

At the component level, program managers are responsible for the information about their respective programs in a central repository known as the Defense IT Portfolio Repository system (DITPR). The component precertification authority is responsible for precertifying that a given system investment is compliant with the BEA, reviewing the system funding requests, and ensuring that the IRB responsible for the investment receives complete, current, accurate, and timely information. The precertification authority is also responsible for “asserting” the status and validity of the investment information by submitting a component precertification letter to the responsible IRB.

At the corporate level, an IRB reviews the precertification letter and related material and makes a recommendation for a specific certification action for each of its investments. After the IRB makes its recommendation, it prepares a certification memorandum that documents the IRB’s decisions and any related conditions. The memorandum is forwarded to the DBSMC, which either approves or disapproves the IRB’s decision and issues a memorandum containing its decision. If the DBSMC disapproves a system investment’s certification, it is up to the component’s precertification authority to decide whether or not to resubmit the investment after it has resolved the reasons for the disapproval.

Under DOD’s approach, there are four types of certification actions: Certify: An IRB certifies the modernization as fully meeting criteria defined in the act and IRB investment review guidance, such as compliance with the BEA and the extent to which the investment is consistent with component and department IT investment portfolios, which are asserted by the component precertification authority.

Certify with conditions: An IRB certifies the modernization with the understanding that it will address specific IRB-imposed conditions. For example, the Army’s Real Estate Management Information System was certified with a condition to provide a plan for how data elements would comply with certain business rules in DOD’s BEA.

Recertify: An IRB certifies the obligation of additional modernization funds for a previously certified modernization investment. For example, the Air Force’s Cargo Movement Operations System was recertified in April 2009 for $6.3 million to be spent in fiscal years 2009 through 2012. This recertification was in addition to the $21.1 million previously certified in fiscal year 2007. In addition, a program must request IRB recertification if the program plans to redistribute previously approved modernization funds among multiple fiscal years and this redistribution will result in the funding for any given fiscal year exceeding the previously approved amount by 10 percent or more.

Decertify: An IRB reduces the amount of modernization funds available to an investment when the entire amount of funding is not to be obligated as previously certified. For example, the Defense Financial Accounting Service’s Standard Disbursing Initiative had about $5.5 million decertified because the funds were no longer needed. An IRB may also decertify a modernization after it has been completed. For example, DOD reported that $213,000 for the Air National Guard’s Reserve Order Writing System was decertified at the time the system was completed because the funds were no longer needed.

DOD’s Report Excludes a Number of Certification Actions Taken on Business System Modernizations

The act requires that DOD’s annual report describe specific actions taken for each business system investment submitted for certification. However, the department’s annual report discusses fiscal year 2009 certification actions on only 116 of the 156 systems on which certification actions were taken. More specifically, the report states that during fiscal year 2009, 92 business system modernizations were certified—32 with and 60 without conditions. For the 32 systems, 58 conditions were collectively reported. Examples of conditions cited in the report are the need for a system to comply with the Standard Financial Information Structure and the need to develop a plan for complying with the data standards of DOD’s Item Unique Identifier Registry. The report also identifies 24 decertifications. For example, the Air Force’s Enhanced Technical Information Management System had about $13.9 million in funding decertified (i.e., reduced), and the Defense Financial Accounting Service’s Standard Disbursing Initiative had about $5.5 million decertified.

However, fiscal year 2009 IRB and DBSMC decision memoranda and meeting minutes show that 40 systems had additional certification actions that were not included in DOD’s annual report. Of these 40 systems, 2 were certified without conditions, 2 were certified with conditions, and 36 were recertified. Collectively, DOD’s annual report omits about 26 percent of its certification actions. (See table 3 for a summary of actual, reported, and unreported certification actions.)

According to BTA officials, the excluded certification actions are all recertifications, which they said are intentionally not reported because they are not new certifications. They also told us that the four new certifications were, in fact, recertifications. However, DBSMC and IRB memoranda and meeting minutes identify these four certification actions as new certifications. Moreover, DOD guidance defines a recertification as a type of certification action, thus making it subject to the act’s reporting requirements. Without complete reporting of its certification actions, DOD is not in full compliance with its guidance, and DOD is limiting congressional visibility into the full scope of its business systems modernization efforts.

IRB Certification Actions Do Not Reflect and Disclose Known Limitations in Programs’ BEA Alignment, Economic Justification, and Management

According to DOD guidance, IRB certification addresses, among other things, a program’s alignment with the BEA and its management relative to factors such as system cost, scope, and complexity. To make a certification decision, IRBs rely on documentation submitted by the component precertification authority, including a certification dashboard, which includes cost and schedule status information; an economic viability analysis, which addresses the investment’s cost and benefits or cost effectiveness; and regulatory and standards compliance determinations. DOD guidance also gives IRBs broad authority in their certification reviews and actions, thus allowing each board to review and consider whatever investment-related information that it deems appropriate. Moreover, BTA and IRB officials told us that an IRB is not limited in the conditions it can place on a program.

The IRB certification actions described in DOD’s latest annual report are limited because they do not reflect significant limitations in the department’s basis for determining an investment’s alignment to the BEA. Specifically, we recently reported that key DOD BEA compliance assessments did not include all relevant architecture products, such as products that specify the technical standards needed to promote interoperability among related systems or examine overlaps with other business systems. In addition, we reported that these compliance assessments were not validated by DOD certification and approval entities. Despite these limitations, business systems modernization programs were certified as compliant with the BEA even though they did not adequately demonstrate such compliance. Accordingly, we recommended that DOD revise its BEA compliance guidance, tool, and IRB verification requirements to address these shortfalls. To date, DOD has yet to implement these recommendations, and thus the compliance determination weaknesses remain. Despite this, DOD’s latest annual report does not disclose these limitations on any of the 116 investment certification actions that it describes.

In addition, the fiscal year 2009 IRB certification actions described in the latest annual report are further limited in that they do not reflect weaknesses we have recently reported with the economic justification for and management of certain programs. For example, We reported in September 2009 that the Defense Readiness Reporting System (DRRS) program was not being effectively managed and made recommendations to address a number of acquisition management weaknesses including the absence of a reliable integrated master schedule and well defined and managed requirements, and adequate testing. As stated in our report, we briefed the DRRS program office on the results of our work prior to its DOD Human Resources Management IRB certification review. However, these results were not disclosed to the IRB. Rather, the certification package that the precertification authority submitted to the IRB stated that DRRS was on track for meeting its cost, schedule, and performance measures and highlighted no program risks. Based on this submission, DRRS was certified by the IRB and approved by the DBSMC to obligate $24.625 million in fiscal years 2009 and 2010. According to the chair of the IRB, the board did not validate the information in the submissions it received, and the results of our review were not disclosed to the IRB.

We reported in July 2008 that the Global Combat Support System-Marine Corps (GCSS-MC) program had not been economically justified on the basis of reliable estimates of both benefits and costs and that key program management controls, such as the use of earned value management and risk management had not been properly implemented. Accordingly, we made recommendations to address these weaknesses. GCSS-MC was certified with conditions and recertified during fiscal year 2009. Neither the weaknesses that we previously reported nor the status of our recommendations to address them were evident in the conditions accompanying the certification, even though our recommendations had yet to be implemented at the time of these certification actions.

We reported in September 2008 that the Navy ERP program did not use important cost estimating practices when economically justifying the program, did not implement key aspects of earned value management, and did not have risk mitigation strategies in place to address the risks described in our report, including risks associated with these issues. Accordingly, we made recommendations to address these weaknesses. However, when Navy ERP was recertified during fiscal year 2009, conditions relative to any of these weaknesses did not accompany the recertification, although our recommendations had yet to be implemented at the time of recertification.

Officials representing each of the IRBs stated that the boards depend on the component precertification authorities to provide them with complete and reliable information about each system investment. Among other things, IRB officials stated that such information should include the results of reviews by us and others. However, DOD guidance does not state that GAO-related information, such as open recommendations or the focus and results of our ongoing reviews, is to be included in the certification packages provided to the IRBs. Further, the Special Assistant to the Deputy Chief Management Officer told us it is each program’s milestone decision authority that is ultimately responsible for addressing known program management issues, including those raised by GAO.

By not having and considering relevant information about the state of each system modernization investment certified and approved, such as the results of our reviews and the status of actions to implement our recommendations that pertain to the investment, DOD’s certification and approval decisions are based on limited information, and thus may not be justified.

DOD’s Report States That Certification Waivers Were Not Granted

The act requires DOD to identify in its annual report any defense business system modernization with an obligation in excess of $1 million during the preceding fiscal year that was not certified and approved according to the act’s provisions, along with any reasons for these requirements being waived. According to DOD’s latest annual report, system investments were certified according to the act’s requirements during fiscal year 2009, and no systems were granted a certification waiver. Similarly, each of DOD’s annual reports since March of 2006 has stated that no systems were approved on the basis of a certification waiver. According to officials representing each of the IRBs, while program officials sometimes seek to be certified on the basis of a waiver, their practice is to ensure that the program office addresses any issues underlying a waiver request before the investment is placed on an IRB’s certification review agenda. As a result, they stated that a system is unlikely to go before an IRB for certification until it can be certified with conditions.

Conclusions

DOD’s latest annual report on its business systems modernization program complies with statutory requirements pertaining to the report’s content, but the scope and completeness of key information that is provided in the report is otherwise limited. In particular, the report omits information on numerous business system investment certification actions taken during fiscal year 2009. In addition, while it includes schedule-focused performance measures and performance against these measures for the modernization investments discussed, as required by statute, it does not include similar information for other performance measures, such as cost, capability, and benefit commitments and performance against these commitments. Collectively, this means that DOD’s annual report does not provide congressional committees with the full range of information necessary to permit meaningful and informed oversight of DOD’s business systems modernization program.

Beyond the scope and content of DOD’s annual report, the basis for the IRB certifications have been limited because DOD guidance does not provide for disclosure of our findings concerning investments being considered. In particular, investments have been certified and approved without conditions even though our prior reports have identified program weaknesses that were unresolved at the time of certification and approval. As a result, these certification and approval decisions may not be sufficiently justified.

Recommendations for Executive Action

To facilitate congressional oversight and promote departmental accountability, we recommend that the Secretary of Defense direct the Deputy Secretary of Defense, as the chair of the DBSMC, to ensure that the scope and content of future DOD annual reports to Congress on compliance with section 332 of the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005, as amended, be expanded to include: Cost, capability, and benefit performance measures for each business system modernization investment and actual performance against these measures.

All certification actions, as defined in DOD guidance, which were taken in the previous year by the department on its business system modernization investments.

To ensure that IRB certification actions are better informed and justified, we further recommend that the Secretary direct the Deputy Secretary to ensure that DOD guidance be revised to include provisions that require IRB certification submissions disclose program weaknesses raised by GAO and the status of actions to address our recommendations to correct the weaknesses.

Agency Comments

In written comments on a draft of this report, signed by the Assistant Deputy Chief Management Officer and reprinted in appendix II, the department agreed with our recommendations.

We are sending copies of this report to interested congressional committees; the Director, Office of Management and Budget; and the Secretary of Defense. This report will also be available at no charge on our Web site at http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

As agreed with congressional defense committees, our objective was to assess the actions by the Department of Defense (DOD) to comply with the requirements of key aspects of section 332 of the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 (the act). To address this, we focused on the extent to which DOD’s annual report to Congress addressed the following provisions of the act: (1) describe milestones and actual performance against specified measures and any revisions, (2) discuss specific improvements in business operations and cost savings resulting from successful business system modernization efforts, (3) describe specific actions on each business system investment submitted for certification, and (4) identify any business system investment with an obligation in excess of $1 million that was not certified during the preceding fiscal year and reasons for the waiver. Our methodology relative to each of the four requirements is as follows: To determine whether the DOD annual report described milestones and actual performance against specified measures and any revisions, we compared information contained in the annual report to what the act required. Further, we compared the types of measures included in the annual report to those commonly associated with program performance, such as those described in prior GAO work related to performance measures. In addition, we interviewed officials from the Business Transformation Agency (BTA) and each of DOD’s investment review boards (IRB) to understand the process used to identify and track milestones and other performance measures. We did not independently validate the accuracy of the milestone dates included in the report.

To determine the extent to which DOD’s annual report discussed specific improvements in business operations and cost savings, we reviewed each of the 18 case-in-point narratives included in the annual report that described examples of business improvements and other benefits. We compared this information with the act’s reporting requirements to identify any variances. We did not validate the accuracy of the improvements or benefits discussed in the case-in-point narratives.

To determine the extent to which DOD’s annual report identified specific actions on each business systems investment submitted for certification, we reviewed and analyzed all Defense Business Systems Management Committee (DBSMC) certification approval memoranda as well as IRB certification memoranda and IRB meeting minutes issued prior to the DBSMC’s final approval decisions for fiscal year 2009 and compared the results to those certification actions described in the annual report to identify differences. We also reviewed DOD IRB guidance to understand the types of actions related to certification of business system modernizations.

For certification actions included in the DBSMC and IRB memoranda but not described in the annual report, we interviewed officials from the BTA, IRBs, the Office of the Assistant Secretary of Defense (Networks and Information Integration)/DOD Chief Information Officer (ASD(NII)/DOD CIO), and the Office of the DOD Deputy Chief Management Officer (DCMO) as to the reason for the differences.

For certification actions included in the report and described in fiscal year 2009 DBSMC and IRB memoranda, we compared information about specific DOD programs from recent GAO reports to the conditions associated with certification actions described in the annual report and the DBSMC and IRB memoranda to determine whether IRBs placed certification conditions related to program weaknesses identified by GAO and whether those conditions addressed those weaknesses. In addition, we interviewed DCMO, BTA, and IRB staff to discuss conditions that were reported as part of certification actions and what is submitted to the IRBs when individual systems request certification.

To determine if DOD’s annual report identified any business system investment with an obligation in excess of $1 million that was not certified during the preceding fiscal year and the reasons for any waivers granted, we reviewed DBSMC and IRB certification memoranda and compared actions taken during fiscal year 2009 to the actions described in DOD’s annual report. We also interviewed DCMO and BTA officials, as well as IRB support staff, to determine if any waivers were issued during fiscal year 2009. Finally, we reviewed DOD’s annual reports from 2005 to present to determine the extent to which these reports identify any waivers issued prior to fiscal year 2009.

We conducted this performance audit at DOD offices in Arlington, Virginia, from January 2010 to May 2010, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact person named above, key contributors to this report were Carl Barden, Justin Booth, Nancy Glover, Michael Holland, Neelaxi Lakhmani (Assistant Director), Kate Nielsen, Constantine Papanastasiou, Christine San, Sylvia Shanks, Jennifer Stavros-Turner, and Adam Vodraska.