Business Systems Modernization: Scope and Content of DOD's Congressional Report and Executive Oversight of Investments Need to Improve
Published: 2010-05-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1995, GAO has designated the Department of Defense's (DOD) multibillion dollar business systems modernization program as high risk, and it continues to do so today. To assist in addressing DOD's modernization challenges, the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 (the act) requires the department to, among other things, report specific information about business system investments, including (1) milestones and actual performance against specified measures and any revisions and (2) actions taken to certify that a modernization investment involving more than $1 million meets defined conditions before obligating funds. The act also directs GAO to review each report. As agreed, GAO focused on the fiscal year 2010 report's compliance with, among other things, these provisions of the act. To do so, GAO compared DOD's report to the act's reporting requirements, interviewed DOD officials, analyzed relevant documentation, and leveraged prior GAO reports.
section_title: What GAO Found
paragraphs: DOD's fiscal year 2010 report to Congress on its business systems modernization program complies with key provisions in the act, but its scope and content are nevertheless limited. Specifically, (1) The report includes milestones, performance against milestones, and milestone revisions for specific investments. However, other important performance measures, such as measures of progress against program cost, capability, and benefit commitments are not included in the report. DOD officials attributed the missing performance-related information to various factors, including that most of the investments addressed in the report have not progressed far enough in their life cycles to measure cost, capability, and benefit performance. However, the report also cites a number of investments that have produced business improvements and cost savings, and thus it follows that performance-related information about investment costs incurred, capabilities delivered, and benefits realized is available and can be reported relative to program expectations. Moreover, programs that have not yet delivered capabilities or realized benefits have nevertheless incurred costs, which DOD can report relative to expected costs. (2) The report identifies certification actions associated with 116 business system modernization investments. However, the report omits certification actions for 40 other investments. According to DOD officials, the omitted actions are not new certifications, but rather are recertifications that were intentionally excluded from the report. However, certification memoranda show this is not the case for four of the actions and DOD guidance defines a recertification as a type of certification action. Further, the underlying bases for a number of reported actions are limited because program weaknesses that GAO's prior work has raised, such as the reliability of the systems' economic analyses and the sufficiency of business enterprise architecture compliance determinations, are not reflected in the reported certification actions. DOD's guidance does not require that certification submissions disclose program weaknesses that GAO has raised, and DOD officials stated that reviews are limited to the information that is submitted. As a result, DOD's annual report does not provide the full range of information that is needed to permit meaningful and informed congressional oversight of the department's business systems modernization efforts. Moreover, the bases for some certification actions exclude relevant information about known investment weaknesses, and thus these actions may not be sufficiently justified.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its timeworn business systems. In 1995, we designated  DOD’s Business Systems Modernization Program as high risk, and we  continue to designate it as such today. In light of the department’s  business systems modernization management challenges, Congress  included provisions in the Ronald W. Reagan National Defense  Authorization Act for Fiscal Year 2005 (the act) for the department to  report annually on its modernization progress. More specifically, the a requires the department’s annual report to, among other things, (1)  describe milestones and actual performance against specified measures and any revisions, (2) discuss specific improvements in business  operations and cost savings resulting from successful business system  modernization efforts, (3) describe specific actions on each business  system modernization submitted for certification, and (4) identify any  business system modernization with an obligation in excess of $1 m that was not certified during the preceding fiscal year and reasons for t waiver. Additionally, the act directs us to submit to the cognizant  congressional committees—within 60 days of DOD’s report submission an assessment of the department’s actions to comply with these  requirements.
As agreed with your offices, the objective of our review was to assess th actions taken by DOD to comply with key aspects of section 332 of the  act. To accomplish this objective, we focused on the extent to which  e  DOD’s annual report addressed the four requirements described above. In  doing so, we compared the nature and scope of the information contained  in the department’s report to Congress, which was submitted on March 25,  2010, as well as the supporting information that DOD used in preparing the  report, to four requirements in the act. In addition, we leveraged our  recent reports related to DOD’s business system investments to evaluate  the content of key aspects of the report. We also met with cognizant DOD  officials to discuss actions taken or planned.
We conducted this performance audit at DOD offices in Arlington,  Virginia, from January to May 2010, in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.  Details on our objective, scope, and methodology are contained in  appendix I.
subsections: section_title: Background
paragraphs: DOD is a massive and complex organization entrusted with more taxpayer  dollars than any other federal department or agency. Organizationally, the  department includes the Office of the Secretary of Defense, the Joint  Chiefs of Staff, the military departments, numerous defense agencies and  field activities, and various unified combatant commands that are  responsible for either specific geographic regions or specific functions.  (See fig. 1 for a simplified depiction of DOD’s organizational structure.)
In support of its military operations, the department performs an  assortment of interrelated and interdependent business functions,  including logistics management, procurement, health care management,  and financial management. As we have previously reported, the DOD  systems environment that supports these business functions is overly  complex and error prone, and is characterized by (1) little standardization  across the department, (2) multiple systems performing the same tasks,  (3) the same data stored in multiple systems, and (4) the need for data to  be entered manually into multiple systems. Moreover, the department’s  nonintegrated and duplicative systems impair its ability to combat fraud,  waste, and abuse. The department recently reported that this systems  environment is composed of approximately 2,480 separate business  systems. For fiscal year 2010, DOD requested about $15.5 billion in funds  to operate, maintain, and modernize its business systems and associated  information technology (IT) infrastructure.
DOD currently bears responsibility, in whole or in part, for 15 of the 31  programs across the federal government that we have designated as high  risk because they are highly susceptible to fraud, waste, abuse, and  mismanagement. Eight of these areas are specific to the department,  and 7 other high-risk areas are shared with other federal agencies.  Collectively, these high-risk areas relate to DOD’s major business  operations that are inextricably linked to the department’s ability to  perform its overall mission, directly affect the readiness and capabilities of  U.S. military forces, and can affect the success of a mission. DOD’s  business systems modernization is one of the high-risk areas, and it is an  essential enabler to addressing many of the department’s other high-risk  areas. For example, modernized business systems are integral to the  department’s efforts to address its financial, supply chain, and information  security management high-risk areas.
subsections: section_title: DOD’s Institutional Approach to Business Systems Modernization
paragraphs: The National Defense Authorization Act (NDAA) for Fiscal Year 2008  designates the Deputy Secretary of Defense as the Chief Management  Officer (CMO) for DOD and created a Deputy CMO position. The CMO’s  responsibilities include developing and maintaining a departmentwide  strategic plan for business reform and establishing performance goals and  measures for improving and evaluating overall economy, efficiency, and  effectiveness and monitoring and measuring the progress of the  department. The Deputy CMO’s responsibilities include recommending to  the CMO methodologies and measurement criteria to better synchronize,  integrate, and coordinate the business operations to ensure alignment in  support of the warfighting mission. The Business Transformation Agency  (BTA) supports the Deputy CMO in leading and coordinating business  transformation efforts across the department. This includes maintaining  and updating the department’s enterprise architecture for its business  mission area.
The CMO and Deputy CMO are to interact with several entities to guide  the direction, oversight, and execution of DOD’s business transformation  efforts, which include business systems modernization. These entities  include the Defense Business Systems Management Committee (DBSMC),  which serves as the highest-ranking investment review and decision- making body for business systems modernization activities and is chaired  by the Deputy Secretary of Defense; the principal staff assistants, who  serve as the certification authorities for business system modernizations  in their respective core business missions; the investment review boards  (IRB), which are chaired by the certifying authorities and form the  review and decision-making bodies for business system investments in  their respective areas of responsibility; and the BTA, which supports IRBs  and leads and coordinates business transformation efforts across the  department. (Table 1 lists these entities and provides greater detail on  their roles, responsibilities, and composition.)
section_title: Overview of DOD’s Tiered Accountability for Business Systems Modernization
paragraphs: Since 2005, DOD has employed a “tiered accountability” approach to  business systems modernization. Under this approach, responsibility and  accountability for business architectures and systems investment  management are assigned to different levels in the organization. For  example, the BTA is responsible for developing the corporate BEA (i.e.,  the thin layer of DOD-wide policies, capabilities, standards, and rules) and  the associated enterprise transition plan (ETP). Each component is  responsible for defining a component-level architecture and transition  plan associated with its own tiers of responsibility and for doing so in a  manner that is aligned with (i.e., does not violate) the corporate BEA.  Similarly, program managers are responsible for developing program-level  architectures and plans and for ensuring alignment with the architectures  and transition plans above them. This concept is to allow for autonomy  while also ensuring linkages and alignment from the program level  through the component level to the corporate level. (Table 2 describes the  four investment tiers and identifies the associated reviewing and  approving entities.)
Consistent with the tiered accountability approach, the NDAA for Fiscal  Year 2008 required the secretaries of the military departments to designate  the department under secretaries as CMOs with primary responsibility for  business operations. Moreover, the Duncan Hunter NDAA for Fiscal Year  2009 required the military departments to establish business  transformation offices to assist their CMOs.
section_title: Summary of Fiscal Year 2005 NDAA Requirements
paragraphs: Congress included provisions in the NDAA for Fiscal Year 2005 that are  aimed at ensuring DOD’s establishment and implementation of effective  investment management structures and processes. According to the act,  DOD is required to develop a BEA; develop an ETP for implementing the  architecture; identify each business system proposed for funding in DOD’s  fiscal year budget submissions; delegate the responsibility for business  systems to designated approval authorities within the Office of the  Secretary of Defense; require each approval authority to establish  investment review structures and processes; and, effective October 1,  2005, not obligate appropriated funds for a defense business system  modernization with a total cost of more than $1 million unless the  approval authority certifies that the business system modernization meets  several conditions.
The NDAA for Fiscal Year 2005 also requires that the Secretary of Defense  submit to congressional defense committees an annual report on the  department’s compliance with the act’s provisions. This report is to  1.  describe actions taken and planned for meeting the act’s requirements,  a)  specific milestones and actual performance against specified  performance measures and any revision of such milestones and  performance measures and b)  specific actions on the defense business system modernizations  submitted for certification under such subsection; 2.  discuss specific improvements in business operations and cost savings  resulting from successful defense business systems modernization  efforts;  3. identify the number of defense business system modernizations  certified; and  4. identify any defense business system modernization with an obligation  in excess of $1 million during the preceding fiscal year that was not  certified as required, and the reasons for the waiver.
section_title: Summary of Recent GAO Reviews of DOD’s Business Systems Modernization and Business Transformation Efforts
paragraphs: Between 2005 and 2008, we reported that DOD had taken increasing steps  to comply with key requirements of the NDAA for Fiscal Year 2005 relative  to architecture development, transition plan development, budgetary  disclosure, and investment review and to satisfy relevant systems  modernization management guidance, but that much remained to be  accomplished relative to the act’s requirements and relevant guidance.  Nevertheless, we concluded that the department had made important  progress in defining and beginning to implement institutional management  controls (i.e., processes, structures, and tools).
Notwithstanding this progress, in May 2009, we reported that the pace of  DOD’s efforts in defining and implementing key institutional  modernization management controls had slowed compared with progress  made in each of the last 4 years, leaving much to be accomplished to fully  implement the act’s requirements and related guidance. For example:    The corporate BEA had yet to be extended (i.e., federated) to the entire  family of business mission area architectures, including using an  independent verification and validation agent to assess the components’  subsidiary architectures and federation efforts.
The fiscal year 2009 budget submission included some, but omitted other  key information about business system investments, in part because of the  lack of a reliable, comprehensive inventory of all defense business  systems.
IT investment management policies and procedures at the corporate and  component levels were not fully defined.
The business system information used to support the development of the  transition plan and DOD’s budget requests, as well as certification and  annual reviews, was of questionable reliability.
Business system investments costing more than $1 million continue to be  certified and approved, but these decisions were not always based on  complete information.
Accordingly, we reiterated existing recommendations to address each of  these areas and further recommended that DOD, among other things,  improve the quality of investment-related information. DOD partially  agreed with our recommendations and described actions being planned or  under way to address them. DOD is currently in the process of addressing  these recommendations.
section_title: DOD’s Annual Report Discusses Programs’ Performance Against Milestones, but Does Not Address Performance Against Other Program Commitments
paragraphs: The act requires that DOD’s report describe milestones for business  system modernization programs and actual performance against  performance measures. In addition, the act requires that the report specify  any revisions to milestones and performance measures.
To its credit, DOD’s annual report includes milestones, performance  against milestones, and milestone revisions for 76 programs. However,  other important performances measures, which typically include measures  associated with determining progress against program cost, capability, and  benefit commitments, are not included in the report.
BTA officials cited various reasons for the scope and content of the  information provided and not provided, but these reasons are at odds with  other aspects of its report. Without including information on program  performance against, and revisions to, such key measures as cost,  capability, and benefit commitments, DOD is not providing Congress with  the information needed to inform its oversight of business system  modernization programs.
subsections: section_title: DOD’s Report Includes Program Milestones as Well as Revisions to and Performance Against These Milestones
paragraphs: Consistent with the act’s requirement that DOD report on specific  milestones and revisions to the milestones, DOD’s March 2010 report  includes a summary of the status of milestones that were to be completed  during fiscal 2009, the revisions associated with these milestones (e.g.  delayed or deleted), and the reason for the revision. Specifically, the  report lists three categories of milestones:    Standard acquisition milestones: key events and dates that are  provided for under DOD’s system acquisition process.
BEA compliance milestones: time frames for addressing specific IRB  certification conditions related to ensuring BEA compliance.
Interim milestones: key events and dates to supplement DOD’s system  acquisition process milestones (e.g., implementing specific system  capabilities or upgrading infrastructure by a given date).
DOD’s report includes a total of 224 milestones that collectively span 76  programs. Of these 224 milestones, 35 are standard acquisition  milestones, 22 are BEA compliance milestones, and 167 are interim  milestones. The report also discusses performance against these  milestones. Specifically, of the 224 milestones, 56 percent are reported to  have been met, while 21 and 23 percent are reported to have been deleted  (i.e., determined to be unnecessary) or not met, respectively. (See fig 2.)  With respect to acquisition and compliance milestones, the percentage of  milestones that were reported as not being met was 66 and 50 percent,  respectively.
section_title: DOD’s Report Does Not Address Performance Against Other Program Commitments
paragraphs: Beyond milestones, the act requires DOD’s annual report to address actual  performance against performance measures and any revision of these  performance measures. As we have previously reported, meaningful  information about program performance is typically measured in terms of  program cost, capability, and benefit commitments, in addition to  schedule commitments. Through such a range of performance measures,  valuable insight into the health and success of a business system  investment can be gained.
As we previously discussed in this report, DOD’s annual report does report  schedule commitments (i.e., milestones) for its modernizing programs.  While DOD’s annual report also includes examples of business  improvements and costs savings, this program performance information is  not reported against performance measure baselines. Further, the report  remains silent with respect to other important performance measures such  as progress made against cost and capability commitments, which would  allow congressional decision makers to understand the extent to which  programs are meeting cost, capability, and benefit commitments.
BTA officials stated they focused the annual report on programs that had  planned milestones during fiscal year 2009. Further, they said they focused  on program milestones because most of the investments covered by the  report have not progressed far enough in their life cycles to measure cost,  capability, and benefit performance. In addition, the annual report states  that DOD does not include performance measures in its annual report for  any system that has reached initial or full operational capability and is no  longer modernizing. Nevertheless, the report includes descriptions of a  number of programs that have progressed to the point where DOD reports  on actual operational efficiencies and dollar savings that have accrued,  which, in turn, means that these programs have progressed to the point  that DOD can report on progress against defined performance  commitments, such as the costs that have been incurred, the capabilities  that have been delivered, and benefits that have been realized. Moreover,  the programs that have not yet delivered capabilities or realized benefits  have incurred costs, which DOD can report relative to expected costs.
Establishing performance measures and monitoring actual-versus- expected performance using the full range of measures are essential to  understanding the health of any IT investment. By not including  information on each program’s performance against defined cost,  capability, and benefit commitments, DOD is not providing Congress with  important information for informing its oversight of business system  modernization investments.
section_title: DOD’s Report Discusses Business Systems Modernization Programs’ Improvements in Business Operations and Cost Savings
paragraphs: The act requires that DOD’s annual report discuss specific improvements  in business operations and cost savings resulting from successful business  system modernization programs. DOD met this requirement by including  18 “case in point” examples in the report. Among other things, each  narrative generally describes the program and provides high-level  information on system capabilities delivered and benefits achieved to date.  Specific examples include:    The Air Force Recruiting Information Support System is to be the  primary Web-based recruiting system for the Air Force. According to the  annual report, the Air Force’s legacy recruiting information system has  been slow and, at times, unavailable to users. Modernization of the system  (e.g., new hardware and software upgrades) is reported to have improved  recruiters’ productivity by reducing the wait time for processing recruiter  requests. Other reported improvements include allowing recruiters to  build applicant files off line and upload them at a later time and reducing  the recruiter’s dependence on Internet connectivity. Future plans include  merging the Air Force Recruiting Information Support System with the Air  Force Reserve Recruiting System to increase functionality and decrease  system response time.
The Army Learning Management System is a Web-based system for  training, scheduling, and career planning for soldiers. According to the  annual report, in 2009 the system contributed to an 88 percent increase in  the number of student accounts, 111 percent increase in the number of  courses offered, and 157 percent increase in the number of course  completions.
Wide Area Work Flow is a Web-based system to centralize and automate  DOD’s largely manual business payment process. The annual report states  that the system has thus far allowed the cost of processing a payment to  decrease from between $22 and $30 to between $6 and $12. Other cited  benefits include allowing suppliers to have a single point of interface with  DOD for payment invoicing, receipt, and acceptance.
The Defense Agencies Initiative is an enterprise resource planning  (ERP) system to standardize and integrate enterprise data to support  financial decisions in real time. According to the annual report, the system  resulted in a reduction in the time it takes to post financial obligations  from 60 days to less than 2 days, and a reduction in the time to close out  monthly financial reports from 4 days to less than 1 day. Further, the  report states that financial information is now available to BTA on a real- time basis and thereby enables proactive management of agency finances.
section_title: DOD’s Annual Report Does Not Describe Certification Actions for All Systems, and Justification of Reported Certification Actions Is Limited
paragraphs: Among other things, the act requires DOD’s annual report to describe  specific actions the department has taken on each business system  modernization investment submitted for certification. More specifically,  the act states that such investments involving more than $1 million in  obligations must be certified by a designated approval authority as  meeting specific criteria, such as demonstrating compliance with DOD’s  BEA. Further, the act requires the DBSMC to approve each of these  certifications.
To its credit, DOD’s annual report identifies IRB certification actions  associated with 116 business system investments. However, certification  actions associated with 40 other investments are not included. Further, the  bases for several of the fiscal year 2009 system certification actions and  subsequent approvals are limited because program weaknesses and issues  that our prior work has raised about, for example, the systems’ economic  analyses and BEA compliance determinations, are not reflected in the  reported certification actions.
According to BTA officials, only new certifications are included in the  report, even though DOD guidance states that recertifications are also  certification actions. Moreover, this guidance does not require programs to  disclose program weaknesses and issues raised by us or others.
By not fully identifying in its annual report the certification actions taken  on all business system modernization investments, DOD is not fully  informing congressional oversight. Further, by not ensuring that all  certifications reflect known program weaknesses, business system  modernization program certification and approval decisions are not being  fully informed and thus may not be adequately justified.
subsections: section_title: DOD Has Established an Approach to Certifying Business System Investments
paragraphs: DOD has established what it describes as a “tiered accountability”  approach to meeting the act’s requirements for certifying business system  investments. Under this approach, investment review begins within the  military departments and defense agencies and advances through a  hierarchy of review and decision-making authorities, depending on the  size, nature, and significance of the investment. For those investments that  meet the act’s dollar threshold, namely those with planned modernizations  in excess of $1 million, this sequence of review and decision making  includes component precertification, IRB certification, and DBMSC  approval. For those investments that do not meet the dollar threshold,  investment decision-making authority remains with the responsible  component.
According to the department’s approach, reviews for modernization  investments of more than $1 million focus on program alignment with the  BEA; alignment with the department’s strategic mission, goals, and  objectives; and oversight commensurate with the program’s cost, scope,  and complexity. The approach further requires that these reviews be  completed before a component can obligate modernization funds.
At the component level, program managers are responsible for the  information about their respective programs in a central repository known  as the Defense IT Portfolio Repository system (DITPR). The component  precertification authority is responsible for precertifying that a given  system investment is compliant with the BEA, reviewing the system  funding requests, and ensuring that the IRB responsible for the investment  receives complete, current, accurate, and timely information. The  precertification authority is also responsible for “asserting” the status and  validity of the investment information by submitting a component  precertification letter to the responsible IRB.
At the corporate level, an IRB reviews the precertification letter and  related material and makes a recommendation for a specific certification  action for each of its investments. After the IRB makes its  recommendation, it prepares a certification memorandum that documents  the IRB’s decisions and any related conditions. The memorandum is  forwarded to the DBSMC, which either approves or disapproves the IRB’s  decision and issues a memorandum containing its decision. If the DBSMC  disapproves a system investment’s certification, it is up to the  component’s precertification authority to decide whether or not to  resubmit the investment after it has resolved the reasons for the  disapproval.
Under DOD’s approach, there are four types of certification actions:    Certify: An IRB certifies the modernization as fully meeting criteria  defined in the act and IRB investment review guidance, such as  compliance with the BEA and the extent to which the investment is  consistent with component and department IT investment portfolios,  which are asserted by the component precertification authority.
Certify with conditions: An IRB certifies the modernization with the  understanding that it will address specific IRB-imposed conditions. For  example, the Army’s Real Estate Management Information System was  certified with a condition to provide a plan for how data elements would  comply with certain business rules in DOD’s BEA.
Recertify: An IRB certifies the obligation of additional modernization  funds for a previously certified modernization investment. For example,  the Air Force’s Cargo Movement Operations System was recertified in  April 2009 for $6.3 million to be spent in fiscal years 2009 through 2012.  This recertification was in addition to the $21.1 million previously certified  in fiscal year 2007. In addition, a program must request IRB recertification  if the program plans to redistribute previously approved modernization  funds among multiple fiscal years and this redistribution will result in the  funding for any given fiscal year exceeding the previously approved  amount by 10 percent or more.
Decertify: An IRB reduces the amount of modernization funds available  to an investment when the entire amount of funding is not to be obligated  as previously certified. For example, the Defense Financial Accounting  Service’s Standard Disbursing Initiative had about $5.5 million decertified  because the funds were no longer needed. An IRB may also decertify a  modernization after it has been completed. For example, DOD reported  that $213,000 for the Air National Guard’s Reserve Order Writing System  was decertified at the time the system was completed because the funds  were no longer needed.
section_title: DOD’s Report Excludes a Number of Certification Actions Taken on Business System Modernizations
paragraphs: The act requires that DOD’s annual report describe specific actions taken  for each business system investment submitted for certification. However,  the department’s annual report discusses fiscal year 2009 certification  actions on only 116 of the 156 systems on which certification actions were  taken. More specifically, the report states that during fiscal year 2009, 92  business system modernizations were certified—32 with and 60 without  conditions. For the 32 systems, 58 conditions were collectively reported.  Examples of conditions cited in the report are the need for a system to  comply with the Standard Financial Information Structure and the need  to develop a plan for complying with the data standards of DOD’s Item  Unique Identifier Registry. The report also identifies 24 decertifications.  For example, the Air Force’s Enhanced Technical Information  Management System had about $13.9 million in funding decertified (i.e.,  reduced), and the Defense Financial Accounting Service’s Standard  Disbursing Initiative had about $5.5 million decertified.
However, fiscal year 2009 IRB and DBSMC decision memoranda and  meeting minutes show that 40 systems had additional certification actions  that were not included in DOD’s annual report. Of these 40 systems, 2  were certified without conditions, 2 were certified with conditions, and 36  were recertified. Collectively, DOD’s annual report omits about 26 percent  of its certification actions. (See table 3 for a summary of actual, reported,  and unreported certification actions.)
According to BTA officials, the excluded certification actions are all  recertifications, which they said are intentionally not reported because  they are not new certifications. They also told us that the four new  certifications were, in fact, recertifications. However, DBSMC and IRB  memoranda and meeting minutes identify these four certification actions  as new certifications. Moreover, DOD guidance defines a recertification  as a type of certification action, thus making it subject to the act’s  reporting requirements. Without complete reporting of its certification  actions, DOD is not in full compliance with its guidance, and DOD is  limiting congressional visibility into the full scope of its business systems  modernization efforts.
section_title: IRB Certification Actions Do Not Reflect and Disclose Known Limitations in Programs’ BEA Alignment, Economic Justification, and Management
paragraphs: According to DOD guidance, IRB certification addresses, among other  things, a program’s alignment with the BEA and its management relative to  factors such as system cost, scope, and complexity. To make a  certification decision, IRBs rely on documentation submitted by the  component precertification authority, including a certification dashboard,  which includes cost and schedule status information; an economic  viability analysis, which addresses the investment’s cost and benefits or  cost effectiveness; and regulatory and standards compliance  determinations. DOD guidance also gives IRBs broad authority in their  certification reviews and actions, thus allowing each board to review and  consider whatever investment-related information that it deems  appropriate. Moreover, BTA and IRB officials told us that an IRB is not  limited in the conditions it can place on a program.
The IRB certification actions described in DOD’s latest annual report are  limited because they do not reflect significant limitations in the  department’s basis for determining an investment’s alignment to the BEA.  Specifically, we recently reported that key DOD BEA compliance  assessments did not include all relevant architecture products, such as  products that specify the technical standards needed to promote  interoperability among related systems or examine overlaps with other  business systems. In addition, we reported that these compliance  assessments were not validated by DOD certification and approval  entities. Despite these limitations, business systems modernization  programs were certified as compliant with the BEA even though they did  not adequately demonstrate such compliance. Accordingly, we  recommended that DOD revise its BEA compliance guidance, tool, and  IRB verification requirements to address these shortfalls. To date, DOD  has yet to implement these recommendations, and thus the compliance  determination weaknesses remain. Despite this, DOD’s latest annual  report does not disclose these limitations on any of the 116 investment  certification actions that it describes.
In addition, the fiscal year 2009 IRB certification actions described in the  latest annual report are further limited in that they do not reflect  weaknesses we have recently reported with the economic justification for  and management of certain programs. For example,    We reported in September 2009 that the Defense Readiness Reporting  System (DRRS) program was not being effectively managed and made  recommendations to address a number of acquisition management  weaknesses including the absence of a reliable integrated master schedule  and well defined and managed requirements, and adequate testing. As  stated in our report, we briefed the DRRS program office on the results of  our work prior to its DOD Human Resources Management IRB  certification review. However, these results were not disclosed to the IRB.  Rather, the certification package that the precertification authority  submitted to the IRB stated that DRRS was on track for meeting its cost,  schedule, and performance measures and highlighted no program risks.  Based on this submission, DRRS was certified by the IRB and approved by  the DBSMC to obligate $24.625 million in fiscal years 2009 and 2010.  According to the chair of the IRB, the board did not validate the  information in the submissions it received, and the results of our review  were not disclosed to the IRB.
We reported in July 2008 that the Global Combat Support System-Marine  Corps (GCSS-MC) program had not been economically justified on the  basis of reliable estimates of both benefits and costs and that key program  management controls, such as the use of earned value management and  risk management had not been properly implemented. Accordingly, we  made recommendations to address these weaknesses. GCSS-MC was  certified with conditions and recertified during fiscal year 2009. Neither  the weaknesses that we previously reported nor the status of our  recommendations to address them were evident in the conditions  accompanying the certification, even though our recommendations had  yet to be implemented at the time of these certification actions.
We reported in September 2008 that the Navy ERP program did not use  important cost estimating practices when economically justifying the  program, did not implement key aspects of earned value management, and  did not have risk mitigation strategies in place to address the risks  described in our report, including risks associated with these issues.  Accordingly, we made recommendations to address these weaknesses.  However, when Navy ERP was recertified during fiscal year 2009,  conditions relative to any of these weaknesses did not accompany the  recertification, although our recommendations had yet to be implemented  at the time of recertification.
Officials representing each of the IRBs stated that the boards depend on  the component precertification authorities to provide them with complete  and reliable information about each system investment. Among other  things, IRB officials stated that such information should include the results  of reviews by us and others. However, DOD guidance does not state that  GAO-related information, such as open recommendations or the focus and  results of our ongoing reviews, is to be included in the certification  packages provided to the IRBs. Further, the Special Assistant to the  Deputy Chief Management Officer told us it is each program’s milestone  decision authority that is ultimately responsible for addressing known  program management issues, including those raised by GAO.
By not having and considering relevant information about the state of each  system modernization investment certified and approved, such as the  results of our reviews and the status of actions to implement our  recommendations that pertain to the investment, DOD’s certification and  approval decisions are based on limited information, and thus may not be  justified.
section_title: DOD’s Report States That Certification Waivers Were Not Granted
paragraphs: The act requires DOD to identify in its annual report any defense business  system modernization with an obligation in excess of $1 million during the  preceding fiscal year that was not certified and approved according to the  act’s provisions, along with any reasons for these requirements being  waived. According to DOD’s latest annual report, system investments were  certified according to the act’s requirements during fiscal year 2009, and  no systems were granted a certification waiver. Similarly, each of DOD’s  annual reports since March of 2006 has stated that no systems were  approved on the basis of a certification waiver. According to officials  representing each of the IRBs, while program officials sometimes seek to  be certified on the basis of a waiver, their practice is to ensure that the  program office addresses any issues underlying a waiver request before  the investment is placed on an IRB’s certification review agenda. As a  result, they stated that a system is unlikely to go before an IRB for  certification until it can be certified with conditions.
section_title: Conclusions
paragraphs: DOD’s latest annual report on its business systems modernization program  complies with statutory requirements pertaining to the report’s content,  but the scope and completeness of key information that is provided in the  report is otherwise limited. In particular, the report omits information on  numerous business system investment certification actions taken during  fiscal year 2009. In addition, while it includes schedule-focused  performance measures and performance against these measures for the  modernization investments discussed, as required by statute, it does not  include similar information for other performance measures, such as cost,  capability, and benefit commitments and performance against these  commitments. Collectively, this means that DOD’s annual report does not  provide congressional committees with the full range of information  necessary to permit meaningful and informed oversight of DOD’s business  systems modernization program.
Beyond the scope and content of DOD’s annual report, the basis for the  IRB certifications have been limited because DOD guidance does not  provide for disclosure of our findings concerning investments being  considered. In particular, investments have been certified and approved  without conditions even though our prior reports have identified program  weaknesses that were unresolved at the time of certification and approval.  As a result, these certification and approval decisions may not be  sufficiently justified.
section_title: Recommendations for Executive Action
paragraphs: To facilitate congressional oversight and promote departmental  accountability, we recommend that the Secretary of Defense direct the  Deputy Secretary of Defense, as the chair of the DBSMC, to ensure that  the scope and content of future DOD annual reports to Congress on  compliance with section 332 of the Ronald W. Reagan National Defense  Authorization Act for Fiscal Year 2005, as amended, be expanded to  include:    Cost, capability, and benefit performance measures for each business  system modernization investment and actual performance against these  measures.
All certification actions, as defined in DOD guidance, which were taken in  the previous year by the department on its business system modernization  investments.
To ensure that IRB certification actions are better informed and justified,  we further recommend that the Secretary direct the Deputy Secretary to  ensure that DOD guidance be revised to include provisions that require  IRB certification submissions disclose program weaknesses raised by GAO  and the status of actions to address our recommendations to correct the  weaknesses.
section_title: Agency Comments
paragraphs: In written comments on a draft of this report, signed by the Assistant  Deputy Chief Management Officer and reprinted in appendix II, the  department agreed with our recommendations.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; and the  Secretary of Defense. This report will also be available at no charge on our  Web site at http://www.gao.gov.
If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: As agreed with congressional defense committees, our objective was to  assess the actions by the Department of Defense (DOD) to comply with  the requirements of key aspects of section 332 of the Ronald W. Reagan  National Defense Authorization Act for Fiscal Year 2005 (the act). To  address this, we focused on the extent to which DOD’s annual report to  Congress addressed the following provisions of the act: (1) describe  milestones and actual performance against specified measures and any  revisions, (2) discuss specific improvements in business operations and  cost savings resulting from successful business system modernization  efforts, (3) describe specific actions on each business system investment  submitted for certification, and (4) identify any business system  investment with an obligation in excess of $1 million that was not certified  during the preceding fiscal year and reasons for the waiver. Our  methodology relative to each of the four requirements is as follows:    To determine whether the DOD annual report described milestones and  actual performance against specified measures and any revisions, we  compared information contained in the annual report to what the act  required. Further, we compared the types of measures included in the  annual report to those commonly associated with program performance,  such as those described in prior GAO work related to performance  measures. In addition, we interviewed officials from the Business  Transformation Agency (BTA) and each of DOD’s investment review  boards (IRB) to understand the process used to identify and track  milestones and other performance measures. We did not independently  validate the accuracy of the milestone dates included in the report.
To determine the extent to which DOD’s annual report discussed specific  improvements in business operations and cost savings, we reviewed each  of the 18 case-in-point narratives included in the annual report that  described examples of business improvements and other benefits. We  compared this information with the act’s reporting requirements to  identify any variances. We did not validate the accuracy of the  improvements or benefits discussed in the case-in-point narratives.
To determine the extent to which DOD’s annual report identified specific  actions on each business systems investment submitted for certification,  we reviewed and analyzed all Defense Business Systems Management  Committee (DBSMC) certification approval memoranda as well as IRB  certification memoranda and IRB meeting minutes issued prior to the  DBSMC’s final approval decisions for fiscal year 2009 and compared the  results to those certification actions described in the annual report to  identify differences. We also reviewed DOD IRB guidance to understand  the types of actions related to certification of business system  modernizations.
For certification actions included in the DBSMC and IRB memoranda but  not described in the annual report, we interviewed officials from the BTA,  IRBs, the Office of the Assistant Secretary of Defense (Networks and  Information Integration)/DOD Chief Information Officer (ASD(NII)/DOD  CIO), and the Office of the DOD Deputy Chief Management Officer  (DCMO) as to the reason for the differences.
For certification actions included in the report and described in fiscal year  2009 DBSMC and IRB memoranda, we compared information about  specific DOD programs from recent GAO reports to the conditions  associated with certification actions described in the annual report and  the DBSMC and IRB memoranda to determine whether IRBs placed  certification conditions related to program weaknesses identified by GAO  and whether those conditions addressed those weaknesses. In addition,  we interviewed DCMO, BTA, and IRB staff to discuss conditions that were  reported as part of certification actions and what is submitted to the IRBs  when individual systems request certification.
To determine if DOD’s annual report identified any business system  investment with an obligation in excess of $1 million that was not certified  during the preceding fiscal year and the reasons for any waivers granted,  we reviewed DBSMC and IRB certification memoranda and compared  actions taken during fiscal year 2009 to the actions described in DOD’s  annual report. We also interviewed DCMO and BTA officials, as well as  IRB support staff, to determine if any waivers were issued during fiscal  year 2009. Finally, we reviewed DOD’s annual reports from 2005 to present  to determine the extent to which these reports identify any waivers issued  prior to fiscal year 2009.
We conducted this performance audit at DOD offices in Arlington,  Virginia, from January 2010 to May 2010, in accordance with generally  accepted government auditing standards. Those standards require that we  plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact person named above, key contributors to this  report were Carl Barden, Justin Booth, Nancy Glover, Michael Holland,  Neelaxi Lakhmani (Assistant Director), Kate Nielsen, Constantine  Papanastasiou, Christine San, Sylvia Shanks, Jennifer Stavros-Turner, and  Adam Vodraska.