DOD Business Systems Modernization: Billions Continue to Be Invested with Inadequate Management Oversight and Accountability
Published: 2004-05-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Despite its significant investment in business systems, the Department of Defense (DOD) continues to have long-standing financial and inventory management problems that prevent it from producing reliable and timely information for making decisions and for accurately reporting on its billions of dollars of inventory. GAO was asked to (1) identify DOD's fiscal year 2004 estimated funding for its business systems, (2) determine if DOD has effective control and accountability over its business systems investments, and (3) determine whether selected business systems will help resolve some of DOD's long-standing problems and whether they are being effectively managed.
section_title: What GAO Found
paragraphs: DOD requested approximately $19 billion for fiscal year 2004 to operate, maintain, and modernize its reported 2,274 business systems. This stovepiped and duplicative systems environment evolved over time as DOD components--each with its own system funding--developed narrowly focused, parochial solutions to their business problems. As a result of this uncontrolled spending, DOD reported over 200 inventory systems and 450 personnel systems. DOD's fundamentally flawed business systems affect mission effectiveness and can contribute to the fraud, waste, and abuse that GAO continues to identify. Further, the number of business systems is likely understated in part because DOD does not have a central systems repository or a standard business system definition. DOD does not have an effective management structure for controlling business systems investments and the business domains' roles and responsibilities have not been defined. Further, DOD does not have reasonable assurance that it is in compliance with the National Defense Authorization Act for Fiscal Year 2003, which requires the DOD Comptroller to determine that system improvements exceeding $1 million meet the criteria specified in the act. Based on limited information provided by DOD, system improvements with at least $479 million of obligations over $1 million were not reviewed by the DOD Comptroller. GAO's two case studies are examples of DOD spending hundreds of millions on business systems that will not result in corporate solutions to its longstanding inventory and related financial management problems. While these efforts should provide some improvement to the Defense Logistics Agency's and the Army's business operations, implementation problems have resulted in schedule slippages, cost increases, and critical capabilities not being delivered. These issues can be attributed, in part, to the lack of disciplined processes in the areas of requirements management and testing. If not corrected, the problems will result in two more costly, nonintegrated systems that only marginally improve DOD business operations and further impede DOD's transformation as envisioned by the Secretary of Defense.

section_title: Letter
paragraphs: The Department of Defense (DOD) spends billions of dollars annually to  operate, maintain, and modernize its business systems.  As we have  reported for years, DOD does not have the ability to produce accurate,  reliable, and timely information to make sound decisions and to accurately  report on its billions of dollars of inventory and other assets.  In addition,  the department’s stovepiped, duplicative systems contribute to its  vulnerability to fraud, waste, and abuse.  Such problems led us in 1990 to  put DOD inventory management on our list of high-risk areas in the federal government and in 1995 to add financial management and business  systems modernization at DOD—designations that continue today.
This report responds to your request for information on the magnitude of  DOD’s enormous investment in business systems, its control and  accountability over these investments, and its management of certain key  business systems modernization projects that are essential to the  department transforming its business systems and operations.  As agreed  with your offices, our objectives were to (1) identify the amount of funding  DOD requested in fiscal year 2004 to operate, maintain, and modernize its  business systems; (2) determine whether DOD has effective control and  accountability over its business systems modernization investments; and  (3) determine whether selected business systems investments will help  resolve some of the department’s long-standing financial and inventory  management problems and whether these projects are being effectively  managed.
To determine how much DOD plans to spend on the operation,  maintenance, and modernization of its business systems in fiscal year 2004,  we analyzed DOD’s information technology (IT) budget request and met  with officials in the office of the DOD Chief Information Officer (CIO) and  military service representatives to obtain an overview of how the IT budget  request is developed.  Additionally, we met with DOD officials and  reviewed available documentation to determine the effectiveness of the  department’s efforts to control and account for its business systems  investments.  Further, we reviewed documentation provided by DOD to  determine if all systems improvements with obligations exceeding   $1 million were reviewed by the DOD Comptroller in accordance with the fiscal year 2003 defense authorization act.  In addition, we selected two  logistics systems modernization efforts—the Defense Logistics Agency’s  (DLA) Business Systems Modernization program (BSM) and the Army’s  Logistics Modernization Program (LMP)—as case studies to determine if  they will help resolve some of the department’s long-standing financial and  inventory management problems.  These two system projects represent 19  percent of the $770 million modernization funding requested in fiscal year  2004 for logistics systems.  In reviewing these two systems, we relied on  project documentation provided by DOD and discussions with program  management officials related to two key processes, requirements  management and testing.  Our work was performed from August 2003  through March 2004 in accordance with U.S. generally accepted  government auditing standards. Details on our scope and methodology are  included in appendix I.  We requested comments on a draft of this report  from the Secretary of Defense or his designee.  Written comments from the  Acting Under Secretary of Defense (Comptroller) are reprinted in appendix  II.
subsections: section_title: Background
paragraphs: DOD is one of the largest and most complex organizations in the world.  In  fiscal year 2003, DOD reported that its operations involved over $1.1 trillion  in assets, over $1.5 trillion in liabilities, approximately 3.3 million military  and civilian personnel—including guard and reserve components, and  disbursements of over $416 billion.  Execution of these operations spans a  wide range of defense organizations, including the military services and  their respective major commands and functional activities, numerous large  defense agencies and field activities, and various combatant and joint  operational commands that are responsible for military operations for  specific geographic regions or theaters of operations.
To execute these military operations, the department performs an  assortment of interrelated and interdependent business functions,  including logistics management, procurement, health care management,  and financial management.  To support its business functions, DOD  reported in April 2003 that it relied on about 2,274 business systems,  including accounting, acquisition, logistics, and personnel systems.  To  support its existing systems environment, DOD requests billions of dollars  annually.  The Assistant Secretary of Defense for Networks and  Information Integration—DOD’s CIO—is responsible for compiling and  submitting the department’s IT budget reports to Congress and the Office  of Management and Budget (OMB). According to a DOD CIO official, the  information in the IT budget request is initially prepared by various DOD  components and processed through their respective CIOs and  comptrollers.  The information is then forwarded to the DOD CIO office,  where it is consolidated before being sent to OMB and Congress.  The DOD  component CIOs and comptrollers are responsible for, and are required to  certify, the reliability of the information about their respective initiatives  that is included in the IT budget request.
subsections: section_title: Serious Financial Management Weaknesses Persist
paragraphs: DOD continues to confront pervasive, decades-old financial and business  management problems related to its systems, processes (including internal  controls), and people (human capital). These problems have (1) resulted in  a lack of reliable information needed to make sound decisions and report  the status of DOD’s activities through financial and other reports;   (2) hindered its operational efficiency; and (3) left the department  vulnerable to fraud, waste, and abuse.  For example:  Of the 481 mobilized Army National Guard soldiers from six GAO case  study Special Forces and Military Police units, 450 had at least one pay  problem associated with their mobilization. DOD’s inability to provide  timely and accurate payments to these soldiers, many of whom risked  their lives in recent Iraq or Afghanistan missions, distracted them from  their missions, imposed financial hardships on the soldiers and their  families, and has had a negative impact on retention.
Some DOD contractors have been abusing the federal tax system with  little or no consequence and DOD is not collecting as much in unpaid  taxes as it could.  Under the Debt Collection Improvement Act of 1996,  DOD is responsible—along with the Department of the Treasury—for  offsetting payments made to contractors to collect funds owed, such as  unpaid federal taxes.  However, we found that DOD had collected only  $687,000 of unpaid taxes as of September 2003.  We estimated that at  least $100 million could be collected annually from DOD contractors  through effective implementation of the levy and debt collection  program.
Our review of fiscal year 2002 data revealed that about $1 of every $4 in  contract payment transactions in DOD’s MOCAS system was for  adjustments to previously recorded payments—$49 billion of  adjustments out of $198 billion in disbursement, collection, and  adjustment transactions. According to DOD, the cost of researching and  making adjustments to accounting records was about $34 million in  fiscal year 2002, primarily to pay hundreds of DOD and contractor  staff.
Tens of millions of dollars are not being collected each year by military  treatment facilities from third-party insurers because key information  required to effectively bill and collect from third-party insurers is often  not properly collected, recorded, or used by the military treatment  facilities.
The long-standing problems continue despite the significant investments  made in DOD business systems each year.  The challenges continue, in part,  because of DOD’s inability to effectively modernize its business systems.   For example, our March 2003 report and testimony concluded that DOD  had not effectively managed and overseen its planned investment of over  $1 billion in four DFAS systems modernization efforts.  DOD has  terminated two of the four DFAS systems modernization projects—the  Defense Procurement Payment System (DPPS) and the Defense Standard  Disbursing System (DSDS).  The DOD Comptroller terminated DPPS in  December 2002 after more than 7 years of effort and an investment of over  $126 million, citing poor program performance and increasing costs.  DFAS  terminated DSDS in December 2003 after approximately 7 years of effort  and an investment of about $53 million, noting that a valid business case for  continuing the effort could not be made.  These two projects were planned  to provide DOD the capability to address some of its long-standing contract  and vendor payment problems.
section_title: Long-standing Inventory Management Deficiencies
paragraphs: Since 1990, we have identified DOD’s management of secondary  inventories (spare and repair parts, clothing, medical supplies, and other  items to support the operating forces) as a high-risk area.  One primary  factor contributing to DOD’s inventory management weaknesses is its  outdated and ineffective systems.  These system deficiencies have hindered  DOD’s ability to (1) support its reported inventory balances; (2) provide  inventory visibility; and (3) provide accurate financial and management  information related to its property, plant, and equipment.  For example:  DOD incurred substantial logistical support problems as a result of  weak distribution and accountability processes and controls over  supplies and equipment shipments in support of Operation Iraqi  Freedom activities, similar to those encountered during the prior Gulf  War. These weaknesses resulted in (1) supply shortages, (2) backlogs of  materials delivered in theater but not delivered to the requesting  activity, (3) a discrepancy of $1.2 billion between the amount of materiel  shipped and that acknowledged by the activity as received,   (4) cannibalization of vehicles, and (5) duplicate supply requisitions.
Inadequate asset visibility and accountability resulted in DOD selling  new Joint Service Lightweight Integrated Suit Technology—the current  chemical and biological protective garment used by our military  forces—on the Internet for $3 each (coat and trousers) while at the  same time buying them for over $200 each.  DOD has acknowledged  that these garments should have been restricted to DOD use only and  therefore should not have been available to the public.
Our analysis of data on more than 50,000 maintenance work orders  opened during the deployments of six battle groups indicated that about  29,000 orders (58 percent) could not be completed because the needed  repair parts were not available on board ship.  This condition was a  result of inaccurate ship configuration records and incomplete,  outdated, or erroneous historical parts demand data. Such problems not  only have a detrimental impact on mission readiness, they may also  increase operational costs due to delays in repairing equipment and  holding unneeded spare parts inventory.
section_title: Efforts to Modernize DOD Business Systems
paragraphs: Transformation of DOD’s business systems and operations is critical to the  department having the ability to provide Congress and DOD management  with accurate and timely information for use in the decision-making  process.  One of the key elements we have reported as necessary to  successfully execute the transformation is establishing and implementing  an enterprise architecture.  In this regard, the department has undertaken a  daunting challenge to modernize its existing business systems environment  through the development and implementation of a BEA or modernization  blueprint.  This effort is an essential part of the Secretary of Defense’s  broad initiative to “transform the way the department works and what it  works on.” As previously noted, the department has designated seven  domain owners to be responsible for implementing the BEA, which  includes (1) performing system reviews and approving initiative funding as  part of investment management and (2) enforcing compliance with the  BEA.
In April 2003, DOD reported that its business systems environment  consisted of 2,274 systems and systems acquisition projects spanning  numerous business operations that were divided into the seven domains  and established a domain leader for each area.  DOD’s efforts to manage  the modernization initiative include a strategy to vest the domains with the  authority, responsibility, and accountability for business transformation,  extension and implementation of the architecture, and investment  management.  We have also recommended that DOD establish an  investment management structure to gain control over business system  investments by (1) establishing a hierarchy of investment review boards  from across the department, (2) establishing a standard set of investment  review and decision-making criteria for its ongoing IT system projects, and  (3) directing the boards to perform a comprehensive review of all ongoing  business system investments.
Two of the business systems modernization efforts DOD has under way to  address some of its inventory problems are DLA’s BSM and the Army’s LMP.   These two business systems represent approximately 19 percent of the  $770 million of the modernization funding requested in fiscal year 2004 for  logistics systems.  DLA and the Army are using the same commercial-off- the-shelf (COTS) enterprise resource planning software package.  DLA  and the Army are using the inventory management portion of the package.
BSM. In November 1999, DLA initiated an effort to replace its materiel  management systems—the Standard Automated Materiel Management  System (SAMMS) and the Defense Integrated Subsistence Management  System—with BSM.   DLA has used the two existing systems for over 30  years to manage its inventory. BSM is intended to transform how DLA  conducts its operations in five core business processes: order fulfillment,  demand and supply planning, procurement, technical/quality assurance,  and financial management.  BSM was deployed in July 2002 and is  operating at the Defense Supply Center Columbus—Columbus, Ohio; the  Defense Supply Center Philadelphia—Philadelphia, Pennsylvania; the  Defense Supply Center Richmond—Richmond, Virginia; the Defense  Distribution Center—New Cumberland, Pennsylvania; the DLA Logistics  Information Service—Battle Creek, Michigan; and DLA headquarters—Fort  Belvoir, Virginia.  The initial deployment included low-volume, low-dollar- value items.  BSM has about 900 users and is populated with over 170,000  inventory items valued at about $192 million.  Once it becomes fully  operational, BSM is expected to have about 5,000 users and control and  account for about 5 million inventory items valued at about $12 billion.   DLA currently estimates that it will invest approximately $850 million to  fully deploy BSM.
LMP.  In February 1998, the U.S. Army Materiel Command (AMC) began an  effort to replace its existing materiel management systems—the  Commodity Command Standard System and the Standard Depot System— with LMP.   The Army has used the existing systems for over 30 years to  manage its inventory and depot maintenance operations.  LMP is intended  to transform AMC’s logistics operations in six core processes: order  fulfillment, demand and supply planning, procurement, asset management,  materiel maintenance, and financial management.  LMP is a 12-year  acquisition requirements contract. LMP became operational at the U.S.  Army Communications and Electronics Command (CECOM), Fort  Monmouth, New Jersey, and Tobyhanna Army Depot, Tobyhanna,  Pennsylvania, in July 2003.  The initial deployment of LMP consisted of  inventory items such as electronics; electronic repair components; and  communications and intelligence equipment such as night vision goggles,  electronic components such as circuit boards, and certain munitions such  as guidance systems included in missiles.  Currently, LMP has 4,500 users at  12 locations and is populated with over 2 million inventory items valued at  about $440 million.  When LMP is fully implemented, its capacity is  expected to include more than 15,000 users at 149 locations and will be  populated with 6 million Army-managed inventory items valued at about  $40 billion.  The Army currently estimates that it will invest approximately  $1 billion to fully deploy LMP.
section_title: Fiscal Year 2004 Budget for DOD’s Stovepiped, Duplicative Business Systems Is Nearly $19 Billion
paragraphs: For fiscal year 2004, DOD requested approximately $28 billion in IT funding  to support a wide range of military operations as well as DOD business  system operations, of which approximately $18.8 billion is for the  reported 2,274 business systems—$4.8 billion for business systems  development/modernization and about $14 billion for operation and  maintenance.  As shown in figure 1, the $28 billion is spread across the  military services and defense agencies.  The $28 billion represents a   $2 billion increase over fiscal year 2003.
The department’s business systems are used to record the events  associated with DOD’s functional areas, such as finance, logistics,  personnel, and transportation.   Table 1 shows how business system  funding is spread across the various DOD components.
OMB requires that funds requested for IT projects be classified as either  steady state (referred to by DOD as “current services”) or as  development/modernization.  Current services are funds for operating and  maintaining systems at current levels (i.e., without major enhancements).   The development/modernization budget category represents funds for  developing new IT systems or making major enhancements to existing  systems.
Some systems, such as BSM, have both current services and  development/modernization funding.  For BSM, while current services are  to be used for operating the system at various DLA locations,  development/modernization funds are to be used for activities such as  developing additional system functionality.  For fiscal year 2004, DLA’s IT  budget request, including BSM, was $452 million for current services and  $322 million for development/modernization.  Generally, current services  are financed through the Operation and Maintenance appropriations,  whereas development/modernization funding can come from any one or  combination of several funding sources, such as the Research,  Development, Test, and Evaluation appropriations; the Procurement  appropriations; or the Defense Working Capital Fund.
As part of DOD’s ongoing business systems modernization efforts, DOD’s  Business Management Modernization Program (BMMP) and Business Management and Systems Integration (BMSI) office are creating a  repository of the department’s existing business systems.  DOD reported  that as of April 2003, this environment consisted of 2,274 systems and  system acquisition projects.  To provide for investment management, DOD  assigned the systems to the seven domains.  For example, DOD assigned  565 systems to the logistics domain, 210 of which primarily perform  inventory functions and 32 of which perform transportation functions.   Similarly, the accounting and finance domain has 542 systems of which 240  primarily perform finance and accounting functions.  Table 2 presents the  composition of DOD’s reported business systems by domain and functional  area.
Table 2 clearly indicates that there are numerous redundant systems  operating in the department today.  For example, DOD has reported that it  has 16 vendor pay systems that are used to pay contractors for services  provided.  A further illustration is the department’s statement that the  Defense Integrated Military Human Resources System, which is to serve as  DOD’s integrated military personnel and pay system, will replace a  reported 79 existing systems.
BMSI officials stated that they are validating the 2,274 different systems  and related functional area categories, as illustrated in table 2, with the  domains.  Although the systems are different, functional area categories  may be the same among the domains.  For example, the Accounting and  Finance and Strategic Planning and Budgeting domains both report having  systems that perform finance and accounting functions.  BMSI officials  have stated that through the BMSI office’s validation efforts, the functional  area categories may be renamed or systems may be reclassified to other  functional areas.  For example, BMSI officials explained that the finance  and accounting functional area within the Strategic Planning and Budgeting  domain may be changed to Budgetary Financial Data.
Although the BMSI office has created an initial repository of 2,274 business  systems to support DOD’s systems modernization efforts, its systems  inventory is currently neither complete nor informative enough for  decision making.  For example, according to logistics domain officials,  there are currently about 3,000 systems just within the logistics domain.  Of  that amount, about 1,900 systems have been validated by the DOD  components as logistics systems—that is, they are not merely a  spreadsheet or a report.  Such a determination has not been made for the  other 1,100.  Our analysis showed that of the 1,900 systems, 253 systems are  included in DOD’s reported 2,274 business systems. According to logistics  domain officials, they are in the process of determining if the remaining  systems should be classified as a business system or a national security  system.
The BMSI office has not reported additional systems since April 2003  because it is continuing to reconcile its inventory with two other  databases—the IT Registry and the Information Technology Management  Application (ITMA).  This reconciliation is necessary because the three  databases are not integrated.  The IT Registry is a database of mission- critical and mission-essential IT systems maintained by the DOD CIO.  As reported by the DOD Inspector General (IG), each DOD component  could determine whether a system should be reported as mission critical or  mission essential in the IT Registry. Since the definitions were subject to  interpretation, the DOD IG concluded that the IT Registry would not  necessarily capture the universe of DOD business systems.  The ITMA is an  application used by the DOD CIO to collect system information for the  development of the department’s annual IT budget request. Each of these  databases—the IT Registry, the ITMA, and the BMMP systems inventory— contains varying information, some of which overlaps.  For example, the IT  Registry includes warfighting systems as well as some business systems,  while the BMMP inventory includes only systems related to the  department’s business operations.  The ITMA includes initiatives and  programs, such as the department’s BEA effort, that are not IT systems.
Although DOD recognizes that it needs an integrated repository of systems  information in order to control and prioritize its IT investments, the  difficulty of developing a single source is compounded by the fact that DOD  has not developed a universal definition of what should be classified as a  business system.  Lacking a standard definition that is used consistently  across the entire department, DOD does not have reasonable assurance  that it has identified all of its business systems. As a result, DOD does not  have complete visibility over its business systems to permit analysis of gaps  and redundancies in DOD’s business systems environment and to assist in  preventing the continuing proliferation of redundant, stovepiped business  systems.
Furthermore, DOD cannot provide reasonable assurance to Congress that  its IT budget request includes all funding for the department’s business  systems. For example, we reported in December 2003, that DOD’s IT  budget submission to Congress for fiscal year 2004 contained material  inconsistencies, inaccuracies, or omissions that limited its reliability. We  identified discrepancies totaling about $1.6 billion between two primary  parts of the submission—the IT budget summary report and the detailed  capital investments reports on each IT initiative. These problems were  largely attributable to insufficient management attention and limitations in  departmental policies and procedures, such as guidance in DOD’s Financial  Management Regulation, and to shortcomings in systems that support  budget-related activities.
section_title: DOD Continues to Have Ineffective Control and Accountability over Business System Investments
paragraphs: DOD continues to lack effective management oversight and control over  business systems modernization investments.  While the domains have  been designated to oversee business systems investments, the actual  funding, as shown in table 1, continues to be spread among the military  services and defense agencies, thereby enabling the numerous DOD  components to continue to develop stovepiped, parochial solutions to the  department’s long-standing financial management and business operation  challenges.  Furthermore, the department does not have reasonable  assurance that it is in compliance with the fiscal year 2003 defense  authorization act, which provides that obligations in excess of $1 million  for systems improvements may not be made unless the DOD Comptroller  makes a determination that the improvements are in accordance with the  criteria specified in the act.  Lacking a departmentwide focus and  effective management oversight and control of business systems  investment, DOD continues to invest billions of dollars in systems that fail  to provide integrated corporate solutions to its business operation  problems.
subsections: section_title: DOD Is Working to Finalize Management Oversight Structure and Guidance
paragraphs: In response to our September 2003 report, DOD said that it was taking  several actions to improve the control and accountability over business  systems investments.  However, as of March 2004, many of these actions  had not been finalized.  As a result, the department has not put into place  the organizational structure and process controls to adequately align  business system investments with the BEA.  Each DOD component  continues to make its own investment decisions, following different  approaches and criteria. The lack of an institutionalized investment  strategy has contributed to the department’s current complex, error-prone,  nonintegrated systems environment and precluded the development of  corporate system solutions to long-standing business problems. In  particular, DOD has not clearly defined the roles and responsibilities of the  domains, established common investment criteria, and conducted a  comprehensive review of its ongoing IT investments to ensure that they are  consistent with the BEA.
As we have previously reported, best practices recommend that  investment review boards be established to control an entity’s systems  investments and that the boards use a standard set of investment review  and decision-making criteria to ensure compliance and consistency with  the architecture.  We have also recommended that the department establish  investment review boards to better control investments and that each  board be composed of representatives from across the department.
DOD has decided that in lieu of the investment review boards, the domains  will be responsible for investment management.  In March 2004, the Deputy  Secretary of Defense signed an IT portfolio investment management policy  and assigned overall responsibility to the domains.  However, the specific  roles and responsibilities of the domains had not been formalized and  standard criteria for performing systems reviews have not been finalized.   According to DOD officials, the related detailed directive and instructions  will outline the specific roles and responsibilities of the domains and how  they are to be involved in the overall business systems investment  management process.  The department is drafting a memorandum that will  require the domains to develop a plan for implementing the investment  management policy.
Further, the department has developed draft system review and  certification process guidance that outlines the criteria that are to be used  by the domains and program managers to assess system compliance with  the BEA.  The systems covered in the review process consist of new system  initiatives, ongoing system developmental projects, and systems in  sustainment.  According to DOD, once a system is placed in sustainment,  modernization funding cannot exceed $1 million.  The system review and  certification process guidance has been integrated with the department’s  existing acquisition guidance—commonly referred to as the DOD 5000  series.  The acquisition guidance requires that certain documentation be  prepared at different stages—known as milestones—within the system’s  life-cycle process.  This documentation is intended to provide relevant  information for management oversight and for decision making on whether  the investment of resources is cost beneficial and technically feasible.   DOD officials noted that the system review process would be further  enhanced because the DOD Comptroller will have to certify that the  proposed investment is consistent and aligned with the BEA at each  milestone decision.  According to DOD, the certification process will help  ensure that the obligation of funds of over $1 million for the modernization  of a system are in accordance with the criteria set forth in the fiscal year  2003 defense authorization act.
While these actions are aimed at improving the control and accountability  over business systems investments, we have previously reported that the  department did not adhere to the milestone decision-making and oversight  processes it established to ensure the economical and technical risks  associated with systems modernizations have been mitigated.  For  example, our March 2003 report noted that DOD had not effectively  managed and overseen its planned investment of over $1 billion in four  DFAS system modernization efforts. One project’s estimated cost had  increased by as much as $274 million, while the schedule slipped by almost  4 years. For each of these projects, DOD oversight entities—DFAS, the  DOD Comptroller, and the DOD CIO—could not provide documentation  that indicated they had questioned the impact of the cost increases and  schedule delays, and allowed the projects to proceed in the absence of the  requisite analytical justification.  Such analyses provide the requisite  justification for decision makers to use in determining whether to invest  additional resources in anticipation of receiving commensurate benefits  and mission value.  Two of the four projects—DPPS and DSDS—were  terminated in December 2002 and December 2003, respectively, after an  investment of approximately $179 million that did not improve the  department’s business operations.
While DOD is continuing to work toward establishing the structure and  processes to manage its business systems investments, it has not yet  conducted a comprehensive system review of its ongoing IT investments to  ensure that they are consistent with its BEA efforts.  The domains have  raised concerns that they did not have sufficient staff to perform the  system reviews.  To assist the domains with their system reviews, in  December 2003, the Deputy Secretary of Defense allotted the domains 54  additional staff.  Despite concerns over the sufficiency of staff resources  and the lack of organizational structure and processes for controlling  system investments, the department has acted to curtail the funding for  some systems.  For example, effective October 2003, the DOD Comptroller  directed that the Defense Joint Accounting System (DJAS) be put into  sustainment.  That is, funding would be provided to operate and maintain  the system, but not to upgrade or modernize the system.  In June 2000, the  DOD Inspector General (IG) reported that DFAS was developing DJAS at  an estimated life-cycle cost of about $700 million without demonstrating  that the program was the most cost-effective alternative for providing a  portion of DOD’s general fund accounting.  DJAS is only being operated at  two locations—Fort Benning, Georgia, and the Missile Defense Agency— and there are no longer any plans to implement the system at other  locations.
Another system that DOD has placed into sustainment is the Joint  Computer Aided Acquisition and Logistics Support (JCALS) system.   JCALS was initiated in June 1992 to enable the services to streamline  DOD’s logistical and acquisition functions through business process  reengineering and eliminating existing systems.  In May 2003, Gartner, Inc.,  reviewed the cost, efficiency, and effectiveness of JCALS and reported that  the program is costly to operate and maintain.  The study recommended  freezing all software and technology spending.  According to DOD’s fiscal  year 2004 IT budget, over $1 billion had been invested in JCALS since the  inception of the program.
Placing DJAS and JCALS in sustainment is a step in the right direction.   However, execution of a comprehensive review of all modernization efforts  by DOD before substantial money has been invested will reduce the risk of  continuing the department’s track record of business systems  modernization efforts that cost more than anticipated, take longer than  expected, and fail to deliver intended capabilities.
Further, in developing the fiscal year 2005 budget request, the DOD  Comptroller denied DFAS’s request for approximately $32 million for the  development of an accounting and budget execution system.  The DOD  Comptroller appropriately noted that there should not be investments in a  new system before the domains define the requirements and the system is  justified through the appropriate DOD approval process.  The DOD  Comptroller also denied DFAS’s request for funding of the Disbursing  Transformation Program, which was a proposed $41 million initiative  through fiscal year 2009.  According to DFAS, the program was to be  funded from resources that were budgeted for DSDS, which, as previously  mentioned, was terminated in December 2003. The DOD Comptroller noted  that the department should not pay for salaries, software development, and  systems modernization for a disbursing system before disbursing  functionality is defined according to the BEA. It was further stated that it is  premature for DFAS to create a new disbursing system when it cannot  explain any of the program’s requirements in broad or detailed terms and  numerous disbursing systems already exist.
It is encouraging to see the DOD Comptroller acting to eliminate budget  requests by DFAS for systems that are not justified.  However, DFAS, which  is under the auspices of the DOD Comptroller, represents a very small  percentage—slightly over 2 percent ($103 million of $4.8 billion)—of the  total modernization funding.  Given that the department lacks a  comprehensive inventory of its business systems, it is unknown how many  other modernization projects should be questioned.  However, since the  roles and responsibilities of the domain owners have not been clarified,  they have not been empowered to make investment decisions similar to  those of the DOD Comptroller.  As we have previously recommended, the  department needs to assess its current systems and limit current  investments to  deployment of systems that have already been fully tested and involve  no additional development or acquisition cost;  stay-in-business maintenance needed to keep existing systems   management controls needed to effectively invest in modernized   new systems or existing system changes that are congressionally  directed or are relatively small, cost-effective, and low risk and can be  delivered in a relatively short time frame.
As noted in our September 2003 report, DOD had not yet defined and  implemented an effective approach for selecting and controlling business  system investments.  Absent the rigors of these stringent criteria, DOD will  continue to invest in systems that perpetuate its existing incompatible,  duplicative, and overly costly systems environment that does not optimally  support mission performance.
section_title: DOD Lacks Reasonable Assurance That It Is in Compliance with Statutory Investment Management Controls
paragraphs: DOD has not yet defined and implemented an effective investment  management process to proactively identify and control system  improvements exceeding $1 million in obligations.  DOD officials have  acknowledged that the department does not have a systematic means to  identify and determine which systems improvements should be submitted  to the DOD Comptroller for review and, in essence, depend on system  owners coming forward to the domain owners and requesting approval.   DOD was unable to provide us comprehensive information on all systems  improvements with obligations greater than $1 million since passage of the  act.  However, based upon limited information provided by the military  services for fiscal years 2003 and 2004, we found that modernizations with  obligations totaling at least $479 million were not submitted to the DOD  Comptroller for any factual determination.
The act states that as a condition of making any obligation in excess of   $1 million for system improvements, the obligation be reviewed by the  DOD Comptroller who must make a determination whether the request is  in accordance with criteria specified in the act. To comply with the  legislative requirement, the DOD Comptroller issued a memorandum on  March 7, 2003, to DOD’s component organizations stating that the BMSI  office—which is responsible for overseeing the development and  implementation of the BEA—must review all system improvements with  obligations in excess of $1 million.  In addition, the memorandum directs  the DOD components, as an integral part of the review and approval  process, to present information to DOD Comptroller officials and relevant  domain owners that demonstrates that each investment (1) complies with  the BEA and (2) is economically justified.  To support that the investment is  economically justified, information on the cost and benefit and return on  investment, including the break-even point, must be provided.
DOD officials acknowledge that the department could utilize the IT budget  to assist in the identification of systems that could be subject to the act’s  requirements. While we recognize that this is budgetary data, rather than  the obligational data referred to in the act, this information could provide a  starting point for the domains identifying potential projects that should be  submitted to the DOD Comptroller.  For example, we analyzed the DOD IT  budget request for fiscal years 2003 through 2005 and identified over 200  systems in each year’s budget, totaling over $4 billion per year that could  involve obligations of funds that exceed the $1 million threshold.  Table 3  presents our summary analysis by DOD component.
The list in table 3 may not be complete.  According to the DOD CIO and  military service officials, the “All Other” category in the IT budget exhibits  includes system projects that do not have to be identified by name because  they fall below the $2 million reporting threshold for budgetary purposes.
In an attempt to substantiate that the obligations for business systems  modernization were in accordance with the act, we requested that DOD  activities provide us with a list of obligations greater than $1 million for  fiscal year 2003 and fiscal year 2004, as of December 2003.  As of February  2004, we received responses from the Army, the Navy, and the Air Force,  but did not receive responses from any of the defense agencies such as  DFAS and DLA.  To ascertain if the DOD Comptroller had made the  determination required in the act, we compared a list of system approvals  provided by the BMSI office with the obligational data (by system)  provided by the military services.  Based upon a comparison of the limited  information available, we identified $479 million in reported obligations  over $1 million by the military services for system improvements that were  not submitted to the DOD Comptroller for review and determination as  required by the act.  Table 4 summarizes our analysis.
Examples of DOD system improvements included in table 4 that were not  submitted include the Air Force obligating over $9 million in fiscal year  2003 and about $4 million in fiscal year 2004 for the Integrated Maintenance  Data System, the Navy obligating about $18 million in fiscal year 2003 and  about $6 million in fiscal year 2004 for the Electronic Military Personnel  Records System, and the Army obligating about $22 million in fiscal year  2003 and about $10 million in fiscal year 2004 for the Transportation  Coordinators' Automated Information for Movements System. Appendix III  provides a list of modernization projects with obligations totaling over   $1 million that were reviewed by the DOD Comptroller as required by the  act. Appendix IV provides a detailed list of the individual systems not  submitted to the DOD Comptroller and the related amount of the total  obligations for fiscal years 2003 and 2004.
The act places limitations on the legal authority of individual program and  government contracting officials to obligate funds in support of the  systems for which they are responsible, but DOD has yet to proactively  manage investments to avoid violations of the limitations and to review  investments in any meaningful way to enforce these statutory limitations.
Until DOD strengthens its process for selecting and controlling business  system investments and adopts an effective governance concept, it remains  exposed to the risk of spending billions of dollars on duplicative,  stovepiped, nonintegrated systems that do not optimize mission  performance and accountability and, therefore, do not support the  department’s transformation goals.
We also identified inconsistencies in how the military services categorized  systems.   For example, the Air Force did not categorize its Global Combat  Support System as a business system, while the Army and the Navy  consider their respective Global Combat Support Systems business  systems.  Additionally, the Navy categorized the Defense Message System  as a business system, but the Army and the Air Force did not.  This  inconsistency further reiterates the need for a standard database and  uniform definition of a business system that properly categorizes DOD’s  numerous systems.
For those systems that were submitted for review, we found that most had  the supporting documentation called for in the DOD Comptroller’s March 7,  2003, memorandum.  For example, the return on investment was identified.   However, the one common element lacking was the assertion that the  system projects were compliant with the BEA or otherwise met the criteria  set out in the act.  As noted earlier, BMMP has developed a draft BEA  system compliance assessment certification for program managers to use;  however, the process has not been finalized.  The inability to assert  compliance with the BEA is consistent with our September 2003 report,  which noted that the BEA lacked the details needed to provide DOD with a  common vision and constrain or control investments.  We also identified  instances in which the justification for the approval was questionable.   These investments were made without DOD knowing whether these  systems are aligned or consistent with part of DOD’s long-term system  modernization strategies.  For example:   In October 2003, the DOD Comptroller approved obligations of   $8 million for the Standard Procurement System (SPS) even though the  supporting documentation noted that there was insufficient  documentation to validate all requirements and some were found to be  noncompliant with the BEA.  We and the DOD IG have previously  reported concerns with the overall management and implementation of  SPS and the ability to deliver its intended capability.  Initiated almost 10  years ago in November 1994, the system was to provide DOD with a  single automated system to perform all functions related to contract  management within DOD’s procurement process for all DOD  organizations and activities.  The system was also intended to replace  the contract administration functions currently performed by MOCAS, a  system implemented in 1968 and still operating today. Further, as will be  discussed later in this report, difficulty with the implementation of SPS  is one of the factors that contributed to the slippage in DLA’s BSM  implementation schedule.
In May 2003, the DOD Comptroller approved funding of about $4 million  for the Army’s Integrated Facilities System (IFS).  Initially, the Director  of the BMSI office denied the funding request in part because it was  noted that the system would be replaced by an enterprise solution.  In  response, the installations and environment domain noted that a final  system solution had not been determined and stated that if IFS was  found to be compliant with the “yet to be determined revised business  process,” it could be designated the enterprisewide solution.  The  response also noted that IFS “might prove to have the best functionality  and technical capabilities for a DOD real property inventory solution.”   However, until the department’s BEA becomes more robust, it remains  unclear if this system will be part of the ultimate system solution.  Until  that decision is made, it is unknown what benefit will be derived from  further investment in this system.
We also identified some instances in which the DOD Comptroller’s  approval depended on specific actions being taken by a given date.   However, prior to December 2003, the BMSI office did not have a process in  place to track and follow up on required actions and did not have  reasonable assurance that the required actions were taken.  For example:  In April 2003, the DOD Comptroller approved the expenditure of about  $53 million for the convergence of four separate Navy enterprise  resources planning solutions into one initiative.  This approval was  subsequent to an approval in February 2003 of about $21 million for the  continuance of two of the four Navy efforts.  The approval  memorandum outlined three specific actions that needed to be taken  and established time frames for the completion of each action.  As of  February 2004, BMSI officials were not able to attest to whether these  actions had been completed.  However, the Navy continues to move  forward with this effort.
The DOD Comptroller approved a pilot project for the National Security  Agency on March 7, 2003, for $13.4 million.  The approval depended on  the completion of an overall planning document that outlined the  various areas that were to be addressed.  This document was to be  completed by March 16, 2003.  As of February 2004, BMSI officials stated  that only minimal supporting documentation had been provided.
Thus, even for the systems modernization efforts approved by the DOD  Comptroller, serious questions remain as to whether these investments are  justified.
section_title: BSM and LMP May Have Difficulty Achieving Cost, Schedule, and Operational Goals
paragraphs: BSM and LMP were initiated in November 1999 and February 1998,  respectively, prior to DOD undertaking the BEA and establishing the  domains.  As such, they are not directed toward a corporate solution to  resolving the department’s long-standing weaknesses in the inventory and  logistics management areas, such as total asset visibility or an integrated  systems environment.  Both projects are more focused on DLA’s and the  Army’s respective inventory and logistics management operations.  If  effectively implemented, BSM and LMP are expected to provide benefits  associated with private industry’s logistics reengineering efforts, such as  inventory reduction, improved cycle time, improved customer satisfaction,  and increased response time.  Additionally, BSM and LMP are intended to  improve supply and demand forecast planning, maintenance workload  planning, provide a single source of data, and improve data quality.
However, the initial deployment of BSM and LMP did not operate as  intended and, therefore, did not meet DLA’s and Army’s component-level  needs.  In large part, these operational problems were due to DLA and the  Army not effectively implementing the disciplined process that are  necessary to manage the development and implementation of BSM and  LMP in the areas of requirements management and testing.  DLA and Army  program officials have acknowledged that requirements and testing defects  were factors contributing to these operational problems as well as  schedule slippages and cost increases.  Further, BSM and LMP have  accumulated numerous lessons learned and have assembled teams to  analyze these lessons and to develop an implementation strategy for  corrective action.  Additionally, to their credit, DLA and the Army have  decided that future deployments of BSM and LMP will not go forward until  they have reasonable assurance that the deployed systems will operate as  expected for a given deployment.
subsections: section_title: BSM and LMP Are Not Corporate Solutions to Long-standing Operational Problems
paragraphs: Effectively managing and overseeing the department’s $19 billion  investment in its business systems is key to the successful transformation  of DOD’s business operations.  The transformation also depends on the  ability of the department to develop and implement business systems that  provide users and department management with accurate and timely  information on the results of operations and that help resolve the  numerous long-standing weaknesses.  As DOD moves forward with  continued development and implementation of its BEA, it needs to ensure  that the department’s business systems modernization projects are part of a  corporate solution to preclude the continued proliferation of duplicative,  stovepiped systems.
Three of the long-standing problems in logistics and inventory management  have been related to total asset visibility, integrated systems, and valuation  of inventory.  We found that BSM and LMP will not resolve problems  associated with total asset visibility and integrated systems and the first  deployment of LMP did not provide for the valuation of inventory at the  depot in accordance with federal accounting standards and departmental  guidance.  Details on each of these areas follow.
subsections: section_title: Total Asset Visibility
paragraphs: Although BSM and LMP are enterprise resource planning systems based on  commercial software that incorporates best business practices for logistics  supply chain management, their planned capabilities do not provide a  corporate solution for total asset visibility—a key gap in DOD’s capabilities  to track and locate items across the department.  A corporate solution for  total asset visibility depends on the successful development and  implementation of other systems.  The time frame and costs associated  with these other system projects have not been fully defined.
To illustrate the lack of asset visibility, in October 2002, a DLA official  testified that BSM would provide improved control and accountability over  the Joint Service Lightweight Integrated Suit Technology (JSLIST)—a  chemical/biological suit. The official stated that the JSLIST suits would be  included in BSM at the earliest practicable date, which was estimated to be  December 2003. BSM, however, is not designed to provide the corporate  total asset visibility necessary to locate and track the suits throughout  DOD’s supply chain.  While the suits are expected to be included in a future  deployment of BSM, program officials have not yet specified a date when  they will be included. Even when the suits are included, BSM is designed to  provide visibility over the suits only within the DLA environment— something DLA has stated already exists within its current legacy system  environment.
As we have previously reported, the lack of integrated systems hinders  DOD’s ability to know how many JSLIST it has on hand and where they are  located once they leave the DLA warehouse.  For example, we found that  military units that receive JSLIST from DLA warehouses maintained  inventory data in nonstandard, stovepiped systems that did not share data  with DLA or other DOD systems. The methods used to control and  maintain visibility over JSLIST at the units we visited ranged from stand- alone automated systems, to spreadsheet applications, to pen and paper.  One military unit we visited did not have any inventory system for tracking  JSLIST.  BSM does not address asset visibility outside of DLA’s supply chain  for the JSLIST, and thus cannot provide total asset visibility for this critical  inventory item.
Having the ability to readily locate sensitive items, such as JSLIST, is  critical, particularly if a defect is found and the items must be recalled.  A  case in point is the JSLIST predecessor, the Battle Dress Overgarment  (BDO).  Over 700,000 of these suits were found to be defective and were  recalled.  Since DOD’s systems did not provide the capability to identify the  exact location of each suit, a series of data calls were conducted, which  proved to be ineffective. We reported in September 2001 that DOD was  unable to locate approximately 250,000 of the defective suits and therefore  was uncertain if the suits were still in the possession of the military forces,  or whether they had been destroyed or sold.  Subsequently, we found that  DOD had sold many of these defective suits to the public as excess,  including 379 that we purchased in an undercover operation.  In addition,  DOD may have issued over 4,700 of the defective BDO suits to local law  enforcement agencies.  This is particularly significant because local law  enforcement agencies are most likely to be the first responders to a  terrorist attack, yet DOD failed to inform these agencies that using these  suits could result in death or serious injury. BSM will not provide DOD with  the capability to readily locate JSLIST for any reason, including the need to  recall defective suits.
Similar to BSM, LMP will not provide the Army with total asset visibility  until a suite of other systems has been developed and implemented.  Specifically, Army officials have stated that LMP will require integration  with other Army systems that are under development in order to achieve  total asset visibility within the Army. These additional systems are the  Product Lifecycle Management Plus (PLM+) and Global Combat Support  System—Army (GCSS–A). According to the Army, PLM+ is to integrate  LMP and GCSS–A to create a seamless end-to-end solution for Army  logistics. According to information provided by the Army, PLM+ was  initiated in December 2003.  No estimates have been developed as to the  cost of this project, nor has a time frame for development and  implementation been established.
The Army has stated that GCSS–A will provide visibility of supplies and  equipment in storage and in transit.  The Army began development of  GCSS–A in fiscal year 1997 and since then has invested approximately   $316 million in this effort.  In May 2003, the Army decided to pursue a COTS  solution for GCSS–A rather than continue to develop the system in house.   The Army recently stated that the total cost of GCSS–A cannot be  accurately estimated until all of the “to be” business processes are  identified, which is expected to occur in October 2004. However, the fiscal  year 2004 capital investment report shows that the Army estimates that it  will invest over $1 billion in GCSS–A through fiscal year 2009.
To help provide for departmentwide total asset visibility, DLA is  undertaking the implementation of the Integrated Data Environment (IDE)  program.  According to DLA, this initiative is intended to provide the  capability for routing data from multiple systems within DLA and DOD into  one system.  According to DLA, the contract was signed in September 2003,  and IDE is expected to reach full operational capability in August 2007.   The current estimated cost of the effort is approximately $30 million.   However, the completion date of August 2007 depends on other  departmental efforts being completed on time, for example, PLM+, for  which a completion date has not been established.
section_title: Integrated Systems
paragraphs: One of the long-standing problems within DOD has been the lack of  integrated systems.  This is evident in the many duplicative, stovepiped  systems among the 2,274 that DOD reported as its systems environment.   Lacking integrated systems, DOD will have a difficult time obtaining  accurate and reliable information on the results of its business operations  and will continue to rely on either manual reentry of data into multiple  systems, convoluted system interfaces, or both.  These system interfaces  provide data that are critical to day-to-day operations, such as obligations,  disbursements, purchase orders, requisitions, and other procurement  activities.  For BSM and LMP, we found that the system interfaces were not  fully tested in an end-to-end manner, and therefore DLA and Army did not  have reasonable assurance that BSM and LMP would be capable of  providing the intended functionality.
We previously reported that Sears and Wal-Mart, recognized as leading- edge inventory management companies, had automated systems that  electronically received and exchanged standard data throughout the entire  inventory management process, thereby reducing the need for manual data  entry. As a result, information moves through the data systems with  automated ordering of inventory from suppliers; receiving and shipping at  distribution centers; and receiving, selling, and reordering at retail stores.  Unlike DOD, which has a proliferation of nonintegrated systems using  nonstandard data, Sears and Wal-Mart require all components and  subsidiaries to operate within a standard systems framework that results in  an integrated system and do not allow individual systems development.
For the first deployment, DLA has had to develop interfaces that permit  BSM to communicate with more than 23 systems, including 3 DFAS, 6 DOD- wide, and 14 DLA systems.  The Army has had to develop 215 interfaces  that permit LMP to communicate with more than 70 systems, including 13  DFAS, 6 DLA, 2 Navy, 5 Air Force, and over 24 Army systems. Figures 2 and  3 illustrate BSM’s and LMP’s numerous required system interfaces.
When BSM and LMP became operational, it became evident that the system  interfaces were not working as intended. Such problems have led BSM,  LMP, and organizations with which they interface—such as DFAS—to  perform costly manual reentry of transactions, which can cause additional  data integrity problems. For example:  BSM's functional capabilities were adversely affected because a  significant number of interfaces were still in development or were being  executed manually once the system became operational. Since the  design of system interfaces had not been fully developed and tested,  BSM experienced problems with receipts being rejected, customer  orders being canceled, and vendors not being paid in a timely manner.   At one point, DFAS suspended all vendor payments for about 2 months,  thereby increasing the risk of untimely payments to contractors and  violating the Prompt Payment Act.
In January 2004, the Army reported that due to an interface failure, LMP  had been unable to communicate with the Work Ordering and Reporting  Communications System (WORCS) since September 2003. WORCS is  the means by which LMP communicates with customers on the status of  items that have been sent to the depot for repair and initiates  procurement actions for inventory items.  The Army has acknowledged  that the failure of WORCS has resulted in duplicative shipments and  billings and inventory items being delivered to the wrong locations.   Additionally, the LMP program office has stated that it has not yet  identified the specific cause of the interface failure. The Army is  currently entering the information manually, which as noted above, can  cause additional data integrity errors.
While these numerous interfaces are necessary because of the existing  stovepiped, nonintegrated systems environment, they should have been  fully developed and tested prior to BSM and LMP being deployed.  In  moving forward with the future deployments of BSM and LMP, it is critical  that program officials ensure that the numerous system interfaces are  operating as intended. Additionally, until the business enterprise  architecture is further developed and DOD has decided which systems will  be part of the future business systems environment, there is uncertainty as  to the number of these systems with which BSM and LMP will continue to  interface.
section_title: Valuation of DOD Inventory
paragraphs: Federal accounting standards require inventories to be valued based on  historical costs or a method that approximates historical costs. DOD’s  inability to account for and control its huge investment in inventories  effectively has been an area of major concern for many years.  DOD’s  antiquated, duplicative systems do not capture the information needed to  comply with federal accounting standards.  BSM and LMP are to provide  DOD the capability to comply with federal accounting standards in the  valuation of its billions of dollars of inventory.  DLA has stated that BSM  has the capability to compute the value of inventory in accordance with  federal accounting standards.  Based upon information provided by DLA  and our analysis, we found that the value of the inventory recorded in BSM  changed each time new items were procured to reflect a moving average  (historical) cost valuation of the inventory—which is an acceptable method  permitted by federal accounting standards and is in accordance with DOD’s  stated policy.
However, the first deployment of LMP did not have the capability to value  all inventory in accordance with federal accounting standards.  In its  evaluation of LMP, the Army Audit Agency found that it had the capability  to compute the value of inventory in accordance with federal accounting  standards at the command level—CECOM—but not at the depot level.  The  Army decided to proceed with deployment of LMP, recognizing that the  issue would have to be resolved prior to further deployments to the other  depots.  The Office of the DOD Comptroller has also directed that there is  to be no further deployment of LMP until the inventory valuation problem  has been fixed.
section_title: Significant Problems Appeared Once BSM and LMP Became Operational
paragraphs: BSM and LMP experienced significant problems once they became  operational at the first deployment sites.  Although BSM and LMP were not  designed to provide a corporate enterprise solution for inventory and  logistics management, the first deployment did not address DLA’s and  Army’s component-level operational needs as intended. These problems  have resulted in schedule slippages and cost increases.   Detecting such  problems after the system is placed into operation leads to costly rework  due to factors such as (1) fixing the defect, (2) entering transactions  manually, and (3) adjusting reports manually.  Furthermore, the manual  processes required to enter the transactions and adjust related reports may  introduce data integrity errors.  Our analysis indicated that many of the  operational problems experienced by DLA and the Army can be attributed  to their inability to effectively implement the disciplined requirements  management and testing processes, as discussed in this report.  In fact,  DLA and Army program officials acknowledged that requirements and  testing defects were factors contributing to the operational problems and  stated that they are working to develop more effective processes.  DLA and  the Army recognized that serious operational problems exist and have  decided that future deployments will not go forward until they have  assurance that the deployed system operates as expected for a given  deployment.  Operational problems include the following:  Army and DFAS officials reported that LMP’s operational difficulties at  CECOM and Tobyhanna Army Depot have resulted in inaccurate  financial management information. More specifically, the depot is not  (1) producing accurate workload planning information; (2) generating  accurate customer bills; and (3) capturing all repair costs, which is  impeding the Army’s ability to calculate accurate future repair prices.  These problems can also hinder the Army’s ability to accurately report  the results of its depot operations and limits customers’ ability to  develop accurate budget estimates.
LMP users experienced difficulty in providing contract information to  MOCAS.  Due to the operational problems, DFAS was unable to  electronically process contract modifications and contract payment  terms and make disbursements to contractors, thereby increasing the  risk of not making timely payments to contractors and violating the  Prompt Payment Act.
BSM experienced significant data conversion problems associated with  purchase requisitions and purchase orders that were created in SAMMS.   Moving the data from SAMMS to BSM proved difficult because BSM  required more detailed information, which was not identified during the  requirements phase.  This additional information needed to be manually  entered into BSM, resulting in numerous errors that caused vendors not  to be recognized and shipments from the depot to be rejected. As a  result of these problems, additional tables, such as vendor master files,  were created within BSM to process orders for the converted purchase  requisitions and purchase orders.
BSM users experienced a number of problems, such as incorrect  information on customer orders, customer orders never being sent, and  vendor invoices not being paid in a timely manner.
These operational problems have been at least partially responsible for  schedule slippages and cost increases for both systems.  In the case of  BSM, it was originally scheduled to achieve full operational capability  (FOC) in September 2005.  However, BSM is now expected to reach FOC  during the second quarter of fiscal year 2006. Further, BSM’s estimated cost  has increased by approximately $86 million since the program was initiated  in November 1999. Figure 4 shows the schedule slippages and cost  increases.
Part of the schedule slippage and cost increase can be attributed to  problems encountered with DLA’s effort to implement SPS, which was to  provide BSM with the required procurement functionality.  Since a large  part of DLA’s overall business is the procurement of inventory items,  difficulties in establishing a viable system solution for this critical aspect of  its business seriously impaired DLA’s ability to meet BSM’s schedule and  cost goals. We have previously reported that DOD’s ineffective  management approach for SPS put the project at risk.
During the initial implementation of BSM, program officials found that SPS  did not have the capability to handle DLA’s large volume of procurement  requisitions.  According to BSM program officials, DLA will spend about   $9 million to resolve the shortcoming in SPS.  Since SPS will not meet its  needs when BSM is fully operational at all sites, DLA has negotiated with  the BSM software developer to purchase new procurement software as the  long-term solution.  DLA estimated that this software would cost  approximately $30 million, which contributed to the increased BSM  program costs.
Similar to BSM, LMP has also experienced schedule slippages and cost  increases since the project was approved in February 1998.  Figure 5 shows  the schedule slippages and cost increases.
As shown in figure 5, as of March 2004, the current estimated cost of LMP is  over $1 billion, with more than $400 million spent to fund the project  during the past 5 years.  In October 1999, we reported that the Army’s  estimated cost of LMP over the 10-year period of the contract was  approximately $421 million.  However, as discussed in that report, the   $421 million estimate did not include an additional $30.5 million per  contract year that would be needed for data processing. The amount  allowed for data processing in the original estimate was based directly on  the percentage of data processing performed by the contractor, with the  Defense Information Systems Agency performing the residual processing.  Further, the original estimate was based on a 10-year contract and the  current estimate is based on a 12-year contract, and each additional  contract year can be as much as $65 million.  Considering these two  factors, a more accurate cost estimate in 1999 would have been  approximately $856 million.
In our discussions with LMP program officials, additional factors were  identified that have caused the cost of LMP to increase to over $1 billion.   For example, since the initiation of LMP, the Army has directed that the  program be (1) integrated with the Army Single Stock Fund effort and   (2) extended to the Army depot maintenance operations.  These additional  capabilities were not part of the standard LMP software package and were  not envisioned to be part of LMP when the original cost estimate was  developed.  Therefore, additional development and implementation costs  were incurred and increased the overall cost of the program by over   $91 million.  Further, the LMP program manager acknowledged that the  1999 estimate did not include adequate DOD program management costs.   The additional program management costs are estimated to be about   $104 million and include such items as personnel and travel.
Additionally, as shown in figure 5, the original FOC date was scheduled for  fiscal year 2004.  However, because of the operational problems that were  identified with the first deployment, the Army is in the process of  developing a new deployment schedule, and as of March 2004, no future  deployment dates had been established.
section_title: Project Management of BSM and LMP Did Not Follow Disciplined Processes
paragraphs: The problems we identified in the areas of schedule, cost, and performance  of the two systems can be linked, at least in part, to DLA’s and the Army’s  failure to follow disciplined processes in the key areas of requirements  management and testing.  While there may have been contributing factors  in other areas of the system acquisition efforts, we selected these two areas  because our assessments, as well as others, have shown that agencies do  not invest adequately for success in these areas, which form the foundation  for success or failure.  Lacking such disciplined processes exposes these  projects to the unnecessary risk that costly rework will be required, which  in turn, will continue to adversely affect these projects’ cost, schedule, and  performance goals.  Our analysis of selected BSM and LMP key  requirements and testing processes found that (1) the functionality to be  delivered was not adequately described or stated to allow for quantitative  evaluation; (2) the traceability among the various process documents (e.g.,  operational requirements documents, functional or process scenarios, and  test cases) was not maintained; and (3) system testing was ineffective.  Because of the weaknesses in these key processes, program officials do not  have reasonable assurance that (1) the level of functionality that will be  provided by a given deployment is understood by the project team and  users and (2) the resulting system will provide the expected functionality.  We have previously reported concerns with BSM’s  lack of a documented  requirements development and management plan.  Such a plan provides a  road map for completing important requirements development and  management activities.  Without it, projects risk either not performing  important tasks or not performing them effectively.  Historically, projects  that experience the types of requirements and testing process weaknesses  found in BSM and LMP have a high probability of not meeting schedule,  cost, and performance objectives.
subsections: section_title: Disciplined Processes Are Key to Successful System Development and Implementation
paragraphs: Disciplined processes have been shown to reduce the risks associated with  software development and acquisition efforts to acceptable levels and are  fundamental to successful systems acquisition. Said another way, a  disciplined software development and acquisition process can maximize  the likelihood of achieving the intended results (performance) within  established resources (costs) on schedule. Although a “standard  cookbook” of practices that will guarantee success does not exist, several  organizations, such as the Software Engineering Institute and the Institute  of Electrical and Electronics Engineers (IEEE), and individual experts  have identified and developed the types of policies, procedures, and  practices that have been demonstrated to reduce development time and  enhance effectiveness. Key to having a disciplined system development  effort is to have disciplined processes in multiple areas, including project  planning and management, requirements management, configuration  management, risk management, quality assurance, and testing. Effective  processes should be implemented in each of these areas throughout the  project’s life cycle since constant changes occur.  In reviewing BSM and  LMP, we focused on requirements management and testing.
Requirements represent the blueprint that system developers and program  managers use to design, develop, and acquire a system. Requirements  should be consistent with one another, verifiable, and directly traceable to  higher-level business or functional requirements. It is critical that  requirements be carefully defined and that they flow directly from the  organization’s concept of operations (how the organization’s day-to-day operations are or will be carried out to meet mission needs). Improperly  defined or incomplete requirements have been commonly identified as a  cause of system failure and systems that do not meet their costs, schedules,  or performance goals. Without adequately defined requirements that have  been properly reviewed and tested, significant risk exists that the system  will need extensive and costly changes before it will achieve its intended  capability.
According to IEEE—a leader in defining the best practices for such  efforts—good requirements have several characteristics, including the  following: The requirements fully describe the software functionality to be  delivered. Functionality is a defined objective or characteristic action of  a system or component. For example, for inventory, key functionality as  previously discussed includes total asset visibility and valuation in  accordance with federal accounting standards.
The requirements are stated in clear terms that allow for quantitative  evaluation. Specifically, all readers of a requirement should arrive at a  single, consistent interpretation of it.
Traceability among various requirement documents is maintained.
Requirements for projects can be expressed at various levels depending  on user needs. They range from agencywide business requirements to  increasingly detailed functional requirements that eventually permit the  software project managers and other technicians to design and build the  required functionality in the new system.  Adequate traceability ensures  that a requirement in one document is consistent with and linked to  applicable requirements in another document.
Industry best practices, as well as DLA’s and Army’s own system planning  documents, indicate that detailed system requirements should be  documented to serve as the basis for effective system testing.  Both  projects documented their high-level or operational requirements and had  designed hierarchical processes for documenting the various requirements  and related documents needed to build and design tests at the transaction  level as well as tests of chains of transactions that flow together to support  multiple business functions and processes.
Because requirements provide the foundation for system testing,  specificity and traceability defects in system requirements preclude an  entity from implementing a disciplined testing process. That is,  requirements must be complete, clear, and well documented to design and  implement an effective testing program. Absent this, an organization is  taking a significant risk that its testing efforts will not detect significant  defects until after the system is placed into production. Industry  experience indicates that the sooner a defect is recognized and corrected,  the cheaper it is to fix. As shown in figure 6, there is a direct relationship  between requirements and testing.
Although the actual testing activities occur late in the development cycle,  test planning can help disciplined activities reduce requirements-related  defects. For example, developing conceptual test cases based on the  requirements derived from the concept of operations and functional  requirements stages can identify errors, omissions, and ambiguities long  before any code is written or a system is configured. Disciplined  organizations also recognize that planning testing activities in coordination  with the requirements development process has major benefits.
section_title: BSM and LMP Functionality to Be Delivered Was Not Adequately Described
paragraphs: Our analysis and evaluation of DLA’s and the Army’s requirements  management and testing processes found that BSM and LMP program  officials did not effectively implement the disciplined processes associated  with requirements management and testing in developing and  implementing their systems.  We identified numerous instances in which  each documented requirement used to design and test the system did not  build upon the next in moving through the hierarchy.  Specifically, the  requirements (1) lacked the specific information necessary to understand  the required functionality that was to be provided and (2) did not describe  how to determine quantitatively, through testing or other analysis, whether  the systems would meet DLA’s and Army’s respective needs.  One reason  that users have not been provided with the intended systems capabilities is  because of the breakdown in the requirements management process.  As a  consequence, DLA and the Army have been forced to implement error- prone, time-consuming manual workarounds as a means to minimize  disruption to critical operations.  DLA and Army officials acknowledged  that improvements in their requirements management processes are  needed and have stated that they are working to develop more specific  requirements that better describe required system functionality and  support more effective system testing.
DLA’s basic hierarchical approach to developing BSM requirements was to  (1) define high-level requirements, commonly referred to as operational  requirements; (2) define more specific blueprint requirements; (3) develop  functional scenarios; (4) define functional designs; (5) define technical  designs; (6) create test cases; and (7) define test conditions. Similarly, the  Army’s basic approach to developing LMP system requirements was to   (1) develop a blueprint of its business processes following the Integration  Definition for Function modeling standards established by the National  Institute of Standards and Technology and IEEE, (2) define high-level  requirements, (3) develop process scenarios, (4) develop test cases, and   (5) use subject matter experts to determine whether the application met  the business processes envisioned by the users and as developed by a  contractor to provide the functionality currently provided by the Army’s  existing systems.  If effectively implemented, either methodology can be  used to develop and implement a system.  The key is that each step of the  process builds upon the previous one.  Accordingly, unidentified defects in  one step migrate to the subsequent steps where they are more costly to fix  and increase the risk that the project will experience adverse impacts on its  schedule, cost, and performance objectives.  The following are examples of  the BSM and LMP requirements that we reviewed that lacked the  specificity necessary to describe the functionality to be delivered.
One BSM requirement stated that the system should be able to reconcile  inventory between the depots (where inventory items are located) and  the inventory control point and that the reconciliation should be  performed daily.  It also stated that the inventory control point must  request that the depot perform a physical count once inventory  differences have met certain criteria, such as dollar value or large  quantities.  However, the various requirement documents did not   (1) define what is meant by “large” or (2) specify how the notification of  the requirement to conduct the inventory was to be accomplished, for  example, by e-mail.  Without such specificity, it is unclear how this  requirement could be tested since an evaluator would not be able to  design a test of the trigger for a physical count because the quantity  difference had not been defined.
For LMP, the operational activity “Manage Assets” did not adequately  describe how to maintain visibility over all assets.  Specifically, the  requirement states that the system “maintains wholesale and retail asset  balances and provides visibility of On-Hand Asset Balances by  identifying assets being repaired, modified, or tested at depots,  contractor and intermediate level repair facilities as well as those on- hand at storage sites, retail activities and other services.” However,  there is no further information that specifies how asset visibility is  maintained or the sources that are to be used in accumulating these  data.  Therefore, the risk is increased that the Army will not be able to  maintain asset visibility over all Army-managed assets. In fact, in  January 2004, the Army reported that it was having difficulty obtaining  accurate data related to material movement (in-transit), assets received,  and assets issued or shipped.
section_title: Requirements Traceability Was Not Maintained
paragraphs: In reviewing the process documents that DLA and Army used to define  their requirements, that is, operational requirement, functional scenario,  functional design, technical design, and test case, we found that the  forward and backward traceability defined by IEEE, and as described by  BSM’s and LMP’s hierarchical approaches and management plan, was not  always maintained. Traceability allows the user to follow the life of the  requirement both forward and backward through these documents and  from origin through implementation. Traceability is also critical to  understanding the parentage, interconnections, and dependencies among  the individual requirements. This information, in turn, is critical to  understanding the impact when a requirement is changed or deleted.   Without an effective traceability approach, it is very difficult to perform  actions such as (1) accurately determining the impact of changes and  making value-based decisions when considering requirement changes,   (2) maintaining the system once it goes into production, (3) tracking the  project's progress, and (4) understanding the impact of a defect discovered  during testing. For almost all of the requirements we analyzed, we found  that traceability was not maintained.  For example:  An operational requirement stated that BSM maintain the effective date  for pricing information.  The subsequent requirements document stated  that all amendments/modifications to the award instrument—purchase  orders and requisitions—should be documented on the prescribed  General Services Administration form.  In our analysis, we were only  able to trace portions of the requirements through BSM’s hierarchical  process.  Since traceability was not maintained through the key  documents, it was unclear why the testing documents included  requirements that were not included in the functional scenarios,  technical design, or test conditions, since these documents should have  provided the detailed information necessary to test the requirements.  Further, since traceability is lacking, it is uncertain how DLA will ensure  that BSM will meet this requirement.
One capability of LMP is to support workload planning for the Army’s  depot maintenance facilities. Data related to scheduled and historical  depot maintenance activities that should be considered in developing  budget requirements, such as assets due in for repair or maintenance,  price data, assets in stock, and maintenance schedules, were included in  the requirement.  However, we found that only the prior month’s sales  data were used in designing the test case—not the information specified  in the requirement.  As a result, the risk is increased that LMP is  determining workload-planning requirements for the Army’s depot  maintenance facilities using incorrect data. This resulted in the Army  reporting in January 2004 that Tobyhanna Army Depot was unable to  develop its working capital fund budget submissions for its operations  and that it will have to perform complex manual calculations to satisfy  its budgetary planning requirements.
section_title: System Testing Was Not Effective
paragraphs: BSM and LMP did not implement disciplined testing activities.  Not  carrying out this recognized best practice materially increases the risk that  defects would not be detected until the systems were placed into  production and that costly rework will be needed to satisfy end-user  requirements, including materiel readiness in support of military  operations.  Testing is the process of executing a program with the intent of  finding errors.  Furthermore, if a requirement has not been adequately  defined, it is unlikely that a test will discover a defect.  System testing is a  critical process utilized by disciplined organizations and improves an  entity’s confidence that the system will satisfy the requirements of the end  user and will operate as intended.  Since requirements provide the  foundation for system testing, requirement defects discussed earlier, such  as the lack of specificity, significantly impaired and will continue to impair  the ability of DLA and the Army to detect defects during system testing.  As  a result of requirement defects and ineffective testing, DLA and the Army  testing activities did not achieve the important goal of reducing the risk  that BSM and LMP would not operate as intended.  For example:  One BSM requirement involved preparing customer payments.  The  system, according to the test case, was required to (1) prepare a  summary bill and (2) present the sales summary report in federal supply  class sequence.  The actual result for one test stated that the system  passed this test even though only one item was used to generate the  summary bill.  It was unclear from this test case whether the system   (1) could summarize multiple items and (2) had any limitations on the  number of items that could be summarized.  Furthermore, the test that  evaluated the sorting of items by federal supply class divided the cost of  the sales summary report by two.  If this result matched the expected  result, BSM passed the test. However, documentation was not available  to explain why the item cost needed to be divided by two.  Based on our  review of the test cases linked to this requirement, we could not validate  that the requirement had been adequately tested.  Therefore, DLA does  not have reasonable assurance that BSM can perform this required  functionality.
Based on our analysis of LMP’s December 2003 and January 2004 project  status reports, we found that the Army continued to experience  problems with the accuracy of data related to budgeting; workload  planning and forecasting and depot maintenance operations; and  accounting records such as customer orders, purchase orders and  requisitions, obligations, and disbursements. DFAS and Army officials  acknowledged that these problems were attributable to relying on  subject matter experts to develop tests for their respective functional  areas, such as budgeting, accounting, and workload planning, and not  performing testing end to end across the various functional areas.   Rather, the testing was stovepiped in that subject matter experts  performed tests for their own respective areas.
section_title: Accuracy of Financial Reporting May Not Be Improved by BSM and LMP
paragraphs: As a result of the specific problems discussed in this report related to BSM  and LMP, such as the lack of total asset visibility, DLA and the Army cannot  be assured that BSM and LMP will routinely generate timely, accurate, and  useful financial information.  The inaccuracy and unreliability of financial  information has been a long-standing DOD weakness.  As mentioned  previously, BSM and LMP rely on information received from and sent  through the various systems.  However, the interfaces with these multiple  systems were not fully developed, nor were they tested when BSM and  LMP become operational.  As a result, DLA and the Army do not have  reasonable assurance that their respective systems are capable of  providing the intended capability.  In fact, the reported operational  problems clearly indicate that BSM and LMP are not providing accurate  data.  For example, the manual workarounds that were required to  compensate for the data conversion problems associated with SAMMS  caused additional errors, which affected the accuracy of data produced.  In  the case of LMP, the Army has acknowledged that accurate information on  its depot operations is not readily available.  This problem severely impairs  the Army’s ability to develop accurate prices for its depot operations.   Inaccurate prices could result in customers being charged too much or too  little for the services provided.  Furthermore, the overall concerns we  raised with regard to DLA and the Army not following disciplined  processes in the key areas of requirements management and testing further  expose BSM and LMP to unnecessary risks.  Specifically, the resulting  systems will not provide the accurate and complete information that is  crucial to making informed decisions and controlling assets so that DOD’s  mission and goals are efficiently and effectively accomplished.
Further, although DLA and the Army have asserted that BSM and LMP,  respectively, are compliant with the requirements of the Federal Financial  Management Improvement Act of 1996 (FFMIA), we have concerns with  the methodology followed in reaching that conclusion. FFMIA builds on  the foundation laid by the Chief Financial Officers Act (CFO) of 1990 by  emphasizing the need for agencies to have systems that can generate  reliable, useful, and timely information with which to make fully informed  decisions and to ensure accountability on an ongoing basis. FFMIA  requires the 23 major departments and agencies covered by the CFO Act to  implement and maintain financial management systems that comply  substantially with (1) federal financial management systems requirements,  (2) applicable federal accounting standards, and (3) the U.S. Government  Standard General Ledger (SGL) at the transaction level.
DLA’s and the Army’s assertions are based upon self-assessments of the  financial management requirements that were reviewed by independent  parties. For both systems, testing of transactions was not performed to  validate that they would be able to process the data as intended.  For  example, in the case of BSM, for one requirement the contractor stated that  “a sample of transactions were reviewed,  it appears that BSM  properly records transactions consistent with the SGL posting rules.”  However, we found no indication that this requirement was tested, and  therefore, we cannot conclude whether BSM has the capability to meet this  requirement.
In the case of LMP, we found that the Army relied upon Joint Financial  Management Improvement Program (JFMIP) testing for 147 requirements  because JFMIP had validated these requirements when it tested the  vendor’s commercial software used for LMP during fiscal year 1999. JFMIP  testing should not be considered a substitute for individual system testing  of the actual data that will be used by the entity.  Further, JFMIP’s tests of  the software do not address entity-specific integrated tests of end-to-end  transactions or system interfaces. Because the Army had to make  modifications to the basic commercial software package to accommodate  some of its business operations, the Army cannot be assured, without  retesting, that these 147 requirements will produce the intended results.
Without adequate documentation to support testing of the FFMIA  requirements and based on our findings, it is questionable whether BSM  and LMP are substantially compliant with FFMIA. As a result, DLA and the  Army cannot provide reasonable assurance that BSM and LMP will  routinely generate timely, accurate, and useful information with which to  make informed decisions and to ensure accountability on an ongoing basis.
section_title: Conclusions
paragraphs: DOD has made limited progress in achieving effective management  oversight, control, and accountability over its $19 billion in business system  investments.  As a result, DOD cannot provide Congress reasonable  assurance that the billions of dollars being spent annually on system  modernizations are not being wasted on projects that will perpetuate the  current costly, nonintegrated, duplicative systems environment.  Our two  cases studies—BSM and LMP—are prime examples of DOD business  system modernization projects costing billions of dollars that are not  directed toward a corporate solution for resolving some of DOD’s long- standing financial and inventory management problems. Rather, these  efforts are more narrowly focused on DLA’s and the Army’s business  operations, but even within that more restricted scope, weaknesses in  project management have resulted in problems in delivering the intended  capabilities.  As the department moves forward with the continued  development and implementation of the business enterprise architecture, it  is critical that actions be taken to gain more effective control over business  system funding.  Maintaining the status quo of permitting each of the  military services and DOD agencies to manage and oversee its business  systems investments only serves to perpetuate the existing nonintegrated  and duplicative systems environment and continues to impede the  department’s overall transformation as envisioned by the Secretary of  Defense.
section_title: Matters for Congressional Consideration
paragraphs: The manner in which business system funding is currently controlled  hampers the development and implementation of broad-based, integrated  corporate system solutions to address DOD-wide problems.  Each military  service and defense agency receives its own funding and is largely  autonomous in deciding how to spend these funds, thereby enabling  multiple system approaches to common problems.  This funding structure  has contributed to the duplicative, nonintegrated, error-prone systems  environment that exists today.  To improve management oversight,  accountability, and control of the department’s business systems funding,  Congress may wish to consider the following four legislative initiatives:    Assign responsibility for the planning, design, acquisition, deployment,  operation, maintenance, modernization, and oversight of business  systems to domain leaders (e.g., the Under Secretary of Defense for  Acquisition, Technology and Logistics and the DOD CIO).
Direct the Secretary of Defense, in coordination with the domain  leaders, to develop a defense business system budget that (1) identifies  each business system for which funding is being requested, (2) identifies  all funds by appropriation type and whether they are for current  services or modernization, and (3) provides justification for expending  funds on system(s) that are not in compliance with the department’s  business enterprise architecture.
Appropriate funds to operate, maintain, and modernize DOD’s business  systems to domain leaders rather than the military services and defense  agencies.
Direct that each domain establish a business system investment review  board that is to be composed of representatives from the military  services and defense agencies who will be responsible for review and  approval of all business system investments.
section_title: Recommendations for Executive Action
paragraphs: To help improve the department’s (1) control and accountability over its  business systems investments and (2) future deployments of BSM and LMP,  we are making the following four recommendations.  We recommend that  the Secretary of Defense direct:   The Under Secretary of Defense (Comptroller) and the Assistant  Secretary of Defense for Networks and Information Integration to  develop a standard definition for DOD components to use to identify  business systems.
The Assistant Secretary of Defense for Networks and Information  Integration to expand the existing IT Registry to include all business  systems.
The Under Secretary of Defense (Comptroller) to establish a mechanism  that provides for tracking all business systems modernization  conditional approvals to provide reasonable assurance that all specific  actions are completed on time.
The Director, Defense Logistics Agency, and the Commanding General,  Army Materiel Command, to take the following actions:  Develop requirements that contain the necessary specificity to  reduce requirements-related defects to acceptable levels. The  requirements management process used to develop and document  the requirements should be adequate to ensure that each requirement  (1) fully describes the functionality to be delivered; (2) includes the  source of the requirement; (3) is stated in unambiguous terms that  allow for quantitative evaluation; and (4) is consistent, verifiable, and  traceable.
Conduct thorough testing before (1) making further deployment  decisions and (2) adding functionality to existing deployment  locations.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the Acting  Under Secretary of Defense (Comptroller) (see app. II).  DOD agreed with  our four recommendations to the Secretary of Defense and two of the four  matters for congressional consideration.  With regard to the  recommendations to the Secretary of Defense, the department identified  actions it has under way and planned to address the concerns discussed in  the report.   For example, the department stated that a system has been  developed that will track all business systems modernization conditional  approvals until all required actions are completed.  In addition, the  department acknowledged that the initial implementations of BSM and  LMP experienced problems that could be attributed to the lack of adequate  requirements determination and system testing.  To address these  inadequacies, the department noted that requirements analysis had been  expanded to include greater specificity and to require the successful  completion of comprehensive testing prior to further implementation of  either system.  The department also stated that industry best practices  would be followed.
With regard to our matters for congressional consideration, the department  disagreed that (1) responsibility for the planning, design, acquisition,  deployment, operation, maintenance, modernization, and oversight of  business systems be assigned to domain leaders (e.g., the Under Secretary  of Defense for Acquisition, Technology and Logistics and the DOD CIO)  and (2) funds to operate, maintain, and modernize DOD’s business systems  be appropriated to domain leaders rather than the military services and  defense agencies. On the first matter, the department stated that it is  developing its business enterprise architecture and its business IT  investment management structure and that these structures will provide  the necessary management and oversight responsibility.  DOD also noted  that business system portfolio management would be an integral part of its  oversight efforts.  Further, DOD noted that the domain leaders will work  closely with component acquisition executives and the DOD CIO, who have  statutory responsibilities for IT related investment activities.
We continue to believe that Congress may wish to consider assigning to the  domains the responsibility for the planning, design, acquisition,  deployment, operation, maintenance, modernization, and oversight of  business systems. DOD components being responsible for these functions  has resulted in the existing business system environment of at least 2,274  systems that are not capable of providing DOD management and Congress  accurate, reliable, and timely information on the results of the department’s  vast operations.  DOD has recently stated that the actual number of  systems could be twice the amount currently reported.  Further, because  the various DOD components are largely autonomous, despite DOD’s  assertion that component acquisition executives will work more closely  with domain leaders under current statutory structure, there is no incentive  for them to seek corporate solutions to problems.  Our two case studies— BSM and LMP—clearly demonstrate that these two system modernization  efforts are not directed toward a corporate solution to resolving the  department’s long-standing weaknesses in areas such as inventory and  logistics management.  Within the current departmental organization  structure, DOD components are able to develop multiple system  approaches to common problems.
With regard to the funding being provided to the domains, the department  stated that the portfolio management process being established—to  include investment review boards—would provide the appropriate control  and accountability over business system investments. DOD also noted that  beginning with the fiscal year 2006 budget review process, the domains will  be actively involved in business system investment decisions.  While the  establishment of the investment review boards is consistent with our  previous recommendations, we continue to believe that appropriating  funds for DOD business systems to the domains will significantly improve  accountability over business system investments.  DOD’s comments  indicate that the domains will be more accountable for making business  system investment decisions, but unless they control the funding, they will  not have the means to effect real change.  Continuing to provide business  system funding to the military services and defense agencies is an example  of the department’s embedded culture and parochial operations.   As a  result of DOD’s intent to maintain the status quo, there can be little  confidence that it will not continue to spend billions of dollars on  duplicative, nonintegrated, stovepiped, and overly costly systems that do  not optimize mission performance and accountability and, therefore, do  not support the department’s transformation goals.
As agreed with your offices, unless you announce the contents of this  report earlier, we will not distribute it until 30 days after its date.  At that  time, we will send copies to the Chairmen and Ranking Minority Members,  Senate Committee on Armed Services; Subcommittee on Defense, Senate  Committee on Appropriations; House Committee on Armed Services;  Subcommittee on Defense, House Committee on Appropriations; Senate  Committee on Governmental Affairs; and House Committee on  Government Reform.  We are also sending copies to the Director, Office of  Management and Budget; the Under Secretary of Defense (Comptroller);  the Under Secretary of Defense (Acquisition, Technology and Logistics);  the Assistant Secretary of Defense (Network and Information Integration);  the Director, Defense Logistics Agency; and the Commanding General,  Army Materiel Command. Copies of this report will be made available to  others upon request.  The report is also available at no charge on GAO’s  Web site at http://www.gao.gov.
If you or your staff have any questions on matters discussed in this report,  please contact Gregory D. Kutz at (202) 512-9505 or kutzg@gao.gov or  Keith A. Rhodes at (202) 512-6412 or rhodesk@gao.gov.  GAO contacts and  key contributors to this report are listed in appendix V.
section_title: Scope and Methodology
paragraphs: We reviewed the Department of Defense’s (DOD) $28 billion fiscal year  2004 information technology (IT) budget request to determine what portion  of the budget relates to DOD business systems.  We reviewed the budget to  determine, of the approximately $19 billion related to the department’s  business systems, the amount allocated for operation, maintenance, and  development.  Additionally, we reviewed DOD’s business systems  inventory, as reported by the department in April 2003, to ascertain if the  systems were identified in the budget request.  To obtain an overview of  how an IT budget request is developed, we also met with officials in the  offices of the DOD Comptroller and DOD Chief Information Officer (CIO),  as well as CIO and financial management officials from the military  services.
To determine the effectiveness of DOD’s control and accountability over its  business systems investments, we met with DOD officials to obtain an  update on the status of our prior recommendations. We also met with  appropriate officials in the DOD Comptroller and DOD CIO offices to  discuss the status of various draft policies and guidance that are aimed at  improving the department’s control and accountability over business  system investments. We also reviewed and analyzed the DOD budget  requests for fiscal years 2003 through 2005 to identify the business systems  investments that could be subject to the requirements of the Bob Stump  National Defense Authorization Act for Fiscal Year 2003, which requires  the DOD Comptroller to review all system improvements with obligations  exceeding $1 million and make a determination whether the improvement  is in accordance with criteria specified in the act.  To assess DOD’s  compliance with the act, we also obtained and reviewed departmental  guidance, memorandums, DOD Comptroller review decisions, and other  documentation provided by the Business Management Systems Integration  (BMSI) office. Additionally, we requested that DOD provide us obligational  data in excess of $1 million for business systems for fiscal years 2003 and  2004, as of December 2003. We received obligational data from the military  services, but did not receive any information from the defense agencies. We  then compared the obligation data provided by the military services with  the information from the BMSI office to determine if the modernizations  were reviewed as stipulated by the act. To augment our document reviews  and analyses, we interviewed officials from various DOD organizations,  including the Office of the Under Secretary of Defense (Comptroller);  Office of the Under Secretary of Defense (Network and Information  Integration)/Chief Information Officer; Office of the Under Secretary of  Defense (Acquisition, Technology and Logistics); and CIO and financial  management officials from the military services.
To determine if selected DOD business system projects are being  effectively managed and will help resolve some of DOD’s long-standing  business operation problems, we selected the logistics domain from which  we chose individual case studies for detailed review.  We selected the  logistics domain because it represents $770 million, or 16 percent, of  modernization funding requested in fiscal year 2004 for the department’s  business systems.  The logistics domain was also selected because of its  significance to DOD operations and its long-standing and inherent  inventory and related financial management weaknesses, such as the  inability to support its inventory balances and provide total asset visibility.   We selected the Defense Logistics Agency’s (DLA) Business Systems  Modernization (BSM) and the Army’s Logistics Modernization Program  (LMP) for detailed review. For these two business systems, we focused on  two key processes, requirements management and testing.
To assess whether DLA and the Army had established and implemented  disciplined processes related to requirements management and testing, we  reviewed DLA’s and the Army’s procedures for defining requirements  management frameworks and compared these procedures to their  current practices;  reviewed guidance published by the Institute of Electrical and  Electronics Engineers and the Software Engineering Institute and  publications by experts to determine the attributes that should be used  for developing good requirements;  reviewed BSM’s system requirement documents related to finance,  order fulfillment, planning, and procurement and LMP’s system  requirement documents related to planning and budget development,  asset management, inventory management, and maintenance analysis  and planning; and  selected 13 of BSM’s 202 system requirements and 12 of LMP’s 293  system requirements and performed an in-depth review and analysis to  determine whether they had the attributes normally associated with  good requirements and whether these requirements traced between the  various process documents.
To augment these document reviews and analyses, we interviewed DLA  and Army program officials and Defense Finance and Accounting Service  (DFAS) officials. To identify the costs associated with BSM and LMP, we  reviewed data provided by DLA and Army program officials. We also  reviewed prior GAO, DOD Inspector General, and service auditors’ reports,  as well as DOD’s agencywide financial statements to obtain further  information on inventory costs.
We conducted our work at the Office of the Under Secretary of Defense  (Comptroller); the Office of the Under Secretary of Defense (Acquisition,  Technology and Logistics); the Office of the Assistant Secretary of Defense  (Network and Information Integration)/Chief Information Officer; DLA; the  Army Materiel Command; and the CIO and financial management offices  for the military services. We also visited two locations—the Defense Supply  Center in Richmond, Virginia, and the Army’s contractor site (Computer  Sciences Corporation) in Moorestown, New Jersey—to gain an  understanding of user involvement in the development and operation of  BSM and LMP, as well as the business processes associated with each  system.
We conducted our work from August 2003 through March 2004 in  accordance with U.S. generally accepted government auditing standards.   We did not verify the accuracy and completeness of the cost information  provided by DOD for the two projects we reviewed.
We requested comments on a draft of this report from the Secretary of  Defense or his designee.  We received written comments on a draft of this  report from the Acting Under Secretary of Defense (Comptroller), which  are reprinted in appendix II.
section_title: Comments from the Department of Defense
section_title: DOD Business Systems Obligations in Excess of $1 Million Approved by the DOD Comptroller
paragraphs: Air Force Financial Information Resource System  Navy Enterprise Resource Planning Pilots  National Security Agency Pilot Initiative Navy Enterprise Resource Planning Program Defense Integrated Military Human Resources System DFAS Mechanization of Contract Administration Services Rehost DFAS PowerTrack (SCR)
Army Integrated Facilities System (SCR)
Navy Enterprise Maintenance Automated Information System DFAS e-Biz Capital Investment Reprogramming DFAS Operational Data Store  (SCR)Composite Health Care System II DFAS General Accounting and Finance System Rehost Air Force Reserve Travel System DFAS Automated Time, Attendance and Production System (SCR)
DFAS Defense Joint Military Pay System—Active Component (SCR)
DFAS Defense Joint Military Pay System—Reserve Component (SCR)
DFAS Defense MilPay Office (SCR)
DFAS Defense Retired and Annuitant Pay System (SCR)
DFAS Marine Corps Total Force System (SCR)
section_title: DOD Business Systems Obligations in Excess of $1 Million for Modernizations Not Submitted to the DOD Comptroller
paragraphs: Transportation Coordinators’ Automated  Information for Movements System II Army Recruiting Information Support System Defense Civilian Personnel Data System- Sustainment MEPCOM Management Information  Reporting System Joint Computer-Aided Acquisition and  Logistics Support Navy Tactical Command Support System Marine Corps Common Hardware Suite Electronic Military Personnel Records  System Navy Standard Integrated Personnel System Conventional Ammunition Integrated  Management System Shipyard Management Information Systems- Financials SPAWAR Financial Management - ERP MSC Afloat Personnel Management Center Transportation Coordinators’ Automated  Information for Movements System II $4.4 (Continued From Previous Page)
Military Sealift Command Financial  Management System Asset Tracking Logistics and Supply System Integrated Logistics System – Supply Depot Maintenance Accounting and  Production System Supply Working Capital Fund Decision  Support System (Keystone)
Reliability and Maintainability Information  System For fiscal year 2004, DOD did not report obligational data.
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: Staff members who made key contributions to this report were Beatrice  Alff, Johnny Bowen, Francine DelVecchio, Stephen Donahue, Francis  Dymond, Jeffrey Jacobson, Jason Kelly, Mai Nguyen, Michael Peacock,  David Plocher, and Katherine Schirano.
section_title: GAO’s Mission
paragraphs: The General Accounting Office, the audit, evaluation and investigative arm of  Congress, exists to support Congress in meeting its constitutional responsibilities  and to help improve the performance and accountability of the federal government  for the American people. GAO examines the use of public funds; evaluates federal  programs and policies; and provides analyses, recommendations, and other  assistance to help Congress make informed oversight, policy, and funding  decisions. GAO’s commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to   e-mail alerts” under the “Order GAO Products” heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs