DOD Business Transformation: Lack of an Integrated Strategy Puts the Army's Asset Visibility System Investments at Risk
Published: 2007-07-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) established a goal to achieve total asset visibility (TAV) over 30 years ago, but to date it has been unsuccessful. GAO was requested to (1) determine whether the Army has a systems strategy for achieving TAV, (2) determine if the Army's business system investment governance structure is consistent with DOD guidance, and (3) evaluate the Army's effort to correct previously reported problems with the Logistics Modernization Program (LMP). GAO obtained an understanding of the Army's efforts to achieve TAV, oversee and manage its business system investments, and address previously reported LMP problems.
section_title: What GAO Found
paragraphs: Supply chain management has been on GAO's high-risk list since 1990. One area that has contributed to this long-standing problem has been DOD's inability to maintain control and accountability over hundreds of billions of dollars of assets. DOD plans to improve its asset management through its business system modernization. In this regard, GFEBS, the Global Combat Support System-Army (GCSS-Army), and LMP are aimed at achieving TAV within the Army. The Army estimates that it will invest approximately $5 billion to develop and implement these systems. However, this investment is being made without a clear integrated strategy. GFEBS, GCSS-Army, and LMP are not being developed in the context of a well-defined Army-wide enterprise architecture. As a result, the Army does not have an informed basis for determining if these systems will fit within the context of future Army business operations and will efficiently and effectively address the Army's long-standing weaknesses associated with the lack of asset visibility. The Army lacks a concept of operations that would describe, at a high level, (1) how the three business systems relate to each other in achieving the Army's TAV goal, and (2) how information flows from and through these systems. Moreover, GAO found that the Army's lack of a concept of operations has contributed to its failure to take full advantage of business process reengineering opportunities that are available when using an enterprise resource planning solution. Without these key foundational elements, the Army is at risk of investing about $5 billion in business systems and still not achieving DOD's and the Army's goal of TAV. Furthermore, while the Army has established a governance structure that is consistent with DOD guidance, its processes are still maturing. The Army's governance structure is designed to certify and review individual business systems rather than to evaluate these investments from a portfolio perspective. Such a perspective permits investments to be viewed in a comprehensive manner to help ensure that the organization's missions and objectives are achieved. GAO also found that the Army did not have reliable processes and analyses, such as an independent validation and verification function or economic analyses, to support its oversight of individual business systems. Until the Army's investment processes mature, it runs the risk of investing in business systems that do not provide the desired functionality and efficiency. Additionally, LMP continues to be plagued by problems that have beset the system since its implementation in July 2003. LMP continues to experience problems with accurately recognizing revenue and billing customers, which can, in part, be attributed to ineffective system testing.

section_title: Letter
paragraphs: The Department of Defense (DOD) has continually struggled to achieve  and maintain efficient and effective management over the hundreds of  billions of dollars it has invested in tangible assets, including inventory,  supplies, and materials (inventory and related property) and property,  plant, and equipment (PP&E). DOD was responsible for almost 72 percent  ($697 billion) of the total $970 billion reported governmentwide value for  these assets, as of September 30, 2006. The nature and severity of DOD’s  financial and business management system deficiencies impede the ability  of DOD managers to receive the full range of information needed to  effectively manage day-to-day operations. Of the 27 areas on GAO’s high-  risk list, DOD has 8 high-risk areas of its own and shares responsibility  for 7 governmentwide high-risk areas.
Visibility over its assets has been a DOD concern for decades. If the  information contained in the asset accountability systems is not accurate,  complete, and timely, DOD’s day-to-day operations could be adversely  affected by investing in inventory, for example, that is not needed to meet  current needs. When this occurs, the department may obligate funds  unnecessarily, which could lead to not having sufficient obligational  authority to purchase needed items. In recognition of the importance of  asset accountability to successful operations, the department established a  goal to achieve total asset visibility (TAV) over 30 years ago. DOD defines  TAV as the ability to provide timely and accurate information on the  location, movement, status, and identity of units, personnel, equipment,  and supplies and having the ability to act on that information. Over the  years, the military services and defense components have undertaken  numerous initiatives to achieve TAV. Within the Army, one such initiative  has been the Logistics Modernization Program (LMP). In May 2004 and  June 2005, we reported that LMP was not providing the Army the promised  capability. DOD’s current estimate for achieving TAV is 2010.
This report provides information in support of your continuing oversight  of DOD’s progress towards resolving the department’s long-standing  problems in achieving TAV. As you requested, our initial effort was  directed at the Army. In September 2006, the Army reported inventory and  related property of about $57 billion and PP&E over $85 billion. The Army  has identified three primary system initiatives directed at achieving TAV  within the service: (1) LMP, (2) Global Combat Support System-Army  Field/Tactical (GCSS-Army), and (3) General Fund Enterprise Business  System (GFEBS). Our objectives were to (1) determine whether the Army  has developed a business system strategy for achieving TAV, (2) determine  if the Army has effectively implemented a governance structure to oversee  and manage its business system investments in accordance with DOD  guidance, and (3) evaluate the extent to which the Army has made  progress in correcting the previously reported problems regarding LMP’s  implementation.
To address the first objective, we met with Army program office officials  for GFEBS, GCSS-Army, and LMP and obtained briefings on the intended  purpose of each system.  In addition, we conducted walkthroughs and  reviewed documentation related to various transactions to obtain an  understanding of how the systems would exchange data and to assess how  the Army intended to use these systems individually and collectively to  achieve TAV. To address the second objective, we reviewed guidance  issued by DOD, the Army, and the Business Transformation Agency (BTA)  related to investment management. We also obtained an understanding of  the Army’s business system investment governance structure and process  for ensuring compliance with the certification and annual system review  processes required by the fiscal year 2005 National Defense Authorization  Act. To address the third objective, we interviewed and obtained briefings  from LMP program management office officials and others, and reviewed  and analyzed LMP system requirement and testing documentation to  assess the extent to which corrective actions had been taken or are  planned to address our prior recommendations. We determined that the  documentation Army prepared and submitted through its business system  governance process to the investment review boards (IRBs) and the  Defense Business Systems Management Committee (DBSMC) as a basis  for approving individual Army business system investments was  sufficiently reliable for our purposes. Our work was performed from May  2006 through June 2007 in accordance with U. S. generally accepted  government auditing standards. Details on our scope and methodology are  included in appendix I. We requested comments on a draft of this report  from the Secretary of Defense or his designee. We received written  comments from the Deputy Under Secretary of Defense (Business  Transformation), which are reprinted in appendix II.
subsections: section_title: Background
paragraphs: TAV has been elusive within DOD. Timely, reliable information on the  location, quantity, and status of the department’s tangible assets could  significantly improve its ability to more efficiently and effectively deliver  needed items to DOD operating forces and thereby enhance military  readiness. The department has recognized the importance of achieving  TAV and included it as part of its overall business transformation initiative,  which includes the development and implementation of a business  enterprise architecture (BEA). The Joint Chiefs of Staff also identified TAV  as one of four fundamental changes needed to transform the department’s  logistics operations.
One of the primary factors contributing to DOD’s inability to provide  management with TAV is DOD’s outdated and ineffective management  information system environment. The Federal Financial Management  Improvement Act (FFMIA) of 1996 and other financial management  reform legislation have emphasized the importance of improving financial  management, which necessarily encompasses proper inventory  management, across the federal government. Built upon the foundation  laid by the Chief Financial Officers Act of 1990, FFMIA emphasizes the  need for agencies to have integrated financial management systems that  can generate timely, accurate, and useful information to make informed  decisions and to ensure accountability on a continuous basis. Lacking  such critical information, government leaders will not be positioned to  invest resources, reduce costs, oversee programs, and hold agency  managers accountable for the manner in which government programs are  operated.
subsections: section_title: Army Initiatives Aimed at Achieving TAV
paragraphs: To improve control and accountability over its assets, the Army has  embarked on a multisystem integration effort that is intended to leverage  commercial ERP software and processes. This integration is the focus of  the Single Army Logistics Enterprise (SALE) initiative. SALE is designed to  integrate information technology requirements, business processes,  business rules, and data from the Army’s logistics, financial, and  acquisition transactions for planning and supporting warfighting logistics  operations. According to the Army, SALE is to provide a coordinated ERP  solution built around two individual logistics system development efforts:  (1) LMP and (2) GCSS-Army. Under the SALE vision, GCSS-Army and LMP  will be integrated into a single solution to provide an Army-wide logistics  environment spanning from “the factory to foxhole.” On the financial side,  the Army is developing GFEBS to provide financial visibility over its  assets. Each of these efforts is described below.
LMP. In February 1998, the U.S. Army Materiel Command began an ERP  effort—LMP—to replace its legacy materiel and maintenance management  systems—the Commodity Command Standard System and the Standard  Depot System—with LMP. The Army has been using the existing legacy  systems for over 30 years. LMP is intended to transform the Army’s  Working Capital Fund logistics operations in six core processes: order  fulfillment, demand and supply planning, procurement, asset management,  materiel maintenance, and financial management. LMP became  operational at the U.S. Army Communications and Electronics Command,  Fort Monmouth, New Jersey, and Tobyhanna Army Depot, Tobyhanna,  Pennsylvania, in July 2003. The initial deployment of LMP consisted of  inventory items such as electronics; electronic repair components; and  communications and intelligence equipment such as night vision goggles,  electronic components such as circuit boards, and certain munitions such  as guidance systems included in missiles. Figure 1 shows the LMP time  line as of March 2007.
The Communications-Electronics Life Cycle Management Command,  Tobyhanna Army Depot, and the Defense Finance and Accounting Service  (DFAS) are the primary LMP users. When LMP is fully implemented, its  capacity is expected to include more than 17,000 users at 149 locations  and it will be populated with 6 million Army-managed inventory items  valued at about $40 billion. LMP is scheduled to reach full operational  capability (FOC) in fiscal year 2010. As of September 30, 2006, the Army  reported that approximately $452 million had been obligated for this  system effort and estimates that it will invest at least another $895 million  in LMP.
GCSS-Army. The GCSS-Army program was initiated in 1997 to overcome  duplicative databases, poor asset visibility, and stovepiped  communications between numerous existing Army logistics systems. The  goal of GCSS-Army is to integrate multiple logistic functions by replacing  numerous legacy systems and interfaces. Since the program’s inception, it  has undergone several revisions, including a change from a custom  software development to a commercial ERP solution, as well as a change  in the prime contractor. According to Army officials, it invested  approximately $95 million in the previous efforts before adopting the ERP  approach. The existing ERP effort was started in September 2003 and is  currently estimated to reach FOC during fiscal year 2014. Figure 2 shows  the time line for GCSS-Army as of March 2007.
GCSS-Army is intended to replace 16 stovepiped, legacy logistics systems  that cover certain types of inventory and PP&E. The system is intended to  be operational at all deployable Army units, and provide asset visibility for  accountable items down to the foxhole. As of September 30, 2006, the  Army reported that it had obligated approximately $203 million for the  ERP version of GCSS-Army. Additionally, the Army estimates that another  $2.1 billion will be invested in GCSS-Army.
Product Lifecycle Management Plus (PLM+). This initiative is the  technical enabler to integrate LMP and GCSS-Army. PLM+ is the means by  which the Army intends to achieve the SALE vision of integrated logistics  systems. PLM+ is intended to provide a single point of entry and exit for  interfaces to external systems. Furthermore, PLM+ is intended to  eliminate duplicative and costly system interfaces. Additionally, the PLM+  implementation schedule and related funding are aligned with that of  GCSS-Army. PLM+ does not stand alone as an independent capability. As  of September 2006, the Army reported that approximately $31 million had  been obligated for PLM+.
GFEBS. While LMP is intended to provide financial control for the Army  Working Capital Fund, GFEBS is intended to provide this important  control over all non-working capital fund inventory, including that which  is reported in GCSS-Army. GFEBS is an ERP solution that was initiated in  October 2004 and is intended to serve as the Army’s general ledger system  for its general fund accounting. As such, GFEBS is intended to improve  the reliability of the Army’s financial information and thereby enhance the  Army’s management decision-making process. GFEBS is expected to  replace 87 legacy systems, including the 30-year old Standard Army  Finance System. Figure 3 shows the GFEBS time line as of June 2007.
The Army estimates that GFEBS will reach FOC by 2010. As of    September 30, 2006, the Army reported that it had obligated $123 million  for the development of GFEBS. In addition, the Army currently estimates  that it will invest another $1.3 billion to implement GFEBS.
section_title: Overview of DOD’s Investment Management Practices
paragraphs: In 2005, DOD adopted a “tiered accountability” approach to improve  control and accountability over the billions of dollars it invests annually in  DOD business systems. Under this approach, executive leadership for the  direction, oversight, and execution of DOD investments is the  responsibility of several entities within DOD and its components. As  shown in figure 4 and described below, the investment control process  begins at the component level and works its way up through a hierarchy of  review and approval authorities, depending on the size and significance of  the investment.
At the DOD enterprise level, key entities involved in maintaining control  and accountability over Army business system investments with systems  modernizations over $1 million include the DBSMC, which serves as the  highest ranking governance body for business systems modernization  activities; the Principal Staff Assistants, who serve as the certification  authorities for business system modernizations in their respective core  business missions; the investment review boards (IRBs), which form the  review and decision-making bodies for business system investments in  their respective areas of responsibility and review each investment for  BEA compliance; and BTA, which provides support to the DBSMC and the  IRBs and is responsible for leading and coordinating business  transformation efforts across the department. The BTA is organized into  seven directorates, one of which is the Defense Business Systems  Acquisition Executive—the component acquisition executive for DOD  enterprise-level (DOD-wide) business systems and initiatives. This  directorate is responsible for developing, coordinating, and integrating  DOD enterprise-level projects, programs, systems, and initiatives— including managing resources such as funding, personnel, and contracts  for assigned systems and programs.
To implement tiered accountability within the Army, the Army designated  its Chief Information Officer (CIO) as the Army’s Pre-Certification  Authority (PCA) for certification and annual reviews of business system  investments. The PCA is accountable for the component’s business  system investments and acts as the component’s principal point of contact  for communication with the IRBs. As such, the PCA (1) validates that the  system information for all business systems modernizations over               $1 million is complete and accessible to the IRBs, (2) reviews  development/modernization investments with total cost of $1 million or  less, (3) reviews system compliance with DOD’s BEA and enterprise  transition plan, (4) verifies the investment’s economic viability analysis,  (5) asserts the status and validity of the investment information by  submitting a component precertification letter to the appropriate IRB for  its review, and (6) provides IT portfolio management policy guidance and  oversight of mission area/domain IT portfolios. Below the Army enterprise  level, the Army has established six functional area domains within its  business mission area. The domains are responsible for implementing the  IT portfolio management process developed by DOD and the Army to  define and justify the portfolio of planned IT expenditures consistent with  strategic objectives and operational requirements.
section_title: GAO Has Previously Reported LMP Issues
paragraphs: Our May 2004 report pointed out that the Army had not effectively  managed its implementation of LMP. The report noted that after LMP was  deployed in July 2003, operational difficulties at the Tobyhanna Army  Depot resulted in inaccurate financial management information. More  specifically, the depot was not (1) producing accurate workload planning  information, (2) generating accurate customer bills, and (3) capturing all  repair costs, necessary for the Army to calculate accurate future repair  prices. The report also pointed out that LMP requirements (1) lacked the  specific information necessary to understand the required functionality  that was to be provided, and (2) did not describe how to determine  quantitatively, through testing or other analysis, whether the system would  meet the Army’s needs. Subsequently, in June 2005, we reported that the  problems with LMP continued to prevent the Tobyhanna Army Depot from  accurately reporting on its financial operations, including gains and losses,  which adversely affected the depot’s ability to accurately set customer  sales prices. Further, the report pointed out that problems persisted with  recognizing revenue and billing customers. We recommended, and the  Army agreed, that the implementation of LMP should be delayed until the  operational problems we identified were resolved. The Army is continuing  to resolve the outstanding issues.
section_title: Army Lacks an Integrated Strategy for Achieving TAV
paragraphs: The Army has estimated that it will invest about $5 billion over the next  several years to complete development and implementation of GFEBS,  GCSS-Army, and LMP. However, the Army is making this significant  investment without a clear integrated strategy for how these systems will  be used to achieve TAV over hundreds of billions of dollars of assets. As  we reported in December 2004, because DOD has not developed a clear  long-range strategy, the military services will be exposed to the risk of  spending billions of dollars on duplicative, stovepiped systems that do not  support the department’s business transformation goals, including  attaining TAV. More specifically, we found that the Army’s current strategy  for achieving TAV, including the implementation of these three systems,  does not embrace two key foundational elements that we had identified as  essential to achieving successful transformation of business systems and  processes: an EA and a concept of operations. Without these key  foundational elements, the Army is at risk of investing billions of dollars in  business systems that may not achieve DOD’s and the Army’s goal of  achieving TAV. Further, the Army’s planned strategy perpetuates some of  the cumbersome and ineffective business processes and data  redundancies that are currently being used in the existing legacy system  environment.
subsections: section_title: Army’s Efforts to Achieve TAV Lack the Benefit of an EA
paragraphs: The Army has yet to develop and implement an Army business EA to help  guide its system efforts, including those aimed at achieving TAV. DOD’s  acquisition policies and guidance, as well as federal and best practice  guidance, recognize the importance of investing in business systems  within the context of an EA. GFEBS, GCSS-Army, and LMP are not being  managed and developed in the context of a well-defined Army-wide EA. In  August 2006, we reported that the Army was in the very early stages of  developing an EA. We recommended that the Secretary of Defense ensure  that the DOD architecture programs we reviewed develop and implement  plans for fully satisfying each of the conditions in our Enterprise  Architecture Management Maturity Framework (EAMMF) for assessing  and improving EA management. In commenting on the report, DOD  agreed with our recommendation.
As detailed in our August 2006 report, we assessed the Army’s efforts to  develop an EA against the criteria specified in EAMMF.  Our EAMMF is a  five-stage architecture framework for managing the development,  maintenance, and implementation of an architecture and understanding  the extent to which effective architecture management practices are being  performed and where an organization is in its progression toward having a  well-managed architecture program. In short, the framework consists of 31  core elements that relate to architecture governance, content, use, and  measurement. These elements reflect research by us and others showing  that architecture programs should be founded upon institutional  architecture commitment and capabilities, and measured and verified  products and results. Our analysis of information provided by the Army  indicated that it had satisfied only 3 percent of all framework elements. In  essence, this means that the Army is at stage 1 of developing and  implementing an EA. While stage 1 agencies may have initiated some  enterprise architecture activity, these agencies’ efforts are ad hoc and  unstructured, lack institutional leadership and direction, and do not  provide the management foundation necessary for successful enterprise  architecture development.
In May 2007, we reported that the Army still had not developed an EA. As  a result, the Army does not have a well-informed basis for determining if  these systems will fit within the context of future Army business  operations and will most efficiently and effectively address the Army’s  long-standing weaknesses associated with the lack of asset visibility.  Improving asset visibility is critical to addressing the problems associated  with supply chain management, which has been on our high-risk list since  1990. Asset visibility is one of the focus areas that are a critical part of  supply chain management.
In addition, without a clear understanding of its current systems and  business processes—commonly referred to as the “As Is” environment— and the business operations it envisions for the future—commonly  referred to as the “To Be” environment—the Army will increase the risk  that misalignments can occur that can introduce redundancies and  incompatibilities that can produce inefficiencies and require costly and  time-consuming rework to fix. GFEBS, GCSS-Army, and LMP have  proceeded without a common, institutional frame of reference (for  example, EA) that can be used to effectively manage their relationships  and dependencies. In November 2006, DOD’s internal review of GFEBS by  BTA noted that the Army lacked a strategy for integrating GFEBS, GCSS- Army, and LMP.
section_title: Army Strategy to Achieve TAV Does Not Utilize an Enterprisewide Perspective
paragraphs: The Army also lacks a concept of operations for how GFEBS, GCSS-Army,  and LMP will collectively achieve TAV. An effective concept of operations  would describe, at a high level, (1) how the three business systems relate  to each other in achieving the Army’s TAV goal, and (2) how information  flows from and through these systems. Moreover, we found that the  Army’s lack of a concept of operations has contributed to its failure to  take full advantage of business process reengineering opportunities that  are available when using an ERP solution. Without a clear long-term  strategy, the Army lacks a key management control for ensuring that time  frames, results-oriented performance measures, and accountability  mechanisms are established and monitored to help achieve TAV.
Specifically, we noted that the Army’s strategy perpetuates some of the  cumbersome and inefficient business processes that are currently being  used in the existing legacy system environment. One of the key benefits of  an ERP is that it can be used to streamline business processes and  eliminate data redundancy. However, some of the basic business  processes being used or expected to be used by GFEBS, GCSS-Army, and  LMP are based on the Army’s existing business processes—many of which  are error prone, labor intensive, and redundant. Considering the billions of  dollars the Army is investing in these systems, it is incumbent upon the  Army to embrace and utilize the most efficient and streamlined ERP  processes to the fullest extent possible in its effort to achieve TAV.
For example, the Army’s planned strategy to improve control and  accountability over its PP&E does not take advantage of the capabilities of  an ERP solution. (See app. III for additional examples in which the Army’s  TAV systems initiatives have not effectively streamlined legacy processes  related to funds control and disbursements.) Financial data, including  acquisition cost and depreciation, for most of the Army’s PP&E are  currently maintained in the Defense Property Accountability System  (DPAS). PP&E accountability data, such as location and quantity, are  maintained in a variety of legacy systems such as the Property Book Unit  Supply Enhanced (PBUSE) system—which is 1 of the 16 systems that is  expected to be replaced by GCSS-Army. In March 2007, Army Program  Executive Office Enterprise Information Systems officials informed us that  the financial data—such as acquisition cost and depreciation—for the  Army’s PP&E will be transferred from DPAS to GFEBS. This maintains the  existing accounting data in an accounting system. On the other hand, the  accountability data—such as location, condition, and number of units—for  PP&E assets will initially be transferred from the current legacy system to  PBUSE or GCSS-Army, depending on whether the items belong to an  installation or an Army unit that may be deployed. The transferring of  items to PBUSE appears to be an interim solution since, according to  DOD’s enterprise transition plan GCSS-Army is to replace PBUSE.  However, GCSS-Army is not expected to assume the control and  accountability over items that are recorded in PBUSE associated with the  Army Working Capital Fund or other Army organizational entities that are  not considered to be units that can be deployed (field/tactical units) in  military operations.
The Army’s decision to segregate its PP&E financial and accountability  data into two separate system solutions reflects its lack of an enterprise  concept of operations and enterprise view for achieving TAV. The Army’s  strategy of moving PP&E data from DPAS to PBUSE, and ultimately to  another system, means that one data conversion will take place to move  the data from DPAS to PBUSE—a system the Army is supposed to  eliminate with the implementation of GCSS-Army—followed by another  data conversion from PBUSE to yet another system, not yet identified.  Further, the Army is continuing to limit the capabilities of LMP by  transferring all information on PP&E from DPAS to PBUSE—which as  noted before is considered a legacy business system. As a result,  information on PP&E that could enhance Tobyhanna’s operations,  especially in the area of workload planning, will not be available within the  LMP environment.
Thus, the approach being taken by the Army is simply to adapt the  processes used by the legacy systems rather than use the streamlined  processes inherent in the ERP solution. If the functionality of an ERP  solution was fully utilized by the Army, it would increase the likelihood  that TAV could be accomplished within each of the three systems for the  specific type of PP&E items each was responsible for. For example,  GFEBS would contain both the financial and accountability information  related to nontactical/non-working capital fund PP&E, GCSS-Army would  have all the information associated with tactical PP&E, and LMP would  have the data related to Working Capital Fund PP&E. Figure 5 illustrates  one way that an ERP approach could be utilized to achieve physical and  financial accountability control of the Army’s PP&E.
While the Army’s approach may result in incremental improvements in its  asset accountability, it does not take full advantage of the (1) benefits of  adopting enterprise processes and (2) functionality that is available in the  ERP solution it has selected to support those processes. Fundamental to a  successful ERP implementation is the reengineering of an organization’s  business processes in a manner that helps ensure that the right resources  (people, material, machinery, and funds) are available in the correct  quantities when needed. While Army has stated that its use of ERP  software would help reengineer its business processes, we found that at  least some business processes that are being used or expected to be used  in the future do not reflect reengineered processes necessary to most  effectively implement an ERP solution. By perpetuating the use of  cumbersome, error-prone, and ineffective business processes in its asset  accountability operations, the Army will diminish its capability to achieve  TAV and improve accountability over its assets.
section_title: Army’s Ability to Effectively Oversee Portfolios of Business Systems Investment Is Not Yet Fully Developed
paragraphs: While the Army has established a governance structure to oversee its  business system investments—including its asset accountability system  investments—that is consistent with DOD guidance, additional actions are  needed to enhance oversight, control, and accountability. The Army’s  business system investment oversight efforts to date have primarily  focused on ensuring that business systems modernizations over $1 million  are reviewed in accordance with the criteria specified in the fiscal year  2005 National Defense Authorization Act. Both DOD and Army officials  acknowledged that the department’s and Army’s investment review  processes, particularly related to their ability to review business system  investments from a portfolio perspective, are in the early stage of maturity.  Portfolio management is a conscious, continuous, and proactive approach  to allocating limited resources among competing initiatives in light of the  investments’ relative benefits. Taking an enterprise view enables an  organization to consider its investment comprehensively, so that  collectively the investments optimally address the organization’s mission,  goals, and objectives. In addition, we found that the Army did not have  reliable processes and analyses to support its oversight of individual  systems modernization program efforts intended to improve asset  visibility. Specifically, (1) the Army has not established or implemented an  independent verification and validation (IV&V) process for the three  systems—GFEBS, GCSS-Army, or LMP and (2) the August 2004 GFEBS  economic analysis was not prepared in accordance with DOD and OMB  guidance.
subsections: section_title: Army’s Oversight of Its Business System Investments Lacks a Portfolio Perspective
paragraphs: The Army’s oversight of its business system investments—including  system investments intended to achieve TAV—continues to be focused on  the review and approval of individual business system investments. In May  2007, we reported that DOD needed to improve its policies and procedures  associated with managing its business system investments as portfolios.  More specifically, we found that DOD had not yet progressed from project- based processes to portfolio-based processes, a key element for effectively  managing business system transformation efforts. DOD informed us that  DOD components are responsible for developing and managing their own  portfolio management processes.
Army IT guidance for managing IT investments requires portfolios to be  managed and monitored using established quantifiable outcome-based  performance measures to determine whether to recommend  continuations, modification, or termination of individual investments  within a portfolio. However, the Army has not implemented processes or  procedures that facilitate an enterprise view toward management and  oversight of portfolios of business system investments intended to  collectively provide a specific capability or functionality. Rather, the  Army’s current business system investment process focuses primarily on  reviewing and approving investment packages submitted by the  responsible program management office for individual systems  improvement efforts that have development/modernization funding over  $1 million. Individual business system investments that exceed the            $1 million systems modernization threshold are required to be reviewed by  the appropriate IRB and approved by the DBSMC.
Army officials acknowledged that the Army’s ability to provide portfolio- based investment review and oversight is in the early stages of  development. For example, while the Army has begun a process to share  investment package information, including cost, schedule, and  performance data, across its mission areas and domains, Army officials  acknowledged that the information provided by a program management  office, as well as the associated oversight and review, is centered largely  on the functionality and capabilities of individual systems. For example,  annual certifications for the three systems, LMP, GCSS-Army, and GFEBS,  which the Army intends to utilize collectively in achieving TAV, are not  submitted or reviewed as a portfolio, but rather as individual system  investments. The Army has not yet implemented the processes to evaluate  and improve its progress toward achieving TAV using portfolio-based  projects such as costs, schedule, performance, and risks. As a result, the  Army’s ability to effectively oversee the development of the portfolio of  systems intended to collectively provide the Army with TAV is limited.  Until the Army’s governance process, policies, and procedures mature,  including its ability to apply a corporate portfolio perspective in managing  and overseeing subportfolios of business system investments intended to  provide a desired capability or function, the Army is at risk of  implementing systems that (1) do not provide the desired capability,  including TAV; (2) are stovepiped in their functionality; (3) do not  interoperate in an efficient manner; and (4) are not supported by efficient  and effective business processes.
section_title: Reliable Processes and Analyses Are Needed to Facilitate Management Oversight of Business System Investments
paragraphs: The Army has not implemented reliable processes and analyses needed to  enhance its management oversight of program management office efforts  to develop and implement its TAV, as well as other business systems on  time, within budget, and with the intended capability. Both DOD’s and the  Army’s business system investment oversight processes are highly  dependent upon the reliability of investment information provided by  Army program management offices. Thus, the reliability of that business  system investment information is critical to both DOD’s and the Army’s  ability to effectively oversee, manage, and redirect—if necessary—the  services’ business system development and modernization efforts.  However, we found that the Army had not yet established an independent  verification and validation function for any of the three systems we  reviewed. In addition, the economic analysis submitted by the program  management office for GFEBS was not prepared in accordance with DOD  or OMB guidance. As a result, the economic analysis submitted by the  program management office to DOD and the Army oversight entities  justifying the Army’s investment in GFEBS was based on questionable  costs and benefits.
subsections: section_title: Independent Verification and Validation Would Enhance Program Oversight
paragraphs: The Army had not yet established an IV&V function for any of the three  systems—GFEBS, GCSS-Army, and LMP. While the Army has established a  verification and validation function for LMP, it was not an independent  review because the reviewer reports directly to the LMP program  management office. Independence is a key component to reliable  verification and validation function.
Best business practices have demonstrated that use of an IV&V function is  an invaluable means to providing management reasonable assurance that a  planned system, or the portfolio of systems, will satisfy its planned use and  users. An effective IV&V review process should provide an independent  assessment to DOD and Army management of the overall status of the  project, including a discussion of any existing or potential revisions to the  project with respect to cost, schedule, and performance. The IV&V reports  should identify to management the issues or weaknesses that increase the  risks associated with the project or portfolio to senior management so that  they can be promptly addressed. These assessments involve reviewing  project documentation, participating in meetings at all levels within the  project, and providing periodic reports and recommendations, if deemed  warranted, to senior management. The IV&V function should report on  every facet of a system project such as:  Testing program adequacy. Testing activities would be evaluated to  ensure they are properly defined and developed in accordance with  industry standards and best practices.
Critical-path analysis. A critical path defines the series of tasks that  must be finished in time for the entire project (or portfolio of projects) to  finish on schedule. Each task on the critical path is a critical task. A  critical-path analysis helps to identify the impact of various project events,  such as delays in project deliverables, and ensures that the impact of such  delays is clearly understood by all parties involved with the project(s).
System strategy documents. Numerous system strategy documents that  provide the foundation for the system development and operations are  critical aspects of an effective system project. These documents are used  for guidance in developing documents for articulating the plans and  procedures used to implement a system. Examples of such documents  include the Life-cycle Test Strategy, Interface Strategy, and Conversion  Strategy.
section_title: Validity of GFEBS’s Economic Analysis Questionable
paragraphs: Our analysis of the August 2004 GFEBS economic analysis found that it  was not prepared in accordance with OMB and DOD guidance. The  purpose of an economic analysis is to give the decision maker insight into  economic factors that have a bearing on accomplishing the stated  objective of the system investment, for example implementation of GFEBS  to improve the Army’s financial accountability. As such, it is important  that the economic analysis reliably identifies factors, such as cost and  performance risks and drivers that can be used to establish priorities and  allocate resource allocations. While it may be appropriate to invest in a  particular IT investment, such as GFEBS, for reasons other than estimated  economic benefits, nonetheless, the issues we identified raise questions as  whether the funds invested in GFEBS were economically justified. Our  specific concerns with the August 2004 GFEBS economic analysis are  highlighted below.
Inappropriate cost savings. At least $142 million of estimated savings  were not savings, but rather, should have been classified as transfers— which do not equate to economic benefits. Transfers represent shifts of  control over resource allocation from one group to another that do not  result in a net change in the value of the resources involved in the transfer.  For example, we found that the Army claimed over $88 million of savings  related to transferring real property management from the Army’s legacy  system to GFEBS. Since the Army’s real property management  responsibilities were not eliminated, only transferred from the legacy  system environment to GFEBS, this represents a transfer rather than an  economic benefit for the Army. Additionally, the Army claimed benefits of  about $54 million related to GFEBS reducing the length of time funds are  borrowed by the Treasury to meet the Army financial obligations by  facilitating the Army’s ability to make just-in-time disbursements. OMB  guidance provides that there are no economic gains from a pure transfer  payment because the benefits to those government entities that receive  such a transfer are matched by the costs borne by those government  entities that provide the transfer. The Army’s inappropriate classification  of transfers resulted in an overstatement of the economic benefits that  would be achieved by implementing GFEBS. Additionally, in November  2006, BTA’s enterprise risk assessment management review of GFEBS  questioned justifying the GFEBS program on factors not fully within the  program’s control.
Lack of sensitivity analysis. The analysis did not include an assessment  of the effects of the uncertainty inherent in estimates of GFEBS benefits  and costs, as stipulated in OMB and DOD guidance. Because an economic  analysis uses estimates and assumptions, it is critical that a sensitivity  analysis be performed to understand the effects of the imprecision in both  underlying data and modeling assumptions. This analysis is required since  the estimates of future benefits and costs are subject to varying degrees of  uncertainty. Sensitivity analysis refers to changing the value of a given  variable in a model to gauge the effect of change on model results. It varies  a single data element or assumption while holding the other data elements  and assumptions constant to determine what amount of change in that  element is required to raise or lower the resulting benefit and cost  elements. In this way, GFEBS data and assumptions can be risk-ranked for  decisionmaking and auditing. The Army’s failure to conduct a sensitivity  analysis of GFEBS, as part of its economic analysis, to identify the effect  of uncertainties associated with different assumptions increases the  chance that decisions regarding GFEBS will be made without a clear  understanding of the possible impact on GFEBS estimates of costs and  benefits. Army officials informed us that they will consider including a  sensitivity analysis of the effect of uncertainties when they update  GFEBS’s economic analysis.
Lack of documentation supporting benefits and cost assumptions.
The GFEBS program management office could not provide us with  benefits and costs data needed to replicate their analyses. As a result, we  were unable to trace the cost and benefit estimates reported in the GFEBS  economic analysis to original source documentation to assess and validate  the reliability and applicability of the data. OMB and DOD guidance stress  the importance of maintaining documentation supporting assumptions  used in a cost-benefit analysis.
section_title: LMP Illustrates Continuing Problems in Implementing Business Systems on Time, within Budget, and with the Promised Capability
paragraphs: LMP continues to be plagued by operational problems that have beset the  system virtually since its initial implementation in July 2003. While, an EA,  concept of operations, and effective IT portfolio management are essential  elements in the Army’s efforts to transform its operations, it is equally  important for the Army to have disciplined processes in place to  implement its business systems on time, within budget, and with the  promised capability. Many of DOD’s approximately 2,980 business systems  are nonintegrated, stovepiped, and not capable of providing departmental  management and the Congress accurate and reliable information on DOD’s  day-to-day operations. LMP is no exception. As previously noted, LMP was  to have reached FOC in fiscal year 2005, but currently FOC is estimated for  fiscal year 2010—a slippage of 5 years. Further, LMP’s estimated program  costs have also increased. Tobyhanna Army Depot—the only depot that is  operating LMP—and DFAS personnel have stated that improvements have  been made in LMP’s operating efficiency. However, problems in the areas  of revenue recognition and billing continue. These problems can, in part,  be attributed to ineffective system testing. The operational issues  confronting LMP and our concerns about the effectiveness of the system  testing are highlighted below and discussed in more detail in appendix IV.
subsections: section_title: Operational Issues Continue
paragraphs: Tobyhanna Army Depot continues to experience problems with LMP  accurately recognizing revenue and billing customers. These problems  have existed virtually since the implementation of LMP in July 2003. While  the Army and its contractor—Computer Sciences Corporation (CSC)— have made numerous attempts to fix the problem, they have not been  successful. Additionally, the accuracy of LMP financial reports continues  to be questionable. For instance, the DOD Inspector General’s audit of the  Army’s Working Capital Fund financial statements for fiscal year 2006  noted that LMP was not properly recording transactions in accordance  with the U.S. Government Standard General Ledger requirements.
For the 3-month period ending January 31, 2007, we found that LMP’s  continuing billing problems resulted in (1) customers not being billed for  costs incurred that should have been billed and (2) customers being billed  too much. Based upon the billing information provided by the LMP  program management office, there were 146 customer orders valued at  approximately $5.4 million that were not billed (or recognized as revenue)  during the January 31, 2007, billing cycle. Customer billings and the  associated revenue provide the means by which the depot finances its day- to-day operations. Similarly, during the January 31, 2007, billing cycle  about 308 customer orders, amounting to about $5.8 million, were shown  as being overbilled and a credit should have been provided to the  customer. Moreover, the continuing billing and revenue recognition  problem may be a factor contributing to higher depot bills from DFAS.  According to DFAS personnel, the recurring billing problems have resulted  in DFAS personnel processing Tobyhanna Army Depot’s bills manually.  Based on information provided by DFAS, the number of hours spent to  provide DFAS accounting services—including billings for the Tobyhanna  Army Depot—has increased from approximately 17,800 hours in fiscal  year 2004 to over 22,600 hours in fiscal year 2006.
section_title: System Testing Remains a Concern
paragraphs: Our prior audits of LMP identified significant weaknesses with the LMP  program management office’s efforts to effectively implement the  requirements management and testing processes needed to reduce risks to  acceptable levels. During our current audit, we found that the  requirements management processes have improved—a critical first  step—but concern remains with respect to the adequacy of the testing.
Our analysis of 10 selected LMP corrective actions identified specific  testing weaknesses in each of the corrective actions reviewed. For  example, none of the test scripts provided adequate information that  linked the items tested to the specific requirement being tested. This  linking is commonly referred to as traceability and is characteristic of a  disciplined testing process. The test scripts reviewed contained headings  for information on the scenario, key data requirements, expected results,  actual results, and whether the test was considered successful. However,  the actual test script did not contain the level of specificity that clearly  delineated how a specific requirement identified in the requirement  document(s) and the associated requirement(s) in that document were  being tested. As a result, it was not possible to determine if all LMP system  requirements were properly tested. Without linking a given requirement to  the tests designed to exercise that requirement, it was impossible to obtain  reasonable assurance that (1) all the requirements were tested and (2) the  applied test provided adequate coverage for each requirement.
After discussing our concerns with the LMP program management office  and its contractor in February 2007, the LMP program office requested that  its verification and validation (V&V) contractor perform an assessment of  the LMP testing process. According to information provided by the LMP  program office and the V&V contractor, in March 2007, the review found  that the LMP testing process was adequately planned but that it did not  support a clear understanding of the (1) requirement being tested and      (2) tests used to determine whether a requirement was adequately  implemented. The V&V contractor’s assessment substantiates our analysis  of the LMP testing process. According to the V&V contractor, these areas  need to be addressed before the Army Test and Evaluation Command  conducts its review of LMP.
According to LMP program management office officials, the CSC is  developing and implementing a testing process that is designed to address  the concerns raised by us and the Army V&V contractor. CSC is expected  to have a testing program that clearly links the requirements to the tests  being conducted and to establish an independent test group to ensure that  the testing process is following best practices. However, until the Army  has reasonable assurance that an effective testing process has been  properly implemented, we are reiterating our June 2005 recommendation  that LMP not be deployed to additional locations until the Army has  assurance that LMP is providing the intended functionality and, more  specifically, that LMP can accurately bill its customers and recognize  revenue.
section_title: Conclusions
paragraphs: The Army’s efforts to develop and implement GFEBS, GCSS-Army, and  LMP may result in incremental improvements in the Army’s accountability  and visibility over its billions of dollars in tangible assets. However, the  Army has not developed or utilized key management tools and concepts  necessary to successfully transform its business processes to achieve TAV.  The primary question that the Army has yet to effectively consider and  address is how these systems and associated processes, individually and  collectively, will provide the desired functionality necessary to achieve  TAV. Until the Army develops and implements an Army EA, concept of  operations, and portfolio-based management and oversight processes, it  continues to risk (1) investing billions of dollars in asset accountability  systems that may not enhance the department’s and the Army’s goal of  achieving TAV, and (2) further delay DOD’s efforts to remove supply chain  management from our high-risk list.
It is also important that the Army take action to implement the disciplined  processes necessary to implement asset accountability systems on time,  within budget, and with the promised capability. Continuing problems  with LMP are illustrative of the consequences in failing to address these  issues. Although LMP became operational 4 years ago, it continues to  encounter operational problems. While there have been some  improvements in LMP, key functionality, such as accurately billing  customers and recognizing revenue, remains problematic. The Army’s  inability to resolve these persistent LMP problems can, in part, be  attributed to inadequate system testing. Until LMP implements an effective  testing process, it will have little assurance that the development and  corrective actions it takes (1) are properly developed, and (2) do not  introduce additional defects into the system.
section_title: Recommendations for Executive Action
paragraphs: To improve the department’s efforts to achieve TAV and further enhance  its efforts to improve the control and accountability over business system  investments, we recommend that the Secretary of Defense direct the  Secretary of the Army and the Director, BTA, to jointly take the following  five actions:    Develop a concept of operations that (1) clearly defines the ERP vision for  accomplishing total asset visibility within the Army; (2) addresses how its  business systems and processes, individually and collectively, will provide  the desired functionality to achieve TAV; and (3) determines the desired  functionality among the selected systems.
Develop policies, procedures, and processes to support the oversight and  management of selected groupings of business systems that are intended  to provide a specific capability or functionality, such as TAV from a  portfolio perspective, utilizing indicators such as costs, schedule,  performance, and risks.
Establish an IV&V function for GFEBS, GCSS-Army, and LMP.
Additionally, direct that all IV&V reports for each system be provided to  Army management, the appropriate IRB, and BTA.
Require that any future GFEBS economic analysis identify costs and  benefits in accordance with the criteria specified by DOD and OMB  guidance, to include a sensitivity analysis.
Direct that LMP utilize systems testers that are independent of the LMP  system developers to help ensure that the system is providing the users of  the system the intended capabilities.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the Deputy  Under Secretary of Defense (Business Transformation), which are  reprinted in appendix II. Overall, DOD concurred with our  recommendations and stated that it would work diligently to implement  them. The comments included two sets of specific responses to our  recommendations—one set provided by BTA and another provided by the  Army. In regard to the response provided by the Army, it concurred with  each of the recommendations. BTA stated that it fully agreed with our  observations. BTA, though, partially concurred with all the  recommendations on the basis that they were directed jointly to the  Secretary of the Army and the Director, BTA. BTA stated that under U.S.  Code Title 10 and in accordance with DOD’s tiered accountability, the  Army has complete authority for execution of its programs and  responsibility for implementing the recommendations. BTA’s comments  noted that it has neither the authority nor the responsibility to direct the  actions of the Army.
We appreciate the department’s willingness to address our  recommendations. With regard to BTA’s concern, our recommendations  do not direct BTA to oversee or direct the Army. Rather, the  recommendations stated that the specific actions should be undertaken  jointly at the direction of the Secretary of Defense. We continue to believe  that a cooperative and effectively coordinated BTA and Army approach to  addressing our recommendations is most likely to achieve the  fundamental business system transformation necessary to achieve the  department’s TAV objective. Further, the involvement of the Office of the  Secretary of Defense, including BTA, in the Army’s investment  management practices is consistent with 10 U.S.C. §§ 186 and 2222, which  provide for DOD policymaking and oversight of Army business system  functions.
In regard to the responses provided by the Army, it acknowledged the  importance of a concept of operations in achieving its goal of TAV and in  optimizing its business system investments. Further, the Army  acknowledged that there are risks involved in attempting to effectively  integrate the various system efforts. Going forward, as we recommended,  it will be important that the Army develop a clear, long-term strategy to  help ensure that time frames, results-oriented performance measures, and  accountability mechanisms are established and monitored to achieve TAV.
We are sending copies of this report to the Secretary of Defense; Acting  Secretary of the Army; Under Secretary of Defense (Acquisition,  Technology, and Logistics); Under Secretary of Defense (Comptroller);  Deputy Under Secretary of Defense (Business Transformation); Deputy  Under Secretary of Defense (Financial Management); Assistant Secretary  of Defense (Networks and Information Integration); Assistant Secretary of  the Army (Acquisition, Logistics, and Technology); Assistant Secretary of  the Army (Financial Management and Comptroller); Army Chief  Information Officer; Army Deputy Chief of Staff (Logistics); Commander,  U.S. Army Materiel Command; and other interested congressional  committees and members. Copies of this report will be made available to  others upon request. In addition, this report is available at no charge on  the GAO Web site at http://www.gao.gov.
Please contact McCoy Williams at (202) 512-9095 or williamsm1@gao.gov,  Keith A. Rhodes at (202) 512-6412 or rhodesk@gao.gov, or William M. Solis  at (202) 512-8365 or solisw@gao.gov if you or your staff have questions on  matters discussed in this report. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine whether the Army has developed a business system strategy  for achieving total asset visibility (TAV), we met with program office  officials for the General Fund Enterprise Business System (GFEBS), the  Global Combat Support System-Army (GCSS-Army), and the Logistics  Modernization Program (LMP). We obtained briefings on the intended  purpose of each system and walkthroughs of various scenarios that  described how the systems would transmit data between them. We also  met with officials from the Office of Program Analysis and Evaluation, the  Office of the Assistant Secretary of Defense (Network and Information  Integration)/the Department of Defense (DOD) Chief Information Officer,  and the Business Transformation Agency (BTA) to identify and discuss  any issues/concerns they have related to the implementation of GFEBS  and GCSS-Army. Furthermore, we reviewed reports by the Army Audit  Agency and the DOD Inspector General to ascertain if they had previously  reported upon system weaknesses related to the three systems and how  those weaknesses would affect the Army’s vision of an integrated systems  strategy.
In order to determine if the Army has effectively implemented a  governance structure to oversee and manage its business system  investment in accordance with DOD guidance, we reviewed guidance  issued by the Office of the Secretary of Defense, the Army, and BTA  related to investment management. We also obtained an understanding of  the Army’s process to comply with the certification and annual system  review processes required by the fiscal year 2005 National Defense  Authorization Act. We also obtained and analyzed documentation related  to the certification and annual system review process. Furthermore, we  reviewed the Selected Capital Investment Review reports as of February  2006 and February 2007 to ascertain if there were any cost, schedule, and  performance issues discussed in these reports for the GFEBS, GCSS-Army,  and LMP. We also reviewed Defense Acquisition Executive Summary  reports for GFEBS to ascertain if there were any cost, schedule, and  performance issues discussed in these reports. Similar reports were not  required for GCSS-Army and LMP. Additionally, we met with BTA officials  to obtain an understanding of how the Enterprise Risk Assessment Model  is to be utilized as part of the department’s investment management  criteria.
To evaluate the extent to which the Army has made progress in correcting  the previously reported problems regarding LMP’s implementation, we  met with officials from the LMP program office and the Computer  Sciences Corporation (CSC). Briefings were provided that detailed the  specific corrective actions taken in response to the various issues  discussed in our two previous reports. To substantiate the corrective  actions taken to address the accounting issues, we reviewed the Defense  Finance and Accounting Service (DFAS) 1307 accounting reports and  discussed the issues detailed in the reports with DFAS personnel  responsible for preparing the reports. We also reviewed the fiscal year  2006 Army Working Capital Fund Financial Statements to ascertain the  specific issues related to LMP. Further, we met with DOD Inspector  General personnel and reviewed their workpapers to obtain an  understanding of the deficiencies discussed in the fiscal year 2006 Army  Working Capital Fund Financial Statements.
To assess whether the Army had established and implemented disciplined  processes related to requirements management and testing we:    obtained an understanding of the Army’s revised procedures for defining  requirements management frameworks and compared these procedures to  its current practices; reviewed guidance published by the Institute of Electrical and Electronics  Engineers and the Software Engineering Institute and publications by  experts to determine the attributes that should be used for developing  good requirements; and   selected 10 requirements and performed an in-depth review and analysis  to determine whether they had the attributes normally associated with  good requirements and whether these requirements traced between the  various process documents. These requirements were to have followed the  revised requirements management process.
To augment these document reviews and analyses, we discussed our  analysis with LMP program office and CSC officials. We also met with  officials from the Office of Program Analysis and Evaluation, the Office of  the Assistant Secretary of Defense (Network and Information  Integration)/DOD Chief Information Officer (CIO), and the Business  Transformation Agency to identify any issues/concerns they have related  to the implementation of LMP.
We performed our audit work from May 2006 through June 2007 in  accordance with U. S. generally accepted government auditing standards.  We conducted our work at the Office of the Assistant Secretary of Defense  (Networks and Information Integration)/DOD Chief Information Officer;  the U.S. Army Materiel Command; the Army CIO; and the Business  Transformation Agency. We visited the Army Program Executive Office  Enterprise Information Systems and GFEBS program office at Ft. Belvoir,  Virginia. We also visited the Army contractor’s site for LMP, Computer  Sciences Corporation in Marlton, New Jersey—and for GCSS-Army— Northrop Grumman in Chester, Virginia. We also visited the Tobyhanna  Army Depot, Tobyhanna, Pennsylvania and DFAS Indianapolis, Indiana.  We requested comments on a draft of this report from the Secretary of  Defense or his designee. We received written comments from the Deputy  Under Secretary of Defense (Business Transformation), which are  reprinted in appendix II.
section_title: Appendix II: Comments from the Department of Defense
paragraphs: The following are GAO’s comments on DOD’s letter dated July 6, 2007.
subsections: section_title: GAO Comments
paragraphs: 1. See the “Agency Comments and Our Evaluation” section of this report.  2. In regard to the GFEBS economic analysis, the program office did not  provide the necessary support for us to replicate the estimated program  savings.  3. In regard to the sensitivity analysis, the Army’s approach of adding the  uncertainty surrounding the costs of interfaces into the estimate is not  consistent with OMB, DOD, and Army guidance, as discussed in the report.  Rather, a sensitivity analysis is accomplished by changing the numerical  value of a given variable to gauge the effect of that change on model  results, such as the benefit-to-cost ratio. A sensitivity analysis identifies  key assumptions and varies a single assumption while holding the others  constant to determine what amount of change in that assumption is  required to raise or lower the resulting dominant benefit or cost estimates  by a set amount.
section_title: Appendix III: Army Strategy Does Not Fully Utilize Capabilities of ERP Solution
paragraphs: Fundamental to successful enterprise resource planning (ERP)  implementation is the reengineering of an organization’s business  processes in a manner that helps ensure that the right resources are  available when needed. While the Army has stated that through the  successful implementation of the General Fund Enterprise Business  System (GFEBS), the Global Combat Support System-Army (GCSS-Army),  and the Logistics Modernization Program (LMP) will reengineer its  business processes, we found some business processes that have been  adopted or are expected to be adopted have not embraced the basic  concept of an ERP solution. Details concerning the Army’s strategy for  handling funds control and disbursements are highlighted below.
subsections: section_title: Funds Control
paragraphs: The funds control concept that will be utilized by GCSS-Army and GFEBS  is that all funds control activities will be handled by GFEBS. Accordingly,  an interface must be developed to transmit the appropriate information  between the two systems in order to ensure that all of the information is in  agreement. Figure 6 illustrates how the process is intended to work.
As shown in figure 6, information will have to be transmitted from GCSS- Army each and every time there is a transaction that is associated with  funds control. This approach does not take full advantage of the  capabilities within the ERP software that will be utilized by GCSS-Army.  Figure 7 shows an example of more efficient and streamlined funds  control process based on ERP concepts that eliminates these various  transactions having to be processed between GFEBS and GCSS-Army.
We were informed by GFEBS program officials that funds control for the  activities associated with GCSS-Army needed to be maintained in GFEBS  because (1) GCSS-Army was not considered an official accounting system,  (2) the version of the software initially selected by GCSS-Army could not  meet the requirements of the Federal Financial Management Improvement  Act of 1996 (FFMIA), and (3) the Army wanted to centrally manage  general fund activities. In effect, the Army’s approach mirrors the  functionality in the stovepiped processes of the legacy systems, dedicated  processes to obtain information for specific reasons rather than ensuring a  given process provided the information needed from an enterprise point of  view. Furthermore, the concern that the version of software being used by  GCSS-Army had not been certified as compliant with federal requirements  can be easily addressed. Army officials have stated that the software  vendor is committed to obtaining the necessary certifications and expects  that these will be granted before GCSS-Army is deployed. From a  corporate perspective, whether one system or two systems is the proper  approach is not necessarily the question that the Army needs to answer.  Rather, the Army needs to address how best to utilize the capabilities of  the ERP to achieve an effective funds control system.
section_title: Disbursements
paragraphs: The Department of Defense (DOD) Inspector General recently reported  that LMP has been implemented without the functionality to match  proposed disbursements with corresponding obligations before making  any payments. This process is referred to as prevalidation, which was  mandated with the passage of the fiscal year 1995 Department of Defense  Appropriations Act because of DOD’s long-standing difficulty in properly  matching disbursements with corresponding obligations. Implementing  LMP without this functionality is an example of suboptimizing the system,  resulting in the failure to address a long-standing weakness in matching  obligations with proposed disbursements for the Army Working Capital  Fund. This functionality is part of the basic software package that is being  used by LMP. We noted that the Army expects to use the same ERP  software package for GFEBS.
The DOD Inspector General reported that, rather than reengineering its  business processes, the Army decided to follow its existing business  processes that were inefficient and precluded the Army from being in  compliance with the fiscal year 1995 legislative requirement. As a result of  this decision, the Army has not taken full advantage of the capabilities  within the ERP software package being used to implement LMP.  Considering that the Army has already invested approximately               $452 million in LMP, and as previously noted is expecting to invest  hundreds of million more, taking full advantage of its capabilities is also a  prudent financial decision. Because the Army failed to reengineer its  business processes, it continues to use an off-line database, managed by  the Defense Finance and Accounting Service (DFAS), to compare  entitlement and accounting data and provide approval for prevalidation  requests. The DOD Inspector General report noted that the database  contained errors in the accounting data, which resulted in prevalidation  failures and the need for additional research and rework by DFAS.
A more efficient and streamlined process could be achieved by adopting  ERP processes for disbursements. For example, by utilizing ERP concepts  LMP would be the accounting system of record and responsible for  ensuring that the applicable federal requirements governing fund control  and disbursements were effectively implemented. Further, the  disbursements would be made directly by the Department of the  Treasury—much like the process used by civilian agencies—rather than  going through DFAS. We recognize that this approach involves some  technical issues, such as building the necessary interfaces with the  Department of the Treasury. However, the commercial-off-the-shelf  (COTS) package adopted by LMP is already required by its Office of  Management and Budget certification to provide the necessary support for  these interfaces. The elimination of the DFAS interfaces, which are not  inherently supported by the COTS package, should help reduce the cost  and risk associated with this change.
section_title: Appendix IV: LMP Problems Continue
paragraphs: The Tobyhanna Army Depot continues to experience financial  management problems because significant operational and developmental  issues related to the Logistics Modernization Program (LMP) have not  been resolved. More specifically, LMP continues to experience problems  with accurately recognizing revenue and billing customers. This problem  has existed virtually since the implementation of LMP in July 2003. While  the Army and its contractor have made numerous attempts to fix the  problem, they have not been successful. While we found that the  requirements management processes have improved—a critical first step  in reducing risk to acceptable levels—we continue to have reservations  about the adequacy of LMP’s testing.
subsections: section_title: Billing Problems Continue to Plague LMP
paragraphs: LMP’s continuing billing problems have resulted in (1) customers not  being billed for costs incurred that should have been billed and                 (2) customers being billed too much. According to information provide by  the LMP program office, 146 customer orders valued at approximately   $5.4 million were not billed during the January 31, 2007, billing cycle.  Customer orders that were not recognized as revenue and billed represent  funds that Tobyhanna Army Depot is entitled to, because the work  requested by the customer was performed. It is essential that revenue is  properly recognized and customers billed for work performed because it is  the means by which the depot finances its day-to-day operations. Similarly,  during the January 31, 2007, billing cycle about 308 customer orders  amounting to about $5.8 million were shown as being overbilled and a  credit should have been issued to the customer. It is critical that the credit  be provided to the customer as soon as possible since the customer may  be able to use these funds to meet other funding demands.
Our analysis of the January 2007 billings for Tobyhanna disclosed that, of  the 146 unbilled customer orders, 61 (about 42 percent) related to fiscal  years 2006 and 2007. Similarly, for the credits, 178 of 308 customer orders  (about 58 percent) were for fiscal years 2006 and 2007. From a dollar  perspective, most of these billings also related to fiscal years 2006 and  2007 transactions. The fact that these billing problems have persisted  clearly indicates that the various efforts by the Army to resolve this critical  aspect of the system have failed. Table 1 shows the dollar value and the  number of customer orders related to unbilled work and credits for 3  months—November and December 2006 and January 2007.
The continuing billing problem at Tobyhanna is also a factor contributing  to higher depot bills from the Defense Finance and Accounting Service  (DFAS)—the Department of Defense’s (DOD) centralized finance and  accounting organization. According to DFAS personnel, the recurring  billing and other problems have resulted in DFAS processing Tobyhanna’s  bills manually and performing other manual actions that are not required  by the other depots that do not use LMP. According to the information  provided by DFAS, the number of hours spent on DFAS accounting  services billings for Tobyhanna has increased between fiscal years 2004  and 2006. Table 2 shows the number of hours and related amounts DFAS  billed Tobyhanna in each year.
Furthermore, according to DFAS personnel, while the accounting needs of  other depots requires the support of one full-time DFAS staff, DFAS uses  three full-time personnel to service Tobyhanna due to the numerous  manual workarounds associated with LMP. The manual workarounds  required to address LMP’s billing issues results in higher DFAS bills for  Tobyhanna, which in turn increases the prices charged by Tobyhanna to  its customers for the work performed. Since Tobyhanna is part of the  Army Working Capital Fund, the costs incurred for accounting services are  passed on to the customer in the form of higher depot prices. During the  course of our audit, LMP program officials stated that they were finalizing  efforts to revise the billing process. At completion of our field work, the  LMP program office stated that it anticipated having the revised process in  place in the August 2007 time frame.
Since billing and revenue recognition have been such a long-standing  issue, and given that the entire process is fundamental to the efficient and  effective operation of the Working Capital Fund, it is incumbent upon the  LMP program office to closely monitor the planned corrective actions and  obtain reasonable assurance that the revised process is meeting the  Army’s expectations. System testing is a critical process that should help  improve an entity’s confidence that the system will satisfy the  requirements of the end user and will operate as intended. If the planned  corrective actions do not resolve the existing problem, Tobyhanna’s  operations will continue to be adversely affected. As we have previously  recommended, LMP should not be deployed to any additional locations.
section_title: Accuracy of Financial Information Reported by LMP Still a Concern
paragraphs: LMP continues to experience difficulty in accurately reporting on the  results of operations. For example, the DOD Inspector General’s fiscal  year 2006 financial statement audit of the Army Working Capital Fund  identified instances in which LMP was not properly recording transactions  in accordance with the U.S. Government Standard General Ledger  requirements. Further, our analysis of the DFAS monthly report on depot  operations—commonly referred to as the 1307 report—found that  inaccurate account balances continue to persist in LMP. These issues have  presented many challenges that are reflected throughout the Army  Working Capital Fund financial statements as adjustments, corrections,  and footnote disclosures. More specific details on the problems are  highlighted below.
The DOD Inspector General Report on Internal Controls and Compliance  with Laws and Regulations noted that the nonfederal accounts payable  account was misstated by approximately $175 million at the end of fiscal  year 2006. This misstatement arose because Army entities using LMP had  not resolved approximately $89 million in abnormal accounts payable  balances reported by the Supply Management business area. An abnormal  account balance is one in which the recorded amount has been incorrectly  classified. During fiscal year 2006, the LMP program office initiated  systemic corrections to reduce the reported abnormal balances. Our  analysis of the first quarter fiscal year 2007 financial statements noted that  the reported balance had been reduced to approximately $81 million. The  LMP program office was continuing to research the issue.
Our analysis of the DFAS 1307 reports found that abnormal balances in  accounts payable have been reported since fiscal year 2004. LMP program  officials stated that abnormal balances were primarily due to the migration  of remaining obligations on open contracts from the legacy systems.  Substantial effort was made in fiscal year 2006 to correct the abnormal  accounts payable balances that were reported by LMP, which resulted in a  reduction from $334 million at the end of fiscal year 2005 to $88 million at  the end of fiscal year 2006. The Army’s efforts to clean up abnormal  account balances continued in fiscal year 2007.
The Army has developed a specific plan to address the financial  management problems recognized in these statements. For example, a  Tobyhanna Army Depot on-site support team has been assembled to  complete the data cleanup, validate that all system and procedural issues  have been identified, document business processes and procedures, and  train Tobyhanna users. However, until these problems are corrected, LMP  will not be able to provide reliable and accurate financial information to  the Congress or Army management.
section_title: Adequacy of LMP Testing Continues to Be a Concern
paragraphs: Our prior audits of LMP identified significant weaknesses with LMP’s  efforts to effectively implement the requirements management and testing  processes needed to reduce risks to acceptable levels. During our current  audit, we found that the requirements management processes have  improved—a critical first step—but we continue to have reservations  about the adequacy of the testing. LMP program officials stated that these  testing weaknesses are being addressed and expect that an improved  testing process will be effectively implemented to reduce risks to  acceptable levels.
subsections: section_title: LMP’s Requirements Management Practices Have Improved
paragraphs: During the current review, Army and contractor Computer Sciences  Corporation (CSC) officials stated that they have acted upon our prior  recommendations and taken steps to improve the program’s requirements  management practices. In setting forth the revised requirements  management process, LMP and CSC officials noted that the new process  generally envisioned having several documents that outlined the  requirements that needed to be addressed with each document containing  specificity needed for the intended audience.
To ascertain if the Army’s stated corrective actions were being adhered to,  we selected 10 requirements for detailed review. Our analysis found that  the LMP documentation in support of the requirements management  practices was generally in compliance with the stated process. More  specifically, the requirements reviewed were generally consistent between  the documents and appeared to contain the necessary specificity to reduce  requirement-related defects to acceptable levels. Our observations were  consistent with the conclusions reached by a subsequent review  performed by the LMP verification and validation contractor, who  examined the same documentation that we analyzed.
section_title: Concerns regarding LMP Testing Processes Remain
paragraphs: Our review of the stated testing processes found that many of the  attributes associated with a disciplined testing process were present.  However, based upon our review and analysis of the 10 selected corrective  actions, we still have concerns that the LMP testing process may not be  adequate since we identified specific weaknesses in the 10 corrective  actions reviewed. System testing is a critical process utilized by  disciplined organizations and improves an entity’s confidence that the  system will satisfy the requirements of the end user and operate as  intended. The stated testing process included the following:    Documenting scenario testing. A scenario is a business process that  typically consists of several events. For example, a scenario could be  developed for small purchases, which would include events such as         (1) initiating a purchase request, (2) approving the request, (3) obligating  the necessary funds, (4) ordering the item, (5) receiving the item, and      (6) paying for the item. One benefit of testing scenarios is that it helps to  ensure that information is accurately passed from one process to another.  For instance, the denial of a requisition prevents the purchase of the item.
Linking requirements to the test cases. Linking requirements and test  cases helps ensure that either (1) all requirements are adequately tested or  (2) the risks associated with not testing a requirement can be properly  evaluated. Because it is not economically feasible to develop a testing  program that can find every defect, it is critical that an approach be  developed to ensure that testing resources are focused on the areas of  highest risk. Accordingly, linking the requirements makes it easier to  determine which combinations of items are not tested, thereby enabling  the Army to evaluate the risks of not conducting those tests.
Regression testing. Regression testing is a process that helps ensure that  changes made to the system have not adversely affected functionality that  was working prior to a change being made to the system. In essence, it is a  process that retests the entire system to ensure that the problem fixed did  not have an adverse effect on other functions within the system.
While the processes described above are an improvement over what we  have seen in the past, we continue to identify weaknesses in the testing  process. For example, none of the test scripts provided adequate  information that linked the items in the test script to a specific  requirement. This linking is commonly referred to as traceability and is  characteristic of a disciplined testing process. The test scripts reviewed  contained headings for information on the scenario, key data  requirements, expected results, actual results, and whether the test was  considered successful. However, the actual test script failed to contain the  level of specificity that clearly delineated how a specific requirement  identified in the requirement document(s) and the associated  requirement(s) in that document were being tested. As a result, it is  virtually impossible to ascertain if the requirement was properly tested.  For example, several requirements defined rules that were expected to be  implemented. Without linking a given requirement to the tests designed to  exercise that requirement, it was impossible to obtain reasonable  assurance that (1) all the requirements had been tested and (2) the test  provided adequate coverage for each requirement.
A review of the test script for one corrective action showed that the  identified expected results were “output of report should show open  accounts receivable items for customers in the range noted.” Our analysis  of the documentation disclosed that the test scripts only tested to make  sure that a report could be printed for (1) a specific customer, (2) a  consecutive range of customers (e.g., customers 1 through 10), (3) federal  and nonfederal customers (the exact condition tested was not specified  since the user was only required to pick one of the conditions), and   (4) receivables associated with a specified general ledger account. While  this testing addressed the requirements that the report be printed using  these conditions, the scripts did not provide evidence that other  requirements were tested and the data presented were valid. For example,  the report was expected to put receivables into 11 different age categories  (for example 0 to 30 days, 31 to 60 days, etc.); however, no evidence was  provided to show that the balances shown on these reports (1) were  consistent with known results and (2) the accounts selected provided  adequate coverage of the ranges specified. The latter is especially  important since two of the ranges overlap. LMP and CSC officials agreed  that the documentation did not always provide the level of detail that was  needed to (1) document which requirements contained in applicable  requirement documents were being tested and (2) ensure that all of the  specific tests that were required to provide reasonable assurance that the  application worked as expected were completed.
Furthermore, we were informed that the system testing was conducted by  the developers and subject matter experts. Using developers and subject  matter experts, and depending on those individuals’ comprehensive body  of knowledge as the foundation for a testing effort, carries significant risks  since this is not a recognized best practice. A basic testing principle is that  a developer should not test his or her own work, nor should a  development organization test its own work. These testing principles are  based on the concept that testing is the process of executing a program  with the intent of finding errors. Testing is normally considered a  “destructive” process while the development activities are normally  considered a “constructive” progress. It is very difficult for a developer to  change the perspective required to successfully develop a program into  the perspective necessary to adequately test a process. This does not  mean that the developers are not involved in the testing process since a  best practice is to have them responsible for testing the code they develop  to ensure that it is ready for the next stage of testing.
Further, as noted in our May 2004 report, our analysis of LMP’s December  2003 and January 2004 project status reports identified numerous  instances in which the Army continued to experience problems with the  accuracy of data related to budgeting; workload planning and forecasting  and depot maintenance operations; and accounting records such as  customer orders, purchase orders and requisitions, obligations, and  disbursements. DFAS and Army officials acknowledged that these  problems were attributable to relying on subject matter experts to develop  tests for their respective functional areas, and not performing end-to-end  testing across the various functional areas.
After discussing our concerns with the LMP program office and CSC  officials in February 2007, the LMP program office requested the  verification and validation (V&V) contractor to perform an assessment of  the LMP testing process. According to information provided by the LMP  program office and the V&V contractor, in March 2007, the review found  that the LMP testing process was adequately defined in the planning  documentation and noted that the LMP testing philosophy was governed  by the concepts that (1) the change should operate as intended, (2) the  change should not be harmful to the existing functional environment,      (3) testing should be practical, and (4) the testing efforts should be cost  efficient. The V&V contractor analysis of the LMP testing process also  found that the testing documentation—which was the same  documentation that we analyzed did not support a clear understanding of  the (1) requirement being tested and (2) tests used to determine whether a  requirement was adequately implemented. According to the V&V  contractor, these key areas need to be addressed. The V&V contractor  assessment substantiates our analysis of the LMP testing process.
section_title: Improved Testing Process Planned
paragraphs: LMP program office officials and the V&V contractor agreed that the  testing process currently being utilized could be improved. They also  noted that the LMP program had begun implementing an initiative to  improve the testing program that should address the weaknesses we  identified. The following outlines two key components of LMP’s improved  testing efforts.
Establishment of an independent test group. This is a critical step  and, if effectively implemented, can serve as a strong foundation for  building an effective testing process. An independent testing group is a  best practice.
Development and implementation of an automated testing  program. CSC is developing an automated testing process which was  expected to reduce the testing burden and eliminate user errors in the  testing process. For example, the user may not detect that the application  did not provide the expected result or record an error when the  application did operate as expected. These are commonly referred to as  testing defects and are to be expected when manual testing activities are  conducted.
LMP program officials are of the opinion that these two actions will help  address the problems of adequately linking the requirements to the testing  and documenting the testing that is actually performed. These actions, if  effectively implemented, are a significant step forward and can help  reduce the risks associated with testing to acceptable levels. However, the  effective implementation will have to overcome a number of past  problems, including ensuring that the tests provide adequate coverage.  While developing automated tests can improve the reliability of the testing  efforts and help facilitate an effective regression testing program, the key  is whether the automated tests are testing the correct items. In our  discussions with CSC, we were informed that the existing test scripts were  the basis for developing the new automated test. Whether this approach  will provide an effective testing process is yet to be seen. As we noted  above, the test scripts that will be used as the basis for developing the  automated test scripts do not provide the necessary link to the  requirements that are being tested or provide the detailed information  necessary to write the detailed test scripts called for in the automated  process.
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the above contacts, the following individuals made key  contributions to this report: J. Christopher Martin, Senior-Level  Technologist; Darby Smith, Assistant Director; Evelyn Logue, Assistant  Director; F. Abe Dymond, Assistant General Counsel; Beatrice Alff; Sunny  Chang; Harold Brumm, Jr.; Francine DelVecchio; K. Eric Essig; Jason  Kelly; Jason Kirwan; Alyson Mahan; and Debra Rucker.