DOD Business Transformation: Lack of an Integrated Strategy Puts the Army's Asset Visibility System Investments at Risk
Released: 2007-07-27T00:00:00

Letter

Background

TAV has been elusive within DOD. Timely, reliable information on the location, quantity, and status of the department’s tangible assets could significantly improve its ability to more efficiently and effectively deliver needed items to DOD operating forces and thereby enhance military readiness. The department has recognized the importance of achieving TAV and included it as part of its overall business transformation initiative, which includes the development and implementation of a business enterprise architecture (BEA). The Joint Chiefs of Staff also identified TAV as one of four fundamental changes needed to transform the department’s logistics operations.

One of the primary factors contributing to DOD’s inability to provide management with TAV is DOD’s outdated and ineffective management information system environment. The Federal Financial Management Improvement Act (FFMIA) of 1996 and other financial management reform legislation have emphasized the importance of improving financial management, which necessarily encompasses proper inventory management, across the federal government. Built upon the foundation laid by the Chief Financial Officers Act of 1990, FFMIA emphasizes the need for agencies to have integrated financial management systems that can generate timely, accurate, and useful information to make informed decisions and to ensure accountability on a continuous basis. Lacking such critical information, government leaders will not be positioned to invest resources, reduce costs, oversee programs, and hold agency managers accountable for the manner in which government programs are operated.

Army Initiatives Aimed at Achieving TAV

To improve control and accountability over its assets, the Army has embarked on a multisystem integration effort that is intended to leverage commercial ERP software and processes. This integration is the focus of the Single Army Logistics Enterprise (SALE) initiative. SALE is designed to integrate information technology requirements, business processes, business rules, and data from the Army’s logistics, financial, and acquisition transactions for planning and supporting warfighting logistics operations. According to the Army, SALE is to provide a coordinated ERP solution built around two individual logistics system development efforts: (1) LMP and (2) GCSS-Army. Under the SALE vision, GCSS-Army and LMP will be integrated into a single solution to provide an Army-wide logistics environment spanning from “the factory to foxhole.” On the financial side, the Army is developing GFEBS to provide financial visibility over its assets. Each of these efforts is described below.

LMP. In February 1998, the U.S. Army Materiel Command began an ERP effort—LMP—to replace its legacy materiel and maintenance management systems—the Commodity Command Standard System and the Standard Depot System—with LMP. The Army has been using the existing legacy systems for over 30 years. LMP is intended to transform the Army’s Working Capital Fund logistics operations in six core processes: order fulfillment, demand and supply planning, procurement, asset management, materiel maintenance, and financial management. LMP became operational at the U.S. Army Communications and Electronics Command, Fort Monmouth, New Jersey, and Tobyhanna Army Depot, Tobyhanna, Pennsylvania, in July 2003. The initial deployment of LMP consisted of inventory items such as electronics; electronic repair components; and communications and intelligence equipment such as night vision goggles, electronic components such as circuit boards, and certain munitions such as guidance systems included in missiles. Figure 1 shows the LMP time line as of March 2007.

The Communications-Electronics Life Cycle Management Command, Tobyhanna Army Depot, and the Defense Finance and Accounting Service (DFAS) are the primary LMP users. When LMP is fully implemented, its capacity is expected to include more than 17,000 users at 149 locations and it will be populated with 6 million Army-managed inventory items valued at about $40 billion. LMP is scheduled to reach full operational capability (FOC) in fiscal year 2010. As of September 30, 2006, the Army reported that approximately $452 million had been obligated for this system effort and estimates that it will invest at least another $895 million in LMP.

GCSS-Army. The GCSS-Army program was initiated in 1997 to overcome duplicative databases, poor asset visibility, and stovepiped communications between numerous existing Army logistics systems. The goal of GCSS-Army is to integrate multiple logistic functions by replacing numerous legacy systems and interfaces. Since the program’s inception, it has undergone several revisions, including a change from a custom software development to a commercial ERP solution, as well as a change in the prime contractor. According to Army officials, it invested approximately $95 million in the previous efforts before adopting the ERP approach. The existing ERP effort was started in September 2003 and is currently estimated to reach FOC during fiscal year 2014. Figure 2 shows the time line for GCSS-Army as of March 2007.

GCSS-Army is intended to replace 16 stovepiped, legacy logistics systems that cover certain types of inventory and PP&E. The system is intended to be operational at all deployable Army units, and provide asset visibility for accountable items down to the foxhole. As of September 30, 2006, the Army reported that it had obligated approximately $203 million for the ERP version of GCSS-Army. Additionally, the Army estimates that another $2.1 billion will be invested in GCSS-Army.

Product Lifecycle Management Plus (PLM+). This initiative is the technical enabler to integrate LMP and GCSS-Army. PLM+ is the means by which the Army intends to achieve the SALE vision of integrated logistics systems. PLM+ is intended to provide a single point of entry and exit for interfaces to external systems. Furthermore, PLM+ is intended to eliminate duplicative and costly system interfaces. Additionally, the PLM+ implementation schedule and related funding are aligned with that of GCSS-Army. PLM+ does not stand alone as an independent capability. As of September 2006, the Army reported that approximately $31 million had been obligated for PLM+.

GFEBS. While LMP is intended to provide financial control for the Army Working Capital Fund, GFEBS is intended to provide this important control over all non-working capital fund inventory, including that which is reported in GCSS-Army. GFEBS is an ERP solution that was initiated in October 2004 and is intended to serve as the Army’s general ledger system for its general fund accounting. As such, GFEBS is intended to improve the reliability of the Army’s financial information and thereby enhance the Army’s management decision-making process. GFEBS is expected to replace 87 legacy systems, including the 30-year old Standard Army Finance System. Figure 3 shows the GFEBS time line as of June 2007.

The Army estimates that GFEBS will reach FOC by 2010. As of September 30, 2006, the Army reported that it had obligated $123 million for the development of GFEBS. In addition, the Army currently estimates that it will invest another $1.3 billion to implement GFEBS.

Overview of DOD’s Investment Management Practices

In 2005, DOD adopted a “tiered accountability” approach to improve control and accountability over the billions of dollars it invests annually in DOD business systems. Under this approach, executive leadership for the direction, oversight, and execution of DOD investments is the responsibility of several entities within DOD and its components. As shown in figure 4 and described below, the investment control process begins at the component level and works its way up through a hierarchy of review and approval authorities, depending on the size and significance of the investment.

At the DOD enterprise level, key entities involved in maintaining control and accountability over Army business system investments with systems modernizations over $1 million include the DBSMC, which serves as the highest ranking governance body for business systems modernization activities; the Principal Staff Assistants, who serve as the certification authorities for business system modernizations in their respective core business missions; the investment review boards (IRBs), which form the review and decision-making bodies for business system investments in their respective areas of responsibility and review each investment for BEA compliance; and BTA, which provides support to the DBSMC and the IRBs and is responsible for leading and coordinating business transformation efforts across the department. The BTA is organized into seven directorates, one of which is the Defense Business Systems Acquisition Executive—the component acquisition executive for DOD enterprise-level (DOD-wide) business systems and initiatives. This directorate is responsible for developing, coordinating, and integrating DOD enterprise-level projects, programs, systems, and initiatives— including managing resources such as funding, personnel, and contracts for assigned systems and programs.

To implement tiered accountability within the Army, the Army designated its Chief Information Officer (CIO) as the Army’s Pre-Certification Authority (PCA) for certification and annual reviews of business system investments. The PCA is accountable for the component’s business system investments and acts as the component’s principal point of contact for communication with the IRBs. As such, the PCA (1) validates that the system information for all business systems modernizations over $1 million is complete and accessible to the IRBs, (2) reviews development/modernization investments with total cost of $1 million or less, (3) reviews system compliance with DOD’s BEA and enterprise transition plan, (4) verifies the investment’s economic viability analysis, (5) asserts the status and validity of the investment information by submitting a component precertification letter to the appropriate IRB for its review, and (6) provides IT portfolio management policy guidance and oversight of mission area/domain IT portfolios. Below the Army enterprise level, the Army has established six functional area domains within its business mission area. The domains are responsible for implementing the IT portfolio management process developed by DOD and the Army to define and justify the portfolio of planned IT expenditures consistent with strategic objectives and operational requirements.

GAO Has Previously Reported LMP Issues

Our May 2004 report pointed out that the Army had not effectively managed its implementation of LMP. The report noted that after LMP was deployed in July 2003, operational difficulties at the Tobyhanna Army Depot resulted in inaccurate financial management information. More specifically, the depot was not (1) producing accurate workload planning information, (2) generating accurate customer bills, and (3) capturing all repair costs, necessary for the Army to calculate accurate future repair prices. The report also pointed out that LMP requirements (1) lacked the specific information necessary to understand the required functionality that was to be provided, and (2) did not describe how to determine quantitatively, through testing or other analysis, whether the system would meet the Army’s needs. Subsequently, in June 2005, we reported that the problems with LMP continued to prevent the Tobyhanna Army Depot from accurately reporting on its financial operations, including gains and losses, which adversely affected the depot’s ability to accurately set customer sales prices. Further, the report pointed out that problems persisted with recognizing revenue and billing customers. We recommended, and the Army agreed, that the implementation of LMP should be delayed until the operational problems we identified were resolved. The Army is continuing to resolve the outstanding issues.

Army Lacks an Integrated Strategy for Achieving TAV

The Army has estimated that it will invest about $5 billion over the next several years to complete development and implementation of GFEBS, GCSS-Army, and LMP. However, the Army is making this significant investment without a clear integrated strategy for how these systems will be used to achieve TAV over hundreds of billions of dollars of assets. As we reported in December 2004, because DOD has not developed a clear long-range strategy, the military services will be exposed to the risk of spending billions of dollars on duplicative, stovepiped systems that do not support the department’s business transformation goals, including attaining TAV. More specifically, we found that the Army’s current strategy for achieving TAV, including the implementation of these three systems, does not embrace two key foundational elements that we had identified as essential to achieving successful transformation of business systems and processes: an EA and a concept of operations. Without these key foundational elements, the Army is at risk of investing billions of dollars in business systems that may not achieve DOD’s and the Army’s goal of achieving TAV. Further, the Army’s planned strategy perpetuates some of the cumbersome and ineffective business processes and data redundancies that are currently being used in the existing legacy system environment.

Army’s Efforts to Achieve TAV Lack the Benefit of an EA

The Army has yet to develop and implement an Army business EA to help guide its system efforts, including those aimed at achieving TAV. DOD’s acquisition policies and guidance, as well as federal and best practice guidance, recognize the importance of investing in business systems within the context of an EA. GFEBS, GCSS-Army, and LMP are not being managed and developed in the context of a well-defined Army-wide EA. In August 2006, we reported that the Army was in the very early stages of developing an EA. We recommended that the Secretary of Defense ensure that the DOD architecture programs we reviewed develop and implement plans for fully satisfying each of the conditions in our Enterprise Architecture Management Maturity Framework (EAMMF) for assessing and improving EA management. In commenting on the report, DOD agreed with our recommendation.

As detailed in our August 2006 report, we assessed the Army’s efforts to develop an EA against the criteria specified in EAMMF. Our EAMMF is a five-stage architecture framework for managing the development, maintenance, and implementation of an architecture and understanding the extent to which effective architecture management practices are being performed and where an organization is in its progression toward having a well-managed architecture program. In short, the framework consists of 31 core elements that relate to architecture governance, content, use, and measurement. These elements reflect research by us and others showing that architecture programs should be founded upon institutional architecture commitment and capabilities, and measured and verified products and results. Our analysis of information provided by the Army indicated that it had satisfied only 3 percent of all framework elements. In essence, this means that the Army is at stage 1 of developing and implementing an EA. While stage 1 agencies may have initiated some enterprise architecture activity, these agencies’ efforts are ad hoc and unstructured, lack institutional leadership and direction, and do not provide the management foundation necessary for successful enterprise architecture development.

In May 2007, we reported that the Army still had not developed an EA. As a result, the Army does not have a well-informed basis for determining if these systems will fit within the context of future Army business operations and will most efficiently and effectively address the Army’s long-standing weaknesses associated with the lack of asset visibility. Improving asset visibility is critical to addressing the problems associated with supply chain management, which has been on our high-risk list since 1990. Asset visibility is one of the focus areas that are a critical part of supply chain management.

In addition, without a clear understanding of its current systems and business processes—commonly referred to as the “As Is” environment— and the business operations it envisions for the future—commonly referred to as the “To Be” environment—the Army will increase the risk that misalignments can occur that can introduce redundancies and incompatibilities that can produce inefficiencies and require costly and time-consuming rework to fix. GFEBS, GCSS-Army, and LMP have proceeded without a common, institutional frame of reference (for example, EA) that can be used to effectively manage their relationships and dependencies. In November 2006, DOD’s internal review of GFEBS by BTA noted that the Army lacked a strategy for integrating GFEBS, GCSS- Army, and LMP.

Army Strategy to Achieve TAV Does Not Utilize an Enterprisewide Perspective

The Army also lacks a concept of operations for how GFEBS, GCSS-Army, and LMP will collectively achieve TAV. An effective concept of operations would describe, at a high level, (1) how the three business systems relate to each other in achieving the Army’s TAV goal, and (2) how information flows from and through these systems. Moreover, we found that the Army’s lack of a concept of operations has contributed to its failure to take full advantage of business process reengineering opportunities that are available when using an ERP solution. Without a clear long-term strategy, the Army lacks a key management control for ensuring that time frames, results-oriented performance measures, and accountability mechanisms are established and monitored to help achieve TAV.

Specifically, we noted that the Army’s strategy perpetuates some of the cumbersome and inefficient business processes that are currently being used in the existing legacy system environment. One of the key benefits of an ERP is that it can be used to streamline business processes and eliminate data redundancy. However, some of the basic business processes being used or expected to be used by GFEBS, GCSS-Army, and LMP are based on the Army’s existing business processes—many of which are error prone, labor intensive, and redundant. Considering the billions of dollars the Army is investing in these systems, it is incumbent upon the Army to embrace and utilize the most efficient and streamlined ERP processes to the fullest extent possible in its effort to achieve TAV.

For example, the Army’s planned strategy to improve control and accountability over its PP&E does not take advantage of the capabilities of an ERP solution. (See app. III for additional examples in which the Army’s TAV systems initiatives have not effectively streamlined legacy processes related to funds control and disbursements.) Financial data, including acquisition cost and depreciation, for most of the Army’s PP&E are currently maintained in the Defense Property Accountability System (DPAS). PP&E accountability data, such as location and quantity, are maintained in a variety of legacy systems such as the Property Book Unit Supply Enhanced (PBUSE) system—which is 1 of the 16 systems that is expected to be replaced by GCSS-Army. In March 2007, Army Program Executive Office Enterprise Information Systems officials informed us that the financial data—such as acquisition cost and depreciation—for the Army’s PP&E will be transferred from DPAS to GFEBS. This maintains the existing accounting data in an accounting system. On the other hand, the accountability data—such as location, condition, and number of units—for PP&E assets will initially be transferred from the current legacy system to PBUSE or GCSS-Army, depending on whether the items belong to an installation or an Army unit that may be deployed. The transferring of items to PBUSE appears to be an interim solution since, according to DOD’s enterprise transition plan GCSS-Army is to replace PBUSE. However, GCSS-Army is not expected to assume the control and accountability over items that are recorded in PBUSE associated with the Army Working Capital Fund or other Army organizational entities that are not considered to be units that can be deployed (field/tactical units) in military operations.

The Army’s decision to segregate its PP&E financial and accountability data into two separate system solutions reflects its lack of an enterprise concept of operations and enterprise view for achieving TAV. The Army’s strategy of moving PP&E data from DPAS to PBUSE, and ultimately to another system, means that one data conversion will take place to move the data from DPAS to PBUSE—a system the Army is supposed to eliminate with the implementation of GCSS-Army—followed by another data conversion from PBUSE to yet another system, not yet identified. Further, the Army is continuing to limit the capabilities of LMP by transferring all information on PP&E from DPAS to PBUSE—which as noted before is considered a legacy business system. As a result, information on PP&E that could enhance Tobyhanna’s operations, especially in the area of workload planning, will not be available within the LMP environment.

Thus, the approach being taken by the Army is simply to adapt the processes used by the legacy systems rather than use the streamlined processes inherent in the ERP solution. If the functionality of an ERP solution was fully utilized by the Army, it would increase the likelihood that TAV could be accomplished within each of the three systems for the specific type of PP&E items each was responsible for. For example, GFEBS would contain both the financial and accountability information related to nontactical/non-working capital fund PP&E, GCSS-Army would have all the information associated with tactical PP&E, and LMP would have the data related to Working Capital Fund PP&E. Figure 5 illustrates one way that an ERP approach could be utilized to achieve physical and financial accountability control of the Army’s PP&E.

While the Army’s approach may result in incremental improvements in its asset accountability, it does not take full advantage of the (1) benefits of adopting enterprise processes and (2) functionality that is available in the ERP solution it has selected to support those processes. Fundamental to a successful ERP implementation is the reengineering of an organization’s business processes in a manner that helps ensure that the right resources (people, material, machinery, and funds) are available in the correct quantities when needed. While Army has stated that its use of ERP software would help reengineer its business processes, we found that at least some business processes that are being used or expected to be used in the future do not reflect reengineered processes necessary to most effectively implement an ERP solution. By perpetuating the use of cumbersome, error-prone, and ineffective business processes in its asset accountability operations, the Army will diminish its capability to achieve TAV and improve accountability over its assets.

Army’s Ability to Effectively Oversee Portfolios of Business Systems Investment Is Not Yet Fully Developed

While the Army has established a governance structure to oversee its business system investments—including its asset accountability system investments—that is consistent with DOD guidance, additional actions are needed to enhance oversight, control, and accountability. The Army’s business system investment oversight efforts to date have primarily focused on ensuring that business systems modernizations over $1 million are reviewed in accordance with the criteria specified in the fiscal year 2005 National Defense Authorization Act. Both DOD and Army officials acknowledged that the department’s and Army’s investment review processes, particularly related to their ability to review business system investments from a portfolio perspective, are in the early stage of maturity. Portfolio management is a conscious, continuous, and proactive approach to allocating limited resources among competing initiatives in light of the investments’ relative benefits. Taking an enterprise view enables an organization to consider its investment comprehensively, so that collectively the investments optimally address the organization’s mission, goals, and objectives. In addition, we found that the Army did not have reliable processes and analyses to support its oversight of individual systems modernization program efforts intended to improve asset visibility. Specifically, (1) the Army has not established or implemented an independent verification and validation (IV&V) process for the three systems—GFEBS, GCSS-Army, or LMP and (2) the August 2004 GFEBS economic analysis was not prepared in accordance with DOD and OMB guidance.

Army’s Oversight of Its Business System Investments Lacks a Portfolio Perspective

The Army’s oversight of its business system investments—including system investments intended to achieve TAV—continues to be focused on the review and approval of individual business system investments. In May 2007, we reported that DOD needed to improve its policies and procedures associated with managing its business system investments as portfolios. More specifically, we found that DOD had not yet progressed from project- based processes to portfolio-based processes, a key element for effectively managing business system transformation efforts. DOD informed us that DOD components are responsible for developing and managing their own portfolio management processes.

Army IT guidance for managing IT investments requires portfolios to be managed and monitored using established quantifiable outcome-based performance measures to determine whether to recommend continuations, modification, or termination of individual investments within a portfolio. However, the Army has not implemented processes or procedures that facilitate an enterprise view toward management and oversight of portfolios of business system investments intended to collectively provide a specific capability or functionality. Rather, the Army’s current business system investment process focuses primarily on reviewing and approving investment packages submitted by the responsible program management office for individual systems improvement efforts that have development/modernization funding over $1 million. Individual business system investments that exceed the $1 million systems modernization threshold are required to be reviewed by the appropriate IRB and approved by the DBSMC.

Army officials acknowledged that the Army’s ability to provide portfolio- based investment review and oversight is in the early stages of development. For example, while the Army has begun a process to share investment package information, including cost, schedule, and performance data, across its mission areas and domains, Army officials acknowledged that the information provided by a program management office, as well as the associated oversight and review, is centered largely on the functionality and capabilities of individual systems. For example, annual certifications for the three systems, LMP, GCSS-Army, and GFEBS, which the Army intends to utilize collectively in achieving TAV, are not submitted or reviewed as a portfolio, but rather as individual system investments. The Army has not yet implemented the processes to evaluate and improve its progress toward achieving TAV using portfolio-based projects such as costs, schedule, performance, and risks. As a result, the Army’s ability to effectively oversee the development of the portfolio of systems intended to collectively provide the Army with TAV is limited. Until the Army’s governance process, policies, and procedures mature, including its ability to apply a corporate portfolio perspective in managing and overseeing subportfolios of business system investments intended to provide a desired capability or function, the Army is at risk of implementing systems that (1) do not provide the desired capability, including TAV; (2) are stovepiped in their functionality; (3) do not interoperate in an efficient manner; and (4) are not supported by efficient and effective business processes.

Reliable Processes and Analyses Are Needed to Facilitate Management Oversight of Business System Investments

The Army has not implemented reliable processes and analyses needed to enhance its management oversight of program management office efforts to develop and implement its TAV, as well as other business systems on time, within budget, and with the intended capability. Both DOD’s and the Army’s business system investment oversight processes are highly dependent upon the reliability of investment information provided by Army program management offices. Thus, the reliability of that business system investment information is critical to both DOD’s and the Army’s ability to effectively oversee, manage, and redirect—if necessary—the services’ business system development and modernization efforts. However, we found that the Army had not yet established an independent verification and validation function for any of the three systems we reviewed. In addition, the economic analysis submitted by the program management office for GFEBS was not prepared in accordance with DOD or OMB guidance. As a result, the economic analysis submitted by the program management office to DOD and the Army oversight entities justifying the Army’s investment in GFEBS was based on questionable costs and benefits.

Independent Verification and Validation Would Enhance Program Oversight

The Army had not yet established an IV&V function for any of the three systems—GFEBS, GCSS-Army, and LMP. While the Army has established a verification and validation function for LMP, it was not an independent review because the reviewer reports directly to the LMP program management office. Independence is a key component to reliable verification and validation function.

Best business practices have demonstrated that use of an IV&V function is an invaluable means to providing management reasonable assurance that a planned system, or the portfolio of systems, will satisfy its planned use and users. An effective IV&V review process should provide an independent assessment to DOD and Army management of the overall status of the project, including a discussion of any existing or potential revisions to the project with respect to cost, schedule, and performance. The IV&V reports should identify to management the issues or weaknesses that increase the risks associated with the project or portfolio to senior management so that they can be promptly addressed. These assessments involve reviewing project documentation, participating in meetings at all levels within the project, and providing periodic reports and recommendations, if deemed warranted, to senior management. The IV&V function should report on every facet of a system project such as: Testing program adequacy. Testing activities would be evaluated to ensure they are properly defined and developed in accordance with industry standards and best practices.

Critical-path analysis. A critical path defines the series of tasks that must be finished in time for the entire project (or portfolio of projects) to finish on schedule. Each task on the critical path is a critical task. A critical-path analysis helps to identify the impact of various project events, such as delays in project deliverables, and ensures that the impact of such delays is clearly understood by all parties involved with the project(s).

System strategy documents. Numerous system strategy documents that provide the foundation for the system development and operations are critical aspects of an effective system project. These documents are used for guidance in developing documents for articulating the plans and procedures used to implement a system. Examples of such documents include the Life-cycle Test Strategy, Interface Strategy, and Conversion Strategy.

Validity of GFEBS’s Economic Analysis Questionable

Our analysis of the August 2004 GFEBS economic analysis found that it was not prepared in accordance with OMB and DOD guidance. The purpose of an economic analysis is to give the decision maker insight into economic factors that have a bearing on accomplishing the stated objective of the system investment, for example implementation of GFEBS to improve the Army’s financial accountability. As such, it is important that the economic analysis reliably identifies factors, such as cost and performance risks and drivers that can be used to establish priorities and allocate resource allocations. While it may be appropriate to invest in a particular IT investment, such as GFEBS, for reasons other than estimated economic benefits, nonetheless, the issues we identified raise questions as whether the funds invested in GFEBS were economically justified. Our specific concerns with the August 2004 GFEBS economic analysis are highlighted below.

Inappropriate cost savings. At least $142 million of estimated savings were not savings, but rather, should have been classified as transfers— which do not equate to economic benefits. Transfers represent shifts of control over resource allocation from one group to another that do not result in a net change in the value of the resources involved in the transfer. For example, we found that the Army claimed over $88 million of savings related to transferring real property management from the Army’s legacy system to GFEBS. Since the Army’s real property management responsibilities were not eliminated, only transferred from the legacy system environment to GFEBS, this represents a transfer rather than an economic benefit for the Army. Additionally, the Army claimed benefits of about $54 million related to GFEBS reducing the length of time funds are borrowed by the Treasury to meet the Army financial obligations by facilitating the Army’s ability to make just-in-time disbursements. OMB guidance provides that there are no economic gains from a pure transfer payment because the benefits to those government entities that receive such a transfer are matched by the costs borne by those government entities that provide the transfer. The Army’s inappropriate classification of transfers resulted in an overstatement of the economic benefits that would be achieved by implementing GFEBS. Additionally, in November 2006, BTA’s enterprise risk assessment management review of GFEBS questioned justifying the GFEBS program on factors not fully within the program’s control.

Lack of sensitivity analysis. The analysis did not include an assessment of the effects of the uncertainty inherent in estimates of GFEBS benefits and costs, as stipulated in OMB and DOD guidance. Because an economic analysis uses estimates and assumptions, it is critical that a sensitivity analysis be performed to understand the effects of the imprecision in both underlying data and modeling assumptions. This analysis is required since the estimates of future benefits and costs are subject to varying degrees of uncertainty. Sensitivity analysis refers to changing the value of a given variable in a model to gauge the effect of change on model results. It varies a single data element or assumption while holding the other data elements and assumptions constant to determine what amount of change in that element is required to raise or lower the resulting benefit and cost elements. In this way, GFEBS data and assumptions can be risk-ranked for decisionmaking and auditing. The Army’s failure to conduct a sensitivity analysis of GFEBS, as part of its economic analysis, to identify the effect of uncertainties associated with different assumptions increases the chance that decisions regarding GFEBS will be made without a clear understanding of the possible impact on GFEBS estimates of costs and benefits. Army officials informed us that they will consider including a sensitivity analysis of the effect of uncertainties when they update GFEBS’s economic analysis.

Lack of documentation supporting benefits and cost assumptions.

The GFEBS program management office could not provide us with benefits and costs data needed to replicate their analyses. As a result, we were unable to trace the cost and benefit estimates reported in the GFEBS economic analysis to original source documentation to assess and validate the reliability and applicability of the data. OMB and DOD guidance stress the importance of maintaining documentation supporting assumptions used in a cost-benefit analysis.

LMP Illustrates Continuing Problems in Implementing Business Systems on Time, within Budget, and with the Promised Capability

LMP continues to be plagued by operational problems that have beset the system virtually since its initial implementation in July 2003. While, an EA, concept of operations, and effective IT portfolio management are essential elements in the Army’s efforts to transform its operations, it is equally important for the Army to have disciplined processes in place to implement its business systems on time, within budget, and with the promised capability. Many of DOD’s approximately 2,980 business systems are nonintegrated, stovepiped, and not capable of providing departmental management and the Congress accurate and reliable information on DOD’s day-to-day operations. LMP is no exception. As previously noted, LMP was to have reached FOC in fiscal year 2005, but currently FOC is estimated for fiscal year 2010—a slippage of 5 years. Further, LMP’s estimated program costs have also increased. Tobyhanna Army Depot—the only depot that is operating LMP—and DFAS personnel have stated that improvements have been made in LMP’s operating efficiency. However, problems in the areas of revenue recognition and billing continue. These problems can, in part, be attributed to ineffective system testing. The operational issues confronting LMP and our concerns about the effectiveness of the system testing are highlighted below and discussed in more detail in appendix IV.

Operational Issues Continue

Tobyhanna Army Depot continues to experience problems with LMP accurately recognizing revenue and billing customers. These problems have existed virtually since the implementation of LMP in July 2003. While the Army and its contractor—Computer Sciences Corporation (CSC)— have made numerous attempts to fix the problem, they have not been successful. Additionally, the accuracy of LMP financial reports continues to be questionable. For instance, the DOD Inspector General’s audit of the Army’s Working Capital Fund financial statements for fiscal year 2006 noted that LMP was not properly recording transactions in accordance with the U.S. Government Standard General Ledger requirements.

For the 3-month period ending January 31, 2007, we found that LMP’s continuing billing problems resulted in (1) customers not being billed for costs incurred that should have been billed and (2) customers being billed too much. Based upon the billing information provided by the LMP program management office, there were 146 customer orders valued at approximately $5.4 million that were not billed (or recognized as revenue) during the January 31, 2007, billing cycle. Customer billings and the associated revenue provide the means by which the depot finances its day- to-day operations. Similarly, during the January 31, 2007, billing cycle about 308 customer orders, amounting to about $5.8 million, were shown as being overbilled and a credit should have been provided to the customer. Moreover, the continuing billing and revenue recognition problem may be a factor contributing to higher depot bills from DFAS. According to DFAS personnel, the recurring billing problems have resulted in DFAS personnel processing Tobyhanna Army Depot’s bills manually. Based on information provided by DFAS, the number of hours spent to provide DFAS accounting services—including billings for the Tobyhanna Army Depot—has increased from approximately 17,800 hours in fiscal year 2004 to over 22,600 hours in fiscal year 2006.

System Testing Remains a Concern

Our prior audits of LMP identified significant weaknesses with the LMP program management office’s efforts to effectively implement the requirements management and testing processes needed to reduce risks to acceptable levels. During our current audit, we found that the requirements management processes have improved—a critical first step—but concern remains with respect to the adequacy of the testing.

Our analysis of 10 selected LMP corrective actions identified specific testing weaknesses in each of the corrective actions reviewed. For example, none of the test scripts provided adequate information that linked the items tested to the specific requirement being tested. This linking is commonly referred to as traceability and is characteristic of a disciplined testing process. The test scripts reviewed contained headings for information on the scenario, key data requirements, expected results, actual results, and whether the test was considered successful. However, the actual test script did not contain the level of specificity that clearly delineated how a specific requirement identified in the requirement document(s) and the associated requirement(s) in that document were being tested. As a result, it was not possible to determine if all LMP system requirements were properly tested. Without linking a given requirement to the tests designed to exercise that requirement, it was impossible to obtain reasonable assurance that (1) all the requirements were tested and (2) the applied test provided adequate coverage for each requirement.

After discussing our concerns with the LMP program management office and its contractor in February 2007, the LMP program office requested that its verification and validation (V&V) contractor perform an assessment of the LMP testing process. According to information provided by the LMP program office and the V&V contractor, in March 2007, the review found that the LMP testing process was adequately planned but that it did not support a clear understanding of the (1) requirement being tested and (2) tests used to determine whether a requirement was adequately implemented. The V&V contractor’s assessment substantiates our analysis of the LMP testing process. According to the V&V contractor, these areas need to be addressed before the Army Test and Evaluation Command conducts its review of LMP.

According to LMP program management office officials, the CSC is developing and implementing a testing process that is designed to address the concerns raised by us and the Army V&V contractor. CSC is expected to have a testing program that clearly links the requirements to the tests being conducted and to establish an independent test group to ensure that the testing process is following best practices. However, until the Army has reasonable assurance that an effective testing process has been properly implemented, we are reiterating our June 2005 recommendation that LMP not be deployed to additional locations until the Army has assurance that LMP is providing the intended functionality and, more specifically, that LMP can accurately bill its customers and recognize revenue.

Conclusions

The Army’s efforts to develop and implement GFEBS, GCSS-Army, and LMP may result in incremental improvements in the Army’s accountability and visibility over its billions of dollars in tangible assets. However, the Army has not developed or utilized key management tools and concepts necessary to successfully transform its business processes to achieve TAV. The primary question that the Army has yet to effectively consider and address is how these systems and associated processes, individually and collectively, will provide the desired functionality necessary to achieve TAV. Until the Army develops and implements an Army EA, concept of operations, and portfolio-based management and oversight processes, it continues to risk (1) investing billions of dollars in asset accountability systems that may not enhance the department’s and the Army’s goal of achieving TAV, and (2) further delay DOD’s efforts to remove supply chain management from our high-risk list.

It is also important that the Army take action to implement the disciplined processes necessary to implement asset accountability systems on time, within budget, and with the promised capability. Continuing problems with LMP are illustrative of the consequences in failing to address these issues. Although LMP became operational 4 years ago, it continues to encounter operational problems. While there have been some improvements in LMP, key functionality, such as accurately billing customers and recognizing revenue, remains problematic. The Army’s inability to resolve these persistent LMP problems can, in part, be attributed to inadequate system testing. Until LMP implements an effective testing process, it will have little assurance that the development and corrective actions it takes (1) are properly developed, and (2) do not introduce additional defects into the system.

Recommendations for Executive Action

To improve the department’s efforts to achieve TAV and further enhance its efforts to improve the control and accountability over business system investments, we recommend that the Secretary of Defense direct the Secretary of the Army and the Director, BTA, to jointly take the following five actions: Develop a concept of operations that (1) clearly defines the ERP vision for accomplishing total asset visibility within the Army; (2) addresses how its business systems and processes, individually and collectively, will provide the desired functionality to achieve TAV; and (3) determines the desired functionality among the selected systems.

Develop policies, procedures, and processes to support the oversight and management of selected groupings of business systems that are intended to provide a specific capability or functionality, such as TAV from a portfolio perspective, utilizing indicators such as costs, schedule, performance, and risks.

Establish an IV&V function for GFEBS, GCSS-Army, and LMP.

Additionally, direct that all IV&V reports for each system be provided to Army management, the appropriate IRB, and BTA.

Require that any future GFEBS economic analysis identify costs and benefits in accordance with the criteria specified by DOD and OMB guidance, to include a sensitivity analysis.

Direct that LMP utilize systems testers that are independent of the LMP system developers to help ensure that the system is providing the users of the system the intended capabilities.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from the Deputy Under Secretary of Defense (Business Transformation), which are reprinted in appendix II. Overall, DOD concurred with our recommendations and stated that it would work diligently to implement them. The comments included two sets of specific responses to our recommendations—one set provided by BTA and another provided by the Army. In regard to the response provided by the Army, it concurred with each of the recommendations. BTA stated that it fully agreed with our observations. BTA, though, partially concurred with all the recommendations on the basis that they were directed jointly to the Secretary of the Army and the Director, BTA. BTA stated that under U.S. Code Title 10 and in accordance with DOD’s tiered accountability, the Army has complete authority for execution of its programs and responsibility for implementing the recommendations. BTA’s comments noted that it has neither the authority nor the responsibility to direct the actions of the Army.

We appreciate the department’s willingness to address our recommendations. With regard to BTA’s concern, our recommendations do not direct BTA to oversee or direct the Army. Rather, the recommendations stated that the specific actions should be undertaken jointly at the direction of the Secretary of Defense. We continue to believe that a cooperative and effectively coordinated BTA and Army approach to addressing our recommendations is most likely to achieve the fundamental business system transformation necessary to achieve the department’s TAV objective. Further, the involvement of the Office of the Secretary of Defense, including BTA, in the Army’s investment management practices is consistent with 10 U.S.C. §§ 186 and 2222, which provide for DOD policymaking and oversight of Army business system functions.

In regard to the responses provided by the Army, it acknowledged the importance of a concept of operations in achieving its goal of TAV and in optimizing its business system investments. Further, the Army acknowledged that there are risks involved in attempting to effectively integrate the various system efforts. Going forward, as we recommended, it will be important that the Army develop a clear, long-term strategy to help ensure that time frames, results-oriented performance measures, and accountability mechanisms are established and monitored to achieve TAV.

We are sending copies of this report to the Secretary of Defense; Acting Secretary of the Army; Under Secretary of Defense (Acquisition, Technology, and Logistics); Under Secretary of Defense (Comptroller); Deputy Under Secretary of Defense (Business Transformation); Deputy Under Secretary of Defense (Financial Management); Assistant Secretary of Defense (Networks and Information Integration); Assistant Secretary of the Army (Acquisition, Logistics, and Technology); Assistant Secretary of the Army (Financial Management and Comptroller); Army Chief Information Officer; Army Deputy Chief of Staff (Logistics); Commander, U.S. Army Materiel Command; and other interested congressional committees and members. Copies of this report will be made available to others upon request. In addition, this report is available at no charge on the GAO Web site at http://www.gao.gov.

Please contact McCoy Williams at (202) 512-9095 or williamsm1@gao.gov, Keith A. Rhodes at (202) 512-6412 or rhodesk@gao.gov, or William M. Solis at (202) 512-8365 or solisw@gao.gov if you or your staff have questions on matters discussed in this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: Scope and Methodology

To determine whether the Army has developed a business system strategy for achieving total asset visibility (TAV), we met with program office officials for the General Fund Enterprise Business System (GFEBS), the Global Combat Support System-Army (GCSS-Army), and the Logistics Modernization Program (LMP). We obtained briefings on the intended purpose of each system and walkthroughs of various scenarios that described how the systems would transmit data between them. We also met with officials from the Office of Program Analysis and Evaluation, the Office of the Assistant Secretary of Defense (Network and Information Integration)/the Department of Defense (DOD) Chief Information Officer, and the Business Transformation Agency (BTA) to identify and discuss any issues/concerns they have related to the implementation of GFEBS and GCSS-Army. Furthermore, we reviewed reports by the Army Audit Agency and the DOD Inspector General to ascertain if they had previously reported upon system weaknesses related to the three systems and how those weaknesses would affect the Army’s vision of an integrated systems strategy.

In order to determine if the Army has effectively implemented a governance structure to oversee and manage its business system investment in accordance with DOD guidance, we reviewed guidance issued by the Office of the Secretary of Defense, the Army, and BTA related to investment management. We also obtained an understanding of the Army’s process to comply with the certification and annual system review processes required by the fiscal year 2005 National Defense Authorization Act. We also obtained and analyzed documentation related to the certification and annual system review process. Furthermore, we reviewed the Selected Capital Investment Review reports as of February 2006 and February 2007 to ascertain if there were any cost, schedule, and performance issues discussed in these reports for the GFEBS, GCSS-Army, and LMP. We also reviewed Defense Acquisition Executive Summary reports for GFEBS to ascertain if there were any cost, schedule, and performance issues discussed in these reports. Similar reports were not required for GCSS-Army and LMP. Additionally, we met with BTA officials to obtain an understanding of how the Enterprise Risk Assessment Model is to be utilized as part of the department’s investment management criteria.

To evaluate the extent to which the Army has made progress in correcting the previously reported problems regarding LMP’s implementation, we met with officials from the LMP program office and the Computer Sciences Corporation (CSC). Briefings were provided that detailed the specific corrective actions taken in response to the various issues discussed in our two previous reports. To substantiate the corrective actions taken to address the accounting issues, we reviewed the Defense Finance and Accounting Service (DFAS) 1307 accounting reports and discussed the issues detailed in the reports with DFAS personnel responsible for preparing the reports. We also reviewed the fiscal year 2006 Army Working Capital Fund Financial Statements to ascertain the specific issues related to LMP. Further, we met with DOD Inspector General personnel and reviewed their workpapers to obtain an understanding of the deficiencies discussed in the fiscal year 2006 Army Working Capital Fund Financial Statements.

To assess whether the Army had established and implemented disciplined processes related to requirements management and testing we: obtained an understanding of the Army’s revised procedures for defining requirements management frameworks and compared these procedures to its current practices; reviewed guidance published by the Institute of Electrical and Electronics Engineers and the Software Engineering Institute and publications by experts to determine the attributes that should be used for developing good requirements; and selected 10 requirements and performed an in-depth review and analysis to determine whether they had the attributes normally associated with good requirements and whether these requirements traced between the various process documents. These requirements were to have followed the revised requirements management process.

To augment these document reviews and analyses, we discussed our analysis with LMP program office and CSC officials. We also met with officials from the Office of Program Analysis and Evaluation, the Office of the Assistant Secretary of Defense (Network and Information Integration)/DOD Chief Information Officer (CIO), and the Business Transformation Agency to identify any issues/concerns they have related to the implementation of LMP.

We performed our audit work from May 2006 through June 2007 in accordance with U. S. generally accepted government auditing standards. We conducted our work at the Office of the Assistant Secretary of Defense (Networks and Information Integration)/DOD Chief Information Officer; the U.S. Army Materiel Command; the Army CIO; and the Business Transformation Agency. We visited the Army Program Executive Office Enterprise Information Systems and GFEBS program office at Ft. Belvoir, Virginia. We also visited the Army contractor’s site for LMP, Computer Sciences Corporation in Marlton, New Jersey—and for GCSS-Army— Northrop Grumman in Chester, Virginia. We also visited the Tobyhanna Army Depot, Tobyhanna, Pennsylvania and DFAS Indianapolis, Indiana. We requested comments on a draft of this report from the Secretary of Defense or his designee. We received written comments from the Deputy Under Secretary of Defense (Business Transformation), which are reprinted in appendix II.

Appendix II: Comments from the Department of Defense

The following are GAO’s comments on DOD’s letter dated July 6, 2007.

GAO Comments

1. See the “Agency Comments and Our Evaluation” section of this report. 2. In regard to the GFEBS economic analysis, the program office did not provide the necessary support for us to replicate the estimated program savings. 3. In regard to the sensitivity analysis, the Army’s approach of adding the uncertainty surrounding the costs of interfaces into the estimate is not consistent with OMB, DOD, and Army guidance, as discussed in the report. Rather, a sensitivity analysis is accomplished by changing the numerical value of a given variable to gauge the effect of that change on model results, such as the benefit-to-cost ratio. A sensitivity analysis identifies key assumptions and varies a single assumption while holding the others constant to determine what amount of change in that assumption is required to raise or lower the resulting dominant benefit or cost estimates by a set amount.

Appendix III: Army Strategy Does Not Fully Utilize Capabilities of ERP Solution

Fundamental to successful enterprise resource planning (ERP) implementation is the reengineering of an organization’s business processes in a manner that helps ensure that the right resources are available when needed. While the Army has stated that through the successful implementation of the General Fund Enterprise Business System (GFEBS), the Global Combat Support System-Army (GCSS-Army), and the Logistics Modernization Program (LMP) will reengineer its business processes, we found some business processes that have been adopted or are expected to be adopted have not embraced the basic concept of an ERP solution. Details concerning the Army’s strategy for handling funds control and disbursements are highlighted below.

Funds Control

The funds control concept that will be utilized by GCSS-Army and GFEBS is that all funds control activities will be handled by GFEBS. Accordingly, an interface must be developed to transmit the appropriate information between the two systems in order to ensure that all of the information is in agreement. Figure 6 illustrates how the process is intended to work.

As shown in figure 6, information will have to be transmitted from GCSS- Army each and every time there is a transaction that is associated with funds control. This approach does not take full advantage of the capabilities within the ERP software that will be utilized by GCSS-Army. Figure 7 shows an example of more efficient and streamlined funds control process based on ERP concepts that eliminates these various transactions having to be processed between GFEBS and GCSS-Army.

We were informed by GFEBS program officials that funds control for the activities associated with GCSS-Army needed to be maintained in GFEBS because (1) GCSS-Army was not considered an official accounting system, (2) the version of the software initially selected by GCSS-Army could not meet the requirements of the Federal Financial Management Improvement Act of 1996 (FFMIA), and (3) the Army wanted to centrally manage general fund activities. In effect, the Army’s approach mirrors the functionality in the stovepiped processes of the legacy systems, dedicated processes to obtain information for specific reasons rather than ensuring a given process provided the information needed from an enterprise point of view. Furthermore, the concern that the version of software being used by GCSS-Army had not been certified as compliant with federal requirements can be easily addressed. Army officials have stated that the software vendor is committed to obtaining the necessary certifications and expects that these will be granted before GCSS-Army is deployed. From a corporate perspective, whether one system or two systems is the proper approach is not necessarily the question that the Army needs to answer. Rather, the Army needs to address how best to utilize the capabilities of the ERP to achieve an effective funds control system.

Disbursements

The Department of Defense (DOD) Inspector General recently reported that LMP has been implemented without the functionality to match proposed disbursements with corresponding obligations before making any payments. This process is referred to as prevalidation, which was mandated with the passage of the fiscal year 1995 Department of Defense Appropriations Act because of DOD’s long-standing difficulty in properly matching disbursements with corresponding obligations. Implementing LMP without this functionality is an example of suboptimizing the system, resulting in the failure to address a long-standing weakness in matching obligations with proposed disbursements for the Army Working Capital Fund. This functionality is part of the basic software package that is being used by LMP. We noted that the Army expects to use the same ERP software package for GFEBS.

The DOD Inspector General reported that, rather than reengineering its business processes, the Army decided to follow its existing business processes that were inefficient and precluded the Army from being in compliance with the fiscal year 1995 legislative requirement. As a result of this decision, the Army has not taken full advantage of the capabilities within the ERP software package being used to implement LMP. Considering that the Army has already invested approximately $452 million in LMP, and as previously noted is expecting to invest hundreds of million more, taking full advantage of its capabilities is also a prudent financial decision. Because the Army failed to reengineer its business processes, it continues to use an off-line database, managed by the Defense Finance and Accounting Service (DFAS), to compare entitlement and accounting data and provide approval for prevalidation requests. The DOD Inspector General report noted that the database contained errors in the accounting data, which resulted in prevalidation failures and the need for additional research and rework by DFAS.

A more efficient and streamlined process could be achieved by adopting ERP processes for disbursements. For example, by utilizing ERP concepts LMP would be the accounting system of record and responsible for ensuring that the applicable federal requirements governing fund control and disbursements were effectively implemented. Further, the disbursements would be made directly by the Department of the Treasury—much like the process used by civilian agencies—rather than going through DFAS. We recognize that this approach involves some technical issues, such as building the necessary interfaces with the Department of the Treasury. However, the commercial-off-the-shelf (COTS) package adopted by LMP is already required by its Office of Management and Budget certification to provide the necessary support for these interfaces. The elimination of the DFAS interfaces, which are not inherently supported by the COTS package, should help reduce the cost and risk associated with this change.

Appendix IV: LMP Problems Continue

The Tobyhanna Army Depot continues to experience financial management problems because significant operational and developmental issues related to the Logistics Modernization Program (LMP) have not been resolved. More specifically, LMP continues to experience problems with accurately recognizing revenue and billing customers. This problem has existed virtually since the implementation of LMP in July 2003. While the Army and its contractor have made numerous attempts to fix the problem, they have not been successful. While we found that the requirements management processes have improved—a critical first step in reducing risk to acceptable levels—we continue to have reservations about the adequacy of LMP’s testing.

Billing Problems Continue to Plague LMP

LMP’s continuing billing problems have resulted in (1) customers not being billed for costs incurred that should have been billed and (2) customers being billed too much. According to information provide by the LMP program office, 146 customer orders valued at approximately $5.4 million were not billed during the January 31, 2007, billing cycle. Customer orders that were not recognized as revenue and billed represent funds that Tobyhanna Army Depot is entitled to, because the work requested by the customer was performed. It is essential that revenue is properly recognized and customers billed for work performed because it is the means by which the depot finances its day-to-day operations. Similarly, during the January 31, 2007, billing cycle about 308 customer orders amounting to about $5.8 million were shown as being overbilled and a credit should have been issued to the customer. It is critical that the credit be provided to the customer as soon as possible since the customer may be able to use these funds to meet other funding demands.

Our analysis of the January 2007 billings for Tobyhanna disclosed that, of the 146 unbilled customer orders, 61 (about 42 percent) related to fiscal years 2006 and 2007. Similarly, for the credits, 178 of 308 customer orders (about 58 percent) were for fiscal years 2006 and 2007. From a dollar perspective, most of these billings also related to fiscal years 2006 and 2007 transactions. The fact that these billing problems have persisted clearly indicates that the various efforts by the Army to resolve this critical aspect of the system have failed. Table 1 shows the dollar value and the number of customer orders related to unbilled work and credits for 3 months—November and December 2006 and January 2007.

The continuing billing problem at Tobyhanna is also a factor contributing to higher depot bills from the Defense Finance and Accounting Service (DFAS)—the Department of Defense’s (DOD) centralized finance and accounting organization. According to DFAS personnel, the recurring billing and other problems have resulted in DFAS processing Tobyhanna’s bills manually and performing other manual actions that are not required by the other depots that do not use LMP. According to the information provided by DFAS, the number of hours spent on DFAS accounting services billings for Tobyhanna has increased between fiscal years 2004 and 2006. Table 2 shows the number of hours and related amounts DFAS billed Tobyhanna in each year.

Furthermore, according to DFAS personnel, while the accounting needs of other depots requires the support of one full-time DFAS staff, DFAS uses three full-time personnel to service Tobyhanna due to the numerous manual workarounds associated with LMP. The manual workarounds required to address LMP’s billing issues results in higher DFAS bills for Tobyhanna, which in turn increases the prices charged by Tobyhanna to its customers for the work performed. Since Tobyhanna is part of the Army Working Capital Fund, the costs incurred for accounting services are passed on to the customer in the form of higher depot prices. During the course of our audit, LMP program officials stated that they were finalizing efforts to revise the billing process. At completion of our field work, the LMP program office stated that it anticipated having the revised process in place in the August 2007 time frame.

Since billing and revenue recognition have been such a long-standing issue, and given that the entire process is fundamental to the efficient and effective operation of the Working Capital Fund, it is incumbent upon the LMP program office to closely monitor the planned corrective actions and obtain reasonable assurance that the revised process is meeting the Army’s expectations. System testing is a critical process that should help improve an entity’s confidence that the system will satisfy the requirements of the end user and will operate as intended. If the planned corrective actions do not resolve the existing problem, Tobyhanna’s operations will continue to be adversely affected. As we have previously recommended, LMP should not be deployed to any additional locations.

Accuracy of Financial Information Reported by LMP Still a Concern

LMP continues to experience difficulty in accurately reporting on the results of operations. For example, the DOD Inspector General’s fiscal year 2006 financial statement audit of the Army Working Capital Fund identified instances in which LMP was not properly recording transactions in accordance with the U.S. Government Standard General Ledger requirements. Further, our analysis of the DFAS monthly report on depot operations—commonly referred to as the 1307 report—found that inaccurate account balances continue to persist in LMP. These issues have presented many challenges that are reflected throughout the Army Working Capital Fund financial statements as adjustments, corrections, and footnote disclosures. More specific details on the problems are highlighted below.

The DOD Inspector General Report on Internal Controls and Compliance with Laws and Regulations noted that the nonfederal accounts payable account was misstated by approximately $175 million at the end of fiscal year 2006. This misstatement arose because Army entities using LMP had not resolved approximately $89 million in abnormal accounts payable balances reported by the Supply Management business area. An abnormal account balance is one in which the recorded amount has been incorrectly classified. During fiscal year 2006, the LMP program office initiated systemic corrections to reduce the reported abnormal balances. Our analysis of the first quarter fiscal year 2007 financial statements noted that the reported balance had been reduced to approximately $81 million. The LMP program office was continuing to research the issue.

Our analysis of the DFAS 1307 reports found that abnormal balances in accounts payable have been reported since fiscal year 2004. LMP program officials stated that abnormal balances were primarily due to the migration of remaining obligations on open contracts from the legacy systems. Substantial effort was made in fiscal year 2006 to correct the abnormal accounts payable balances that were reported by LMP, which resulted in a reduction from $334 million at the end of fiscal year 2005 to $88 million at the end of fiscal year 2006. The Army’s efforts to clean up abnormal account balances continued in fiscal year 2007.

The Army has developed a specific plan to address the financial management problems recognized in these statements. For example, a Tobyhanna Army Depot on-site support team has been assembled to complete the data cleanup, validate that all system and procedural issues have been identified, document business processes and procedures, and train Tobyhanna users. However, until these problems are corrected, LMP will not be able to provide reliable and accurate financial information to the Congress or Army management.

Adequacy of LMP Testing Continues to Be a Concern

Our prior audits of LMP identified significant weaknesses with LMP’s efforts to effectively implement the requirements management and testing processes needed to reduce risks to acceptable levels. During our current audit, we found that the requirements management processes have improved—a critical first step—but we continue to have reservations about the adequacy of the testing. LMP program officials stated that these testing weaknesses are being addressed and expect that an improved testing process will be effectively implemented to reduce risks to acceptable levels.

LMP’s Requirements Management Practices Have Improved

During the current review, Army and contractor Computer Sciences Corporation (CSC) officials stated that they have acted upon our prior recommendations and taken steps to improve the program’s requirements management practices. In setting forth the revised requirements management process, LMP and CSC officials noted that the new process generally envisioned having several documents that outlined the requirements that needed to be addressed with each document containing specificity needed for the intended audience.

To ascertain if the Army’s stated corrective actions were being adhered to, we selected 10 requirements for detailed review. Our analysis found that the LMP documentation in support of the requirements management practices was generally in compliance with the stated process. More specifically, the requirements reviewed were generally consistent between the documents and appeared to contain the necessary specificity to reduce requirement-related defects to acceptable levels. Our observations were consistent with the conclusions reached by a subsequent review performed by the LMP verification and validation contractor, who examined the same documentation that we analyzed.

Concerns regarding LMP Testing Processes Remain

Our review of the stated testing processes found that many of the attributes associated with a disciplined testing process were present. However, based upon our review and analysis of the 10 selected corrective actions, we still have concerns that the LMP testing process may not be adequate since we identified specific weaknesses in the 10 corrective actions reviewed. System testing is a critical process utilized by disciplined organizations and improves an entity’s confidence that the system will satisfy the requirements of the end user and operate as intended. The stated testing process included the following: Documenting scenario testing. A scenario is a business process that typically consists of several events. For example, a scenario could be developed for small purchases, which would include events such as (1) initiating a purchase request, (2) approving the request, (3) obligating the necessary funds, (4) ordering the item, (5) receiving the item, and (6) paying for the item. One benefit of testing scenarios is that it helps to ensure that information is accurately passed from one process to another. For instance, the denial of a requisition prevents the purchase of the item.

Linking requirements to the test cases. Linking requirements and test cases helps ensure that either (1) all requirements are adequately tested or (2) the risks associated with not testing a requirement can be properly evaluated. Because it is not economically feasible to develop a testing program that can find every defect, it is critical that an approach be developed to ensure that testing resources are focused on the areas of highest risk. Accordingly, linking the requirements makes it easier to determine which combinations of items are not tested, thereby enabling the Army to evaluate the risks of not conducting those tests.

Regression testing. Regression testing is a process that helps ensure that changes made to the system have not adversely affected functionality that was working prior to a change being made to the system. In essence, it is a process that retests the entire system to ensure that the problem fixed did not have an adverse effect on other functions within the system.

While the processes described above are an improvement over what we have seen in the past, we continue to identify weaknesses in the testing process. For example, none of the test scripts provided adequate information that linked the items in the test script to a specific requirement. This linking is commonly referred to as traceability and is characteristic of a disciplined testing process. The test scripts reviewed contained headings for information on the scenario, key data requirements, expected results, actual results, and whether the test was considered successful. However, the actual test script failed to contain the level of specificity that clearly delineated how a specific requirement identified in the requirement document(s) and the associated requirement(s) in that document were being tested. As a result, it is virtually impossible to ascertain if the requirement was properly tested. For example, several requirements defined rules that were expected to be implemented. Without linking a given requirement to the tests designed to exercise that requirement, it was impossible to obtain reasonable assurance that (1) all the requirements had been tested and (2) the test provided adequate coverage for each requirement.

A review of the test script for one corrective action showed that the identified expected results were “output of report should show open accounts receivable items for customers in the range noted.” Our analysis of the documentation disclosed that the test scripts only tested to make sure that a report could be printed for (1) a specific customer, (2) a consecutive range of customers (e.g., customers 1 through 10), (3) federal and nonfederal customers (the exact condition tested was not specified since the user was only required to pick one of the conditions), and (4) receivables associated with a specified general ledger account. While this testing addressed the requirements that the report be printed using these conditions, the scripts did not provide evidence that other requirements were tested and the data presented were valid. For example, the report was expected to put receivables into 11 different age categories (for example 0 to 30 days, 31 to 60 days, etc.); however, no evidence was provided to show that the balances shown on these reports (1) were consistent with known results and (2) the accounts selected provided adequate coverage of the ranges specified. The latter is especially important since two of the ranges overlap. LMP and CSC officials agreed that the documentation did not always provide the level of detail that was needed to (1) document which requirements contained in applicable requirement documents were being tested and (2) ensure that all of the specific tests that were required to provide reasonable assurance that the application worked as expected were completed.

Furthermore, we were informed that the system testing was conducted by the developers and subject matter experts. Using developers and subject matter experts, and depending on those individuals’ comprehensive body of knowledge as the foundation for a testing effort, carries significant risks since this is not a recognized best practice. A basic testing principle is that a developer should not test his or her own work, nor should a development organization test its own work. These testing principles are based on the concept that testing is the process of executing a program with the intent of finding errors. Testing is normally considered a “destructive” process while the development activities are normally considered a “constructive” progress. It is very difficult for a developer to change the perspective required to successfully develop a program into the perspective necessary to adequately test a process. This does not mean that the developers are not involved in the testing process since a best practice is to have them responsible for testing the code they develop to ensure that it is ready for the next stage of testing.

Further, as noted in our May 2004 report, our analysis of LMP’s December 2003 and January 2004 project status reports identified numerous instances in which the Army continued to experience problems with the accuracy of data related to budgeting; workload planning and forecasting and depot maintenance operations; and accounting records such as customer orders, purchase orders and requisitions, obligations, and disbursements. DFAS and Army officials acknowledged that these problems were attributable to relying on subject matter experts to develop tests for their respective functional areas, and not performing end-to-end testing across the various functional areas.

After discussing our concerns with the LMP program office and CSC officials in February 2007, the LMP program office requested the verification and validation (V&V) contractor to perform an assessment of the LMP testing process. According to information provided by the LMP program office and the V&V contractor, in March 2007, the review found that the LMP testing process was adequately defined in the planning documentation and noted that the LMP testing philosophy was governed by the concepts that (1) the change should operate as intended, (2) the change should not be harmful to the existing functional environment, (3) testing should be practical, and (4) the testing efforts should be cost efficient. The V&V contractor analysis of the LMP testing process also found that the testing documentation—which was the same documentation that we analyzed did not support a clear understanding of the (1) requirement being tested and (2) tests used to determine whether a requirement was adequately implemented. According to the V&V contractor, these key areas need to be addressed. The V&V contractor assessment substantiates our analysis of the LMP testing process.

Improved Testing Process Planned

LMP program office officials and the V&V contractor agreed that the testing process currently being utilized could be improved. They also noted that the LMP program had begun implementing an initiative to improve the testing program that should address the weaknesses we identified. The following outlines two key components of LMP’s improved testing efforts.

Establishment of an independent test group. This is a critical step and, if effectively implemented, can serve as a strong foundation for building an effective testing process. An independent testing group is a best practice.

Development and implementation of an automated testing program. CSC is developing an automated testing process which was expected to reduce the testing burden and eliminate user errors in the testing process. For example, the user may not detect that the application did not provide the expected result or record an error when the application did operate as expected. These are commonly referred to as testing defects and are to be expected when manual testing activities are conducted.

LMP program officials are of the opinion that these two actions will help address the problems of adequately linking the requirements to the testing and documenting the testing that is actually performed. These actions, if effectively implemented, are a significant step forward and can help reduce the risks associated with testing to acceptable levels. However, the effective implementation will have to overcome a number of past problems, including ensuring that the tests provide adequate coverage. While developing automated tests can improve the reliability of the testing efforts and help facilitate an effective regression testing program, the key is whether the automated tests are testing the correct items. In our discussions with CSC, we were informed that the existing test scripts were the basis for developing the new automated test. Whether this approach will provide an effective testing process is yet to be seen. As we noted above, the test scripts that will be used as the basis for developing the automated test scripts do not provide the necessary link to the requirements that are being tested or provide the detailed information necessary to write the detailed test scripts called for in the automated process.

Appendix V: GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

In addition to the above contacts, the following individuals made key contributions to this report: J. Christopher Martin, Senior-Level Technologist; Darby Smith, Assistant Director; Evelyn Logue, Assistant Director; F. Abe Dymond, Assistant General Counsel; Beatrice Alff; Sunny Chang; Harold Brumm, Jr.; Francine DelVecchio; K. Eric Essig; Jason Kelly; Jason Kirwan; Alyson Mahan; and Debra Rucker.