Defense IRM: Critical Risks Facing New Materiel Management Strategy
Released: 1996-09-06T00:00:00

Letter

Objectives, Scope, and Methodology

During the course of our review Defense changed its strategy for materiel management systems. We therefore refocused our review to include evaluating the risks associated with the new migration strategy and the extent to which it will facilitate improvement of DOD’s materiel management operations. We reviewed DOD’s directives, instructions, and guidance for developing and implementing systems under the CIM initiatives, as these projects relate to the materiel management business area. We interviewed officials responsible for the old and new materiel management strategies as well as those in charge of developmental testing and those who participated in the early deployment of some applications. We also analyzed system design documents, program assessments, acquisition methodologies, and strategies. Our audit was performed from January through May 1996 in accordance with generally accepted government auditing standards. We performed our work primarily at the offices of the Deputy Under Secretary of Defense for Logistics (DUSD(L)) in Washington D.C.; the Joint Logistics Systems Center (JLSC) at Wright-Patterson Air Force Base, Ohio; and the MMSS prototype site at the Marine Corps Logistics Base, Albany, Georgia. Appendix I details our scope and methodology. The Deputy Under Secretary of Defense for Logistics provided written comments on a draft of this report. These comments are discussed at the end of this report and presented, along with our evaluation, in appendix II.

Background

Materiel management involves determining the type and amount of consumable and reparable items needed for daily military operations, such as ammunition, fuel, paint, and spare parts; purchasing these materials from private vendors or manufacturing agencies within DOD; and tracking materials from their purchase to end use. Materiel management business operations incorporate four major business activities—asset management, requirements determination, supply, and technical data support. Annually, the Department spends about $19 billion on materiel management operations, and it currently manages a reported 6.3 million inventory items valued at a reported $73.6 billion.

Materiel Management Business Strategies Are Changing

DOD’s worldwide logistics operation includes about 1,400 warehouses at 27 distribution depots and other locations that provide supplies such as electronics, construction, and industrial items to the military services. The military services use large amounts of these items to maintain and repair weapon systems and other equipment. For example, the three services operate a total of about 25 facilities that perform detailed, time-consuming maintenance of major weapon system components such as radars, navigation aids, and various types of communication equipment. The services also store supplies (known as retail inventory) in warehouses at or near these maintenance facilities. From these warehouses, supplies are distributed to the mechanics or end-users when needed. The mechanics also hold some of these same items in nearby storage bins.

We have previously reported that DOD’s large inventory levels reflect the management practice of buying and storing supplies at both wholesale and retail locations to ensure that they are available to customers—sometimes years in advance of when actually needed. DOD often stores inventories in as many as four different layers between suppliers and end-users. Storing inventory at many different locations results in inventory that turns over slowly, which produces large amounts of old, obsolete, and excess items. DOD’s multilayered supply system also increases the amount of supply on hand and drives up the cost of holding inventory. As discussed in previous reports in 1993 and 1994, this is a philosophy that private firms have moved away from in an attempt to lower the cost of doing business, provide better service, and remain competitive.

For example, during the past decade, the private sector instituted a logistics management philosophy that provides a sharp contrast to DOD’s methods of managing and distributing inventories. Some private sector companies have avoided inventory management problems by using modern, “just-in-time” business practices that shift responsibilities for storing and managing inventory to suppliers. In fact, companies that are using the most aggressive practices no longer store inventory in intermediate locations at all; their suppliers deliver inventory to them only when needed. Some companies have achieved large savings for certain items by standardizing items being used, eliminating bulk storage locations and, most importantly, relying on prime vendors to deliver small quantities when and where needed.

A major characteristic of these new logistics practices is the way the companies buy supplies. Companies have reduced the number of suppliers they use by establishing long-term agreements with only a few key suppliers. Typically, suppliers are contracted to provide a company’s supplies for a particular commodity. Thus, most of the management responsibilities are shifted from the company to the suppliers. The suppliers take on these responsibilities because they are promised a long-term relationship with the company. Other steps companies have taken to change their inventory management practices include using direct delivery programs, primarily through the use of a prime vendor . By using direct delivery programs, companies bypass the need for intermediate storage and handling locations. Once the end-users order supplies, the suppliers deliver the items directly to the user’s facility close to the time when the items are needed. Also, to facilitate communication channels with the suppliers, electronic ordering systems and the use of bar coding are often used to eliminate paperwork and speed up the ordering process.

DOD is beginning to move away from its current multilayered inventory management philosophy by developing initiatives, similar to those adopted in the private sector, which involve some combination of long-term contracting agreements, direct delivery of items from suppliers to the services, and electronic data interchange for streamlining the ordering process. The use of these initiatives is allowing DOD to (1) decrease procurement lead times, (2) increase accuracy in forecasting future item demands, (3) reduce paperwork, and (4) reduce inventory levels. While DOD has used these commercial practices, the initiatives generally have been limited in scope and represent only a small portion of its overall operations.

Materiel Management Systems Intended to Improve Various Business Processes

Defense wants to develop and deploy materiel management systems to improve business operations and processes nationally for DOD materiel management and reduce the costs associated with inventory, personnel, and inefficient systems. Historically, while the services and DLA have had similar logistics activities, they employed widely different processes and supporting information systems. Currently, Defense relies on over a reported 500 legacy systems to carry out wholesale logistics operations. As these systems become fragmented, outdated, and inefficient, they require billions of dollars in maintenance costs. According to Defense, because today’s materiel managers do not have access to timely, accurate, and reliable logistics information, they increasingly make unnecessary requisitions, which, in turn, result in excess inventory and waste. In addition, according to Defense, fragmented systems and the lack of current technology have severely affected the ability to achieve greater asset visibility; quickly adjust requirements for materiel supply items to support military operations; standardize planning, requisitioning, and inventory control; and provide greater support with fewer resources.

By embarking in 1992 on a strategy to develop the materiel management standard system (MMSS), Defense sought to replace hundreds of service-unique legacy systems being used to acquire, manage, move, and maintain inventory items with nine standard systems. These are the (1) Central Secondary Item Stratification System, (2) Configuration Management Information System, (3) Deficiency Reporting System, (4) Initial Requirements Determination/Readiness Based Sparing System, (5) Maintenance Planning and Execution System, (6) Production Definition Support System, (7) Provisioning Cataloging Technical Support System, (8) Requirements Computation System, and (9) Stock Control System. The specific functions of each of these nine systems are described in appendix III. Generally, these systems are intended to improve business operations in the following ways:

Asset management—provide greater asset visibility from the time of purchase to use and the capability to track and monitor product quality using automated deficiency reports during the wholesale process.

Requirements determinations—better define initial and repair requirements for supply items based on readiness scenarios and automate the computation of repair schedules and budgets.

Supply and technical data—automate paper copy guidebooks, procedures, and regulations needed to catalog new inventory items and provide managers with greater configuration control of inventory items.

Defense referred to this program as the “Big Bang” strategy because it involved installing the entire suite of applications at each of the 17 inventory control points, rather than deploying each application as it was developed. The new systems were to be integrated so that the services and DLA could communicate and exchange data with each other and across business activities outside materiel management, such as finance, procurement, personnel, and logistics. JLSC planned to field MMSS across all 17 inventory control points and have the system be fully implemented in 7 to 8 years. JLSC expected this standard system to save billions of dollars in logistics costs by consolidating and streamlining management operations, improving the responsiveness, accuracy, and timeliness of data, and eliminating the cost of maintaining some information systems that support the same business processes.

Problems in the Materiel Management Migration Strategy

From 1992 to late 1995, Defense spent about $714 million developing standard systems with minimal results. During that time, there were dramatic changes in the goals and expectations for the program and only one application was partially deployed. Because of changes in objectives and scheduling and problems in development, prospects for achieving the original objective of implementing a standard suite of integrated materiel management systems appeared dim. At the same time, the services and DLA were asking for quicker system deployments.

As table 1 shows Defense began the migration strategy in 1992 with the intent of implementing an integrated MMSS system in 7 to 8 years. But only a year later, it decided to implement the system in 3 years. About 2 years after that, the program was completely rebaselined because of funding cuts, cost overruns, schedule slippages, and poor contractor performance. During the same time period, the objectives of the program changed: at the start of the program, business process improvements were to be identified while systems were under development; in 1993, improvements were to be identified after implementation. Taken together, these changes raised serious concerns about MMSS among the services and DLA.

For the one application that was deployed (the Stock Control System), the development and scheduling problems were particularly evident. As appendix V further details, JLSC shifted system testing onto the users in order to meet milestone dates. One official told JLSC that its scheduling of testing actually extended time frames and resulted in a loss of confidence from users. This, coupled with other SCS problems related to resolving deficiencies discovered during user testing, poor training, and inadequate system documentation, prevented the application from providing the benefits originally anticipated.

Because of the development and scheduling problems, the services and DLA reported serious reservations about implementing several of the new systems because they believed that some of their existing legacy systems were better than the planned standard systems. They concluded that in some cases, the new systems, such as the Stock Control System and the Initial Requirements Determination/Readiness Based Sparing system, either would not meet their operational requirements or lacked the necessary functionality to allow them to shut down existing legacy systems as planned. Nevertheless, the services and DLA claimed that some of their legacy systems were quickly deteriorating and that they could not fund necessary upgrades. Therefore, they demanded deployment of the new systems as quickly as possible based on their individual service needs.

In April 1995, a Defense evaluation team, comprised of representatives from the Office of the Secretary of Defense, JLSC, DLA, the military services, and independent contractors, reviewed selected DOD inventory control point processes and concluded that the migratory systems approach to standardizing and upgrading materiel management automated data processing systems in DOD is not workable. The team recommended that JLSC discontinue its current efforts to develop MMSS and advised the Secretary of Defense to redirect JLSC, the services, and DLA toward a long-term effort to develop a unified automated data processing supply system using an independent contractor to design, develop, and prototype the system.

Nevertheless, in December 1995, because of pressure from the services and DLA and the problems they were experiencing with the MMSS migration strategy, DOD dramatically changed the MMSS scope and implementation approach. Program officials believed that if systems were not deployed quickly, the entire materiel management system program would be vulnerable to additional funding cuts, thus jeopardizing the entire program and risking total failure. The commander of JLSC stated that if customers did not see immediate results, his organization would be in danger of “going out of business.” In order to accelerate deployments under the revised approach, JLSC no longer plans to deploy a standard materiel management system. Instead, it will deploy applications incrementally as they are developed.

New Materiel Management Strategy Faces Serious Flaws

In December 1995, Defense embarked on an accelerated deployment strategy of the nine applications that make up MMSS in order to meet services and DLA priorities and to realize operational benefits sooner than originally planned. Under this strategy, JLSC will no longer deploy an integrated suite of standard MMSS systems. Rather, it now plans to individually deploy each of the nine system applications as they are developed at selected sites from fiscal year 1996 through fiscal year 1999. The services and DLA will choose which applications they want, when and where they will be deployed and, as a result, some inventory control points may never receive new systems. Deployment will be constrained by available funding. JLSC and DUSD(L) refer to the new strategy as “deploy or die” since program officials believed that unless these systems were deployed quickly, the entire materiel management system program would be vulnerable to additional funding cuts, thus placing the program in serious jeopardy of total failure. The current deployment schedule of these systems is provided in appendix IV. Table 2 reflects the differences in the two strategies.

In turning to the new materiel management systems strategy, Defense is now intent on delivering new applications to customers as soon as possible. But this haste puts the new materiel management systems development at higher risk than the previous one. As the following sections discuss, Defense will begin deploying new systems before it clearly defines its approach, ensures adequate oversight, and plans for economic and technical risks. Defense will also begin deployments without considering the effects of major upcoming changes to materiel management operations. In addition, Defense will be deploying all applications before critical necessary testing is complete. We believe that these steps are all critical to ensuring that Defense gets the most from each dollar it invests in materiel management systems. If Defense neglects to address them, it will likely incur substantial additional costs associated with maintaining legacy systems, interfacing them with the new systems, funding the rework to correct problems surfacing after deployment, and adapting its approach to expected dramatic changes in operations and systems.

New Strategy Still Lacks Direction and Justification

Under the new strategy, services will be keeping their legacy systems longer than anticipated and many will not be shut down. This is a major departure from Defense’s previous goal of eliminating hundreds of redundant legacy systems and varied business processes in order to move to standard integrated systems and processes. Yet for such a significant change in direction, Defense first did not conduct assessments that would ensure that the strategy would be cost-effective and beneficial. It also did not incorporate into the strategy plans to consolidate and privatize operations and other alternatives being considered to enhance existing systems. In addition, the change was not justified within the Department’s own oversight process, nor were documents critical to defining the programs objectives, costs, goals, and risk mitigation strategies prepared.

Economic and Risk Assessments Not Completed

JLSC is proceeding with the new strategy without first conducting critical economic and risk assessments that would estimate project costs, benefits, and risks and evaluate system choices based on these analyses. Without these assessments, DOD has no assurance that the best or most cost-effective systems are selected for migration nor can it plan actions designed to avoid or lessen the potential for project delay, overspending, or failure. These evaluations are particularly important at this time because, according to program officials, the estimated MMSS lifecycle costs and expected benefits in the July 1995 economic analysis do not reflect the most recent strategy change. These evaluations would also help DOD in planning to mitigate some of the additional costs associated with maintaining legacy systems that will be incurred as a result of the new strategy.

The benefits of DOD using these analyses, for example, could have been realized in choosing the systems to deploy first in the new strategy. As table 3 shows, three of the four systems (the Configuration Management Information System (CMIS), the Product Definition Support System (PDSS), and the Deficiency Reporting System (DRS)) scheduled for deployment in fiscal year 1996 have very low projected benefits. The benefits listed in table 3 were taken from the July 1995 economic analysis.

According to program officials, IRD/RBS, CMIS, PDSS, and DRS were chosen for deployment first in the new strategy because they are further along in development. However, we believe that had Defense analyzed costs, benefits, and risks associated with all selections, it would have had to seriously consider whether the benefits associated with RCS, SCS and MP&E made it imperative to concentrate on their development first.

Additionally, by not analyzing and anticipating costs and risks associated with the new strategy, JLSC officials told us that they do not know how much it will cost to maintain the legacy systems that will remain under the new strategy and what it will cost to interface the new applications with the legacy systems. Because these systems will not be deployed as an integrated suite at all inventory control points, the services and DLA will have to operate many of their legacy systems for a substantially longer period of time. In turn, the large number (and complexity) of interface designs is likely to increase development and deployment costs significantly and delay implementation schedules. In November 1995, JLSC reported that the identification and development of more than 3,000 interfaces to existing legacy systems in support of multiple deployments is the prime technical risk facing the program.

Finally, economic and risk analyses would reveal potential conflicts between available funding and planned scheduling of deployments. According to the MMSS program manager, the number of actual deployments for both fiscal years 1996 and 1997 will be contingent on available funding. In April 1996, the manager reported that the revised fiscal year 1997 schedule is too ambitious given the funding projected to be available.

Impact of Potential Changes to Materiel Management Program and Systems Not Assessed

In addition to not assessing economic and technical risks, Defense has not assessed the impact that a number of potential changes under consideration for material management operations and systems could have on the program. These changes, and their implications, include the following.

Recent DOD initiatives focus on privatizing materiel management operations or consolidating inventory control points. For example, the Commission on Roles and Missions of the Armed Forces recommended, in May 1995, that Defense outsource materiel management activities relating to cataloging, inventory management, and warehousing. If outsourcing occurs, Defense may end up spending millions of dollars on systems for functions that are later outsourced or on inventory control points that are later consolidated.

As discussed in the background section of this report, DOD is beginning to move away from its multilayered inventory management philosophy by embarking on initiatives similar to those adopted in the private sector, which involve some combination of long-term contract agreements, direct delivery of items from suppliers to the services, and electronic data interchange for streamlining the ordering process. These initiatives have not been a part of the system migration strategy; however, as they are expanded, they will significantly impact the processes the systems support.

According to program officials, Defense is considering implementing a “data-focused approach” to materiel management systems starting in fiscal year 1998, which would enhance interoperability and logistics modernization efforts through the use of “middleware” software.

Middleware permits an application to see the data stored in other applications as if they were in a single, logical data repository. In doing so, it precludes the need to radically redesign the legacy systems and implement data standardization. If pursued, the middleware alternative could extend deployment schedules and drive up maintenance costs for existing systems. It also will not result in the consolidation or elimination of legacy systems.

New Strategy Lacks Adequate Oversight

Through its own oversight process for major information system projects,DOD has established a basis by which decisionmakers—who make up the Major Automated Information System Review Council (MAISRC)—can ensure that sound business practices are followed for major information technology system investments. Under MAISRC guidelines, a project should be reviewed and approved at each of five decision milestones before substantial funds are obligated. An important aspect of the review process is that it lays the groundwork for ensuring that major initiatives are clearly defined, user requirements will be met, and sound acquisition and testing strategies are in place. Documents that the Council reviews—such as the mission needs statement and the acquisition and test plans—justify the program’s existence and define economic and technical risks.

In January 1996, the Deputy Assistant Secretary of Defense for Command, Control, Communications, and Intelligence placed the MMSS project under the MAISRC oversight review process for the first time. However, the decision to make such a drastic change for the materiel management strategy in the first place was never presented to MAISRC. As a result, key Defense decisionmakers did not have a chance to evaluate the program in order to decide whether to continue the current program, make minor changes, redirect, or terminate the program before it began.

When JLSC entered the review process in April 1996 (before a working-level team versus the high-level Council), it requested approval to continue incrementally fielding individual applications, but did not have strategic plans and other required documentation that the Council could use to reach a decision. Based on the working-level team’s review, the Council withheld authorization to proceed until certain documents were submitted and approved.

In its May 13, 1996, decision memorandum, the Council directed the MMSS program manager to prepare the basic documents required for MAISRC review over the next 180 days. These include a mission need statement, which sets the goals of the program and defines projected capabilities and needs in broad operational terms; an acquisition strategy to guide the entire acquisition process throughout the system development life cycle and serve as the framework for planning, directing, and managing the program; an operational requirements document to document user objectives and minimum acceptable requirements for systems and to become the basis for operational performance criteria and testing; and a plan for preparing an economic analysis.

Primarily to enable DOD to meet contractual obligations, the Council plans to hold another working-level session to evaluate and approve the acquisition and test and evaluation strategies before the end of fiscal year 1996. According to the Council, if these strategies are approved, JLSC will be authorized to proceed with deployment. However, when this decision is made, the other critical documents, such as the plan for preparing an economic analysis and the operational requirements document, will not be available for the Council’s review. Therefore, JLSC may be authorized to proceed with deployment before key decisionmakers at Defense have reviewed a cost-benefit analysis, funding profile, and other important information that would shed light on the risks and costs associated with the new strategy. We believe that the risks associated with the new strategy and problems experienced with the old strategy warrant a full, high-level Council review of all MAISRC-related documents before deployments proceed.

Premature Deployment Poses Additional Risks for the New Strategy

The new deployment schedule for materiel management systems does not accommodate the time required for testing the new systems. In fact, all systems scheduled to be deployed in fiscal year 1996 and 1997 will only have met a minimal testing level, that is, developmental testing by the contractor. As a result, the risk is greatly increased that Defense will experience problems associated with shifting testing to system users and curtailing the levels of testing normally done. This has already been the case with the one application that was deployed at the MMSS prototype site—the Stock Control System.

The four applications scheduled for deployment in fiscal year 1996 are still in the latter stages of development and have yet to complete required developmental and integration testing. In an attempt to meet the revised deployment schedule, JLSC is shifting developmental testing responsibilities from the development contractors to system users where the application is to be initially deployed. In some instances, JLSC will also forgo system qualification and integration testing altogether—which are critical to determining whether the applications will work as planned. According to program officials, the intent is to demonstrate interoperability of the database concept in a customer environment and to obtain customer feedback more quickly.

However, not successfully completing these tests prior to deployment increases the risk that software problems will go undetected until the later phases of the system lifecycle. According to the Test Director, it is much more expensive and time consuming to correct errors once the applications are operational. Program officials also acknowledge that problems detected at the user sites will be more expensive to fix and could offset or exceed up-front investment savings.

In the absence of an approved test and evaluation master plan, JLSC is negotiating memorandums of agreement (MOA) with respective users to define test conditions, assumptions, and responsibilities. Our review of the one approved MOA covering the Navy’s deployment of the CMIS application in fiscal year 1996 raises concerns about the test program. For example, the MOA shows that the application will be an “as is” version which has not been accepted by JLSC and will not be interfaced with any legacy systems during the test period. Although the Navy is required to prepare a lesson-learned report after it completes testing, no formal test plans or test reports are required under the testing process. Without these key documents, JLSC has little assurance that all necessary tests will be completed and that problems encountered with the system are thoroughly documented.

Stock Control System Illustrates Problems Associated With Early Deployment and User Testing

The early deployment of the Stock Control System application under the previous migration strategy illustrates problems associated with shifting testing to users. As discussed in appendix V, the first MMSS prototype site, the Marine Corps Logistics Base, Albany, Georgia, was activated on May 1, 1995. At that time, JLSC deployed an early version of the SCS application, which had only about 50-percent functionality of the asset management software. The Marine Corps expected that this release would resolve its core asset management system deficiencies and demonstrate operational functionality and practical business process improvements.

As of May 1996, according to system users, the system has failed to provide substantial improvement over the legacy applications being used at Albany. Because the project contractor delivered the application basically untested, with very limited functionality and inadequate user documentation, the Marine Corps has had to perform extensive and costly amounts of rework, debugging, and on-site testing. The Marine Corps has initiated 65 changes to correct major functional deficiencies; however, only 42 have been funded to date. Until the remaining deficiencies are corrected, SCS will not be able to meet all of the Marine Corps’ requirements.

Because of continuous problems in defining requirements and schedule slippages, JLSC stopped all development work on SCS in December 1995. At the time, SCS development was about 55 percent complete. In May 1996, the Logistics Management Institute (a contractor hired by JLSC to provide technical support) recommended that JLSC terminate SCS development and maintain legacy asset management systems rather than invest an additional 2 years and as much as $100 million to correct the problems. JLSC still plans to deploy SCS; however, it will limit additional functional enhancements and will deploy the system only to the Marine Corps and the Air Force.

Conclusions

To provide service on demand, Defense made a major change in its materiel management migration system policy. In doing so, it is clearly on a course to accelerate system deployments before critical steps are taken that would help ensure that good business decisions are made and that risks are minimized. As a result, Defense may likely deploy systems that will not be significantly better than the hundreds of legacy systems already in place, and it could waste millions of dollars resolving problems that result from the lack of developing and implementing a clear and cohesive strategy. Before proceeding with any new strategy, it is imperative that Defense take the necessary steps to fully define its approach, plan for risks, ensure adequate oversight and complete testing of the new systems.

Recommendations

We recommend that the Secretary of Defense stop the materiel management system development and deployment until (1) DUSD(L) completes an economic analysis and a comprehensive implementation plan, including actions to be taken, schedules, milestones, and performance measures, and a technical risk plan and (2) the full MAISRC reviews and approves these plans.

Agency Comments and Our Evaluation

The Department of Defense provided written comments on a draft of this report. The Deputy Under Secretary of Defense for Logistics generally agreed with our findings but disagreed with our recommendations. Defense’s specific comments are summarized below and presented, along with our rebuttals, in appendix II.

In commenting on a draft of this report, Defense effectively acknowledged that the first materiel management strategy failed and it agreed on the need to mitigate risks confronting the new strategy. However, it did not agree with our recommendations to stop materiel management system development and deployment until it takes necessary steps to define its approach, plan for risks, and ensure adequate oversight. Instead, Defense believes it is addressing the concerns expressed in our report under the logistics business systems strategy it is currently developing. The latest strategy focuses on creating a common operating environment for logistics. According to Defense, under the common operating environment, guidelines and standards specifying how to reuse existing software and build new software will facilitate system interoperability and allow for continually evolving computer capabilities.

We agree with the Department’s contention that its materiel management strategy has failed and commend it for pursuing alternative strategies, such as privatization and developing a common operating environment based on commercial off-the-shelf (COTS) systems. Such alternatives may well solve some of the past problems associated with materiel management systems. However, we disagree that the common operating environment strategy being developed will address our recommendations. Without first conducting required economic analyses, Defense has no assurance that the systems it is currently deploying, which were selected under the failed strategy, will fully support the new strategy or by themselves still be good investments. In addition, by not conducting these analyses, decisionmakers will lack necessary information to make sound, informed decisions for selecting the best among competing alternatives and understanding how upcoming major changes to materiel operations will impact their strategy and alternatives. These shortcomings led to the failure of the first strategy, and we believe that, unless they are addressed, Defense risks failing a second time.

Further, conducting economic and risk analyses and providing for adequate oversight over system development, is not only required by Defense’s own regulations but also by the recently enacted Information Technology Management Reform Act (ITMRA), which took effect August 8, 1996. The intent of this legislation is to prevent failures similar to the material management standard system strategy. Under ITMRA, DOD is required to design and implement a process for selecting information technology investments using criteria such as risk-adjusted return-on-investment and specific criteria for comparing and prioritizing alternative information system projects. If implemented properly as part of the new strategy, Defense can have a means for senior management to obtain timely information regarding progress in terms of costs, capability of the system to meet performance requirements, timeliness, and quality. Without implementing an effective investment process for the new strategy, Defense will continue to risk encountering unmanaged development risks, low-value or redundant information technology projects, and too much emphasis on maintaining old systems at the expense of using technology to redesign outmoded work processes.

We are sending copies of this report to the Senate Committee on Governmental Affairs and the House Committee on Government Reform and Oversight; the Secretaries of Defense, the Army, the Navy, the Air Force; the Director of the Defense Logistics Agency; the Director of the Office of Management and Budget; and other interested parties. Copies will be made available to others on request.

If you have any questions about this report, please call me at (202) 512-6240 or Carl M. Urie, Assistant Director, at (202) 512-6231. Major contributors to this report are listed in appendix VI.

Objectives, Scope, and Methodology

As mandated by the National Defense Authorization Act for Fiscal Year 1996 (Public Law 104-106), we reviewed the Department of Defense’s Materiel Management Standard System (MMSS). The original objectives of our review were to determine (1) the mission and the economic and technical basis for selecting MMSS as the migrating system and (2) the extent to which this strategy has or will improve DOD’s materiel management operations. To accomplish our objectives, we (1) interviewed program officials and contractors responsible for developing, implementing, and managing MMSS projects, (2) reviewed pertinent program and contractor documentation such as cost performance reports, metrics, quarterly reports on major automated information systems, economic analyses, implementation and migration plans, and the test and evaluation plan, (3) examined system design documents, program assessments, and acquisition methodologies and strategies to support the MMSS, and (4) interviewed senior Defense officials responsible for approving and directing the MMSS development and acquisition regarding their efforts to minimize MMSS development risks and improve materiel management operations.

However, shortly after we began the review, Defense stopped its strategy to develop a standard materiel management system and started making plans to separately deploy individual applications. Consequently, we refocused our review to include evaluating the risks associated with this new migration strategy and the extent to which it will facilitate improvements in materiel management operations. We interviewed DOD and program officials to determine the rationale behind the strategy change and the potential economic and technical risks threatening the successful implementation of the new strategy. We obtained and examined budgetary and cost data, reviewed project status reports, and pertinent program documents such as the revised deployment schedule, test procedures, and program decision papers. For applications scheduled to be deployed in fiscal year 1996, we compared the test procedures with deployment schedules to evaluate potential program risks of deploying software applications before successfully completing required testing. We were hampered in our attempt to assess the potential improvements to DOD’s materiel management operations because critical strategic documents such as the revised economic analysis, acquisition strategy, and mission need statement had not been completed. To determine if field locations had experienced problems resulting from insufficient testing, we also interviewed officials at the Marine Corps Logistics Base, Albany, Georgia who had participated in the early deployment of some MMSS applications.

Our work was performed from January 1996 through May 1996 in accordance with generally accepted government auditing standards. We performed our work primarily at the offices of the Deputy Under Secretary of Defense for Logistics, Washington D.C.; the Joint Logistics Systems Center, Wright-Patterson Air Force Base, Ohio; and the Marine Corps Logistics Base, Albany, Georgia.

Comments From the Department of Defense

GAO Comments

1. The wording in the paragraph cited has been modified but addresses the same issue. 2. Defense states that it is documenting a revised logistics business systems strategy at the same time that the first aspects of the strategy are being executed. The revised strategy is to be based on the new common operating environment (COE) approach for building interoperable systems, a collection of reusable software components, a software infrastructure for supporting mission area applications, and a set of guidelines and standards. The standards will specify how to reuse existing software and how to build new software to facilitate system interoperability. We are currently reviewing the COE. While it may address technical infrastructure problems associated with logistics business systems, we believe that Defense must still incorporate into the new strategy the essential ingredients that ensure sound decisionmaking as we recommended in our report: conducting required economic and technical risk analyses and providing adequate oversight for the systems currently being deployed. In doing so, Defense can ensure that the systems being deployed now will be compatible with systems or work processes developed under the common operating environment approach. Further, it can ensure that sound business decisions are being made as it finalizes the new approach.

Additionally, while the military services and Defense agencies have requested materiel management deployments, because the systems have not been fully tested, they may not be an improvement over existing legacy systems. In addition, in the near-term, these systems will require the services and agencies to continue to maintain their legacy systems. We believe that before these systems are deployed to the services and the agencies, Defense needs to ensure that testing is sufficient. This should help reduce the risks of software problems surfacing in later phases of the system lifecycle. 3. As stated in our report, the economic analysis completed in 1995 was not reflective of the new approach to materiel management systems. An analysis for the new strategy may well have identified the additional risks that have not been addressed, such as those associated with maintaining the legacy systems that will now remain under the new strategy. In addition, developing an economic analysis after deployments have begun will not give Defense decisionmakers an opportunity to ensure that good business decisions are being made before funds are committed. Both ITMRA and the Office of Management and Budget’s November 1995 guide for evaluating information technology investments call for such analyses prior to making information technology investments as it allows senior managers to examine trade-offs among competing proposals and to ensure that each project is cost-effective and beneficial. 4. We support Defense’s efforts to communicate its intentions for materiel management systems with management across the services and within logistics operations. Obtaining support of decisionmakers is critical to the success of the new strategy. However, because Defense has not complied with its own regulations for ensuring sound decisionmaking, these managers still do not have the information necessary for making informed decisions. Further, the Logistics Information Board, which Defense has established to review execution of the new strategy is no substitute for a full MAISRC review. For example, while the Logistics Information Board consists of members who participate in logistics business operations, MAISRC comprises high-ranking officials separate from logistics business who have a more independent perspective in reviewing the strategy. Further, while the Logistics Information Board can play an effective role in development and implementation of the strategy, MAISRC plays a vital role in developing and rigorously verifying the cost-benefit and alternative analyses that are fundamental to making investment decisions. 5. Defense contends that the systems it is developing will be flexible enough to support private contractors assuming responsibility for materiel management operations if privatization is pursued. This contention, however, presumes that contractors will want to use these systems rather than acquire their own systems and that these systems will support new work processes adopted by contractors. Further, because most of the migratory systems being deployed are based on an out-of-date system architectures, there is no assurance they will facilitate interoperability between the legacy and COTS environments, as Defense contends. 6. The testing strategy DOD describes is a common practice in the commercial world. However, it should be noted that this practice is intended to mitigate risk and reduce costs prior to full production of the system. In DOD’s case, the strategy is being employed during full production of the system. Additionally, under DOD’s approach, the services will be required in many cases to continue spending operational funds on their legacy systems to make up for the lack of full functionality in the fielded new systems. Further, Defense historically has encountered significant cost increases to software-intensive systems as a result of fielding them before they are adequately tested. As our report discusses, these problems were especially evident with the Stock Control System. In its comments, Defense did not dispute that the Marine Corps has had to make extensive and costly changes to the system chiefly because the application was delivered basically untested and with very limited functionality. Finally, we do not believe it is appropriate to deploy these systems until the testing strategy is approved by the full MAISRC. 7. We disagree with Defense’s contention that the costs to maintain legacy systems will remain the same. Since a bare bones approach to legacy maintenance has been sustained for the past several years, we believe that these systems will require more maintenance as they get older. Additionally, we believe the maintenance costs for legacy systems will increase since fewer systems will be terminated under the new strategy than anticipated under the original strategy. As discussed in our report, these remaining systems will also require costly interfaces with new systems. Further, we are not recommending that Defense delay moving forward with a new logistics business systems strategy. Rather, we are recommending that Defense delay continuing to implement pieces of an admittedly failed migration system strategy until it can be assured that the systems it wants to deploy are good investments. 8. We agree that the old strategy is not viable. However, we disagree that there are no alternatives other than the current strategy. Our report, in fact, discusses alternatives to the new strategy currently being considered by Defense, such as privatizing materiel management functions. Also, by conducting required economic analyses, Defense would be able to fully identify the available alternatives and consider their associated costs, benefits and risks. 9. Until Defense completes the documentation associated with its own oversight process—which includes a complete definition of the new strategy; an analysis of costs, benefits, and alternatives; and a test plan—its decisionmakers will not have assurance that they are choosing the best system solutions. Further, the contention that the full MAISRC review of major information system investments merely adds “another level of review” goes against Defense’s original intention in implementing this process: ensuring that the essential ingredients to making sound business decisions are incorporated into all major technology investment decisions, and that senior managers are making the final decisions and held accountable for them.

Description of Materiel Management Standard System Applications

Central Secondary Item Stratification (CSIS): Stratifies the requirements computed in the other systems across financial programs and is the basis for budgeting and funding allocations.

Configuration Management Information System (CMIS): Provides configuration identification, configuration status accounting, electronic change control, and configuration audits.

Deficiency Reporting System (DRS): Collects, processes, and stores quality deficiency and discrepancy data on weapon systems and equipment.

Initial Requirements Determination/Readiness Based Sparing (IRD/RBS): Computes initial spare requirements for new systems and computes requirements based on readiness scenarios.

Maintenance Planning & Execution (MP&E): Manages repair requirements and monitors the performance of maintenance facilities.

Product Definition Support System (PDSS): Creates and moves a complete requirements package from the requirements determination system to the contracting system.

Provisioning Cataloging Technical Support System (PCTSS): Supports the selection of items for new end items/weapon systems, obtains and maintains national stock numbers and associated data.

Requirements Computation System (RCS): Provides demand-based requirements computations for recoverable and consumable items.

Stock Control System (SCS): Provides asset visibility through requisition processing, receipt processing, and inventory processing.

New Strategy Deployments

Tables IV.1 and IV.2 below show the deployment schedule for fiscal years 1996 and 1997. Four system applications—the Configuration Management Information System, Deficiency Reporting System, Initial Requirements Determination/Readiness Based Sparing, and the Product Definition Support System—will be deployed in fiscal year 1996 at an estimated cost of $3.1 million at selected sites across three of the five services based primarily on need. These systems will be delivered to the user organizations by JLSC “as is,” that is, with limited functionality and system testing. According to the MMSS program manager, the number of actual deployments for both fiscal year 1996 and 1997 will be contingent on available funding. In April 1996, the materiel management program manager reported that the revised fiscal year 1997 schedule is likely too ambitious given the funding projected to be available.

Defense hired a contractor to conduct site surveys at each deployment site to determine the physical plant and architectural requirements, that is, communication, electrical, and computer hardware and software configurations needed to support applications. As of May 31, 1996, the contractor had completed 11 of the 21 required site surveys. According to JLSC officials, to meet their deployment schedule, some applications will be deployed in fiscal year 1996 even though the site surveys may not be done.

Deployment of the Stock Control System at the Marine Corps Logistics Base in Albany, Georgia

By fielding the Stock Control System (SCS), JLSC expected to achieve immediate benefits and to demonstrate major progress in support of DOD objectives. The benefits to the Marine Corps—the first service to receive the system—included replacing an outdated system and moving from a batch system to on-line processing. The monetary benefits were expected to exceed $56.7 million with an implementation cost of $27.2 million.MCLB-Albany was selected as the SCS production site in August 1993, and the system was declared operational in April 1995. However, SCS was not fielded with full functionality.

In addition to the challenges of learning a new system, users continue to experience problems with the system’s operability. They cited the following reasons for the difficulties encountered with the SCS:

The majority of users believed that the initial testing of SCS was inadequate. Since the system’s deployment, users have experienced problems that should have been detected during the system’s testing.

Most users reported not receiving timely training prior to the system’s deployment. Given that training was provided up to 8 months before system implementation, the majority agreed that the training was too early for them to retain the knowledge they needed to operate the system. An MCLB-Albany official stated that training was conducted so far in advance because the deployment date was officially scheduled to be 6 months earlier. Three of the users stated that training was ineffective because instructors were unfamiliar with Marine Corps processes or too general in their presentation. Recognizing the need for additional training, MCLB-Albany conducted a refresher class just prior to the implementation of SCS, which some of the users thought was beneficial.

Most of the users believe that the SCS system documentation is insufficient and several thought the manual was useless and ever changing. One inventory manager never even received a user’s manual. All of the users seek answers to their problems by consulting with designated SCS system analysts or other users.

All of the users stated that the SCS no longer allows them to perform certain job related tasks. In specific instances, some of the users reported going to a separate, manual source to complete these tasks. An MCLB-Albany official anticipates resolving this problem with later versions of system.

Most of the users have experienced problems in accessing SCS, which has been unavailable for periods of time ranging from a few minutes to several days. Given that users spend as much as 90 percent of their day in SCS, this problem could inhibit their ability to do their jobs. MCLB-Albany officials stated that the inaccessibility of SCS is often due to problems with their local area network rather than problems with the SCS.

An official with MCLB-Albany’s Defense Accounting Office reported that financial data is not able to pass directly from the SCS to the accounting system.

All nine inventory managers interviewed liked that SCS gave them immediate, on-line processing of information. The legacy system employed batch processing, making users wait until the day after they input information to receive the results.

Needing a success story, JLSC set unrealistic milestone dates and pushed the system through testing and onto the users. One official told JLSC that its scheduling of testing actually extended time frames and resulted in a loss of confidence from the users. This official and a lessons-learned report emphasized that the manner in which the system tests were conducted has exacerbated the problems with SCS. They cited that some problems were not identified in testing; the tests only addressed one area of the system at a time, as if in a vacuum; the tests were conducted on a different operating environment from the Marine Corps’; some areas of SCS were not tested; and insufficient testing at contractor facility led to additional on-site testing to correct problems.

Albany has devoted many resources to implement interim solutions to resolve interface and operating environment problems. Analysts have resorted to these measures because SCS has on-line processing capabilities, while the legacy systems use batch processing. Additionally, SCS uses a different operating environment from the legacy systems. Even with these interim solutions, MCLB-Albany cannot ensure that data is carried from one operating environment to the next. Additionally, because MCLB-Albany has SCS and legacy systems operating in tandem, it spends more time and money maintaining and reconciling the systems and to purify and convert data. Although MCLB-Albany officials acknowledge that a cost exists for all of these interim solutions, they were unable to determine the cost of addressing interface problems.

According to Program Managers, JLSC ceased to provide feedback to MCLB-Albany on SCS monthly activity reports. At the beginning of the effort, JLSC required status reports from MCLB-Albany, but they later directed Albany to discontinue these reports. Considering that SCS still has unresolved problems, JLSC’s instructions to cease the flow of progress reports is puzzling. MCLB-Albany officials later informed us that informal communication with JLSC has subsequently improved with the new direction of MMSS; however, they still receive no feedback on their progress reports. MCLB-Albany continues to generate the status reports for their own benefit.

Although the users continue to experience problems using SCS, program management at MCLB-Albany believes that SCS will be its asset management system into the future. MCLB-Albany is scheduled to receive an update to SCS in 1996, which includes an upgrade to a modernized language because the older version will no longer be supported by the commercial market. The new language is expected to alleviate incompatibility problems in the operating systems. The Marines consider this update a top priority. Although officials at MCLB-Albany continue to request the functions that they did not get when SCS was deployed, JLSC has not responded. These missing functions are now under initiatives of other DOD agencies.

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

Chicago/Dayton Field Office

Atlanta Regional Office

Carl L. Higginbotham, Senior Advisor Christopher T. Brannon, Senior Evaluator Valerie A. Paquette, Staff Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.