Defense Management: Selection of Depot Maintenance Standard System Not Based on Sufficient Analyses
Released: 1995-07-13T00:00:00

Letter

Scope and Methodology

We based our review on an assessment of DOD’s implementation of its own directives and instructions for new automated information systems or the selection and implementation of standard migratory systems under the CIM initiative, as these projects relate to the depot maintenance business area. These directives, referred to as Life-Cycle Management, contain the same steps and milestones as GAO’s own methodology for reviewing large automated information systems/projects.

Our audit was performed between April 1994 and March 1995 in accordance with generally accepted government auditing standards. We performed our work primarily at the offices of the Deputy Under Secretary of Defense for Logistics in Washington, D.C., and the Joint Logistics Systems Center, Wright-Patterson Air Force Base, Ohio. Appendix I details our scope and methodology. The Deputy Under Secretary of Defense for Logistics provided written comments on a draft of this report. These comments are discussed at the end of this report and presented, along with our evaluation, in appendix II.

Significant Problems in DOD Depot Maintenance

Each year DOD spends about $13 billion to manufacture, overhaul, and repair more than 2 million items at its more than 27 maintenance depots. The depots have primary responsibility for the maintenance, overhaul, and repair of large items, such as tanks, ships, and airplanes, and small and intricate ones, such as communications and electronic components. Depot maintenance consists of three basic business processes: project management (maintenance of major-end items, such as airplanes, ships, and tanks), reparables management (maintenance of items, such as engines, transmissions, and radios), and specialized support (various individual functions, such as tracking hazardous materials, tools, and test samples).

For years, GAO and DOD have reported on major problems facing the depot maintenance area, principally that DOD’s depot management structure has not resulted in substantial competition, interservicing, or reduction of excess capacity and duplication of effort. For example: In 1983, GAO testified that DOD had not moved quickly to eliminate duplicate capability and excess capacity within depot maintenance because of (1) parochial interests, (2) lack of central authority, and (3) absence of DOD-wide planning. In 1993, the Joint Chiefs of Staff reported that closing a significant number of depots was needed to reduce excess capacity and that significant savings could come from consolidating depot workload across service boundaries. In May 1993, we testified, that the Joint Chiefs of Staff identified 25 to 50 percent more depot capacity than will be needed in the future and that this problem had been exacerbated by (1) the end of the cold war, (2) reduction of defense systems and equipment, (3) retirement of less reliable and more maintenance-intensive systems, and (4) the private sector’s push for a greater share of the depot maintenance workload. In 1993, we reported that internal controls at Army depots did not adequately safeguard millions of dollars of weapons and equipment during the maintenance processes. Specifically, we reported that poor storage practices increased maintenance costs, depot inventory records were not accurate, and the Army’s depot cost accounting system did not capture actual job costs. In 1995, DOD reported to the Congress that its financial systems and databases were inadequate to provide the type of information to determine the cost-effectiveness of greater public-private competition for providing depot maintenance services.

Over the last several years, DOD has taken a number of actions to correct these problems. One of these actions is its Corporate Information Management initiative, which was established to prepare DOD for future budget reductions and post-cold war readiness requirements through (1) streamlining business processes, (2) integrating essential data systems, and (3) eliminating duplicate or redundant information systems across the Department. The DMSS project was undertaken as part of this effort.

Strategy for Addressing Depot Maintenance Problems

To improve its depot maintenance operations and manage its resources more efficiently, the Principal Staff Assistant (PSA) for logistics, in November 1991, established the Joint Logistics System Center (JLSC). JLSC is to facilitate the improvement of depot maintenance processes by identifying business process improvements and managing the development and deployment of a standard depot maintenance system to replace service-unique systems currently used.

In January 1994, JLSC prepared an economic analysis recommending development and deployment of a standard depot maintenance information system—called the Depot Maintenance Resource Planning (DMRP) system, which consisted of four system components. Currently, the standard information system consists of eight components and is called DMSS. The following table identifies the core depot maintenance business processes and the eight system components selected to support them.

By implementing DMRP, DOD expected a return on its investment of $2.6 billion through business process improvements and savings derived from replacing more than 60 service-unique automated depot maintenance information systems. Specifically, these benefits are to be derived from (1) reduced direct and indirect labor costs, (2) reduced direct and indirect material costs, (3) reduced costs associated with shutting down old information technology (legacy) systems, (4) shorter cycle time for certain types of maintenance and inspections, and (5) automation of many currently paper-based work processes.

Our concerns with this strategy are twofold. First, DOD did not base its decision to develop and deploy DMSS on convincing analyses of expected system development and deployment costs or detailed assessments of DMSS’s economic and technical risks. Further, Defense did not obtain the independent reviews by the MAISRC and approvals by the MDA of the project’s milestones, which are designed to ensure the decision was consistent with sound business practice. Second, we believe that DOD needs to consider reengineering entire processes before implementing system changes if it is to achieve the dramatic reductions in operational support costs called for by CIM.

DUSD(L) Did Not Use Sufficient Analyses in Selecting DMSS

In selecting DMSS as DOD’s initial step toward improving defense maintenance depot operations, the Deputy Under Secretary of Defense for Logistics (DUSD(L)) did not base its decision on sufficient analyses of expected system development and deployment costs or detailed assessments of DMSS’s economic and technical risks. Further, DUSD(L) did not obtain independent milestone reviews and approvals which are designed to ensure (1) decisions are consistent with sound business principles and (2) risks inherent in large information systems projects are adequately managed. Thus, even the marginal improvements Defense expects from DMSS may never be achieved.

Defense directives require that decisions to develop and deploy information systems be based on convincing, well-supported estimates of project costs, benefits, and risks. These directives establish a disciplined process for selecting the best projects based on comparisons of competing alternatives.

Defense’s principal means for comparing various alternatives is a functional economic analysis. For each alternative, it identifies resource, schedule, and other critical project characteristics and presents estimates of the costs, benefits, and risks. The Office of the Assistant Secretary of Defense for Program Analysis and Evaluation is required to validate these estimates to help ensure that the economic analysis presents compelling quantitative data for each of the alternatives being evaluated. Once an alternative is chosen, the analysis becomes the basis for project approval. Any significant change in the project’s expected costs, benefits, or risks requires that the project selection and direction be reevaluated.

Also, DOD directives established the Major Automated Information System Review Council (MAISRC) to provide oversight of individual major information system projects. At each development milestone for proposed information system projects, MAISRC reviews these projects to determine if they are consistent with DOD policies and directives. MAISRC then recommends continuation, redirection, or termination of each project to the project’s Milestone Decision Authority (MDA). DOD’s current policy is to ensure that funds are not obligated for any automated information system until the MAISRC milestone review and MDA approval are complete.

In January 1994, following the logistics CIM migration strategy, the JLSC evaluated three alternatives for improving the core Defense depot maintenance functions. The alternatives considered involved (1) maintaining status quo by allowing each service to continue to operate its own information system with some new development under JLSC’s purview, (2) choosing a corporate information system from among the services and establishing it as the DOD-wide standard system—deploying it either immediately and then enhancing it over a 3-year period or deploying it after enhancements, and (3) developing a new system.

In selecting an alternative, DUSD(L) did not evaluate sufficiently accurate cost data and detailed assessment of risks, nor did it obtain milestone reviews and approvals designed to ensure automated information systems are selected consistent with sound business practices.

DUSD(L) Selected DMSS Without Sufficient Cost Data

DUSD(L) selected DMSS without analyzing the system’s full development and deployment costs. Instead, it relied on a functional economic analysis of a previously proposed project—the Depot Maintenance Resource Planning (DMRP) system. This analysis significantly understated DMSS costs by including costs for only some components, and it understated costs for the components it did include. In early 1994, the JLSC Commander recognized that the DMRP economic analysis did not reflect DMSS as defined.

According to JLSC officials, DUSD(L) used the DMRP functional economic analysis as a basis for selecting DMSS because it was the best available at the time. The DMRP analysis estimated project costs at $988 million—$582 million to develop and deploy and $406 million to operate and support over a 10-year period. These officials stated that the DMRP analysis fairly represented the DMSS project. However, the Office of the Assistant Secretary of Defense for Program Analysis and Evaluation reviewed this analysis and found its level of detail insufficient to validate either cost or benefit estimates.

Although we also found insufficient details supporting cost and benefit estimates, we believe that DMSS will cost significantly more than the DMRP. As shown in table 2, the DMRP economic analysis included costs for only three of the eight DMSS system components. Therefore, the analysis understated DMSS costs by the amount necessary to develop and deploy the five additional system components. As of February 1995, JLSC had not completed a cost estimate for these five additional components.

In addition, the DMRP economic analysis underestimated costs for system components common to both DMRP and DMSS projects. Specifically, it underestimated licensing costs for using commercially owned software, costs to exchange data with other information systems, and costs to install the system.

One example of underestimated licensing costs is in a key DMSS component—the Air Force’s Depot Maintenance Management Information System (DMMIS). Over the last 10 years, the Air Force spent over $200 million to develop DMMIS for use in its maintenance depots. Originally designed around a core of commercially available application and database software, the Air Force chose to extensively modify this proprietary software to better meet its unique depot maintenance requirements. However, all software versions remain the sole property of the commercial developers. As a result, to use the DMMIS system, DOD will have to pay license fees to several commercial software developers.

Although the DMRP economic analysis did not specify DMMIS license fee costs, JLSC officials stated that $1.6 million per site was included in the deployment cost totals. In February 1995, JLSC estimated that DMMIS license fees for just the development facility and two operational sites would exceed $13 million, including a one-time payment of over $5 million and nearly $850,000 each year over the system’s life. As of April 1995, JLSC expected to run DMMIS at three additional sites. Licensing agreements had yet to be negotiated for these sites.

The DMRP analysis also underestimated costs to develop interfaces needed to allow system components to exchange data with the information systems currently used by the services to accomplish their missions. While the analysis recognized that system components must interface with other systems, it did not include the full cost of these interfaces. According to JLSC officials, some costs to interface the DMMIS and Programmed Depot Maintenance Scheduling System were included in the $37.7 million estimate for developing the system’s software applications. However, they did not specify these costs.

Although JLSC has yet to identify them, DMSS will require numerous system interfaces if it is to be the corporate depot maintenance system. For example, prior work done by the Air Force to deploy DMMIS, before it was selected as a DMSS component, identified 73 required interfaces just to meet Air Force requirements. As a DMSS system component, additional DMMIS interfaces will be needed to meet Army, Navy, and Marine Corps requirements. Further, interfaces for the remaining seven DMSS system components must be identified and developed. In February 1995, JLSC’s Deputy Director for Depot Maintenance estimated that $70 million not included in the DMRP economic analysis would be needed to develop the DMSS interfaces.

Finally, the DMRP economic analysis underestimated costs for deploying the system. The analysis estimates $497 million for system deployment. This estimate includes nonrecurring costs of $17 million to install the system at each operational site. Since DMSS was initiated, JLSC has identified that an additional $60 million would be needed to deploy the system.

In May 1994, the JLSC Commander told the DOD Comptroller about the DMRP economic analysis. The Commander stated that the economic analysis briefed to DUSD(L) in December 1993 and submitted for the DOD Comptroller’s review in early 1994 did not reflect DMSS as it was then defined. Further, he stated that to accommodate changes requested by the Comptroller and the office of Program Analysis and Evaluation and to reflect the current DMSS, JLSC was developing a new analysis. According to JLSC officials, the final economic analysis is expected to be completed in July 1995. However, by this time Defense will have spent more than $200 million to develop and deploy DMSS.

DUSD(L) Selected DMSS Without Fully Assessing Risks

Although any large automated information system development project is inherently a high-risk venture, DUSD(L) decided to develop and deploy DMSS without first fully assessing the risks to the project’s success. Without a detailed risk assessment, DOD has no assurance that DUSD(L) selected the best information system alternative for improving defense depot maintenance operations, nor can it plan actions designed to avoid or lessen the potential for project delay, overspending, or failure.

DOD has long recognized that project success relies on its ability to manage risk. The Defense Systems Management College guide on risk management states that, as a minimum, a prudent manager should attempt to understand system specific risks and quantify their potential impact for each alternative. While the earlier DMRP analysis identified several potential risks associated with each alternative being considered, it did not quantitatively or qualitatively compare these risks. Additionally, it did not contain any plans to mitigate potential project risks.

After DUSD(L) selected DMSS, JLSC convened a customer advisory team in April 1994 to identify and generate ideas on how to mitigate DMSS risks. This team, with membership from all the military services, identified a number of risks facing DMSS, such as (1) incomplete design and testing of the two core DMSS systems—Depot Maintenance Management Information System and the Baseline Advanced Industrial Management System, (2) not enough personnel to implement and maintain the system, (3) inability to obtain service cooperation needed to successfully build and deploy the system, (4) numerous external and internal interface issues, and (5) depot maintenance workers’ reluctance to work with an entirely new system.

JLSC requested another high-level risk analysis of the depot maintenance standard system strategy from the Defense Information Systems Agency’s Center For Integration & Interoperability (CFI&I). In a July 1994 briefing to JLSC, CFI&I said that program management posed the greatest risks to DMSS success. CFI&I said the project lacked (1) integrated detailed planning specifying the activities and milestones to be achieved at each depot and (2) coordination of events necessary to implement the system, and that, as a result, there was no assurance that DMSS could meet cost, schedule, and performance expectations.

In addition, CFI&I identified a number of technical risks to DMSS implementation, including (1) no encompassing data migration strategy, (2) incomplete and inadequate understanding of the requirement to interface DMSS with other current service systems, (3) difficulties associated with maintaining modified commercially owned software, and (4) incomplete development and testing of two of the system components.

In October 1994, JLSC began an iterative detailed assessment of DMSS to quantify risks, identify possible mitigation or avoidance steps, and develop a risk management plan. As of April 1995, JLSC was continuing this assessment.

Project Milestone Reviews and Approvals Not Obtained

Although Defense directives establish MAISRC review and MDA approval procedures to ensure that decisions to develop major automated information systems are based on sound business principles, as of February 1995, DUSD(L) had not scheduled a date for an initial milestone review of the entire DMSS project. Under MAISRC guidelines, a project should be reviewed and approved at each of five decision milestones before substantial funds are obligated. Despite this DOD policy, DUSD(L) spent nearly $180 million in fiscal years 1993 and 1994 on DMSS, and budgeted $111.2 million in fiscal year 1995 and $95.1 million in fiscal year 1996. These budgeted amounts are for the development and deployment of DMSS and do not include amounts to maintain and operate the current systems.

According to the director of logistics systems development within DUSD(L), DMSS will be submitted for MAISRC review and MDA approval during 1995. However, we found that as of February 1995, DMSS was on the MAISRC review schedule for 1995 but no date for the review had been established. The director also indicated that continued implementation of DMSS at selected prototype sites is justified based on past MAISRC reviews and MDA approvals of the DMMIS component of the project. However, Defense directives require programs which consist of a number of component systems to be reviewed by MAISRC and approved by MDA as a single project. Without these reviews and approvals, DOD has less assurance that the decision to select DMSS was consistent with sound business practices. Also, DOD did not have an opportunity afforded by the MAISRC review and MDA approval to redirect or terminate DMSS before investing significant amounts of money.

DUSD(L) Did Not Consider Reengineering Depot Maintenance Processes Before Selecting DMSS

In evaluating alternatives to improve depot maintenance operations, DUSD(L) did not consider reengineering alternatives which offer opportunities to dramatically improve depot maintenance business processes and greatly reduce the costs of operations. Even if successful, DOD’s strategy to develop and deploy an information system designed to incrementally improve depot maintenance processes will only provide marginal cost reductions and productivity increases rather than the fundamental and dramatic changes needed to meet the challenges of maintaining military readiness in the 1990s.

Reengineering of Business Processes Can Offer Dramatic Improvement

The defense community must make fundamental changes in the way it performs its activities if it is to provide the nation with the defense it requires and demands. ... Incremental improvements...will not shift the Department to a higher plateau of performance. Breakthrough innovation and change—a new paradigm for defense activities—is needed to meet the challenges of the 1990’s.

In January 1991, the Deputy Secretary of Defense endorsed a CIM implementation plan in which DOD would “reengineer,” or thoroughly study and redesign, its business processes before it standardized its information systems. The Deputy Secretary understood that DOD would have to improve the way it does business to achieve dramatic cost reductions and productivity increases and that it could not merely standardize old, inefficient processes and systems. Simply stated, doing the same thing faster will not provide dramatic improvement.

Though reengineering efforts in DOD have been limited in scope and represent a small portion of operations, significant improvements have been achieved through reengineering specific logistics business areas. For example, in 1980, the Defense Construction Supply Center established a contractor-operated parts depot program that reduced order and delivery time from 70 to 35 days—a 50-percent reduction.

In addition, the private sector, which also has major industrial centers that use similar maintenance and repair supplies for regularly scheduled maintenance of equipment, has undergone successful reengineering efforts when faced with increasing costs associated with acquiring supplies, spare parts, and raw materials. For example, since 1986, through customized agreements with suppliers and the use of new inventory management practices, an Ohio steel firm, Timken Company, reduced maintenance and repair inventories by $4 million (32 percent). The company also eliminated six inventory storerooms, improved inventory availability, and increased the accuracy of physical inventories.

We have also reported that by adopting certain commercial practices, Defense could similarly dramatically improve depot maintenance. In 1993, for example, we found that a number of private firms provide third-party logistics transportation services, such as freight bill processing, pre-auditing, verifying, and generating management reports with freight payment. Two of these firms proposed to perform transportation services for DOD at a cost ranging to $.75 to $1.25 per government bill of lading. DOD spends about $5.70 per freight bill to provide these same services. If DOD used these firms or changed its process to obtain similar performance, it could reduce costs for these services by more than 75 percent.

Reengineering Not Considered for Improving Depot Maintenance

Instead of first considering opportunities to reengineer business processes, DUSD(L) chose a strategy that focuses on the development and deployment of a DOD standard depot maintenance information system. Under this strategy, business processes are to be incrementally improved as DMSS is deployed. Reengineering of these processes will be considered only after system deployment. Currently, DMSS deployment is expected to be completed by fiscal year 1999. Accordingly, fundamental and dramatic changes to the depot maintenance processes will be delayed for years.

According to DOD officials, the vast number of different logistics processes and supporting information systems across the Department must be reduced before significant improvements can be made. These officials further stated that, once fully deployed, the Defense standard information systems will form the foundation upon which significant improvements to current depot maintenance practices can be made. This foundation will eliminate the need to implement major changes across a multitude of information systems and business processes that exist throughout the services.

Additionally, JLSC officials emphasized that improvements are being made to depot maintenance processes as DMSS is being deployed. According to these officials, benefits being achieved from these improvements include (1) cost reductions of $7 million in shop floor material recovered at the Air Logistics Center in Ogden, Utah, and a $8 million reduction in purchase of hazardous material at Hill Air Force Base and (2) performance increases from a 30-percent reduction in labor hours for overhauls of the Los Angeles class submarine, and two additional B-1 bombers processed through the Oklahoma City Air Logistics Center.

While these examples show that incremental improvements are being made, JLSC estimated that, overall, the DMSS project would reduce depot operational costs by $2.6 billion over a 10-year period ending in fiscal year 2003 from $112.9 billion to $110.3 billion over this period—a net cost reduction of about 2.3 percent.

We believe that standardizing existing information systems and incrementally improving business processes will not position DOD for reengineering its processes or dramatically improve their operations. Government and private industry have learned that initial focus on information system deployment may make future reengineering efforts more difficult by entrenching inefficient and ineffective work processes.

Accomplishing order-of-magnitude improvements in both government and private organizations requires reengineering—fundamental redesign—of critical work processes. Information system initiatives that do not first reengineer business processes typically fail or attain only a fraction of their potential. In addition, case studies of private organizations presented in Reengineering The Corporation - A Manifesto For Business Revolution,revealed that companies often commit a fundamental error in viewing automation as the answer to enhancing or streamlining their business operations. They spend billions of dollars to automate existing processes so they can perform the same work faster. Companies that initially focused on information technology managed only to entrench inefficient processes and made future change to these processes more difficult. ...Defense has focused on trying to pick the best of its hundreds of existing automated systems and standardizing their use across the military components without thoroughly analyzing the technical, cost, and performance risks of this approach. As a result, Defense may lock itself into automated ways of doing business that do not service its goals for the future and cannot provide promised benefits and cost savings.

Our review of DUSD(L)’s depot maintenance standard system strategy confirms this. The benefits it expects from implementing DMSS are relatively meager when compared with results other organizations are achieving through reengineering.

Conclusion

We agree with DOD’s concern over depot maintenance operations. Further, we agree that accurate information on depot operations and costs is critical to improving this important readiness-related support process. However, the decision to develop DMSS was based on insufficient cost data and with little consideration of identified risks. Efficient, cost-effective depot maintenance operations are important to supporting the Department’s military operations. Major investment decisions—such as DMSS—represent significant opportunities to make dramatic improvements in core business processes. Further, DOD’s proposed solution was made without due consideration of reengineering alternatives which offer dramatic improvements and greatly reduce costs of depot operations. DOD’s failure to consider reengineering alternatives and to fully consider the costs and risks associated with DMSS will likely limit those opportunities.

Recommendations

To achieve the dramatic improvements in effectiveness and efficiency of its depot maintenance operations that Defense has stated are critical to meet the challenges of the 1990s and beyond, we recommend that the Secretary of Defense direct the Deputy Under Secretary of Defense for Logistics to complete the following actions.

Prepare a full set of project documentation that describes the project and validates that it is the best alternative for improving depot operations. At a minimum, this documentation should include the following.

A final functional economic analysis containing a comprehensive evaluation of information system alternatives. This analysis should formulate and compare estimates of the total costs and benefits of each alternative.

Identification of economic and technical risks associated with success of each project alternative and development of a plan to avoid or mitigate these risks.

A comprehensive implementation plan that identifies actions to be taken, schedules, and milestones for these actions, and performance measures to be used to manage the system deployment.

Obtain the Major Automated Information Systems Review Council review and Milestone Decision Authority approval of the project documentation prior to spending any fiscal year 1996 funds on DMSS development and deployment.

Conduct a thorough study of opportunities to reengineer the depot maintenance business processes. Reengineering alternatives identified by this study should be analyzed as part of the final functional economic analysis and submitted for MAISRC review and MDA approval.

Agency Comments and Our Evaluation

The Department of Defense provided written comments on a draft of this report. The Deputy Under Secretary of Defense for Logistics generally disagreed with our findings, but partially concurred with our recommendations. Defense’s specific comments are summarized below and presented, along with our rebuttals, in appendix II.

In its comments, Defense took the following positions.

The DMSS functional economic analyses of March 1993 and January 1994 provided sufficient cost, benefit, and risk information to select the best alternative for improving depot maintenance business processes.

DMSS is not one system requiring MAISRC oversight and that the individual system components meeting MAISRC oversight criteria have been reviewed and approved.

Process reengineering is being accomplished concurrently with DMSS development and deployment. Defense asserts that by following this strategy, it has achieved substantial depot maintenance improvements yielding significant cost reductions. Defense expects even more dramatic improvements and savings in the future.

We disagree with Defense’s positions on these matters. Specifically:

The March 1993 and January 1994 FEAs were insufficient because they did not include cost and benefit estimates for the DMSS, contained cost estimates of questionable accuracy, and did not include cost and benefit estimates for five of the DMSS system components.

Defense CIM guidance specifically directs that information system projects be reviewed and approved in accordance with Defense life-cycle management directives. Under these directives DMSS is required to be reviewed by MAISRC and approved by the MDA at five milestone decision points before any funds are spent to develop the system. Further, these directives state that projects consisting of several components shall be considered as a single automated system.

Defense’s approach to improving depot maintenance business processes focuses on the selection of the best currently operating information systems and implementation of these selected systems across all Defense components. While this approach may improve overall DOD business processes and may provide incremental benefits, it cannot be construed as reengineering. DMSS is designed to provide only incremental improvements to existing business processes and it is clear from Defense’s own benefit projections that it will not result in the dramatic improvements that are possible by considering reengineering-based solutions. While it claims that DMSS has improved depot maintenance processes and resulted in some reductions in operational costs, DUSD(L)’s focus on information system selection and implementation may inhibit reengineering efforts by entrenching current work processes.

Although Defense disagreed with our findings, it agreed with our recommendation to prepare a full set of project documentation that describes DMSS and validates that it is the best alternative to improve depot maintenance. It partially concurred with our recommendation on obtaining MAISRC review but specifically disagreed with our recommendation concerning thoroughly studying opportunities to reengineer depot maintenance business processes. Our recommendation for MAISRC review is consistent with review requirements established in Defense life-cycle management directives. Further, because reengineering offers order-of-magnitude improvement and cost reductions, Defense cannot afford to deploy DMSS beyond the first five prototype sites until it has fully assessed reengineering alternatives.

We are sending copies of this report to the Ranking Minority Member of the Subcommittee, the Chairmen and Ranking Minority Members of the Senate and House Committees on Appropriations, the Senate Committee on Armed Services, the Senate Committee on Governmental Affairs, and the House Committee on Government Reform and Oversight; the Secretaries of Defense, the Army, the Navy, and the Air Force; the Director of the Office of Management and Budget; and other interested parties. Copies will be made available to others on request.

If you have any questions about this report, please call me at (202) 512-6240 or Carl M. Urie, Assistant Director, at (202) 512-6231. Major contributors to this report are listed in appendix IV.

Objectives, Scope, and Methodology

We based our review on an assessment of DOD’s implementation of its own directives and instructions for new automated information systems or the selection and implementation of standard migratory systems under the CIM initiative, as such projects relate to the depot maintenance business area. These directives, referred to as Life Cycle Management, contain the same steps and milestones as GAO’s own methodology for reviewing large automated information systems/projects.

Specifically, to determine whether the Department based its selection of DMSS on convincing analyses of costs and benefits, we reviewed policies, procedures, directives, and memoranda establishing criteria for the successful acquisition of automated information systems under the CIM initiative. We compared the Department’s actions and plans for selecting and implementing DMSS with these criteria. To further assess the adequacy of the selection, we examined the cost and benefit data available to senior Defense officials responsible for selecting DMSS. Because the level of detail was insufficient, we did not evaluate these cost and benefit data. Also, we interviewed Defense logistics officials to obtain the rationale behind the DMSS selection. To identify expected DMSS costs and benefits, we analyzed available functional economic analyses (FEA). We did not validate the costs and benefits presented in the FEA used to justify DMSS since (1) our objective was to examine DOD’s decision given the cost and benefit information available to it at the time and (2) the FEA was based on a different project—the DMRP. We interviewed JLSC officials to determine changes made to project scope, costs, or benefits occurring since early 1994 and any additional analyses currently being done. We also met with numerous program and functional officials, including JLSC managers responsible for implementing the eight DMSS system components, and depot officials at the Air Force’s repair depot in Ogden, Utah, and the Army’s depot in Tobyhanna, Pennsylvania.

To determine whether the Department had fully assessed economic and technical risks threatening the successful implementation of DMSS and identified actions to avoid or mitigate these risks, we reviewed risk assessments available when DUSD(L) decided to develop and deploy DMSS. Additionally, we examined risk analyses conducted by the Joint Logistics Systems Center, other Defense organizations, and industry experts completed since the DMSS selection was made. We interviewed program and technical officials to obtain opinions on the potential impact of risks identified by these analyses on project success and to identify actions for avoiding or mitigating those risks most likely to result in project failure, delay, and overspending.

To determine whether the Department selected a strategy that would dramatically improve depot maintenance processes, we reviewed DOD documents detailing challenges of meeting the defense mission in the post-cold war environment, CIM goals and objectives to meet these challenges, and the plans and strategies for implementing CIM across the Department. We compared these DOD strategies and plans to the Logistics Migration Approach established to implement the CIM initiative in the logistics business area. We then compared the level of improvement expected from a standard depot maintenance information system to the DOD stated requirement to meet the challenges of the future defense environment. To identify alternatives to information system approaches, we reviewed private industry studies and past GAO reports of lessons learned by private and public organizations that have successfully improved their business processes. We compared these lessons learned and case studies with the approach being implemented through the development and deployment of DMSS.

Our work was performed between April 1994 and March 1995 primarily at the offices of the Deputy Under Secretary of Defense for Logistics in the Pentagon, Washington D.C., and the Joint Logistics Systems Center, Wright-Patterson Air Force Base, Ohio. We also performed work at the offices of the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence in Washington D.C.; the Center for Integration & Interoperability, Defense Information Systems Agency, Blacklick, Ohio; the Air Force Air Logistics Center, Hill Air Force Base, Utah; and the Tobyhanna Army Depot, Tobyhanna, Pennsylvania.

Comments From the Department of Defense

The following are GAO’s comments on the Department of Defense’s letter dated May 30, 1995.

GAO Comments

1. The March 1993 Phase I Functional Economic Analysis (FEA) and January 1994 Phase II FEA for the DMRP processes did not provide well-supported estimates of project costs, benefits, and risks upon which to approve the Depot Maintenance Standard System (DMSS). As stated in our report, there are three reasons why these FEAs did not sufficiently support the DMSS selection.

First, these FEAs did not include cost and benefit estimates for the DMSS. The DMSS was not defined as a project until March 1994—more than a year after the Phase I FEA was completed and 2 months after the Phase II FEA was submitted for review. The Commander of Joint Logistics Systems Center told the DOD Comptroller that the functional economic analysis briefed to DUSD(L) and submitted for the Comptroller’s review in early 1994 did not include DMSS. The Commander also stated that JLSC was developing a new economic analysis to (1) accommodate changes requested by the Comptroller and the Office of Program Analysis and Evaluation and (2) reflect the current DMSS.

Secondly, the analyses contained estimates of questionable accuracy. The Office of the Assistant Secretary of Defense for Program Analysis and Evaluation, which is required to validate automated information system project estimates to help ensure that the economic analyses present compelling quantitative data for project selection, found that the level of detail was insufficient to validate either cost or benefit estimates. Even with this limitation, we determined that the Phase II FEA underestimated the cost to develop and deploy the three system components that later became part of DMSS by at least $100 million.

Information system options and opportunities that support the functional management strategy and process improvement efforts are evaluated based on technical feasibility, cost, schedule, performance, risk, and conformance to architectural guidelines and standards. Information system development/modernization must comply with life cycle management policy...The SDP will be part of the approval decision package supporting the designation of the AIS as a migration system by the OSD Principal Staff Assistant. The SDP will also support an in-process review (IPR) or milestone review, as appropriate, by the designated milestone decision authority (MDA)...When AIS changes are part of the process improvement alternative(s) selected for more detailed analysis, the Functional Activity Program Manager’s evaluation decision is a filter that precedes other reviews required by DoDD 8120.1...The Functional Activity Program Manager is responsible for ensuring that the AIS-related aspects of the process improvement proposal are reviewed and approved in accordance with DoDD 8120.1, in addition to being reviewed and approved by the OSD Principal Staff Assistant as part of the complete process improvement proposal.

As stated in our report, Defense Directive 8120.1, Life-Cycle Management (LCM) of Automated Information Systems (AISs); and Defense Instruction, 8120.2, Automated Information System (AIS) Life-Cycle Management (LCM) Process, Review, and Milestone Approval Procedures, establish MAISRC review and MDA approval procedures to ensure that decisions to develop or modernize major automated information systems are based on sound business principles. Under these procedures, a project should be reviewed and approved at each of five decision milestones before substantial funds are obligated. Despite this policy, DUSD(L) spent over $200 million to implement the DMSS initiative without receiving approval for even the initial milestone decision point.

Also, DUSD(L)’s claim that the MAISRC did review the DMSS initiative on March 16, 1993, is not accurate. On this date the MAISRC completed an In-Process Review (IPR) of the overall Logistics CIM strategy. An IPR is defined as “An LCM review between LCM milestones to determine the current program status, progress since the last LCM review, program risks and risk-reduction measures, and potential program problems.” Further, as admitted by DUSD(L), the DMSS initiative was not approved until early 1994—more than a year after this review.

For the purpose of determining whether an AIS is major, the separate AISs that constitute a multi-element program, or that make up an evolutionary or incremental development program, or make up a multi-component AIS program, shall be aggregated and considered a single AIS.

Based on these directives, DUSD(L) is required to obtain MAISRC review and approval for the entire DMSS initiative at each of five milestone decision points before any additional funds are spent. 3. DUSD(L) officials contend that reengineering of depot maintenance processes is occurring concurrently with the deployment of the DMSS. Further, they assert that these reengineering efforts will provide dramatic economic benefits, and cite cost savings and productivity increases accrued from initial implementation of four DMSS system components as support. DUSD(L)’s approach focuses on the selection of the best currently operating information systems and implementation of these selected system across all Defense components. While this approach may improve overall DOD business processes and may provide incremental benefits, it is not the fundamental rethinking and radical redesign of depot maintenance processes and will not provide the dramatic cost reductions and productivity gains available from process reengineering. At best, it will allow DOD to accomplish current depot maintenance processes faster and more efficiently. At worst, DUSD(L)’s focus on information system selection and implementation will make future reengineering efforts more difficult by entrenching current work processes.

In January 1991, the Deputy Secretary of Defense endorsed a Corporate Information Management initiative implementation plan that directed business processes be reengineered before information systems are standardized. However, DUSD(L) did not consider reengineering opportunities as alternatives to the DMSS initiative. As discussed in the report, the functional economic analysis used by DUSD(L) to approve the DMSS initiative compared only three alternatives. All three of these alternatives focused on using automated information systems to improve current depot maintenance functions.

Further, as stated in DMSS documentation, the initiative is designed to provide only incremental improvements to existing business processes. It is clear from Defense’s own benefit projections that DMSS will not result in dramatic improvements possible from consideration of reengineering-based solutions. DUSD(L) projected DMSS would reduce the costs to DOD depot maintenance operational costs over a 10-year period from $112.9 billion to $110.3 billion. A cost reduction of $2.6 billion or only 2.3 percent over this period does not constitute a dramatic increase in efficiency.

In late 1994, the Office of the Secretary of Defense for Command, Control, Communications, and Intelligence, responsible for CIM initiatives across DOD, found major flaws in the overall implementation. It concluded that, as opposed to the private sector which uses a very different approach, “DOD has virtually no chance of making high impact/quantum changes using the current approach.” Further, the Commission on Roles and Missions of the Armed Forces, charged by the Congress to provide an independent review of the roles and missions of the armed services, has found that “ather than reengineering its processes, DOD has spent its energies in closing excess capacity (base and facilities) and in standardizing its management information systems” and concluded that DOD will achieve a more compact, more standardized version of its traditional logistics approach.The Commission confirmed that DOD must radically reengineer its logistics processes to achieve meaningful improvements. 4. While information technology is critical to any reengineering effort because it provides a tool for breaking old rules and creating new ways of working, it should not be the driver of the reengineering effort. Such an approach may make future reengineering efforts more difficult by entrenching inefficient and ineffective work processes. Reengineering offers order-of-magnitude improvement compared to the incremental improvements DMSS is designed to provide. DUSD(L) can not afford to deploy DMSS departmentwide beyond the first five prototype sites until it has first determined which old rules need to be broken and what new ways of accomplishing depot maintenance are most efficient and effective. The Commission on Roles and Missions of the Armed Forces has identified a number of alternatives for changing the way DOD conducts its depot maintenance. These alternatives could serve as a starting point for a thorough study by DUSD(L) of its reengineering opportunities.

Description of DMSS Component Systems

This appendix provides brief descriptions of the eight information systems selected as Depot Maintenance Standard System components to support the DOD-wide depot maintenance function.

Baseline Advanced Industrial Management System: Supports allocation decisions on resource application, schedules, and job management of maintenance projects. It allows timely review of cost and schedule performance at any level of the work breakdown structure. One of this system’s major modules, Programmed Depot Maintenance Scheduling System, provides project schedules of individual maintenance operations and critical path of work requirements for maintenance of major end items.

Depot Maintenance Management Information System (DMMIS): Provides depot maintenance managers with an automated capability to forecast workloads; schedule repair activities; track and control inventories; program staffing, materials, and other resources; and track and manage production costs.

Enterprise Information System: Provides the ability to interface to existing data sources, extract relevant data, and package the information to support decisionmakers with timely summary information.

Facilities and Equipment Maintenance: Provides an integrated tracking and control system for equipment and facility maintenance, preventive maintenance, and calibration of precision measurement equipment.

Depot Maintenance Hazardous Materiel Maintenance System: Records the receipt and issue of all hazardous material within a maintenance depot. Provides inventory visibility of all hazardous material to control the issue of hazardous material to authorized users.

Interservice Material Accounting and Control System: Provides the tracking of Depot Maintenance Interservice Support agreements and visibility and control for interservice workloads.

Laboratory Information Management System: Provides the monitoring and control of laboratory data such as sample order status, order tracking, backlog, scheduling, location tracking, workload prediction, pricing, and invoicing. Automates tracking and archiving for depot material samples and test results.

Tool Inventory Management Application: Provides total inventory tracking and accountability of both hard and perishable (consumable) tools and tooling assets. Tracks issues and receipts of assets to both individuals and in tool kits.

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

Cincinnati Regional Office

Related GAO Reports

Government Reform: Using Reengineering and Technology to Improve Government Performance (GAO/T-OGC-95-2, Feb. 2, 1995).

Defense Management: Impediments Jeopardize Logistics Corporate Information Management (GAO/NSIAD-95-28, Oct. 21, 1994).

Commercial Practices: DOD Could Reduce Electronic Inventories by Using Private Sector Techniques (GAO/NSIAD-94-110, Jun. 29, 1994).

Executive Guide: Improving Mission Performance Through Strategic Information Management and Technology (GAO/AIMD-94-115, May 1994).

Commercial Practices: Leading Edge Practices Can Help DOD Better Manage Clothing and Textile Stocks (GAO/NSIAD-94-64, Apr. 13, 1994).

Defense Management: Stronger Support Needed for Corporate Information Management Initiative To Succeed (GAO/AIMD/NSIAD-94-101, Apr. 12, 1994).

Defense IRM: Business Strategy Needed for Electronic Data Interchange Program (GAO/AIMD-94-17, Dec. 9, 1993).

Defense Transportation: Commercial Practices Offer Improvement Opportunities (GAO/NSIAD-94-26, Nov. 26, 1993).

Defense Inventory: Applying Commercial Purchasing Practices Should Help Reduce Supply Costs (GAO/NSIAD-93-112, Aug. 6, 1993).

Commercial Practices: DOD Could Save Millions by Reducing Maintenance and Repair Inventories (GAO/NSIAD-93-155, Jun. 7, 1993).

DOD Food Inventory: Using Private Sector Practices Can Reduce Costs and Eliminate Problems (GAO/NSIAD-93-110, Jun. 4, 1993).

Defense ADP: Corporate Information Management Must Overcome Major Problems (GAO/IMTEC-92-77, Sep. 14, 1992).

DOD Medical Inventory: Reductions Can Be Made Through the Use of Commercial Practices (GAO/NSIAD-92-58, Dec. 5, 1991).

Commercial Practices: Opportunities Exist to Reduce Aircraft Engine Support Costs (GAO/NSIAD-91-240, Jun. 28, 1991).

Defense Logistics: Observations on Private Sector Efforts to Improve Operations (GAO/NSIAD-91-240, Jun. 13, 1991).

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Address Correction Requested