Information Technology: DLA Should Strengthen Business Systems Modernization Architecture and Investment Activities
Published: 2001-06-04T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Defense Logistics Agency (DLA) plays a critical role in supporting America's military forces worldwide. DLA employs about 28,000 civilian and military workers at about 500 sites in all 50 states and 28 countries; in round numbers, it manages 4 million supply items and processes 30 million annual supply distribution actions. This report reviews DLA's efficiency and effectiveness in managing it is Business Systems Modernization (BSM) acquisition program. Specifically GAO determines (1) whether DLA is using an enterprise architecture to guide and constrain its investment in (BSM) and (2) whether DLA is investing in BSM in an incremental manner.
section_title: What GAO Found
paragraphs: GAO found that DLA does not have an enterprise architecture to guide its investment in BSM. DLA plans call for creating an architecture as a byproduct of BSM's implementation. In addition, GAO found that DLA has not been managing its investment in this program in an incremental manner; that is, DLA has not treated the first of its four planned incremental releases of BSM as a separate investment decision. Instead, DLA has thus far treated the entire BSM program as a single investment decision, according to BSM officials. DLA now plans to take an incremental approach to future releases. This change would ensure that DLA's investment decisions are consistent with statutory requirements and federal guidance.

section_title: Letter
paragraphs: The Defense Logistics Agency (DLA) plays a critical role in supporting America’s military forces worldwide. To fulfill this role, DLA employs about 28,000 civilian and military workers, located at about 500 sites in all 50 states and 28 countries; in round numbers, it manages 4 million supply items and processes 30 million annual supply distribution actions. In fiscal year 2000, DLA reported that these operations resulted in sales to the military services of about $13 billion.
This report is one in a series of products to satisfy our mandate under the fiscal year 2001 Defense Authorization Act. The act directed that we review DLA’s efficiency and effectiveness in meeting customer requirements, application of best business practices, and opportunities for improving DLA operations. As agreed with your offices, our first review of DLA’s information technology (IT) management practices focused on DLA’s $900 million Business Systems Modernization (BSM) acquisition. As further agreed with your offices, our objectives were to determine (1) whether DLA is using an enterprise architecture to guide and constrain its investment in BSM and (2) whether DLA is investing in BSM in an incremental manner. Using enterprise architectures—institutional blueprints for business and technological change—and investing incrementally in large modernization programs are legislative and federal requirements and best industry practices. This review did not address other system modernization best practices, such as whether DLA is employing effective system acquisition process controls, and did not evaluate the specific commercial, off-the-shelf product that DLA chose as its BSM system solution. Details on our objectives, scope, and methodology are in appendix I.
subsections: section_title: Background
paragraphs: DLA is DOD’s logistics manager for all DOD consumable items and some DOD repair items; its primary business function is providing supply support to sustain military operations and readiness. In addition to this primary function, which DLA refers to as either materiel management or supply-chain management, DLA performs five other major business functions: distributing materiel ordered from its inventory, purchasing fuels for DOD and the U.S. government, storing strategic materiel,marketing surplus DOD materiel for reuse and disposal, and providing numerous information services, such as item cataloging, for DOD and the U.S. and selected foreign governments.
DLA consists of a central command authority supported by a number of field commands that manage the agency’s six business functions. Table 1 shows DLA’s field commands, their respective business functions, and examples of the consumable and/or repair items each manages.
To support the current materiel management business function, DLA reports that it relies on a collection of mainframe-based Cobol systems, which are not integrated and are more than 30 years old. These legacy systems support DLA’s traditional mission as a manager of physical inventory; that is, they support materiel management functions such as acquisition/ procurement, inventory control, requirements forecasting, requisition processing, technical data maintenance, and financial management. The systems consist of two primary systems—the Standard Automated Materiel Management System (SAMMS) and the Defense Integrated Subsistence Management System (DISMS)—and two subsidiary systems—SPEDE (SAMMS Procurement by Electronic Data Exchange) and DPACS (Defense Pre-award Contracting System). The functions of these systems are summarized in table 2.
DLA reports that these legacy systems are the product of decades of accumulated and divergent business practices and use obsolete technology that is no longer supported by the original equipment manufacturers and the software support provider. Further, DLA maintains that these systems, consisting of several million lines of unstructured and expensive-to-maintain code, provide inadequate analytical capability and no real-time data access.
subsections: section_title: DOD Has Issued Strategic Plans for Modernizing Logistics Management
paragraphs: Joint Vision 2020, published in June 2000, was preceded by Joint Vision 2010, published in 1996. It was issued by the Chairman, Joint Chiefs of Staff. force the right assets (personnel, equipment, and supplies) in the right place, at the right time, and in the right quantity. To achieve this goal, DOD envisions a real-time, Web-based information system providing both logisticians and field commanders with total asset visibility within a given theater of operations. Additionally, this system is to include decision support tools to improve requirements analysis and planning, as well as provide real-time control of the logistics supply chain, regardless of whether the requirements are to be fulfilled by the commercial sector or within DOD. 21st Century Logistics calls for the military services and DLA to pursue system modernization efforts to bring about the focused logistics goals described in Joint Vision 2020. The systems are to be an integrated collection of applications sharing a common data environment, which are to be deployed to all forces by fiscal year 2006.
To fulfill Joint Vision 2020, the Deputy Secretary of Defense issued a memorandum in March 2000 to the military services and DLA requiring submission of logistics transformation plans, documenting, on an annual basis, planned actions and related resources for implementing focused logistics, as well as any other logistics initiatives. The military services and DLA were directed to satisfy the 21st Century Logistics objectives by fiscal year 2006, including developing logistics process and system modernization plans by the end of fiscal year 2001 and implementing these plans by fiscal year 2006. DLA was directed to submit its transformation plan to the Under Secretary of Defense for Acquisition, Technology, and Logistics by July 1, 2000.
section_title: DLA Has Developed a Strategic Plan and Logistics Transformation Plan
paragraphs: To fulfill its direction under DOD’s 21st Century Logistics plan, DLA has outlined strategic goals and objectives in its Strategic Plan 2000: DLA 21and Logistics Transformation Plan. Under Goal 2 of DLA 21, DLA plans to reduce costs, improve efficiency, and increase effectiveness through organizational redesign, business systems modernization, strategic sourcing, infrastructure consolidation, and optimally sized inventories.
To achieve this goal, DLA is first focusing on transforming its materiel management (supply-chain management) function, because according to the DLA Director, this is DLA’s most critical business function. More specifically, it plans to implement a Web/network-based logistics system using commercial, off-the-shelf (COTS) products.
To select the most appropriate COTS products, DLA evaluated potential solutions against business processes for certain of its commodities (including clothing, aircraft parts, and medical supplies). According to DLA, it chose important, widely differing commodities so that its analysis would be sufficiently comprehensive to be extrapolated to the rest of the commodities within the materiel management business function. Further, DLA chose to evaluate solutions against business processes that had been previously reengineered, so that the system solution chosen would be based on modernized business processes. Through this analysis, which used scripted demonstrations of the reengineered requirements, DLA sought to ensure that the chosen applications could effectively meet DLA’s materiel management/supply-chain management requirements.
The resulting COTS-based system is to be used to reengineer the agency’s materiel management business processes. DLA refers to its acquisition and implementation of this COTS-based system, and the associated business process reengineering, as Business Systems Modernization (BSM).
subsections: section_title: DLA Has Begun Implementing its Business Systems Modernization Program
paragraphs: DLA’s BSM program is intended to modernize DLA’s materiel management business function, changing DLA from solely a provider and manager of physical inventory to primarily a manager of supply chains. In this role, DLA would link customers with appropriate suppliers and track physical and financial assets. DLA expects this approach to enhance supply-chain visibility and capabilities, resulting in reduced logistics cycle times, increased customer satisfaction, reduced customer wait time, and decreased materiel costs. In short, DLA’s goal is “to do business as business does business.”
According to BSM program documents, DLA’s transformation from its current to its future role requires modernization of the IT systems that now support DLA’s materiel management business function, subfunctions, and processes. The BSM system solution is envisioned as consisting of Web- and COTS-based applications supported by an enterprise application tool. Through the use of COTS products, DLA expects to reduce materiel management systems operating costs and take advantage of the continuous technology insertion and continuous functional improvement that COTS packages allow. The BSM COTS will consist of enterprise resource planning (ERP) software.
DLA plans to acquire and deploy its BSM COTS system solution through a series of four system releases/increments. First, it plans to demonstrate successful application of its new concept of doing business to selected commodities, namely, earth-moving equipment, medical/pharmaceutical supplies, and F/A-18 engine components. These commodities were chosen for the first release because each is an important responsibility of one of the three Defense Supply Centers. Thus, the first release will be deployed to all three Defense Supply Centers at once. Once this first release is successfully demonstrated, DLA plans to expand the system solution to other commodities and to other users within the materiel management business function.
According to DLA’s fiscal year 2001 budget submission, DLA will invest approximately $897 million to acquire and implement BSM from fiscal years 2000 through 2007 (see table 3). Thus far, DLA has obligated about $150 million for BSM. From this total, about $56 million was obligated in fiscal year 2000 to prepare and validate functional requirements, conduct pre-award contract functions for selection of the systems integration contractor, perform initial work on the program by awarding the integration contract, and begin detailed business process reengineering planning for the first increment of the program.
The remaining approximately $93 million was obligated in fiscal year 2001 for the development of a materiel management enterprise architecture (referred to by DLA as the BSM Business Architecture Blueprint) and for the initial implementation, including testing, of the BSM concept demonstration. Of the unobligated portion of the BSM budget, about $549 million is to fund implementation and sustainment of the COTS solution (system releases 1 through 4) within the materiel management business area, including its extension to the commodities not covered by the concept demonstration. DLA officials told us that the remaining unobligated funds for fiscal years 2006 and 2007, about $201 million, are to be used for extending the BSM ERP software to the distribution and cataloging business areas.
Only about 1 percent of the BSM budget is for new network and computing platform infrastructure (hardware and systems software). This is because DLA’s current and future investments in IT infrastructure, which will support ongoing DLA operations as well as the BSM program, are funded separately. The infrastructure investments in fiscal year 2001 amount to about 45 percent of DLA’s approximately $680 million IT budget.
BSM is expected to take 6 years to acquire and implement for materiel management (fiscal years 2000 through 2005). DLA has structured the BSM life cycle to comply with DOD 5000.2 milestone phases for major systems acquisitions. Figure 1 summarizes the BSM development timeline.
The first milestone phase (phase 0, December 1999 through July 2000) of the BSM program focused on evaluation of alternative concepts and potential system integrators.
The second and current milestone phase (phase I/IIA) covers program definition (design), risk reduction, and concept demonstration. The current phase includes implementation and testing/evaluation of the system solution for certain commodities, as well as such implementation activities as training and site preparation. This phase is scheduled to run from September 2000 through September 2002.
The third phase (phase II/III) is referred to as initial fielding/deployment and operational support. This phase focuses on implementing the system beyond the scope of the concept demonstration, operating and maintaining the system, and modifying and upgrading the system as required. This phase is scheduled to run from October 2002 through March 2005.
Investment decisionmaking for BSM (and other DLA major modernization programs) is vested in the DLA 21 Executive Board, chaired by the Chief Acquisition Executive of DLA, who is the Vice Director. BSM life-cycle management is the responsibility of the BSM Program Manager, who runs the BSM program office and reports to the DLA Program Executive Officer, who oversees the program in coordination with the DLA Chief Information Officer (CIO). Figure 2 shows this BSM program management and oversight structure.
section_title: BSM Is Not Being Guided by Either a DLA Architecture or a DOD-wide Logistics Architecture
paragraphs: Using enterprise architectures to guide and constrain information technology investments is called for by law, OMB, and DOD, and is an industry best practice. An enterprise architecture is a blueprint for guiding and constraining business and technological change for the enterprise, which can be an organization (e.g., DLA or the services) or a functional or mission area spanning more than one organization (e.g., logistics management or financial management). In some cases, both organizational and functional/mission area architectures are appropriate, because organizations interrelate closely, sharing functional and mission area responsibilities. This is the case for DOD and its component organizations.
DLA is managing its BSM program without having either a DLA enterprise architecture or a DOD-wide logistics management enterprise architecture. In doing so, DLA risks investing in significant business process and system change that does not optimally support enterprise (DLA and DOD) requirements (operational, system, and technical). To their credit, officials from both DLA and the Office of the Deputy Under Secretary for Logistics and Materiel Readiness recognize the need to take an enterprise approach to transforming logistics management, and at the conclusion of our work, they committed to doing so.
subsections: section_title: Enterprise Architectures: A Cornerstone of Successful Modernization Programs
paragraphs: Enterprise architectures are essential tools for effectively and efficiently reengineering business processes and for implementing and evolving their supporting systems. Enterprise architectures systematically capture—in useful models, diagrams, and narrative—the full breadth and depth of the mission-based mode of operation for a given enterprise, which can be (1) a single organization or (2) a functional or mission area that transcends more than one organizational boundary (e.g., financial management, acquisition management, logistics management). An architecture describes the enterprise’s operations in both (1) logical terms, such as interrelated business processes, information needs and flows, work locations, and system applications, and (2) technical terms, such as hardware, software, data, communications, and security attributes and performance standards. It provides these perspectives both for the enterprise’s current or “as is” environment and for its target or “to be” environment, as well as an IT capital investment road map for moving between the two environments.
The development, implementation, and maintenance of enterprise architectures are recognized hallmarks of successful public and private sector organizations. Managed properly, an enterprise architecture can clarify and help optimize the interdependencies and interrelationships among an organization’s business operations and the underlying IT infrastructure and applications that support these operations. Employed in concert with other important IT management controls, such as portfolio investment management (selection, control, and evaluation) practicesand continuous information security management practices, enterprise architectures can greatly increase the chances that modernization programs will succeed. Our experience with federal agencies has shown that attempting to define and build major IT systems without first completing an enterprise architecture often results in IT systems that are duplicative, are not well integrated, are unnecessarily costly to maintain and interface, and do not effectively optimize mission performance.
Congress, OMB, and the federal CIO Council have recognized the importance of enterprise architectures. The Clinger-Cohen Act, for example, requires that agency CIOs develop, maintain, and facilitate the implementation of enterprise architectures as a means of integrating business processes and agency goals with IT. Further, OMB has issued guidance on the development and implementation of agency IT architectures. Among other things, OMB guidance directs that agency investments in information systems be based on these architectures.Similarly, the CIO Council has issued guidance providing (1) a federal framework for the content and structure of an enterprise architecture,(2) a process for assessing investment compliance with an enterprise architecture, and (3) a set of management controls for developing, implementing, and maintaining an enterprise architecture.
According to CIO Council guidance, it is critically important that an enterprise architecture be derived through a “top-down” incremental approach, consistent with the hierarchical architectural views that are the building blocks of published architecture frameworks, including DOD’s. It is equally important, according to this guidance, that the higher level views span the entire enterprise. Only through such an approach can an organization develop enterprisewide understanding of the interrelationships and interdependencies among business operations and supporting technology. Such understanding is vital for informed decisionmaking about whether the enterprise, and thus the enterprise architecture, can be divided into segments without sacrificing the goal of optimizing enterprisewide performance and accountability.
DOD has also issued enterprise architecture policy, including a framework defining an architecture’s content and structure. Specifically, in February 1998, DOD directed its components and activities to use the C4ISR (Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance) Architecture Framework, Version 2.0. According to DOD, this framework is a critical tool in achieving its strategic direction, and all DOD components and activities should use the framework for all functional areas and domains within the department. This framework is also recognized in the CIO Council published guide as a model architecture framework. Figure 3 shows the views required by the C4ISR framework and their relationships.
Briefly, the C4ISR Architecture Framework decomposes an enterprise architecture into three primary views (windows into how the enterprise operates): the operational, systems, and technical views. According to DOD, these three interdependent views are needed to ensure that IT systems are developed and implemented in an interoperable and cost- effective manner. Each of these views is summarized below.
The operational architecture view defines the operational elements, activities, tasks, and information flows required to accomplish or support an organizational mission or business function. According to DOD, it is useful for facilitating a number of actions and assessments across DOD, such as examining business processes for reengineering or defining operational requirements to be supported by physical resources and systems.
The systems architecture view defines the systems and their interconnections supporting the organizational or functional mission, including how multiple systems link and interoperate, and may describe the internal construction and operations of particular systems. According to DOD, this view has many uses, such as helping managers to evaluate interoperability improvement and to make investment decisions concerning cost-effective ways to satisfy operational requirements.
The technical architecture view defines a minimum set of standards and rules governing the arrangement, interaction, and interdependence of system applications and infrastructure. It provides the technical standards, criteria, and reference models upon which engineering specifications are based, common building blocks are established, and applications are developed.
Within the three architectural views, the C4ISR Architecture Framework identifies 26 graphical, textual, and tabular architectural artifacts or products. Of the 26 products, DOD specifies that 7 are essential and must be developed for each enterprise architecture. Table 4 briefly describes the content of each essential product.
section_title: DLA’s Approach to Developing an Architecture Is Not Enterprise Focused
paragraphs: BSM officials, including the BSM program manager and chief architect, have acknowledged that DLA does not have an enterprise architecture. However, these officials also maintain that BSM still has had enterprisewide architectural direction, because although the program did not have C4ISR-compliant architectural artifacts, the strategic direction of the agency (as defined in strategic plans) was the primary focus of BSM.Nevertheless, these officials also stated that they planned to develop a C4ISR-compliant DLA enterprise architecture.
Subsequently, the BSM program office prepared a draft enterprise architecture development plan that focused first on developing a materiel management architecture, which DLA calls the BSM Business Architecture Blueprint, as a by-product of acquiring and implementing its BSM COTS solution. According to the draft plan, once BSM is completed (scheduled for fiscal year 2005), DLA will add other business functions to the BSM Business Architecture Blueprint so that the end result is an enterprisewide architecture (see fig. 4).
DLA’s approach to developing and implementing a C4ISR-based, agencywide enterprise architecture is thus one of compliance after the fact (that is, after acquiring and implementing its system solution), rather than development before the fact (and application during the fact). DLA plans to fill out the essential products of the C4ISR framework as by-products of implementing its COTS solution, rather than developing these products beforehand, so that they could be used as the basis for guiding and constraining its acquisition and implementation of system solutions.
Although DLA has not yet formally developed the seven essential products in the C4ISR format, BSM program officials point out that some of the information needed for these products is available, and DLA is relying on that information in its direction of BSM. For example, BSM officials acknowledged as enterprise requirements the following: technology infrastructure standards, a transition strategy, technical reference models, standards profiles, and information assurance policies at the enterprise level. In our view, such information would be valuable and useful in building a DLA enterprise architecture.
However, DLA nevertheless does not have a complete enterprise architecture as defined by DOD’s C4ISR framework, and its plans for developing one are not consistent with a fundamental best practice in CIO Council guidance. That is, DLA’s approach is focused on only part of its enterprise (albeit the largest), whereas CIO Council guidance promotes a process to ensure development of an enterprisewide perspective. According to this guidance, the scope of the architecture for the operational view needs to encompass the entire enterprise. Only then can the agency understand the relationships and dependencies among its business areas and position itself to make informed decisions about the level of depth and detail to include in the architecture. Necessarily, this level of depth and detail will be tailored to each enterprise based on its size and complexity, as well as the purpose of the architecture and the riskiness of the modernization effort to be undertaken.
Although DLA’s approach does not preclude success, it does introduce a risk of misalignment between the COTS solution and the enterprise’s strategic operational vision. To manage this risk, the enterprise would have to fully evaluate the COTS solution against the business requirements for the whole enterprise, not just a portion of it. Such an enterprisewide perspective is what Strategic Plan 2000: DLA 21 and the Logistics Transformation Plan envision. (For instance, DLA 21 refers to logistics solutions that are integrated from two standpoints: throughout the supply chain and throughout the agency.) An approach that lacks an enterprisewide perspective does not adequately ensure that the COTS system solution being acquired and implemented will optimally support all logistics business functions.
Despite the existence of some enterprisewide architectural information, DLA’s plan for building an architecture for its future operational, systems, and technical environment does not have an enterprisewide perspective. Instead, DLA is defining and implementing its architecture by first focusing on one of its six business functions (materiel management) separately from the others. For DLA, this is particularly problematic because DLA’s six business functions are interrelated and interdependent. For example:   Materiel management and cataloging are related, because how an item is procured is determined in part by how an item is classified during the cataloging process.
Materiel management and distribution are related, because decisions about whether DLA will distribute the materiel (or the vendor will deliver directly to the requisitioner) are made at the same time the materiel is bought.
Distribution and cataloging are related, because how materiel is stored and transported is determined by how it is classified during cataloging (e.g., hazardous materiel has special storage and transportation needs).
DLA adopted its approach to developing and implementing an enterprise architecture because the agency believed that using materiel management—its largest business process—as a starting point would lead to a solution with a baseline functionality that could be augmented as DLA evolves. Further, DLA expected that addressing materiel management would lead to the inclusion of some aspects of its other business functions, such as distribution and cataloging, for that part of materiel management operations to be performed via private sector supply chains.
However, DLA also recognized that postponing consideration of some business functions might lead to disconnects later in the development. According to DLA, if the later DLA business functions could not be incorporated into the BSM Business Architecture Blueprint without major redesign, then DLA would construct linkages between BSM and the other business areas to produce an enterprisewide architecture. This approach introduces the risk that DLA will sacrifice optimizing enterprisewide performance and accountability in order to optimize the performance and accountability of its individual components—the risk that well developed and implemented enterprise architectures are intended to prevent.
At the conclusion of our review, DLA officials stated that they intend to expand the scope of their enterprise architecture program to provide for an agencywide, C4ISR-compliant architecture.
section_title: DLA Does Not Have Effective Management Structures and Processes for Developing and Implementing an Enterprise Architecture
paragraphs: CIO Council guidance defines a set of recognized key practices (management structures and processes) for developing and implementing an enterprise architecture that are hallmarks of successful public and private sector organizations. Among other things, these practices include the following:   Because the enterprise architecture is a corporate asset for systematically managing institutional change, the head of the enterprise should support and sponsor the architecture, giving it a clear mandate in the form of an enterprise policy statement. Such support is crucial to gaining the commitment of all organizational components of the enterprise, all of whom should participate in developing and implementing the enterprise architecture.
The enterprise architecture effort should be directed and overseen by an executive body, empowered by the head of the enterprise, with members who represent all stakeholder organizations and have the authority to commit resources and to make and enforce decisions for their respective organizations.
The enterprise architecture effort should be led by a Chief Architect who reports to the enterprise CIO, and it should be managed as a formal program. A formal program entails creating a program office, committing core staff, implementing a program management plan that details work breakdown structure and schedules, allocating resources and tools, performing basic program management functions (e.g., risk management, change control, quality assurance, configuration management), and tracking and reporting progress against measurable goals.
The enterprise architecture should conform to a specified framework.
DLA is following some of these recognized key practices: it plans to follow the DOD C4ISR Architecture Framework, and a BSM Chief Architect has been designated. However, DLA is not following most key practices. It is not, for example,   approaching the architecture as a corporate endeavor, with explicit support and sponsorship from the DLA Director in the form of an agency policy statement;   using an executive body consisting of stakeholders from across DLA to direct and oversee the architecture effort; or   managing the architecture effort as a formal program.
Additionally, while an architecture development plan was drafted following our inquiries, neither the DLA CIO nor any DLA executive body has taken any action to approve the draft plan (dated July 2000), and thereby demonstrate their commitment and accountability for the architecture.
Moreover, the Chief Architect is assigned to the BSM program rather than to an organization with authority and responsibility for DLA-wide logistics modernization, and the draft architecture development plan focuses solely on materiel management, only providing a notional sequence for possibly incorporating DLA’s five other business functions at some time after 2005. Further, no work breakdown structures and schedules are provided for adding these other business functions, no estimates of resources are provided for implementing the plan, no measures are provided for managing progress in developing the architecture, and no enterprise stakeholders outside the BSM program participated in drafting the plan.
At the conclusion of our review, DLA officials stated that they will expand the architecture program to provide an agencywide focus, assign responsibility and accountability for the program accordingly, and formalize their approach to its management.
section_title: DOD Does Not Have a Departmental Logistics Management Enterprise Architecture
paragraphs: Beyond the need for a DLA organization-based enterprise architecture, DLA needs a DOD-wide logistics enterprise architecture, which would also serve DOD’s other component organizations that have interrelated and interdependent logistics management missions. However, DOD does not yet have such an architecture, although it has begun efforts to develop one.
Despite the lack of a DOD-wide logistics architecture, in March 2000, the former Deputy Secretary of Defense directed DLA and the military services to develop plans for modernizing their respective logistics processes and systems by July 2001 and to implement their respective plans by September 2006. No DOD-wide architectural blueprint to promote operational and system commonality and integration accompanied this direction.
According to the Director of Logistics Systems Modernization within the Office of the Deputy Under Secretary of Defense for Logistics and Materiel Readiness (the DOD organization responsible for overseeing the implementation of systems modernization efforts in 21st Century Logistics), DOD needs a departmental logistics management enterprise architecture to achieve its logistics management vision and optimize DOD- wide logistics management performance and accountability. Without such an architecture, the Director stated that the services’ and DLA’s investments in new logistics management systems would result in operations and systems that, although modernized, would continue to be “stovepiped,” rather than optimized and integrated across DOD. Accordingly and as an outgrowth of Year 2000 testing, efforts were started to develop a DOD-wide logistics enterprise architecture concurrently with DOD components’ efforts to modernize their respective operations and systems. According to the Director, modernization of component operations and systems is the single most important factor in being able to effectively and efficiently optimize and integrate operations and systems across DOD. The Director’s position is that components’ modernization efforts and DOD-wide logistics architecture efforts should proceed in parallel.
To date, according to the Director,   a plan has been drafted for creating a departmental logistics management   a description of the department’s current or “as is” logistics management enterprise architecture is being developed; and   descriptions of both near-term and long-term future (“to be”) logistics management enterprise architectures are being developed.
However, the Director would not provide us a copy of the enterprise architecture development plan, noting that it has yet to be approved and that a milestone for completing the enterprise architecture was not available. He also would not provide us copies of any “as is” architectural products, acknowledging that these products were based on data obtained under DOD’s Year 2000 Conversion Program and thus needed to be modified in light of the ongoing modernization activities in the military service and DLA. He added that the near-term and long-term “to be” architecture efforts are being managed by two different organizations within the Office of the Deputy Under Secretary for Logistics and Materiel Readiness, and that these architectures are to be based on yet-to-be- developed business rules and operational and technical agreements among the services and DLA. The Director specifically cited efforts under way to develop an architecture for DOD-wide management of conventional ammunition, DOD’s most critical commodity for the warfighter, as the prototype effort for developing a logistics management architecture and policies for architecture-based portfolio investment management.
The Director also told us that the Office of the Deputy Under Secretary of Defense for Logistics and Materiel Readiness would control these investments using traditional means, such as exercising oversight of the acquisition and budget processes. This oversight role will be augmented, according to the Director, by a DOD enterprise integration consortium, established in December 2000, made up of component organizations that are pursuing logistics enterprise resource planning system solutions, and led by the Director. The purpose of this consortium is to update plans for community services and DOD-wide logistics rules, develop enterprisewide interfaces between logistics and other functional areas, and develop mechanisms for sharing development plans and technical artifacts among programs and for sharing lessons learned.
These controls, however, are not sufficient substitutes for having a well- defined enterprise architecture. As discussed earlier, an enterprise architecture defines the business and technical rules, standards, and protocols that govern how the entire enterprise will operate in the future, and provides a common road map for getting to this future operational state. It thus serves as the explicit, documented change management tool that the Office of the Deputy Under Secretary for Logistics and Materiel Readiness needs to effectively discharge the very acquisition and budget oversight role that the Director cited. Without an enterprise architecture to guide and constrain the components’ modernization efforts, there is not adequate assurance of common understanding of the nature and content of the change.
At the conclusion of our review, an official in the Deputy Under Secretary’s office acknowledged that DOD needs to move swiftly to develop and use a DOD-wide logistics management architecture.
section_title: DLA Has Not Invested Incrementally in BSM
paragraphs: Incremental investment management involves three fundamental components: (1) developing/acquiring a large system in a series of smaller projects or system increments; (2) individually justifying investment in each separate increment on the basis of costs, benefits, and risks; and (3) monitoring actual benefits achieved and costs incurred on ongoing increments and modifying subsequent increments/investments to reflect lessons learned. Using this approach prevents agencies from discovering too late that their systems are not cost beneficial and allows them to reduce the enormous risks associated with large, expensive projects. Such an approach does not preclude overlap and smooth transition among increments, because lessons learned from the actual results of ongoing increments should be monitored and evaluated continuously so that these results are available for use in defining and justifying future increments.
DLA has not yet followed an incremental approach to investment in BSM. To date, DLA has treated the entire BSM program as one investment decision, justified by a single economic analysis, because this approach was consistent with DOD policy for major system acquisitions that were in effect until January 2001. DLA invested in its preliminary work (the analysis of candidate COTS systems against previously reengineered business processes) and in the initial release of BSM on the basis of this economic analysis. As a result, DLA did not justify and decide on investing about $150 million thus far in BSM on the basis of release-specific analyses of costs, benefits, and risks.
This kind of approach to making investment decisions has historically resulted in agencies investing huge sums of money in systems that do not provide commensurate benefits, and thus has been abandoned by successful organizations. The need to avoid this pitfall was a major impetus for the Clinger-Cohen Act investment management reforms.
At the conclusion of our review, DLA officials told us that in future, they plan to make investment decisions on BSM incrementally.
subsections: section_title: Incremental Investment Management Spreads the Risk of Large Programs Across Smaller, Incremental Parts
paragraphs: Capital Programming Guide, Version 1.0, OMB Circular A-11, part 3, Supplement, Planning, Budgeting, and Acquisition of Capital Assets (July 2000), pp. 35–37.
Separability: Each increment should be fully funded, have substantial programmatic use that is not dependent on any subsequent increment, and be capable of performing its principal functions even if no subsequent increments are acquired. Interoperability: Each increment should comply with a common architecture or commercially acceptable technology standards and should be compatible with and capable of being integrated with other increments.
Performance requirements: Each increment’s performance requirement should be consistent with the performance requirements of the completed overall system and should address interface requirements with other increments.
In short, incremental investment helps to mitigate the risks inherent in large IT acquisitions/developments by breaking apart a single large project into smaller, independently useful components with known and defined relationships and dependencies. Making investment decisions up front on large-scale, long-term projects is generally risky: their economic justification is based on costs, benefits, and risks that are difficult to forecast reliably, partially because later increments are not well understood or defined, and partially because they are subject to change in light of experience on nearer term increments and changing business needs. Through incremental investment management, organizations can reduce the level of project risk and complexity faced at any one time by spreading this risk and complexity across a series of smaller investments;   permit the delivery of some part of the expected business value earlier rather than waiting until later for the total, but more uncertain, business value;   continuously monitor and evaluate the delivery of cost and benefit expectations on ongoing increments and use this information to better define and economically justify these increments; and   permit later increments to exploit technology advances or accommodate evolving business needs.
In January 2001, DOD issued a change to its major system acquisition policy requiring incremental investment management; specifically, the policy notes that a program’s milestone decision authority must verify that each increment meets part of the mission need and delivers a measurable benefit, independent of future increments.
section_title: DLA Has Not Yet Made Investment Decisions Incrementally
paragraphs: Although DLA plans to acquire and implement its BSM system solution in four increments, it has not so far managed BSM investments incrementally. Specifically, (1) DLA has not justified proposed investment in the first BSM increment on the basis of costs and benefits, (2) the DLA 21 Executive Board has not made decisions about whether to invest in each discrete BSM increment, and (3) the BSM program office has not measured actual return on investment from each increment and used the results to better inform decisionmaking about future increments.
DLA’s position is that BSM was being managed according to DOD Instruction 5000.2, which until January 2001 required a single economic analysis of a system’s life-cycle costs and benefits before the system entered the concept demonstration phase. Accordingly, DLA prepared a program life-cycle economic analysis for BSM, according to which the program is expected to produce a positive net present value over its 15- year life cycle and improve military readiness. As a result, DLA officials stated that investing about $900 million to acquire and implement the BSM system solution is prudent, and DLA committed itself to funding, acquiring, and implementing all four BSM phases or increments.
Relying on a single economic analysis to make an investment decision for a program that is large and risky, involving many things to be done over many years, is neither prudent nor consistent with the principles of incremental investment management embodied in the Clinger-Cohen Act and OMB requirements. Approaching the BSM investment decision in this way was especially risky, since at the time that DLA prepared the BSM program life-cycle economic analysis, it had defined BSM business requirements for only the reengineered part of the materiel management business area. Thus, DLA was not only attempting the daunting task of accurately estimating the costs and benefits of many things over many years, it was doing so without knowing its BSM requirements beyond the reengineered parts of the materiel management business area. Our experience in reviewing IT investments across the federal government has shown that such estimating does not produce reliable results and cannot provide a sufficient basis for informed investment decisionmaking.
To date, the BSM program office has not analyzed the costs, benefits, and risks of what DLA defines as its first BSM release. Program officials told us that they justified this release, which is to cost $93 million (not including the $56 million that was spent on BSM before the first release), on the basis of the program life-cycle economic analysis.
At the conclusion of our review, however, DLA officials informed us that they do now plan to make decisions on the last three releases incrementally: they plan to justify release 2 on the basis of its costs, benefits, and risks, and to verify that the first release produced benefits commensurate with costs before deciding to invest in release 2. They also stated that they would follow this approach to investing in later releases. According to program officials, because DLA is still working on its first release (concept demonstration), the investment decision for release 2 will not be made until fiscal year 2002.
However, the program office does not yet have plans or measures to determine, once an increment is implemented, whether its expected value is actually accruing. The implementation plans for the first BSM increment (release 1) were completed in April 2001, but no specific plans or measures were developed to determine whether benefit versus cost expectations would be met for this increment. Clinger-Cohen requires that agencies identify quantifiable measurements for determining costs and benefits. Until the program office has some means of measuring expected value, DLA will not be able to determine whether this first increment is performing at a level equal to or better than its current materiel management systems—SAMMS and DISMS.
According to the BSM program manager, the investment management plans and measures are not in place because DLA’s policies and procedures on investment portfolio management are still in draft. When these are completed, they should govern, for example, (1) how investment increments are to be defined, prioritized, and sequenced, (2) how performance criteria are to be applied to each increment, and (3) how accrued earned value from each investment increment will be measured.
Until these investment management plans and measures are in place, the program manager stated that incremental investment will be accomplished through a series of 11 planned contract task orders, consisting of 105 more specific subtasks, which are to span the four BSM increments. However, merely structuring contract task orders into incremental pieces is not a sufficient substitute for making investment approval and funding decisions incrementally.
section_title: Conclusions
paragraphs: DLA has already invested significant sums of money in its BSM program, and its plans call for investing hundreds of millions of dollars more. To invest this money wisely and in a way that minimizes risks, DLA will need to acquire BSM within the context of an enterprise architecture and on an incremental basis. To date, DLA has not used an enterprise architecture or applied management controls for developing, implementing, and maintaining one, as described in CIO Council guidance. Additionally, DLA has not employed incremental investment practices.
Officials of DLA and the Office of the Secretary of Defense have committed to correcting these management weaknesses. We view these commitments as positive first steps for two primary reasons. First, if DLA continues to invest large sums of money in BSM without a DLA- or DOD- wide logistics enterprise architecture, it runs the serious risk that it will acquire a system solution that focuses narrowly on DLA materiel management performance and accountability at the expense of DLA- and DOD-wide performance and accountability. Second, until DLA begins justifying the return on its BSM investment incrementally, making BSM investment decisions incrementally, and measuring actual return on this investment incrementally, it will not be able to make well-informed and prudent investment decisions, and it will not know whether its BSM system solution is cost-effective until it has already spent hundreds of millions of dollars.
section_title: Recommendations
paragraphs: To address DLA’s need for an enterprise architecture to guide and constrain its IT capital investments, including BSM, we recommend that the Secretary of Defense direct the DLA Director to designate and treat development, implementation, and maintenance of a DLA enterprise architecture as an agency priority.
We recommend that in fulfilling this direction from the Secretary, the DLA Director (1) issue a policy governing development, implementation, and maintenance of an enterprise architecture and (2) establish a DLA enterprise architecture steering committee, chaired by the DLA Vice Director, to provide a DLA-wide direction and focus to the enterprise architecture, and to ensure that one is developed and maintained in a manner consistent with the CIO Council published guide on managing enterprise architectures.
To provide for the effective development and maintenance of the DLA enterprise architecture, we recommend that the DLA Director make the CIO accountable to the enterprise architecture steering committee for developing and maintaining the agency’s enterprise architecture. We recommend that in fulfilling this responsibility, the CIO appoint a Chief Architect for DLA and establish a program office responsible for developing and maintaining a DLA-wide enterprise architecture. We further recommend that the CIO direct the Chief Architect to work collaboratively with the appropriate offices within the Office of the Deputy Under Secretary for Logistics and Materiel Readiness to appropriately align the DLA enterprise architecture with this office’s ongoing efforts to develop a DOD-wide logistics enterprise architecture. Additionally, we recommend that the CIO have the Chief Architect, as appropriate, follow the steps outlined in the CIO Council’s published guide for managing enterprise architectures, including   obtaining executive buy-in and support,   establishing architecture management structure and controls,   defining the architecture process and approach,   developing the baseline architecture, the target architecture, and the sequencing plan, facilitating the use of the architecture, and   maintaining the architecture.
To ensure that, once developed, the enterprise architecture is effectively implemented, we also recommend that the Director require the DLA 21 Executive Board and its subsidiary investment review boards to ensure that only architecturally compliant IT investments are approved and funded, unless the investment decisionmaking authority issues a written waiver in response to a written justification.
Until a DLA enterprise architecture is developed and can be used to effectively guide and constrain DLA IT investment, we also recommend that the DLA Director limit future investment in BSM to only its first incremental release.
To address DLA’s need to incrementally invest in BSM, we recommend that the DLA Director impose three further conditions on investment in BSM beyond its first increment. Specifically, such investment should not occur until plans have been developed and processes implemented for (1) measuring and validating whether ongoing BSM increments are actually producing benefits commensurate with costs, (2) developing economic justifications for each subsequent increment that consider the actual performance of ongoing increments, and (3) ensuring that decisions on each subsequent increment are based on these incremental economic justifications.
To ensure that the approach to DLA’s logistics modernization (as well as that of other DOD component organizations) recognizes the logistics management interdependencies and interrelationships among DOD components and aims to optimize departmentwide performance and accountability, we recommend that the Secretary of Defense direct the Under Secretary of Defense for Acquisition, Technology, and Logistics to (1) expedite development of a DOD-wide logistics management enterprise architecture in a manner consistent with CIO Council published guidance and (2) establish effective controls for ensuring that component investments in modernized logistics systems are compliant with the architecture or are otherwise granted an explicit waiver to this requirement on the basis of analytical justification.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report (reprinted in appendix II), the acting Deputy Under Secretary of Defense for Logistics and Materiel Readiness agreed with us on the importance of developing and maintaining enterprise architectures and making BSM investment decisions incrementally, concurring with eight of our ten recommendations. DOD also acknowledged that the absence of these IT management controls increased program risk. Despite these areas of substantial agreement, the Deputy Under Secretary disagreed with our recommendations that the DLA Director limit future investment in BSM to the first incremental release until (1) a “full-blown” (as DOD termed it) DLA enterprise architecture is developed and (2) the BSM program manager has the means in place to measure actual benefit accrual on the first incremental release and to use this information in deciding on investment in the next release.
According to DOD, our findings and conclusions regarding the overall level of risk to the successful achievement of BSM and other DOD logistics modernization program objectives are not supported by the single risk factor addressed in this report—the use of enterprise architectures. DOD’s stated view is that the risks associated with delaying BSM are greater than the risks of modernizing while concurrently developing the architecture. DOD also stated that while the conditions that we recommended regarding measurement and use of actual benefit accrual are valuable and will be performed by DOD, they should not be performed by the program manager and should not be on the critical path for undertaking future incremental releases.
While DOD agrees that the absence of an “end-to-end” enterprise architecture for DLA is a risk, DOD strongly believes that this risk is amply mitigated by measures already taken. As examples, the Deputy Under Secretary’s comments refer to the preparation of an enterprisewide logistics implementation plan for community data management services that defines the future information and business rules interaction among DOD components; the existence of an “as is” logistics architecture (documented during the Y2K campaign); the recent establishment of an OSD-led enterprise integration consortium that, according to the comments, will ensure optimal collaboration among programs; and the commercial architecture and process reengineering procedures being followed by DLA’s BSM program.
Notwithstanding the considerable level of agreement between DOD and us on this report, we have a differing view on the level of risk facing DOD logistics modernization programs in general, and BSM in particular, in the absence of the two risk mitigators addressed in this report: (1) effective enterprise architecture development and implementation and (2) effective incremental investment management.
Our experience in reviewing other federal agency modernization programs has shown that the absence of these program management controls are two of the root causes for other programs’ lack of success.  The two recommendations that DOD disagrees with are intended to strike a reasonable balance between the risk that DOD will suffer the same fate as these other agencies and the risk that DOD points out of delaying BSM. These recommendations permit DLA to make progress on the first BSM increment while developing and implementing the two missing management controls and thus becoming better equipped to move forward on subsequent increments. Therefore, we continue to believe that DOD should limit investment in BSM beyond the first increment until certain conditions are met.
We agree with the Deputy Under Secretary that the risk mitigation strategies that DOD has followed can lower the risk associated with the lack of a complete enterprise architecture. However, we continue to believe that these efforts are not a sufficient substitute for having an explicit enterprise architecture and effectively using it to guide and constrain modernization investment decisions. At most, these mitigation strategies should be viewed as temporary controls until the architecture is available, as well as useful starting points for the development and use of an enterprise architecture.
With regard to risk mitigation, the Deputy Under Secretary comments further that commercial implementation of enterprise resource planning (ERP) is a proven means for achieving an enterprise business strategy, adding that a key requirement of DLA’s BSM program is to move to a commercial approach for achieving its end-state enterprise architecture. We agree that the use of commercial products can be a less risky approach than the development of custom applications, but we believe that success still depends on effective implementation and in particular on ensuring that the chosen products meet the needs of the enterprise; this requirement is best fulfilled by having and using an enterprise architecture. The main point of our recommendations is that DLA has yet to define its end-state enterprise architecture and therefore cannot know whether its chosen ERP solution will satisfy requirements beyond the materiel management business area. Accordingly, we stand by our recommendation in this area, which is intended to significantly lower the risk for the BSM program in relation to its role in developing DLA’s enterprise architecture, not just in achieving DLA’s requirements for modernizing its materiel management systems.
We would add that DOD’s characterization of the enterprise architecture that we recommend as “full-blown” and “end-to-end” suggests that we are prescribing the full depth and detail of this architecture. We are not. In fact, effective architecture management practices recognize that there is no one-size-fits-all architecture solution and that the driving goal in developing an architecture is to make it useful for the task at hand, meaning that its depth and detail will vary from enterprise to enterprise.
The Deputy Under Secretary also disagreed with our recommendation to impose additional conditions on investment in BSM beyond its first increment, specifically concerning the development of processes for performing cost/benefit analyses of ongoing and completed increments. According to the Deputy Under Secretary, the BSM program does plan to prepare a business case to support each of four scheduled releases and to include in each of these increments a projection of business process performance and an evaluation of empirical results of the program to that point. These are the very activities that we recommend as conditions for investing beyond the first increment. The Deputy Under Secretary’s disagreement with our recommendation appears to center on who should be required to measure and use actual benefit accrual in making decisions on continuing the program to the next increment. It is not our intent to suggest that only the BSM program manager should measure and use actual benefit accrual or that one increment must be fully completed and fully evaluated before another can begin. In fact, effective incremental investment management necessitates some overlap among increments, and our recommendations do not contradict this. On the matter of who should measure and validate actual benefit accrual, we are not opposed to DLA assigning this responsibility to someone other than the program manager, and we acknowledge that the program manager should not be solely responsible for ensuring that investment decisions on subsequent increments are based on economic justifications. Accordingly, we have modified the recommendation to reflect this.
DOD also provided specific comments on other aspects of the report. These comments have been incorporated throughout the report where appropriate.
We are sending copies of this report to the Chairmen and Ranking Minority Members of the Senate Appropriations Subcommittee on Defense; the Subcommittee on Readiness and Management Support, Senate Committee on Armed Services; the House Appropriations Subcommittee on Defense; and the Subcommittee on Readiness, House Committee on National Security. We are also sending copies to the Director, Office of Management and Budget, and the Director, Defense Logistics Agency. Copies will be made available to others upon request.
If you have any questions regarding this report, please contact me at (202) 512-6256 or Carl M. Urie, Assistant Director, at (202) 512-6231. We can also be reached by e-mail at hiter@gao.gov and uriec@gao.gov. Other key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine (1) whether DLA is using an enterprise architecture to guide and constrain its investment in its Business Systems Modernization (BSM) program and (2) whether DLA is investing in BSM in an incremental manner. Using enterprise architectures—institutional blueprints for business and technological change—and investing incrementally in large modernization programs are legislative and federal requirements and best industry practices. This review did not address other system modernization best practices, such as whether DLA is employing effective system acquisition process controls, and did not evaluate the specific commercial, off-the-shelf product that DLA chose as its BSM system solution.
To determine whether DLA has an enterprise architecture to guide BSM, we identified the DOD organizations involved in efforts to reform and modernize DOD logistics management operations and systems, as well as organizations responsible for DOD policy and guidance on enterprise architectures, including the Office of the Under Secretary of Defense (Acquisition, Technology, and Logistics); the Director of Logistics Systems Modernization; the DLA Logistics Operations Directorate; and the DLA Director of Information Operations, who is in charge of corporate information policy. From each of these organizations, we solicited information on plans and activities that defined the form and content of these reform and modernization efforts. We then questioned officials from each organization about planned and existing architectural artifacts and obtained copies of all such plans and artifacts. Next, we analyzed the information provided, including DOD’s Joint Vision 2020, 21st Century Logistics: DOD Logistics Strategic Plan, and available DLA enterprise architecture artifacts, against DOD’s C4ISR Architecture Framework to determine the extent to which these organizations individually or collectively had produced architectural artifacts that satisfied DOD requirements. In our analysis, we also considered other published architectural guidance, including Office of Management and Budget memorandums and the Federal Enterprise Architecture Framework of the Chief Information Officers Council.
We also obtained and reviewed the federal CIO Council’s A Practical Guide to Federal Enterprise Architecture, Version 1.0, as well as published information on the enterprise architecture best practices that the guide is based upon. We compared DLA’s Enterprise Information Architecture Plan (draft, 28 July 2000) with this guidance.
To determine whether DLA is following an incremental investment strategy for BSM, we compared DLA’s Single Acquisition Management Plan, Operational Requirements Document, and other program-related material against the Clinger-Cohen Act, OMB’s Capital Programming Guide, and the associated assessment method. We evaluated policies, procedures, and guidance related to DLA’s IT modular contracting activities. We evaluated task order plans and performance data against commonly accepted IT investment methodologies.
Additionally, we reviewed economic analysis of DLA’s BSM program and the related cost justifications. We also used DOD’s Logistics Transformation guide and the Defense Fiscal Year 2001 Budget to obtain information about other DOD and military service supply-chain logistics initiatives.
We conducted our work at DLA headquarters, located at Fort Belvoir, VA, from February to April 2001 in accordance with generally accepted government auditing standards.
section_title: Appendix II: Comments From the Department of Defense
subsections: section_title: GAO Comments
paragraphs: 1.  We agree that the BSM program should not be held “hostage” to the architecture and that the program and the architecture can proceed in parallel, but only to a point. As we have recommended, this point is the start of the next BSM increment, which according to DLA is August/September 2002. Our recommendation affords DLA ample time to develop and use the missing architectural definition without delaying the program. 2.  We disagree that this report draws conclusions about the program’s “overall” risk based on a single risk factor. Our report is not intended to and does not provide a comprehensive BSM risk profile. Rather, it identifies important risks associated with the two IT management controls that were the focus of our work: (1) the use of an enterprise architecture and (2) the use of incremental investment controls. Further, while we acknowledge that DLA has taken risk mitigation measures, as cited in the draft report provided to DOD for comment, these measures are not a sufficient substitute for having an explicit enterprise architecture and effectively using it to guide and constrain modernization investment decisions. At most, they should be viewed as temporary controls until the architecture is available. In fact, while DOD believes that it cannot delay modernization, it nevertheless states in its comments that it supports the need for enterprise architectures and the integration of these architectures into its investment management processes. 3.  We agree that one should not conclude from this report that there is unacceptable risk associated with modernizing systems while developing an enterprise architecture. We also agree that there are risks associated with delaying modernization. Accordingly, our recommendation strikes a reasonable balance between the need for an architecture and the need to modernize legacy systems by allowing DOD to continue modernization while it develops an architecture. However, it does not allow the two to proceed in parallel indefinitely, because the longer DLA proceeds without an architecture, the greater the risks will become. 4.  We do not disagree and our report does not question the use of an ERP solution. Our point is that DLA has yet to define its end-state enterprise architecture and therefore cannot be sure that it is implementing its chosen ERP solution in a manner that will not suboptimize DLA enterprisewide performance and accountability, in order to optimize performance and accountability of one business area. 5.  See comments 1 and 2. Also, we do not agree that BSM is currently guided by sound architectural discipline. Such discipline would be evidenced by the existence and use of architectural artifacts developed in accordance with the practices outlined in CIO Council guidance.While we acknowledge the existence of some enterprise architecture information, as cited in the draft report provided to DOD for comment, and state in this report that that information would be valuable and useful in building a DLA-wide architecture, the fact remains that DLA does not currently have such an architecture. 6.  We acknowledge that the Deputy Under Secretary of Defense for Logistics and Materiel Readiness has pursued risk-reducing strategies, as noted in this report. However, they are not a sufficient substitute for a DOD-wide logistics architecture. 7.  We agree. The incremental investment actions described, if effectively implemented, are consistent with our recommendations. 8.  See comments 1 and 2. 9.  See comment 1. 10. We generally agree. The activities described in the comment are the very activities that we recommend as conditions for investing beyond the first increment. Further, it was not our intent to suggest that only the BSM program manager should measure and use actual benefit accrual or that one increment must be fully completed and fully evaluated before another can begin. In fact, effective incremental investment management necessitates some overlap among increments, and our recommendations do not preclude this. On the matter of who should measure and validate actual benefit accrual, we are not opposed to DLA assigning this responsibility to someone other than the program manager, and we acknowledge that the program manager is not the appropriate person to ensure that investment decisions on subsequent increments are based on economic justifications. Accordingly, we have slightly modified the recommendation and clarified language elsewhere in the report to reflect this. 11. We agree. We have changed the title of the report. 12. We disagree. The statement in our report is supported by DLA’s draft Enterprise Information Architecture Plan, which clearly shows that the initial component of DLA’s enterprise architecture will be, by design, limited to the materiel management business area. The plan also shows that DLA’s other business areas are to be incorporated into the enterprise architecture following completion of the BSM program, which, according to BSM documents, is 5 years from now. 13. Our report has given the department credit for its DOD Logistics Strategic Plan, as well as Joint Vision 2020 and other strategic planning directions. However, as previously stated, these plans are not a sufficient substitute for an effective DOD-wide logistics management enterprise architecture. Moreover, in its comments, DOD states that having an architecture-based approach to investing in modernized systems is a more efficient means to optimizing enterprise performance and accountability. DOD also states that having an architecture would provide a more efficient and effective means for developing and overseeing the implementation of a common vision. 14. We disagree. While we have recognized these “interim measures” in our report, our position is that these are just what DOD terms them— temporary controls until DOD has in place the architecture-based approach to modernization that its comments state are needed to achieve end-to-end optimization. See also comment 3. 15. We disagree. Notwithstanding DLA’s recent commitments to incrementally invest in BSM, DLA has so far treated BSM as a single investment decision. The economic analysis used to justify DLA’s decision to invest in BSM is dated April 2000, and this analysis was for the entire program. We requested any further economic analyses developed to justify the ongoing BSM increment as well as any plans to produce such analyses. DLA did not provide either. In June 2001, DLA provided an updated economic analysis, dated March 30, 2001, which continued to treat BSM as a whole. DOD’s comments are consistent with statements made by BSM officials at the conclusion of our review, which were cited in the draft report. We are encouraged by DOD’s comment that DLA will prepare a business case for each release and will include a projection of business process performance as well as a retrospective evaluation of empirical results of the program to that point. 16. We have modified our report to reflect this change. 17. We disagree. As we state in the report, enterprise architectures are essential tools for effectively and efficiently reengineering business processes and for implementing and evolving their supporting systems. Their development, implementation, and maintenance are recognized hallmarks of successful public and private sector organizations. Congress, OMB, and the federal CIO Council have recognized the importance of enterprise architectures. 18. We disagree. See comments 2 and 6. Further, our report does not state that the lack of an enterprise architecture will result in “bad” effects. Our report concludes that this lack increases the probability of such effects, and our report recognizes that other IT management weaknesses can also be contributing factors. 19. See comment 5. 20. See comment 12. 21. We disagree that our report implies that there is little interaction and consideration of the enterprise cataloging requirements. Our report refers to DLA’s expectation that some aspects of other business functions would be included in the BSM blueprint. At the time of our review, however, DLA recognized that postponing some business functions might lead to disconnects, requiring later linkages between BSM and other business areas to produce an architecture with an enterprisewide scope. Our point is that because these functions are interrelated, it makes sense to construct an enterprise architecture to guide and constrain BSM implementation and thereby optimize DLA- wide performance and accountability. 22. See comments 2 and 21. 23. See comment 13. 24. We have modified the report to reflect this comment. 25. We have modified the report to incorporate the additional context provided in this comment and to recognize when efforts to develop a DOD-wide logistics enterprise architecture began. 26. We disagree that we have been provided a copy of the document defining the mission-critical logistics “threads” used in DOD’s Year 2000 testing “in their current status.” While we did receive in 1999 a copy of the document defining these mission-critical logistics threads, the Director also told us that these were being modified in light of ongoing modernizations, and DOD’s comments acknowledge that they have since been modified. 27. We have modified the report to reflect this comment. 28. See comment 23. Also, we have slightly modified the language of our report.
section_title: Appendix III: GAO Contact and Staff Acknowledgment
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the person named above, Katherine Chu-Hickman, John Christian, Barbara Collier, and Greg Donnellon made key contributions to this report.