Defense Communications: Performance Measures Needed to Ensure DISN Program Success
Published: 1996-11-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to congressional request, GAO reviewed the steps taken by the Department of Defense (DOD) in selecting and implementing its acquisition strategy for the Defense Information System Network (DISN) Continental United States (CONUS), focusing on whether: (1) DOD considered alternative approaches, such as use of an integrated bid, in its selection of an acquisition strategy; and (2) the selected acquisition strategy will yield the best value to the government over DISN's life-cycle.
section_title: What GAO Found
paragraphs: GAO found that: (1) DOD considered several options prior to selecting an acquisition strategy for DISN, including an approach that would have involved using a single comprehensive service provider to furnish an integrated set of services to the government and another one that involved separately acquiring component services with the government integrating those components itself; (2) DOD considered the advantages and disadvantages of each option in terms of five factors: requirements; technology enhancement; schedule; management; and cost; (3) after evaluating its options and receiving industry comments on its draft request for proposals, DOD ultimately decided on an approach that calls for the Defense Information Systems Agency (DISA) to separately acquire and integrate component services itself, using contracts awarded on a staggered schedule; (4) DOD believes that this strategy will best meet national security needs at a reasonable cost; (5) in reviewing DOD's DISN efforts in 1995, GAO reported that DOD had yet to define the program's minimal acceptable requirements; (6) GAO also reported that DOD had not yet developed an estimate of what it would cost to acquire, operate, and sustain the DISN infrastructure; (7) without this information, DOD has no objective cost and performance baseline for measuring DISN's success; (8) without this baseline, GAO cannot determine whether the selected acquisition strategy will yield the best value to the government over the course of DISN's life cycle, which is estimated to be over 10 years; (9) once this baseline is developed, DOD must also establish effective measures for tracking DISN's progress; (10) at present, DOD is far from meeting federal requirements for establishing performance measures; and (11) by developing measures that focus on benefits, costs, and risks, DOD management can target problem areas, highlight successes, and ensure DISN meets its cost and performance goals.

section_title: Letter
paragraphs: The Department of Defense (DOD) is in the process of acquiring the transmission services and switching technology needed to construct its future telecommunications network known as the Defense Information Systems Network (DISN). Defense’s overall DISN strategy calls for the Defense Information Systems Agency (DISA) to acquire and implement DISN transmission and switching services across three regions: the continental United States (CONUS), the Pacific, and Europe. These long-haul services will, in turn, interconnect Defense’s base-level and deployed communications networks.
This report responds to your request that we review the steps taken by Defense in selecting and implementing its acquisition strategy for DISN CONUS, which Defense is focusing on first. Specifically, you asked that we determine whether (1) Defense considered alternative approaches, such as use of an integrated bid, in its selection of an acquisition strategy and (2) the selected acquisition strategy will yield the best value to the government over DISN’s life-cycle. Your office also requested that we determine what performance measures Defense managers are using or plan to use to objectively evaluate the outcomes and effectiveness of their DISN implementation efforts.
subsections: section_title: Background
paragraphs: Defense has long operated multiple telecommunications systems to meet an array of mission needs, ranging from the command and control of military forces to its payroll and logistics support functions. Because military services and other Defense agencies independently procured and operated their own networks, Defense’s communications environment has been fragmented and redundant. To improve the effectiveness and efficiencies of its military communications services, Defense began in 1991 to plan and implement DISN to serve as the Department’s primary worldwide telecommunications and information transfer network to support national security and defense operations.
Defense’s DISN strategy focuses on replacing its older data communications systems, using emerging technologies and cost-effective acquisition strategies that provide secure and interoperable voice, data, video, and imagery communications services in support of military operations. Under Defense’s DISN concept, the military services and Defense agencies will still be responsible for acquiring telecommunications services for their local bases and installations, as well as deployed communications networks.
DISA will be responsible for acquiring the long-haul services that will interconnect these base-level and deployed networks within and between the continental United States, Europe, and the Pacific. DISA’s current efforts focus on acquiring and implementing DISN CONUS services. For 10 years, Defense users obtained switched voice, data, video teleconferencing, and transmission services within the United States through the Defense Commercial Telecommunications Network (DCTN) contract with AT&T. The DCTN contract expired in February 1996. Since then, these services have been provided through a follow-on, sole-source DISN Transition Contract (DTC) with AT&T until Defense can fully implement its new DISN services. Defense estimates that DTC costs are approximately $18.5 million per month.
In July 1995, we reported on Defense’s efforts to plan and implement DISN. At that time, we recommended that Defense ensure that DISN plans and program decisions were based on a validated statement of DISN’s operational requirements. By defining the minimal acceptable requirements for DISN as well as the critical technical characteristics, the operational requirements document would provide the basis for determining DISN’s effectiveness. We also recommended that Defense develop an estimate of the acquisition, operations, maintenance, and support costs for DISN over its life-cycle. While Defense concurred with these recommendations, it has not yet completed either action.
Nevertheless, given the expiration of its DCTN contract in February 1996, and its desire to limit the term of the sole-source DISN Transition Contract, DISA is proceeding with its DISN implementation efforts and has issued four RFPs supporting DISN’s implementation:
DISN Support Services - Global, to provide engineering, operations, network management, and other support services worldwide.
DISN Switched/Bandwidth Manager Services - Continental United States (CONUS), to provide the capability to switch network traffic and provide bandwidth manager devices at designated service delivery points within the continental United States.
DISN Transmission Services - CONUS, to provide access transmission services and transmission services connecting the bandwidth managers and switches provided under the switched/bandwidth manager contract, and to connect Defense installations with the DISN network.
DISN Video Services - Global, to provide worldwide video teleconferencing through three video network hubs located in the continental United States.
The timetable for receipt of proposals and contract awards is shown in table 1.
DISA awarded the support services contract to Boeing Information Services, Inc., in June 1996, and awarded the switched/bandwidth manager services contract to MCI Corporation in August 1996. The evaluation of these proposals and subsequent contract awards addressed four factors: cost, technical, management, and past performance. DISA plans to award the video services contract on the same basis. Because transmission is a basic commodity service, Defense advised that it intends to award the transmission services contract primarily on the basis of lowest price. Defense plans full implementation of its DISN system within the continental United States by July 1997.
The switched/bandwidth manager, transmission services, and video services acquisitions were subject to a bid protest in December 1995 by AT&T, which was adjudicated by the General Accounting Office (GAO). In this protest, AT&T argued that DISA arbitrarily refused to allow offerors to submit and have evaluated a single, comprehensive proposal, what AT&T termed an “integrated bid,” as an alternative to submitting individual proposals under each RFP. GAO’s decision, issued on May 1, 1996, upheld the legality of the acquisition strategy that DISA has followed.
section_title: Objectives, Scope, and Methodology
paragraphs: To obtain information about Defense’s acquisition strategy, and the steps taken by Defense in determining and selecting that strategy, we obtained and analyzed copies of the DISN solicitations from DISA staff in the Washington, D.C., area. We analyzed studies prepared by DISA staff during April and May 1995 that identified and evaluated DISN acquisition alternatives. We reviewed Defense’s DISN architecture and were briefed on steps taken to develop the DISN design by engineering staff at DISA’s Joint Interoperability and Engineering Organization, Center for Systems Engineering, in Reston, Virginia. In addition, in conducting our review, we used supporting documentation from our bid protest decision.
To obtain information about the specific evaluation methods and factors used to select a DISN acquisition strategy, we interviewed several DISA officials including the DISN Program Manager and the DISN Contracting Officer in Arlington, Virginia. Our review was conducted from August 1996 through October 1996 in accordance with generally accepted government auditing standards.
section_title: Defense’s Analysis of DISN Acquisition Alternatives
paragraphs: In developing its DISN acquisition approach, Defense considered several acquisition alternatives in April and May 1995 including one—using a single contractor to furnish a comprehensive set of services to the government—that is similar to the integrated approach that AT&T had advocated. Defense also evaluated the costs and benefits of separately acquiring component services with the government integrating those components itself, and other alternative approaches as well.
In reviewing Defense’s analyses of alternatives, we found that Defense evaluated the advantages and disadvantages of each acquisition alternative in terms of relative cost and how it (1) met DISN requirements, (2) facilitated technology insertion and enhancement, (3) could be implemented within schedule constraints, and (4) supported Defense’s control of the network.
DISA selected an acquisition strategy that divided the acquisition into four components with four separately awarded contracts. Under this plan, DISA, with the assistance of the support services contractor, would acquire, integrate, operate, and maintain the separate DISN components rather than employ a comprehensive service provider to integrate and operate DISN. Defense believed that breaking the program into functional components facilitated control over network interoperability, integration, surge capacity, technology insertion, and security. It also concluded that by breaking the program into pieces, more vendors could bid for contracts, thus increasing competition. Further, in Defense’s view, multiple contracts with frequent options made it easier to negotiate technological upgrades, and created incentives for vendors to maintain high standards of performance. Finally, Defense believed that the strategy encouraged vendors to offer their lowest prices on each separate contract instead of just offering prices that were averaged across the entire network.
After issuing solicitations to implement this strategy, Defense received comments from industry contending that vendors could offer significant economies if they could submit one comprehensive, or integrated, bid for all of the business offered under the switched/bandwidth manager, transmission, and video services RFPs. Defense responded with an approach which staggers contract awards such that a vendor who wins the switched/bandwidth manager contract can use any economies that might accrue to its advantage when bidding for the remaining contracts. According to DISA, this approach enables the government to reap the potential cost savings of an integrated bid while maintaining maximum flexibility for cost-effective technical enhancements and continuing competition over the life of the program.
section_title: Ability of DISN Acquisition Strategy To Yield Best Value
paragraphs: Defense believes that it has selected the acquisition strategy that will yield the best value to the government over the course of DISN’s life cycle. However, Defense lacks the baseline information needed for us to ensure whether this is the case. We recommended in July 1995 that Defense ensure that the DISN approach was based on valid operational requirements and that it identify the additional life-cycle acquisition, maintenance, and support costs that would be incurred in developing and operating DISN. In making these recommendations, we concluded that without this important information, Defense would lack a starting point for ensuring that DISN facilities and services effectively and efficiently met their requirements. While Defense concurred with our recommendations, it has not fully implemented them. Given the current advanced state of the DISN acquisition and the need to replace the high-cost transition contract, we are not questioning the need to continue to move forward with DISN. However, Defense still needs this baseline information to gauge the performance of DISN as it is being implemented.
DISN program officials in DISA and staff from the Office of the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence have told us that DISN’s requirements are known and documented because they are based on the requirements developed for Defense’s current communications systems. However, we believe that the operational requirements in the existing systems are not valid for DISN because they do not consider several important factors.
First, with the growth of worldwide telecommunications networks such as the Internet, the information warfare threat to Defense, and thus the need for security requirements, has significantly increased in the past decade. For example, we recently reported that Defense may have experienced as many as 250,000 computer attacks last year and that Defense estimates that these attacks are successful 65 percent of the time. We also reported that the number of attacks is doubling each year, as Internet use increases along with the sophistication of computer attackers and their tools.Second, since the new strategy calls for diversifying contractors, integration risks are significantly higher than those accompanying the previous contract and system management is much more complex. Third, users now have greater expectations for network services as telecommunications technology has made significant strides in recent years. Taken together, these changes clearly demonstrate the need for Defense to document and validate with DISN users the operational requirements for the new strategy.
section_title: Defense Lacks Performance Measures Critical to DISN’s Success
paragraphs: By better establishing its operational requirements and life cycle costs for DISN, Defense would lay the groundwork for assessing whether the system is meeting its cost and performance goals. The next step would be to develop effective measures for tracking DISN’s progress against this baseline cost and performance information. Defense has not yet established any performance measures that would allow it to track whether DISN is meeting its objectives. Since Defense plans to begin implementing DISN CONUS in less than 8 months, the absence of these measures raises concerns that the Department will not be able to effectively manage DISN’s implementation and operation.
Establishing good performance measures is not only critical because of the risks confronting the DISN program, it is central to the success of any significant information system undertaking. We have previously reported, for example, that successful organizations rely heavily upon performance measures to achieve mission goals and objectives, quantify problems, evaluate alternatives, allocate resources, track progress, and learn from mistakes. For service-oriented programs such as DISN, these may include such measures as the percent of mission improvements resulting from the new service in terms of cost, time, quality, and quantity; the percent of customers satisfied with certain telecommunications services; or the number of problems resolved within target times. Once the right measures are chosen, they help management target problem areas, highlight successes, and generally increase the rate of performance improvement through enhanced learning.
Further, several statutory requirements call for Defense to define cost, schedule, and performance goals for major defense acquisition programs and for each phase of the acquisition cycle of such programs. These include the Federal Acquisition Streamlining Act (FASA) of 1994 and the recently enacted Clinger-Cohen Act of 1996. The requirement to establish program cost estimates and performance measures of operational effectiveness are also embodied in Defense acquisition guidance.
At present, Defense is far from meeting any of these requirements. For example, even basic objectives, such as DISN’s ability to provide its users with the needed quality and volume of communications services, have not been validated by users and lack evaluation criteria upon which to measure success. Without this type of information, Defense has no way of knowing whether it will be spending billions of dollars acquiring, operating, and maintaining DISN facilities and services that efficiently and effectively meet its needs.
section_title: Conclusions
paragraphs: Defense is striving to fully implement its DISN CONUS system by July 1997. However, it has yet to establish the basic cost and performance baseline information critical to laying the groundwork for assessing DISN’s success. We continue to believe that Defense should expeditiously implement our previous recommendation to develop and document DISN operational requirements and to identify DISN life cycle costs. In addition, Defense has not established performance measures that would determine how the implementation of this multibillion dollar initiative measures up to its cost and operational goals. Establishing such measures now for DISN would markedly improve DOD’s and the Congress’ ability to manage and oversee implementation of this system by providing the basis for independent analysis and evaluation.
section_title: Recommendation
paragraphs: We recommend that the Secretary of Defense direct the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence to establish the objective measures needed to gauge DISN’s success. At a minimum, these measures should include the concerns of DISN customers and should correspond to the five factors—requirements, technology enhancement, schedule, management, and cost—that DISA used to select its acquisition strategy.
section_title: Agency Comments and Our Evaluation
paragraphs: We obtained written comments on a draft of this report and have incorporated those comments where appropriate. These comments are presented in appendix I.
In commenting on the draft report, Defense concurred with our recommendation. We are encouraged that Defense intends to develop cost estimates and performance measures for major DISN components from this point forward. It is likewise important that Defense does so for the DISN-CONUS component currently being implemented. As stated in our report, these actions are critical in order for Defense to have an objective cost and performance baseline for measuring the success of this acquisition.
As agreed with your office, we will send copies of this report to the Ranking Minority Member of the Senate Committee on Governmental Affairs, Chairman and Ranking Minority Member of the House Committee on Government Reform and Oversight, other interested congressional committees, the Secretary of Defense, and the Director of the Office of Management and Budget. Copies will be sent to others upon request. Please contact me at (202) 512-6240 if you or your staff have any questions. Major contributors to this report are listed in appendix II.
section_title: Comments From the Department of Defense
section_title: Major Contributors to This Report
subsections: section_title: Accounting and Information Management Division, Washington, D.C.
paragraphs: Linda D. Koontz, Associate Director Franklin W. Deffer, Assistant Director Kevin E. Conway, Senior Information Systems Analyst Mary T. Marshall, Information Systems Analyst Cristina T. Chaplain, Communications Analyst The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.