Defense Acquisition: Advanced Concept Technology Demonstration Program Can Be Improved
Published: 1998-10-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the current Advanced Concept Technology Demonstration (ACTD) program, focusing on: (1) whether the selection process includes criteria that are adequate to ensure that only mature technologies are selected for ACTD prototypes; (2) whether guidance on transitioning to the normal acquisition process ensures that a prototype appropriately completes product and concept development and testing before entering production; and (3) the Department of Defense's (DOD) current practice of procuring more ACTD prototypes than needed to assess the military utility of a mature technology.
section_title: What GAO Found
paragraphs: GAO noted that: (1) through the determination of military value of mature technologies and their use in the acquisition process, ACTDs have the potential to reduce the time to develop and acquire weapon systems; (2) however, several aspects of the ACTD program can be improved; (3) DOD's process for selecting ACTD candidates does not include adequate criteria for assessing the maturity of the proposed technology and has resulted in the approval of ACTD projects that included immature technology; (4) DOD has improved its guidance on the maturity of the technologies to be used in ACTD projects but the revised guidance describes several types of exceptions under which immature technologies may be used; (5) where DOD approves immature technologies as ACTD program candidates and time is spent conducting developmental activities, the goal of reduced acquisition cycle time will not be realized; (6) further, guidance on entering technologies into the normal acquisition process is not sufficient to ensure that a prototype completes product and concept development and testing before entering production; (7) the guidance does not mention the circumstances when transition to development may be appropriate or the kinds of developmental activities that may be appropriate; (8) while commercial items that do not require any further development could proceed directly to production, many ACTDs may still need to enter the engineering and manufacturing development phase to proceed with product and concept development and testing before production begins; (9) through the ACTD early user demonstration, DOD is expected to obtain more detailed knowledge about its technologies before entering into the acquisition process; (10) however, in the one case in which an ACTD has proceeded into production, DOD made that decision before completing product and concept development and testing, thereby accepting programmatic risks that could offset the schedule and other benefits gained through the ACTD process; (11) DOD's current practice of procuring prototypes beyond those needed for the basic ACTD demonstration and before completing product and concept development and testing is unnecessarily risky; and (12) this practice risks wasting resources on the procurement of items that may not work as expected or may not have sufficient military utility and risks a premature and excessive commitment to production.

section_title: Letter
paragraphs: The Advanced Concept Technology Demonstration (ACTD) program was initiated by the Department of Defense (DOD) to determine the extent to which a given mature technology will improve military capabilities before entering the normal acquisition process and, by using mature technology, reduce the length of time to develop and acquire weapon systems. According to the ACTD program documentation, the ACTD program was established in 1994 in response to recommendations of the 1986 Packard Commission and a 1991 Defense Science Board study. The Packard Commission concluded that major improvements could be made in defense acquisition by emulating the practices of successful commercial companies. Accordingly, the Commission recommended, among other things, building and testing prototypes to assess military utility and provide a basis for realistic cost estimates before committing to acquisition. The Defense Science Board’s study discussed the need for early dialogue between the potential system’s user and producer for a proper analysis of cost, risk, and operational capability.
ACTDs have been the subject of congressional interest since the program’s inception. Congressional committees have expressed concerns about the validity of several technologies selected for the ACTD program and the number of assets procured for the projects. At your request, we assessed the current ACTD program. Specifically, we determined whether (1) the selection process includes criteria that are adequate to ensure that only mature technologies are selected for ACTD prototypes and (2) guidance on transitioning to the normal acquisition process ensures that a prototype appropriately completes product and concept development and testing before entering production. Finally, we assessed DOD’s current practice of procuring more ACTD prototypes than needed to assess the military utility of a mature technology.
subsections: section_title: Background
paragraphs: Through fiscal year 1998, about $172 million has been allocated to the ACTD program and 48 projects have been approved. DOD’s budget request for fiscal year 1999 for the ACTD program is $116.4 million. An additional 10 to 15 projects are expected to be funded in fiscal year 1999.
Under the current ACTD program, DOD builds prototypes to assess the military utility of mature technologies, which are used to reduce or avoid the time and effort usually devoted to technology development. Demonstrations that assess a prototype’s military utility are structured to be completed within 2 to 4 years and require the participation of field users (war fighters).
ACTD projects are not acquisition programs. The ACTD program seeks to provide the war fighter with the opportunity to assess a prototype’s capability in realistic operational scenarios. From this demonstration, the war fighter can refine operational requirements, develop an initial concept of operation, and make a determination of the military utility of the technology before DOD decides whether the technology should enter into the normal acquisition process.
Not all projects will be selected for transition into the normal acquisition process. The user can conclude that the technology (1) does not have sufficient military utility and that acquisition is not warranted or (2) has sufficient utility but that additional procurement is not necessary. Of the 11 ACTD projects completed as of August 1998, 2 were found to have insufficient utility to proceed further, 8 were found to have military utility but no further procurement was found to be needed at the time, and 1 was found to have utility and has transitioned to the normal acquisition process.
ACTD funding is to be used to procure enough prototypes to conduct the basic demonstration of military utility. At the conclusion of the basic demonstration, ACTD projects are expected to provide a residual operational capability for the war fighter. Under the current practice, ACTD funding is also to be available to support continued use of ACTD prototypes that have military utility for a 2-year, post-demonstration period. The 2 years of funding is to support continued use by an operational unit and provide the time needed to separately budget for the acquisition of additional systems. Further, if the ACTD prototypes—such as missiles—will be consumed during the basic demonstration, additional prototypes are to be procured.
As stated in the ACTD guidance, a key to successfully exploiting the results of the demonstration is to enter the appropriate phase of acquisition without loss of momentum. ACTDs are intended to shorten the acquisition cycle by reducing or eliminating technology development and maturation activities during the normal acquisition process. Further, DOD can concentrate more on technology integration and demonstration activities. Time and effort usually devoted to technology development can be significantly reduced or avoided and the subsequent acquisition process reduced accordingly, if the project is deemed to have sufficient military utility.
ACTD candidates are nominated from a variety of sources within the defense community, including the Commanders in Chief, the Joint Chiefs of Staff, the Office of the Secretary of Defense agencies, the services, and the research and development laboratories. The candidates are then reviewed and assessed by staff from the Office of the Deputy Under Secretary of Defense (Advanced Technology). After this initial screening, the remaining candidates are further assessed by a panel of technology experts. The best candidates are then submitted to the Joint Requirements Oversight Council, which assesses their priority. The final determination of the candidates to be funded is made within the Office of the Deputy Under Secretary of Defense (Advanced Technology), with final approval by the Under Secretary of Defense (Acquisition and Technology).
section_title: Inadequate Guidance for Assessing Maturity of Technology
paragraphs: By limiting consideration to prototypes that feature mature technology, the ACTD program avoids the time and risks associated with technology development, concentrating instead on technology integration and demonstration activities. The information gained through the demonstration of the mature technology could provide a good jump start to the normal acquisition process, if the demonstration shows that the technology has sufficient military value. Time and effort usually devoted to technology development could be reduced or avoided and the acquisition process shortened accordingly.
Program officials stated that they have a mechanism in place to ensure that only those projects using mature technology are allowed to become ACTDs. These officials explained that an ACTD candidate’s technology is assessed by high-ranking representatives from the services and the DOD science and technology community before candidates are selected. Program personnel stated that determining technology maturity is important before a candidate is selected because ACTD program funding is not intended to be used for technology development. According to program guidance, the ACTD funding is to be used for (1) costs incurred when existing technology programs are reoriented to support ACTD, (2) costs to procure additional assets for the basic ACTD demonstration, and (3) costs for technical support for 2 years of field operations following the basic ACTD demonstration. We were told that no ACTD money was to be used for technology development activities.
However, the project selection process does not ensure that only mature technologies enter the ACTD program. We found examples where immature technologies were selected and technology development was taking place after the approval and start of the ACTD program. The current operations manager of the Combat Identification project, which began in fiscal year 1996, told us that one of his major concerns has been that some of the ACTD funding was being used for technology development, and not exclusively used for designing and implementing the assessment. However, during the ACTD project, technical or laboratory testing was still necessary to evaluate the acceptability of many of the 12 technologies included in the initial project. Eventually, 6 of the 12 technologies had to be terminated. According to the demonstration manager, 2 of the 6 technologies were terminated because they were immature. According to the manager, that is one of the reasons the project is currently behind schedule.
Another example of the inclusion of immature technology occurred in the Outrider Unmanned Aerial Vehicle project. According to the management plan for the project, one of the individual technologies to be incorporated into the vehicle was a heavy fuel engine. According to a program official, it was later deemed that this individual technology was too immature and an alternate technology had to be used. However, trying to use this immature technology has already caused schedule slippage and cost overruns in the ACTD project.
section_title: Insufficient Guidance for Transitioning ACTDs to the Normal Acquisition Process
paragraphs: To complete the basic demonstration within the prescribed 2 to 4 year period, ACTDs typically use early prototypes. If the demonstrated technology is deemed to have sufficient military utility, many ACTD projects will still need to enter the normal acquisition process to complete product and concept development and testing to determine, for example, whether the system is producible and can meet the user’s suitability needs. These attributes of a system go beyond the ACTD’s demonstration of military utility to address whether the item can meet the full military requirement. Commercial items that do not require any further development could proceed directly to production. However, other non-software related ACTDs should enter the engineering and manufacturing development phase to proceed with product and concept development and testing.
According to ACTD guidance, if further significant development is needed, a system might enter the development portion of the engineering and manufacturing development phase. However, the guidance states that, if the capability is adequate, the ACTD can directly enter production. The guidance does not specifically define what is considered an “adequate capability” to allow an ACTD system to enter low-rate production.
In 1994, we reported on numerous instances of weapon systems that began production prematurely and later experienced significant operational effectiveness or suitability problems. In our best practices report, we reported that typically DOD programs allowed much more technology development to continue into the product development phase than is the case in commercial practices. Turbulence in program outcomes—in the form of production problems and associated cost and schedule increases—was the predictable consequence of DOD’s actions. In contrast, commercial firms gained more knowledge about a product’s technology, performance, and producibility much earlier in the product development process. Commercial firms consider not having this type of knowledge early in the acquisition process an unacceptable risk. In responding to that report, the Secretary of Defense stated that DOD is vigorously pursuing the adoption of such business practices. Specifically, he stated that DOD has taken steps to separate technology development from product development through the use of ACTDs. The ACTD guidance and DOD’s current practice do not appear to reflect this emphasis.
In the case of the Predator ACTD, the one ACTD that has proceeded into production, DOD decided to enter the technology into production before proceeding with product and concept development and testing, thereby accepting programmatic risks that could offset the schedule and other benefits gained through the ACTD process. In the early operational assessment of the Predator’s ACTD demonstration, the Director, Operational Test and Evaluation, did not make a determination of the system’s potential operational effectiveness or suitability. However, the system was found to be deficient in several areas, including mission reliability, documentation, and pilot training. The assessment also noted that the ACTD demonstration was not designed to evaluate several other areas such as system survivability, supportability, target location accuracy, training, and staffing requirements.
The basic ACTD demonstration may have clarified the Predator’s military utility but it did not demonstrate its system requirements or its suitability. Thus, instead of using the knowledge acquired during the demonstration to complete the Predator’s development through the product and concept development and testing stages of acquisition, DOD allowed it to directly enter production.
section_title: Procuring ACTD Prototypes Beyond Those Needed for Basic Demonstration Is Unnecessarily Risky
paragraphs: DOD’s practice is to procure sufficient ACTD prototypes to provide a 2-year residual capability. When it determines that the original prototypes will be consumed during the basic demonstration, additional prototypes are procured for potential use after the basic ACTD demonstration. However, these additional assets—like the basic demonstration prototypes—have not been independently tested to determine their effectiveness and suitability. Procuring additional ACTD prototypes before product and concept development and testing is completed risks wasting resources on the procurement of items that may not work as expected or may not have sufficient military utility. Representatives from the service test agencies did not support this practice and agreed that it had the potential for problems. Without a meaningful independent assessment of a product’s suitability, effectiveness, and survivability, users cannot be assured that it will operate as intended and is supportable.
Congress has expressed concern about the amount of equipment being procured beyond what is needed to conduct the basic ACTD demonstration. Its concern is that DOD is making an excessive commitment to production before military utility is demonstrated and before appropriate concepts of operation are developed. For example, DOD plans to procure 192 Enhanced Fiber Optic Guided missiles at an estimated cost of $27 million and 144 Line-of-Sight Anti-Tank missiles at an estimated cost of $28 million beyond the quantities of missiles required for the ACTD demonstrations—64 and 30 missiles, respectively. The production of these additional missiles will follow the production of the missiles needed for the basic demonstration and will continue on a regular basis throughout the 2-year, post-demonstration period. If the prototypes are deemed to have sufficient military utility, the service involved will be expected to fund the production of additional missiles beyond these quantities. By establishing a regular pattern of procurement in this way, DOD risks committing to a continuing production program before a determination is made about the technology’s military utility and before there is assurance that the system will meet validated requirements and be supportable.
section_title: Conclusions
paragraphs: The strength of the ACTD program is in conducting basic demonstrations of mature technology in military applications before entering the normal acquisition process. This practice could significantly reduce or eliminate the time and effort needed for technology development from the acquisition process. For this to occur, it is essential that DOD use only mature technology in its ACTDs. DOD’s criteria for selecting technologies for ACTD candidates should be clarified to ensure the selection of mature technology with few, if any, exceptions.
Further, ACTDs may not, by themselves, result in an effective and safe deployment of military capability. It is important that product and concept development as well as test and evaluation processes be allowed to proceed before the service commits to the production of the demonstrated technology. If an ACTD project is shown to have military value, the normal acquisition processes can and should be tailored—but not bypassed—before DOD begins production. Lastly, emphasizing the need to complete concept and product development and testing before procuring more items than needed for the basic demonstration would reduce the risk of prematurely starting production.
section_title: Recommendations
paragraphs: We recommend that the Secretary of Defense clarify the ACTD program guidance to (1) ensure the use of mature technology with few, if any, exceptions and (2) describe when transition to the development phase of the acquisition cycle is necessary and the types of development activity that may be appropriate. Further, we recommend that the Secretary of Defense limit the number of prototypes to be procured to the quantities needed for early user demonstrations of mature technology until the item’s product and concept development and testing have been completed.
section_title: Agency Comments and Our Evaluation
paragraphs: “. . . new technologies proposed for incorporation into an ACTD should not be in the 6.1 (basic research) or 6.2 (applied research) budget categories. Furthermore, the technologies must have been successfully demonstrated at the subsystem or component level and at the required performance level prior to the start of the ACTD.”
While this guidance is improved over previous versions, the new guidance permits the selection of immature technology—even as the primary or core technology—provided that it is demonstrated prior to the ACTD demonstration. Also, some recent ACTD projects have been approved without the technologies having been identified. Moreover, the new guidance goes on to describe several types of exceptions under which immature technologies may be permitted to be used in an ACTD. As our report states, the use of immature technologies has delayed programs and we continue to believe DOD needs to focus the ACTD program on the use of mature technology with few, if any exceptions.
DOD also agreed that some but not all ACTDs may require additional product and concept development before proceeding into production. DOD states that a mandatory engineering and manufacturing development phase would not be appropriate for all ACTD projects. We agree, however, the existing ACTD guidance focuses on the transition directly to production and provides too little guidance concerning a possible transition to development. As stated in our recommendation, the guidance should specify when a transition to development may be appropriate and the kinds of developmental activities that may be appropriate.
Finally, DOD agreed that the number of ACTD prototypes to be procured should be limited until the Under Secretary can confirm that sufficient testing has been satisfactorily completed to support any additional procurement. We agree with DOD that test results should form the basis for starting limited procurement. However, DOD’s equating a determination of military utility (based on an ACTD demonstration) with a determination of a system’s readiness to begin production is inappropriate because production decisions require more testing data. We have long held the view and have consistently recommended that DOD use extreme caution to avoid premature commitments to production.
section_title: Scope and Methodology
paragraphs: To determine the adequacy of the ACTD program’s selection criteria in assessing technology maturity and guidance for transitioning to the normal acquisition process, we reviewed existing program guidance, published reports, the Office of the Inspector General’s April 1997 ACTD report, and the recommendations of the 1986 Packard commission and the 1996 Defense Science Board. We discussed selection criteria, transitioning to the acquisition process, and all 34 of the individual ACTD programs approved through fiscal year 1997 with representatives from the Office of the Deputy Under Secretary of Defense (Advanced Technology), Washington, D.C.; the Army’s Deputy Chief of Staff for Operations and Plans, Office of Science and Technology Programs, Washington, D.C.; the Air Force’s Director for Operational Requirements, Rosslyn, Virginia; the Navy’s Requirements and Acquisition Support Branch, Washington, D.C.; the Marine Corps’ Combat Development Command Office of Science and Innovation, Quantico, Virginia; the Joint Staff’s Acquisition and Technology Division and Requirements Assessment Integration Division, Washington, D.C.; and the Office of the Commander in Chief, U.S. Atlantic Command, Norfolk, Virginia.
We discussed the issue of procuring additional residual assets for early deployment with representatives from DOD’s Office of the Director, Operational Test and Evaluation, Washington, D.C.; the Army’s Test and Evaluation Management Agency, Washington, D.C.; the Army’s Operational Test and Evaluation Command, Alexandria, Virginia; the Marine Corps’ Operational Test and Evaluation Activity, Quantico, Virginia; the Air Force’s Test and Evaluation Directorate, Washington, D.C.; and the Navy’s Commander, Operational Test and Evaluation Force, Norfolk, Virginia.
We conducted our review from September 1997 to July 1998 in accordance with generally accepted government auditing standards.
As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies to other interested congressional committees; the Secretaries of Defense, the Army, the Air Force, and the Navy; the Commandant of the Marine Corps; the Director, Office of Management and Budget; and other interested parties. We will also make copies available to others upon request.
Please contact me at (202) 512-4841, if you or your staff have any questions concerning this report. The major contributors to this report were  Bill Graveline, Laura Durland, and John Randall.
section_title: Comments From the Department of Defense
paragraphs: The following are GAO’s comments on the Department of Defense’s (DOD) letter, dated August 31, 1998.
subsections: section_title: GAO Comments
paragraphs: “. . . new technologies proposed for incorporation into an ACTD should not be in the 6.1 (basic research) or 6.2 (applied research) budget categories. Furthermore, the technologies must have been successfully demonstrated at the subsystem or component level and at the required performance level prior to the start of the ACTD.” “. . .Strategies and approaches are described to facilitate transitioning from an ACTD to the acquisition process as defined in DOD 5000.2R. The suggested approaches are based on lessons learned. The focus of the suggestions are ACTDs that are planned—if successful—to enter the acquisition process at the start of LRIP.”
Although there is a basic recognition that the transition to development may be possible, the bulk of the guidance is on how and when to transition to production. As pointed out in the report, the guidance does not describe when a transition to development or what types of development activity may be appropriate. In our view, the guidance needs to be more balanced between the possibility of transition to development and the transition of ACTD projects directly to production. 5. As discussed in the report, the independent operational testing agencies are observers in the ACTD demonstrations and not active participants. While the Office of the Director of Operational Test and Evaluation was an observer during the Predator demonstration, a determination was not made that Predator was potentially effective and suitable. 6. We agree that ACTDs address the technology’s suitability. However, the ACTD focus on suitability is in a very general sense and extensive data is not collected on the system’s reliability, maintainability, and other aspects of suitability needed to support production decisions. 7. As our report states, the Predator was rushed into low-rate initial production prematurely given the limited amount of testing conducted at that time and the problems that were uncovered during that limited testing. 8. DOD’s equating a determination of military utility (based on an ACTD demonstration) with a determination of a system’s readiness to begin production is inappropriate because production decisions require more testing data. During our review, we noted that sufficient information was not obtained from an ACTD demonstration to make a commitment to limited production. Commercial practice would dictate that much more information be obtained about a product’s effectiveness, suitability, producibility, or supportability before such a commitment is made. We believe the ACTD guidance needs to be more balanced and should anticipate that ACTD prototypes may need to conduct more product and concept development and testing prior to production. We have long held the view and have consistently recommended that DOD use extreme caution to avoid premature commitments to production. 9. We are not suggesting that a lengthy development phase be conducted on all ACTD products nor, as DOD appears to suggest, that an ACTD prototype may be ready to start limited production immediately after its basic demonstration. As DOD stated in its intent to establish the ACTD program, we believe the benefit of the ACTD process is in eliminating or reducing technology development, not in making early commitments to production or in postponing product and concept development and testing activities until after production starts. 10. While ACTD demonstrations are performed in operational environments, they are not operational tests. During the course of our work, we held several discussions with officials from the operational test community. Those officials were in favor of the user demonstrations featured in the ACTD program, but none considered those demonstrations as substitutes for operational testing because of their informality, lack of structure, and the lack of a defined requirement by which to measure performance. 11. DOD appears not to recognize the very real possibility that the ACTD demonstration may find the technology in question to have little or no military utility or to be unaffordable in today’s budgetary and security environment. In fact, due to budget constraints, the Army was forced to prioritize its procurement programs, and the planned procurement funding for Enhanced Fiber Optic Guided missiles has been reallocated. 12. While we agree with DOD that test results should form the basis for starting limited procurement, the testing needed goes beyond the basic demonstration of military utility provided by the ACTD program.
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.