Best Practices: Stronger Practices Needed to Improve DOD Technology Transition Processes
Published: 2006-09-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) relies on its science and technology community to develop innovative technologies for weapon systems, spending $13 billion on basic, applied, and advanced technology research. Several GAO reports have addressed problems in transitioning technologies to the acquisition community. This report, which was prepared under the Comptroller General's authority to conduct evaluations, compares DOD's technology transition processes with commercial best practices. Specifically, GAO identifies technology transition techniques used by leading companies and assesses the extent to which DOD uses the techniques
section_title: What GAO Found
paragraphs: Leading commercial companies use three key techniques for successfully developing and transitioning technologies, with the basic premise being that technologies must be mature before transitioning to the product line side. (1) Strategic planning at the corporate level: Strategic planning precedes technology development so managers can gauge market needs, identify the most desirable technologies, and prioritize resources. (2) Gated management reviews: A rigorous process is used to ensure a technology's relevancy and feasibility and enlist product line commitment to use the technologies once the labs are finished maturing them. (3)Corroborating tools: To secure commitment, technology transition agreements solidify and document specific cost, schedule, and performance metrics labs need to meet for transition to occur. Relationship managers address transition issues within the labs and product line teams and across both communities. Meaningful metrics gauge project progress and process effectiveness. Not only does DOD lack the breadth and depth of these techniques, the department routinely accepts high levels of technology risk at the start of major weapon acquisition programs. The acquisition community works with technologies before they are ready to be transitioned and takes on responsibility for technology development and product development concurrently. A defined phase for technology transition is not evident. These shortcomings contribute significantly to DOD's poor cost and schedule outcomes. A stark contrast exists between DOD's and private industry's environments for developing technology. The numerous examples of DOD programs that have incurred cost overruns, schedule delays, and reduced performance serve as reminders that inserting a few best practices and changing the mechanics of technology transition processes without changing the environment that determines incentives may not produce better outcomes.

section_title: Letter
paragraphs: The Department of Defense (DOD) relies on the technological superiority  of its weapon systems and armed forces to protect U.S. interests at home  and abroad. The DOD science and technology (S&T) community is tasked  with ensuring that technologies are mature when DOD’s acquisition  community takes over and integrates the technologies into weapon  systems. In fiscal year 2006, DOD plans to spend approximately $13 billion  in its science and technology efforts to develop technologies that are as  innovative as stealth and global positioning were when they were first  developed. In doing its work, the S&T community must strike a balance  between meeting the short-term needs of today’s warfighters and the long- term needs of future years’ warfighters. Achieving this balance is made  more challenging because of the need to keep pace with or exceed the  pace of innovation and to counter technologies developed by potential  U.S. adversaries.
Although the United States has produced the best weapons in the world,  its acquisition programs often incur cost overruns, schedule delays, and  performance shortfalls that undermine DOD’s buying power. This dilemma  is due in part to DOD’s difficulty transitioning technologies from a  technology development environment to an acquisition program. Because  the acquisition community frequently pulls technologies too early, it takes  on the additional task of maturing the technologies—an activity that is the  primary responsibility of technology developers—at the start of an  acquisition program. The start of a program ushers in a high-cost,  delivery-oriented phase in which the acquisition community is supposed to  be focused on integrating subsystems and working on system development  and demonstration. DOD has continued to allow the acquisition  community to take over this task before the S&T community considers the  technologies ready for transition. Numerous GAO reports have addressed  the problems of proceeding with immature technologies and have  explained what leading commercial companies do to ward off such  problems.
This report examines DOD’s technology transition processes through the  prism of best practices. Specifically, our objectives are to (1) identify  techniques that commercial companies use to transition mature  technologies before the start of product development and (2) assess the  extent to which DOD is using these techniques. We also describe how the  environments at private companies and DOD differ with regard to  technology development. The overall effort to hand off and integrate  mature technologies is referred to in this report as technology transition.  Also in this report, the private sector acquisition community is generally  referred to as a product line or business unit. GAO has prepared this  report under the Comptroller General’s authority to conduct evaluations  on his own initiative as part of a continued effort to assist Congress in  overseeing DOD’s technology and acquisition investments.
In conducting our work, we interviewed lab and product line managers at  four leading commercial companies—Boeing, IBM, Motorola, and 3M—to  identify processes for successfully transitioning mature technologies.  These companies develop a variety of products such as the 787 airplane,  network servers, integrated radio communications and information  solutions, and fuel cell technology for cars, buses, trains, homes, and  businesses. We also conducted interviews and collected pertinent  documents from each of DOD’s military services and the Defense  Advanced Research Projects Agency (DARPA), which is the central  science and technology organization for DOD. We compared commercial  and DOD practices to identify potential areas for improvement. Appendix I  includes additional details about how we performed our review, which  was conducted from June 2005 to July 2006 in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: Mature technologies are pivotal in developing new products. Without  mature technologies at the outset, a product development program will  almost certainly incur cost and schedule problems. Effectively managing  technology as a separate process from product development can improve  the potential for on-time product delivery at reduced cost. Leading  companies know this and have established disciplined processes to  prioritize their technology investments based on market needs, eliminate  technologies that are not relevant or feasible, balance technology push and  customer pull, and then transition technologies to product development  efficiently. Overall, effective management of technology facilitates the  delivery of new, innovative products to the user in less time. While DOD is  very aware of the need for new advanced technology in its weapon  systems, it has not always been effective in transitioning mature and  relevant technologies to product development.
Development of DOD’s new weapon systems depends on two distinct  phases: technology development and product development. DOD relies on  its S&T community to identify, pursue, and develop new technologies that  improve and enhance military operations and ensure technological  superiority over adversaries. This includes the development of  technologies for new weapons programs as well as those that will be  inserted into existing systems. The acquisition community takes these new  technologies, develops weapon system programs, and delivers the  products—that is, the weapon systems—to the warfighter. Table 1 shows  technology-related activities that take place during technology  development and product development, who performs the work, fiscal  year 2006 funding, and overall goals.
DOD uses research, development, test, and evaluation appropriations to  fund its technology activities. The budget is divided into categories that  generally follow a sequential path for developing technologies. Figure 1  shows the amount of money DOD spends in each category. (App. III  contains additional details about the budget.)
The S&T community controls the budget for basic research, applied  research, and advanced technology development. The budget category of  advanced component development and prototypes involves testing and  evaluating prototypes of systems or subsystems in a high-fidelity or  realistic environment before product development starts. DOD officials  say it is assumed that either the S&T community or the acquisition  community could carry out this work, but traditionally the acquisition side  prevails. After this point, additional technology-related activities are  completed as part of product development under the authority of the  acquisition community, namely, the program manager for a weapon  system.
Prior GAO reports have said that DOD launches new weapon programs  with immature technology. We found this inability to mature the  technology before the start of product development to be a major  contributor to weapon system cost and schedule growth. In our March  2006 review of 52 major DOD weapon programs, we found that 90 percent  of the programs started with immature technologies. More than half of the  programs were working with immature technologies at design review,  when DOD acquisition policy expects the design to be stable. And by the  time production began, one-third of programs still did not have mature  technologies.
Not surprisingly, we found that DOD research, development, test, and  evaluation cost estimates increased dramatically for programs having  immature technologies at program start. Figure 2 shows the average cost  growth of DOD programs we reviewed when technologies were mature  and immature at program start. design components matured late in development, contributing to research,  development, test, and evaluation cost growth and four Nunn-McCurdy  unit cost breaches. Instead of purchasing five satellites, the Air Force now  plans to buy three at a program acquisition unit cost of about $3.4 billion, a  315 percent increase.
section_title: Leading Companies Rely on Strategic Planning, a Gated Process, and Tools to Transition Needed Technologies
paragraphs: To successfully develop and transition technologies from their labs to  their product lines, leading commercial companies depend on three key  techniques:    strategic planning at the corporate level;    a gated management review process that ensures a technology’s relevancy,  feasibility, and transition readiness; and   effective tools to solidify commitment, address transition issues, and  gauge project progress and process effectiveness.
Overall, corporations are incentivized to follow these critical techniques  because the opportunity cost of not meeting customer demand is late  delivery and lost revenues and market share for the company. Through  rigorous adherence to these practices, leading companies increase  chances of eventual success because the strategy for developing the  technology fits the company’s objectives, commitment is strong for  incorporating the technology, and only after the technology is considered  mature enough for use in a certain product does product development  begin. Because the cost of developing new and breakthrough technologies  can be high, funding typically comes from the corporate level rather than  from a single product line unit, enabling the company’s product lines to  manage only product development risk, not technology risk as well.
The central focus of this report is on how the lab and the product line  communities work together to solidify the final steps of technology  transition. Figure 3 depicts the general flow of technology development  and where technology transition resides in the process.
This report touches briefly on strategic planning, which precedes  technology development. As the lab community identifies and develops  high-priority technologies during the technology development phase,  product line managers develop the business case for a new product by  identifying the market potential of the new product and using systems  engineering to set product requirements. This report does not focus on  these product line activities or on product development.
subsections: section_title: Strategic Planning Is a Critical Precursor to Successful Transition
paragraphs: Strategic planning—the effort to identify desirable technologies and  prioritize resources—is an important early step in a company’s ability to  eventually deliver the highest-priority technologies to various product  lines. Leading companies organize their research and development  activities into technological “thrusts”, or competencies that represent the  core markets of their businesses. 3M and the other companies we visited  undergo strategic planning at least annually, and this process enables  corporate management to conduct portfolio analysis, identify long-term  market needs, and match the projects in each thrust area to market needs.  Eventually, managers determine which projects appear to be relevant and  feasible, which ones are applicable for which products, whether the right  projects are getting the right resources, whether the company wants to be  first to market, and whether the final products should be released to the  marketplace as soon as possible or several years down the road. Managers  may decide to establish new thrust areas as new ideas come to light, rely  on outside suppliers for some technology, or hire new people to fill  technology voids. Projects that no longer are relevant or feasible are  eventually terminated. This type of strategic planning is critical to ensuring  that the right technologies ultimately transition to the right product line in  an economical and timely way.
Each company we visited funds technology differently, however, all  ensure that funding for early technology projects is protected at the  corporate level and not immediately beholden to product lines. 3M, for  example, has 17 technological thrust areas, and the projects are directly  funded with seed money from the corporate level. IBM receives the  majority of its funding from the corporate level. IBM labs also receive and  manage funding from product lines for specific projects. Both Motorola  and Boeing require business units to fund a portion of lab costs each year  based on their historical usage of lab resources, but the labs have freedom  to use those funds in line with corporate strategy. Decisions are made at  least annually on the composition of projects in the portfolio so it includes  the most relevant and feasible projects.
section_title: Leading Companies Use Gated Reviews to Validate Readiness for Technology Transition
paragraphs: After technology projects are selected for funding, they enter the  development “pipeline” where a gated process is used to manage and  oversee technology exploration, development, and transition. At each  gated review, lab managers assess technology progress and ensure that  certain criteria are met before technologies can enter the next stage of  development. The final phase is dedicated to technology transition  activities. The number and names of the gates vary by company, but the  type of information collected and knowledge obtained are similar. For  example, 3M uses a three-gate process, Boeing has a four-gate process,  and Motorola has a five-gate process. Regardless of where the funds came  from—a corporate or a product line unit—labs are responsible for  managing, overseeing, and using the money to fund technology projects  until the projects have transitioned to a product line. Figure 4 generally  depicts the technology development gates that companies use, lists types  of deliverables expected during each gate, and shows who provides  funding.
Explore gate: Lab technologists turn technology proposals into viable,  executable projects that ideally will meet future market needs, while lab  managers determine the amount of resources they need to invest in the  projects. Funding goes to projects that are deemed relevant. Prior to the  gated review, technologists deliver to lab managers a preliminary  assessment of the competitiveness of the technology and a road map for  completion. Technologists must address key deliverables related to such  areas as manufacturing, intellectual property, marketing, life cycle  management, and plans for the next gate.
Develop gate: Technologists develop prototypes of the project and  measure performance for relevancy and feasibility on an eventual product.  Technologists deepen their understanding of the technology, refine the  technology solution, identify the most attractive market segments for  introducing the technology, and select a product line that will incorporate  the technology into a new product. 3M and Motorola require product line  commitment to transition a technology before the gated review draws  near. This prevents labs from wasting valuable time and resources  developing technologies that product lines do not want. Projects that do  not gain product line commitment will either be terminated or go through  a different process to demonstrate their relevancy and feasibility. Motorola  leadership, for example, has a special program called an early stage  accelerator under which selected projects can obtain funds to build  additional prototypes to demonstrate the value of technology to product  line managers or potential customers. Senior lab managers carefully  monitor these projects through quarterly reviews to determine how they  are progressing. If the project is not making sufficient progress, the project  is quickly terminated so resources can be spent on other projects.
Transition gate: Technologists demonstrate a prototype in an  operational environment. The product line maintains a “customer” role  until managers are confident that the technology will work in the intended  product. Then gradually, the responsibility for funding and further  developing the technology shifts from the lab to the product line. This shift  in responsibility requires up-front and continuous planning by both the lab  and product line communities to ensure a smooth transition. As shown in  figure 5, companies we visited plan for technology transition to occur  prior to program start. This provides product line managers sufficient time  to gain additional knowledge about a technology’s attributes as they are  maturing it to its final form, fit, and function. Product line managers, who  concurrently have been working on the business case for the product, are  able to validate that the technology can be integrated into the preliminary  product design. In addition, they can develop more reliable product  development cost and revenue estimates before they are locked into  needing this technology as a product feature.
During the transition phase, labs and product lines must complete a  number of activities for transition to go smoothly. For example, labs must  demonstrate that the technology meets product line cost, schedule, and  performance requirements. In addition, production costs must be  identified and acceptable to the product line. According to 3M officials,  past experience has shown that costly manufacturing is a major reason for  product line managers deciding not to transition a technology. Labs must  also address intellectual property concerns, a step that is crucial to the  company’s ability to be a market leader. Product lines must address any  people or organizational issues, such as the transition of jobs and training  requirements that may result from using the new technology. They must  also work out any agreements with the labs for on-going support. At the  end of the phase, technical documentation related to the technology is  transitioned from the labs to the product line.
Boeing Commercial Airplanes assesses the extent to which these  activities, as well as others, have been completed to determine whether a  particular technology is ready to transition. The information is then  summarized on a scorecard that lab and product line managers can use to  quickly identify areas that need additional attention. The scorecard is  updated continually and measures a technology’s readiness from a  business and production standpoint as well as its technological readiness.  Boeing’s labs have three phases for technology development, which they  call discovery, feasibility, and practicality, plus a technology transition  phase. Each technology is evaluated in 10 readiness categories, and bars  are plotted horizontally across the scorecard to indicate how much  objective evidence exists for technology readiness. Once a technology has  demonstrated full readiness in all 10 categories, it is ready to be  transitioned. Figure 6 shows a notional picture of a technology’s readiness  at one point in time. Because the hypothetical technology has reached full  readiness in only five categories and is halfway through transition in one  category, the technology is not ready for transition. The lab and product  line would have to address issues involving readiness categories 3, 4, 6 and  9 before transition can begin or the technology is in jeopardy of not being  applied to a product.
section_title: Technology Transition Is Bolstered by Formal Agreements, Relationship Managers, and Metrics
paragraphs: Companies we visited use three tools to aid transition, including  technology transition agreements, relationship managers, and metrics.  Technology transition agreements are formal documents that detail the  specific cost, schedule, and performance attributes of the technology that  labs must demonstrate before transition can occur. Relationship managers  address the details and issues surrounding transition, and metrics allow  managers to identify the effectiveness of their technology development  and transition processes and make adjustments when necessary. These  tools require the active involvement of the labs and product lines to ensure  successful transition of technologies to new products.
subsections: section_title: Technology Transition Agreements
paragraphs: Labs and product lines use technology transition agreements to facilitate  each technology’s transition to product development. What the  agreements specify varies by company but typically consists of specific  technology and business readiness metrics, such as cost, schedule, and  performance characteristics that labs must demonstrate for product line  managers to agree to accept the technology. As figure 7 shows, the  agreements should answer some basic questions.
These formal agreements may also include identification of product line  resources needed to transition the technology, such as nonrecurring costs  to further mature the technology, as well as consideration of recurring  manufacturing costs associated with integrating the new technology on  the product. Metrics collected in the agreements are used by product line  managers to manage technology risk. For example, if the labs cannot  develop needed technology within cost and schedule parameters or with  specified performance characteristics, the product lines can terminate the  agreement and go with an alternative technology.
At Motorola, agreements enable technology project leaders to customize  their work for a particular product line’s needs. Product lines may request  that the labs identify alternative technologies for a particular product,  produce a detailed report about a specific technology, or conduct various  tests to demonstrate the relevancy or feasibility of a technology. Each of  these items becomes a “deliverable” in the agreement. 3M uses transition  agreements only for technologies that are expected to go into a product.  3M’s agreements incorporate many of the same metrics used by Motorola,  including feasibility, relevancy, and application of the technology. In  addition, agreements include an assessment of the product line’s ability  and commitment to transition technology. For example, they evaluate a  product line’s resource constraints that could inhibit a technology from  transitioning, such as asking if the technology is too expensive or if  manufacturing it would be too costly. 3M officials told us that some  agreements include loaning key lab technologists to the product line for a  period of time so product line teams can maintain momentum after the lab  has signed off and moved on to developing technology for other parts of  the company.
section_title: Relationship Managers
paragraphs: 3M, IBM, and Motorola use lab and product line relationship managers to  smooth transition. Relationship managers foster effective transition  practices by preventing the labs from pushing technologies that product  line managers do not want and by preventing product line managers from  pulling immature technologies from the labs. A Motorola official referred  to relationship managers as the most important communication tool  because communication occurs more frequently, thus allowing problems  to be identified and addressed more quickly.
Motorola designates key people at three different levels in both the labs  and the product line to serve as relationship managers. As a technology is  matured and demonstrated, communication intensifies and additional  levels of relationship managers become involved. Figure 8 depicts how  Motorola’s relationship managers communicate with each other within the  lab or product line and with their designated counterparts in the other  community.
Executive managers in Motorola’s labs and product lines are responsible  for interfacing with each other on a periodic basis. Altogether, Motorola  has eight executive managers—four in the labs and four on the product  line side. Motorola officials told us that this number provides a one-to-one  match between the labs and the product lines. Lab executive managers are  responsible for ensuring product line needs are identified and new  technology projects are started or existing projects are reprioritized to  meet those needs. Their counterparts have the final word on what priority  each technology project has with respect to the needs of the product line.  Lab and product line executive managers are required to sign off on key  technology projects and milestones for the current year. In addition, they  try to remove any roadblocks within their own units and work with their  counterparts to address any roadblocks between the two communities.
Motorola considers liaisons the most critical in the process. Again, there  are four liaisons in the labs and four on the product line side. They are the  primary interface for coordination, collaboration, and communication  during technology development and transition. Liaisons in the lab have  broad oversight of technologies being developed and share information  about technology breakthroughs with their counterparts on the product  line side, and remove roadblocks for lab technologists. These lab liaisons  work intimately with their product line counterparts to approve  technology transition agreements and assess technology readiness. The  lab liaisons oversee 40 to 60 technology transition agreements at any one  time. Liaisons on the product line side are responsible for providing  information about the changing needs of the product lines on a more real- time basis. They determine product line needs and priorities during the  annual planning process and remove roadblocks for technologists on the  product line side. Lab and product line liaisons are incentivized through  their annual performance assessments and pay increases to work together  to ensure the successful transition of technologies.
The most direct and constant communication is between the lab’s  technologist, who developed the technology, and the product line’s  technologist, who is responsible for maturing the technology for inclusion  into the product. These two technologists stay in continuous contact  because they are the ones who have the most working knowledge about  the technology. The lab technologist is expected to spend as much time as  needed to make sure that transition happens as smoothly as possible.  While the lab technologist does not become an official asset of a product  line, he or she might have to spend a period of time working in the product  line’s development lab.
section_title: Metrics
paragraphs: Leading companies use product metrics—such as weight, power, and  reliability measurements—to assess the readiness of transitioning  technologies and process metrics—such as profit growth and cycle time— to gauge the impact of their technology development processes and  identify areas for improvement. The companies analyze data gained from  the use of these metrics to evaluate how well they deliver on what is  promised, better understand the value of their respective science and  technology activities, and identify cases of inefficient investment in  technologies or underutilization of lab technologies. Table 2 provides a  composite example of metrics used at the companies we visited and the  value of the metrics to management.
Motorola’s labs use a suite of metrics, many of which are shown above, to  monitor technology projects and lab performance. For example, the  company tracks the estimated and actual development time and costs for  each project to determine estimation accuracy. In addition, it identifies  and tracks the number of lab and product line people (headcount) needed  for development and transition. These metrics are not treated with the  same rigor as metrics that are used after product development begins.  Corporate managers understand that technology development must be  managed with more latitude to accommodate calculated risk taking. In  addition, Motorola asks each product line to fill out satisfaction surveys  twice a year to assess lab performance. Because relationships are at the  heart of the process, the product line’s perceptions are of paramount  importance, making internal customer satisfaction a key metric. A  Motorola official indicated that satisfaction surveys are a ubiquitous part  of the Motorola culture. Not only do they help the labs improve, but they  help the company improve overall, the official said. The surveys also  provide compelling anecdotal information about performance problems  that need to be addressed by certain labs.
In addition to technology-specific metrics, Boeing tracks the number of  technologies that were actually integrated into products. While the intent  is that all completed technologies will make it into a product, this does not  always happen because of changes in market requirements, timing, or  funding constraints. Boeing also has metrics that assess the technology’s  impact on orders for new airplanes and on its ability to reduce  manufacturing costs. The labs, for example, have a responsibility for  improving the way Boeing designs and builds its products. As such, they  have goals for reducing recurring and nonrecurring costs and cycle time.  Finally, Boeing uses a metric that encourages the transfer of people, along  with the technologies, to the product line side. The company tracks the  rotation of people from its labs to its product lines specifically to improve  its transition processes and to refresh staff with “program people.” As a  result, company officials believe they can better align their labs to their  product lines strategies.
Company officials also told us that, similar to DOD, they have struggled to  assess return on investment for technologies they develop. This is because  technology can undergo a metamorphosis after it leaves the labs and can  be applied to multiple products or altered before going into the intended  product. Company officials said they are actively trying to track  technology utilization and the impact their technology investments have  had on company revenues. 3M has a technology database it uses to track  each project it funds. The database contains cost, schedule, and  performance data and summarizes other information, such as how long a  project took to complete and lab and product line funding used to mature  the technology. 3M then compares the cost information with worldwide  sales, tracked through bar codes, to determine its return on investment for  a particular technology. Company officials indicated that there are still  problems with this methodology for estimating return on investment and  that they are continuing to refine their approach.
section_title: DOD Lacks Breadth and Depth of Techniques That Leading Companies Use to Effectively Transition Technologies
paragraphs: DOD has taken some steps over the past few years to improve its  technology transition processes, but the practice of accepting high levels  of technology risk at the start of major weapon system acquisition  programs continues to be the norm. This shortcoming is a major  contributor to DOD’s poor cost and schedule outcomes. Many of DOD’s  problems can be attributed to deficiencies in strategic planning for critical  technologies, processes for technology development and transition, and  tools that support transition. A recent Defense Science Board report states  that DOD is not as well positioned as it should be from a strategic  standpoint to meet the challenges and exploit the opportunities offered by  technology. The report identified several opportunities for improvement at  the Director, Defense Research and Engineering level (DDRE) to help the  department jointly identify, prioritize, develop, and deliver the  technologies most relevant and critical to meeting weapon system  requirements in a timely manner. DOD’s technology development process  is undisciplined and lacks criteria for maintaining a close connection  between fledgling technologies and the products that will need them to  meet the warfighter’s future needs. As a result, technologies are often not  ready when they are needed, and acquisition programs pull the  technologies into their programs too early, leading to inefficiency during  product development and cost and schedule increases. DOD’s process for  transitioning technologies to product development is funded and managed  by its acquisition community, the opposite approach to that taken by the  companies we visited.
We identified some initiatives within DOD’s S&T community that emulate  some of the tools we found in the commercial world. They hold promise,  but must be accepted, improved, and replicated significantly more than  currently to have a positive impact. For example, the military services are  using technology transition agreements and have established boards to  select and oversee some of its technology portfolio. Also, DARPA uses  relationship managers to expedite the efficient transition of technologies  to products and DOD has established other programs that institutionalize  some best practices. The collection and use of meaningful metrics,  however, remain a problem.
On the basis of our previous review of 52 major weapons programs, we  found that DOD’s typical path involves starting programs with immature  technologies and concurrently working on technology development and  product development. Figure 9 is a depiction of this path.
subsections: section_title: Strategic Planning Affects DOD’s Ability to Meet Warfighters’ Needs
paragraphs: DOD has an annual strategic planning process that involves the  development of several plans at the corporate level and within the services  and defense agencies for determining how to invest its S&T funding, which  amounted to about $13 billion in fiscal year 2006. Despite these efforts, a  recent Defense Science Board report stated that DOD is not as well  positioned as it should be at the DDRE level (the corporate level) to meet  the challenges and exploit the opportunities offered by technology. The  report identified several opportunities for improvement. Until  improvements are made, it is likely that the there will be strategic gaps and  overlaps in technology coverage as the services and agencies develop their  own approaches to meet critical warfighter needs.
Similar to private industry, the department goes through an annual  strategic planning process led by its corporate level. As part of its  responsibilities, DDRE develops the Defense Science and Technology  Strategy, which serves as the foundation for DOD’s science and  technology strategic planning process. The strategy identifies five generic  technology thrust areas—information assurance, battlespace awareness,  force protection, reduced cost of ownership, and maintaining basic  research—that have high priority in the department, and is supported by  four other documents, including the    Basic Research Plan, which presents the DOD objectives and investment  strategy for DOD-sponsored basic research performed by universities,  industry, and service laboratories;   Defense Technology Area Plan, which presents the DOD objectives and  the strategies for applied research and advanced technology development  investments;   Joint Warfighting Science and Technology Plan, which takes a joint  perspective across the applied research and advanced technology  development plans to ensure that the science and technology program  supports priority future joint warfighting capabilities; and   Defense Technology Objectives, which identify specific technology  advancements that will be developed or demonstrated, the anticipated  date of availability, and specific expected benefits.
Together, these documents, as well as the supporting S&T plans of the  military services and defense agencies, are supposed to provide the  framework for decisionmaking throughout the science and technology  community.
However, a recent Defense Science Board report states over time there  has been a relative decline in the influence of DDRE at the corporate level  on strategic matters. In the 1960s and 1970s, the corporate level was more  proactive and provided high-level direction that drove several decisive  technological developments, including stealth; standoff precision strike;  and tactical intelligence, surveillance, and reconnaissance systems that  have transformed U.S. military capabilities. Since the 1980’s DDRE has  used a decentralized management approach, relying on the services and  agencies to determine and prioritize the most needed technology projects.  The report states that as a result, the department is not wellpositioned to  implement new operational capabilities needed for future warfighters,  including  identifying and tracking terrorists,   addressing commercialization and globalization issues,  rapidly evolving technologies such as bio- and nanotechnologies, and identifying the proper mix of short- and long-term projects to work on.
One of the recommendations included in the report is for the  Undersecretary of Defense (Acquisition, Technology, and Logistics) to  develop a strategic technology plan and that DDRE be tasked with  assuring that all research and development organizations are  implementing DDRE’s strategic technology guidance. The board also  believes DDRE needs to be more involved in strategic challenges related  to (1) gathering and nurturing technology from a variety of sources,   (2) developing and exploiting technology to enable new disruptive  capabilities; (3) identifying and countering disruptive capabilities  developed by adversaries using readily available or advanced technology,  and (4) ensuring an adequate level of long-term research for DOD needs.  Without effective corporate level leadership, warfighter needs are  addressed in a decentralized, uncoordinated manner, thereby increasing  the risk that the department will not fulfill these needs.
section_title: DOD Does Not Use a Disciplined, Gated Process to Test for Relevancy, Feasibility, and Transition Readiness
paragraphs: DOD’s 5000 series acquisition policy specifies that technology  development should be separated from product development and that a  project shall not exit the technology development phase until the  technology has been demonstrated in a relevant environment. This is in  line with the practices we found at leading companies, with the exception  that the companies require technologies to be demonstrated in an  operational environment before program start, which is a higher degree of  readiness. However, the department lacks a structured, gated process for  managing technology development and transition, as well as criteria that  would allow decision makers to know when technology is ready to  progress from the technology development environment into an  acquisition program to begin product development. As a result, the  services continue to launch new programs with immature technologies,  and acquisition programs take on technology development  responsibilities, an activity that is considered too risky for commercial  companies we visited. The following figure shows where the technology  development phase occurs in DOD’s acquisition process and the entrance  and exit criteria for that phase.
As shown in the figure above, DOD has two phases prior to program start  (also referred to in DOD nomenclature as milestone B). The first is the  concept refinement phase, when acquisition programs refine the initial  concept and develop a technology development strategy that is supposed  to guide activities during the technology development phase. According to  the acquisition policy, the strategy should include    a discussion of the planned acquisition approach, including a summary of  the considerations and rationale concerning the approach;   a discussion of the planned strategy to manage research and development;    a complete description of the first technology demonstration; and    a test plan.
The policy, however, does not address the role of the S&T community in  designing or supporting a technology development strategy. And according  to S&T officials, their role in this phase is minimal. We recently reported  that DOD skipped a formal milestone meeting that should take place at the  end of the concept refinement phase, referred to as milestone A, for   80 percent of the programs we reviewed. This inhibits the S&T and  acquisition communities’ ability to create a realistic plan for developing  and maturing needed technologies prior to program start and executing a  plan once the program begins.
During the technology development phase, the acquisition policy calls for  DOD to focus on the development, maturation, and demonstration of the  technologies needed for the capability under consideration. To exit this  phase, technologies should be demonstrated to be sufficiently mature.  DOD defines this as demonstrating technologies in a relevant environment.  (See app. IV for definitions of technology readiness.)
When comparing the DOD process with that used by commercial  companies we visited, we identified two major differences. First, although  DOD policy states that the S&T community “shall enable rapid, successful  transition … to useful military products,” there is no defined transition  phase with criteria to facilitate assessment of a technology’s readiness to  transition. Instead, the services have senior-level boards that review  projects on an annual basis to determine if they are on track relative to  cost, schedule, and performance goals set out for the program and rely on  a technology readiness tool to gauge development progress. The reviews  do not include a formal assessment of many of the technical and business  criteria contained in the Boeing scorecard, such as determining if the  costs, benefits, and risk are well understood and technology is affordable.  As a result, program managers often pull a technology from technology  developers before the program manager has the opportunity to validate  that the technology can be integrated into the preliminary product design  and to develop good product development estimates prior to the start of  system development. As stated earlier, commercial companies we visited  had a dedicated phase at the end of the technology development process  for shifting technology responsibilities to their product lines. We recently  reported that almost three-fourths of the programs started since the  acquisition policy was revised in 2000 began with critical technologies that  were not ready for product development. Seven of the nine programs we  reviewed more indepth, about 80 percent, were approved to begin  development even though program officials reported that technologies  were below readiness levels required for entering product development.  This included programs like the Future Combat System and the Joint  Strike Fighter.
We found DOD generally views immature critical technologies at the  beginning of development as an acceptable risk as long as there is a plan  to mature the technologies by the time the program reaches its design  readiness review. In effect, the department views risk management plans  as an acceptable substitute for demonstrated knowledge. For example, the  Navy’s Multi-Mission Maritime Aircraft program had none of its critical  technologies mature at program initiation. Instead of holding the program  to the acquisition policy criteria for entering development, the decision  maker simply directed the Navy to work with the Office of the Secretary of  Defense to implement risk mitigation and technology maturation plans  during the integration phase of system development.
Second, under DOD’s current funding structure, the transition of  technology—which should occur prior to the beginning of product  development and milestone B—is funded and managed by acquisition  programs. This contrasts with the approach used by commercial  companies, where the lab manages and funds these activities. Commercial  firms find that holding their labs accountable for the management and  funding of technology as it transitions to a new product forces them to  deliver more quickly and efficiently and allows the product lines to focus  on product development and the risks associated with it. Figure 11  illustrates the difference in responsibility.
As shown above, in the commercial companies we visited, product  developers are allowed to act like customers for emerging technologies;  they are not required to accept, manage, and fund technology risk. This  significantly improves the chances of their products reaching the market  quickly, at predictable cost, and with high quality. In DOD, major weapon  system acquisition programs pull technologies that are not yet ready for a  product in order to meet that product’s requirements. The S&T  community, although still called upon to review technology readiness, has  no funds at stake and is no longer responsible for the risk from technology  at that point. This is a major contributor to the significant cost overruns  and late deliveries of major weapon systems to the warfighter in recent  years.
section_title: DOD Is Adopting Transition Tools, but Use Is Not Widespread
paragraphs: DOD is using several tools we found were also being used by commercial  companies to facilitate the transition of technology to weapons programs.  For example, the services are using technology transition agreements for  some of their technology projects, DARPA is using relationship managers  to address transition issues, the services have established boards to  oversee and manage a portion of their technology portfolios, and DOD has  several new programs, including the Joint Capability Technology  Demonstration program to speed a technology’s transition to a weapon  system. However, because of the newness or small amount of funding  associated with these tools, widespread use has yet to occur. We also  found that DOD is using technology-specific and status metrics, but does  not have sufficient measures in place to assess the impact of their  technology investments or technology transition processes.
subsections: section_title: Technology Transition Agreements
paragraphs: The services have recently begun using technology transition agreements  to formalize technical expectations labs must demonstrate in order for  acquisition programs to transition technologies into their programs. DOD  science and technology officials believe the agreements are a useful way  to hold the technology developers accountable and gauge lab progress  toward meeting technology specifications. We are encouraged by their use  of agreements, particularly in the Navy. Agreements we reviewed,  however, did not contain some of the information or metrics commercial  companies believe is valuable to track, and use thus far has been limited.
The agreements we reviewed contained some of the same elements  included in agreements used by leading companies we visited, such as a  description of the technology project, key technology developer and  weapon system personnel associated with the project, and specific  performance characteristics that the lab must meet for transition to occur.  In addition, they identify the amount of funding acquisition programs will  have for transition and when that funding will be available. Unlike the  leading companies we visited, however, these agreements did not typically  require the technology developer to demonstrate cost metrics for the  technology to be included in a weapons program. Commercial companies,  for example, include an assessment of manufacturing costs that could be  expected if a technology were to be included on a new product and they  usually demand prototypes of the technology as demonstration of  readiness to transition. A recent Defense Science Board Report  highlighted the importance of addressing manufacturing concerns during  technology development. Excerpts are shown in figure 12.
We also found that the services’ use of technology transition agreements  has been very limited. For example, we found that the services have a  combined total of 224 applied and advanced technology projects that they  have identified as candidates for transition, of which 146—or about   65 percent—have technology transition agreements. The Navy accounted  for the greatest percentage, with 90 of its 115 projects having an  agreement, almost 80 percent. However, there are hundreds of other  military services applied and advanced technology research projects that  were not selected as candidates where agreements may also be useful. As  stated earlier, commercial companies we visited had agreements for nearly  all technology projects. The agreements were updated at least annually or  sooner if particular tasks were completed. Lab officials use metrics  included in the agreements, as well as other criteria, to determine if the  project should be continued.
It should also be pointed out that while technology transition agreements  are useful tools for firming weapons program commitment to transition,  they do not guarantee transition success. For example, a Navy official told  us that about one-third of the projects that had been expected to transition  in fiscal year 2005 did not transition for the following reasons:    The technology did not meet the cost parameters required for transition to  occur.
Weapon system requirements changed, forcing changes to the original  agreement.
Program offices did not identify dollars in its budget to transition the  technology.
A program office decided the technology was too risky compared with  current hardware in production.
Air Force and Army officials told us they do not track information on why  their technologies do not transition. However, they have experienced  similar problems as the Navy. They also stated that some lab technologies  do not transition because prime contractors decide to use technologies  developed in-house, even though DOD technology developers met the  metrics included in the transition agreements. Because agreements are  usually established between the technology developers and program  offices prior to the selection of prime contractors, there is no assurance  that the technologies will transition. Even in cases where a prime  contractor is known, such as for ongoing development programs, the  prime contractor is not a party in the agreement.
section_title: Relationship Managers
paragraphs: With the exception of DARPA, DOD does not use relationship managers in  the same manner as leading private companies. According to an Air Force  lab official, relationship managers market technologies being developed by  the labs or gather data about ongoing projects for senior lab management.  Most communication about technology transition in DOD takes place  through integrated product teams or during annual reviews of technology  projects by the senior-level oversight boards for each of the services. Use  of relationship managers for these purposes are helpful, but the managers  do not necessarily serve as points of contact within the labs and  acquisition communities, do not devote time toward efficiently  transitioning technologies to multiple weapon system programs, and do  not help identify and address systemic transition problems.
Within DARPA, senior officers, called operational liaisons, focus on  marketing and transitioning DARPA-sponsored technologies. According to  the DARPA director, the liaisons have been very helpful with transitioning  technologies because they are well practiced at using the command chain  of their respective services and finding the right service contact at the  right time. The liaisons    provide operational advice for planning and strategy development;    provide an understanding of service perspectives, issues and needs so that  potential customers can be identified and effective agreements can be  written;    draft and coordinate agreements between DARPA and the services; and    direct technology insertion in the services.
Figure 13 describes the impact that operational liaisons have had on  transitioning a DARPA-sponsored program called Boomerang. We believe  that DARPA’s approach could serve as a model for how the military  services might establish more formal roles for communication between  the S&T and acquisition communities.
The military services and the Office of the Secretary of Defense have a  variety of programs to help transition technologies to weapons systems or  directly to the warfighter. These programs, some of which are relatively  new and others that have been around for several years, have met with  some success. However, these programs represent a small portion of   $13 billion DOD spends on science and technology development. As such,  they cannot single-handedly overcome transition problems, but rather  demonstrate various ways to ease transition.
In recent years, each of the military services has established senior-level  boards to oversee technology programs that include advanced and in some  cases applied research projects. This includes the Air Force’s Advanced  Technology Demonstration program, the Army Technology Objectives  program, and the Navy’s Future Naval Capabilities program. The boards,  which are comprised of representatives from the lab, acquisition, and  warfighter communities, are responsible for selecting projects to be  developed, allocating funding to those projects, and reviewing the  projects’ progress on an annual basis. The boards emphasize transition  planning and can address obstacles to successful transition. According to  officials, these programs represent about $1.3 billion of DOD’s S&T  budget.
The Advanced Concept Technology Demonstration program and the newly  established Joint Capability Technology Demonstration program were  initiated by DOD as a way to get technologies that meet critical military  needs into the hands of users faster and at less cost than by the traditional  acquisition process. Under these programs, military operators test  prototypes that have already been developed and matured in realistic  settings. If they find the items to have military utility, DOD may choose to  buy additional quantities or just use the items remaining after the  demonstration. Fiscal year 2006 was the first year of a 3- to 5-year period  where the current Advanced Concept Technology Demonstration program  will be phased out in favor of the new Joint Capability Technology  Demonstration program. Department officials believe the new joint  program offers improvements over the earlier program in that capabilities  will be demonstrated 1-2 years earlier, there is a greater focus on  combatant command needs during the selection process, and the Office of  the Secretary of Defense provides significantly more funding during the  first 2 years of the demonstration project.
A big difference between the two programs is that the new joint program  is expected to provide a better path for the transition of technologies  because it includes funding for both advanced technology development  and advanced component development and prototypes. The Advanced  Concept Technology Demonstration program only includes funding for  advanced technology development. It must rely on the acquisition  community to identify advanced component development and prototype  funding for transitioning the technologies. An additional benefit is that the  Office of the Secretary of Defense is partnering with the Naval  Postgraduate School to conduct business case analysis for completed  demonstration projects. This type of analysis is expected to aid decision  makers in evaluating alternative approaches to the allocation of scarce  resources competing for transition funds.
The Manufacturing Technology Program is aimed at quickly identifying  and solving production problems associated with technology transition. It  focuses on the needs of weapon system programs for affordable, low-risk  development and production. For example, one project, which was given  $730,000, was geared toward improving the reliability and strength of  large-diameter fasteners used to attach various components on the  Seawolf and Virginia Class submarines. Corrosion concerns required that  the existing fasteners be replaced at periodic intervals to preclude  catastrophic failure. Navy and commercial companies evaluated a wide  variety of materials for strength and corrosion resistance to determine the  best candidate material and then adjusted the material manufacturing  process to obtain the required strength and toughness levels to ensure the  fasteners could be produced without extraordinary measures or a  deterioration of the material properties. Among other things, program  officials expect the project to result in cost avoidance of $1.1 million per  fielded submarine per year and improved reliability.
There are other, smaller programs that also focus on transition. Two of  these are the Foreign Comparative Testing Program, which focuses on  identifying, evaluating, and procuring technologies that have already been  developed and tested in other countries, and the Technology Transition  Initiative, which focuses on speeding the transition of technologies  developed by DOD’s S&T programs into acquisition programs. These  programs received about $37 million and $29 million in fiscal 2006,  respectively. The Foreign Comparative Testing Program, for example,  successfully evaluated a South African mine-protected clearance vehicle  that will protect soldiers from the effects of landmine explosions during  route clearance operations. As of June 2006, 61 of these vehicles have been  delivered to the Army, Navy, and Marine Corps.
section_title: Metrics
paragraphs: For the most part, the military services track and use only a few of the  metrics we found at commercial companies. As stated earlier, they have  some technology-specific metrics in their technology transition  agreements. In addition, they have status metrics to track some of the   on-going, completed, and transitioned projects that are funded by the Air  Force, Army, and Navy S&T communities as well as the number of  projects with technology transition agreements. However, the services  have few metrics that would enable them to gauge the impact of their  investments and the effectiveness of their processes for developing and  transitioning technology.
DOD officials told us that establishing good, quantifiable metrics to  measure transition success is difficult to do. Some of the challenges they  face include determining    how long to track a technology after it has transitioned,    how to track and distinguish transition success when the same technology  is used on multiple weapons programs, and   how much credit to take when only portions of a technology transition to a  weapon program.
Further, DOD S&T officials do not believe that they have enough resources  to track technology once technology developers finish working on a  project. They say that it would be very labor-intensive to track long-term  measures, such as the impact of transitioned technology in terms of cost  savings and improved performance. Instead, the services rely heavily on  projected cost, schedule, and performance improvements that are required  by S&T program management as part of the project selection process.  Although progress in meeting technology expectations is monitored  throughout technology development, once technology developers are  finished working on a technology, little is done to determine if it actually  went onto a weapon system and if it is being used in the field.
Last year, we reported that the Office of the Secretary of Defense had  difficulty tracking the impact of three small technology transition  programs it oversees—the Technology Transition Initiative, the Defense  Acquisition Challenge Program, and the Quick Reaction Fund.  Nevertheless, we pointed out that there may already be readily available  starting points within the department to capture more information  regarding return on investments. For example, we pointed out that the  DOD Foreign Comparative Testing Program has established metrics to  measure the health, success, and cost-effectiveness of the program and  has a database to facilitate return on investment analyses. Further, a study  by the Naval Postgraduate School to identify metrics for the Advanced  Concept Technology Demonstration program may also be useful to the  military services for assessing transition success and impact.
section_title: Differences in Environment and Incentives Contribute to Different Practices and Outcomes
paragraphs: There are critical differences between the environments and cultures of  private world-class companies and DOD that must be recognized before  tangible progress can be made in establishing more efficient practices for  transitioning technologies to major weapon system acquisition programs.  Examples from past initiatives serve as reminders that just changing the  mechanics of technology transition processes, without changing the  environment that determines incentives, may not produce better  outcomes. Private companies operate in a competitive environment that  demands speedy delivery of innovative, high-quality technologies and  products to satisfy market needs. If the company cannot meet those  criteria, it will cease to exist in that environment. On the other hand, DOD  operates in an environment with a complex variety of “customers.” These  complex relationships often hinder the ability of DDRE, its “corporate”  component, to provide strategic leadership similar to that of the private  company. Success in this environment is often based on a single service’s  ability to launch a program to address critical needs and to secure annual  funding for the program. The challenge for DOD and congressional  decision makers may not lie so much in the “how to” aspects of technology  transition as in creating stronger and more uniform incentives that  encourage the S&T and acquisition communities to work together to  deliver mature technologies to programs.
Private firms must continually deliver innovative, reliable products to  market very quickly to prosper. The delivery-oriented environment that  private firms live in creates a need for strategic simplicity and directness  from leadership and forces product lines and labs to measure success in  terms of lower costs and increased revenue and market share. The ability  to deliver new, innovative, reliable products of high quality to market as  soon as possible drives revenue and market share. This environment  creates incentives to maintain an efficient technology base that is focused  on market needs and can efficiently transition feasible technologies into  new products, keeping costs down while achieving greater revenues.  Because commercial firms understand that they do not succeed until a  new, innovative product is delivered to the customer, they view successful  transition of technologies to products as a critical part of their value chain.  The corporation, therefore, insists on accountability from and supports its  technology developers with a strong strategic process for ensuring that its  product portfolio is balanced. It does not want too many programs vying  for scarce investment dollars.
For example, IBM officials stated that poor strategic planning was one of  the problems the company had to address after it incurred an   $8 billion loss in 1992-1993. According to company officials, technologies  no longer could be developed for the sake of development; they had to be  aligned with a product line and have market potential to be funded. IBM  labs now conduct a situation analysis that takes an internal and external  look at technologies being developed and a gap analysis to determine  where IBM might be behind in technology development.
At 3M, meeting market imperatives is critical and senior leadership has set  a goal to increase sales 5 to 8 percent annually through new products. In  this environment, the company has established a companywide initiative  referred to as 2X/3X. Generally, the goal is to double the number of new  ideas going into the product development pipeline and to triple the market  success of the products coming out of the pipeline. Lab and product line  managers are evaluated and held accountable for the success of the  initiative based upon their ability to meet sales goals and properly  resource their projects. According to 3M’s 2004 annual report, the  initiative is producing strong results. 3M officials believe their culture is  well aligned with this competitive environment. For example,  technologists participate in companywide technology forums to discuss  problems and share information. A technology council, comprised of  about 80 lab and technical directors, meets monthly to share ideas,  experiences, and best practices among the labs.
At Motorola, the corporate vision is known as “seamless mobility.” Senior  leaders have aligned the company’s global resources—including its   $3.1 billion annual research and development budget and a worldwide  research network—to help achieve this goal. According to Motorola  officials, the seamless mobility vision has been instrumental in uniting  Motorola labs and product lines over the past 3 years toward a common  goal.
On the other hand, DOD operates in an environment that often hinders the  ability of its corporate component—DDRE—to provide strategic  leadership similar to the private company. Not only must the department  worry about national security imperatives, it must also answer to Congress  and its oversight agencies; manage a stove-piped, parochial culture; ensure  the public’s trust that tax dollars are wisely and fairly spent; and  incentivize an industrial base to deliver cutting-edge, often very risky  technologies and products. This is a complex set of deliverables by any  measure. It creates an environment that makes it difficult for DDRE to  lead and has contributed to an undisciplined strategic planning process.  Success in this environment is often based on a service’s ability to launch  a program to address critical needs and to secure annual funding for the  program. It is characterized by fierce competition among the services to  win scarce funding to begin major weapon system acquisition programs,  regardless of the readiness of technologies to meet far-reaching  requirements. As a result, programs are pressured to distinguish  themselves from other programs previously developed by promising new,  enhanced features such as being faster and/or more lethal than anything  else. It is very common for these new programs to include requirements  for new technologies with overly optimistic assessments of technological  feasibility, risk, and delivery schedules. This environment and these  incentives breed uncertainty and risk and, as a result, cost overruns and  delivery delays.
section_title: Conclusions
paragraphs: In DOD, delivering mature technologies to weapon system acquisition  programs at the right time continues to be a challenge. Rather than  addressing technology issues in a science and technology environment  before product development starts, acquisition programs carry significant  technology risk into product development. This brings with it a high risk  that costs will rise and deliveries to the warfighter will be delayed. In fact,  there is strong evidence that acquisition programs that start with immature  technologies encounter significantly poorer acquisition outcomes than  others. This approach is in sharp contrast to the approach taken in the  commercial world. High-performing companies solve technology  challenges in the S&T environment. To do this, they have put in place  processes and adopted techniques that are pertinent to DOD. Strong  strategic planning defines critical investment priorities, and a structured  process defines the path towards a technology’s transition to product  development. This transition is supported by technology transition  agreements that hold the research labs accountable for what they must  deliver. They also include clarification of the responsibilities the product  developer has in accepting new technologies. Metrics are used to force  demonstration of relevancy and feasibility at key points in the process and  gauge the success of individual projects and the process itself. Funding for  technology development largely comes from the corporate level, with the  research labs having responsibility for technology development until  technology is matured and transitioned to the product line. This is critical  to success, because it allows product developers to play a customer role.  They are allowed to say no to technologies that are not ready for their  programs and can focus on their job at hand—product integration,  supplier management, and quality.
DOD has adopted some of these practices. The department has begun  using technology transition agreements and relationship managers and has  initiated programs that place greater emphasis on technology transition  planning. However, the reach of these initiatives is limited, and there is no  unified, corporate approach to using them at this point. We recognize that  the environment and incentives for DOD are very different than those of  commercial firms we visited. For commercial practices to work on a broad  scale, the DOD environment must be conducive for applying such  practices. DOD senior leadership is a critical factor in providing this  direction and vision as well as in maintaining the culture of the  organization. The department must devote greater attention to strategic  planning and technology development processes so that resources are  spent on technologies that can and will be transformed into capabilities  for the warfighter. In addition, the S&T and acquisition communities must  work together to expand the use of technology transition agreements and  relationship managers. Finally, the department should examine the way it  currently funds technology transition. It may benefit from an examination  of commercial companies’ methods in this regard. They hold their  technology developers accountable for delivering relevant technologies  that are ready to be integrated into new products, but they empower them  to succeed by providing adequate funding to get the job done.
section_title: Recommendations for Executive Action
paragraphs: DOD should take steps to improve its transition of technologies to more  efficiently deliver capabilities to its warfighters. The Defense Science  Board has recommended strengthening of DOD’s strategic planning  function, and we believe this would move the department in the right  direction. We believe a disciplined, gated approach for technology  development, supported by technical and business criteria, would provide  adequate knowledge to acquisition program managers about the risk of  including particular technologies on specific weapons programs. It would  also provide a more systematic way for the S&T community to continually  assess the relevancy, feasibility, and potential transition commitment of its  technologies and make decisions about future investments. DOD’s current  process lacks specific decision points with “go/no go” decisions. As such,  we recommend that the Secretary of Defense take the following actions:    develop a gated process for developing and transitioning technologies that  establishes a transition phase and defines activities that should occur  during this phase, and  include specific criteria to support continued funding of specific projects  in that process.
We also believe greater use of tools, such as technology transition  agreements, relationship managers, and metrics, could help the  department improve its ability to deliver mature technologies when  needed, address transition issues more quickly, and gauge the impact of  their science and technology investments and lab processes. Therefore, we  recommend that the Secretary of Defense:    expand the use of technology transition agreements to applied and  advanced development projects; include additional metrics in technology transition agreements to provide  S&T and acquisition program managers demonstrated knowledge about  the manufacturing readiness, producibility, other benefits, and risks of  including the technology on a weapons program;   expand the use of relationship managers by designating people at various  levels in both the S&T and acquisition communities to address systemic  transition issues and those related to specific weapon system programs.  Also, define responsibilities for each level of relationship manager;    adopt additional process-oriented metrics, such as the percentage of  advanced technologies that—once past milestone A of the acquisition  process—transitioned into a weapons program or were fielded and the  cycle time from milestone A to milestone B as a way to measure the  effectiveness of S&T processes and the impact of science and technology  investments; Commercial companies fund technology development and transition  activities in their labs and hold the labs accountable for delivering mature  technology to their product lines. As such, we are recommending that the  Secretary of Defense    Set aside a portion of advanced component development and prototype  funds for the S&T community to manage the transition of technologies to  acquisition programs. For this funding to be used effectively, it will require  the discipline provided by corporate leadership in defining priorities,  processes, and metrics.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report. DOD  generally concurred with the recommendations in our report to improve  its ability to transition technology to our warfighters. In doing so, it  emphasized that that commercial industry program managers operate in  an environment driven by profit and market opportunities, while DOD is  organized to support our warfighters. We understand the basic differences  in the environments, but believe there are many lessons that can be  gleaned from our commercial visits. While it is true that private companies  are motivated by profit, they can not achieve a profit without being  focused on the timely delivery of products to the market. We note in this  and other best practices reports that commercial firms have significant  pressures on them to deliver cutting edge technologies quickly. They often  bet their very existence on delivering new products and technologies  before anyone else does so. In some cases, such as medical systems, their  customers have urgent, life-or-death needs, similar to those of the  warfighters, that demand very complex technological solutions. The real  difference between these companies and the department is that they  understand the futility of promising more than can be delivered to the  market at any given point. They understand that, in the final analysis, they  only succeed—or survive—only by delivering needed capability. On the  other hand, the department’s acquisition programs consistently accept  immature technologies into product development that become a major  cause of cost increases, schedule delays, and ultimately very late delivery  to the warfighter who needs the equipment. The list of programs that have  delivered late is extensive.
Rather than focus on the different environment in the commercial world,  DOD should focus on the practices used in the commercial world that  allow it to deliver mature technology to product development (then, to the  user) quickly and efficiently, an outcome woefully lacking in the  department.
Specially, of the six recommendations we made, DOD concurred with two  recommendations, partially concurred with two others, and did not concur  with the final two. DOD’s comments appear in appendix II.
DOD concurred with the recommendation that called for establishing a  gated process for developing and transitioning technologies. In its  response, the department stated that it has a gated process for programs  that have a formal milestone A decision point and that technology maturity  and technology readiness assessments are used in this process. DOD  explained that the technology maturity assessments, in particular, often  lead to risk mitigation plans, which include explicit gates that immature  technologies must pass through to become qualified for adoption.
We are not confident that DOD’s implementation plans are fully responsive  to this recommendation. First, the department does not have a process  similar to those that we found at commercial companies to actively  manage and make investment decisions across all technologies and  determine if an individual technology is ready to transition. In its  comments, DOD refers to a process for programs beginning at milestone  A. However, we have reported that 80 percent of the programs we  reviewed that passed milestone B since 2000 did not have a milestone A.
Second, we found that even when technology readiness assessments  indicated that technologies were not ready to be included in an acquisition  program, DOD often decided to use them anyway. Third, risk mitigation  plans used by DOD are subjective engineering judgments and are of  limited value when evaluating the transition readiness of technologies.
The department also concurred with our recommendation that it expand  the use of technology transition agreements to applied and advanced  development projects. However, in its response, DOD did not identify  specific actions it plans to take to implement this recommendation.
DOD partially concurred with the recommendation that it include  additional metrics in technology transition agreements. In its response, the  department indicated that it has taken steps to incorporate manufacturing  readiness assessments as a means to improve technology readiness and  that any additional metrics should be project-specific and included as exit  criteria as part of the technology transition agreement. We are encouraged  by DOD’s recent development of manufacturing readiness levels and  believe they should be a mandatory tool for the S&T and acquisition  program managers to use to assess the manufacturability of a new  technology. While we agree that the department should develop additional  metrics on a project-by-project basis, we also believe that a more holistic  approach, as depicted earlier in the report, is needed to determine the  transition readiness of a particular technology. Transition readiness is not  solely dependent upon the technical maturity achieved by S&T, but is also  dependent upon the readiness of an acquisition program to accept the new  technology and therefore should include an assessment of other factors  such as cost, benefit, risk, scalability, and acquisition program  endorsement.
DOD also partially concurred with the recommendation to expand the use  of relationship managers to address systemic transition issues and those  related to specific weapon system programs. In its response, DOD  indicated that it relies on written documents—the technology maturity  assessments and technology transition agreements—to facilitate  communication between the S&T and acquisition communities,  particularly at the executive level. As stated earlier, these documents do  not address many of the factors that could hinder transition. As DOD  states in its comments, technology transition is referred to as a “body- contact sport.” We agree, and this is why our recommendation is aimed at  more direct person-to-person communication. Leading companies believe  relationship managers play a key role in their ability to successfully  transition technology. While DOD did acknowledge that its S&T  investment would benefit from expanded emphasis by staff at the  execution levels on technology transition issues, it did not directly address  the need for midlevel relationship managers. We continue to believe DOD  would benefit from the use of midlevel relationship managers. These  managers provide the back and forth communications between the S&T  and acquisition communities that help ensure that the right technology is  being developed in the time frame needed. These managers are also  uniquely positioned to ensure that technology is applied to multiple  platforms when applicable and can identify systemic transition problems  that should be addressed.
DOD did not concur with the recommendation that it should develop  additional process-oriented metrics, even though it stated that it is  installing a process to do this through the use of technology maturity and  technology readiness assessments. DOD stated that it does not want to  commit to overly burdensome metrics that may be more oriented to  measuring the process for the sake of measurement. Like DOD, we do not  believe the department should develop metrics just for the sake of  measurement. Rather, we believe DOD should develop and use metrics  that allow DOD, Congress, and taxpayers to gauge the effectiveness of  DOD S&T investments, which are expected to reach about $13 billion this  year. The metrics that DOD references—those derived from technology  maturity and technology readiness assessments—are project-level metrics  rather than portfolio and process metrics that would allow the department  to analyze its investment and make adjustments appropriately.
DOD also did not concur with the recommendation to set aside a portion  of advanced component development and prototype funds for the S&T  community to manage the transition of technologies to acquisition  programs. In its response, DOD stated that acquisition programs are best  suited for transitioning technology because they have the training and  discipline to field systems and have the responsibility to ensure a stable  design, identify a responsive and responsible contractor, manage  execution, and plan for life cycle support of the system. While DOD  discusses transition to the warfighter, our recommendation is aimed at  transitioning technology from S&T to acquisition programs. We found that  in the commercial world, the training and discipline for transitioning  technologies to product development is managed by the technology  development community and is adequately funded. DOD believes its  current approach of setting aside a small portion of S&T funds for  transition through programs such as the Advanced Concept Technology  Demonstration program, Joint Capability Technology Demonstration  program, the Technology Transition Initiative, Manufacturing Technology  program, and the Foreign Comparative Testing program is appropriate. We  continue to believe DOD’s approach to funding transition is flawed and  that small pots of money for specific transition activities offer a piecemeal  solution to a more systemic problem. DOD currently uses some of its  advanced component development and prototype funds for S&T activities,  including transition. However, acquisition programs typically carry out  and fund these activities. We believe the S&T community should be  responsible for these activities and as such, should be given the  appropriate level of funding to carry them out. We believe DOD would be  better positioned to develop and deliver weapon systems more quickly to  its warfighters if the S&T community was responsible for developing,  maturing, and transitioning mature technologies to the acquisition  community and if the acquisition community focused solely on product  development activities and delivering weapon systems to the warfighter  That being said, we recognize that the acquisition and warfighting  communities play critical roles in this process and therefore must continue  to work toward setting realistic program requirements and establishing an  evolutionary approach for developing new weapon programs.
We are sending copies of this report to the Secretary of Defense, the  Director of the Office of Management and Budget, and interested  congressional committees. We will also make copies available at no charge  on the GAO Web site at http://www.gao.gov.
If you have any questions about this report or need additional information,  please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. Key contributors to this report were   Karen Zuckerstein, Assistant Director; Cheryl Andrew; Lily Chin;   Sameena Ismailjee; and Sean Merrill.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report examines the Department of Defense’s (DOD) efforts to  improve its technology transition processes, with a focus on identifying  specific management, funding, and organizational practices that could  improve technology transition and weapon system outcomes. Specifically,  our objectives were to (1) identify techniques that commercial companies  use to transition mature technologies before the start of product  development and (2) assess the extent to which DOD is using these  techniques.
We used a case study approach to compare and contrast DOD and leading  commercial companies’ practices. Companies were selected on the basis  of such factors as the amount of money spent on research and  development activities over the past several years and the percentage of  change in research and development spending. For the most part, we  selected Fortune 500 companies that were in the top 100 for research and  development spending and had not experienced major cutbacks in this  funding. We also took into account the type of products each company  develops and selected companies representing several different business  sectors. Below are descriptions of the four companies featured in this  report.
subsections: section_title: Boeing
paragraphs: Boeing is a leading aerospace company and the largest manufacturer of  commercial jetliners and military aircraft combined, with capabilities in  rotorcraft, electronic and defense systems, missiles, satellites, launch  vehicles, and advanced information and communication systems. Boeing  Phantom Works develops advanced systems solutions, such as advanced  homeland security and air traffic management, as well as breakthrough  technologies, such as advanced avionics and composite materials that are  intended to significantly improve the performance, quality, and  affordability of aerospace products and services. We met with research  officials, as well as product line officials representing both its commercial  and defense sectors in Seattle, Washington.
section_title: 3M
paragraphs: 3M is a diversified technology company with a worldwide presence in  various markets, including consumer and office; display and graphics;  electronics and communications; health care; industrial; safety, security,  and protection services; and transportation. With more than 55,000  products, 3M invests more than $1 billion annually in research and  development and related activities associated with 30-plus core  technologies. The company was awarded nearly 500 U.S. patents in 2005.
We met with research, business unit, and government programs officials at  3M headquarters in St. Paul, Minnesota.
IBM is one of the world’s largest technological companies, spending about  $3 billion annually on research and development activities. It is the largest  supplier of hardware, software, and information technology services and  pioneered the development and implementation of on-demand business.  With 3,248 U.S. patents, IBM earned more patents than any other company  for the 12th consecutive year in 2004. In the past 4 years, IBM inventors  received more than 13,000 patents—approximately 5,400 more than any  other patent recipient. We met with research and product development  officials at the Watson Research Center in Hawthorne, New York.
section_title: Motorola
paragraphs: Motorola is a Fortune 500 global communications leader that provides  seamless mobility products and solutions across broadband, embedded  systems and wireless networks for products in homes, automobiles, and  workplaces. Motorola spent about $3.5 billion in 2005 on research and  development activities. Approximately 22,000 professional employees  were engaged in research activities during 2005. We met with research  officials and product development officials, as well as partnership  development managers at its offices in Schaumburg, Illinois.
For each of the companies, we interviewed senior management officials  knowledgeable about research and development activities to gather  consistent information about processes, practices, and metrics the  companies use to transition technology smoothly. In particular, we  discussed their (1) strategic planning process for identifying and  prioritizing customer needs, (2) technology development process used to  fund and mature technologies required to meet customer needs, (3) tools  used to facilitate communication between labs and product lines to  transition technology, and (4) technology transition process, including  when transition occurs, the organizations involved, and how technology is  funded throughout the transition phase. We synthesized information from  GAO’s past best practices work about technology and product  development.
To determine DOD’s practices for transitioning technology, we met with  science and technology (S&T) and acquisition officials to discuss the same  categories of questions listed above that we asked leading commercial  companies. The following is a list of the organizations we met with:    Director, Defense Research and Engineering, Alexandria, Virginia    Director, Plans and Programs    Deputy Under Secretary of Defense, Advanced Systems and Concepts    Director, Defense Advanced Research Projects Agency    Office of the Assistant Secretary of the Air Force for Acquisition  (Science, Technology, and Engineering), Alexandria, Virginia    Air Force Material Command, Wright Patterson Air Force Base, Ohio    Air Force Research Labs, Wright Patterson Air Force Base, Ohio    Joint Unmanned Combat Air Systems Program Office, Wright Patterson  Air Force Base, Ohio    Assistant Secretary of the Army Acquisition, Logistics and Technology,    Deputy Assistant Secretary for Research and Technology Chief  Scientist, Alexandria, Virginia    Joint Tactical Radio System Cluster 5 Program Office, Alexandria,    Deputy Assistant Secretary of Navy for Research, Development, Test  and Evaluation, Alexandria, Virginia    Office of Naval Research, Alexandria, Virginia    Naval Air Systems Command, Patuxent River Naval Air Station,    Naval Sea Systems Command, Washington Navy Yard, D.C.    CVN-21 Program Office, Washington Navy Yard, D.C.    Multi-mission Maritime Aircraft Program Office, Patuxent River Naval    Aircraft Launch and Recovery Equipment Program Office, Patuxent  River Naval Air Station, Maryland  At each of these locations, we collected appropriate documents that  describe the various programs, organizations, responsibilities, and  funding. We obtained examples of key documents, such as technology  transition agreements or memorandums of agreement that are used to  solidify agreements made between labs and weapon programs. We  reviewed DOD and military service strategic plans and research,  development, test, and evaluation funding documents. In addition, we  reviewed documents required as part of the selection and oversight  process for the Air Force Advanced Technology Development program,  the Army Technology Objectives program, and the Future Naval  Capabilities program. We relied on previous GAO best practices and  weapon system reports that highlight cost and schedule impacts of  launching new weapon programs with immature technology. A list of these  reports can be found at the end of this report.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: DOD Research, Development, Technology, and Engineering Budget
paragraphs: Budget (fiscal  year 2006)
1  Basic research is systematic study directed toward greater  knowledge or understanding of the fundamental aspects of  phenomena and of observable facts without specific  applications towards processes or products in mind. It  includes all scientific study and experimentation directed  towards increasing fundamental knowledge and  understanding in those fields of the physical, engineering,  environmental, and life sciences related to long-term national  security needs. It is farsighted high-payoff research that  provides the basis for technological progress.  2  Applied research is systemic study to understand the means  to meet a recognized and specific need. It is a systematic  expression and application of knowledge to develop useful  materials, devices, and systems or methods. Applied research  may translate promising basic research into solutions for  broadly defined military needs, short of system development.  Applied research precedes system-specific technology  investigations or development.  3  Advanced technology development includes development of  subsystems and components and efforts to integrate them  into system prototypes for field experiments and/or tests in a  simulated environment. The results of this type of effort are  proof of technological feasibility and assessment of  subsystem and component operability and producibility rather  than the development of hardware for service use. Projects in  this category have a direct relevance to identified military  needs. Program elements in this category involve pre- acquisition efforts, such as system concept demonstration,  joint and service-specific experiments, or technology  demonstrations, and generally have technology readiness  levels (TRLs) of 4, 5, or 6. Projects in this category do not  necessarily lead to subsequent development or procurement  phases, but should have the goal of moving out of science  and technology and into the acquisition process within years  defense program. Upon successful completion of projects that  have military utility, the technology should be available for  transition.  4  Advanced component development and prototypes consists of  efforts necessary to evaluate integrated technologies or  prototype systems in a high fidelity and realistic operating  environment. These activities include system-specific efforts  that help expedite technology transition from the laboratory to  operational use. Emphasis is on proving component and  subsystem maturity prior to integration in major and complex  system sand may involve risk reduction initiatives. Advanced  component development and prototypes efforts are to occur  before an acquisition program starts product  5  System development and demonstration consists of newly  initiated acquisition programs and includes engineering and  manufacturing development tasks aimed at meeting validated  requirements prior to full-rate production. Characteristics of  this activity involve mature system development, integration,  and demonstration to support a production decision.  6  RDT&E management support includes efforts to sustain  and/or modernize the installations or operations required for  general RDT&E. Such efforts may related to test ranges,  military construction, maintenance support of laboratories,  operation and maintenance of test aircraft and ships, and  studies and analyses I support of the RDT&E program.  7  Operational system development includes development  efforts to upgrade systems that have been fielded or have  received approval for full-rate production and anticipate  production funding in the current or subsequent fiscal year.
section_title: Appendix IV: DOD Technology Readiness Levels
subsections: section_title: 1. Basic principles observed and reported. Lowest level of technology readiness. Scientific research begins to be translated into
paragraphs: applied research and development. Examples might include paper studies of a  technology’s basic properties.
Invention begins. Once basic principles are observed, practical applications can be  invented. Applications are speculative and there may be no proof or detailed analysis to  support the assumption. Examples are still limited to analytic studies.
Active research and development is initiated. This includes analytical studies and  laboratory studies to physically validate analytical predictions of separate elements of the  technology. Examples include components that are not yet integrated or representative.
Basic technological components are integrated to establish that they will work together.  This is relatively “low fidelity” compared to the eventual system. Examples include  integration of “ad hoc” hardware in a laboratory.
Fidelity of breadboard technology increases significantly. The basic technological  components are integrated with reasonably realistic supporting elements so it can be  tested in a simulated environment. Examples include “high fidelity” laboratory integration  of components.
Representative model or prototype system, which is well beyond that of TRL 5, is tested  in a relevant environment. Represents a major step up in a technology’s demonstrated  readiness. Examples include testing a prototype in a high-fidelity laboratory environment  or in simulated operational environment.
Prototype near, or at, planned operational system. Represents a major step up from TRL  6, requiring demonstration of an actual system prototype in an operational environment  such as in an aircraft, vehicle, or space. Examples include testing the prototype in a test  bed aircraft.
Technology has been proven to work in its final form and under expected conditions. In  almost all cases, this TRL represents the end of true system development. Examples  include developmental test and evaluation of the system in its intended weapon system to  determine if it meets design specifications.
Actual application of the technology in its final form and under mission conditions, such as  those encountered in operational test and evaluation. Examples include using the system  under operational mission conditions.
section_title: Related GAO Products
paragraphs: Defense Acquisitions: Assessments of Selected Major Weapon  Programs. GAO-06-391. Washington, D.C.: March 31, 2006.
Best Practices: Better Support of Weapon System Program Managers  Needed to Improve Outcomes. GAO-06-110. Washington, D.C.: November 1,  2005.
DOD Acquisition Outcomes: A Case for Change. GAO-06-257T.  Washington, D.C.: November 15, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to  Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393.  Washington, D.C.: March 1, 2004.
Best Practices: Setting Requirements Differently Could Reduce Weapon  Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February  11, 2003.
Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept  Technology Demonstration. GAO-03-52. Washington, D.C.: December 2,  2002.
Best Practices: Capturing Design and Manufacturing Knowledge Early  Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,  2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best  Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to  Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,  2001.
Best Practices: A More Constructive Test Approach Is Key to Better  Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,  2000.
Defense Acquisition: Employing Best Practices Can Shape Better  Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April  26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System  Programs Implement Best Practices. GAO/NSIAD-99-206. Washington,  D.C.: August16, 1999.
Best Practices: Better Management of Technology Development Can  Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington,  D.C.: July 30, 1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program  Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T- NSIAD-98-123. Washington, D.C.: March 17, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon  System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires  Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:  February 24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer  Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,  1996.