GAO-04-386SP 

USING A 

KNOWLEDGE-BASED

APPROACH TO IMPROVE

WEAPON ACQUISITION

PRACTICES

BEST
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

USING A KNOWLEDGE-BASED APPROACH 

TO IMPROVE WEAPON ACQUISITIONS

The Department of Defense (DOD) is on the threshold of 

several major investments in acquisition programs that are 

likely to dominate budget and doctrinal debates well into the 

next decade. Over the next 5 years alone, DOD’s overall 

investments are expected to average $150 billion a year as 

DOD works to keep legacy systems as well as transform our 

national defense capabilities for the future. To meet this 

challenge, it is essential that sound foundations for 

investments in systems be laid now so that the resulting 

programs can be executed within estimates of available 

resources. 

At the request of the Congress, we have been examining ways DOD can 

optimize its investment in weapon systems, drawing on lessons learned from 

the best, mostly commercial, product development efforts. Leading 

commercial firms we have studied have developed increasingly sophisticated 

products in less time and at lower cost. Key to their success is their 

knowledge-based approach to the acquisition of new products. A knowledgebased approach is supported by incentives that encourage realism and candor. 

This booklet highlights the results of our work to date. We continue to explore 

additional facets of the acquisition process to identify best practices. More 

details on our work can be found in the reports cited in this brochure. 

GAO’s best practices work is done under the direction of Katherine V. 

Schinasi. For more on this work, go to www.gao.gov/bestpractices. If you have 

questions or would like to discuss our reviews, please contact Ms. Schinasi at 

(202) 512-4841 or schinasik@gao.gov. 

Jack L. Brock Jr. 

Managing Director 

Acquisition and Sourcing Management 

THIS BOOKLET DESCRIBES THE 

KNOWLEDGE-BASED APPROACH

SUCCESSFUL ORGANIZATIONS 

GENERALLY FOLLOW TO 

DELIVER SOPHISTICATED 

PRODUCTS IN LESS TIME AND 

AT LOWER COSTS. 
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

A KNOWLEDGE-BASED APPROACH PUTS ACQUISITION

PROGRAMS IN A BETTER POSITION TO SUCCEED

Leading commercial firms expect that their managers will deliver high quality 

products on time and within budgets. Doing otherwise could result in losing a 

customer in the short term and losing the company in the long term. Thus, 

these firms have adopted practices that put their individual program in a good 

position to succeed in meeting these expectations on individual products. 

Collectively, these practices ensure that a high level of knowledge exists about 

critical facets of the product at key junctures during its development and is 

used to deliver capability as promised. DOD has recognized the need to adopt 

ways of doing business that enable it to achieve similar results for its weapons 

acquisitions. 

Our reviews have shown that there are three critical junctures at which firms 

must have knowledge to make large investment decisions. First, before a 

product development is started, a match must be made between the 

customers’ needs and the available resources—technical and engineering 

knowledge, time, and funding. Second, a product’s design must demonstrate its 

ability to meet performance requirements and be stable about midway through 

development. Third, the developer must show that the product can be 

manufactured within cost, schedule, and quality targets and is demonstrated 

to be reliable before production begins. If the knowledge attained at each 

juncture does not confirm the business case on which the acquisition was 

originally justified, the program does not go forward. 

In applying the knowledge-based approach, the most-leveraged decision point 

of the three junctures is matching the customer’s needs with the developer’s 

resources. This initial decision sets the stage for the eventual outcome—

desirable or problematic. The match is ultimately achieved in every 

development program, but in successful development programs, it occurs 

before product development. In successful programs, negotiations and tradeoffs occur before product development is started to ensure that a match exists 

between customer expectations and developer resources. Leading firms thus 

make an important distinction between technology development and product 

development. Technologies that are not mature continue to be developed in 

the technology base—they are not included in a product development. 

With achievable requirements and commitment of sufficient investment to 

complete the development, programs are better able to deliver products at cost 

and on schedule. When knowledge lags, a number of risks are introduced into 

the acquisition process that can result in cost overruns, schedule delays, and 
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

inconsistent product performance. 

An approach that enables organizations to achieve a match between needs and 

resources is evolutionary product development. Under this approach, basic 

requirements are achieved first, with additional capabilities planned for future 

generations of the product. Because product development is incremental, 

achieving knowledge is more manageable. Commercial companies have found 

that trying to capture the knowledge needed to stabilize the design of a 

product with considerable new technical content is an unwieldy task—

especially if the goal is to reduce development cycle times and get the product 

to the marketplace as quickly as possible. With evolutionary development, a 

product’s design uses only components and subsystems whose reliability has 

been proven through past use or testing. 

Knowledge Points at a Glance 

 

The knowledge-based process followed by leading organizations is highlighted 

and further discussed below. 

 Knowledge point 1: Resources and needs match. Knowledge point 1 

occurs when a sound business case is made for the product—that is, a 

match is made between the customer’s requirements and the product 

developer’s available resources in terms of knowledge, time, and money. To 

determine their available resources, successful companies rely on current 

and valid information from predecessor programs, new technologies that 

have demonstrated a high level of maturity, system engineering data, and 

experienced people. Successful companies also communicate extensively 

with customers to match their wants and needs with the firm’s available 

resources and with its ability to manufacture an appropriate product. 

 Knowledge point 2: Product design is stable. Knowledge point 2 occurs 

when a company determines that a product’s design is stable—that is, it 

will meet customer requirements and cost and schedule targets. A best 
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

practice is to achieve design stability at the product’s critical design review, 

usually held midway through development. In DOD, the critical design 

review occurs when the first phase of product development—product 

integration—has been completed and the second phase—product 

demonstration—is about to begin. Completion of at least 90 percent of 

engineering drawings at the critical design review provides tangible 

evidence that the design is stable. 

 Knowledge point 3: Production processes are mature. This level of 

knowledge is achieved when it has been demonstrated that the product can 

be manufactured within cost, schedule, and quality targets. A best practice 

is to ensure that all key manufacturing processes are in statistical control—

that is, they are repeatable, sustainable, and capable of consistently 

producing parts within the product’s quality tolerances and standards—at 

the start of production. It is important that the product’s reliability be 

demonstrated before production begins, as investments can increase 

significantly if defective parts need to be repaired or reworked. 

 

Measuring Success at Key Knowledge Points 

 

The organizations we have studied apply a variety of tools and measures to 

gauge the level of knowledge that they have attained before making major 

investment decisions. 

 

 Measuring technology readiness

Many programs now use an analytical tool—technology readiness levels 

(TRL)—that can assess the maturity level of technology as well as the risk 

that maturity poses if the technology is included in a product development. 

The experiences of the DOD and commercial technology development 

cases we reviewed indicate that demonstrating a high level of maturity 

before allowing new technologies into product development programs puts 

those programs in a better position to succeed. Simply put, the more 

mature technology is at the start of the program, the more likely the 

program will succeed in meeting its objectives. 

There are nine TRLs, each denoting a level of demonstrated performance, 

beginning with concept—the lowest level of readiness—to application, 

where the technology has been “flight proven” under mission conditions. 

The higher the TRL, the smaller the gap between the technology’s maturity 

and the product’s requirements, and the lower the risk of including the 

technology in the product’s development. Our best practices work has 
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

shown that TRL 7—demonstration of a technology in an operational 

environment—is the level of technology needed to minimize risks when 

launching an acquisition program. 

Using TRLs to assess a technology’s maturity helps decisionmakers make 

informed choices about product development. If a technology lacks 

maturity, decisionmakers can choose to delay product development until 

the technology has matured sufficiently or lower the product’s 

requirements so that a less advanced but proven technology can be used. 

The more a technology has been proven, the more likely the product is to 

meet its objectives and the more likely the company can minimize 

unexpected problems and avoid potential schedule delays and cost 

overruns. 

 Measuring design readiness 

The organizations we studied also understood the importance of having 

disciplined design reviews and getting agreement from the stakeholders 

that the product’s design had been demonstrated to meet requirements 

before beginning initial manufacturing. Each organization had a design 

review process that began at the component level, continued through the 

subsystem level, and culminated with a critical design review of the 

integrated system to determine if the product was ready to progress to the 

next phase of development. 

In addition to design engineers, a cross-functional team of stakeholders in 

the process included key suppliers, manufacturing representatives, and 

service and maintenance representatives. From past experience, leading 

organizations have discovered that cross-functional teams provide a 

complete perspective of the product. 

At critical design review, the leading organizations we studied generally 

require that at least 90 percent of the engineering drawings be completed. 

They consider engineering drawings to be a good measure of the 

demonstrated stability of a product’s design because the drawings represent 

the language used by engineers to communicate to the manufacturers the 

details of a new product design—what it looks like, how its components 

interface, how it functions, how to build it, and what critical materials and 

processes are required to fabricate and test it. 
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

Other Important Tools and Practices 

Our reports have identified other tools and practices that enable successful 

implementation of the approach. These include: 

 Systems engineering for identifying gaps between requirements and 

resources so that they can be reconciled through effective trade-offs before 

product development. 

 Employment of integrated product teams to bring together in a single 

organization the different functions needed to design and manufacture a 

product, such as engineering, finance, test and evaluation, and 

manufacturing. 

 Supplier management approaches that optimize relationships with both 

contractors and subcontractors. 

 Using earned value management techniques to track a projects progress 

and assess its ability to meet cost and scheduling goals. 

 The use of a variety of testing and evaluation tools and techniques to 

validate a product’s performance early and throughout development. 

 The use of targeted, hands-on methods to ensure that program offices are 

trained on key practices. 

 Collection of statistical process control data to ensure manufacturing 

processes are consistently producing parts within quality standards. 

 Considering reasonable operating and support costs and the readiness 

or availability of equipment as requirements equal in importance to other 

performance characteristics. 

Coupling the knowledge-based approach with sound tools and techniques for 

oversight, strengthening the workforce, and ensuring the right knowledge is 

attained at the right times, increases an organization’s potential to meet cost, 

scheduling, and performance targets. It is still essential, however, that the 

right incentives and resources be in place to encourage program managers to 

employ such measures. 
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

FOLLOWING A KNOWLEDGE-BASED ACQUISITION PROCESS IS 

CRITICAL FOR WEAPON ACQUISITIONS

The majority of weapon system acquisitions we have reviewed over the past 

several decades experienced problems during acquisition that drove up costs 

and schedules and increased technical risks. Many programs have been 

restructured by DOD in the face of delays and cost growth; a few have been 

canceled. We have found that these problems are largely rooted in the failure 

to match customer’s needs with the developer’s resources—technical 

knowledge, timing, and funding--when starting product development. In other 

words, commitments were made to delivering capability without knowing 

whether technologies being pursued could really work as intended. Time and 

costs were consistently underestimated. Problems that surfaced early cascaded 

throughout development and magnified the risks facing the program. 

On a number of programs, DOD has shown that it can manage its weapon 

system acquisition process to ensure important knowledge about technology 

and requirements, design, and manufacturing is captured and used to make 

informed and timely decisions before committing to substantial development 

and production investments. Moreover, over the past few years, DOD has 

WHEN KNOWLEDGE IS NOT ATTAINED

AT KEY JUNCTURES

 Launching an acquisition program before 

requirements and available resources are 

matched can result in a product that fails 

to perform as expected, costs more, or 

takes longer to develop. 

 Failure to ensure design stability about 

halfway through product development can 

result in design changes that are more 

costly to correct later in development or 

after the product is fielded. 

 Entering production before manufacturing 

processes are under control can result in 

product defects that require additional 

resources to rework or scrap—a costly and 

inefficient practice.
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

made constructive changes to its acquisition policy to embrace best 

practices—especially those related to technology maturity and separating 

technology development from product development. If faithfully implemented, 

DOD’s policies of evolutionary acquisition and phased requirements will make 

it easier to match resources with needs before starting a new product 

development. Funding pressures and the need to modernize systems across 

the Department will continue to make it paramount for DOD to adopt 

practices that can save money and deliver new capabilities quicker. 

However, a number of incentives—many tied to funding—continue to 

undermine DOD’s ability to achieve a match between needs and resources 

at the onset of weapon acquisition programs. Unlike the commercial world 

where the focus is on delivering a product to market, DOD’s system focuses 

on competing for resources. In the competition for funding, managers are 

encouraged to launch product developments before technologies are 

mature. Because a proven way to win support for a new weapons 

acquisition program is to promote unprecedented performance features 

and design characteristics, managers have implicit incentives to rely on 

immature technologies. Moreover, because funding is competitive and 

DOD’s forecasts of cost, schedule, and performance are largely based on 

immature technologies and other unknowns, estimates tend to be squeezed 

into insufficient profiles of available funding. Other factors, such as short 

tenures and career pressures, discourage program managers from saying no 

to requirements that are later discovered to have been unreasonable. Thus, 

to meet the investment challenge of modernizing its forces, DOD will need 

not only to implement policies that embrace evolutionary, knowledgebased acquisitions, it will need to instill incentives that encourage realism 

and candor in the acquisition process and sustain its commitment to 

improving business practices. 
BEST PRACTICES

JANUARY 2004 GAO-04-386SP U.S. GENERAL ACCOUNTING OFFICE

BEST PRACTICES REPORTS

 

These reports can be found on the GAO Web site at www.gao.gov. 

Defense Acquisitions: Assessments of Major Weapon Programs, GAO-03-476, 

Washington, D.C.: May 2003. 

Best Practices: Setting Requirements Differently Could Reduce Weapon 

Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 

2003. 

Best Practices: Capturing Design and Manufacturing Knowledge Early 

Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002. 

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. 

GAO-02-469T. Washington, D.C.: February 27, 2002. 

Best Practices: DOD Teaming Practices Not Achieving Potential Results. GAO01-510, Washington, D.C.: April 2001. 

Best Practices: Better Matching of Needs and Resources Will Lead to Better 

Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001. 

Best Practices: A More Constructive Test Approach Is Key to Better Weapon 

System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000. 

Defense Acquisitions: Employing Best Practices Can Shape Better Weapon 

System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. 

Best Practices: DOD Training Can Do More to Help Weapon System Programs 

Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 

1999. 

Best Practices: Better Management of Technology Development Can Improve 

Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 

1999. 

Best Practices: DOD Can Help Suppliers Contribute More to Weapon System 

Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. 

Best Practices: Successful Application to Weapon Acquisition Requires 

Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: 

February 24, 1998. 

Best Practices: Commercial Quality Assurance Practices Offer Improvements 

for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.