United States General Accounting Office

GAO Testimony

Before the Subcommittee on Acquisition and Technology,

Committee on Armed Services, U.S. Senate

For Release on Delivery

Expected at

9:30 a.m.,EST

Wednesday,

March 18, 1998

DEFENSE ACQUISITION

Improved Program

Outcomes Are Possible

Statement of Louis J. Rodrigues, Director, Defense

Acquisitions Issues, National Security and International

Affairs Division

GAO/T-NSIAD-98-123

Mr. Chairman and Members of the Subcommittee:

We are pleased to be here today to discuss issues facing the Department of

Defense (DOD) in its acquisition of weapon systems, related spare parts,

and other goods and services. In response to the many changes that have

been witnessed in the defense acquisition environment over the last few

years, DOD has begun broad-based changes to its acquisition and

contracting processes. However, weapon programs continue to have

questionable requirements; unrealistic cost, schedule, and performance

estimates; and strategies that begin production before adequate testing has

been completed. This discussion of acquisition issues is well-timed, as DOD

implements plans to increase its procurement budget to $60 billion in

fiscal year 2001—a 40-percent increase over last fiscal year’s budget.

My testimony focuses on a different approach to improving weapon

acquisition outcomes based on best commercial practices and an

understanding of the acquisition culture. My testimony also includes some

observations on (1) DOD’s management of its acquisition workforce and

organization, (2) DOD’s experience with commercial pricing of spare parts,

(3) the effectiveness of DOD’s mentor-protege pilot program, and

(4) federal agencies’ use of multiple award task- and delivery-order

contracts.

Results in Brief Improved outcomes from the weapon systems acquisition process—that

is, acquiring systems better, cheaper, and faster—are possible if the

incentives that drive behaviors are changed. The best commercial

companies have found processes and decision-making practices that are

based on knowledge and focused on production to be successful.

Employing such processes and practices can improve weapon acquisitions

if DOD and the Congress are able to foster an environment that provides

program managers with incentives for applying best practices. In our

February and March 1998 reports, we recommended that the Secretary of

Defense separate technology development from product development,

send signals through decisions on individual programs that encourage

acquisition managers to identify unknowns and ameliorate their risks early

in development, and develop and disseminate throughout DOD and the

defense industry a policy that promotes productive supplier relationships

and their importance to improving program outcomes. In addition, we

asked the Congress to consider supporting these efforts through its

funding and oversight mechanisms.

Page 1 GAO/T-NSIAD-98-123 Defense Acquisition
Over the last several years, the Congress has mandated cuts in DOD’s

acquisition workforce and called on DOD to submit plans to streamline and

restructure its acquisition organizations. DOD has been able to reduce the

acquisition workforce, but it has had more difficulty changing the

structure that underlies decision-making in the acquisition process. We

have previously reported that the connection among DOD’s acquisition

workforce, organizations, and outcomes of the acquisition process should

be considered in making changes.

For an increasing number of sole-source spare parts, DOD is transitioning

from a cost-based pricing environment to a market-based or commercial

pricing environment where price analysis is the principal means used to

negotiate the reasonableness of prices. Regarding sole-source,

commercially priced spare parts for which DOD is the predominant buyer,

some DOD contracting personnel expressed concerns about (1) how to

determine whether the prices offered are fair and reasonable, (2) future

contract negotiations where recent cost-based historical prices may not be

available, and (3) the sometimes conflicting pressures between obtaining

fair and reasonable prices and negotiating contracts in time to meet

customer needs and avoid backlogs. So far, our work indicates that while

some contractors are now offering commercial prices that are significantly

higher than DOD paid in the past, there are some questions about how well

contracting officers understand the basis of such prices.

DOD has spent over $200 million on a mentor-protege program to provide

incentives for major DOD contractors (mentors) to furnish disadvantaged

small business owners (proteges) with assistance designed to enhance

their capabilities and increase their participation as suppliers under DOD,

other federal government, and commercial contracts. However, DOD lacks

information needed to determine whether the program is effective.

Strengthened performance reviews could ensure that sufficient and

reliable information is available to assess the program’s effectiveness.

Our work on multiple award task- and delivery-order contracts shows that

federal agencies are not consistently achieving competition when placing

orders against these contracts. We also found that the fees charged to

agencies that place orders on another agency’s contract varied greatly.

Weaknesses in agency accounting and management systems prevented us

from determining if the fees were reasonable. We found that the use of

multiple award contracts did not impair small business’ ability to compete

for such contracts.

Page 2 GAO/T-NSIAD-98-123 Defense Acquisition
The Need for a

Different Approach to

Improving Weapon

Acquisition Outcomes

This morning we will not focus on individual weapon system problems;

rather, we will concentrate on the underlying reasons for such problems

and what can be done about them. In two recent reports prepared at the

request of this Subcommittee,1

 we identified several commercial practices

in the areas of product development and supplier relationships that have

the potential for significantly improving weapon system outcomes. Of

particular note are the different incentives we found that operate in the

two sectors and their primacy in determining program management

practices. Simply put, practices are adopted and persist because they

work—they help programs succeed in their environment. Thus, the way to

get lasting reform is to realign the incentives of the weapons acquisition

process with desired program outcomes; specific practices can then show

the way to better outcomes. Changing these incentives will take the efforts

of the Congress as well as DOD and the services, for all participants in the

acquisition process influence its incentives.

Clear Differences Between

Best Commercial Practices

and Weapon Acquisition

Practices Exist

Our work on the transition of major products to production and on the

management of suppliers shows that best commercial practices and DOD

practices differ considerably. Before discussing these differences, let me

clarify that we do not hold the view that commercial is good and DOD is

bad. Not all commercial firms exhibit best practices and the leading firms

make their share of mistakes. Also, we found some promising practices in

weapon systems that could have application to other programs.

In our transition to production work, we characterized knowledge on

product developments in terms of three junctures: when a match is made

between the customer’s requirements and the available technology, when

the product’s design is determined to be capable of meeting performance

requirements, and when the product is determined to be producible within

cost, schedule, and quality targets. For the purposes of comparing

commercial and DOD product developments, we have characterized the

points at which virtual certainty of each of these aspects of a product is

achieved as a “knowledge point.” Figure 1 illustrates the three knowledge

points and the differences between the commercial and military product

developments in terms of when they attain knowledge.

1

Best Practices: Successful Application to Weapon Acquisitions Requires Changes in DOD’s

Environment (GAO/NSIAD-98-56, Feb. 24, 1998) and Best Practices: DOD Can Help Suppliers

Contribute More to Weapon System Programs (GAO/NSIAD-98-87, Mar. 18, 1998).

Page 3 GAO/T-NSIAD-98-123 Defense Acquisition
Figure 1: Comparison of Three Key Knowledge Points for Commercial and Military Product Developments

| |

| |

Knowledge Point 1

Knowledge Point 2

Knowledge Point 3

Commercial Product Development

Program Launch Production Start

DOD Product Development

Program Launch Production Start

Knowledge Point 1

Knowledge Point 2

Knowledge Point 3

Unknowns Knowns

Knowledge Point 1: Knowledge that a match exists between technology and requirements.

Knowledge Point 2: Knowledge that the design will work as required.

Knowledge Point 3: Knowledge that the design can be produced within cost, schedule, and 

quality targets.

Page 4 GAO/T-NSIAD-98-123 Defense Acquisition
Commercial firms gain more knowledge about a product’s technology,

performance, and producibility much earlier in the product development

process than DOD. Product development in commercial ventures is a

clearly defined undertaking for which firms insist on having the

technology in hand to meet customer requirements before starting. Once

underway, these firms demand—and get—specific knowledge about a new

product before production begins. The process of discovery—the

accumulation of knowledge and the elimination of unknowns—is

completed for the best commercial programs well ahead of production.

Not having this knowledge when demanded constitutes a risk the firms

find unacceptable. Immature or undeveloped technology cannot meet

these demands and is kept out of commercial product development

programs; this technology is managed separately until it can meet the

demands for product development.

In contrast, DOD programs allow technology development to continue into

product development. Consequently, the programs proceed with much

less knowledge—and thus more risk—about required technologies, design

capability, and producibility. The programs’ discovery process persists

much longer, even after the start of production. Not having the same level

of knowledge as commercial firms explains much of the turbulence in DOD

program outcomes as the transition to production is made. It is a

predictable consequence that can be forecast early by the use of

knowledge points or other metrics. It is complicated by the fact that

although DOD accepts more unknowns on its programs than commercial

firms, it understates the risks present.

Knowledge Point 1: Customer

Requirements and

Technological Capability Are

Matched

To minimize the amount of technology development that occurs during

product development, the companies we visited employ a disciplined

process to match requirements with technological capability before the

product development process begins. This process is grounded in

production realities that demand proof that the technology will work and

can be produced at an acceptable cost, on schedule, and with high quality.

The companies bring proven technological knowledge to the requirements

process in the form of current, high-fidelity information from predecessor

programs, people with first-hand experience on those programs, and new

technologies deemed mature as a result of having “graduated” from a

disciplined technology development and screening process. In addition,

they communicate extensively with customers to match their wants and

needs to the firms’ available technology and ability to manufacture an

appropriate product. They do not stray far from their technological

foundation.

Page 5 GAO/T-NSIAD-98-123 Defense Acquisition
We found examples of best commercial practices for matching

requirements to technology at Boeing, Hughes, and Ford. Boeing

communicated with the airlines to set achievable requirements for the

777-200 airplane and tested the design early. Boeing also applied lessons

learned from past programs to (1) ensure that technology was mature

before the 777 was launched2

 and (2) eliminate additional requirements

once the development program began. Hughes used a technology

development process that graduated new technologies from concept into a

product development program, enabling the firm to make what it saw as

quantum performance increases with mature technologies on its new

satellite, the HS-702. Ford uses its technology deployment process and

“Wall of Invention” to separate immature technology from a new product’s

development.

DOD programs did not attain a match between technology and

requirements at the time of launch. DOD accepted varying—but

consistently higher—degrees of technological risk on the four programs

we reviewed. Some examples from the F-22 illustrate that a match

between requirements and technology is still not certain in that program.

The F-22 program includes 10 newly developed derivatives of existing

materials that are important to its low-observability feature. The

performance and maintainability of these materials will not be completely

verified until 2 years after production begins. Similarly, it will be after

production before it is certain that the avionics software, which features a

level of integration not previously achieved in a fighter, is a match for the

performance requirements. The F-22 engines have many advanced features

to meet aggressive performance requirements and program officials

acknowledge that it is still unclear whether the engines will meet all of the

requirements. Leading commercial firms do not do this—they do not

tolerate basic unknowns about the performance of a new product to

persist this long. In fact, resolving these kinds of unknowns is a precursor

to starting a commercial product development.

Another example that illustrates the different knowledge standards are the

commercial and DOD decisions made on a lightweight aircraft

material—aluminum lithium. Boeing had initially decided to use the alloy

on its 777-200 aircraft but rejected its use early in development because it

was expensive, its manufacturing processes were not well understood, and

its availability was limited. It was willing to pay a weight penalty rather

than accept the unknowns associated with aluminum lithium. DOD

2

We define program launch or start as the point at which organizations define a product’s performance,

cost, and schedule estimates and commit to making the financial investment needed to complete

development and bring the product into production.

Page 6 GAO/T-NSIAD-98-123 Defense Acquisition
accepted these risks and used the alloy on the C-17 aircraft. The first

production aircraft contained 2,200 pounds of the alloy. Its application

proved unsuccessful as some of its unknowns became problems.

Consequently, the decision was made to discontinue aluminum lithium

and it will be phased out of the program by the 51st production airplane.

Knowledge Point 2: The Design

Will Perform as Required

The commercial firms we visited achieved near certainty that their product

designs would meet customer requirements and had gone a long way to

ensure that the product could be produced before the halfway point of

product development. Both DOD and commercial firms hold a critical

design review (CDR) to review engineering drawings, confirm the design is

mature, and “freeze” it to minimize changes in the future. The completion

of engineering drawings and their release to manufacturing organizations

signify that program managers are confident in their knowledge that the

design performs acceptably and is mature. The drawings are critical to

documenting this knowledge because they are not only precision

schematics of the entire product and all of its component parts—they also

reflect the results of testing and simulation and describe the materials and

manufacturing processes to be used to make each component. Both DOD

and commercial companies consider the design to be essentially complete

when about 90 percent of the engineering drawings are completed. 

Figure 2 compares what knowledge, in the form of released drawings, was

in hand at the time of the critical design review for the commercial and

DOD programs we reviewed.

Page 7 GAO/T-NSIAD-98-123 Defense Acquisition
Figure 2: Comparison of When Commercial and DOD Programs Achieve Knowledge About Their Product’s Design

Hughes HS-702

Boeing 777-200

DOD F-22

DOD C-17

Program Launch Production Start

CDR and Knowledge Point 2

CDR and Knowledge Point 2

CDR Knowledge Point 2

CDR Knowledge Point 2

Unknowns Knowns

Program Launch Production Start

Commercial firms such as Boeing and Hughes told us they typically had

over 90 percent of these drawings available for the CDR. Boeing began

releasing engineering drawings when product development began in 1990

and completed the release process in 1992, less than 2 years later. Once

the CDR was complete, Boeing strictly enforced the design freeze for the

777-200. For example, Boeing incorporated a customer requirement to

include folding wingtips, along with the supporting bulkheads, into the 

777 design at a cost of nearly $40 million. Later, the customer decided the

folding wingtips were not necessary; however, Boeing left the bulkheads

Page 8 GAO/T-NSIAD-98-123 Defense Acquisition
in the wings anyway because all of the engineering drawings were

completed and the risk of introducing changes, even though the changes

would have saved weight, was considered too high relative to cost and

schedule targets.

The C-17 and the F-22 programs had less knowledge—in the form of test

results or engineering drawings—about their designs than commercial

companies did at the time they held their CDRs. The programs did not get

or were not projected to get to the same level of completion on the

drawings until later in the development cycle, which placed greater

reliance on the lesser information available at the time of the review.

Specifically, the C-17 program had less than 60 percent and the 

F-22 program less than one-third of the drawings available for the CDR.

Over one-fifth of the C-17’s drawings became available after production

began, and the aircraft experienced a number of problems in production

as difficulties with the design were worked out. Several key technologies

are still unproven on the F-22, and some will not be proven out until after 

40 aircraft have entered production. Nonetheless, the risks of proceeding

with the rest of development as planned at the time of the CDRs for both

programs were deemed acceptable.

Even though it is still too early to predict outcomes on the AIM-9X missile

and the Joint Direct Attack Munitions (JDAM) programs, their prospects

appear promising because they have chosen mostly proven technology

from existing programs to achieve performance requirements.

Knowledge Point 3: Production

Units Will Meet Cost, Quality,

and Schedule Objectives

The companies we visited reached the point at which they knew that

manufacturing processes would produce a new product conforming to

cost, quality, and schedule targets before they began fabricating

production articles. Reaching this point meant more than knowing the

product could be manufactured; it meant that all key processes were

under control, such that the quality, volume, and cost of their output were

proven and acceptable. As indicated in figure 3, the DOD programs

demanded less proof of the design’s producibility before the product

transitions to production.

Page 9 GAO/T-NSIAD-98-123 Defense Acquisition
Figure 3: Comparison of When Commercial and DOD Programs Achieve Knowledge That Processes Can Produce a

Product

Knowledge Point 3

Knowledge Point 3

Knowledge Point 3 has 

not yet been reached.

Hughes HS-702

Boeing 777-200 

DOD F-22 

Program Launch Production Start

Unknowns Knowns

Knowledge Point 3 will

be reached well after

production begins.

Program Launch Production Start

DOD C-17

The commercial firms relied on known manufacturing processes and

statistical process control data to achieve this knowledge early and, in

fact, had all their key processes under statistical control when production

began. The ability to establish control for key processes before production

began was the culmination of all the practices employed to identify and

reduce risk. All of the companies we visited agreed that knowledge about

technology and design up front in the process makes the control of

processes possible and the transition to production smooth.

Page 10 GAO/T-NSIAD-98-123 Defense Acquisition
The C-17 program began production in 1989 and still has less than

13 percent of its key manufacturing processes in control. The 

F-22 program is currently faring better than the C-17: The contractor

believes it has almost 40 percent of its key manufacturing processes in

control, 2 years before production is scheduled to begin. However, the

program does not plan to have all key processes in control until about

4 years into production. Both programs experienced basic producibility

problems that were not discovered until late in development or early in

production. Problems occurred despite completing production readiness

reviews that were intended to reduce producibility problems and having

exit criteria to ensure that risks were acceptable and enough knowledge

had been gained to enter the next development phase.

The C-17 program discovered major design changes of the wings, flaps,

and slats were required after the critical design review. These changes

caused costly changes to processes; forced the manufacturers to develop

workaround plans; and resulted in high rates of scrap, rework, and repair.

The production preparations for the F-22 illustrate the limitation of a

review mechanism when a substantial amount of knowledge is unattained.

The initial production readiness review, held in 1995 when only about

one-third of the engineering drawings were released, did not report any

high risks in manufacturing or producibility. In 1996, an independent team

mandated by the Air Force reviewed the program and discovered

numerous manufacturing and producibility problems, such as

underestimated complexity in manufacturing processes, understated labor

requirements, immature definition of avionics flight test requirements, and

concerns about software integration.

Management of Suppliers Leading commercial companies have found that more cooperative

business relationships with suppliers have led to lower costs, higher

quality, and shorter cycle times. In both commercial and defense products,

suppliers account for much, if not most, of the product content and

technical innovation. In our review of supplier relationships, we also

found key differences between best commercial practices and DOD

practices. The best practices of commercial firms recognized as industry

leaders in the area of supplier relationships can be aggregated into four

traits:

• providing the central support necessary to optimize supplier relations,

which were seen as essential to maximizing product success;

• implementing a rigorous supplier selection process, which created a

manageable pool of strong suppliers;

Page 11 GAO/T-NSIAD-98-123 Defense Acquisition
• creating channels for open communication and continuous assessment of

performance for both customer and supplier; and

• creating an environment whereby the suppliers also benefited from

superior performance.

The leading commercial firms go beyond simple supplier relationships that

are limited to the purchase of goods and services in return for payment.

Their relationships evolved to the sharing of information and interaction

on a variety of business functions in a joint effort to make a better quality

product, more quickly, and less expensively. Both the firm responsible for

the complete product and its suppliers benefited from the process. Thus,

we see the four traits as the components of a self-sustaining system,

shown in figure 4.

Figure 4: System of Four Traits Seen in Commercial Best Practices

Strong

Central

Support

Rigorous

Supplier

Selection

Clear

Communications

and Feedback

Investment In Supplier Relations

Better Product Outcomes Greater Supplier Contribution To Product

Rewarding

Environment

For Suppliers

We discussed supplier relationships with several defense prime

contractors and did detailed work on the Brilliant Anti-armor Submunition

program, referred to as BAT, and the JDAM program. We found that in a

more traditional program, like BAT, the four traits do not comprise as

powerful a system as is formed by the best commercial practices. While a

number of the practices that make up the middle two traits have been

adopted, their impact on the BAT program was blunted by weaknesses in

central support and providing a mutually rewarding environment. The

commitment of the prime contractor to improve supplier relationships was

not perceived by some key suppliers as having been much more than

Page 12 GAO/T-NSIAD-98-123 Defense Acquisition
procedural changes. Part of the reason is that although DOD shares

responsibility for determining what is important in managing an individual

program, its traditional approach has been to maintain an “arm’s length”

relationship with prime contractors and have little involvement with

suppliers. On the JDAM, DOD was able to create a better environment for

fostering mutual benefits between defense prime contractors and their

suppliers.

Differences in Practices

Reflect Different

Incentives

The differences in the practices employed by the leading commercial firms

and DOD are not necessarily explainable by differences in tools, techniques,

or talent. Rather, the differences in the actual practices reflect the

different demands imposed on programs by the culture or environment in

which they were managed. Indeed, the way success and failure are defined

for commercial and defense product developments differs considerably,

which creates a different set of incentives and different behaviors from the

people managing the programs. Specific practices take root and are

sustained because they help a program succeed in its environment. In this

sense, practices are adopted because they work—not because they are

textbook solutions. In our transition to production review, we observed

that, with the possible exception of having more experience with repeated

product developments than a DOD program manager, commercial program

managers were not better or more ethical than their DOD counterparts. On

supplier relations, we noted that it is a major undertaking for a firm to

commit the resources to implement an active supplier policy. Such a

commitment is not based on altruism or management theory; rather, the

commitment comes from the desire to maintain a competitive edge in

preserving or increasing a firm’s market share.

Commercial Practices Are

Driven by the Customer’s

Acceptance of the Finished

Product

The commercial firms we contacted launch a product development

program only when a solid business case can be made. The business case

basically revolves around the ability to produce a product that will sell

well enough to make an acceptable return on investment. The point of sale

occurs after product development is complete; program success is

determined in production when the customer buys the finished product. If

the firm has not made a sound business case or has been unable to deliver

on one or more of the business case factors, it faces a very real prospect of

failure in the form of the customer “walking away.” Production is a

dominant concern throughout the product development process and

forces discipline and tradeoffs in the design process. This environment

encourages realistic assessments of risks and costs; doing otherwise

would threaten the business case and invite failure. For the same reasons,

Page 13 GAO/T-NSIAD-98-123 Defense Acquisition
the environment places a high value on knowledge for making decisions.

Incentives favor identifying unknowns early, designating them an

appropriate high risk, and aggressively eliminating them. Practices, such

as achieving statistical process control before production, are adopted

because they help ensure success.

Boeing described the business case for the 777-200 product development

as a “money wheel” that must be balanced across all of its factors. These

factors include a market opportunity, a product whose technical features

can satisfy the market, available investment capital, a cycle time short

enough to get the product to market on time, and a unit production cost

that will yield an acceptable return on investment. Boeing informed us that

if any factor gets out of line, either through estimating errors or changing

conditions, the “wheel” will not turn, and profitability—and perhaps

corporate reputation—could be lost. The program manager is judged by

these standards, unlike in DOD.

Once a company decides to launch a product development, strong

incentives—both positive and negative—serve to keep the programs on

track. To meet market demands, leading commercial companies build

relatively short cycle times into decisions to begin a product’s

development. Boeing’s 777-200 went to production less than 5 years after

development began, Hughes’ HS-702 took about 26 months, and Chrysler

developed its Dodge Durango sport utility vehicle in 24 months. These

short timeframes make the day of reckoning—sale of the produced item to

the customer—close at hand. Consequently, production—on time, at rate,

at cost, and with quality—looms as a near-term reality that continues to

greatly influence subsequent design and configuration decisions within the

framework of the business case. The incentives that operate in the

commercial environment encourage program managers to want risks

identified early, be intolerant of unknowns, and not rely on testing as the

main vehicle for discovering the performance characteristics of the

product. By protecting the business case as the key to success, program

managers are conservative in their estimates and aggressive in risk

reduction. Ultimately, preserving the business case enables them to say

“no” to pressures to accept risks or unknowns.

DOD Practices Reflect the Need

to Succeed in Funding and

Managing the Development

Effort

The business case for a major weapon system is quite different. It is

characterized by a stated need for a military capability; a proposed

weapon system for which the demands of a successful launch dictate

optimistic technological, cost, and schedule estimates; and a development

Page 14 GAO/T-NSIAD-98-123 Defense Acquisition
effort for which candor about progress is curbed by the competition for

continued funding.

Traditionally, needs for new weapons have been generated by individual

branches within each service. Once a need has been established, a product

development vying for launch faces intense competition for initial funding.

DOD typically defines and launches a program years earlier in the process

than a commercial product development, and thus, the case for the

product is made when much less is known about technology, cost, and

schedule. The knowledge required to make the business case to launch a

commercial product development is generally not available for a DOD

program until well into the engineering and manufacturing development

phase. In a very real sense, the point of sale begins much earlier on a DOD

program and recurs more often because the customer (DOD and the

Congress) pays for the product on an annual installment basis from

program launch. Success, then, for most of the product development

cycle, is measured in terms of ability to secure the next installment.

Because this approval must be won every year, it creates incentives to

make the program’s case look attractive.

The competition for funding at the time of launch encourages aspiring DOD

programs to include performance features and design characteristics that

rely on immature technologies. Untempered by knowledge to the contrary,

the risks associated with these technologies are deemed acceptable.

Because production can be 15 years from the launch decision, it is difficult

for production realities and concerns to exert as much influence on a DOD

product development as they do on commercial products. Instead, design

features and performance are more dominant. More unknowns are

accepted on a DOD program, and their attendant risks are often

understated. This combination, which can be devastating to a commercial

business case, can help a weapon system program get launched and

survive.

Other pressures on DOD programs at launch make tough demands for

knowledge that does not yet exist. A product development deemed worthy

cannot be launched unless development and production funding is

available over the right time period. The product’s development and

production cost, as well as timing, must fall within available funding.

Because DOD relies largely on forecasts of cost, schedule, and performance

that are comparatively soft at this stage, funding competition encourages

the cost and schedule estimate to be squeezed into profiles of available

Page 15 GAO/T-NSIAD-98-123 Defense Acquisition
funding. Additional requirements, such as high reliability and ease of

maintenance, serve to make the fit even tighter.

As a product development proceeds in DOD, its success is still measured in

terms of the funding it receives. Success translates into getting the funding

request approved each year; failure can mean anything from a significant

funding cut to cancellation. This view of success is reinforced by the fact

that, unlike commercial programs, DOD programs do not receive full

corporate support throughout development. Individual programs face

scrutiny by service executives, the Office of the Secretary of Defense,

independent cost estimating and test agencies, audit agencies, and several

committees and subcommittees of the Congress. Given this amount of

competition and oversight, the detection of a problem on an individual

program makes that program vulnerable to criticism and possible loss of

funding support. Ironically, it is these same pressures that encourage

overreaching at the time of program launch. By the time a DOD program

makes it through this development process and begins production, the

customer is deeply invested and unlikely to walk away. Thus, success, in

terms of program continuance, is substantially ensured before end items

are produced.

The pressures and incentives in the DOD environment explain why the

behaviors of managers and other sponsors of product developments differ

from those in commercial programs. According to a 1994 study done for

the Under Secretary of Defense for Acquisition, government program

managers found their formal role of objective program management at

odds with their informal role of program advocates.3

 According to the

study:

“A feeling of responsibility for program advocacy appears to be the primary factor causing

government managers to search aggressively and optimistically for good news relating to

their programs, and to avoid bad news, even when it means discrediting conventional

management tools that forecast significant negative deviations from plan.”

In this environment, risks in the form of ambitious technology

advancements and tight cost and schedule estimates are accepted as

necessary for a successful launch. Problems or indications that the

estimates are decaying do not help sustain the program in subsequent

years, and thus, their admission is implicitly discouraged. An optimistic

production cost estimate makes it easier to launch a product development

and sustain annual approval; admission that costs are likely to be higher

3

Critical Issues in the Defense Acquisition Culture, Defense Systems Management College, Dec. 1994.

Page 16 GAO/T-NSIAD-98-123 Defense Acquisition
could invite failure. There are few rewards for discovering and recognizing

potential problems early in the DOD product development. For commercial

product developments, an optimistic production cost estimate will mean

failure of sales or profit; admission of cost increases early invites

aggressive problem-solving behaviors to restore the business case. The

behavior of tolerating unknowns and not assigning them the same risk

value as in the commercial environment is rational in the DOD environment

because there is little incentive to admit to high risks before it is

absolutely necessary, as long as the resulting estimates are accepted by

DOD and the Congress. In fact, admitting risk may doom the program.

Behaviors toward testing follow a similar logic. On commercial product

developments, much more is known about the product’s performance at

the beginning of development. Testing is used to confirm knowledge and

identify weaknesses or limits in the product. It is consistent with a firm’s

anxiety to eliminate unknowns to preclude failure in production. DOD

product developments are much more dependent on testing to discover

technical performance characteristics and answer the question of whether

the product will work. DOD tests serve more than the purpose of

discovering or confirming performance characteristics—they are

examinations on which the program must get good grades or face failure

in the form of withdrawal of support. Good test results can help a

program, whereas negative test results are equated with failure.

Unknowns, then, present a safer course of action; if testing does not occur

until late in the product development, forecasts of product performance

will serve as the best information available.

A Rewarding Environment Is

Key to Fostering the Best

Supplier Relationships

In our work on supplier relationships, we also found that the environment

DOD created on an individual program greatly affected the contribution

suppliers made to the outcomes of the program. The best supplier

practices were sustained when a commercial firm created an environment

in which it became an attractive customer. Firms did this by not only

rewarding superior suppliers with future business but by building

partnerships, allowing top suppliers to participate in product planning and

design, sharing business plans, and relaxing the procedures for doing

business together. In turn, the key suppliers were willing to go the extra

mile, commit their own resources to enhance prospects for future

business, and comply with the rigor that the source selection and

evaluation mechanisms demanded. The suppliers’ responses improved

product output and reinforced the initial commitment that the product

developer made to strengthening supplier relationships.

Page 17 GAO/T-NSIAD-98-123 Defense Acquisition
Mutual trust—earned through action—was essential to creating this

environment. For example, Chrysler’s relationships with some suppliers

had evolved to the point that it no longer needed to make large

investments in some key technology areas because of the relationships it

had developed with some suppliers. Instead, the suppliers made the

technology investment themselves and had enough confidence in their

relationship with Chrysler that they did not fear the long-term commitment

that this entailed. For its part, Chrysler trusted the suppliers to make the

investments that would keep their vehicles competitive.

On weapon system programs, DOD shares responsibility with the prime

contractors for the acquisition policies that shape the suppliers’

environment. Thus, the role it plays on individual programs has a direct

bearing on the sophistication of supplier relationships and the success of

best supplier practices. The supplier relationships on the BAT program

reflect DOD’s traditional role of distancing itself from suppliers. This role

can be traced, in part, to the fact that DOD has not articulated a particular

supplier policy to guide program managers. By default, DOD’s concerns

over interfering with the contractual relationship between the prime and a

supplier have encouraged an arms-length approach to suppliers by

managers.

Nonetheless, some key BAT suppliers did not see their environment as

conducive to such relationships. They viewed their role as only complying

with the design requirements handed down to them by the upper-tier

firms. They believed that attempts to do more—such as offer design

suggestions or make long-term investments—would not reap benefits.

Some suppliers believed no consideration was given to their years of

working together when it came to the low-rate production contract

proposal.

On the JDAM program, DOD was much more proactive and involved with the

suppliers. Its status as a pilot program that was afforded early statutory

and regulatory relief helped support the program office’s involvement in

seeing that best supplier practices were used. As a result, high performing

suppliers were selected, all tiers of suppliers participated in meeting the

program’s priorities, and long-term benefits were offered to the prime

contractor and its suppliers for good performance. The ultimate success of

this approach in producing a weapon that will perform as required remains

to be seen. Nonetheless, suppliers praised the approach for the

relationships it fostered.

Page 18 GAO/T-NSIAD-98-123 Defense Acquisition
A Broader Perspective of

the Acquisition Culture Is a

Precursor to Reform

Our findings relative to the different environments that commercial

product developments and weapon system programs operate in are

consistent with the results of a study we completed in 1992.4

 At that time,

we took a retrospective look at the previous 15 years of our reviews of

weapon systems and the acquisition process to identify the underlying

factors that contributed to recurring acquisition difficulties. We concluded

that if changes in the acquisition of weapons were to be of a lasting nature,

acquisition problems needed to be looked at from another perspective—as

the consequences of a way of acquiring weapons that had become deeply

rooted over the years. Those findings have since been reinforced by the

1994 Defense Systems Management College study, which included the

participation of 80 experienced government and defense industry

acquisition managers.

The acquisition culture can be defined as the collective behavior of the

various participants5

 in the acquisition process and the forces that

motivate their behavior. In fact, the process may be more realistically

portrayed as the interaction of its participants than the methodological

procedure depicted on paper. This culture has evolved as the acquisition

process has become a vehicle for meeting the diverse needs of its

participants. This depiction of the acquisition culture does not stem from a

pejorative view of individual participants or organizations. Rather, they do

what they believe is right given the pressures they face. The difficulty lies

in the fact that there is no consensus on what is right. In the absence of

such a consensus, the acquisition process serves to satisfy the diverse

needs of its participants within the umbrella of providing U.S. forces with

the best weaponry. In so doing, the incentives of the process—both

positive and negative—favor maximizing programs.

The acquisition culture offers an explanation for why problems with

weapon acquisitions persist despite numerous attempts at reform.

Reforms, in essence, have championed sound management practices, such

as realistic estimating, thorough testing, and accurate reporting without

necessarily affecting why they did not happen in the first place. For

example, there have been recommendations aimed at improving the

realism of cost estimates, but these are hard to implement when the

acquisition process itself does not reward realism. The same can be said

about streamlining organizational structure to increase the flow of realistic

4

Weapons Acquisition: A Rare Opportunity for Lasting Change (GAO/NSIAD-93-15, Dec. 1992).

5

Acquisition process participants include the military services, the Offices of the Secretary of Defense

and the Joint Chiefs of Staff, independent oversight organizations, contractors, professional

associations, and the Congress.

Page 19 GAO/T-NSIAD-98-123 Defense Acquisition
information, advocating a “fly before buy” approach to testing weapons

before major production commitments are made, and baselining programs

in exchange for stable funding and minimal interference.

DOD has embarked on several initiatives that draw lessons from

commercial practices, such as cost as an independent variable, integrated

product teams, use of past performance data, and performance

specifications. These initiatives are based on best commercial practices

and are not as compliance-oriented as some past reforms. They could have

a positive effect on the outcomes of weapon system programs if the

environment for launching programs and appraising risks can be changed

to provide the right incentives. In our work on supplier relationships, we

found that attempts to apply techniques, such as integrated product teams,

were not seen by some suppliers as being effective because their

environment had not changed. On the JDAM program, where conditions

allowed for a somewhat unique approach, reform initiatives were more

effective. Ultimately, if incentives still exist to launch a program with an

overly optimistic cost, schedule, and performance baseline, then cost as an

independent variable and integrated product teams will not succeed.

Charting a Course of

Action for Better

Outcomes

We believe the acquisition reforms underway have a sound basis and have

the potential for improving the outcomes of weapon systems. We also

believe the current leadership is genuinely committed to making a

difference in the status quo. However, we are convinced that lasting

improvements in the outcomes of acquisition programs will not be realized

unless the incentives that drive behaviors in the acquisition process are

changed. The first steps to making such changes involve decisionmakers:

• Accepting collective responsibility for the incentives that drive the

acquisition process, rather than placing blame on individuals or procedure.

• Agreeing that a cultural focus on acquisition reform—which addresses the

“why” or the incentives that affect behavior—is needed to complement the

traditional focus on the “how” (process and control) and the “who”

(organization).

• Accepting that it is the actions taken and the decisions made on individual

programs that communicate the broader message of “what will work” to

others in the process. Within the current process, circumstances can

usually make a compelling case for taking actions on individual programs

that would otherwise contradict sound principles.

Page 20 GAO/T-NSIAD-98-123 Defense Acquisition
Against this backdrop, it is possible for acquisition reforms and

commercial practices for gaining knowledge and assessing risks to help

produce better outcomes on DOD acquisitions. For such practices to work,

however, the knowledge they produce must help a DOD program succeed

in its environment. The Congress and DOD can help redefine success in

weapons acquisition to make the acquisition environment conducive to

such practices. At least two factors are critical to fostering such an

environment. First, program launch decisions must be depressurized. That

is, they must be relieved of the need to overpromise on performance and

resource estimates. This may require altering how decisions are made on

determining what weapons are needed. Second, once a program is

underway, it must be made acceptable for program managers to identify

unknowns as high risks so that they can be aggressively worked on earlier

in development. Following are specific actions we have reported on that

are in line with addressing the deeper causes of acquisition problems.

Recommendations we have made to the Secretary of Defense include:

• Redefine the point for launching programs as the point at which

technology development ends and product development begins.

• On individual program decisions, send the signals that create incentives

for acquisition managers to identify unknowns and ameliorate their risks

early in development.

• Develop a policy that promotes productive supplier relationships and

emphasizes the importance of suppliers in improving acquisition outcomes

and communicate this policy throughout the acquisition workforce and the

defense industry through training and other means.

Matters we have asked the Congress to consider include:

• Support the Secretary of Defense’s efforts to create the right environment

through changes to the acquisition process that provide program managers

clear incentives for gaining sufficient knowledge at key points in weapon

acquisition programs.

• Provide the funds needed to manage technology development efforts

outside the bounds of individual weapon system programs, if the Secretary

of Defense takes steps to separate technology development from product

development.

• Help create the right incentives on individual programs by favorably

considering DOD funding requests to mitigate high risks early in a program

Page 21 GAO/T-NSIAD-98-123 Defense Acquisition
and cautiously considering late requests for funds to resolve problems that

should have been addressed earlier.

DOD has concurred with these recommendations and says that it will take

action. We are encouraged by this and hope to see these actions reflected

and supported in funding and other decisions made on individual

programs.

Linking Workforce

Reductions With

Better Program

Outcomes

Over the last several years, the Congress has mandated cuts in DOD’s

acquisition workforce and called on DOD to submit plans to streamline and

restructure its acquisition organizations. With DOD’s desire to increase

procurement funding partially through savings from infrastructure

reductions and the need to have better weapon system program outcomes,

such actions are called for.

In defense authorization acts for fiscal years 1996, 1997, and 1998, DOD was

mandated to cut people from its acquisition workforce. We have reported

that DOD most likely will achieve the mandated cuts of 25 percent over 5

years in the acquisition workforce, although the cuts will be offset

somewhat by individuals moving elsewhere in DOD and additional

functions being performed by contractors. These mandates allow the

Secretary of Defense wide latitude to determine how and how much more

to cut.

Legislative mandates for significant streamlining and restructuring actions

have also been placed on DOD. These actions have had much less success.

Section 277 of the National Defense Authorization Act for Fiscal Year 1996

required DOD to develop a 5-year plan to restructure and consolidate

laboratory and test and evaluation centers. DOD developed the Vision 21

plan, but its implementation has been put on hold pending decisions on

another base realignment and closure round. Section 906 of the same act

required the Secretary to submit a plan on how to restructure current DOD

acquisition organizations. DOD’s response did not assess specific

streamlining and restructuring options but rather concluded that its efforts

had been sufficient because the workforce had been reduced. In section

912 of the National Defense Authorization Act for Fiscal Year 1998, DOD

was again required to address streamlining and consolidating acquisition

organizations, this time by submitting an implementation plan to do so.

The next report is due on April 1, 1998.

Page 22 GAO/T-NSIAD-98-123 Defense Acquisition
DOD has several related actions underway to redefine and restructure the

acquisition workforce. In redefining the acquisition workforce, we

encourage DOD to do so in a way that enhances its ability to:

• effectively deliver the training needed to operate successfully in today’s

unfamiliar environment;

• understand the cost of acquisition functions, whether they be performed

by government employees or contractors; and

• accurately identify overlap and duplicative functions between and among

the services in setting requirements for, developing and procuring, and

supporting weapon systems.

As for restructuring the workforce, DOD has taken actions to reshape the

Office of the Secretary of Defense in the Secretary’s Defense Reform

Initiative to focus on corporate level tasks. The Office’s responsibilities

must be balanced with the responsibilities of the individual services for

equipping the forces to ensure that organizations at all levels support

incentives needed to rationalize weapon system requirements and acquire

them cheaper, better, and faster. Such issues have yet to be addressed. As

the congressional directions indicate, the size, organization, and capability

of the workforce DOD relies on to buy equipment is important in

determining the outcomes of acquisition programs. And as DOD’s actions

illustrate, reducing workforce levels is easier than changing the structure

that underlies decision-making in the acquisition process. Whether DOD’s

acquisition workforce and organizations drive the outcomes of the

acquisition process or are a reflection of them, they are connected in a

way that should be considered in contemplating solutions.

Commercial Pricing of

Sole-Source Spare

Parts

Mr. Chairman, I would now like to discuss commercial pricing of

sole-source spare parts. At this Subcommittee’s request, we have begun to

examine how and with what information and guidance, DOD contracting

officers determine whether prices for sole-source commercial items are

fair and reasonable in an environment where there may not be sufficient

competitive market forces to control prices. This universe is a small

portion of the $100-plus billion that DOD spends on contracting each year.

Based on fiscal year 1997 DOD contracting data, about 9 percent of DOD’s

contracting dollars, or $10.8 billion, went for commercial item purchases,

and about 2 percent, or $2.7 billion, represented commercial items

purchased without the benefits of competition. We examined selected

sole-source contracts to compare commercial prices with prices

previously paid by DOD. While our work is not complete, we have been

Page 23 GAO/T-NSIAD-98-123 Defense Acquisition
asked to share our preliminary observations. These observations are based

principally on our discussions with contracting officers and other

contracting personnel, and on our review of contract files at three Defense

Supply Center buying activities, one Air Force buying activity, and one

Navy buying activity.

For an increasing number of sole-source spare parts, DOD is transitioning

from a cost-based pricing environment that relied on certified cost or

pricing data to establish prices to a market-based or commercial pricing

environment where price analysis is the principal means used to

determine the reasonableness of prices. It is premature, at this time, to

draw conclusions about DOD’s ability to respond to this new environment.

Views of Contracting

Personnel

In our discussions, some DOD contracting officers expressed concern

about the breadth of the commercial item definition and are struggling

with how to determine whether commercial prices offered by contractors

for sole-source items are fair and reasonable, particularly those that are

predominately or only sold to DOD. While this concern about the

commercial pricing of sole-source spare parts exists today, some

contracting personnel expressed more concern about future commercial

price negotiations. This is because contracting officers now have recent

cost-based prices to evaluate current prices, and they rely heavily on this

data to establish a negotiating benchmark for determining price

reasonableness. For the future, they expressed concern about their ability

to negotiate fair and reasonable prices without the benefit of recent

cost-based prices.

Another point that came out in our discussions was the sometimes

conflicting pressures on contracting officers to obtain fair and reasonable

prices for sole-source commercial items and to get contracts negotiated so

that purchases can be made to meet the customer’s needs and avoid

backlogs. In fact, one buying unit provided us information showing that

unfilled backorders for aircraft and ship spare parts are increasing.

According to the buying unit’s data, unfilled backorders for spare parts

increased from around 12,000 in October 1996 to around 29,000 in

December 1997. The backlog, according to contracting officials, is caused

in large part by the increasing difficulty of negotiating prices in the new

commercial pricing environment. At this point in our work, we do not

know the impact this backlog is having on aircraft or ship readiness, or

how widespread the problem may be.

Page 24 GAO/T-NSIAD-98-123 Defense Acquisition
Review of Selected

Sole-Source Commercial

Item Contracts

Our review of selected sole-source contracts indicates that some

contractors are now offering commercial prices significantly higher than

DOD paid in the past. While these price differences, discussed below, are

significant, DOD officials and others emphasized that comparing current

commercial prices with historical prices alone may not tell the full story,

and that one must analyze the makeup of the price, the terms and

conditions of the contract, and compare services offered under previous

contracts to those to be provided under the new contract. Clearly, DOD

contracting officers must understand the basis of commercial prices. Our

work raises concerns about how well contracting officers are responding

to higher commercial prices.

For example, the Navy purchased material used to repair naval aircraft

from a contractor for under $100 a yard prior to its being offered as a

commercial item. The manufacturer for this material was acquired by

another company, and the acquiring company raised the price to $800 a

yard, which represented a discount from its $950 catalog price. The Navy

purchased about $1.1 million of this material at the $800 price. According

to the Navy contracting officer, the government is the only buyer of this

material and it requested the contractor to provide cost information

supporting the higher price. The contractor declined to provide cost

information and informed the Navy that the offered price was based on the

company’s financial goals, not on its costs. The contractor referred the

contracting officer to its price catalog. The reasonableness of the 

$800 price could not be tested against commercial sales since there were

none. According to the contracting officer, even though he considered the

price high, the Navy needed the material to solve a readiness problem.

In another example, on three different occasions from 1994 to 1996, the Air

Force purchased from a sole-source contractor an electronic engine

control unit as a spare part. Because this component was bought as a

commercial item, no cost data was requested or provided. In all, the Air

Force bought 54 electronic engine control units for about $19 million.

Each time, the Air Force paid the full commercial catalog price, ranging

from about $330,000 to $370,000 per unit.

The contracting officer performed no additional price analysis comparing

either historical prices or quantities ordered against commercial sales.

Such an analysis would have shown that the company recently had sold

949 of these items, of which 82 percent were sold to commercial

customers at other than catalog prices. In 1989, the Air Force paid $80,000

each for 18 of these same units. These units were not purchased directly

Page 25 GAO/T-NSIAD-98-123 Defense Acquisition
from the electronic engine control manufacturer, but were procured under

a contract with the aircraft manufacturer. Finally, the Air Force, in 1995,

purchased from the engine control manufacturer 90 electronic engine

controls used on another aircraft engine at a $63,200 unit price.

With regard to commercial prices, our ongoing work has identified other

cases where commercially offered or catalog prices are significantly

higher than prices previously paid by DOD. For example, the Air Force

recently received two such proposals, not yet negotiated. In the first case,

in response to an Air Force solicitation for 96 aircraft repair kits, the

contractor submitted a commercial unit price of about $3,144, for a total

price of almost $302,000. The Air Force’s analysis of historical prices

showed that it paid only about $900 a unit for 80 repair kits in 1996.

In the second case, the contractor offered a $1,748 unit price for 

792 aircraft engine fuel control kits, for a total of about $1.4 million.

According to the Air Force’s procurement history records, the unit price

paid for this item in 1993 was only $464 for 150 of these kits. We were told

that after the Air Force questioned the contractor’s proposed price as

being too high, the contractor verbally offered a 40-percent discount from

its catalog, reducing the proposed price to about $1,049 per unit.

In a third case, the Navy received a unit price offer of about $100,000 for

communications test set equipment. According to the Navy’s procurement

history file, the equipment was purchased in early 1995 for around $25,000.

The Navy is currently attempting to negotiate this contact. However, to

date, the contractor has not provided an explanation for the price increase

or any commercial sales data.

The fact that commercially proposed or catalog prices are high relative to

past paid prices presents contracting officers with a negotiating challenge.

It is clear that initial contractor proposals or catalog prices are the

beginning point of the negotiation process and should not be accepted as

being fair and reasonable. Sound price analysis and aggressive

negotiations are fundamental to DOD’s ability to achieve fair and

reasonable commercial prices. For example, in a recently negotiated Air

Force contract, the contractor had initially offered a 30-percent discount

off its commercial catalog prices for about 8,000 engine spare parts.

However, the Air Force determined that a significantly greater discount

was warranted based on its analysis of historical prices. The Air Force’s

price analysis was based on escalated historical prices paid for 120 items

purchased on a sole-source basis from this contractor and on projected

Page 26 GAO/T-NSIAD-98-123 Defense Acquisition
future demands for these items. The historical prices used in its analysis

for many of the items were based on certified cost or pricing data

previously obtained from the contractor. After the Air Force raised

concerns about the offered discount, the contractor proposed a 61-percent

discount off the catalog prices, which the Air Force accepted. Based on

the Air Force’s projected future buy of the 120 sole-source items, the

61-percent discount represents about a $25-million reduction from the

contractor’s initial offer.

We support DOD’s effort to reform its acquisition process and adopt more

efficient commercial practices. Currently, DOD is increasing its effort to

purchase commercial products and adopt commercial pricing methods.

The current contracting environment for commercial items, particularly

for sole-source items where there may be insufficient market forces to

contain prices, presents negotiating challenges for DOD contracting

officers.

While today’s DOD contracting environment is difficult, tomorrow’s may be

even more challenging. At the present time, contracting officers rely

heavily on cost-based historical prices as a baseline for negotiating fair

and reasonable commercial prices. In the future, this data will be less

available. Whether the risks will be moderated as DOD contracting officers

receive additional training in commercial pricing and become more

comfortable with price analysis as the primary tool to determine price

reasonableness and negotiate commercial prices remains to be seen.

Effectiveness of

DOD’s Mentor-Protege

Program

Section 831 of the National Defense Authorization Act for Fiscal Year 1991

established the Pilot Mentor-Protege Program. The purpose of the program

is to provide incentives for major DOD contractors (mentors) to furnish

disadvantaged small business concerns6

 (proteges) with assistance

designed to enhance their capabilities and increase their participation as

suppliers under DOD contracts, other federal government contracts, and

commercial contracts. Over the last few years, we have been asked to

review various aspects of the mentor-protege program. Most recently, the

National Defense Authorization Act for Fiscal Year 1998 requires us to

report, on or before March 31, 1998, on DOD’s implementation of the

program and the extent to which the program is achieving the purposes

established by the Congress.

6

To qualify as a disadvantaged small business, a company must not exceed the Small Business

Administration’s standards for number of employees or annual sales and must be independently

owned (at least 51 percent) and operated by socially and economically disadvantaged individuals.

Page 27 GAO/T-NSIAD-98-123 Defense Acquisition
Our reports have shown a number of implementation problems. For

example, in our 1992 report, we recommended, among other things, that

DOD develop and implement adequate internal controls in the application

and approval process and in the oversight of protege development.7

 In our

1994 report, we were not able to recommend that the pilot program be

extended because sufficient information was not available to determine

whether the program’s purposes could be achieved or whether

reauthorization and extension was warranted.8

In our latest review, we found that continuing data limitations preclude

assessing whether the program is achieving the purposes established by

the Congress. DOD has undertaken actions to review the program that are

intended to provide the basis for such an assessment. Such actions include

conducting a survey of mentors and proteges and requesting the Defense

Contract Management Command to conduct performance evaluations of

each agreement. However, we believe shortcomings in the survey

methodology and incomplete performance evaluations will limit DOD’s

ability to assess the program’s overall effectiveness.

The Congress has appropriated about $233 million for the program since

fiscal year 1992. The funding was generally obligated through either

cooperative agreements where both the government and contractor work

together to obtain a common purpose, separate contracts or line items in

DOD prime contracts. DOD has decided that the services and the defense

agencies should be responsible for managing reimbursable mentor-protege

agreements. In addition, the services have been inconsistent in paying fees

to mentors for providing assistance to proteges and reimbursing proteges

for various expenses.

After spending over $200 million on its pilot mentor-protege program, DOD

lacks information needed to determine the program’s effectiveness.

Accordingly, in our imminent report, we are recommending that DOD

strengthen its performance reviews to ensure that sufficient and reliable

information is gathered on planned and actual mentor assistance as well

as on the protege firms’ business development. In addition, we are

suggesting that the Congress may wish to clarify mentor-protege program

legislation as to whether mentors can be paid fees in addition to expenses

and proteges can be reimbursed for various expenses. Neither the

7

Defense Contracting: Interim Report on Mentor-Protege Program for Small Disadvantaged Firms

(GAO/NSIAD-92-135, Mar. 30, 1992).

8

Defense Contracting: Implementation of the Pilot Mentor-Protege Program (GAO/NSIAD-94-101,

Feb. 1, 1994).

Page 28 GAO/T-NSIAD-98-123 Defense Acquisition
mentor-protege program legislation nor DOD’s implementing regulation

specifically addresses these expenses.

Multiple-Award

Contracts

Concerned that federal agencies were avoiding competitive requirements

when ordering under task- and delivery-order contracts, the Congress

directed agencies to consider awarding multiple contracts—rather than a

single contract—when a task- or delivery-order contract format was

planned.9

 To provide for competition in ordering, agencies are to provide

each of the multiple contractors a fair opportunity to be considered for

orders placed under the contract.

Based on our ongoing work, agencies have not been consistent in

achieving competition for orders.10 One agency issued a high

proportion—64 percent—of orders on a sole-source basis. This

multiple-award contract has a potential value of over a billion dollars. In

another multiple-award contract having the potential to exceed several

billion dollars, agency announcements of planned orders identify

“recommended” firms specifically invited to submit proposals. This

practice has resulted in just one proposal being received on most orders.

We also noted that several agencies will increase the value of orders after

contractors had been provided an opportunity to be considered. While it

may be appropriate to award follow-on or additional work to an

incumbent contractor in many circumstances—such as when the work

involves continuing development of a system the incumbent initiated—we

are concerned that it may not be appropriate in every instance, and that

competition could be undermined.

Most contracts included in our review permit other federal agencies to

place orders under the contracts. The primary agency responsible for

overall contract administration may charge a fee, when other federal

agencies place orders under the contract. Fees should be limited to

recovering the actual costs under the contract. While agencies generally

intend their fees to recover actual costs, weaknesses in accounting and

management systems at some agencies obscure comparisons of fees to the

costs incurred. For example, one agency charged fees that ranged from

$125 to $99,000 for administering a single order under the contract. This

9

A task- and delivery-order contract provides for an indefinite quantity, within stated limits, of supplies

or services to be furnished during a fixed period, with deliveries or performance to be scheduled by

placing orders with the contractor.

10Our work to date has focused on multiple-award contracts administered by the Defense Information

Systems Agency, the Department of Transportation, the General Services Administration, the National

Institutes of Health, and the U.S. Air Force’s Electronic Systems Center and Standard Systems Group.

Page 29 GAO/T-NSIAD-98-123 Defense Acquisition
agency did not maintain accounting records or develop analyses that

would justify this disparity.

Finally, use of the multiple-award contracting mechanism need not impair

the ability of small business to compete for federal contracts. Concerns

had been expressed that—because multiple-award contracts sometimes

consolidated the requirements of numerous programs and offices, and

provided for a large and diverse scope of work—small business could not

realistically compete for such contracts. The Office of Federal

Procurement Policy, the Small Business Administration, and the agencies

where we did our work had all taken steps to ensure that use of

multiple-award contracts did not exclude small business from the federal

marketplace. An adverse effect on small business was not apparent in the

statistical data available to date. In particular, small business received half

or more of the multiple-award contracts awarded at two agencies, and

won orders proportionate with the number of contracts received. The

experience of these two agencies suggests that multiple-award contracts

can be structured to help participation by small business.

Mr. Chairman, this concludes my prepared statement. I would be happy to

respond to any questions you or other Members of the Subcommittee may

have.

(707339) Page 30 GAO/T-NSIAD-98-123 Defense Acquisition
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100