GAO

July 2001 CONTRACT

MANAGEMENT

Benefits of the DOD

Mentor-Protégé

Program Are Not

Conclusive

GAO-01-767

Report to Congressional Requesters

United States General Accounting Office
Page i GAO-01-767 Contract Management

Letter 1

Results in Brief 2

Background 3

DOD Exceeds 5-Percent Goal, but Program’s Contribution Cannot

Be Fully Determined 4

No Explicit Link Between Mentoring and Protégés’

Competitiveness, Financial Independence, or Business

Development 5

Effect of Program Funding on Defense Contractor Participation Is

Unknown 6

Conclusions 6

Matter for Congressional Consideration 7

Recommendations 7

Agency Comments and Our Evaluation 7

Scope and Methodology 8

Appendix I Performance Measurements Reported in the

September 1999 Semi-Annual Reports 10

Appendix II Comments From the Department of Defense 12

Appendix III GAO Contact and Staff Acknowledgments 15

Tables

Table 1: Distribution of Protégés and the Value of Contracts

Received 10

Table 2: Distribution of Protégés and Change in Revenue 11

Table 3: Distribution of Protégés and Change in the Number of

Employees 11

Figure

Figure 1: Percentage of Total Dollar Value of DOD Contract

Awards and Subcontracts Awarded to Small

Disadvantaged Businesses 4

Contents
Page 1 GAO-01-767 Contract Management

July 19, 2001

The Honorable Carl Levin

Chairman

The Honorable John Warner

Ranking Minority Member

Committee on Armed Services

United States Senate

The Honorable Bob Stump

Chairman

The Honorable Ike Skelton

Ranking Minority Member

Committee on Armed Services

House of Representatives

To boost the participation of small disadvantaged businesses as

subcontractors and suppliers under Department of Defense (DOD)

contracts, the Congress authorized1

 the Pilot Mentor-Protégé Program. The

program provides incentives for major defense contractors (mentors) to

assist small disadvantaged businesses (protégés) in strengthening their

ability to compete for work (for example, by providing computer training

or help in the areas of proposal writing and contract administration).

However, DOD has been criticized in the past for not establishing

compelling evidence about the program’s overall effectiveness.

To ensure that the program is focused on a results-oriented approach to

assessing program performance, the Congress asked DOD to obtain

information that could be used to assess the program’s effectiveness. The

Congress also mandated us to study the program’s effectiveness.2

 In

responding to this mandate, we

• reviewed the relationship between the results of the Mentor-Protégé

Program and the statutory goal3

 of awarding 5 percent of the total

 1

 Section 831 of the National Defense Authorization Act for Fiscal Year 1991 (P.L. 101-510,

10 U.S.C. 2302 Note).

2

 Section 811 of the National Defense Authorization Act for Fiscal Year 2000 (P.L. 106-65).

3

 10 U.S.C. 2323.

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-01-767 Contract Management

dollar amount contracted by DOD and subcontracted by DOD prime

contractors to small disadvantaged businesses;

• assessed whether the Mentor-Protégé Program enhanced the “business

competitiveness, financial independence, and business development of

protégé firms”; and

• evaluated whether program funds had been used as an effective

incentive for mentor firms to participate in the program.

In performing our work, we relied primarily on data the DOD

Mentor-Protégé Program Office collected. Program Office officials said

that, in response to our last review,4

 that data they collected would be

reliable and sufficient enough to evaluate program success. A complete

statement of our methodology begins on page 8 of this report.

DOD lacks data integral to assessing the success of the Mentor-Protégé

Program. Specifically, DOD does not have sufficient information to

explicitly

• Determine the relationship between the program and the goal of

awarding 5 percent of the total dollar amount contracted to small

disadvantaged businesses. Although DOD has consistently achieved

this goal since 1992, the overall contribution of the Mentor-Protégé

Program in achieving this goal is unknown.

• Assess whether the Mentor-Protégé Program enhanced the business

competitiveness, financial independence, and business development of

protégé firms. Although there are small disadvantaged business

program participants that have realized increased success, data is not

available to attribute this success to the program.

Program funds have been used to encourage major defense contractors to

provide developmental assistance for small disadvantaged businesses.

However, data is not available to determine whether program funds are

needed to continue to encourage major defense contractors to establish

business relationships with small disadvantaged businesses.

 4 Defense Contracting: Sufficient, Reliable Information on DOD’s Mentor-Protégé

Program Is Unavailable (GAO/NSIAD-98-92, Mar. 30, 1998).

Results in Brief
Page 3 GAO-01-767 Contract Management

This report recommends that DOD better demonstrate the benefits of the

Mentor-Protégé Program. In commenting on a draft of this report, DOD

partially concurred with our recommendations as discussed on page 7.

DOD’s written comments are in appendix II.

In 1986, congressional concern about the low participation of small

disadvantaged businesses within DOD’s procurement system resulted in

section 1207 of the National Defense Authorization Act for Fiscal Year

1987 (P.L. 99-661, later codified as amended at 10 U.S.C. 2323). This

section established a goal of awarding 5 percent of the total dollar amount

contracted by DOD and subcontracted by DOD prime contractors to small

disadvantaged businesses. The 5-percent goal was not met in the years

immediately following its establishment because, according to large DOD

prime contractors, there were not enough qualified small disadvantaged

businesses available as DOD subcontractors. Subsequently, the National

Defense Authorization Act for Fiscal Year 1991 established the Pilot

Mentor-Protégé Program. The purpose of the program is to provide

incentives for major DOD contractors (mentors) to furnish small

disadvantaged businesses (protégés) with assistance designed to enhance

their capabilities and increase their participation as subcontractors and

suppliers under DOD contracts, other federal government contracts, and

commercial contracts.

Under the Mentor-Protégé Program, protégés are to benefit from

developmental assistance provided by mentors in such areas as

(1) infrastructure development, including organizational, financial, and

personnel management; proposal writing; contract administration; and

overall general business development; and (2) technology transfer such as,

training in the areas of production, quality control, manufacturing,

engineering, computer hardware and software, and assistance in obtaining

production and accounting certifications needed to work on large DOD

contracts. Mentors are to benefit from a strengthened cadre of

subcontractors and DOD is to benefit from a resultant robust and

competitive supplier base. The metrics DOD uses to measure program

performance are presented in appendix I.

Since the Pilot Mentor-Protégé Program was authorized, it has been

continuously extended and, with the Fiscal Year 2000 National Defense

Authorization Act, DOD is now authorized to approve new agreements

through September 30, 2002. Program performance is authorized through

September 30, 2005.

Background
Page 4 GAO-01-767 Contract Management

Through fiscal year 2000, the Congress has appropriated $287 million for

the program. Funding covers program operational costs and reimburses

mentors for providing developmental assistance. Mentor-protégé

agreements are now limited to a 3-year period, with the expectation that

the developmental assistance provided will allow protégés to grow their

businesses.

Figure 1 shows that DOD has achieved or exceeded its 5-percent goal of

awarding prime contracts and subcontracts to small disadvantaged

businesses since 1992. However, DOD has not been able to attribute this

success to the program.

Figure 1: Percentage of Total Dollar Value of DOD Contract Awards and

Subcontracts Awarded to Small Disadvantaged Businesses

While data contained in figure 1 indicates success in bringing small

disadvantaged businesses into DOD contract work, the contribution of the

Mentor-Protégé Program to this achievement is not fully known, in part,

because DOD does not fully track the contributions of the 360 former

protégés in meeting the goal. The requirement to track the progress of

former protégés was established by the Fiscal Year 2000 National Defense

Authorization Act, for those agreements entered into on or after October 1,

1999. Because agreements can last 3 years, post-agreement reporting may

not start until October 1, 2002.

DOD Exceeds

5-Percent Goal, but

Program’s

Contribution Cannot

Be Fully Determined
Page 5 GAO-01-767 Contract Management

Data is not available to establish a cause-and-effect relationship between

mentoring provided and protégés’ competitiveness, financial

independence, or business development. DOD officials contend it is

difficult to establish a cause-and-effect relationship between mentoring

provided and protégé success. DOD officials said protégés that have

successfully received mentor-provided assistance might have graduated

from the program before receiving contracts that can be attributed to the

program.

A reliable way to determine program impact is to compare the success of a

group of participants to the success of an equivalent group of

non-participants. Implicit in such a comparison is that differences found

would not have occurred without program participation. Although impact

studies are sometimes difficult and expensive, they are the most reliable

way to determine if a program, such as the Mentor-Protégé Program, is

producing real results and answer the question, “Is the program making a

difference?”

DOD tries to measure program effectiveness by requiring mentors to

report on such performance metrics as protégé revenue and employee

growth. Our analysis of that data shows that protégé revenue increased by

$456 million and the number of protégé employees grew by 2,909

personnel.5

 However, the revenue and employee growth cannot be

explicitly linked to the Mentor-Protégé Program. The revenue and

employee changes include all changes, not just those that may have

resulted from the Mentor-Protégé Program. The same holds true for the

performance metrics captured by DOD’s data collection instrument.

Appendix I contains additional details on the DOD performance metrics.

Other data problems make it further difficult to determine the success of

the program. Specifically,

• Our review of the 1999 and 2000 semi-annual reports6 submitted by

mentors revealed that many of the reports are incomplete. For

example, about 41 percent of the reports have no evidence of protégé

concurrence or nonconcurrence, which is expected to provide

 5

 This growth was measured from the time a protégé entered the program through

September 30, 1999.

6

 Mentor firms are required by DOD to report, semiannually, on the progress made under

active mentor-protégé agreements for the periods ending March 31st and September 30th

.

No Explicit Link

Between Mentoring

and Protégés’

Competitiveness,

Financial

Independence, or

Business

Development
Page 6 GAO-01-767 Contract Management

corroborating support of the data submitted by mentor firms. Further,

as of January 2001, less than half of the September 2000 semi-annual

reports had been received.

• To meet protégé business needs, mentors and protégés are to jointly

fashion business development plans, known as mentor-protégé

agreements. These agreements outline the developmental assistance

mentors are to provide to their protégés. DOD officials said they rely

on the Defense Contract Management Agency to verify that agreements

have been met. We found that over 22 percent of the Defense Contract

Management Agency reviews of the agreements had no indication of

whether the mentor-protégé agreements were being met.

Program funds have been used to encourage major defense contractors to

provide developmental assistance for small disadvantaged businesses.

However, sufficient data is not available to determine whether program

funds are needed to continue to encourage major defense contractors to

establish business relationships with small disadvantaged businesses.

A study completed in December 2000 for DOD by the Logistics

Management Institute (LMI) indicated that 71 percent of the 93 active

mentors, at the time the survey was administered, would withdraw from

the Mentor-Protégé Program if funding for reimbursable agreements were

eliminated. LMI recommended that DOD continue to fund reimbursable

agreements. It also recommended that DOD explore other incentives, such

as assigning additional points during the source selection evaluation

process to those contractors who choose to participate in the

Mentor-Protégé Program, and through the use of award fees and

procurement set-asides to encourage continued mentor assistance to

protégés. In exploring the need for continued funding, LMI asked program

beneficiaries whether additional incentives are needed.

Direct surveys of a program’s beneficiaries are not the best way to analyze

a program because beneficiaries would not be inclined to report that they

did not need government funds after receiving them. Accordingly, it is

unknown how the absence of funding would affect overall provisions of

developmental assistance by mentors or discourage them from

establishing business relationships with small disadvantaged businesses.

There are small disadvantaged businesses that have participated in DOD’s

Mentor-Protégé Program and are receiving contract and subcontract

awards, increasing their revenues by hundreds of millions of dollars. At

Effect of Program

Funding on Defense

Contractor

Participation Is

Unknown

Conclusions
Page 7 GAO-01-767 Contract Management

the same time, DOD is meeting its 5-percent contracting goal with these

businesses. Nevertheless, DOD has not been able to demonstrate the

success of the program—first because it does not have complete data on

program results from mentors and second because it has not performed an

impact study to compare the success of a group of participants to the

success of an equivalent group of non-participants. Implicit in such a

comparison is that differences found would not have occurred without

program participation. Although impact studies are sometimes difficult

and expensive, they can be a reliable way to determine whether a

program, such as the Mentor-Protégé Program, is making a difference.

Before extending the Pilot Mentor-Protégé Program, the Congress may

wish to consider directing DOD to conduct a more conclusive assessment

of the Program’s impact and its contributions.

We recommend that DOD (1) gather more complete information by asking

protégés how the Mentor-Protégé Program has resulted in additional

protégé contracting and subcontracting, and (2) determine if

mentor-protégé agreements have been met. We also recommend that DOD

not accept incomplete mentor-reported data.

In commenting on a draft of this report, DOD contends that the

Mentor-Protégé Program is successful. However, DOD recognizes the

need to strengthen its data collection and performance reviews to enable

more complete and consistent reporting. DOD partially concurred with

our recommendation that it (1) gather more complete information by

asking protégés how the Mentor-Protégé Program has resulted in

additional protégé contracting and subcontracting, and (2) determine if

mentor-protégé agreements have been met. DOD agreed to establish a

team to review the current semi-annual reporting mechanism to determine

and implement improvements to the data collection instrument and the

performance review process. DOD believes that asking protégés how the

program has resulted in additional contracting and subcontracting would

be an imposition to protégés. We disagree and believe that DOD should

ask protégés if and how the program has resulted in contracting and

subcontracting opportunities within the context of the existing statutory

reporting requirement on progress made in employment, revenues, and

participation in DOD contracts. Further, the gathering of such information

is consistent with the advancement of the statutory purpose of the

program, namely, to increase the participation of protégés as

Matter for

Congressional

Consideration

Recommendations

Agency Comments

and Our Evaluation
Page 8 GAO-01-767 Contract Management

subcontractors and suppliers under DOD contracts and other contracts

and subcontracts.

DOD concurred with our recommendation that it not accept incomplete

mentor-reported data. DOD’s written comments are in appendix II.

Our congressional mandate required that we study the relationship

between the results of the Mentor-Protégé Program and the objectives

established in 10 U.S.C. 2323, which established a goal of awarding

5 percent of the total dollar amount contracted by DOD and subcontracted

by DOD prime contractors to small disadvantaged businesses. However,

sufficient information is not available to establish a relationship between

the Mentor-Protégé Program and its effect on the 5-percent goal.

The congressional mandate also required that we determine whether that,

as a result of the program, protégés are more competitive, financially

independent, and what the correlation of program funding is to protégé

business development. In response to our last review,7

 DOD officials wrote

that they were developing a program database that would be reliable and

sufficient enough to evaluate program success. We attempted to answer

our mandate by using this database. We found, however, that the DOD

database does not have sufficient information to answer these mandated

requirements.

To evaluate the effectiveness of the incentives provided to mentor firms to

participate in the program, we interviewed program officials and reviewed

a Mentor-Protégé Program contractor study, completed in December 2000,

which examined (1) whether it would be feasible to operate the MentorProtégé Program without reimbursement and (2) whether alternative

incentives would motivate mentors to participate without reimbursement.

We also were asked to evaluate how program funds had been used as

incentives for mentor firms to participate in the program. Because we

could not correlate funding and protégés’ business development, we did

not further report on the manner in which funds were obligated and on the

average amount spent on individual agreements.

 7 Defense Contracting: Sufficient, Reliable Information on DOD’s Mentor-Protégé

Program Is Unavailable (GAO/NSIAD-98-92, Mar. 30, 1998).

Scope and

Methodology
Page 9 GAO-01-767 Contract Management

During our review, we held discussions with (1) the President of the

National Association of Small Disadvantaged Businesses, which consists

of about 300 member firms; (2) current and former Mentor-Protégé

Program participants, consisting of four mentors and eight protégés;

(3) Mentor-Protégé Program Office officials; and (4) the Mentor-Protégé

Program Office supporting contractor responsible for maintaining the

Mentor-Protégé Program database. In addition, we reviewed MentorProtégé Program Office documentation and minutes of a Mentor-Protégé

Program Office sponsored mentor meeting. We also held discussions with

officials from the Small Business Administration.

We performed this phase of work between December 2000 and April 2001

in accordance with generally accepted government auditing standards.

We will send copies to the Chairmen and the Ranking Minority Members

of other appropriate congressional committees; the Secretary of Defense;

and the Director, Office of Management and Budget; and other interested

parties. We will also make copies available to others on request.

Please contact me at (202) 512-4841 if you or your staff have any questions

concerning this report. Additional contact and key contributors to this

report are acknowledged in appendix III.

David E. Cooper

Director

Acquisition and Sourcing Management
Appendix I: Performance Measurements

Reported in the September 1999 Semi-Annual

Reports

Page 10 GAO-01-767 Contract Management

The Department of Defense (DOD) requires mentor firms to report

semiannually on the progress made under active mentor-protégé

agreements for the periods ending March 31st and September 30th. The

September semi-annual report is to summarize progress made for the

entire fiscal year from the start of each mentor-protégé agreement. The

metrics listed in the tables below were developed from the data in the

September 1999 semi-annual reports1

 because, as of January 2001, less

than half of the September 2000 semi-annual reports had been received

and, therefore, were not available for our review.

The semi-annual report is not designed to establish a cause-and-effect

relationship between mentoring provided and reported success. A protégé

may realize additional revenue because of mentor-related business

opportunities. However, the semi-annual report does not isolate mentorrelated revenue growth from growth that is unrelated to the program. The

same holds true for the other metrics that the semi-annual report captures

and that are presented below, such as employment growth and contracts

awarded.

Table 1: Distribution of Protégés and the Value of Contracts Received

Dollars in millions

Distribution of protégés

Value of awarded

contractsa

 3 $519.5

10 242.6

16 155.1

13 52.7

54 53.2

76 0

a

Contracts cannot be explicitly linked to the Mentor-Protégé Program. The amounts include all

protégé contracts from DOD, not just those that may have resulted from the Mentor-Protégé Program.

 1

 We reviewed 186 semi-annual reports dated September 30, 1999. However, 14 of these

reports were incomplete and were not used in this appendix.

Appendix I: Performance Measurements

Reported in the September 1999 Semi-Annual

Reports
Appendix I: Performance Measurements

Reported in the September 1999 Semi-Annual

Reports

Page 11 GAO-01-767 Contract Management

Table 2: Distribution of Protégés and Change in Revenue

Dollars in millions

Distribution of protégésa

Change in

revenueb

12 $240.2

17 113.4

25 71.1

19 24.4

54 25.1

 6 0

22 (28.3)

a

Semi-annual reports submitted on 17 protégés either did not list the protégés’ revenue at the start of

the agreement or the protégés’ revenue at the time of the semi-annual report. Therefore, these

reports were not included in this analysis.

b

These revenue changes cannot be explicitly linked to the Mentor-Protégé Program. The revenue

increase includes all protégés’ revenue increases, not just those that may have resulted from the

Mentor-Protégé Program.

Table 3: Distribution of Protégés and Change in the Number of Employees

Distribution of protégésaChange in number of employeesb

12 1,750

16 868

24 501

15 173

44 190

17 0

35 (573)

a

Semi-annual reports submitted on nine protégés either did not list the number of employees at the

start of the agreement or the number of employees at the time of the semi-annual report. Therefore,

these reports were not included in this analysis.

b

These employee changes cannot be explicitly linked to the Mentor-Protégé Program. The employee

changes include all protégés’ employee changes, not just those that may have resulted from the

Mentor-Protégé Program.
Appendix II: Comments From the Department of Defense

Page 12 GAO-01-767 Contract Management

Appendix II: Comments From the

Department of Defense
Appendix II: Comments From the Department of Defense

Page 13 GAO-01-767 Contract Management
Appendix II: Comments From the Department of Defense

Page 14 GAO-01-767 Contract Management
Appendix III: GAO Contact and

Staff Acknowledgments

Page 15 GAO-01-767 Contract Management

Ralph Dawn (202) 512-4544

In addition to the name above, Catherine Baltzell, Daniel Hauser, Russ

Reiter, and Adam Vodraska made key contributions to this report.

Appendix III: GAO Contact and Staff

Acknowledgments

GAO Contact

Acknowledgments

(120029)

The first copy of each GAO report is free. Additional copies of reports are

$2 each. A check or money order should be made out to the

Superintendent of Documents. VISA and MasterCard credit cards are also

accepted.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St., NW (corner of 4th and G Sts. NW)

Washington, DC 20013

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. To

receive facsimile copies of the daily list or any list from the past 30 days,

please call (202) 512-6000 using a touchtone phone. A recorded menu will

provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an email message with “info” in the body to:

Info@www.gao.gov

or visit GAO’s World Wide Web home page at:

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• E-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

Ordering Information

To Report Fraud,

Waste, and Abuse in

Federal Programs