United States General Accounting Office

GAO Report to Congressional Requesters

March 1998 DEFENSE

CONTRACTING

Sufficient, Reliable

Information on DOD’s

Mentor-Protege

Program Is Unavailable

GAO/NSIAD-98-92

GAO United States

General Accounting Office

Washington, D.C. 20548

National Security and

International Affairs Division

B-278784 

March 30, 1998

The Honorable Strom Thurmond

Chairman

The Honorable Carl Levin

Ranking Minority Member

Committee on Armed Services

United States Senate

The Honorable Floyd Spence

Chairman

The Honorable Ike Skelton

Ranking Minority Member

Committee on National Security

House of Representatives

In 1990, Congress passed section 831 of the National Defense

Authorization Act for Fiscal Year 1991 establishing the Pilot

Mentor-Protege Program. The purpose of the program is to provide

incentives for major Department of Defense (DOD) contractors (mentors)

to furnish disadvantaged small business concerns1

 (proteges) with

assistance designed to enhance their capabilities and increase their

participation as subcontractors and suppliers under DOD contracts, other

federal government contracts, and commercial contracts.

The fiscal year 1998 National Defense Authorization Act requires us to

report on DOD’s implementation of the program and the extent to which

the program is achieving the purposes established by Congress. We

reviewed (1) DOD’s management of the program, including its efforts to

evaluate the program’s effectiveness and (2) the manner in which program

funds have been obligated. We also attempted to assess the extent to

which the pilot program benefited the proteges.

Background DOD’s Office of Small and Disadvantaged Business Utilization (OSADBU) is

responsible for the mentor-protege program. A signed mentor-protege

agreement for each mentor-protege relationship must be submitted to

OSADBU and approved before developmental assistance costs may be

incurred. OSADBU has reimbursed mentors through cooperative agreements

where both the government and a contractor work together to achieve a

1

To qualify as a disadvantaged small business, a company must not exceed the Small Business

Administration’s standards for number of employees or annual sales and must be independently

owned (at least 51 percent) and operated by socially and economically disadvantaged individuals.

Page 1 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

common purpose. OSADBU also approved credit agreements that were

negotiated on a nonreimbursable basis.2

 In addition, the services and the

defense agencies requested OSADBU’s approval of mentor-protege

agreements that reimbursed mentors pursuant to separate contracts or

line items in DOD contracts. OSADBU is to conduct periodic performance

reviews of the progress and accomplishments realized under approved

mentor-protege agreements.

This report is one of a series we have issued on the mentor-protege

program. In our first report, we recommended, among other things, that

DOD develop and implement adequate internal controls in the application

and approval process and in the oversight of protege development.3

 In our

second report, we were not able to recommend that the pilot program be

extended because sufficient information was not available to determine

whether the program’s purposes could be achieved or whether

reauthorization and extension were warranted.4

Results in Brief Data limitations preclude our assessing the extent to which the program is

achieving the purposes established by Congress. OSADBU has recently

undertaken actions to review the program that were intended to provide

the basis for such an assessment. It has initiated a survey of mentors and

proteges and enlisted the Defense Contract Management Command to

conduct performance evaluations of each agreement. However, we believe

shortcomings in the survey methodology and incomplete coverage in the

performance evaluations will limit the usefulness of their results in

assessing the program’s overall effectiveness.

Congress has appropriated about $233 million for the program since fiscal

year 1992. The funding was generally obligated through either cooperative

agreements, separate contracts, or line items in DOD prime contracts. While

OSADBU has managed cooperative agreements in the past, it has decided

that the services and the defense agencies should be responsible for

managing reimbursable mentor-protege agreements. In addition, the

services have been inconsistent in their policies on paying fees to mentors

2

By using a credit agreement, a mentor is reimbursed for developmental assistance costs that are

allowable, allocable, and reasonable through their inclusion in indirect cost pools and is also given

credit toward meeting small business contracting goals as if these assistance costs were contracts

awarded to small and disadvantaged businesses.

3

Defense Contracting: Interim Report on Mentor-Protege Program for Small Disadvantaged Firms

(GAO/NSIAD-92-135, Mar. 30, 1992).

4

Defense Contracting: Implementation of the Pilot Mentor-Protege Program (GAO/NSIAD-94-101,

Feb. 1, 1994).

Page 2 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

for providing assistance to proteges and reimbursing proteges for various

expenses.

Because sufficient and reliable information on program performance was

unavailable, the overall extent to which the pilot program benefited the

proteges could not be assessed.

Management Actions

Are Underway

In our prior two reports, we identified the lack of management data,

oversight of individual agreements, and evaluation of results of program

participation as major impediments to measuring progress toward

accomplishing congressional goals. Within the last 16 months, OSADBU has

initiated actions to evaluate program performance and improve

administration of the program.5

 Actions to evaluate program performance

include conducting a survey of program participants and establishing a

performance review process. However, low survey response rates and

incomplete performance review data weaken OSADBU’s efforts to evaluate

and report on the overall success of the program.

Our Prior Reviews of the

Program

Our 1992 report pointed out that DOD needed to develop and implement

adequate internal controls in the application and approval process and in

the oversight of protege development. In its response, OSADBU officials

agreed to verify the progress reports, ensure the agreed upon assistance

was provided, and evaluate (through communication with the proteges)

whether the assistance had increased the number of subcontracts

awarded.

In our 1994 report, we again pointed out that OSADBU had neither

(1) assessed progress and accomplishments realized under any of the

agreements nor (2) evaluated program success. In its comments to our

report, OSADBU acknowledged that monitoring and oversight of

mentor-protege agreements had not occurred and added that it had plans

to do so once resources were available and enough time had elapsed for

significant activity under the agreements to have occurred.

5

OSADBU officials told us that actions have been taken to improve program administration, including

tracking financial information, abandoning centrally managed cooperative agreements, moving toward

service-funded agreements, implementing a phased approach to reduce cost of agreements, increasing

the use of incrementally funding of agreements, and improving outreach briefings to provide clear

guidance and lessons learned to participants.

Page 3 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

OSADBU Has Initiated

Action

In November 1996, OSADBU, with contractor support, began a data

collection effort by conducting a survey of program participants to

evaluate program effectiveness. Also, OSADBU requested that the Defense

Contract Management Command (DCMC) conduct performance reviews of

all mentor-protege agreements approved since the program’s inception,

and thereafter, to review all active agreements annually. The purpose of

these reviews is to establish and maintain a regular performance review

process.

Survey Provides Limited Data

on Program’s Effectiveness

During our review, OSADBU was conducting a questionnaire survey of the

program. OSADBU officials told us that the questionnaire responses will, in

part, provide the data needed to ascertain and report overall progress

toward achieving the program’s statutory goals. OSADBU plans to issue its

report by late spring 1998.

Separate questionnaires for mentors and proteges were sent to current

participants as well as those who had completed or terminated agreements

as of November 4, 1996. The questionnaires requested general background

data on the participants and the agreements, information on types of

assistance provided, data on the dollar value and number of contracts as

well as subcontracts received and/or awarded during the agreements, and

experiences of the participants and their views of the program.

A successful survey relies on a high response rate, an analysis of why

nonrespondents in the population did not respond, and an assessment of

the validity and reliability of the data. As of January 1998, OSADBU’s

response rates of about 60 percent for the mentor survey and about

40 percent for the protege survey were well below that which would be

required to draw statistically meaningful conclusions about the population

of program participants or the mentor-protege program as a whole.6

Further, OSADBU has not conducted a systematic analysis of why

nonrespondents did not participate in the survey to assess any patterns for

their lack of participation as compared to the respondents.

Also, the financial data on contracting and subcontracting were not

systematically verified with an independent source such as progress

reports, contractor reports, or other records. Attempts were made to

reconcile conflicting information between the mentors and proteges, but

only if the discrepancies were large or if totally inconsistent information

was provided by each respondent.

6

To make plausible generalizations about the overall pilot program, the effective response rate should

usually be at least 75 percent for each variable measure—a goal used by most practitioners. See

Developing and Using Questionnaires (GAO/PEMD-10.1.7, Oct. 1993).

Page 4 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

Performance Evaluations Have

Shortcomings

On November 6, 1996, OSADBU, in an effort to conduct annual performance

reviews of all mentor-protege agreements, requested DCMC to review all

305 agreements that had been approved since the program’s inception. In

response, DCMC sent a memorandum dated January 2, 1997, to its districts

setting out its policy on conducting the performance reviews. The reviews

were to include evaluations of costs, planned and actual mentor

assistance, and data on protege firms’ business development (contract

awards and improvements in technical capabilities) resulting from

program participation. Although DCMC requested that all the reports be

sent to OSADBU by December 31, 1997, OSADBU had received only 138

reports on the 305 agreements, or about 45 percent, as of that date.

We reviewed 66 of the 138 DCMC reports and found that many of the reports

were missing information or did not address all the required areas outlined

in the DCMC policy memorandum. The differences in reporting all

information, as well as incomplete data, will limit the usefulness of the

performance reviews in analyzing trends, tracking performance of

program agreements, and assessing the program’s effectiveness. OSADBU’s

support contractor is reviewing the reports to identify those that are

missing and the general quality and consistency of the reviews to

determine what additional DCMC guidance is necessary for its districts.

Manner of Obligating

Funds Has Changed

The funding appropriated from fiscal year 1992 to 1998 was generally

obligated through either cooperative agreements, separate contracts, or in

a line item in DOD prime contracts. While OSADBU has managed cooperative

agreements in the past, it has decided that the services and the defense

agencies should be responsible for managing reimbursable mentor-protege

agreements. In addition, the services were inconsistent in paying fees to

mentors and reimbursing proteges for various expenses.

Changes in the Ways Funds

Are Obligated

Mentor-protege funding of about $233 million7 was generally obligated

through cooperative agreements, separate contracts, or line items in DOD

contracts. OSADBU spent about $88.3 million on 50 cooperative agreements

that resulted in 99 mentor-protege agreements ranging in value from

$135,000 to about $3 million, or an average of about $892,000 per

mentor-protege agreement. In addition, the services and the defense

7

In addition to the $233 million, an undetermined amount was believed to be reimbursed indirectly

through 142 credit agreements by including the costs of mentor assistance in indirect costs pools.

Page 5 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

agencies have reimbursed costs amounting to about $108.5 million,8

pursuant to 143 separate contracts or line items in DOD contracts that have

ranged in value from $38,525 to $6.9 million, or an average of about

$759,000 per mentor-protege agreement. Funds amounting to about

$7.2 million were also disbursed by OSADBU for historically Black colleges

and universities and minority institutions, contractor support, and other

miscellaneous expenses. As of February 1998, the remaining funds of

$28.9 million included $10.6 million that had been sent to the services but

not yet obligated, $16.6 million that had not been distributed,9

 and

$1.7 million that had expired.

According to the OSADBU program manager, centrally managed cooperative

agreements were useful for initially getting the program started because

they successfully produced interest in the program. OSADBU used a

“program announcement” whereby potential mentors submitted proposals

for funding a mentor-protege agreement through a cooperative agreement.

OSADBU has decided to discontinue its use of centrally managed

cooperative agreements. The OSADBU program manager informed us that

the level and type of technical assistance provided to the proteges were

primarily business in nature, and not related to DOD-specific program

areas. The program manager further pointed out that OSADBU does not have

the staff to centrally manage, review, monitor, and serve as the responsible

official on multiple cooperative agreements.

OSADBU has decided that the services and the defense agencies should be

responsible for managing reimbursable mentor-protege agreements. The

OSADBU program manager informed us that by linking the agreements to a

service or a defense agency, the likelihood of a protege receiving technical

assistance that is mission related increases significantly. Further, by

incorporating the program into the acquisition process, it becomes an

integral part of that process, thus mitigating the need for additional

resources to administer the program and monitor program performance.10

8

In addition to the $108.5 million, Air Force funding of $11.6 million was used to support four

agreements developing technologies for the F-22 aircraft, while Navy funding of $900,000 was used to

support three agreements for the F/A-18 aircraft.

9

Included in this amount was $10 million available for fiscal year 1998, but withheld by the DOD

Comptroller.

10Since the mentor-protege agreements are incorporated into contracts, the service OSADBU program

managers generally view the administration of the agreements to be a function of the contracting

officers within the program offices.

Page 6 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

Services Are Inconsistent

in Reimbursing Various

Expenses

Neither the mentor-protege program legislation nor the implementing

regulation specifically addresses either the payment of fees to mentors or

the reimbursement of proteges for their costs in participating in the

program.

In our March 1992 report, issued shortly after the program actually began,

we noted low mentor participation and recommended that DOD consider

ways to enhance program incentives for mentor participation. At that time,

we noted that the appeal of the cash reimbursement incentive was

diminished because mentors were prohibited by regulation from earning a

fee on the development assistance provided to proteges. In 1992, DOD

eliminated the regulatory prohibition.

In our current review, we found that the payment of fees to mentors varied

among the services. We found no instance where the Navy has paid a fee,

while the Army and the Air Force have paid fees to mentors. According to

the OSADBU program manager, mentors have received fees in addition to

their expenses because mentor-protege agreements are reimbursed by

means of a line item in DOD contracts that allow a fee and the program’s

legislation does not prohibit the payment of fees to the mentor.

In May 1997, the Air Force asked OSADBU about reimbursing protege

salaries while the proteges were undergoing training. The Air Force noted

that it had found no prohibition on such payments in either the program

legislation or the regulation. OSADBU approved the payment of such

incidental expenses in July 1997, so long as they were otherwise,

allowable, allocable, and reasonable and did not exceed 10 percent of the

total reimbursable cost of the approved mentor-protege agreement. We

found that the Army and the Air Force permit these payments; the Navy

does not.

Extent Proteges

Benefited From

Funding Has Not Been

Determined

While the mentor-protege program has provided incentives to mentors to

provide assistance to proteges, the extent that the funding has enhanced

the business competitiveness and financial independence of protege firms

has not been determined because sufficient and reliable information on

program performance has not been available. OSADBU is developing a

program database that it believes will be the source for periodic analysis

of program results.

About 180 mentors received reimbursement of certain costs incurred in

providing assistance to their protege firms, credit in meeting small

Page 7 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

business goals,11 or both. Of the 180 mentors, 92 used only reimbursable

agreements, 69 used only credit agreements, and 19 used both

reimbursement and credit agreements. We found that 86 mentors have

active agreements, while 94 mentors have completed or terminated their

agreements.

Of the 340 proteges that received assistance from mentors, 201 received

assistance through reimbursable agreements, 113 received assistance

through credit agreements, and 26 used both types of agreements. We

found that 147 proteges have active agreements, while 193 proteges have

completed or terminated their agreements.

OSADBU Is Developing a

Program Database

OSADBU is developing a program database on all participants, including

responses to the questionnaire surveys. The database is expected to

incorporate information gathered from the mentor application, the

agreements, and the data gathered as a result of the DCMC reviews. In

addition, a reference resource is expected to provide more easily

understood program guidelines and streamlined procedures for program

participation and guidance on costs eligible for reimbursement, lessons

learned, and best practices.

Conclusion After spending over $200 million on the pilot mentor-protege program,

OSADBU does not know the overall extent to which the program is achieving

the purposes Congress has established because sufficient and reliable

information on program performance is unavailable. While OSADBU has

initiated an assessment of the program by conducting a survey and

establishing a performance review process, shortcomings in the

assessment will limit its usefulness. For example, a low survey response

rate will limit the usefulness of its findings in assessing the program’s

overall effectiveness. In addition, performance reviews were missing

information or did not address all the required areas, such as

improvements in protege capabilities and contract awards. While it would

be difficult to correct the shortcomings in the survey to assess the overall

effectiveness of the program, the weaknesses in the performance reviews

can be overcome.

Neither the mentor-protege program legislation nor the implementing

regulation specifically addresses either the payment of fees to mentors or

11In acquisitions above $500,000 ($1 million for construction), prime contractors are required to

establish a subcontracting plan that specifies the percentage goal of subcontract awards to small and

disadvantaged businesses.

Page 8 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

the reimbursement of proteges for their costs in participating in the

program. We found that there is no uniformity among the services on

making these payments.

Recommendation We recommend that the Secretary of Defense instruct the Director,

OSADBU, to strengthen OSADBU’s performance reviews to ensure that

sufficient and reliable information on program performance is gathered on

planned and actual mentor assistance as well as on the protege firms’

business development.

Matter for

Congressional

Consideration

Congress may wish to clarify mentor-protege program legislation as to

whether mentors can be paid fees in addition to expenses and proteges

can be reimbursed for various expenses.

Agency Comments DOD concurs with the recommendation to strengthen performance reviews

and will continue to make efforts to improve the reliability and sufficiency

of data on the program. However, DOD states that there have been a

significant number of proteges that have benefited from the program and

provides nine examples of mentor-protege agreements that it believes

illustrate the effectiveness of the program. We found that there has been a

lack of sufficient and reliable information on close to 400 agreements

approved by OSADBU to assess the program’s overall effectiveness and do

not believe that nine examples, while positive in themselves, are a

sufficient basis for this assessment. (See app. I for DOD’s written

comments.)

DOD acknowledges that the services have been inconsistent in their

policies on paying fees to mentors for providing assistance to proteges and

reimbursing proteges for various expenses. By allowing the diversity of

approaches, DOD states that it is testing the range of options for program

execution and the impact of various incentives upon participation in the

program. Given that the mentor-protege program legislation and the

implementing regulation do not specifically address either paying fees to

mentors or reimbursing proteges for their costs in participating in the

program, we continue to believe that Congress may wish to clarify existing

legislation as to the use of mentor-protege funding.

Page 9 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

Scope and

Methodology

The congressional mandate states, in part, that we will study the extent to

which the program is achieving the purposes established by Congress.

Because DOD maintained performance review and evaluation data12 on the

program that was insufficient and unreliable, we considered making an

independent data collection survey. We did not make such a survey

because OSADBU and a trade association were completing separate surveys

of the same population of mentors and proteges. We also did not make the

survey because repeated surveys of the same population at closely spaced

intervals can lead to “history effects,” which can effect both the response

rate and the quality of the data collected in unpredictable ways. We

focused our review on the management of the program.

To review program management, we reviewed our past reports on the

program that identified weaknesses in the management, obtained DOD’s

response to our findings, and identified OSADBU’s actions initiated to

improve program administration and evaluation. We also contacted

OSADBU’s contractor who is responsible for administering and summarizing

the DOD survey questionnaire and DCMC to identify its progress in

conducting performance reviews of the mentor-protege agreements.

To review the manner in which the funds were obligated, we met with

OSADBU and Washington Headquarters Service personnel who transfer

mentor-protege funding from OSADBU to the responsible services and the

defense agencies. As such, we obtained overall funding information

dealing with the program from fiscal year 1992 to 1998 that showed a

breakdown of the funding by responsible organization. To verify the

funding information, we contacted officials within the services’ and

defense agencies’ offices of Small and Disadvantaged Business Utilization

responsible for administering the mentor-protege agreements. When

necessary, these officials contacted supporting commands to gather more

reliable information regarding the obligation of the funding.

We attempted to assess the extent to which the funds benefited the

proteges by reviewing OSADBU’s files and contacting service and defense

agency officials to identify the number of mentors and proteges that have

participated in the program. Sufficient and reliable information on

program performance, however, was unavailable on the overall extent that

the proteges benefited from the funding provided by the program.

12According to OSADBU officials, the data were useful in making program administrative decisions

such as abandoning centrally managed cooperative agreements and moving toward service-funded

agreements.

Page 10 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
B-278784 

We performed this phase of work between August 1997 and February 1998

in accordance with generally accepted government auditing standards.

We will send copies to the Chairmen and the Ranking Minority Members

of other appropriate congressional committees; the Secretary of Defense;

and the Director, Office of Management and Budget. We will also make

copies available to others upon request.

If you have any questions regarding this report, please call me on

(202) 512-4841. Major contributors to this report are listed in appendix II.

Louis J. Rodrigues

Director, Defense Acquisitions Issues

Page 11 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
Contents

Letter 1

Appendix I 

Comments From the

Department of

Defense

14

Appendix II 

Major Contributors to

This Report

17

Abbreviations

DCMC Defense Contract Management Command

DOD Department of Defense

OSADBU Office of Small and Disadvantaged Business Utilization

Page 12 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
Page 13 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
Appendix I 

Comments From the Department of Defense

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

Page 14 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
Appendix I 

Comments From the Department of Defense

See comment 1.

See comment 2.

See comment 3.

See comment 1.

Page 15 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
Appendix I 

Comments From the Department of Defense

The following are GAO’s comments on the Department of Defense’s (DOD)

letter dated February 26, 1998.

GAO Comments 1. DOD states that a case study approach can sufficiently portray the impact

of the program upon the protege firms. In this regard, DOD provided nine

examples of mentor-protege agreements that it believed illustrated the

effectiveness of the program. However, there have been close to 400

agreements approved by OSADBU since the inception of the program. We do

not believe the nine examples, while positive in themselves, are sufficient

to assess the program’s overall effectiveness. Therefore, we have chosen

not to include the enclosure in this report.

2. We have modified the report to properly attribute to OSADBU officials the

actions that they believe have improved program administration.

3. We have incorporated DOD’s technical comments in the text where

appropriate.

Page 16 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
Appendix II 

Major Contributors to This Report

National Security and

International Affairs

Division, Washington,

D.C.

Charles D. Groves

Stanley J. Kostyla

Russell R. Reiter

Patricia D. Slocum

Charles W. Thompson

(707292) Page 17 GAO/NSIAD-98-92 DOD’s Mentor-Protege Program
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100