Defense Contracting: Sufficient, Reliable Information on DOD's Mentor-Protege Program Is Unavailable
Released: 1998-03-30T00:00:00

Letter

Background

DOD’s Office of Small and Disadvantaged Business Utilization (OSADBU) is responsible for the mentor-protege program. A signed mentor-protege agreement for each mentor-protege relationship must be submitted to OSADBU and approved before developmental assistance costs may be incurred. OSADBU has reimbursed mentors through cooperative agreements where both the government and a contractor work together to achieve a common purpose. OSADBU also approved credit agreements that were negotiated on a nonreimbursable basis. In addition, the services and the defense agencies requested OSADBU’s approval of mentor-protege agreements that reimbursed mentors pursuant to separate contracts or line items in DOD contracts. OSADBU is to conduct periodic performance reviews of the progress and accomplishments realized under approved mentor-protege agreements.

This report is one of a series we have issued on the mentor-protege program. In our first report, we recommended, among other things, that DOD develop and implement adequate internal controls in the application and approval process and in the oversight of protege development. In our second report, we were not able to recommend that the pilot program be extended because sufficient information was not available to determine whether the program’s purposes could be achieved or whether reauthorization and extension were warranted.

Management Actions Are Underway

In our prior two reports, we identified the lack of management data, oversight of individual agreements, and evaluation of results of program participation as major impediments to measuring progress toward accomplishing congressional goals. Within the last 16 months, OSADBU has initiated actions to evaluate program performance and improve administration of the program. Actions to evaluate program performance include conducting a survey of program participants and establishing a performance review process. However, low survey response rates and incomplete performance review data weaken OSADBU’s efforts to evaluate and report on the overall success of the program.

Our Prior Reviews of the Program

Our 1992 report pointed out that DOD needed to develop and implement adequate internal controls in the application and approval process and in the oversight of protege development. In its response, OSADBU officials agreed to verify the progress reports, ensure the agreed upon assistance was provided, and evaluate (through communication with the proteges) whether the assistance had increased the number of subcontracts awarded.

In our 1994 report, we again pointed out that OSADBU had neither (1) assessed progress and accomplishments realized under any of the agreements nor (2) evaluated program success. In its comments to our report, OSADBU acknowledged that monitoring and oversight of mentor-protege agreements had not occurred and added that it had plans to do so once resources were available and enough time had elapsed for significant activity under the agreements to have occurred.

OSADBU Has Initiated Action

In November 1996, OSADBU, with contractor support, began a data collection effort by conducting a survey of program participants to evaluate program effectiveness. Also, OSADBU requested that the Defense Contract Management Command (DCMC) conduct performance reviews of all mentor-protege agreements approved since the program’s inception, and thereafter, to review all active agreements annually. The purpose of these reviews is to establish and maintain a regular performance review process.

Survey Provides Limited Data on Program’s Effectiveness

During our review, OSADBU was conducting a questionnaire survey of the program. OSADBU officials told us that the questionnaire responses will, in part, provide the data needed to ascertain and report overall progress toward achieving the program’s statutory goals. OSADBU plans to issue its report by late spring 1998.

Separate questionnaires for mentors and proteges were sent to current participants as well as those who had completed or terminated agreements as of November 4, 1996. The questionnaires requested general background data on the participants and the agreements, information on types of assistance provided, data on the dollar value and number of contracts as well as subcontracts received and/or awarded during the agreements, and experiences of the participants and their views of the program.

A successful survey relies on a high response rate, an analysis of why nonrespondents in the population did not respond, and an assessment of the validity and reliability of the data. As of January 1998, OSADBU’s response rates of about 60 percent for the mentor survey and about 40 percent for the protege survey were well below that which would be required to draw statistically meaningful conclusions about the population of program participants or the mentor-protege program as a whole.Further, OSADBU has not conducted a systematic analysis of why nonrespondents did not participate in the survey to assess any patterns for their lack of participation as compared to the respondents.

Also, the financial data on contracting and subcontracting were not systematically verified with an independent source such as progress reports, contractor reports, or other records. Attempts were made to reconcile conflicting information between the mentors and proteges, but only if the discrepancies were large or if totally inconsistent information was provided by each respondent.

Performance Evaluations Have Shortcomings

On November 6, 1996, OSADBU, in an effort to conduct annual performance reviews of all mentor-protege agreements, requested DCMC to review all 305 agreements that had been approved since the program’s inception. In response, DCMC sent a memorandum dated January 2, 1997, to its districts setting out its policy on conducting the performance reviews. The reviews were to include evaluations of costs, planned and actual mentor assistance, and data on protege firms’ business development (contract awards and improvements in technical capabilities) resulting from program participation. Although DCMC requested that all the reports be sent to OSADBU by December 31, 1997, OSADBU had received only 138 reports on the 305 agreements, or about 45 percent, as of that date.

We reviewed 66 of the 138 DCMC reports and found that many of the reports were missing information or did not address all the required areas outlined in the DCMC policy memorandum. The differences in reporting all information, as well as incomplete data, will limit the usefulness of the performance reviews in analyzing trends, tracking performance of program agreements, and assessing the program’s effectiveness. OSADBU’s support contractor is reviewing the reports to identify those that are missing and the general quality and consistency of the reviews to determine what additional DCMC guidance is necessary for its districts.

Manner of Obligating Funds Has Changed

The funding appropriated from fiscal year 1992 to 1998 was generally obligated through either cooperative agreements, separate contracts, or in a line item in DOD prime contracts. While OSADBU has managed cooperative agreements in the past, it has decided that the services and the defense agencies should be responsible for managing reimbursable mentor-protege agreements. In addition, the services were inconsistent in paying fees to mentors and reimbursing proteges for various expenses.

Changes in the Ways Funds Are Obligated

Mentor-protege funding of about $233 million was generally obligated through cooperative agreements, separate contracts, or line items in DOD contracts. OSADBU spent about $88.3 million on 50 cooperative agreements that resulted in 99 mentor-protege agreements ranging in value from $135,000 to about $3 million, or an average of about $892,000 per mentor-protege agreement. In addition, the services and the defense agencies have reimbursed costs amounting to about $108.5 million,pursuant to 143 separate contracts or line items in DOD contracts that have ranged in value from $38,525 to $6.9 million, or an average of about $759,000 per mentor-protege agreement. Funds amounting to about $7.2 million were also disbursed by OSADBU for historically Black colleges and universities and minority institutions, contractor support, and other miscellaneous expenses. As of February 1998, the remaining funds of $28.9 million included $10.6 million that had been sent to the services but not yet obligated, $16.6 million that had not been distributed, and $1.7 million that had expired.

According to the OSADBU program manager, centrally managed cooperative agreements were useful for initially getting the program started because they successfully produced interest in the program. OSADBU used a “program announcement” whereby potential mentors submitted proposals for funding a mentor-protege agreement through a cooperative agreement.

OSADBU has decided to discontinue its use of centrally managed cooperative agreements. The OSADBU program manager informed us that the level and type of technical assistance provided to the proteges were primarily business in nature, and not related to DOD-specific program areas. The program manager further pointed out that OSADBU does not have the staff to centrally manage, review, monitor, and serve as the responsible official on multiple cooperative agreements.

OSADBU has decided that the services and the defense agencies should be responsible for managing reimbursable mentor-protege agreements. The OSADBU program manager informed us that by linking the agreements to a service or a defense agency, the likelihood of a protege receiving technical assistance that is mission related increases significantly. Further, by incorporating the program into the acquisition process, it becomes an integral part of that process, thus mitigating the need for additional resources to administer the program and monitor program performance.

Services Are Inconsistent in Reimbursing Various Expenses

Neither the mentor-protege program legislation nor the implementing regulation specifically addresses either the payment of fees to mentors or the reimbursement of proteges for their costs in participating in the program.

In our March 1992 report, issued shortly after the program actually began, we noted low mentor participation and recommended that DOD consider ways to enhance program incentives for mentor participation. At that time, we noted that the appeal of the cash reimbursement incentive was diminished because mentors were prohibited by regulation from earning a fee on the development assistance provided to proteges. In 1992, DOD eliminated the regulatory prohibition.

In our current review, we found that the payment of fees to mentors varied among the services. We found no instance where the Navy has paid a fee, while the Army and the Air Force have paid fees to mentors. According to the OSADBU program manager, mentors have received fees in addition to their expenses because mentor-protege agreements are reimbursed by means of a line item in DOD contracts that allow a fee and the program’s legislation does not prohibit the payment of fees to the mentor.

In May 1997, the Air Force asked OSADBU about reimbursing protege salaries while the proteges were undergoing training. The Air Force noted that it had found no prohibition on such payments in either the program legislation or the regulation. OSADBU approved the payment of such incidental expenses in July 1997, so long as they were otherwise, allowable, allocable, and reasonable and did not exceed 10 percent of the total reimbursable cost of the approved mentor-protege agreement. We found that the Army and the Air Force permit these payments; the Navy does not.

Extent Proteges Benefited From Funding Has Not Been Determined

While the mentor-protege program has provided incentives to mentors to provide assistance to proteges, the extent that the funding has enhanced the business competitiveness and financial independence of protege firms has not been determined because sufficient and reliable information on program performance has not been available. OSADBU is developing a program database that it believes will be the source for periodic analysis of program results.

About 180 mentors received reimbursement of certain costs incurred in providing assistance to their protege firms, credit in meeting small business goals, or both. Of the 180 mentors, 92 used only reimbursable agreements, 69 used only credit agreements, and 19 used both reimbursement and credit agreements. We found that 86 mentors have active agreements, while 94 mentors have completed or terminated their agreements.

Of the 340 proteges that received assistance from mentors, 201 received assistance through reimbursable agreements, 113 received assistance through credit agreements, and 26 used both types of agreements. We found that 147 proteges have active agreements, while 193 proteges have completed or terminated their agreements.

OSADBU Is Developing a Program Database

OSADBU is developing a program database on all participants, including responses to the questionnaire surveys. The database is expected to incorporate information gathered from the mentor application, the agreements, and the data gathered as a result of the DCMC reviews. In addition, a reference resource is expected to provide more easily understood program guidelines and streamlined procedures for program participation and guidance on costs eligible for reimbursement, lessons learned, and best practices.

Conclusion

After spending over $200 million on the pilot mentor-protege program, OSADBU does not know the overall extent to which the program is achieving the purposes Congress has established because sufficient and reliable information on program performance is unavailable. While OSADBU has initiated an assessment of the program by conducting a survey and establishing a performance review process, shortcomings in the assessment will limit its usefulness. For example, a low survey response rate will limit the usefulness of its findings in assessing the program’s overall effectiveness. In addition, performance reviews were missing information or did not address all the required areas, such as improvements in protege capabilities and contract awards. While it would be difficult to correct the shortcomings in the survey to assess the overall effectiveness of the program, the weaknesses in the performance reviews can be overcome.

Neither the mentor-protege program legislation nor the implementing regulation specifically addresses either the payment of fees to mentors or the reimbursement of proteges for their costs in participating in the program. We found that there is no uniformity among the services on making these payments.

Recommendation

We recommend that the Secretary of Defense instruct the Director, OSADBU, to strengthen OSADBU’s performance reviews to ensure that sufficient and reliable information on program performance is gathered on planned and actual mentor assistance as well as on the protege firms’ business development.

Matter for Congressional Consideration

Congress may wish to clarify mentor-protege program legislation as to whether mentors can be paid fees in addition to expenses and proteges can be reimbursed for various expenses.

Agency Comments

DOD concurs with the recommendation to strengthen performance reviews and will continue to make efforts to improve the reliability and sufficiency of data on the program. However, DOD states that there have been a significant number of proteges that have benefited from the program and provides nine examples of mentor-protege agreements that it believes illustrate the effectiveness of the program. We found that there has been a lack of sufficient and reliable information on close to 400 agreements approved by OSADBU to assess the program’s overall effectiveness and do not believe that nine examples, while positive in themselves, are a sufficient basis for this assessment. (See app. I for DOD’s written comments.)

DOD acknowledges that the services have been inconsistent in their policies on paying fees to mentors for providing assistance to proteges and reimbursing proteges for various expenses. By allowing the diversity of approaches, DOD states that it is testing the range of options for program execution and the impact of various incentives upon participation in the program. Given that the mentor-protege program legislation and the implementing regulation do not specifically address either paying fees to mentors or reimbursing proteges for their costs in participating in the program, we continue to believe that Congress may wish to clarify existing legislation as to the use of mentor-protege funding.

Scope and Methodology

The congressional mandate states, in part, that we will study the extent to which the program is achieving the purposes established by Congress. Because DOD maintained performance review and evaluation data on the program that was insufficient and unreliable, we considered making an independent data collection survey. We did not make such a survey because OSADBU and a trade association were completing separate surveys of the same population of mentors and proteges. We also did not make the survey because repeated surveys of the same population at closely spaced intervals can lead to “history effects,” which can effect both the response rate and the quality of the data collected in unpredictable ways. We focused our review on the management of the program.

To review program management, we reviewed our past reports on the program that identified weaknesses in the management, obtained DOD’s response to our findings, and identified OSADBU’s actions initiated to improve program administration and evaluation. We also contacted OSADBU’s contractor who is responsible for administering and summarizing the DOD survey questionnaire and DCMC to identify its progress in conducting performance reviews of the mentor-protege agreements.

To review the manner in which the funds were obligated, we met with OSADBU and Washington Headquarters Service personnel who transfer mentor-protege funding from OSADBU to the responsible services and the defense agencies. As such, we obtained overall funding information dealing with the program from fiscal year 1992 to 1998 that showed a breakdown of the funding by responsible organization. To verify the funding information, we contacted officials within the services’ and defense agencies’ offices of Small and Disadvantaged Business Utilization responsible for administering the mentor-protege agreements. When necessary, these officials contacted supporting commands to gather more reliable information regarding the obligation of the funding.

We attempted to assess the extent to which the funds benefited the proteges by reviewing OSADBU’s files and contacting service and defense agency officials to identify the number of mentors and proteges that have participated in the program. Sufficient and reliable information on program performance, however, was unavailable on the overall extent that the proteges benefited from the funding provided by the program.

We performed this phase of work between August 1997 and February 1998 in accordance with generally accepted government auditing standards.

We will send copies to the Chairmen and the Ranking Minority Members of other appropriate congressional committees; the Secretary of Defense; and the Director, Office of Management and Budget. We will also make copies available to others upon request.

If you have any questions regarding this report, please call me on (202) 512-4841. Major contributors to this report are listed in appendix II.

Comments From the Department of Defense

The following are GAO’s comments on the Department of Defense’s (DOD) letter dated February 26, 1998.

GAO Comments

1. DOD states that a case study approach can sufficiently portray the impact of the program upon the protege firms. In this regard, DOD provided nine examples of mentor-protege agreements that it believed illustrated the effectiveness of the program. However, there have been close to 400 agreements approved by OSADBU since the inception of the program. We do not believe the nine examples, while positive in themselves, are sufficient to assess the program’s overall effectiveness. Therefore, we have chosen not to include the enclosure in this report. 2. We have modified the report to properly attribute to OSADBU officials the actions that they believe have improved program administration. 3. We have incorporated DOD’s technical comments in the text where appropriate.

Major Contributors to This Report

National Security and International Affairs Division, Washington, D.C.

Charles D. Groves Stanley J. Kostyla Russell R. Reiter Patricia D. Slocum Charles W. Thompson The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.