Small Business: Action Needed to Determine Whether DOD's Comprehensive Subcontracting Plan Test Program Should Be Made Permanent
Released: 2015-11-16T00:00:00

Letter

Background

In fiscal year 2014, DOD’s prime contractors subcontracted for $133 billion in goods and services to support DOD’s missions, of which $44 billion, or 33 percent, was awarded to small businesses. The Federal Acquisition Regulation (FAR) generally requires proposed prime contractors to have individual subcontracting plans in place for contracts (including modifications) of more than $650,000—or $1.5 million for construction contracts—whenever subcontracting opportunities exist. These plans are to document subcontracting goals as a specific dollar amount planned for small business awards and as a percentage of total available subcontracting dollars to various socioeconomic categories of small businesses. The plans also are to identify the types of products and services suitable for subcontracting awards.

Under the authorizing statute, as implemented in regulation, each participant in the Test Program negotiates and reports on subcontracting goals and achievements for a specific fiscal year on a plant, division, or corporate-wide basis. A comprehensive plan may cover a large number of individual contracts. For example, one participant told us about a plan that covered more than 3,000 contracts that otherwise would require individual subcontracting plans. Reporting small business subcontracting activity in a comprehensive plan means that less data may be available on the subcontracting activities for specific contracts or programs. In addition, comprehensive subcontracting plans are to include various initiatives to enhance small business subcontracting opportunities through specific programs or other actions. According to a DOD official, these initiatives are not specific to any one contract, but can be completed across the entire scope of defense work the prime contractor performs. Table 1 highlights the key differences between the comprehensive subcontracting plans used in the Test Program and individual subcontracting plans.

Current Test Program eligibility is limited by statute to defense contractors that performed at least three DOD prime contracts for supplies and services worth a combined value of at least $5 million during the preceding fiscal year. In addition, the contractor must have achieved a small disadvantaged business subcontracting participation rate of at least 5 percent during the preceding fiscal year. Participation in the Test Program is voluntary and, as shown in table 2, there are currently 12 participants.

According to data in the Federal Procurement Data System—Next Generation, the federal government’s contract reporting system, 8 of these contractors are ranked among the top 10 U.S. defense contractors, based on contract dollars obligated in fiscal year 2014.

According to DOD officials, the DOD Office of Small Business Programs (OSBP) is responsible for overseeing the Test Program, but delegates the management and oversight of comprehensive subcontracting plan annual negotiations and performance evaluation to the Defense Contract Management Agency (DCMA). DCMA is responsible for reviewing and approving Test Program participants’ proposed comprehensive subcontracting goals and initiatives to ensure that they are challenging yet realistic. DCMA is also responsible for reviewing the achievements of the participants at the end of the fiscal year and rating their performance. The Test Program does not have an overall measure for demonstrating success in creating small business subcontracting opportunities; instead, metrics are established for each individual participant. The performance of participants in the Test Program is measured by their achievement of negotiated goals and initiatives.

Congress has extended the program eight times but has not made the program permanent in part because of a lack of data on program performance. The latest extension was in the National Defense Authorization Act for Fiscal Year 2015 in which Congress temporarily extended the Test Program to December 31, 2017, and made a number of amendments to the program. These amendments included, among other things: an increase in the eligibility requirements; additional reporting requirements on subcontracting activities and costs to increase visibility at the contract, program, and military department levels; and an additional consequence for failure to make good faith efforts to comply with the program.

DOD commissioned reviews of the Test Program from two consulting groups in 2002, 2007, and 2013. These reviews examined the performance of Test Program participants against their goals and initiatives and whether administrative savings had been achieved. According to DOD officials, DOD has not released these reviews publicly or submitted them to the Congress. All of the reviews found that the program resulted in administrative costs savings and enhanced subcontracting opportunities for small businesses.

Some in the small business community have publicly raised concerns about the Test Program. The primary concern is that the lack of data available to evaluate the program precludes a determination of the program’s effectiveness and impact on small businesses. Specifically, some small business advocates believe that more data are needed to evaluate whether the program has resulted in more awards in areas such as innovative technology research that traditionally have not been made available to small businesses.

Test Program Allows Participants and DOD to Avoid Millions in Administrative Costs and May Provide Additional Benefits

Prior DOD reviews estimated that use of the Test Program resulted in the avoidance of millions of dollars in administrative costs for both participants and DOD and may offer other benefits. Significant one-time administrative costs could result if the Test Program were canceled or allowed to expire and comprehensive subcontracting plans could no longer be used. Test Program participants and DOD officials we interviewed stated that the use of comprehensive subcontracting plans results in benefits other than administrative cost avoidance as well. However, DOD has not taken action to address the program’s status.

Test Program Reduces Administrative Costs for Participants and DOD

DOD’s reviews estimated that the program resulted in millions of dollars in cost avoidance for the program’s participants. According to these reviews, cost avoidance is enabled primarily by the use of a single comprehensive subcontracting plan for multiple contracts rather than an individual subcontracting plan for each contract. Our examination of the methodologies and data used in these reviews, as well as our own analysis of more recent data on contracts covered by comprehensive subcontracting plans, support these conclusions. Table 3 shows the total estimated annual administrative costs avoided by the participants, according to the DOD reviews.

Some of the Test Program participant officials we interviewed explained that they have not quantified their administrative cost avoidance under the program; however, they stated that savings likely accrued from negotiating a single plan for multiple contracts as opposed to individual plans. One participant official stated that utilizing comprehensive subcontracting plans also allows them to use fewer people for the administrative tasks of developing and monitoring subcontracting plans. Certain participants noted that the resources that would have gone to the development and administration of individual plans can be used instead for increased small business outreach activities such as more small business forums. Expansion of the Test Program from division-level to company-level could increase the cost avoidance for certain participants, but officials from one division-level participant we interviewed stated that the continued test status of the program served to inhibit that expansion. The 2007 DOD review also found the cost avoided by Test Program participants and DOD would increase substantially if more prime contractors used the program, but that this was unlikely due to their reluctance to enter the program because of its test status.

In addition to the costs avoided by program participants, DOD also may benefit from avoiding administrative costs as a result of the Test Program. The 2007 DOD review found that DOD benefited from negotiating, administering, and monitoring consolidated small business subcontracting plans rather than multiple individual plans. For example, for fiscal year 2005, the review estimated that the administrative cost avoided by DOD under the Test Program was at least $45 million. Defense officials stated that additional participation in the Test Program by other prime contractors could increase the department’s cost avoidance, but stated that they are not considering new participants for the program due to its test status.

Administrative Costs to Convert Existing Test Program Plan Contracts to Individual Contracting Plans Could Be Substantial

DOD’s 2013 review noted that the program participants at that time had about 7,000 contracts and subcontracts that would require individual subcontracting plans if the Test Program did not exist. The estimated cost to convert the contracts to individual subcontracting plans if the Test Program were canceled or allowed to expire was $21.7 million. The review also noted that the combined cost for the participants to prepare, submit and negotiate their comprehensive subcontracting plans on an annual basis under the Test Program was approximately $660,000. In addition, most of the current participants have been in the Test Program for at least 10 years; two have been in the program since its inception 25 years ago. As a result, converting from comprehensive subcontracting plans to individual subcontracting plans may require participants to add personnel and change their current administrative systems. One participant noted that reverting to individual subcontracting plans would require an additional 44 employees to handle the additional administrative workload and an estimated $2 million in system changes.

Our analysis of similar data supports these estimates. We analyzed data from DCMA for contracts that were active as of March 2015 for 6 of the 12 Test Program participants, including those with the largest number of contracts, and found a total of 3,299 contracts that would require individual subcontracting plans in the absence of the Test Program. Using estimates provided by DOD for the cost and number of hours required to develop an individual subcontracting plan, we determined that these six participants would need to spend between $6.3 million and $9.5 million to convert existing contracts to individual subcontracting plans. One participant we interviewed estimated that for fiscal year 2013, the cost to convert all of the contracts under the Test Program to individual plans would have been approximately $6.6 million. The cost to this participant to prepare, review, and negotiate its fiscal year 2013 comprehensive subcontracting plan was $159,000.

As is the case with the Test Program participants, DOD would likely experience an increase in costs for negotiating, administering, and monitoring individual small business subcontracting plans if the program were to end. Defense officials noted that DOD would have to hire additional personnel in DCMA and the military services to manage the thousands of additional individual subcontracting plans. These personnel—which would include contracting officers, small business specialists, and cost analysts—all require specific levels of expertise and training depending on the cost and complexity of the contract. According to DOD officials, it would be challenging to provide the amount of resources required with the limited acquisition workforce DOD employs and available budgets for hiring new personnel.

Test Program May Result in Other Benefits for Participants, DOD, and Small Busineses but DOD Has Not Taken Action to Address Program Status

According to some of the Test Program participants we interviewed, the use of comprehensive subcontracting plans as opposed to individual subcontracting plans also provides additional non-monetary benefits. For example, certain participants stated that they benefit from the approach of consolidating small business subcontracting, as it allows them to consider or leverage small business subcontracting opportunities across the whole organization rather than on a contract-by-contract basis. Some participants stated that as a result of this approach they have increased small business subcontracting outside of their defense contracts by utilizing the same small businesses cultivated under the Test Program for other work. All of the participants we interviewed noted that the issue of small business subcontracting has greater visibility and awareness with corporate leadership as a result of the Test Program and that their entities might be less inclined to award subcontracts to small businesses in its absence.

DOD officials agreed that, in addition to administrative cost avoidance, the Test Program gives them leverage they may not have in negotiating individual contracting plans. The ability to require the contractor to look for opportunities across multiple contracts increases the likelihood of small business outreach and subcontract awards. Officials stated, however, that there are some concerns within the department about the Test Program. For example, they stated that reporting of small business subcontracting activity at a division or corporate level reduces the department’s visibility into small business activity on a contract or programmatic level. In addition, DOD officials said that the continuing test status of the program has made it difficult for OSBP and DCMA to develop the policies and guidance they believe are needed to enhance program activities.

The 2007 review strongly recommended that DOD work with Congress to make the Test Program permanent. According to DOD officials, DOD has not acted on this recommendation, by drafting a legislative proposal or taking other actions, primarily due to concerns that the program, at least as structured prior to the latest legislative changes, may limit visibility into small business subcontracting activities for individual programs or contracts. However, implementation of the increased reporting requirements legislated in the National Defense Authorization Act for Fiscal Year 2015 may alleviate these concerns as they include reporting on small business subcontracting activities at, among other things, the contract and major defense acquisition program level. Working with Congress to address the program’s status, for example by providing information on the effectiveness of the Test Program as identified in the three DOD-commissioned reviews and our analysis, could help eliminate the uncertainty associated with the program.

Test Program Enhanced Small Business Opportunities, Though Participant Performance Varied

In reviewing participants’ performance in the Test Program, DCMA looks at whether participants have achieved the negotiated initiatives and goals associated with their comprehensive subcontracting plans, and assigns an overall rating for that fiscal year’s performance. Our analysis found that 87 percent of DCMA’s reviews indicated that participants made acceptable progress on their initiatives to enhance small business opportunities, but that participants’ performance in meeting their negotiated subcontracting goals varied. Our analysis also found that while participants did not always meet individual goals, their aggregate performance resulted in subcontract awards to small business that exceeded aggregate goals by approximately $5.4 billion from fiscal years 2006 through 2013. When compared to the performance of contractors not participating in the program (nonparticipants), Test Program participants have lower percentages of dollars awarded to small businesses. Participants we interviewed explained that this is due, among other things, to the nature of the contracts they perform and the way the Test Program tracks achievement of goals. Finally, DCMA’s annual performance reviews, which assess achievements against both negotiated initiatives and goals, have been largely positive.

Implementation of Initiatives Enhanced Subcontracting Opportunities for Small Businesses

Participants made acceptable progress, as assessed by DCMA, on their negotiated initiatives in 74 of the 85 reviews we analyzed—an 87 percent success rate. Our analysis indicates that completion of these initiatives has resulted in tangible subcontracting opportunities for small businesses. DOD officials stated the achievement of these initiatives is not tied to any one contract within the comprehensive subcontracting plan; any of the defense work being subcontracted by a prime contractor can be used to meet the initiative. According to a participant and DOD officials, the initiatives offer a distinct advantage over individual subcontracting plans. Examples of initiatives include: redirecting subcontracts from large businesses to small businesses; targeting increased small business subcontracts in technical fields such as integrated circuits, computer information technology products, and engineering and technical services; and participating in the Small Business Innovation Research (SBIR) and Mentor-Protégé Programs.

Some of these initiatives target the socioeconomic categories of small businesses for which the participants had the most difficulty achieving performance goals. For example, one participant failed to meet its subcontracting goals for the small disadvantaged business category over a 3-year period. For each of these years the participant had an initiative to increase performance in that category.

According to our analysis of DCMA reviews, participants are generally successful at achieving their initiatives. In our review period of fiscal years 2006 through 2013, we found

16 initiatives resulted in the redirect of approximately $93 million in subcontracts from large business to small businesses; a 72-percent success rate in participants meeting milestones for increasing small business subcontracts in targeted industries;

24 mentor-protégé initiatives resulted in 61 new mentor-protégé relationships between participants and small businesses; and

11 SBIR projects that identified 83 new small business suppliers.

The 2013 DOD review estimated that initiatives completed by Test Program participants could amount to as much as $1.8 billion per year in increased small business opportunities, and that participants spent as much as $5.5 million annually on small business subcontracting enhancements and initiatives.

Participants’ Performance in Meeting Goals Varies

Our analysis found that Test Program participants’ performance in meeting their negotiated subcontracting goals varied. The Test Program does not have overall small business subcontracting goals similar to those that DOD negotiates annually with the Small Business Administration, but does have goals on a contractor-by-contractor basis. The goals, as well as the achievements, for small business subcontracting are expressed in terms of (1) the total dollars awarded to small businesses, and (2) the dollars awarded to small business as a percent of total subcontract dollars.

DCMA is responsible for annually negotiating and approving Test Program participants’ dollar and percentage goals to ensure that they are realistic and challenging. In all the negotiation support memorandums we reviewed, DCMA attempted to negotiate with the participants to establish challenging goals. According to DCMA documentation, this was accomplished by, for example, analyzing a participant’s 5-year subcontracting performance trends and comparing them to submitted goals, or assessing a participant’s documented support of its proposed goals to determine if a goal higher than that presented by the participant was realistic.

As shown in figure 1, the number of Test Program participants that met or exceeded goals varied by fiscal year.

According to some participants we interviewed, there are a number of reasons that individual participants may not meet their annual goals. One reason is that the goals are intended to be “stretch goals” beyond what the participants are confident they know can be achieved. Our analysis also shows that when participants failed to meet a goal they generally missed the goal by a small percentage. For example, in fiscal year 2011, the eight participants who missed their goals all did so by less than 10 percent and in two cases by less than 1 percent. Some participants also stated that their inability to renegotiate goals to account for changes in the dollars available for subcontracting may also negatively affect their achievements. DCMA officials said that they generally prefer not to allow for renegotiation, as it is a time consuming process.

When examining the performance of Test Program participants as a group, we found that they exceeded their combined goals for subcontracting dollars awarded to small businesses in each year of our review period, as shown in figure 2, for a total of approximately $5.4 billion above the combined goals from fiscal years 2006 through 2013.

For the goal of percentage of total subcontracting dollars awarded to small business, we found that, at an aggregate level, participants exceeded goals in 3 of the 8 fiscal years we reviewed and missed achieving those goals by less than 4 percent in every other fiscal year.

Various Reasons Given for Disparity in Percentage of Subcontracting Dollars Awarded to Small Businesses by Participants Compared to Other Defense Contractors

In our discussion with one small business advocate, one of the criticisms of the Test Program raised was that participants generally awarded a lower percentage of their subcontracting dollars to small businesses than nonparticipants. Our analysis, as shown in figure 3, shows this to be the case.

For Test Program participants, the percentage of subcontract dollars awarded to small businesses ranged from a low of around 25 percent in fiscal year 2011 to a high of around 29 percent in fiscal year 2008. However, for nonparticipants the percentage was a low of around 33 percent in fiscal year 2008, and a high of nearly 43 percent in fiscal year 2006.

We found that over the 8-year review period, Test Program participants also experienced less success in achieving their negotiated goals for the various socioeconomic categories of small businesses when compared to nonparticipants’ achievements. Test Program participants consistently awarded, on the basis of percentage of subcontract dollars awarded, lower percentages than nonparticipants. The only fiscal year that Test Program participants’ awarded higher amounts than nonparticipants was in fiscal year 2008 and then only for 2 of the 5 subcategories measured.

According to some participants we interviewed, there are a number of factors that can adversely affect their ability to achieve their annual percentage and dollar subcontracting goals. Examples of these factors include funding changes that affect the value of a contract or other changes to contracts that affect the amounts available for subcontracting. These factors are not unique to Test Program participants and could apply to any contractor involved in federal contracting; however, there are some factors that may have a disproportionate effect on Test Program participants. These factors include the following:

Test Program participants represent some of DOD’s largest prime contractors. As a result, many of the contracts they receive from DOD are large, technically complex, and include a scope of work that may require teaming arrangements with other large businesses, reducing the dollars available for subcontracting to small businesses. For example, the F-35 Joint Strike Fighter program has a prime contract with Lockheed Martin Corporation. For this highly complex and technical work, Lockheed Martin has subcontracted significant portions of the work to Northrup Grumman Corporation and BAE Systems, neither of which is considered a small business.

Test Program participants can only count certain small business subcontracting activities toward their goals. While small businesses may be involved at some level of subcontracting—either through a direct award from the prime contractor as a first-tier subcontractor or as a second-tier subcontractor through an award from a first-tier subcontractor—the ability of a Test Program participant to count small business subcontracting dollars towards their performance goals is limited. According to Test Program participants and DOD officials we interviewed, federal acquisition regulations require that goals for small business subcontracting be based on the prime contractor’s first-tier subcontracts. This government-wide practice does not permit prime contractors to report subcontracting below the first-tier for purposes of demonstrating the achievement of small business subcontracting goals. Therefore, participants can only take credit for awards made at the first-tier of subcontracting. For example, for the F-35 program Lockheed Martin has approximately 500 first-tier suppliers and 1,250 second-tier suppliers. Lockheed Martin does not receive credit for small business subcontract achievements below the first-tier, which could represent a significant amount of small business subcontracting.

If a small business subcontractor grows into a large business, its subcontracts are no longer counted toward participants’ goals. The determination of a subcontractor’s size as small for subcontracting purposes is set on the date that it self-certifies that it is small for the subcontract and is not typically revisited for the duration of the contract. Some participants we interviewed stated that, for their comprehensive subcontracting plans, the size determination is revisited when assessing progress against annual goals. For example, if a small business becomes successful and wins contracts that grow it into a larger business during the comprehensive subcontracting plan’s annual assessment period, a Test Program participant cannot count those subcontracting dollars toward the attainment of its small business goals. For instance, if a participant subcontracts work to a veteran-owned small business which then wins contracts from other sources and grows beyond the small business size standards, those subcontract dollars are excluded from the calculation of goal attainment.

Make or buy decision. Participants may make decisions to produce components in-house rather than buying them from a supplier in order to reduce the overall contract cost. This decision thus reduces the amount of subcontracting dollars available to small businesses.

Overall Ratings on Participants’ Performance Are Generally Positive

To achieve another perspective on Test Program performance, we analyzed DCMA performance ratings for the individual participants. In our review of 85 annual performance reviews for fiscal years 2006 through 2013, Test Program participants generally received positive annual performance ratings from DCMA. DCMA is responsible for annually evaluating the performance of Test Program participants and making recommendations as to their continued participation in the program. When evaluating the participants, DCMA is to consider their performance on both goals and initiatives and, if they failed to achieve these, whether the participants made a “good faith effort” to do so. Participants receive an annual overall program rating ranging from “Unsatisfactory” to “Outstanding.” Participants that do not receive at least an “Acceptable” rating are required to submit to DCMA a detailed corrective action plan to account for and improve on known deficiencies.

For example, for fiscal year 2013, the 12 participating firms received the following ratings:

Five participants received an Outstanding rating—meaning they generally exceeded the annual negotiated small business goals and two additional socioeconomic category goals and had exceptional success with numerous specific initiatives to assist, promote, and utilize small businesses.

Four participants received a Highly Successful rating—meaning they generally met or exceeded negotiated goals, including three small business categories, and had moderate success with some initiatives to assist, promote, and utilize small businesses.

Three participants received an Acceptable rating—meaning they generally demonstrated a good faith effort to meet all of their annual subcontracting goals and provided reasonable and supportable explanations why certain goals could not be met.

In the period we reviewed, fiscal years 2006 through 2013, no participants received a rating of “Unsatisfactory” and three participants received a rating of “Marginal.” Also, none of the participants we reviewed in the period were found by DCMA to not have made a “good faith effort” in seeking to meet their goals, no matter what their overall rating. According to DCMA annual performance documentation, the “Marginal” ratings were assigned generally because the participants were deficient in meeting key subcontracting plan elements, or the contractor failed to satisfy one or more requirements of a corrective plan from a prior review. Two of these participants achieved ratings at the “Acceptable” level or above in subsequent reporting periods and one voluntarily exited the program.

DOD officials said they have never terminated a participant from the program, but that some have voluntarily exited the program. DOD officials stated that ratings are taken into consideration by both DOD and participants when negotiating future participation in the program and may be taken into consideration as part of the determination of past performance by contracting officers in awarding future government contracts. One participant representative highlighted that the company uses its high ratings as a marketing tool to attract small businesses.

Conclusions

Because of its large contracting operations, DOD is critical to the success of federal programs designed to provide opportunities for small businesses. The Test Program is aimed at enhancing these opportunities and reducing participants’ administrative costs. The evidence collected by DOD and our analysis of that evidence indicates that the program has achieved these goals. The use of comprehensive subcontracting plans allows both DOD and Test Program participants to avoid millions of dollars in administrative costs and has led to demonstrable enhancements in small business subcontracting opportunities, thereby meeting the criteria established by Congress. While there may be concerns about the visibility of small business subcontracting on particular contracts or programs, these concerns may diminish when legislative changes made to the program as part of the fiscal year 2015 National Defense Authorization Act are implemented. Given that the Test Program has been in existence for 25 years, and therefore has become a de facto permanent program for both DOD and participants, termination of the program in favor of individual subcontracting plans would likely require substantial increases in manpower and fiscal resources. However, DOD has not acted on a 2007 review recommendation to work with Congress to make the Test Program permanent. Continually extending the program rather than making it permanent creates uncertainty among participants and DOD, inhibiting the expansion of the program by some participants, the inclusion of new participants, and the formulation of DOD policies and additional guidance that could enhance the program’s results. Working with Congress to address the program’s status, for example by providing information on the effectiveness of the Test Program as identified in the three DOD-commissioned reviews and our analysis, could help eliminate the uncertainty associated with the program.

Matter for Congressional Consideration

To help ensure continued reductions in administrative costs to DOD and program participants and enhance subcontracting opportunities for small businesses, Congress should consider making the Test Program permanent.

Recommendation for Executive Action

We recommend that the Secretary of Defense work with Congress to determine the status of the Test Program. In doing so, the Secretary could provide Congress with information on the effectiveness of the Test Program as discussed in the three DOD-commissioned reviews.

Agency Comments and Our Evaluation

In written comments, DOD did not concur with the draft report’s recommendation to draft a legislative proposal to make the program permanent and questioned the finding related to the Test Program enhancing small business subcontracting opportunities. The department agreed to work with Congress to determine the status of the Test Program. Given DOD’s disagreement, we added a matter for Congressional consideration to this report and modified the recommendation as discussed below. DOD’s comments are provided in full in appendix II.

Related to the finding on enhancing small business subcontracting opportunities, with its comments on our draft report, DOD provided a chart that shows a decrease in the percentage of subcontract dollars awarded to small business by Test Program participants from fiscal years 1996 through 2014. The department stated in its response that “his trend shows that the current practices associated with the do not enhance opportunities for small businesses.” We do not agree with that assessment for a number of reasons.

While the percentage of subcontracting dollars awarded to small business by program participants declined since fiscal year 1996, more recent data through fiscal year 2013 show that the performance of program participants has remained relatively stable, as shown in Figure 3. Although the percentages for program participants are lower than for nonparticipants, the trends are generally the same for both. The differences between the groups may be attributable to some of the factors we discussed in our report—such as the large, technically complex nature of some participant’s contracts and their inability to count certain small business subcontracting activities towards their goals—which do not suggest shortcomings in the program.

Enhancing opportunities for small business is measured by more than just the percentage of subcontract dollars awarded. For example, the value of the small business subcontracts awarded by program participants grew during this period from about $895 million to more than $7.7 billion. Small businesses therefore have received a relatively smaller portion of a significantly larger amount.

As discussed in the report, the Test Program also provided enhanced opportunities to small businesses through a variety of initiatives completed by the participants. These initiatives, among other things, resulted in the redirection of millions of dollars of subcontracts from large businesses to small businesses, and increased small business participation in targeted industries, including innovative research.

Finally, regardless of the overall percentages, we found that program participants met or nearly met all of the “stretch” percentage goals negotiated with the department in the fiscal years we reviewed. In total, the Test Program participants exceeded the dollar goals by approximately $5.4 billion.

Thus, we continue to believe the program achieved its goals of enhancing opportunities for small businesses, and as discussed in the report, reducing participants’ and DOD’s administrative costs. Given these findings, and that DOD had not acted on the 2007 review recommendation to work with Congress to make the program permanent, we included a recommendation in the draft report for DOD to draft a legislative proposal to make the program permanent or otherwise work with Congress to determine the status of the Test Program. The department agreed to work with Congress to determine the status of the Test Program; however, DOD did not concur with the recommendation to draft a legislative proposal. Consequently we modified the recommendation and added a matter for Congressional consideration to help ensure continued reductions in administrative costs to DOD and program participants and enhance subcontracting opportunities for small businesses.

We are sending copies of this report to the appropriate congressional committees and to the Secretary of Defense. This report will also be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by email at woodsw@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Section 821(e) of the National Defense Authorization Act for Fiscal Year 2015 included a provision for us to report on the results of the Department of Defense’s Test Program for Negotiation of Comprehensive Small Business Subcontracting Plans (Test Program). This report addresses the extent to which the Test Program (1) reduces administrative costs, and (2) enhances subcontracting opportunities for small businesses.

To address if the Test Program reduces administrative costs for program participants, we collected and analyzed three reviews conducted for the Department of Defense (DOD) on the Test Program in 2002, 2007, and 2013 that included estimates of the costs associated with the program. We assessed the methodologies and assumptions utilized—including the number of contracts and cost of labor to compile program documentation—and used data collected for DOD’s 2013 study to validate the findings. To help assess the validity of the data, we discussed with DOD officials how the data was collected. We determined that the methodologies were valid and the data were reliable for our purposes. In addition, we interviewed DOD officials and 5 of the 12 Test Program participants to obtain their views on any cost savings generated by having the Test Program, as well as any benefits they experienced from a reduced administrative burden. Test Program participants we interviewed represented different levels of participation in the program, including two corporate-level participants and three division-level participants, as well as range of subcontracting activity. The views expressed by these participants provided insight into the operation of the program but are not generalizable to all program participants.

In order to determine if the conversion costs articulated by the 2013 review were reasonable, we assessed the methodology used, obtained the data used for the estimate, discussed how the data was collected with DOD, and performed our own analysis using more recent data that found the conclusions of the 2013 review to be reasonable. To perform this analysis, we obtained March 2015 contract data from the Defense Contract Management Agency (DCMA) for the Test Program participants. From the contract listings we received, we removed those contracts for which we could not determine contract costs, as well as those that were below reporting thresholds that would require an individual subcontracting plan, to determine the total number of contracts that would have required individual subcontracting plans in the absence of the Test Program. DOD officials also provided a range of estimates for the number of hours required to develop an individual subcontracting plan, as well as the cost per hour, which we used to determine the cost for conversion. To validate the March 2015 contract data received from DCMA, we also obtained data from one corporate-level participant with a high level of subcontracting activity that we interviewed. This allowed us to estimate its cost for converting contracts under their comprehensive subcontracting plan to individual subcontracting plans. We determined that the data were reliable for our purposes.

To address whether the effect of Test Program initiatives enhanced small business subcontracting opportunities, we selected the period of fiscal years 2006 through 2013 for analysis. This selection was made because prior DOD reviews released in 2002 and 2007 assessed participants’ performance against goals and initiatives through fiscal year 2005 and fiscal year 2013 was the last full year of data available. The 2002 study used data from fiscal years 1991 to 2000, and the 2007 study used data from fiscal years 2001 to 2005. We reviewed the available plans, memorandums, and reviews used by DCMA to assess participant performance from fiscal years 2006 through 2013. This included: 80 annual comprehensive subcontracting plans submitted by participants to DCMA, 60 DCMA memorandums documenting negotiations with the participants, and 85 annual Form 640 reviews performance reviews by DCMA. We also analyzed the three DOD-commissioned reviews. To provide context for the negotiation process, as well as to gather different viewpoints about the initiatives as a whole, we interviewed representatives from DCMA’s Small Business Programs Division, DOD’s Office of Small Business Programs (OSBP), and five Test Program participants.

To address whether the Test Program enhances subcontracting opportunities for small businesses by successfully achieving its annual goals, we reviewed Test Program participants’ comprehensive subcontracting plans and negotiation support memorandums and analyzed performance data from fiscal years 2006 through 2013. We compared the actual performance contained in the Form 640 reviews against the approved subcontracting goals. We also obtained information about the small business subcontracting performance of DOD in general from DOD’s OSBP. Combining this information with that obtained for program participants, we identified trends in small business subcontracting for Test Program participants in comparison to DOD small business subcontracting performance in general. We also reviewed legislation, agency guidance, Federal Register notices, and relevant GAO and DOD reports. We also interviewed officials from DOD’s OSBP, DCMA’s Small Business Programs Division, 5 of the 12 prime contractors participating in the Test Program, and two small business advocacy groups chosen for their representation of the small business community, for their views on the Test Program, and factors that contribute to or undermine its success. The views expressed by these groups provided insight into the operation of the program but are not generalizable to all small businesses.

We conducted this performance audit from March to November 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, J. Kristopher Keener (Assistant Director), Kathryn (Emily) Bond, Joe Hunter (Analyst-in-Charge), Cale Jones, Julia Kennon, Stephen V. Marchesani, Sylvia Schatz, and Roxanna Sun made key contributions to this report.