Small Business: Action Needed to Determine Whether DOD's Comprehensive Subcontracting Plan Test Program Should Be Made Permanent
Published: 2015-11-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1990, DOD has been conducting a congressionally directed test program related to how contractors report their subcontracting activities. The purpose of the program is to test whether using comprehensive subcontracting plans that cover multiple contracts across contractor plants, divisions, or entire companies can yield administrative cost savings and enhance small business subcontracting opportunities. Despite the 25-year existence of the program, little is publicly known about its effectiveness.
The National Defense Authorization Act for Fiscal Year 2015 included a provision for GAO to report on the results of the program. This report addresses the extent to which the program (1) reduces administrative costs, and (2) enhances subcontracting opportunities for small businesses. GAO analyzed prior DOD reviews and data on estimates of administrative costs savings; reviewed program participants' performance for enhancing small business subcontracting opportunities for fiscal years 2006 through 2013; and interviewed officials from DOD, program participants, and small business advocacy groups.
section_title: What GAO Found
paragraphs: Reviews commissioned by the Department of Defense (DOD) concluded that the Test Program for Negotiation of Comprehensive Small Business Subcontracting Plans (Test Program) has resulted in the avoidance of millions of dollars in administrative costs for both program participants and DOD. According to the review conducted in 2013, the 12 firms then participating in the program avoided about $18.5 million in costs through the use of single comprehensive subcontracting plans rather than multiple individual subcontracting plans. Also, a 2007 review estimated that DOD avoided administrative costs of at least $45 million in fiscal year 2005. GAO reviewed the methodologies used for these reviews and took other steps to validate their findings. According to DOD officials, if the Test Program were terminated or allowed to expire, a significant one-time administrative cost of about $22 million could result to participants. GAO's analysis confirms this conclusion. Test Program participants and DOD officials GAO interviewed stated that the program also has resulted in non-financial benefits, including greater company-wide awareness of small business subcontracting opportunities. The participants GAO interviewed said that without the program their companies might be less inclined to award subcontracts to small businesses. They emphasized, however, that the program's continuing test status creates uncertainty and inhibits further expansion. The 2007 review recommended that DOD work with Congress to make the Test Program permanent; however, DOD has not acted on this recommendation. Doing so could help eliminate uncertainty with the program.
GAO found that the Test Program enhanced small business subcontracting opportunities, although participants' performance in meeting individual goals has varied. Participants are evaluated on their achievement of negotiated initiatives and goals in their comprehensive subcontracting plans. GAO's analysis of performance reports found that participants made acceptable progress on their initiatives 87 percent of the time, thus providing tangible subcontracting opportunities for small businesses. For example, during fiscal years 2006 through 2013, program participants redirected nearly $93 million in subcontracts from large businesses to small businesses. Participants also achieved a 72-percent success rate in increasing small business subcontracts in areas such as integrated circuits and information technology, thus addressing a concern among some small businesses that high-end technical work was not being subcontracted under the program. The 2013 DOD review estimated that participant initiatives could amount to as much as $1.8 billion per year in increased small business opportunities. GAO's analysis found that participants did not always meet individual goals, in part due to the challenging nature of these goals. However, GAO also found that their combined performance from fiscal years 2006 through 2013 resulted in subcontract awards to small businesses that exceeded aggregate goals by about $5.4 billion. The annual performance reviews of the participants, which take into account performance on both initiatives and goals, have been largely positive.
section_title: What GAO Recommends
paragraphs: Congress should consider making the program permanent. GAO also recommends that DOD work with Congress on the program's status. DOD agreed. DOD disagreed with a recommendation to draft a legislative proposal to make the program permanent. GAO subsequently modified the recommendation and added the matter for Congress.

section_title: Letter
paragraphs: Contractors on certain defense contracts are required to develop plans  specifying how much of the work will be performed by subcontractors. In  1989, Congress authorized the Test Program for Negotiation of  Comprehensive Small Business Subcontracting Plans (Test Program),  which permits participating contractors to submit a single subcontracting  plan covering all of their contracts rather than a separate plan for each  contract. The purpose of the Test Program is to reduce the administrative  burden on contractors and enhance subcontracting opportunities for small  businesses. Although the Test Program’s initial performance period was  3 years, Congress has extended its duration eight times, with the most  recent extension scheduled to end December 31, 2017. Despite the 25- year tenure of the Test Program, little is publicly known about its results,  as the Department of Defense (DOD) has not published any reports on its  performance.
The National Defense Authorization Act for Fiscal Year 2015 included a  provision for us to report on the results of the Test Program. This report  addresses the extent to which the Test Program (1) reduces  administrative costs, and (2) enhances subcontracting opportunities for  small businesses.
To address whether the Test Program reduces administrative costs, we  analyzed DOD-commissioned reviews of the program performed in 2002,  2007, and 2013; each of which included estimates of the cost savings or  avoidance associated with the program. We assessed the methodologies  employed and assumptions used, including estimates of the number of  contracts and cost of labor involved in preparing program documentation,  to arrive at the estimates for savings. To validate the data used in the  2013 study, we interviewed DOD officials to discuss how the data were  collected and obtained, and we analyzed and compared the data to more  current information we obtained independently. We found the  methodologies in all three studies to be reasonable, and the data were  sufficiently reliable for our purposes. We interviewed officials from DOD  and 5 of the 12 Test Program participants, selected to represent different  levels of corporate participation in the program and varying volumes of  small business subcontracting activity, to obtain their views on any  reduction in administrative costs and other benefits they may have  realized from their participation in the program. The views expressed by  these participants provided insight into the operation of the program, but  are not generalizable to all program participants.
To address whether the Test Program enhances subcontracting  opportunities for small businesses, we reviewed legislation and agency  guidance for the Test Program and relevant GAO and DOD reports. We  also reviewed Test Program participants’ comprehensive subcontracting  plans from fiscal years 2006—the first year not covered by a prior DOD  review—through 2013—the last year for which full data were available.  We compared the approved subcontracting goals and initiatives as stated  in the plans to the results for each participant and as reflected in annual  performance reviews, known as Form 640 reviews, conducted by the  Defense Contract Management Agency (DCMA) to establish trends for  participant performance. We also interviewed officials from DOD’s Office  of Small Business Programs, DCMA’s Small Business Programs Division,  the five Test Program participants described above, and two small  business advocacy groups chosen for their representation of the small  business community for their views on the Test Program. The views  expressed by these groups provided insight into the operation of the  program, but are not generalizable to all small businesses. Appendix I  provides additional details on our objectives, scope, and methodology.
We conducted this performance audit from March to November 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: In fiscal year 2014, DOD’s prime contractors subcontracted for $133  billion in goods and services to support DOD’s missions, of which $44  billion, or 33 percent, was awarded to small businesses. The Federal  Acquisition Regulation (FAR) generally requires proposed prime  contractors to have individual subcontracting plans in place for contracts  (including modifications) of more than $650,000—or $1.5 million for  construction contracts—whenever subcontracting opportunities exist.  These plans are to document subcontracting goals as a specific dollar  amount planned for small business awards and as a percentage of total  available subcontracting dollars to various socioeconomic categories of  small businesses. The plans also are to identify the types of products  and services suitable for subcontracting awards.
Under the authorizing statute, as implemented in regulation, each  participant in the Test Program negotiates and reports on subcontracting  goals and achievements for a specific fiscal year on a plant, division, or  corporate-wide basis. A comprehensive plan may cover a large number of  individual contracts. For example, one participant told us about a plan that  covered more than 3,000 contracts that otherwise would require individual  subcontracting plans. Reporting small business subcontracting activity in  a comprehensive plan means that less data may be available on the  subcontracting activities for specific contracts or programs. In addition,  comprehensive subcontracting plans are to include various initiatives to  enhance small business subcontracting opportunities through specific  programs or other actions. According to a DOD official, these initiatives  are not specific to any one contract, but can be completed across the  entire scope of defense work the prime contractor performs. Table 1  highlights the key differences between the comprehensive subcontracting  plans used in the Test Program and individual subcontracting plans.
Current Test Program eligibility is limited by statute to defense contractors  that performed at least three DOD prime contracts for supplies and  services worth a combined value of at least $5 million during the  preceding fiscal year. In addition, the contractor must have achieved a  small disadvantaged business subcontracting participation rate of at least  5 percent during the preceding fiscal year. Participation in the Test  Program is voluntary and, as shown in table 2, there are currently 12  participants.
According to data in the Federal Procurement Data System—Next  Generation, the federal government’s contract reporting system, 8 of  these contractors are ranked among the top 10 U.S. defense contractors,  based on contract dollars obligated in fiscal year 2014.
According to DOD officials, the DOD Office of Small Business Programs  (OSBP) is responsible for overseeing the Test Program, but delegates the  management and oversight of comprehensive subcontracting plan annual  negotiations and performance evaluation to the Defense Contract  Management Agency (DCMA). DCMA is responsible for reviewing and  approving Test Program participants’ proposed comprehensive  subcontracting goals and initiatives to ensure that they are challenging  yet realistic. DCMA is also responsible for reviewing the achievements of  the participants at the end of the fiscal year and rating their performance.  The Test Program does not have an overall measure for demonstrating  success in creating small business subcontracting opportunities; instead,  metrics are established for each individual participant. The performance  of participants in the Test Program is measured by their achievement of  negotiated goals and initiatives.
Congress has extended the program eight times but has not made the  program permanent in part because of a lack of data on program  performance. The latest extension was in the National Defense  Authorization Act for Fiscal Year 2015 in which Congress temporarily  extended the Test Program to December 31, 2017, and made a number  of amendments to the program. These amendments included, among  other things: an increase in the eligibility requirements; additional  reporting requirements on subcontracting activities and costs to increase  visibility at the contract, program, and military department levels; and an  additional consequence for failure to make good faith efforts to comply  with the program.
DOD commissioned reviews of the Test Program from two consulting  groups in 2002, 2007, and 2013. These reviews examined the  performance of Test Program participants against their goals and  initiatives and whether administrative savings had been achieved.  According to DOD officials, DOD has not released these reviews publicly  or submitted them to the Congress. All of the reviews found that the  program resulted in administrative costs savings and enhanced  subcontracting opportunities for small businesses.
Some in the small business community have publicly raised concerns  about the Test Program. The primary concern is that the lack of data  available to evaluate the program precludes a determination of the  program’s effectiveness and impact on small businesses. Specifically,  some small business advocates believe that more data are needed to  evaluate whether the program has resulted in more awards in areas such  as innovative technology research that traditionally have not been made  available to small businesses.
section_title: Test Program Allows Participants and DOD to Avoid Millions in Administrative Costs and May Provide Additional Benefits
paragraphs: Prior DOD reviews estimated that use of the Test Program resulted in the  avoidance of millions of dollars in administrative costs for both  participants and DOD and may offer other benefits. Significant one-time  administrative costs could result if the Test Program were canceled or  allowed to expire and comprehensive subcontracting plans could no  longer be used. Test Program participants and DOD officials we  interviewed stated that the use of comprehensive subcontracting plans  results in benefits other than administrative cost avoidance as well.  However, DOD has not taken action to address the program’s status.
subsections: section_title: Test Program Reduces Administrative Costs for Participants and DOD
paragraphs: DOD’s reviews estimated that the program resulted in millions of dollars  in cost avoidance for the program’s participants. According to these  reviews, cost avoidance is enabled primarily by the use of a single  comprehensive subcontracting plan for multiple contracts rather than an  individual subcontracting plan for each contract. Our examination of the  methodologies and data used in these reviews, as well as our own  analysis of more recent data on contracts covered by comprehensive  subcontracting plans, support these conclusions. Table 3 shows the total  estimated annual administrative costs avoided by the participants,  according to the DOD reviews.
Some of the Test Program participant officials we interviewed explained  that they have not quantified their administrative cost avoidance under the  program; however, they stated that savings likely accrued from  negotiating a single plan for multiple contracts as opposed to individual  plans. One participant official stated that utilizing comprehensive  subcontracting plans also allows them to use fewer people for the  administrative tasks of developing and monitoring subcontracting plans.  Certain participants noted that the resources that would have gone to the  development and administration of individual plans can be used instead  for increased small business outreach activities such as more small  business forums. Expansion of the Test Program from division-level to  company-level could increase the cost avoidance for certain participants,  but officials from one division-level participant we interviewed stated that  the continued test status of the program served to inhibit that expansion.  The 2007 DOD review also found the cost avoided by Test Program  participants and DOD would increase substantially if more prime  contractors used the program, but that this was unlikely due to their  reluctance to enter the program because of its test status.
In addition to the costs avoided by program participants, DOD also may  benefit from avoiding administrative costs as a result of the Test Program.  The 2007 DOD review found that DOD benefited from negotiating,  administering, and monitoring consolidated small business subcontracting  plans rather than multiple individual plans. For example, for fiscal year  2005, the review estimated that the administrative cost avoided by DOD  under the Test Program was at least $45 million. Defense officials stated  that additional participation in the Test Program by other prime  contractors could increase the department’s cost avoidance, but stated  that they are not considering new participants for the program due to its  test status.
section_title: Administrative Costs to Convert Existing Test Program Plan Contracts to Individual Contracting Plans Could Be Substantial
paragraphs: DOD’s 2013 review noted that the program participants at that time had  about 7,000 contracts and subcontracts that would require individual  subcontracting plans if the Test Program did not exist. The estimated cost  to convert the contracts to individual subcontracting plans if the Test  Program were canceled or allowed to expire was $21.7 million. The  review also noted that the combined cost for the participants to prepare,  submit and negotiate their comprehensive subcontracting plans on an  annual basis under the Test Program was approximately $660,000. In  addition, most of the current participants have been in the Test Program  for at least 10 years; two have been in the program since its inception 25  years ago. As a result, converting from comprehensive subcontracting  plans to individual subcontracting plans may require participants to add  personnel and change their current administrative systems. One  participant noted that reverting to individual subcontracting plans would  require an additional 44 employees to handle the additional administrative  workload and an estimated $2 million in system changes.
Our analysis of similar data supports these estimates. We analyzed data  from DCMA for contracts that were active as of March 2015 for 6 of the  12 Test Program participants, including those with the largest number of  contracts, and found a total of 3,299 contracts that would require  individual subcontracting plans in the absence of the Test Program. Using  estimates provided by DOD for the cost and number of hours required to  develop an individual subcontracting plan, we determined that these six  participants would need to spend between $6.3 million and $9.5 million to  convert existing contracts to individual subcontracting plans. One  participant we interviewed estimated that for fiscal year 2013, the cost to  convert all of the contracts under the Test Program to individual plans  would have been approximately $6.6 million. The cost to this participant  to prepare, review, and negotiate its fiscal year 2013 comprehensive  subcontracting plan was $159,000.
As is the case with the Test Program participants, DOD would likely  experience an increase in costs for negotiating, administering, and  monitoring individual small business subcontracting plans if the program  were to end. Defense officials noted that DOD would have to hire  additional personnel in DCMA and the military services to manage the  thousands of additional individual subcontracting plans. These  personnel—which would include contracting officers, small business  specialists, and cost analysts—all require specific levels of expertise and  training depending on the cost and complexity of the contract. According  to DOD officials, it would be challenging to provide the amount of  resources required with the limited acquisition workforce DOD employs  and available budgets for hiring new personnel.
section_title: Test Program May Result in Other Benefits for Participants, DOD, and Small Busineses but DOD Has Not Taken Action to Address Program Status
paragraphs: According to some of the Test Program participants we interviewed, the  use of comprehensive subcontracting plans as opposed to individual  subcontracting plans also provides additional non-monetary benefits. For  example, certain participants stated that they benefit from the approach of  consolidating small business subcontracting, as it allows them to consider  or leverage small business subcontracting opportunities across the whole  organization rather than on a contract-by-contract basis. Some  participants stated that as a result of this approach they have increased  small business subcontracting outside of their defense contracts by  utilizing the same small businesses cultivated under the Test Program for  other work. All of the participants we interviewed noted that the issue of  small business subcontracting has greater visibility and awareness with  corporate leadership as a result of the Test Program and that their entities  might be less inclined to award subcontracts to small businesses in its  absence.
DOD officials agreed that, in addition to administrative cost avoidance,  the Test Program gives them leverage they may not have in negotiating  individual contracting plans. The ability to require the contractor to look for  opportunities across multiple contracts increases the likelihood of small  business outreach and subcontract awards. Officials stated, however, that  there are some concerns within the department about the Test Program.  For example, they stated that reporting of small business subcontracting  activity at a division or corporate level reduces the department’s visibility  into small business activity on a contract or programmatic level. In  addition, DOD officials said that the continuing test status of the program  has made it difficult for OSBP and DCMA to develop the policies and  guidance they believe are needed to enhance program activities.
The 2007 review strongly recommended that DOD work with Congress to  make the Test Program permanent. According to DOD officials, DOD has  not acted on this recommendation, by drafting a legislative proposal or  taking other actions, primarily due to concerns that the program, at least  as structured prior to the latest legislative changes, may limit visibility into  small business subcontracting activities for individual programs or  contracts. However, implementation of the increased reporting  requirements legislated in the National Defense Authorization Act for  Fiscal Year 2015 may alleviate these concerns as they include reporting  on small business subcontracting activities at, among other things, the  contract and major defense acquisition program level. Working with  Congress to address the program’s status, for example by providing  information on the effectiveness of the Test Program as identified in the  three DOD-commissioned reviews and our analysis, could help eliminate  the uncertainty associated with the program.
section_title: Test Program Enhanced Small Business Opportunities, Though Participant Performance Varied
paragraphs: In reviewing participants’ performance in the Test Program, DCMA looks  at whether participants have achieved the negotiated initiatives and goals  associated with their comprehensive subcontracting plans, and assigns  an overall rating for that fiscal year’s performance. Our analysis found  that 87 percent of DCMA’s reviews indicated that participants made  acceptable progress on their initiatives to enhance small business  opportunities, but that participants’ performance in meeting their  negotiated subcontracting goals varied. Our analysis also found that while  participants did not always meet individual goals, their aggregate  performance resulted in subcontract awards to small business that  exceeded aggregate goals by approximately $5.4 billion from fiscal years  2006 through 2013. When compared to the performance of contractors  not participating in the program (nonparticipants), Test Program  participants have lower percentages of dollars awarded to small  businesses. Participants we interviewed explained that this is due, among  other things, to the nature of the contracts they perform and the way the  Test Program tracks achievement of goals. Finally, DCMA’s annual  performance reviews, which assess achievements against both  negotiated initiatives and goals, have been largely positive.
subsections: section_title: Implementation of Initiatives Enhanced Subcontracting Opportunities for Small Businesses
paragraphs: Participants made acceptable progress, as assessed by DCMA, on their  negotiated initiatives in 74 of the 85 reviews we analyzed—an 87 percent  success rate. Our analysis indicates that completion of these initiatives  has resulted in tangible subcontracting opportunities for small businesses.  DOD officials stated the achievement of these initiatives is not tied to any  one contract within the comprehensive subcontracting plan; any of the  defense work being subcontracted by a prime contractor can be used to  meet the initiative. According to a participant and DOD officials, the  initiatives offer a distinct advantage over individual subcontracting plans.  Examples of initiatives include:  redirecting subcontracts from large businesses to small businesses;  targeting increased small business subcontracts in technical fields  such as integrated circuits, computer information technology products,  and engineering and technical services; and participating in the Small Business Innovation Research (SBIR) and  Mentor-Protégé Programs.
Some of these initiatives target the socioeconomic categories of small  businesses for which the participants had the most difficulty achieving  performance goals. For example, one participant failed to meet its  subcontracting goals for the small disadvantaged business category over  a 3-year period. For each of these years the participant had an initiative to  increase performance in that category.
According to our analysis of DCMA reviews, participants are generally  successful at achieving their initiatives. In our review period of fiscal years  2006 through 2013, we found
16 initiatives resulted in the redirect of approximately $93 million in  subcontracts from large business to small businesses; a 72-percent success rate in participants meeting milestones for  increasing small business subcontracts in targeted industries;
24 mentor-protégé initiatives resulted in 61 new mentor-protégé  relationships between participants and small businesses; and
11 SBIR projects that identified 83 new small business suppliers.
The 2013 DOD review estimated that initiatives completed by Test  Program participants could amount to as much as $1.8 billion per year in  increased small business opportunities, and that participants spent as  much as $5.5 million annually on small business subcontracting  enhancements and initiatives.
section_title: Participants’ Performance in Meeting Goals Varies
paragraphs: Our analysis found that Test Program participants’ performance in  meeting their negotiated subcontracting goals varied. The Test Program  does not have overall small business subcontracting goals similar to  those that DOD negotiates annually with the Small Business  Administration, but does have goals on a contractor-by-contractor basis.  The goals, as well as the achievements, for small business  subcontracting are expressed in terms of (1) the total dollars awarded to  small businesses, and (2) the dollars awarded to small business as a  percent of total subcontract dollars.
DCMA is responsible for annually negotiating and approving Test  Program participants’ dollar and percentage goals to ensure that they are  realistic and challenging. In all the negotiation support memorandums we  reviewed, DCMA attempted to negotiate with the participants to establish  challenging goals. According to DCMA documentation, this was  accomplished by, for example, analyzing a participant’s 5-year  subcontracting performance trends and comparing them to submitted  goals, or assessing a participant’s documented support of its proposed  goals to determine if a goal higher than that presented by the participant  was realistic.
As shown in figure 1, the number of Test Program participants that met or  exceeded goals varied by fiscal year.
According to some participants we interviewed, there are a number of  reasons that individual participants may not meet their annual goals. One  reason is that the goals are intended to be “stretch goals” beyond what  the participants are confident they know can be achieved. Our analysis  also shows that when participants failed to meet a goal they generally  missed the goal by a small percentage. For example, in fiscal year 2011,  the eight participants who missed their goals all did so by less than 10  percent and in two cases by less than 1 percent. Some participants also  stated that their inability to renegotiate goals to account for changes in the  dollars available for subcontracting may also negatively affect their  achievements. DCMA officials said that they generally prefer not to allow  for renegotiation, as it is a time consuming process.
When examining the performance of Test Program participants as a  group, we found that they exceeded their combined goals for  subcontracting dollars awarded to small businesses in each year of our  review period, as shown in figure 2, for a total of approximately $5.4  billion above the combined goals from fiscal years 2006 through 2013.
For the goal of percentage of total subcontracting dollars awarded to  small business, we found that, at an aggregate level, participants  exceeded goals in 3 of the 8 fiscal years we reviewed and missed  achieving those goals by less than 4 percent in every other fiscal year.
section_title: Various Reasons Given for Disparity in Percentage of Subcontracting Dollars Awarded to Small Businesses by Participants Compared to Other Defense Contractors
paragraphs: In our discussion with one small business advocate, one of the criticisms  of the Test Program raised was that participants generally awarded a  lower percentage of their subcontracting dollars to small businesses than  nonparticipants. Our analysis, as shown in figure 3, shows this to be the  case.
For Test Program participants, the percentage of subcontract dollars  awarded to small businesses ranged from a low of around 25 percent in  fiscal year 2011 to a high of around 29 percent in fiscal year 2008.  However, for nonparticipants the percentage was a low of around 33  percent in fiscal year 2008, and a high of nearly 43 percent in fiscal year  2006.
We found that over the 8-year review period, Test Program participants  also experienced less success in achieving their negotiated goals for the  various socioeconomic categories of small businesses when compared to  nonparticipants’ achievements. Test Program participants consistently  awarded, on the basis of percentage of subcontract dollars awarded,  lower percentages than nonparticipants. The only fiscal year that Test  Program participants’ awarded higher amounts than nonparticipants was  in fiscal year 2008 and then only for 2 of the 5 subcategories measured.
According to some participants we interviewed, there are a number of  factors that can adversely affect their ability to achieve their annual  percentage and dollar subcontracting goals. Examples of these factors  include funding changes that affect the value of a contract or other  changes to contracts that affect the amounts available for subcontracting.  These factors are not unique to Test Program participants and could  apply to any contractor involved in federal contracting; however, there are  some factors that may have a disproportionate effect on Test Program  participants. These factors include the following:
Test Program participants represent some of DOD’s largest  prime contractors. As a result, many of the contracts they receive  from DOD are large, technically complex, and include a scope of work  that may require teaming arrangements with other large businesses,  reducing the dollars available for subcontracting to small businesses.  For example, the F-35 Joint Strike Fighter program has a prime  contract with Lockheed Martin Corporation. For this highly complex  and technical work, Lockheed Martin has subcontracted significant  portions of the work to Northrup Grumman Corporation and BAE  Systems, neither of which is considered a small business.
Test Program participants can only count certain small business  subcontracting activities toward their goals. While small  businesses may be involved at some level of subcontracting—either  through a direct award from the prime contractor as a first-tier  subcontractor or as a second-tier subcontractor through an award  from a first-tier subcontractor—the ability of a Test Program  participant to count small business subcontracting dollars towards  their performance goals is limited. According to Test Program  participants and DOD officials we interviewed, federal acquisition  regulations require that goals for small business subcontracting be  based on the prime contractor’s first-tier subcontracts. This  government-wide practice does not permit prime contractors to report  subcontracting below the first-tier for purposes of demonstrating the  achievement of small business subcontracting goals. Therefore,  participants can only take credit for awards made at the first-tier of  subcontracting. For example, for the F-35 program Lockheed Martin  has approximately 500 first-tier suppliers and 1,250 second-tier  suppliers. Lockheed Martin does not receive credit for small business  subcontract achievements below the first-tier, which could represent a  significant amount of small business subcontracting.
If a small business subcontractor grows into a large business, its  subcontracts are no longer counted toward participants’ goals.  The determination of a subcontractor’s size as small for  subcontracting purposes is set on the date that it self-certifies that it is  small for the subcontract and is not typically revisited for the duration  of the contract. Some participants we interviewed stated that, for their  comprehensive subcontracting plans, the size determination is  revisited when assessing progress against annual goals. For  example, if a small business becomes successful and wins contracts  that grow it into a larger business during the comprehensive  subcontracting plan’s annual assessment period, a Test Program  participant cannot count those subcontracting dollars toward the  attainment of its small business goals. For instance, if a participant  subcontracts work to a veteran-owned small business which then wins  contracts from other sources and grows beyond the small business  size standards, those subcontract dollars are excluded from the  calculation of goal attainment.
Make or buy decision. Participants may make decisions to produce  components in-house rather than buying them from a supplier in order  to reduce the overall contract cost. This decision thus reduces the  amount of subcontracting dollars available to small businesses.
section_title: Overall Ratings on Participants’ Performance Are Generally Positive
paragraphs: To achieve another perspective on Test Program performance, we  analyzed DCMA performance ratings for the individual participants. In our  review of 85 annual performance reviews for fiscal years 2006 through  2013, Test Program participants generally received positive annual  performance ratings from DCMA. DCMA is responsible for annually  evaluating the performance of Test Program participants and making  recommendations as to their continued participation in the program.  When evaluating the participants, DCMA is to consider their performance  on both goals and initiatives and, if they failed to achieve these, whether  the participants made a “good faith effort” to do so. Participants receive  an annual overall program rating ranging from “Unsatisfactory” to  “Outstanding.” Participants that do not receive at least an “Acceptable”  rating are required to submit to DCMA a detailed corrective action plan to  account for and improve on known deficiencies.
For example, for fiscal year 2013, the 12 participating firms received the  following ratings:
Five participants received an Outstanding rating—meaning they  generally exceeded the annual negotiated small business goals and  two additional socioeconomic category goals and had exceptional  success with numerous specific initiatives to assist, promote, and  utilize small businesses.
Four participants received a Highly Successful rating—meaning they  generally met or exceeded negotiated goals, including three small  business categories, and had moderate success with some initiatives  to assist, promote, and utilize small businesses.
Three participants received an Acceptable rating—meaning they  generally demonstrated a good faith effort to meet all of their annual  subcontracting goals and provided reasonable and supportable  explanations why certain goals could not be met.
In the period we reviewed, fiscal years 2006 through 2013, no participants  received a rating of “Unsatisfactory” and three participants received a  rating of “Marginal.” Also, none of the participants we reviewed in the  period were found by DCMA to not have made a “good faith effort” in  seeking to meet their goals, no matter what their overall rating. According  to DCMA annual performance documentation, the “Marginal” ratings were  assigned generally because the participants were deficient in meeting key  subcontracting plan elements, or the contractor failed to satisfy one or  more requirements of a corrective plan from a prior review. Two of these  participants achieved ratings at the “Acceptable” level or above in  subsequent reporting periods and one voluntarily exited the program.
DOD officials said they have never terminated a participant from the  program, but that some have voluntarily exited the program. DOD officials  stated that ratings are taken into consideration by both DOD and  participants when negotiating future participation in the program and may  be taken into consideration as part of the determination of past  performance by contracting officers in awarding future government  contracts. One participant representative highlighted that the company  uses its high ratings as a marketing tool to attract small businesses.
section_title: Conclusions
paragraphs: Because of its large contracting operations, DOD is critical to the success  of federal programs designed to provide opportunities for small  businesses. The Test Program is aimed at enhancing these opportunities  and reducing participants’ administrative costs. The evidence collected by  DOD and our analysis of that evidence indicates that the program has  achieved these goals. The use of comprehensive subcontracting plans  allows both DOD and Test Program participants to avoid millions of  dollars in administrative costs and has led to demonstrable  enhancements in small business subcontracting opportunities, thereby  meeting the criteria established by Congress. While there may be  concerns about the visibility of small business subcontracting on  particular contracts or programs, these concerns may diminish when  legislative changes made to the program as part of the fiscal year 2015  National Defense Authorization Act are implemented. Given that the Test  Program has been in existence for 25 years, and therefore has become a  de facto permanent program for both DOD and participants, termination  of the program in favor of individual subcontracting plans would likely  require substantial increases in manpower and fiscal resources. However,  DOD has not acted on a 2007 review recommendation to work with  Congress to make the Test Program permanent. Continually extending  the program rather than making it permanent creates uncertainty among  participants and DOD, inhibiting the expansion of the program by some  participants, the inclusion of new participants, and the formulation of DOD  policies and additional guidance that could enhance the program’s  results. Working with Congress to address the program’s status, for  example by providing information on the effectiveness of the Test  Program as identified in the three DOD-commissioned reviews and our  analysis, could help eliminate the uncertainty associated with the  program.
section_title: Matter for Congressional Consideration
paragraphs: To help ensure continued reductions in administrative costs to DOD and  program participants and enhance subcontracting opportunities for small  businesses, Congress should consider making the Test Program  permanent.
section_title: Recommendation for Executive Action
paragraphs: We recommend that the Secretary of Defense work with Congress to  determine the status of the Test Program. In doing so, the Secretary  could provide Congress with information on the effectiveness of the Test  Program as discussed in the three DOD-commissioned reviews.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments, DOD did not concur with the draft report’s  recommendation to draft a legislative proposal to make the program  permanent and questioned the finding related to the Test Program  enhancing small business subcontracting opportunities. The department  agreed to work with Congress to determine the status of the Test  Program. Given DOD’s disagreement, we added a matter for  Congressional consideration to this report and modified the  recommendation as discussed below. DOD’s comments are provided in  full in appendix II.
Related to the finding on enhancing small business subcontracting  opportunities, with its comments on our draft report, DOD provided a  chart that shows a decrease in the percentage of subcontract dollars  awarded to small business by Test Program participants from fiscal years  1996 through 2014. The department stated in its response that “his  trend shows that the current practices associated with the   do not enhance opportunities for small businesses.” We do not agree with  that assessment for a number of reasons.
While the percentage of subcontracting dollars awarded to small  business by program participants declined since fiscal year 1996,  more recent data through fiscal year 2013 show that the performance  of program participants has remained relatively stable, as shown in  Figure 3. Although the percentages for program participants are lower  than for nonparticipants, the trends are generally the same for both.  The differences between the groups may be attributable to some of  the factors we discussed in our report—such as the large, technically  complex nature of some participant’s contracts and their inability to  count certain small business subcontracting activities towards their  goals—which do not suggest shortcomings in the program.
Enhancing opportunities for small business is measured by more than  just the percentage of subcontract dollars awarded. For example, the  value of the small business subcontracts awarded by program  participants grew during this period from about $895 million to more  than $7.7 billion. Small businesses therefore have received a  relatively smaller portion of a significantly larger amount.
As discussed in the report, the Test Program also provided enhanced  opportunities to small businesses through a variety of initiatives  completed by the participants. These initiatives, among other things,  resulted in the redirection of millions of dollars of subcontracts from  large businesses to small businesses, and increased small business  participation in targeted industries, including innovative research.
Finally, regardless of the overall percentages, we found that program  participants met or nearly met all of the “stretch” percentage goals  negotiated with the department in the fiscal years we reviewed. In  total, the Test Program participants exceeded the dollar goals by  approximately $5.4 billion.
Thus, we continue to believe the program achieved its goals of enhancing  opportunities for small businesses, and as discussed in the report,  reducing participants’ and DOD’s administrative costs. Given these  findings, and that DOD had not acted on the 2007 review  recommendation to work with Congress to make the program permanent,  we included a recommendation in the draft report for DOD to draft a  legislative proposal to make the program permanent or otherwise work  with Congress to determine the status of the Test Program. The  department agreed to work with Congress to determine the status of the  Test Program; however, DOD did not concur with the recommendation to  draft a legislative proposal. Consequently we modified the  recommendation and added a matter for Congressional consideration to  help ensure continued reductions in administrative costs to DOD and  program participants and enhance subcontracting opportunities for small  businesses.
We are sending copies of this report to the appropriate congressional  committees and to the Secretary of Defense. This report will also be  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841 or by email at woodsw@gao.gov. Contact  points for our Office of Congressional Relations and Public Affairs may be  found on the last page of this report. Staff who made key contributions to  this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Section 821(e) of the National Defense Authorization Act for Fiscal Year  2015 included a provision for us to report on the results of the Department  of Defense’s Test Program for Negotiation of Comprehensive Small  Business Subcontracting Plans (Test Program). This report addresses  the extent to which the Test Program (1) reduces administrative costs,  and (2) enhances subcontracting opportunities for small businesses.
To address if the Test Program reduces administrative costs for program  participants, we collected and analyzed three reviews conducted for the  Department of Defense (DOD) on the Test Program in 2002, 2007, and  2013 that included estimates of the costs associated with the program.  We assessed the methodologies and assumptions utilized—including the  number of contracts and cost of labor to compile program  documentation—and used data collected for DOD’s 2013 study to  validate the findings. To help assess the validity of the data, we discussed  with DOD officials how the data was collected. We determined that the  methodologies were valid and the data were reliable for our purposes. In  addition, we interviewed DOD officials and 5 of the 12 Test Program  participants to obtain their views on any cost savings generated by having  the Test Program, as well as any benefits they experienced from a  reduced administrative burden. Test Program participants we interviewed  represented different levels of participation in the program, including two  corporate-level participants and three division-level participants, as well  as range of subcontracting activity. The views expressed by these  participants provided insight into the operation of the program but are not  generalizable to all program participants.
In order to determine if the conversion costs articulated by the 2013  review were reasonable, we assessed the methodology used, obtained  the data used for the estimate, discussed how the data was collected with  DOD, and performed our own analysis using more recent data that found  the conclusions of the 2013 review to be reasonable. To perform this  analysis, we obtained March 2015 contract data from the Defense  Contract Management Agency (DCMA) for the Test Program participants.  From the contract listings we received, we removed those contracts for  which we could not determine contract costs, as well as those that were  below reporting thresholds that would require an individual subcontracting  plan, to determine the total number of contracts that would have required  individual subcontracting plans in the absence of the Test Program. DOD  officials also provided a range of estimates for the number of hours  required to develop an individual subcontracting plan, as well as the cost  per hour, which we used to determine the cost for conversion. To validate  the March 2015 contract data received from DCMA, we also obtained  data from one corporate-level participant with a high level of  subcontracting activity that we interviewed. This allowed us to estimate its  cost for converting contracts under their comprehensive subcontracting  plan to individual subcontracting plans. We determined that the data were  reliable for our purposes.
To address whether the effect of Test Program initiatives enhanced small  business subcontracting opportunities, we selected the period of fiscal  years 2006 through 2013 for analysis. This selection was made because  prior DOD reviews released in 2002 and 2007 assessed participants’  performance against goals and initiatives through fiscal year 2005 and  fiscal year 2013 was the last full year of data available. The 2002 study  used data from fiscal years 1991 to 2000, and the 2007 study used data  from fiscal years 2001 to 2005. We reviewed the available plans,  memorandums, and reviews used by DCMA to assess participant  performance from fiscal years 2006 through 2013. This included: 80  annual comprehensive subcontracting plans submitted by participants to  DCMA, 60 DCMA memorandums documenting negotiations with the  participants, and 85 annual Form 640 reviews performance reviews by  DCMA. We also analyzed the three DOD-commissioned reviews. To  provide context for the negotiation process, as well as to gather different  viewpoints about the initiatives as a whole, we interviewed  representatives from DCMA’s Small Business Programs Division, DOD’s  Office of Small Business Programs (OSBP), and five Test Program  participants.
To address whether the Test Program enhances subcontracting  opportunities for small businesses by successfully achieving its annual  goals, we reviewed Test Program participants’ comprehensive  subcontracting plans and negotiation support memorandums and  analyzed performance data from fiscal years 2006 through 2013. We  compared the actual performance contained in the Form 640 reviews  against the approved subcontracting goals. We also obtained information  about the small business subcontracting performance of DOD in general  from DOD’s OSBP. Combining this information with that obtained for  program participants, we identified trends in small business  subcontracting for Test Program participants in comparison to DOD small  business subcontracting performance in general. We also reviewed  legislation, agency guidance, Federal Register notices, and relevant GAO  and DOD reports. We also interviewed officials from DOD’s OSBP,  DCMA’s Small Business Programs Division, 5 of the 12 prime contractors  participating in the Test Program, and two small business advocacy  groups chosen for their representation of the small business community,  for their views on the Test Program, and factors that contribute to or  undermine its success. The views expressed by these groups provided  insight into the operation of the program but are not generalizable to all  small businesses.
We conducted this performance audit from March to November 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, J. Kristopher Keener (Assistant  Director), Kathryn (Emily) Bond, Joe Hunter (Analyst-in-Charge), Cale  Jones, Julia Kennon, Stephen V. Marchesani, Sylvia Schatz, and  Roxanna Sun made key contributions to this report.