text: Background Overview of the Scorecard Process According to SBA, the purposes of the scorecard program are to monitor government-wide performance in meeting small business contracting goals and to provide accurate and transparent information through the public reporting of small business procurement data for individual agencies and government-wide. SBA uses its scorecard methodology to calculate a numeric score for each agency annually. SBA then converts those numeric scores to letter grades on an A+ through F scale. Each year, SBA negotiates small business prime contracting goals with each federal agency with procurement authority such that, in the aggregate, the federal government meets its overall 23-percent goal for the percentage of prime contract dollars awarded to small businesses. In setting annual agency goals, SBA considers prior-year achievement and other factors. In addition to an overall prime contracting goal, Congress also established statutory contracting goals for various socioeconomic subcategories of small businesses. These small business subcategories are small disadvantaged businesses, women-owned small businesses, service-disabled veteran-owned small businesses, and businesses located in Historically Underutilized Business Zones (HUBZone). SBA does not negotiate agency-specific goals for prime contracting and subcontracting achievement within each small business socioeconomic subcategory. Instead, each agency’s goal is the same as the government- wide goals. Prime contracting and subcontracting achievement goals for each subcategory are shown in table 1 below. Procurement Data Systems SBA uses two government-wide data systems maintained by the General Services Administration (GSA) to measure agencies’ small business contracting activity. SBA uses the Federal Procurement Data System- Next Generation (FPDS-NG) to calculate agencies’ prime contracting awards to small businesses. Federal agencies are required to report to FPDS-NG all contracts whose estimated value is $3,500 or more, and FPDS-NG also records whether the contract has gone to a small business. GSA requires that agencies annually certify the accuracy of data submitted. To measure subcontracting, SBA uses the Electronic Subcontracting Reporting System (eSRS), which captures data on spending on first-tier subcontracts, including spending directed to small businesses. Prime contractors that hold one or more government contracts totaling more than $700,000 are required to report their small business subcontracting activity in eSRS. Role of the OSBDUs In 1978 Congress amended the Small Business Act to require that all federal agencies with procurement powers establish an Office of Small and Disadvantaged Business Utilization (OSDBU). These offices are intended to advocate for small businesses in procurement and contracting processes, and thus work with agencies to achieve contracting goals. OSDBUs have multiple functions and duties that are codified in section 15(k) of the Small Business Act, as amended. In addition to their agency responsibilities, OSDBU directors serve with the SBA administrator or a designee on the Small Business Procurement Advisory Council, which was established in 1994. The council’s duties include identifying best practices for maximizing small business utilization in federal contracting and conducting peer reviews of each OSDBU to determine compliance with section 15(k). SBA has included the results of this peer review as part of its scorecard calculations for several years. SBA Made Several Revisions to the Scorecard for Fiscal Year 2017 but Has Not Completed a Plan to Evaluate Those Changes Scorecard Revisions Focused Largely on Mandated Changes SBA revised the scorecard methodology prior to fiscal year 2017 to make it consistent with changes required by the 2016 NDAA. Specifically, SBA reduced the proportion of the total scorecard results related to prime contracting performance from 80 percent to 50 percent and added an element to calculate changes in the number of small business prime contractors compared to the prior year. SBA officials said they considered, but did not add, a scorecard element that calculated changes in the number of small business subcontractors, which the 2016 NDAA required to be included if data were available. Officials said that unlike prime contracting data, which are validated by agencies, subcontracting data are recorded by the prime contractor and are based on contracting plans and not obligated federal funds. As a result, SBA officials said they determined that data were not available to implement this change. SBA also made other changes to the scorecard methodology, as the agency was permitted to do under the 2016 NDAA. SBA adjusted the weights of other scorecard elements, increasing subcontracting performance from 10 percent to 20 percent of the total scorecard result and increasing the peer review evaluation element from 10 percent to 20 percent. SBA also established that the new statutorily required element to assess changes in the number of prime contractors would be weighted at 10 percent. (See fig. 1 for a summary of revisions to the scorecard methodology.) Officials said they increased the subcontracting weight because it was an increasingly important area of small business procurement activity. In addition, SBA officials and other Small Business Procurement Advisory Council members revised the peer review evaluation methodology in an effort to facilitate a more in-depth review of agencies’ compliance with section 15(k) requirements. SBA included the results from this new peer review process in its revised scorecard methodology. Specifically, the council changed the peer review process in an effort to have peer reviewers make compliance determinations for categories that directly corresponded to the individual subparts of section 15(k). The prior peer review process asked reviewers to assign scores in seven areas, which the process termed “success factors.” For the fiscal year 2017 scorecard, SBA asked peer reviewers to assess and provide scores for 18 of the 21 individual subparts. Categories for the three remaining 15(k) subparts were incorporated starting with the fiscal year 2018 scorecard methodology. SBA officials said members of the Small Business Procurement Advisory Council were active participants in determining the revisions to the scorecard methodology. For example, SBA officials said the council members gave input on proposed revisions and recommended changes prior to the adoption of the new scorecard methodology. OSDBU directors also discussed potential methodological revisions in meetings of the Federal OSDBU Directors Interagency Council. SBA officials said the OSDBU directors’ input was incorporated into SBA’s revised scorecard guidance and, as a result, the criteria within the scorecard were more robust. Officials we interviewed from SBA and other agencies said the adopted scorecard revisions were the result of a consensus among Small Business Procurement Advisory Council members, although no formal votes were taken. Revisions to the scorecard methodology were outlined in a memorandum circulated to agencies in August 2016, about 8 weeks before the start of fiscal year 2017. SBA officials said that many agencies were tracking their progress toward goals using the revised methodology before results were issued. Agencies also had an opportunity to review preliminary scorecard results for fiscal year 2017 before the official scorecard results were published in May 2018. Fiscal Year 2017 Scorecard Outcomes Were Similar to Those of Prior Years Scorecard results under the revised methodology were similar to those of prior years. For example, in fiscal year 2017, the distribution of agencies’ letter grade results was similar to those of fiscal years 2014 through 2016, with between 19 and 21 of the 24 scored agencies achieving at least an A grade each year (see table 2). Prime contracting achievement. Agencies’ performance in small business prime contracting was similar in fiscal year 2017 and fiscal year 2016 (see table 3). In both years, 18 of 24 agencies met their overall prime contracting goals. In fiscal year 2017, 15 of 24 agencies met at least three of the four small business subcategory goals—one fewer than in fiscal year 2016. Subcontracting achievement. In fiscal year 2017, 15 of 24 agencies met their subcontracting goals compared to 16 of 24 in the prior year. However, among the small business subcategories, more agencies met at least three subcategory goals in 2017 (14 agencies) than in fiscal year 2016 (10 agencies) (see table 4). Peer review evaluations element. The fiscal year 2017 government- wide score for the peer review of section 15(k) compliance (a score of 19.25 out of a maximum 20.00) was nearly identical to the government- wide score for fiscal year 2016, once we adjusted for changes in the scoring scale between the 2 years. The government-wide score in fiscal year 2016 was 9.60 out of 10, which equates to 19.20 on a 20-point scale. Number of small business prime contractors. The overall number of small business prime contractors declined between fiscal years 2016 and 2017. The number of prime contractors overall decreased from 120,009 in fiscal year 2016 to 117,480 in fiscal year 2017, a decrease of approximately 2 percent. However, the 24 agencies, in aggregate, had more small business prime contractors in three of the four small business subcategories in fiscal year 2017 than in the prior year (see table 5). Comparison with prior scorecard weighting formula. We found that agencies’ numerical scores for fiscal year 2017 were generally lower under the revised scorecard methodology than they would have been under the fiscal year 2016 methodology’s weighting of scorecard elements. Twenty-two of 24 agencies had a lower score than they would have had under the prior methodology’s weighting. The revised methodology adjusted the weight of multiple scorecard elements, and there are a variety of reasons why an agency might have received a lower score than under the fiscal year 2016 methodology’s weighting. However, reducing the weight for prime contracting achievement under the revised methodology could explain at least part of the lower score for 21 of the 22 agencies. The overall median score for fiscal year 2017 was about 7 points lower than it would have been under the weighting formula used in fiscal year 2016. (The median score for fiscal year 2017 scorecards was 111 and would have been 118 under the prior methodology’s weighting formula.) SBA Said It Was Preparing but Had Not Completed a Plan to Evaluate the Effects of Scorecard Revisions In June 2018, SBA officials told us they were not preparing a plan for evaluating the effects of scorecard revisions because they thought such a plan would be premature. At that time, SBA officials said they had identified some aspects of the revised methodology for further review, including two issues related to the peer review evaluations—the peer review scoring scale and whether agencies believed SBA’s requests for supporting information were reasonable. In July 2018, however, SBA officials said that, in response to our preliminary findings, they had begun to develop a plan for evaluating the revised scorecard methodology’s effects, if any, on meeting the government-wide procurement goals. The officials did not provide us a draft plan or details about the plan. They said they expected to complete the evaluation plan by October 2018 and to complete the evaluation itself by the end of December 2018. Federal internal control standards state that management should use quality information to achieve the entity’s objectives, such as those in an agency’s strategic plan. These standards also call for management to design control activities to achieve goals and respond to risks—for example, activities to monitor performance measures and indicators. SBA’s strategic plan includes an objective to ensure federal contract and innovation set-aside goals are met or exceeded. The agency uses scorecard results to measure progress toward meeting or exceeding the statutory goal of 23 percent for overall small business prime contracting. Scorecard results are also used to measure progress toward other goals for the small business socioeconomic subcategories. We have previously identified key attributes of effective program evaluation design, which include the following: clear criteria for making comparisons that would lead to strong, defensible evaluation conclusions; an established evaluation scope that would ensure that the evaluation is tied to its research questions, effectively defines the subject matter to be assessed, and can be completed in a timely fashion to meet reporting deadlines; clear and specific research evaluation questions that use terms that can be readily defined and measured; and carefully thought-out data and analysis choices, which can enhance the quality, credibility, and usefulness of the evaluation. A comprehensive evaluation of revisions to the scorecard that includes the key attributes outlined above could aid SBA officials in determining whether the revised scorecard provides better information and whether the scorecard revisions are designed and implemented appropriately. Such an evaluation also could assist SBA in understanding whether the scorecard revisions may contribute to maximizing contract dollars awarded to small businesses, which is one of the goals in SBA’s strategic plan. In addition, the 2016 NDAA requires that SBA report to Congress by March 31, 2019, about changes stemming from the revised methodology and recommend whether the scorecard program should continue or be further modified. Such an evaluation could also be used by SBA to inform its report to Congress and future decisions about the scorecard methodology and program. SBA Uses Available Procurement Data to Calculate Scorecard Outcomes, but the Process for Producing Scorecards Has Weaknesses Subcontracting Data Have Known Limitations That May Affect the Reliability of Scorecard Calculations The two data systems SBA uses to measure agencies’ small business contracting activity—FPDS-NG and eSRS—are the best available sources of procurement data for calculating scorecard results, according to SBA. However, eSRS has limitations that agency officials cited and that we have previously identified that could hinder the reliability of scorecard results on subcontracting. Federal law prohibits SBA from requiring agencies to use alternative data collection methods for the purposes of the scorecard calculations. GSA intends to replace both systems as part of an initiative to consolidate the functions of several existing data systems, according to GSA documents. As we reported in 2014, this new system is intended to better link prime contracting and subcontracting data. Agency officials we interviewed said eSRS has limitations that make it challenging to verify the accuracy of reported subcontracting activity, and we also have identified eSRS limitations in our prior work. Prime contractors are responsible for reporting their subcontracting activity to the federal government, and the self-reported nature of these data is a limitation that could hamper the accuracy of eSRS data, agency officials said. Although prime contractors generally are required to submit a plan describing planned subcontracting activity, officials explained that eSRS did not provide a method to allow agency officials to verify that actual subcontracting activity matched the levels described in prime contractors’ plans. In addition, not all prime contractors are required to file subcontracting plans. Exceptions to the requirement include, for example, when the prime contract is for goods or services worth $700,000 or less or if the prime contractor is exempt. Small business prime contractors are one example of an exempt group that is not required to prepare subcontracting plans. SBA officials added that measuring subcontracting activity also is challenging because there are no federal funds obligated for subcontracts. Therefore, the federal government does not have a verified record of who performed subcontracting work and the amount paid. In addition, our previous work has found that eSRS was not designed to provide a list of subcontractors associated with a particular prime contract and that linking small business subcontractors to prime contracts when there is a subcontracting plan that pertains to multiple contracts is especially difficult. In addition, our previous work has identified some limitations with FPDS- NG focused on specific agencies and small business programs, although we have not more broadly assessed the reliability of the FPDS-NG data fields that SBA uses to compile scorecard results. For example, we found mismatches between certain accounting records from the Department of Veterans Affairs and data captured in FPDS-NG, and we identified challenges in using FPDS-NG data to monitor the eligibility of Alaska Native Corporations for certain small business contracts available to small disadvantaged businesses. However, officials from SBA and two departments we interviewed for this work said prime contracting data in FPDS-NG generally do not have the same weaknesses they identified with subcontracting data in eSRS. Errors in Published Scorecard Results Weaken Reliability and Perceived Integrity of Scorecard Program Scorecard results originally published by SBA on May 22, 2018, contained errors, including one agency scorecard published with an incorrect letter grade. SBA officials said they discovered the publication errors within approximately 2 days of publication and published corrected versions. However, these corrections occurred after SBA issued a public announcement highlighting the new results, and interested parties may have downloaded erroneous results prior to the corrected versions being posted on SBA’s website. We identified errors from SBA’s originally published scorecards independent of SBA’s determination that the agency had published scorecards containing errors. The errors we and SBA identified were concentrated in the scorecard for the Department of Education and the government-wide scorecard: The scorecard for the Department of Education showed an incorrect letter grade of A+, rather than the correct grade of A. The published scorecard also showed an incorrect overall numeric score. The Department of Education’s score for the peer review component of the scorecard was incorrect. The government-wide scorecard showed incorrect scores for changes in the number of women-owned small business contractors and the number of service-disabled veteran-owned small business contractors. SBA did not initially document on the corrected scorecards how they had been changed from the original scorecards. However, SBA later added documentation that the scorecards for the Department of Education and government-wide results had been corrected. SBA took this step after we inquired about the absence of documentation about revisions that had been made to the fiscal year 2017 scorecards. SBA officials said they performed accurate calculations for determining agencies’ performance and that inaccuracies in the published scorecards were the result of transcription errors associated with formatting the results for publication. Officials said SBA used new software to publish the fiscal year 2017 scorecards so that they could be accessible to visually impaired readers. Making the scorecards more accessible required some additional steps and at times required manual data entry due to limitations in SBA’s software. These additional steps resulted in errors, officials said. One set of errors—the inaccurate government-wide scores for changes in the number of women-owned small business contractors and the number of service-disabled veteran-owned small business contractors—canceled each other out and did not lead to erroneous overall scorecard results. SBA officials said they review the scorecard data and calculations before they are prepared for publication. However, the agency does not have a process to review formatted scorecards prior to publication to confirm that the version for publication matches actual calculations. Agency officials said they believed that such a process was not necessary. Additionally, agency officials said SBA has instituted a process to update previously issued scorecards to make them accessible for the visually impaired. SBA officials said they intend to review the accuracy of these updated scorecards for characteristics such as accurate letter grades as agency resources permit. Both the Office of Management and Budget and SBA have issued policies related to transparency and integrity of government data. The Office of Management and Budget has issued government-wide guidance on transparency in sharing government data and instructed federal agencies to develop their own policies. SBA’s policy on information quality says the policy is intended, in part, to ensure the integrity of information SBA disseminates. SBA’s policy also says the agency should have full, accurate, transparent documentation and should identify and disclose to users any error sources affecting data quality. In addition, federal internal control standards cite the need for management to design controls— including controls over information processing—to achieve objectives. Errors in the published scorecards may impair the other agencies’ or Congress’s access to quality information to make informed decisions and evaluate an agency’s performance in meeting small business goals. The scorecard errors that we and SBA identified after publication—and the lack of any indicator that scorecards had been corrected—also may undermine confidence in the integrity and transparency of the scorecard data. Agency Officials and Other Stakeholders Expected the Revised Scorecard to Have Little Impact on Small Business Opportunities Agency officials and representatives of small business groups we spoke with generally expected the revised scorecard methodology for fiscal year 2017 to have little impact on small business procurement opportunities. OSDBU officials in the four agencies we interviewed said their offices, in general, are not altering existing efforts at advocating for small business opportunities as a result of scorecard revisions. Some agency officials also said they would need additional years of scorecard data before making any changes to their efforts or reassessing how their priorities align with the revised scorecard’s formula. However, officials from one agency said they updated their agency’s internal monitoring of subcontracting activity as a result of the revised scorecard methodology’s increased emphasis on subcontracting measures. Officials said they updated the monitoring process so the agency would place more emphasis on small business subcontracting activity. Officials said the change to this agency’s internal monitoring process took effect for fiscal year 2018. Officials from three of the four federal departments and representatives from the three small business groups we interviewed said they had not seen any changes in opportunities for small business prime contracting as a result of the scorecard’s methodological changes. Instead, representatives from three small business groups and officials from two departments said any changes in prime contracting opportunities that might have occurred would be influenced by other government-wide procurement initiatives. Specifically, representatives from the three small business groups said the federal government’s emphasis on “category management” was resulting in fewer prime contracts available to all government contractors, including small business contractors. Under the category management initiative, the federal government groups commonly purchased goods and services into categories to streamline procurement processes with the goal of eliminating redundancies and reducing costs. However, representatives of small business groups said these policies result in fewer contract awards and opportunities for small businesses. Representatives from the three small business groups said that the new scorecard element that calculates the annual changes in the number of small business contractors could help highlight the effects of these prime contracting trends on procurement opportunities. According to agency officials and small business representatives, subcontracting opportunities are also unlikely to be impacted by the revised scorecard methodology, which increased the weight of subcontracting performance. Officials from two of the four departments we interviewed told us that their agencies have stable purchasing patterns and that subcontracting activity is not likely to change as a result of scorecard revisions. Representatives from two of the three small business groups said the influence of the scorecard revisions in incentivizing agencies to focus on subcontracting opportunities is limited by the reliability of available subcontracting data, discussed previously. For example, one agency told us that the shift from prime to subcontracting performance reduces the agency’s ability to influence scorecard outcomes because the agency has no means of validating the subcontracting data that are recorded. Similarly, representatives from two of the three small business groups said that because the data on subcontracting are entered by the prime contractors at the time of proposed contracting rather than confirmed contracting, the data do not include verification of subcontracting activity and therefore might not be an accurate measure of subcontracting activity. Representatives from agencies and small business groups said the scorecard program has generally played a role in drawing attention to agencies’ performance in identifying small business procurement opportunities. For example, SBA officials said the scorecard results provide public information about how well the government performed overall in providing small business procurement opportunities and help to ensure that all agencies are contributing toward those goals. Officials at one agency told us that the scorecard was an important factor in driving internal goals and opportunities for small businesses. Another agency said that while it had been reaching its overall prime contracting goal, its performance in certain small business subcategories was falling short of goals. As a result, the agency has directed additional outreach efforts to those types of small businesses. In addition, representatives of all three small business groups said because results are public, the scorecard has created additional pressure on agencies to meet procurement goals. Conclusions SBA uses its scorecard program to monitor federal agencies’ compliance with goals set by Congress to promote small business participation in federal contracting, and SBA has identified having agencies meet or exceed those participation benchmarks as one of its agency-wide goals in its strategic plan. The effects of recent changes to the scorecard and their potential benefits for improving federal contracting opportunities for small businesses are uncertain. SBA recently began to develop a plan for evaluating whether or how changes to the scorecard might facilitate SBA’s ability to meet government-wide procurement goals. Completing such an evaluation and making sure the evaluation plan is aligned with key attributes for effective evaluations could help SBA management: determine whether the revised scorecard provides quality information—consistent with federal internal control standards—and whether it helps meet the agency’s strategic goals; fully address whether the revisions are effective in measuring and creating small business procurement opportunities; and make a well-supported recommendation about whether to continue or modify the scorecard program. Congress required that SBA recommend by March 31, 2019, whether to continue or modify the scorecard program. In addition, the scorecard appears to have played a role in drawing attention to agencies’ performance in identifying small business procurement opportunities. However, there were errors in the initial fiscal year 2017 scorecards published on SBA’s website, and SBA did not initially take steps to notify the public after it made corrections. SBA officials said that SBA does not have a process to ensure that published scorecard results are accurate. Errors in the published scorecards and a lack of timely disclosure about corrections may impair other agencies’ or Congress’s access to quality information to make informed decisions. Recommendations We are making the following two recommendations to SBA: The SBA Administrator or her designee should complete the design and implementation of a comprehensive evaluation of the Small Business Procurement Scorecard aligned with key attributes of effective program evaluations to assess the effectiveness of the revised scorecard in measuring agency performance and promoting small business procurement opportunities. (Recommendation 1) The SBA Administrator or her designee should institute a process to review Small Business Procurement Scorecards for accuracy prior to publication and a mechanism for publicly identifying when issued scorecards have been revised. (Recommendation 2) Agency Comments and Our Evaluation We provided a draft of this report to SBA for review and comment. In written comments, reproduced in appendix II, SBA generally agreed with both of our recommendations. Regarding our recommendation that SBA design and implement an evaluation of the revised scorecard methodology, SBA said it planned to evaluate the changes to the scorecard methodology mandated by the 2016 NDAA. As discussed in our report, in revising the scorecard, SBA also made other changes not specifically mandated by the 2016 NDAA, such as increasing the emphasis on small business subcontracting activity and incorporating a revised peer review process to facilitate a more in-depth review of agencies’ compliance with section 15(k) requirements. As stated in our report, we recommend that SBA plan and implement an evaluation of all aspects of the revised scorecard methodology. SBA also indicated that it will not complete the evaluation until after it has validated data for the fiscal year 2018 procurement scorecard. We note that SBA can prepare an evaluation plan and begin to consider potential evaluation findings using available scorecard data from fiscal year 2017. We also note that our recommendation states that SBA’s evaluation plan should be aligned with the key attributes of effective evaluation design. Regarding our recommendation that SBA institute a process to review scorecards for accuracy prior to publication and a mechanism for publicly identifying when issued scorecards have been revised, SBA said it had taken several steps to revise the processes for publishing accurate scorecard results, including adding steps to compare the prepared scorecard documents to source documents prior to publication and to annotate any score corrections that are made to published scorecards. While we have not yet had the opportunity to assess SBA’s actions, the steps SBA describes in response to our recommendation could improve other agencies’ or Congress’s access to quality information. We will send copies to the Administrator of SBA and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Appendix I: Objectives, Scope, and Methodology This report describes (1) revisions to the Small Business Procurement Scorecard (scorecard) methodology for fiscal year 2017 and results of the fiscal year 2017 scorecard, as well as the extent to which the Small Business Administration (SBA) plans to evaluate the effects of revisions; (2) the extent to which SBA’s revised scorecard methodology uses relevant and reliable information and SBA publishes accurate scorecards; and (3) views of selected federal agencies and industry stakeholders on the extent to which SBA’s revised scorecard methodology may encourage agencies to expand small business procurement opportunities. To examine the changes SBA made to the Small Business Procurement Scorecard and the rationale for these changes, we reviewed relevant documents, including the National Defense Authorization Act for Fiscal Year 2016, SBA’s descriptions of the prior and revised scorecard methodology, and revised peer review guidance used for the scorecard element that assesses compliance with section 15(k) of the Small Business Act. We also interviewed officials from SBA and four other agencies about the revisions to the scorecard calculation methodology, the peer review guidance, the process for providing input on scorecard revisions, and how revisions were implemented. The four agencies (the Departments of Agriculture, Defense, Energy, and Homeland Security) represented a judgmental, nongeneralizable sample of federal agencies with procurement powers, selected based on small business procurement volume, recent improvement in scorecard results, and level of participation in discussions with SBA and other agencies about potential changes to the scorecard. We also interviewed SBA officials about their plans to evaluate the effects of scorecard revisions on small business procurement opportunities and about their plans, if any, to evaluate the revised scorecard. In addition, we reviewed federal internal control standards and GAO’s key attributes for designing effective evaluations. We analyzed the distribution of agencies’ letter grade results (A+, A, B, C, D, and F) from the fiscal year 2017 scorecard and compared this distribution to fiscal years 2014 through 2016, which used a different scorecard methodology. We also reviewed the distribution of results of fiscal year 2017 individual scorecard elements—specifically, results of prime contracting achievement, subcontracting achievement, and peer reviews—and compared this distribution to results for fiscal year 2016. We compared agencies’ prime contracting and subcontracting performance against their small business procurement goals for fiscal years 2016 and 2017. To compare peer review results across years, we made adjustments to account for changes in the value of peer review results (raised from 10 points to 20 points from fiscal years 2016 to 2017). To adjust for this difference, we doubled the value of fiscal year 2016 scores to put both years’ scores on a 20-point scale. Finally, we compared actual fiscal year 2017 scorecard results to the results if SBA had used the 2016 scorecard weighting. To do this, we increased the weighting of fiscal year 2017 prime contracting results from 50 percent to 80 percent of each agency’s total scorecard grade, decreased the weight of subcontracting results from 20 percent to 10 percent, and decreased the weight of peer review results from 20 percent to 10 percent. We also excluded results from the new scorecard element calculating changes in the number of small business contractors, which was not part of the 2016 methodology. To examine the extent to which SBA’s revised scorecard methodology considers relevant and reliable information, we interviewed officials from SBA and the Departments of Agriculture, Defense, Energy, and Homeland Security. We reviewed documents describing the prior and revised scorecard methodology. We discussed limitations, if any, in the electronic data systems that capture government-wide data on prime contracting and subcontracting (which SBA uses to calculate those respective scorecard elements). We also reviewed our prior work that assessed these data systems. To assess the data reliability of the published scorecards, we reviewed them for obvious errors and interviewed SBA officials about the cause of errors we identified. We found the scorecards to be reliable for analyzing scorecard results for fiscal year 2017. We also compared SBA’s revised scorecard methodology against the agency’s policies on information quality and against GAO’s standards for internal control in the federal government. To collect views on the extent to which SBA’s revised scorecard methodology may encourage agencies to expand small business procurement opportunities, we interviewed officials from SBA and the four selected departments cited above, as well as representatives from three organizations representing the interests of small businesses. These three organizations were selected to represent a mix of small business types: one (The American Small Business Chamber of Commerce) represented all types of small businesses; one (Women Impacting Public Policy) represented a small business socioeconomic subcategory with a 5 percent goal for prime contracting and subcontracting (as a percentage of total prime contracting and subcontracting); and one (The Task Force for Veterans’ Entrepreneurship, also known as Vet-Force) represented a small business subcategory with a 3 percent goal for prime contracting and subcontracting. We conducted this performance audit from January 2018 to September 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Comments from the Small Business Administration Appendix III: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Andy Pauline (Assistant Director), Steve Robblee (Analyst in Charge), William Chatlos, Holly Hobbs, Marc Molino, Jessica Sandler, and Jennifer Schwartz made key contributions to this report.
target: Each year SBA produces a scorecard measuring federal contract spending allocated to small businesses. The 2016 NDAA included a provision for SBA to revise the scorecard's methodology and for GAO to evaluate the effects of those revisions for fiscal year 2017. This report discusses, among other things, (1) SBA's changes to the scorecard methodology and plans, if any, to evaluate the effects of these changes, (2) the extent to which SBA has processes to disseminate reliable information, and (3) views of selected stakeholders on the scorecard's effects on small business procurement opportunities. GAO analyzed SBA's prior and revised scorecard methodology and results and interviewed officials from SBA, four other federal agencies selected based on small business procurement volume and other attributes, and three groups representing the interests of small businesses. For fiscal year 2017, the Small Business Administration (SBA) revised the methodology for its Small Business Procurement Scorecard, which is used to assess federal agencies' progress toward small business procurement goals. SBA made revisions to address requirements specified in the National Defense Authorization Act for Fiscal Year 2016 (2016 NDAA). SBA (1) reduced the share of the total scorecard grade devoted to prime contracting achievement, which is the dollar amount of contracts awarded directly to small businesses, and (2) added an element calculating changes in the number of small businesses receiving prime contracts. SBA made two additional revisions—with input from other agencies' representatives—to increase the share of subcontracting achievement results and peer review of required activities designed to facilitate small business procurement (see figure). In July 2018, officials said they had begun developing a plan to evaluate the effects of the revised scorecard methodology but did not provide a draft plan. Conducting a well-designed and comprehensive evaluation could aid SBA in determining whether the scorecard is an effective tool for helping to achieve the agency's strategic goals. (Scorecard elements are expressed as a percentage of total scorecard grade.) The published fiscal year 2017 scorecards originally contained errors, including an incorrect grade and numeric score for one agency, and SBA does not have a process to ensure that scorecard results are published accurately. Although SBA later corrected the errors, the agency did not initially document that scorecards had been changed, which is inconsistent with SBA's policy on information quality. SBA officials said that errors occurred in the process of formatting scorecards for publication. Errors in the published scorecards—and the initial lack of disclosure about corrections—weaken data reliability and may undermine confidence in scorecard data. Agency officials and representatives of small business groups that GAO interviewed generally expected the scorecard revisions to have little impact on small business procurement opportunities. However, one agency's officials said they would focus more on tracking subcontracting activity as a result of changes to the scorecard.