Small Business Research Programs: Most Agencies Met Spending Requirements, but DOD and EPA Need to Improve Data Reporting
Published: 2017-05-31T00:00:00

Fast Facts:
Federal agencies fund research and development projects at small businesses with the goal of helping them develop and commercialize innovative technologies.
The two small business R&D programs we looked at require participating agencies to spend a certain percentage of their external research funding on the programs. Most agencies' spending data showed that they met these requirements.
However, we couldn't determine if the Department of Defense or the Environmental Protection Agency met the requirements because they did not collect or submit the data. We recommended that DOD and EPA establish procedures to collect and submit their data.

Highlights:
section_title: Why GAO Did This Study
paragraphs: Through the SBIR and STTR programs, federal agencies have awarded contracts and grants totaling about $44 billion to small businesses to develop and commercialize innovative technologies. The Small Business Act requires agencies with extramural R&D obligations that meet certain thresholds—$100 million for SBIR and $1 billion for STTR—to spend a percentage of these funds on the programs. The agencies are to report on their activities to SBA and, in turn, SBA is to report to Congress.
The 2011 reauthorization of the programs includes a provision for GAO to review agencies' compliance with SBIR and STTR spending and reporting requirements and other program aspects. This report examines, for fiscal year 2015, (1) the extent to which participating agencies met spending requirements, (2) the extent to which agencies and SBA met certain reporting requirements, (3) the potential effects of basing spending requirements on total R&D budget authority instead of on extramural R&D obligations, and (4) what is known about the amounts agencies spent to administer the programs. GAO reviewed agency spending data and reports and interviewed program officials from SBA and the 11 participating agencies.
section_title: What GAO Found
paragraphs: Data submitted to the Small Business Administration (SBA) indicate that most of the agencies participating in the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs met spending requirements for fiscal year 2015. Nine of the 11 agencies participating in the SBIR program and 4 of the 5 agencies participating in the STTR program submitted the required data on obligations for extramural research or research and development (R&D)—which is generally conducted by nonfederal employees outside of federal facilities—needed to determine whether they met spending requirements. Of the agencies that submitted the required obligations data, 8 of 9 met SBIR spending requirements and all 4 met STTR spending requirements for fiscal year 2015. GAO could not determine compliance for 2 agencies—the Department of Defense (DOD) and the Environmental Protection Agency (EPA)—because these agencies did not submit the required obligations data. Officials at most agencies, including DOD and EPA, identified difficulties in using extramural R&D obligations data to calculate spending requirements and determine compliance. A working group of several participating agencies is considering proposing an alternative methodology to calculate spending requirements, according to an agency official involved in leading the effort. Until DOD and EPA find ways to provide obligations data or until the calculation methodology is changed, these 2 agencies will not know whether they have met the spending requirements, which limits SBA's reporting of that information to Congress.
Agencies inconsistently met certain requirements to report to SBA on their methodologies for calculating their extramural R&D budgets in fiscal year 2015, and SBA officials said they are taking steps to expedite SBA's required reporting to Congress. Five of the 11 agencies submitted their required methodology reports on time. Most agencies provided detailed information in these reports, but they did not consistently provide all of the information required. Regarding SBA's own reporting requirements, as of May 2017, SBA had not submitted its report for fiscal year 2015. GAO has previously made recommendations to SBA to address these issues, and SBA is in the process of implementing them.
GAO's analysis shows that basing SBIR and STTR spending requirements on an agency's total R&D budget authority instead of on its extramural R&D obligations for fiscal year 2015 would have increased spending requirements and the number of agencies required to participate in the programs. Officials from each of the participating agencies identified benefits or drawbacks of this potential change. For example, agency officials identified more awards to small businesses as a benefit, but officials from several agencies also stated that such a change could limit resources for other R&D programs.
Total administrative spending for the SBIR and STTR programs for fiscal year 2015 is unknown, but 9 agencies participating in an administrative pilot program—which permits agencies to fund new administrative activities—reported spending about $33.9 million on these activities for that year. Agencies are not required to and generally do not track their total administrative costs, so GAO could not determine total spending.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD and EPA establish procedures to report obligations data or propose to Congress an alternative methodology for calculating spending requirements. DOD and EPA agreed with GAO's recommendation.

section_title: Letter
paragraphs: Federal agencies support research or research and development (R&D)  projects at small businesses through the Small Business Innovation  Research (SBIR) and Small Business Technology Transfer (STTR)  programs. Through these programs, federal agencies awarded about  5,200 contracts and grants, totaling about $2 billion, in 2015 and have  awarded about 159,000 contracts and grants, totaling about $44 billion,  since the programs’ inceptions in 1982 and 1992, respectively. These  awards are intended to support the development and commercialization  of innovative technologies. For example, one award from 2015 was  intended to help a company develop a tool to prevent surgical wound  infections. Another award from 2015 was intended to help a company  identify and evaluate improved materials and designs for components of  aerospace wind tunnel facilities. The SBIR and STTR programs are  similar in that participating agencies identify topics for R&D projects and  support small businesses, but the STTR program requires the small  business to partner with a nonprofit research institution, such as a  nonprofit college or university or a federally funded research and  development center.
Federal agencies with obligations of $100 million or more for extramural  R&D are required to participate in the SBIR program, and federal  agencies with obligations of $1 billion or more for extramural R&D are  required to also participate in the STTR program. Currently, 11 agencies  participate in the SBIR program, and 5 of these agencies also participate  in the STTR program, as shown in table 1.
The Small Business Act, which authorizes the SBIR and STTR programs,  establishes the minimum percentage of an agency’s extramural R&D  obligations that it must spend on the programs annually. In fiscal year  2015, agencies participating in the SBIR program were required to spend  at least 2.9 percent of their extramural R&D obligations on the program,  and agencies participating in the STTR program were required to spend  at least 0.40 percent of their extramural R&D obligations on the program.  The SBIR and STTR policy directives, which provide direction to the  participating agencies for the general operation of the programs, require  participating agencies to submit data to the Small Business  Administration (SBA) each year on the amount of their extramural R&D  obligations and the amount they obligate for SBIR and STTR awards,  among other information.
The Small Business Act also establishes certain reporting requirements  for participating agencies and SBA. Among other things, within 4 months  of the enactment of agencies’ annual appropriations, the agencies must  report to SBA on their methodologies for calculating their extramural R&D  obligations. Furthermore, SBA must annually report to Congress on the  participating agencies’ SBIR and STTR programs.
Both programs were reauthorized in 2011. The SBIR/STTR  Reauthorization Act of 2011 (reauthorization act) directed SBA to allow  agencies to participate in a pilot program, known as the administrative  pilot program, that permitted the funding of administrative and certain  other costs in fiscal years 2013 through 2015. This administrative pilot  program, as outlined in the reauthorization act and policy directives,  allows agencies to use not more than 3 percent of the funding allocated to  the SBIR program for new activities, including program administration;  outreach; commercialization; standardization and simplification of  program procedures; prevention of waste, fraud, and abuse; and  reporting. The SBIR and STTR policy directives specifically note that  funding under the pilot program may not replace current agency  administrative funding for SBIR or STTR activities. Instead, the  administrative pilot program is intended to supplement the agency’s  existing administrative efforts. In November 2015, the National Defense  Authorization Act for Fiscal Year 2016 extended the pilot program through  September 30, 2017. In December 2016, the National Defense  Authorization Act for Fiscal Year 2017 extended the SBIR and STTR  programs through fiscal year 2022 but did not extend the pilot program.
The reauthorization act includes a provision for us to annually review the  participating agencies’ compliance with spending and reporting  requirements for the programs, as well as other aspects of the programs.  We issued four reports from September 2013 through May 2016 in  response to this provision. This report covers fiscal year 2015 and  addresses (1) the extent to which participating agencies met SBIR and  STTR program spending requirements, (2) the extent to which  participating agencies and SBA met certain program reporting  requirements, (3) the potential effects of basing the spending  requirements for the programs on an agency’s total R&D budget authority  instead of on its extramural R&D obligations, and (4) what is known  about the amounts agencies spent to administer the programs.
To accomplish these objectives, we generally followed the methodology  that we used for our previous four reports on these issues. Specifically, to  examine the extent to which participating agencies met SBIR and STTR  program spending requirements in fiscal year 2015, we used the data that  agencies submitted to SBA and calculated each agency’s spending  requirement by applying the mandated percentages to the agency’s  reported extramural R&D obligations for fiscal year 2015. We then  compared the spending requirements we calculated with the total  program obligations data the agency submitted to SBA for fiscal year  2015. We reported that an agency met its spending requirement if the  agency’s spending for these programs was greater than or equal to the  spending requirement we calculated. To assess the reliability of the  data, we interviewed agency officials about the sources and quality  control procedures for the data; examined the integrity of the data, which  included looking for outliers and obvious errors; and reviewed relevant  documentation. We found the data to be sufficiently reliable for the  purposes of assessing whether agencies met their spending  requirements. In cases in which agencies did not submit extramural R&D  obligations data, we interviewed officials from those agencies and SBA  about the data submitted to identify the reasons an agency did not  provide the required data. In these cases, we determined the data these  agencies submitted could not be used to determine whether these  agencies met their spending requirements. We also discussed agencies’  compliance with the spending requirements—including reasons agencies  did not meet the spending requirements—with program managers at each  of the participating agencies.
To examine the extent to which participating agencies and SBA met  certain program reporting requirements for fiscal year 2015, we compared  information in the reports that agencies submitted to SBA to describe the  methodology for calculating their extramural R&D obligations, referred to  as their methodology reports, with requirements in the Small Business Act  and program guidance. We reported that a participating agency met its  reporting requirements if the agency submitted a methodology report to  SBA, submitted the report on time (within 4 months of receiving  appropriations), itemized exclusions from its extramural R&D obligations,  and explained those exclusions. We also examined whether the  methodology reports included other information requested by SBA.  Additionally,  we discussed the status of SBA’s required reports to  Congress with SBA program officials.
To examine the potential effects of basing spending requirements for the  SBIR and STTR programs on an agency’s total R&D budget authority  instead of on its extramural R&D obligations, we used fiscal year 2015  data on total R&D budget authority as reported by the Office of  Management and Budget. We used these data to calculate potential  spending requirements for each federal agency under alternate  scenarios, assuming that the same spending percentages currently  required by the Small Business Act would apply to total R&D budget  authority. We also assumed that current spending thresholds that, under  the current law, require agencies to participate in SBIR and STTR  programs would apply to the total R&D budget authority. As in our  previous reports, we compared the spending requirements from the  alternate scenarios with requirements under current law to determine the  potential effects that changing the methodology for calculating spending  requirements would have had in fiscal year 2015. We also analyzed  additional scenarios that used smaller percentages than those currently  required by the Small Business Act to determine whether they would lead  to different outcomes. We spoke to program managers at all participating  agencies about how potential changes to the calculation methodology  could affect their programs.
To examine what is known about how much agencies spent to administer  the SBIR and STTR programs in fiscal year 2015, we requested and  analyzed administrative spending data from the 11 participating agencies  and spending data associated with the administrative pilot program from  the 9 agencies that participated in the pilot. We discussed these data  with officials and determined that the administrative spending data for the  participating agencies were incomplete and came from such varied  sources that we could not calculate the total amount the agencies spent  to administer their programs. We also reviewed the total amounts the 9  agencies had estimated they would spend on the administrative pilot  program in fiscal year 2015 as described in their proposals for the  program; we then compared those amounts with the amounts the  agencies reported spending. Further, we reviewed how the agencies  used administrative pilot program funds to meet program goals. To  assess the reliability of the data associated with the administrative pilot  program, we discussed the completeness and accuracy of the data with  officials from the 9 agencies that participated in the program. We  determined that the data were sufficiently reliable for the purposes of  determining the amounts that agencies spent on the administrative pilot  program.
We conducted this performance audit from July 2016 to May 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The SBIR program was initiated in 1982 and has four main purposes: (1)  use small businesses to meet federal R&D needs, (2) stimulate  technological innovation, (3) increase commercialization of innovations  derived from federal R&D efforts, and (4) encourage participation in  technological innovation by small businesses owned by women and  disadvantaged individuals. The STTR program was initiated a decade  later, in 1992, and has three main purposes: (1) stimulate technological  innovation, (2) foster technology transfer through cooperative R&D  between small businesses and research institutions, and (3) increase  private sector commercialization of innovations derived from federal R&D.  Legislation enacted in 2011 reauthorized the programs for fiscal years  2012 through 2017, and legislation enacted in 2016 extended the  programs for fiscal years 2018 through 2022.
The SBIR and STTR programs each include three phases:  In phase I, agencies make awards to small businesses to determine  the scientific and technical merit and feasibility of ideas that appear to  have commercial potential. Phase I awards normally do not exceed  $150,000. For SBIR, phase I awards generally last up to 6 months.  For STTR, these awards generally last 1 year.
In phase II,  small businesses with phase I projects that demonstrate  scientific and technical merit and feasibility, in addition to commercial  potential, may compete for awards of up to $1 million to continue the  R&D for an additional period, normally not to exceed 2 years.
In phase III,  small businesses pursue commercialization of technology  developed in prior phases. Phase III  work derives from, extends, or  completes an effort made under prior phases, but it is funded by  sources other than the SBIR and STTR programs. In this phase, small  businesses are expected to raise funds from private investors, the  capital markets, or funding sources within the agency that made the  initial award other than the agency’s SBIR or STTR program.  Agencies can also participate in phase III  by, for example, purchasing  the technology developed in prior phases.
SBA’s Office of Investment and Innovation  is responsible for overseeing  and coordinating the participating agencies’ efforts for the SBIR and  STTR programs. As part of SBA’s oversight and coordination role, the  agency issued SBIR and STTR policy directives to explain and outline  requirements for agencies’ implementation of these programs. The  policy directives include a list of the data that agencies must submit to  SBA annually, such as their extramural R&D obligation amounts and the  amounts they obligated for awards for their SBIR and STTR programs.
Each participating agency must manage its SBIR and STTR programs in  accordance with program laws and regulations and the policy directives  issued by SBA. In general, the programs are similar across participating  agencies. All of the participating agencies follow the same general  process to obtain proposals from and make awards to small businesses  for both the SBIR and STTR programs. However, each participating  agency has considerable flexibility  in designing and managing specific  aspects of these programs, such as determining research topics,  selecting award recipients, and administering funding agreements. At  least annually, each participating agency issues a solicitation requesting  proposals for projects in topic areas determined by the agency. Each  participating agency uses its own process to review proposals and  determine which proposals should receive awards. The agencies that participate in both SBIR and STTR programs usually use the same  process for both programs. Also, each participating agency determines  whether to provide the funding for awards as grants or contracts.
section_title: Agency Data Indicate Most Participating Agencies Met Spending Requirements, but Not All Submitted the Required Data
paragraphs: Agency data indicate that most of the agencies participating in the SBIR  and STTR programs met spending requirements for fiscal year 2015.  Nine of the 11 agencies that participate in the SBIR program and 4 of the  5 agencies that also participate in the STTR program submitted the  extramural R&D obligations data required to determine compliance with  spending requirements. For 2 of the 11 agencies, we could not determine  compliance with the spending requirements because these agencies did  not submit required obligations data to SBA. Of the 9 agencies that  submitted the required data, 8 met their SBIR spending requirements,  and all of the 4 that also participate in the STTR program met their  spending requirements for fiscal year 2015.
subsections: section_title: Most Agencies Submitted the Obligations Data Required to Determine Whether Spending Requirements Were Met, But Two Agencies Did Not Collect and Submit These Data
paragraphs: Nine of the 11 agencies participating in the SBIR program and 4 of the 5  agencies also participating in the STTR program submitted to SBA the  extramural R&D obligations data required to determine whether they met  the respective program spending requirements for fiscal year 2015. The  Small Business Act requires agencies to spend a minimum percentage of  their extramural R&D obligations on the SBIR and STTR programs  annually. Determining whether an agency met the minimum spending  requirement for its SBIR or STTR program requires data on the agency’s  obligations for the program as well as its total extramural R&D  obligations. All participating agencies reported obligations data to SBA for  their spending on the programs for fiscal year 2015. However, 2  agencies—the Department of Defense (DOD) and Environmental  Protection Agency (EPA)—did not report total extramural R&D obligations  data; instead, they reported budget authority amounts for extramural  R&D.
According to a SBA report and officials, SBA has been working with  agencies for several years to clarify that they should be reporting  extramural R&D obligations data to SBA. In our June 2014 report  covering SBIR and STTR spending for fiscal year 2012, we  recommended that SBA provide additional guidance to agencies about  submitting data and calculating spending requirements. In response to our recommendation, SBA provided guidance in the template that  agencies used to report data for fiscal year 2013; this guidance directed  agencies to report actual obligations for both extramural R&D and the  SBIR and STTR programs. SBA’s reporting template for fiscal year 2015  continued this effort by providing the definition of total extramural R&D  obligations and asking agencies to confirm that the amount reported was  the amount actually obligated. SBA’s efforts to clarify the required data  have improved agencies’ reporting. In fiscal year 2013, only 2 of the 11  agencies reported extramural R&D obligations data to SBA. This number  increased to 8 agencies in fiscal year 2014 and 9 agencies in fiscal year  2015.
Officials from DOD and EPA told us their agencies did not report  extramural R&D obligations for fiscal year 2015 because their program  offices have no procedures to capture total extramural R&D obligations  for a particular fiscal year. Moreover, they said they have no plans to  establish such procedures, and establishing them would be challenging.  For example, EPA officials told us that their agency’s financial system  does not distinguish extramural from intramural R&D funds, and that the  agency’s multi-year appropriations, which allow funds to be obligated over  2 years, make it hard to know in which fiscal year the extramural R&D  funding was obligated. DOD officials told us that obtaining obligations  data would require requesting information from more than 10 individual  program offices that, in turn, would have to request the information from  various DOD comptrollers, which would be a major effort. In contrast, all  of the other agencies reported fiscal year 2015 obligations.
Officials at most agencies, including DOD and EPA, identified difficulties  in using extramural R&D obligations data to determine spending  requirements. As we also reported in May 2016, officials from several  agencies told us that they believe it is unfair or impractical to hold their  agencies to a requirement based on obligations, which are not known  until after the end of the fiscal year, when it is not possible to obligate  additional money for the SBIR and STTR programs. Because of these  difficulties, a working group of several participating agencies is  considering proposing a recommendation to base spending requirements  on extramural R&D budget authority, according to a Department of  Energy (DOE) official involved in leading the effort. If Congress were to  take up such a proposal, it might eliminate the data collection challenges  that some agencies have identified.
Until DOD and EPA find ways to provide the required data, or the  spending requirements are changed and the identified data collection  challenges are resolved, DOD and EPA will not know whether they have  met the spending requirements. Further, without DOD and EPA reporting  the required data, SBA will continue to be limited in its ability to accurately  report that information to Congress, as we previously found.
section_title: Agency Data Indicate Most Agencies Met SBIR and STTR Spending Requirements, and Some Have Developed Practices to Help Meet Them
paragraphs: Eight of the 9 agencies that submitted the required extramural R&D  obligations data met or exceeded their SBIR spending requirements for  fiscal year 2015, according to the agencies’ data (see fig 1). Those 8  agencies spent from 2.9 percent to 5.6 percent of their extramural R&D  obligations on the program. The remaining agency, the U.S. Department  of Agriculture (USDA), spent 2.5 percent—less than the required 2.9  percent. We could not determine compliance with spending requirements  for DOD and EPA because, as previously discussed, those agencies did  not submit the required obligations data. However, the data those 2  agencies submitted indicate that DOD spent $956.9 million on the SBIR  program, while EPA spent $4.7 million on the program. Appendix I  provides more details on agencies’ SBIR spending and spending  requirements.
In our September 2013 report, we recommended that SBA direct  participating agencies to include in their annual reports if they did not  meet the spending requirements, the reasons why they did not, and how  they planned to meet the spending requirements in the future. SBA  updated the annual report template for fiscal year 2015 data to request  such information from agencies that did not meet the spending  requirements. Given that this change may have improved agencies’  compliance with spending requirements for fiscal year 2016, we are not  making a new recommendation at this time.
Of the 5 agencies that also participated in STTR, 4 agencies submitted  the required obligations data, and all 4 met their STTR spending  requirements for fiscal year 2015, according to the agencies’ data (see  fig. 2). According to the agencies’ data, these 4 agencies spent from  0.40 percent to 0.45 percent of their extramural R&D obligations on their  STTR programs, meeting or exceeding the required minimum spending  requirement of 0.40 percent. We could not determine compliance with the  spending requirement for the fifth participating agency—DOD—because it  did not submit the required extramural R&D obligations data. However,  DOD’s submitted data shows that DOD spent $125.5 million on the STTR  program for fiscal year 2015. Appendix II provides more details on  agencies’ STTR spending and spending requirements.
Officials, including those from agencies that met their SBIR and STTR  spending requirements and those from the agency that did not, cited a  number of potential difficulties with meeting the requirements, including:
Minimum spending requirement is unknown until the end of the fiscal  year. Consistent with our previous reviews, agency officials said that it  is difficult to meet the minimum spending requirements because the  spending requirements are based on extramural R&D obligations,  which are not known until after the end of the fiscal year. If an  agency’s actual extramural R&D obligations at the end of the fiscal  year turn out to be higher than estimated, the agency’s estimate of  spending requirements for the programs may turn out to be too low,  according to some agency officials. For example, USDA officials told  us that their agency estimated total extramural R&D spending for  fiscal year 2015 would be $697 million, which set their SBIR program  budget at $20.2 million. USDA data submitted to SBA shows that  USDA spent $22.7 million, which was more than the amount  estimated earlier in the year. However, some agencies, including  USDA, receive multi-year appropriations, which generally may be  carried forward from one fiscal year to the next. According to the  USDA officials, other programs that provide extramural R&D funding  carried over about $240 million from fiscal year 2014, which increased  extramural R&D obligations in fiscal year 2015 and thus increased the  agency’s SBIR spending requirement by about $6 million beyond the  amount officials had estimated based on 2015 appropriations. As a  result, USDA did not meet the spending requirement for fiscal year  2015 even though, according to USDA program managers, the  agency spent the amount it budgeted for the SBIR program.
Agency cannot always issue all expected awards. Agency officials  said that they cannot always make all of the awards that they  anticipated at the beginning of the fiscal year. For example, officials at  DOE and the Department of Commerce (Commerce) told us of  instances when, late in the award process, a small business declined  an award or was found to be ineligible to receive the award and the  agency was not able to make an award to another small business  before the end of the fiscal year, hindering its ability to meet the  minimum spending requirements.
Agency carries over program funds to the next fiscal year. According  to some agency officials, agencies do not always spend their SBIR or  STTR funding in the fiscal year in which it is appropriated for various  reasons. For example, DOD officials told us that circumstances such  as receiving agency appropriations later in the fiscal year can delay  spending, as we have previously reported. This can make it difficult  to meet the spending requirement for that year. USDA officials  explained that they may choose to spend their SBIR funds over  multiple fiscal years to help spend the funds properly and efficiently.  However, DOE officials told us that if an agency carries over SBIR or  STTR funds into the next fiscal year, it might not spend enough to  meet the spending requirement in the current fiscal year, although the  carried-over funds may help the agency meet or exceed the spending  requirement in the following year.
Despite the difficulties cited above, officials at most agencies told us that  they had developed practices to help ensure they meet the minimum  spending requirements. Those practices include:
Budget more than the minimum estimated spending requirement.
Commerce officials told us that in fiscal year 2015, one of their  subunits estimated its SBIR budget using a higher percentage than  the minimum spending requirement. According to agency officials, this  helps ensure that the subunit meets the spending requirement, even if  actual extramural R&D obligations exceed the amount estimated.  DOE officials told us that they estimated the required SBIR budget  based on the statutory minimum percentage but used some  assumptions that led to an estimate of the agency’s total extramural  R&D obligations that exceeded the actual obligation amount, which  helped DOE exceed its minimum spending requirement in fiscal year  2015. Department of Education (Education) officials told us they  budget a set amount each year, $7.5 million, which has been higher  than their minimum SBIR spending requirement.
Spend funds beyond the minimum estimated spending requirement  budgeted. Department of Homeland Security (DHS) and Department  of Health and Human Services (HHS) officials told us their agencies  generally spend funds beyond the minimum spending requirements  for which they budgeted. EPA officials told us that, in addition to funds  designated for SBIR awards, EPA provides annual administrative  funding that typically exceeds the actual administrative costs for the  agency’s SBIR program. As a result, EPA can potentially fund an  additional SBIR award with the excess funding.
Spend program funds across agency subunits. HHS officials told us  that agency subunits independently spend SBIR funds. However, if  one subunit realizes that it cannot spend all of its SBIR funds, HHS  officials identify a meritorious application to award from another  subunit, which enables HHS to spend all the funds in that fiscal year.
Spend multi-year funds in the year appropriated. Officials from EPA  and National Science Foundation (NSF) said that they try to spend all  of their multi-year funds for SBIR in the same year that they are  appropriated to better ensure that they meet the minimum spending  requirement.
section_title: Participating Agencies Inconsistently Met Certain Reporting Requirements, and SBA Officials Report Having Taken Steps to Expedite Its Reporting to Congress
paragraphs: All of the agencies participating in the SBIR and STTR programs  submitted methodology reports to SBA for fiscal year 2015, and about  half of these agencies submitted them on time. While most agencies  provided detailed information in their methodology reports, they did not  consistently provide all of the information required by SBA’s policy  directive. As of May 2017, SBA had not submitted its annual report on the  programs to Congress for fiscal year 2015, but SBA officials said they  have taken steps to expedite reporting.
subsections: section_title: About Half of the Agencies Submitted Their Methodology Reports on Time, but Agencies Did Not Consistently Provide Required Information
paragraphs: All 11 participating agencies submitted their required methodology reports  to SBA for fiscal year 2015, and about half of these agencies submitted  their reports on time. According to the Small Business Act, agencies must  submit to SBA a report describing the methodology they use to calculate  their extramural R&D budgets, and the submission must occur within 4  months of enactment of their annual appropriations. Most participating  agencies’ appropriations for fiscal year 2015 were enacted in mid- December 2014, so their methodology reports were due by mid-April  2015. Five agencies—DHS, DOE, EPA, HHS, and the National  Aeronautics and Space Administration (NASA)—submitted their required  methodology reports on time in fiscal year 2015. More agencies provided timely submissions in fiscal year 2015 than in fiscal year 2014,  when only 2 of the 11 agencies submitted their methodology reports on  time.
Those agencies that submitted their methodology reports late generally  did so only after SBA sent an e-mail in August 2015 requesting that the  agencies submit their reports. Officials at Education, USDA, Commerce,  and the Department of Transportation (DOT) told us that they had not  known that they needed to submit the methodology report or had not  known the deadline. In contrast, officials from the agencies that reported  on time told us that they have routine procedures to submit the  methodology report each year.
As we found in our September 2013 report, the late submission of the  methodology reports makes it difficult for SBA to promptly analyze the  methodologies and provide agencies with timely feedback to assist them  with accurately calculating their spending requirements. Without such  review and feedback, agencies may be calculating their extramural R&D  obligations incorrectly, which could lead to agencies spending less than  the required amounts on the SBIR and STTR programs. In our June 2014  report, we recommended that SBA request that the agencies submit their  methodology reports within 4 months of the enactment of appropriations.  SBA generally agreed with the recommendation, and SBA officials sent  an e-mail to participating agencies in February 2016 requesting that they  submit their fiscal year 2016 methodology reports on time. Given that  SBA’s request may have improved the timeliness of agencies’  submissions for fiscal year 2016, we are not making a new  recommendation on requesting the methodology reports at this time.
While most agencies provided detailed information in their methodology  reports, they did not consistently provide all of the information required by  SBA. According to the program policy directives, in addition to describing  their methodology for calculating their extramural R&D budgets, agencies  are to itemize the programs they exclude from their extramural R&D  calculations and the reasons for the exclusions. For fiscal year 2015, all  11 participating agencies submitted descriptions of their methodologies, 9  agencies at least partially itemized exclusions from their calculations of  extramural R&D, and 8 agencies at least partially explained the reasons  for these exclusions.
Further, where agencies did provide the required information, they did so  using different levels of detail. For example, 5 of the 11 agencies provided  dollar amounts for total, intramural, and extramural R&D budgets. EPA  and Commerce officials told us that providing such information reflects  guidance that SBA had provided in prior years. Six agencies provided the  dollar amounts of their planned SBIR funding. This funding information  was explicitly requested in SBA’s August 2015 e-mail, but SBA sent this  guidance to only those agencies that had not yet submitted their reports.
In our April 2015 report, we concluded that the varying detail in agencies’  methodology reports makes it difficult for SBA to analyze the reports and  provide accurate information to Congress, and we recommended that  SBA assess the methodology reporting requirement and revise its  guidance, if necessary, to require adequate information. SBA included  updated reporting requirements in a proposed policy directive. If  implemented, the language in the proposed policy directive would expand  the required information to include the planned amount of SBIR funding,  which SBA officials said would make the reporting more useful for SBA’s  analysis of agencies’ program funding. SBA’s proposed policy directive was not in force for fiscal year 2015, so we could not evaluate the effect  of this guidance on methodology reporting, and we are not making  additional recommendations at this time.
section_title: SBA Has Not Submitted Its Required Report to Congress for Fiscal Year 2015, but Officials Said Steps Have Been Taken to Expedite Reporting
paragraphs: The Small Business Act requires SBA to report to certain congressional  committees on the SBIR and STTR programs—including an analysis of  the agencies’ methodology reports—not less than annually, but the act  does not specify a date when the report is due. SBA issued its report to  Congress on the SBIR and STTR programs for fiscal year 2013 in March  2016, but it has not submitted its report for fiscal years 2014 or 2015.
In our September 2013 report, we concluded that without more rigorous  oversight by SBA and its more timely and detailed reporting, it would be  difficult for SBA to ensure that intended benefits of the SBIR and STTR  programs are being attained and that Congress is receiving critical  information to oversee these programs. In that report, we recommended  that SBA provide Congress with a timely annual report. SBA agreed and  stated that it planned to implement the recommendation.
However, SBA has not yet done so, and as we found previously and in  this review, it continues to lag in its submissions to Congress. SBA issued  its fiscal year 2012 report to Congress in November 2014, 19 months  after the agencies’ data were due to SBA. SBA issued its fiscal year 2013  report to Congress in March 2016, 2 years after the agencies’ data were  due to SBA. As of May 2017, SBA was continuing to develop the reports  for fiscal years 2014 and 2015, about 26 months and 14 months after the  agencies’ data were due to SBA, respectively. SBA officials said that they  have not set issuance dates for either the fiscal year 2014 or fiscal year  2015 report to Congress, in part because it is difficult to predict the  completion of an Office of Management and Budget review that is not in  their control. Additionally,  according to these officials, report production  has been slowed by SBA staff leaving the agency. Given that SBA  officials told us that they are working to respond to our previous  recommendation by taking steps to expedite the submission of the  reports, such as by hiring contractors to help prepare the reports, we are  not making additional recommendations at this time.
section_title: Basing Spending Requirements on Total R&D Budget Authority Would Have Increased Spending Requirements and Participation with Potential Benefits and Drawbacks
paragraphs: Changing the methodology for calculating SBIR and STTR spending  requirements to use total R&D budget authority rather than extramural  R&D obligations would have increased agencies’ spending requirements  and the number of agencies required to participate in the programs for  fiscal year 2015. Agency officials identified several benefits and  drawbacks that changing the calculation methodology could have for their  agencies’ SBIR and STTR programs.
subsections: section_title: Changing the Calculation Methodology to Total R&D Budget Authority Would Have Increased Spending Requirements and Participation in the Programs
paragraphs: Changing the methodology for calculating SBIR and STTR spending  requirements to use an agency’s total R&D budget authority rather than  its extramural R&D obligations would have increased spending  requirements for fiscal year 2015. Specifically, we estimated that the total  spending requirements for the SBIR and STTR programs in fiscal year  2015 would have increased from $2.5 billion to $4.4 billion, or by 76  percent, assuming the same percentages and participation thresholds  defined in current law. The increase would have occurred for two  reasons: (1) an agency’s total R&D budget authority is larger than its  extramural R&D obligations, and therefore agencies that currently  participate would have been required to spend more on the programs,  and (2) additional agencies would have been required to participate.  Figure 3 shows the effects at each agency of changing the calculation  methodology from current law, which is based on a percentage of  extramural R&D obligations, to an alternative scenario that applies the  same percentages to total R&D budget authority. These effects are  consistent with our findings in previous reports.
As shown in figure 3, some agencies’ spending requirements would have  increased more than others under the alternative scenario. This variation  is due primarily to differences in the relative proportions of the agencies’  extramural and intramural R&D obligations, but it is also affected by how  many programs are excluded by statute. Under the alternative scenario,  agencies that fund primarily extramural research would have seen smaller  increases to their spending requirements for the SBIR and STTR  programs, while agencies that fund more intramural research would have  seen larger increases in their spending requirements, a finding consistent  with those of our previous reports. Examples are as follows:
NSF used 96.4 percent of its total R&D budget authority to fund  extramural research in fiscal year 2015 and was required, based on  data it submitted to SBA, to spend $156 million on its SBIR program  that year. Under the alternative scenario, NSF’s SBIR spending  requirement would have been $162 million, an increase of about 4  percent.
Commerce, on the other hand, used 24.5 percent of its total R&D  budget authority to fund extramural research in fiscal year 2015 and  was required to spend about $9 million on its SBIR program that year.  Under the alternative scenario, Commerce’s spending requirement  would have more than quadrupled to about $37 million. Furthermore,  assuming that the thresholds for participating in the program did not  change, this scenario would have required Commerce to spend  approximately $5 million on a new STTR program in fiscal year 2015.  The alternative scenario would have required Commerce to spend  about $33 million more on the SBIR and STTR programs in fiscal year  2015, an increase of more than 360 percent.
As noted above, changing the calculation methodology from basing the  spending requirement on extramural R&D obligations to basing it on total  R&D budget authority would also require additional agencies to  participate in the SBIR and STTR programs, assuming that the dollar  thresholds for participation remain the same. Under the alternative  scenario, 3 additional agencies—the Department of Veterans Affairs, the  Department of the Interior, and the U.S. Agency for International  Development—would have been required to participate in SBIR during  fiscal year 2015. Adding these agencies to the SBIR program would  have increased total SBIR spending requirements by $65 million, in  addition to the $1.6 billion increase in spending requirements at the 11  agencies that currently participate in the SBIR program. Likewise, under  the alternative scenario, 3 additional agencies—USDA, Commerce, and  the Department of Veterans Affairs—would have been required to  participate in STTR. Adding these agencies to the STTR program would  have increased total STTR spending requirements by $19 million, in  addition to the $210 million increase in spending requirements at the 5  agencies that currently participate in STTR.
Alternatively,  basing the SBIR and STTR spending requirements on an  agency’s total R&D budget authority and applying a lower percentage  than required under current law could result in total spending  requirements similar to those under current law. However, our analysis  shows that such a scenario would lower spending requirements at some  agencies and raise them at others. As shown in figure 4, if the percentage  applied to an agency’s total R&D budget authority had been 1.65 percent  for the SBIR program and 0.23 percent for the STTR program in fiscal  year 2015, and the thresholds for participation had remained the same,  total required spending on the programs would be similar to required  spending under current law. Using these lower percentages, spending  requirements would have increased at agencies that primarily fund  intramural research, such as EPA and Commerce. In contrast, spending requirements would decrease at agencies that primarily fund extramural  research, such as HHS and NSF. In this scenario, the spending  requirement reductions, including $203.7 million at HHS and $63.8 million  at NSF, were large enough to offset increases in spending requirements  at other agencies.
section_title: Agency Officials Identified Potential Benefits and Drawbacks to Changing the Calculation Methodology to Total R&D Budget Authority
paragraphs: Officials at each of the participating agencies identified potential benefits  or drawbacks to changing the calculation methodology for their SBIR and  STTR spending requirements to use total R&D budget authority rather  than extramural R&D obligations. For example, agency officials told us  that changing the calculation method could simplify how the spending  requirement is determined, increase the number and dollar amount of  SBIR and STTR awards, and speed up disbursements of funds to  program offices. However, officials also told us that the change could also  have drawbacks for their agencies. First, changing the spending  requirement to total R&D budget authority and keeping the same  percentage requirement could negatively affect other R&D efforts,  including intramural R&D, by shifting resources away from them to the  SBIR and STTR programs, according to officials at several agencies.  Shifting these resources could more broadly undermine their agency’s  ability to accomplish its mission, according to officials from Commerce’s  National Institute of Standards and Technology. For example, these  officials said that work that is done within the agency may need highly  specialized and unique equipment that may not be available to small  businesses. In addition, agency officials at DOD, DOT, and NASA told us  that a significant increase in the spending requirements resulting from the  change could also strain their administration of the SBIR and STTR  programs by requiring additional staffing and resources to manage the  larger programs. As a result, officials at these agencies suggested that if  the calculation methodology is changed to base spending requirements  on total R&D budget authority, the percentages should be lowered to  avoid shifting resources away from other programs and straining the  administration of the SBIR and STTR programs.
section_title: Total Administrative Spending for Fiscal Year 2015 Is Unknown, but Nine Agencies Spent $33.9 Million on the Administrative Pilot Program
paragraphs: Total administrative spending for the SBIR and STTR programs for fiscal  year 2015 is unknown because the agencies that participate in the  programs generally do not—and are not required to—fully track these  costs. The 9 agencies that participated in the administrative pilot program  reported spending $33.9 million in fiscal year 2015, but this amount does  not represent total administrative spending. Officials from all 11 agencies  participating in the SBIR and STTR programs told us that the  administrative pilot program is beneficial, but officials from a few agencies  also told us there are constraints to participating in it.
subsections: section_title: Total Administrative Spending for Fiscal Year 2015 Is Unknown
paragraphs: The total amount that the 11 participating agencies spent to administer  their SBIR and STTR programs for fiscal year 2015 is unknown because  the agencies generally do not—and are not required to—track these  costs. Officials at 9 of these 11 agencies told us that it is challenging to  accurately estimate total administrative costs because staff and  contractors that help administer the program also spend time on other  programs, and the agencies do not have systems in place to track the  time they spend specifically on the SBIR and STTR programs. Officials at  1 of the 2 remaining agencies, Education, told us that it is not challenging  to estimate the expenses because the department has one person  engaged nearly full time with its SBIR program, and it also has estimates  for the hours that staff from other program offices contributed to the SBIR  program. Officials from the other remaining agency, DOT, told us that  they have implemented a system that allows them to track most of their  administrative expenses for the programs, but they acknowledged that  they are not able to track all their expenses, particularly for staff that do  not spend much time on SBIR activities.
In response to our requests for data on their fiscal year 2015  administrative costs, 9 of the 11 participating agencies provided  information on some categories of administrative costs and partial  estimates of costs. These estimates ranged from about $288,000 to  $10.4 million. As with the cost data for fiscal years 2011 through 2014 we  obtained from the agencies for our previous reports, these data were  incomplete and unverifiable.
section_title: Nine Agencies Participated in the Administrative Pilot Program in Fiscal Year 2015 and Spent $33.9 Million
paragraphs: Nine agencies—Commerce, DOD, DOE, DOT, Education, HHS, NASA,  NSF, and USDA—participated in the administrative pilot program in fiscal  year 2015, and these agencies reported spending $33.9 million on  administrative and oversight activities as part of the program. This  represents an increase in the number of participating agencies from  previous years—6 agencies participated in fiscal year 2013, and 7  agencies participated in fiscal year 2014.
The policy directives require each agency that participates in the  administrative pilot program to submit to SBA a work plan that includes,  among other information, a prioritized list of initiatives, the estimated  percentage of funds or amounts to be spent on each initiative, and the  expected results to be achieved. After the fiscal year has ended, agencies  are to report on their actual spending on the administrative pilot program.
In fiscal year 2015, agencies proposed that they would spend $59.8  million on the administrative pilot program, but our analysis of the work  plans and data that agencies submitted to SBA shows that the agencies  obligated $33.9 million, or 57 percent of the proposed amount (see table  2). This represents an increase over previous years; agencies obligated  $12.3 million (21 percent of the proposed amount) in fiscal year 2013 and  $19.1 million (34 percent of the proposed amount) in fiscal year 2014.  Officials from 2 agencies told us that they propose the maximum  allowable amount in their work plans to give them the ability to spend  those funds, but they do not necessarily intend to spend the full amount.  Officials from other agencies told us they either proposed what they  planned to spend or higher than what they planned but not the maximum amount. The 9 agencies that participated in the administrative pilot  program varied greatly in the extent to which they actually obligated what  they proposed to spend on the program. For example, DOE obligated  almost all of its proposed amounts, but Education obligated little of its  proposed amount. Officials at Education told us they proposed to spend  funds on several outreach activities but obligated far less when they  found they had time to attend only one conference.
Although funding for the pilot program cannot replace current agency  administrative funding, the administrative pilot program provides agencies  with new flexibility  to undertake activities they otherwise could not have  undertaken. Agencies that participated in the administrative pilot  program in fiscal year 2015 used funds towards six goals developed by  SBA in consultation with agency officials, as shown in table 3. Our review  of the funds agencies spent on the administrative pilot program showed  that some goals received more attention than others. In particular, 8 of  the agencies that participated in the administrative pilot program spent  funds to increase outreach to underserved communities, 5 agencies  spent funds to help awardees commercialize their products, and 4  agencies spent funds to streamline and simplify the application and award  process. Four agencies also spent funds on preventing fraud, waste, and  abuse, and fewer agencies spent funds on congressional and interagency  reporting.
Within each of these six goals, agencies developed a variety of specific  uses for their funds for the administrative pilot program. Table 4 provides  examples of these uses.
section_title: Officials Identified Benefits and Constraints that Affect Participation and Spending on the Administrative Pilot Program
paragraphs: Officials from all 11 agencies participating in the SBIR and STTR  programs told us that the administrative pilot program is beneficial. For  example, officials at 4 agencies told us that benefits include allowing them  to conduct outreach and other activities, which would not have been  possible without funding from the administrative pilot program. Further,  officials at 1 agency told us that the increased outreach seems to have  resulted in more applications, including from underserved states.
However, officials also identified the following constraints to participating  in the administrative pilot program:
Administrative pilot program authority is temporary. Officials at 2  agencies told us that the temporary nature of the administrative pilot  program constrained them from implementing longer-term activities.  For example, officials at 1 agency said that the temporary nature limits  them to projects that can be completed within 1 year, and officials at  the other agency told us that the temporary nature prevents them from  hiring additional staff that would allow them to more fully implement  the pilot. Program managers at 8 of the 9 agencies participating in the  administrative pilot program told us they supported making it  permanent. SBA officials acknowledged that the temporary nature of  the pilot program is its biggest constraint. As of May 2017, the  administrative pilot program has not been extended past fiscal year  2017.
Funds can be used only for new activities, and some program officials  were unsure about how long certain activities would be considered as  new. Agencies can only spend the funds from the administrative pilot  program on new activities, and officials at 4 agencies told us that  written clarification regarding how SBA defined new activities would  be helpful, particularly because the administrative pilot program has  been in effect for a few years.
SBA is required to collect data and report on the use of funds to achieve  the objectives of the administrative pilot program. In May 2016, we  recommended that SBA could include in its annual report to Congress an  evaluation of the constraints that have hindered agencies’ participation in  the administrative pilot program and steps to address these constraints.  SBA officials told us they probably will not discuss potential constraints in  the fiscal year 2014 report to Congress because SBA is not required to  discuss such constraints in its annual reports and has not collected  information on constraints from all the participating agencies. As of April  2017, SBA officials said that SBA is in the process of completing its  follow-up with agencies to document agencies’ utilization of the  administration pilot program. Officials said they will draft a report once  SBA receives the information requested from the agencies. As of April  2017, a formal date for completing and submitting the report has not been  established.
We continue to believe that having SBA include an evaluation of potential  constraints to participating in the administrative pilot program could be  useful if Congress decides to continue the program in the future. Without  such information, SBA and Congress will not have the information they  need to address the constraints and help ensure agencies are  implementing the administrative pilot program to the fullest extent.
section_title: Conclusions
paragraphs: Under the SBIR and STTR programs, federal agencies have awarded  billions of dollars to small businesses to develop and commercialize  innovative technologies. In our previous reports on these programs, we  identified some steps SBA could take to better ensure agencies’  compliance with spending and reporting requirements and to improve  management of the administrative pilot program. SBA has taken some of  these steps, such as revising its annual reporting template to clarify the required data that agencies should submit. One key remaining issue is  ensuring that all agencies submit the data required for SBA to determine  whether they have met spending requirements. In fiscal year 2015, 2  participating agencies—DOD and EPA—did not submit these data  because they do not have procedures for collecting the data, and they  face challenges in collecting these data. A working group of agencies  participating in the SBIR and STTR programs is considering proposing an  alternative methodology to calculate spending requirements for the  programs. Until DOD and EPA find ways to provide the required data or  until the spending requirements are changed or the identified data  collection challenges are resolved, DOD and EPA will not know whether  they have met the spending requirements, and SBA will be limited in its  ability to accurately report that information to Congress.
section_title: Recommendation for Executive Action
paragraphs: To ensure full compliance with SBIR and STTR spending and reporting  requirements, we recommend that the Secretary of Defense and the EPA  Administrator establish procedures to collect and submit obligations data  or—through SBA, independently,  or through a working group of agencies  participating in the SBIR and STTR programs—propose to Congress an  alternative methodology for calculating spending requirements for their  agencies.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to SBA and the 11 participating  agencies for review and comment. In written comments, DOD and EPA  agreed with the recommendation, and EPA stated that it will work with  SBA to develop an alternative methodology for calculating spending  requirements. DOD’s and EPA’s comments are reproduced in appendixes  III  and IV. Three agencies—Commerce, HHS, and NSF—provided  technical comments, which we incorporated as appropriate. Seven  agencies—DHS, DOE, DOT, Education, NASA, SBA, and USDA—had no  technical or written comments.
We are sending copies of this report to the appropriate congressional  committees; the Secretaries of Agriculture, Commerce, Defense,  Education, Energy, Health and Human Services, Homeland Security, and  Transportation; the Administrators of the Small Business Administration,  the Environmental Protection Agency, and the National Aeronautics and  Space Administration; the Director of the National Science Foundation;  and other interested parties. In addition, the report is available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or neumannj@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made key contributions to  this report are listed in appendix V.
section_title: Appendix I: Agencies’ Spending and Spending Requirements for the Small Business Innovation Research Program for Fiscal Year 2015, According to Agency Data
paragraphs: Nine of the 11 agencies participating in the SBIR program submitted the  extramural research or research and development (R&D) obligations data  required to determine compliance with program spending requirements  for fiscal year 2015. Eight of these 9 agencies met or exceeded their  SBIR spending requirements for fiscal year 2015, according to the  agencies’ data. For the purposes of this report, we defined compliance as  spending at least 2.9 percent of an agency’s reported extramural R&D  obligations on the SBIR program in fiscal year 2015, as required by the  Small Business Act. This method is consistent with the Small Business  Administration’s approach for calculating spending requirements in its  reports to Congress on the program. We could not determine compliance  with spending requirements for the Department of Defense and the  Environmental Protection Agency because those agencies did not submit  the required extramural R&D obligations data. Table 5 shows agency  SBIR spending for all 11 participating agencies and spending  requirements for the 9 agencies that submitted the required obligations  data.
section_title: Appendix II: Agencies’ Spending and Spending Requirements for the Small Business Technology Transfer Program for Fiscal Year 2015, According to Agency Data
paragraphs: Four of the five agencies participating in the STTR program submitted the  extramural research or research and development (R&D) obligations data  required to determine compliance with program spending requirements  for fiscal year 2015. All four of these agencies met or exceeded their  STTR spending requirements for fiscal year 2015, according to the  agencies’ data. For the purposes of this report, we defined compliance as  spending at least 0.40 percent of an agency’s reported extramural R&D  obligations on the STTR program in fiscal year 2015, as required by the  Small Business Act. This method is consistent with the Small Business  Administration’s approach for calculating spending requirements in its  reports to Congress on the program. We could not determine compliance  with spending requirements for the Department of Defense because it did  not submit the required extramural R&D obligations data. Table 6 shows  agency STTR spending for all five participating agencies, and spending  requirements for the four agencies that submitted the required obligations  data.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: Comments from the Environmental Protection Agency
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Hilary Benedict (Assistant  Director), John Barrett, Antoinette  Capaccio, Ellen Fried, Cindy Gilbert,  Terry Hanford, and Sara Sullivan made key contributions to this report.