Small Business Credit Programs: Treasury Continues to Enhance Performance Measurement and Evaluation but Could Better Communicate and Update Results
Released: 2014-12-11T00:00:00

Letter

Background

Small Business Lending Fund

For the purpose of the SBLF program, the Small Business Jobs Act of 2010 defines qualified small business lending—as defined by and reported in an institution’s quarterly regulatory filings, also known as Call Reports—as one of the following: owner-occupied nonfarm, nonresidential real estate loans; commercial and industrial loans; loans to finance agricultural production and other loans to farmers; and loans secured by farmland.

In addition, qualifying small business loans cannot be for an original amount of more than $10 million, and the business may not have more than $50 million in annual revenue. The act specifically prohibits Treasury from accepting applications from institutions that are on the Federal Deposit Insurance Corporation’s (FDIC) problem bank list or have been removed from that list during the previous 90 days. The initial baseline small business lending amount for the SBLF program was the average amount of qualified small business lending that was outstanding for the four full quarters ending on June 30, 2010, and the dividend or interest rates paid by an institution are adjusted by comparing future lending against this baseline. Also, the institution is required to report any loans resulting from purchases, mergers and acquisitions so that its qualified small business lending baseline is adjusted accordingly.

Risk-weighted assets are weighted according to credit risk and are used in the calculation of required capital levels. Specifically, all assets are assigned a risk weight according to the credit risk of the obligor or the nature of the exposure and the nature of any qualifying collateral or guarantee, where relevant.

Purchase Program (CPP).required to submit a small business lending plan to its regulator describing how the applicant’s business strategy and operating goals would allow it to address the needs of small businesses in the area it serves.

Some banking institutions are formed as C-Corporations. C-Corporations pay federal and state income tax on earnings. When earnings are distributed to shareholders as dividends, they are subject to taxation. C-Corporations, unlike S-Corporations, are taxed separately from their owners. is designed to encourage CDLFs to repay the capital investment by the end of the 8-year period. Treasury allows an SBLF participant to exit the program at any time, with the approval of its regulator, by repaying the funding provided along with dividends or interest owed for that period.

Under the act, Treasury has a number of reporting requirements to Congress related to SBLF: (1) a monthly report describing all of the transactions made under the program during the reporting period; (2) a semiannual report (for the periods ending each March and September) providing all projected costs and liabilities and all operating expenses; and (3) a quarterly report, known as the Lending Growth Report, detailing how participants have used the funds they have received under the program.

State Small Business Credit Initiative

SSBCI was established to support existing and new state programs that support private financing to small businesses and small manufacturers that, according to Treasury, are not obtaining the loans or investments they need to expand and to create jobs. SSBCI provides direct support to participants for use in programs designed to increase access to credit for small businesses. Using a formula contained in the Small Business Jobs Act of 2010, Treasury calculated the amount of SSBCI funding for which each of the 50 states, as well as the District of Columbia, the Commonwealth of Puerto Rico, the Commonwealth of the Northern Mariana Islands, Guam, American Samoa, and the United States Virgin Islands, were eligible to apply. This formula takes into account a state’s job losses in proportion to the aggregate job losses of all states. In addition to states, the act granted permission to municipalities to apply directly for funding under SSBCI in the event that their state did not apply for funding. Municipalities in Alaska, North Dakota, and Wyoming used this option. Participants are expected to use their SSBCI funds to leverage private financing and investment that is at least 10 times the amount of their SSBCI allocation (a leverage ratio of 10 to 1) by December 31, 2016. Forty-seven states; the District of Columbia; the Commonwealth of Puerto Rico; the Commonwealth of the Northern Mariana Islands; Guam; American Samoa; the United States Virgin Islands; Anchorage, Alaska; Carrington, North Dakota; Mandan, North Dakota; and Laramie, Wyoming, currently participate in the program.

The act allowed Treasury to provide SSBCI funding for two state program categories: capital access programs (CAP) and other credit support programs (OCSP). For both CAPs and OCSPs, lenders are required to have a meaningful amount of their own capital at risk. Loan terms, such as interest and collateral, are typically negotiated between the lender and the borrower, although in some cases loan terms are subject to SSBCI participant approval and, in many cases, the SSBCI participant and lender will discuss and negotiate loan terms and options prior to reaching agreement.

A CAP is a loan portfolio insurance program wherein the borrower and lender, such as a small business owner and a bank, contribute to a reserve fund held by the lender. Under a CAP, when a participating lender originates a loan, the lender and borrower combine to contribute an amount equal to a percentage of the loan to a loan reserve fund, which is held by the lender. Under SSBCI, the contribution must be from 2 percent to 7 percent of the amount borrowed. Typically, the contribution ranges from 3 percent to 4 percent. The SSBCI participant then matches the combined contribution and sends that amount to the lender, which deposits the funds into the lender-held reserve fund. Under SSBCI, approved CAPs are eligible to receive federal contributions to the reserve funds held by each participating financial institution in an amount equal to the total amount of the contributions paid by the borrower and the lender on a loan-by-loan basis.

In addition, the following OCSPs are examples of programs eligible to receive funding under the act:

Collateral support programs: A Collateral Support Program is designed to enable financing that might otherwise be unavailable due to a collateral shortfall. It provides pledged cash collateral to lenders to enhance the collateral coverage of individual loans. The SSBCI participant and lender negotiate the amount of cash collateral to be pledged by the SSBCI participant.

Loan participation programs: SSBCI participants may structure a loan participation program in two ways: (1) through purchase transactions, also known as purchase participations, in which the SSBCI participant purchases a portion of a loan originated by a lender, or (2) by participating in a loan as a co-lender, where a lender originates a senior loan and the SSBCI participant originates a second loan to the same borrower that is usually subordinate to the lender’s senior loan should a default occur. SSBCI loan participation programs encourage lending to small businesses because the lender is able to reduce its potential loss by sharing its exposure to loan losses with the SSBCI participant.

Loan guarantee programs: These programs enable small businesses to obtain a term loan or line of credit by providing the lender with the necessary security in the form of a partial guarantee. In most cases, the SSBCI participant sets aside funds in a dedicated reserve or account to collateralize the guarantee of a specified percentage of each approved loan. The guarantee percentage is determined by the participants and lenders but, under SSBCI, may not exceed 80 percent of loan losses.

Venture capital programs: These programs provide investment capital to create and grow start-ups and early-stage businesses, often in one of two forms: (1) an SSBCI participant-run venture capital fund (which may include other private investors) that invests directly in businesses, or (2) a fund of funds, which is a fund that invests in other venture capital funds that in turn invest in individual businesses.

Direct loan programs: Although Treasury does not consider these programs to be a separate SSBCI program type, it acknowledges that some states may identify programs that they plan to support with SSBCI funds as direct loan programs. The programs that some participants label as direct loan programs are viewed by Treasury as co-lending programs categorized as loan participation programs, which have lending structures that are allowable under the statute.

OCSPs approved to receive SSBCI funds are required to target small businesses with an average size of 500 or fewer employees and to target support toward loans with an average principal amount of $5 million or less. In addition, these programs cannot lend to borrowers with more than 750 employees or make any loans in excess of $20 million.

After their applications were approved, the SSBCI participants entered into Allocation Agreements with Treasury before they received their funds. SSBCI Allocation Agreements are signed by Treasury and participants, and they outline how recipients are to comply with program requirements. The act requires that each participant receive its SSBCI funds in three disbursements or tranches of approximately one-third of its approved allocation. As part of its request to receive the second and third disbursements, a participant must certify that it has expended, transferred, or obligated 80 percent or more of the previous disbursement. When participants request subsequent disbursements, Treasury may also review a sample of the participant’s transactions for compliance with SSBCI requirements. All SSBCI Allocation Agreements will expire on March 31, 2017. Treasury may terminate any portion of a state’s allocation that Treasury has not yet transferred to the participant within 2 years of the date on which its SSBCI Allocation Agreement was signed. Treasury may also reduce, suspend, or terminate a state’s allocation at any time during the term of the Allocation Agreement upon an event of default under the agreement. Under the act, participants are required to submit quarterly and annual reports on their use of SSBCI funds.

Changes to participants’ Allocation Agreements generally must be approved by Treasury through a program modification. For example, participants must submit a program modification to Treasury for approval in order to use SSBCI funds for a new program that was not originally approved by Treasury, to materially change the scope or purpose of an approved program, or to reapportion and transfer allocated SSBCI funds among approved programs when the cumulative amounts transferred exceed 20 percent of the participant’s total SSBCI allocation. Treasury has procedures for participants to apply for modifications and for its processing of modification approval requests. In general, the procedures require participants to submit, among other things, justification of the need for the modification and the impact of the change on program performance, including the 10 to1 private leverage expectation. As of June 30, 2014, Treasury had approved 60 modifications of SSBCI participants’ programs.

SBLF Participants Continue to Lend to Small Businesses and Additional Participants Have Exited the Program

Treasury’s 2014 second quarter Lending Growth Report showed that SBLF participants’ aggregate qualified small business lending has continued to exceed their baseline levels since the program began. As of June 30, 2014, total qualified small business lending for SBLF participants increased by $13.5 billion over an aggregate baseline of $33 billion. Banks participating in the SBLF program increased their qualified small business lending by $13.1 billion (41 percent) over a $32.2 billion baseline. CDLFs participating in the SBLF program increased their qualified small business lending by $394.6 million (50 percent) over a $794.5 million baseline. Of the 241 participating banks, 229 (95 percent) increased their qualified small business lending, and 45 out of 49 CDLFs (92 percent) increased their qualified small business lending.

Banks and CDLFs had made about $255 million in dividend, interest rate, and fee payments to Treasury as of June 30, 2014. Banks paid $249 million and CDLFs paid $5.6 million in dividend, interest, and fee payments. See figure 1 for the numbers of participants in different dividend or interest rate categories. Between July 1, 2013, and June 30, 2014, five SBLF participants were required to pay approximately $427,000 in CPP lending incentive fees to Treasury.

From July 1, 2013, to June 30, 2014, the number of participants that had exited the program more than doubled. In our previous report, we found that as of July 1, 2013, 16 of the 332 institutions (5 percent) that received SBLF funds had fully redeemed Treasury’s investment and exited the program. As of June 30, 2014, a total of 41 institutions (12 percent) had exited the program. These 41 institutions included 39 banks and 2 CDLFs. Of the 39 banks that exited, 23 had used SBLF funds to repay an earlier investment from TARP’s CPP, according to Treasury. As of October 2014, Treasury had not had to write off any of its investments due to bankruptcy, according to agency officials.

In August 2014, Treasury reported in its Second Annual SBLF Lending Survey report that of the 299 institutions participating in SBLF at the time the survey was administered, 247 (83 percent) planned to fully redeem Treasury’s investment. These 247 institutions included 208 banks and 39 CDLFs. Of the 208 banks that indicated in the survey that they plan to fully redeem, 168 (81 percent) reported that they are planning to redeem in full by the end of first quarter 2016 when the statutorily required rate increase to 9 percent begins. Of the 39 CDLFs that indicated in the survey that they plan to fully redeem, 35 reported that they plan to redeem Treasury’s investment by the end of third quarter 2019, when the 9 percent interest rate begins.

Participants Have Used over Half of the SSBCI Funds and Treasury Continues to Address Challenges in Using Program Funds

As of June 30, 2014, SSBCI participants have used over half of the total allocated funds, but the rate of funds used among participants continues to vary. Some of the SSBCI participants we interviewed for this report faced challenges using funds that were consistent with the challenges we previously reported, and Treasury and SSBCI participants have continued to undertake activities to address these challenges. The fiscal year 2015 president’s budget recommends legislation authorizing a $1.5 billion extension of SSBCI funding.

SSBCI Participants Have Used over Half of the Total Allocated Funds, but Participants’ Rate of Fund Usage Continues to Vary

Overall, SSBCI participants have used about $869 million (60 percent) of the $1.5 billion in allocated funds, as shown in figure 2. Between June 30, 2013, and June 30, 2014, participants used about $320 million in SSBCI funds.

As of June 30, 2014, participants’ use of SSBCI funds continued to vary widely, as shown in figure 3. For example, 7 participants have used 25 percent or less of their allocations, while 10 have used 90 percent or more of their allocations. Three participants had used 1 percent or less of their allocations. According to Treasury officials, factors such as interest among local lenders, state marketing efforts, specific program design features, and local economies may account for differences in participants’ use of funds.

According to Treasury’s quarterly report on participants’ progress in using SSBCI funds, all 57 participants had received their first disbursement, 47 had received their second disbursement, and 20 participants had received their third disbursement, as of June 30, 2014. As previously discussed, participants can request subsequent SSBCI disbursements when they have used at least 80 percent of their current tranche of SSBCI funds. In addition, 13 states have reused SSBCI funds by using interest or principal repayments on existing SSBCI loans and investments to make additional financing available to other small businesses. As of June 30, 2014, Massachusetts had recycled about 40 percent of the SSBCI funds it had used. Following Massachusetts, Idaho and Michigan recycled about 20 and 12 percent of the funds they had used, respectively. According to a Massachusetts state official, the high rate of recycled funds in the state likely is attributable to, in part, the high interest rate on SSBCI-funded loans, which results in loans that are typically paid off before maturity. The official explained that the SSBCI-funded loan program in Massachusetts is a preexisting program intended to help small business owners who may not qualify for lower rate loans with typical lenders. These loans provide a source of financing for these borrowers until they are more stable and able to refinance at lower rates in the private market.

Participants used a variety of different program types to distribute their allocated funds. Figure 4 shows the number of different program types that had used SSBCI funds as of June 30, 2014. Similar to data we reported in our 2013 report, loan participation and venture capital programs continue to be the most common program types. Some participants chose to use only one program type, whereas others used multiple programs. For example, 14 participants used their allocation to fund one program type while others used up to five program types.

SSBCI participants distributed their allocated funds using 150 programs. Participants had the option to use existing programs to distribute their allocated funds or create new programs specifically to utilize their SSBCI funds. As shown in figure 5, most of the programs using SSBCI funds are new. Specifically, of the 150 programs, 84 (56 percent) were new programs compared to 66 (44 percent) existing programs.

Treasury and SSBCI Participants Continue to Address Challenges in Using Program Funds and Promoting the Program

Some SSBCI participants we interviewed for this report faced difficulties in using program funds that were consistent with some of the challenges we previously reported in 2013.interviewed for this report, 2 participants stated they did not encounter significant challenges in using program funds. Three participants we interviewed indicated that Treasury has improved its guidance since the program began, stating that the information on Treasury’s website has been helpful. Some participants, however, stated they continued to face challenges related to a lack of clarity in guidance and some banks’ reluctance to participate. Specifically: Of the 10 SSBCI participants we

Treasury officials stated that they have established a process by which they provide written responses to questions raised by states, but officials from three participating states indicated that there continues to be a lack of clarity in Treasury’s guidance regarding the use of SSBCI funds for certain transactions. One participant explained that, while this lack of clarity is not unexpected since SSBCI is still a relatively new program, it may have resulted in some missed opportunities to use SSBCI funds. The participant stated that Treasury responded in writing to questions it raised but the additional guidance Treasury provided was still vague.

According to officials in three participating states, some banks were reluctant to participate in the program because they were unfamiliar with it or perceived that it would increase scrutiny from regulators. Specifically, one participant stated that the local banking community generally was not familiar with state lending programs and some have been reluctant to participate in government loan programs overall. The participant explained that some local banks prefer to work with programs that they were more familiar with, such as programs from the Small Business Administration. Another participant highlighted the ongoing concerns about being subject to increased regulatory scrutiny for using SSBCI programs. The participant explained that, although regulators had made efforts to address banks’ concerns about the potential for increased scrutiny from regulators because of the risk associated with loans needing SSBCI funds, banks continued to be reluctant to participate for this reason.

SSBCI participants have continued to undertake the types of activities we reported in 2013 to promote the use of SSBCI and address challenges in using program funds. Several of the SSBCI participants we interviewed for this report conducted activities, similar to those we reported in 2013, to promote the program, including holding face-to-face meetings, conducting presentations to interested parties, and maintaining websites, among other activities. In addition, similar to what we reported in 2013, some participants used program modifications to address challenges in using program funds. Specifically, 6 of the 10 participants we interviewed stated they had requested and received approval for program modifications to make use of SSBCI funds. For example, according to state officials, Treasury approved a program modification request from Connecticut in July 2014, which will allow the state to transfer funds from its CAP to a newly created venture capital program. As of June 30, 2014, Connecticut had used only 1 percent of its allocation within its CAP because, according to a state official, a Small Business Administration (SBA) program and a new state program decreased demand for their SSBCI- funded CAP. As a result of the modification allowing the state to transfer funds to a venture capital program, Connecticut officials told us they will partner with the state’s quasi- public entity that supports small business financing that includes venture capital funding and expect to be able to deploy the remainder of the state’s SSBCI allocation within 18 months.

Treasury has continued existing activities as well as undertaken new ones to address challenges states face in using program funds and promoting SSBCI. Specifically, Treasury officials stated that they have continued to offer technical assistance to states and utilize relationship managers, who are responsible for working with an assigned group of participants to support their outreach to the lending industry and help them successfully allocate funds to lenders and subsequently borrowers. In addition, Treasury has continued to facilitate information sharing among states through its national conferences and various working groups. For example, Treasury published reports summarizing the discussions and observations of its working groups on venture capital and addressing underserved communities in April and October 2014, respectively. Since our 2013 report, Treasury has undertaken new activities to facilitate information sharing among states such as organizing multistate sessions and other activities. To date, Treasury has facilitated three regional multistate sessions, which provide a forum for state officials and other stakeholders to share information and learn from each other. Officials explained that, although Treasury organizes the sessions, participants determine the content. Treasury officials also told us they facilitate information sharing on venture capital through quarterly calls for SSBCI program managers and contractors that administer venture capital programs. Treasury officials stated that they plan to publish a compilation of case studies on using SSBCI funds to target underserved communities, which is intended to complement its prior work on targeting underserved communities.

The Fiscal Year 2015 President’s Budget Proposed an Additional $1.5 Billion in SSBCI Funding

The fiscal year 2015 president’s budget recommended new legislation that would reauthorize SSBCI and provide an additional $1.5 billion in program funding. Under the proposal, $1 billion of the funds would be awarded on a competitive basis and $500 million would be awarded through a need-based formula. The budget justification states that a second round of funding, which would begin in 2015 and run through 2021, will expand on the success of the first round, capitalize on new working relationships between states and small business lenders and investors, and strengthen the federal government’s relationships with state economic development agencies. In May 2014, members of the United States Senate and House of Representatives proposed companion bills, titled the Small Business Access to Capital Act of 2014, which would reauthorize SSBCI. The bills would reauthorize SSBCI and provide an additional $1.5 billion to be allocated on a similar competitive and need-based formula basis.

In developing the budget justification and providing information to the Office of Management and Budget (OMB) on the amount and structure of the proposed additional SSBCI funding, Treasury officials stated they considered data on participants’ progress in using SSBCI funds and considered other information, such as market conditions and lessons learned. In addition, Treasury officials stated they considered data from participants’ quarterly reports and considered states’ disbursement and modification requests to support the proposed $1.5 billion in additional SSBCI funding. Treasury officials indicated that there is demand for additional funding as participants approach their third disbursement. In addition, the rate at which some participants are using their funds indicates additional demand. Specifically, Treasury officials stated they were confident that some participants could use the additional funding based on the 12-month rate of expended, obligated, and transferred funds among participants.

To support the proposal for both a competitive process and need-based formula, Treasury officials stated they considered the variation among participants regarding the speed at which funds were disbursed and the impact of the funds, as well as current market conditions in certain areas. Treasury officials stated that they introduced a competitive funding element because some participants were able to more quickly disburse SSBCI funds, demonstrated greater impact using the funds, and demonstrated greater capacity to establish relationships with private lenders and investors. The budget proposal states that the competitive funds would be awarded to states on the basis of their ability to target underserved groups, leverage federal funding, and evaluate results. Officials stated that they proposed maintaining a need-based allocation method because their analysis of market conditions indicated that some areas continued to have significant needs for increased access to capital. The proposal states that the need-based award would be allocated based on economic factors such as job losses and pace of economic recovery. Treasury has not proposed additional details regarding the structure of the proposed additional funding, according to agency officials. However, officials stated that they would recommend that certain features of the program be maintained, such as keeping the five program types, maintaining the 10 to 1 public to private leverage ratio, and emphasizing increasing access to underserved communities.

Some participants we interviewed said they were better positioned than others to continue their efforts in the absence of additional SSBCI funding. Participants recognized that additional funds would allow them to meet current demand for existing programs, provide funds more consistently, or explore adding additional programs to meet other economic development needs in the state. Eight of the 10 participants we interviewed stated that recycled funds would play a role in continuing their SSBCI-funded efforts. Two participants indicated that some discussions had been held at the state level about providing state funds to continue efforts currently funded by SSBCI. Conversely, two participants indicated that state funds were not an option for continuing their efforts. For example, one participant explained that the state constitution does not allow public state funds to be used for private lending.

Treasury Has Enhanced Performance Measures and Evaluation but Has Not Effectively Communicated or Updated Its SBLF Evaluation

Treasury has taken steps to enhance performance measures and evaluation, consistent with our prior recommendations. For SBLF, Treasury conducted an impact evaluation using statistical methods to compare lending between SBLF banks and a control group of non-SBLF banks that are as similar as possible to participating SBLF banks. This rigorous approach estimated the impact of the program and is a significant improvement over its previous comparison assessments of the SBLF program which did not attempt to isolate impact. However, while Treasury discussed the results of its previous comparisons in the body of its Lending Growth Reports, it discussed the results of its more rigorous impact evaluation less prominently and included them only in the appendixes of the reports. In addition, Treasury has not updated the results of its impact evaluation. For SSBCI, Treasury developed some new performance measures and also developed targets for some of its measures. In addition, officials stated they reached out to and received input from internal agency subject-matter experts and experts from OMB on the design of the planned SSBCI evaluation and have reached out to congressional staff for feedback on its proposed design.

Treasury Conducted an Impact Evaluation of SBLF Using a Generally Accepted and Rigorous Approach

In 2011 and 2013, we recommended that Treasury include in its evaluation of SBLF measures that isolate the net effect of the SBLF program apart from other factors that could also affect small business lending. See GAO-12-183 and GAO 14-135. adjust for differences between treatment and control groups.instance, SBLF banks are considered the treatment group, and banks that are similar to participating SBLF banks yet did not participate in the program are considered the control group. To conduct its propensity score matching approach, Treasury selected a control group of banks that are as similar as possible to participating SBLF banks along a number of relevant and observable participant characteristics. Specifically, Treasury incorporated a broad set of 71 observable financial and market variables that could affect a bank’s likelihood of participating in the program. These variables include balance sheet measures, financial performance measures, capitalization, and loan composition, among others. Using the observable financial and market variables, Treasury achieved similarity between the two groups of banks by estimating the propensity to participate in the program for participants and nonparticipants alike and then matching nonparticipants to participants with a similar propensity to participate. Treasury officials stated they chose the propensity score matching approach because it makes similar comparisons as their previous assessments in that the propensity score matching approach also compares lending between SBLF banks and a selected control group of banks.

Treasury’s impact evaluation of the program estimated that SBLF resulted in a median increase of 23 percentage points in small business lending, relative to the most similar control group of banks that did not participate in SBLF. In other words, when controlling for observable variables, the evaluation estimates that 23 percentage points of the increase in small business lending among SBLF banks is attributable to the program. Because the propensity score matching approach results in a carefully selected control group of non-SBLF banks that is as similar as possible to participating SBLF banks, lending among the control group banks can be used to estimate lending among SBLF banks if they had not participated in the program. Specifically, the propensity score matching approach estimated that the average lending of SBLF banks may have increased by 29 percent in the absence of the program (i.e., the median change in lending over baseline among propensity score matched banks). However, the average lending of participating SBLF banks increased by 52 percent, 23 percentage points more than non-SBLF banks. This difference represents the estimated impact of SBLF. It is difficult to interpret the magnitude of the results of Treasury’s impact evaluation as other evaluations on entrepreneurial assistance—including support for small businesses—are not comparable because they did not evaluate program impact on increasing small business lending. However, because SBLF participants might have increased their lending by 29 percent in the absence of the program, 23 additional percentage points in small business lending is a meaningful increase.

Treasury has developed and refined its approach to assessing SBLF, and its impact evaluation is a distinct improvement over its previous comparisons between SBLF banks and groups of non-SBLF banks. While some differences between participants and nonparticipants may be unobserved, such as anticipated demand for small business lending, the control group Treasury developed based on propensity score matching addresses the observable differences among SBLF and non-SBLF banks in an effort to isolate the impact of the program. Previous assessments, however, did not adequately address observable differences between SBLF and non-SBLF banks and did not attempt to isolate the impact of the program. In its earliest approach to assess SBLF, Treasury has compared lending among SBLF banks to small and medium-sized banks, which suggested that SBLF banks increased business lending by 45 percentage points more than non-SBLF banks as of September 30, 2013 (see fig. 6). To improve its approach to assessing the program, Treasury also conducted a peer group comparison by which it compared SBLF banks to a group of peer banks matched according to size, geography, and the Texas Ratio, a measure of asset quality. Treasury officials described these characteristics as common and intuitive for comparing financial institutions. The peer group comparison suggested that SBLF banks increased business lending by about 41 percentage points more than peer group banks as of September 30, 2013. As we stated in our prior reports, these two analyses did not fully identify how SBLF has affected participants’ lending compared to other factors that While the peer group comparison could explain the increase in lending.method was valuable because the peer group was more similar to SBLF banks than the general population of small banks, it omitted important characteristics that could influence small business lending among SBLF participants and nonparticipants, such as liquidity or loan growth rate. The propensity score matching approach attempted to address these limitations because it identified a control group of banks that is similar to SBLF participants along a very broad set of characteristics, which provides greater confidence that any differences observed between participants and nonparticipants are attributable to the program.

Treasury Has Not Effectively Communicated or Updated the Results of Its Impact Evaluation

Treasury adopted a generally accepted and rigorous approach to conduct an impact evaluation, as we recommended, but its presentation did not communicate the results as effectively as possible. In its Lending Growth Reports, Treasury discussed the results of its prior, simpler assessments—those that compare lending among SBLF banks to that of small and medium-sized banks and peer matched banks—more prominently than the more rigorous results of its impact evaluation. Specifically, it discussed the simpler assessments in the body, but published the results of its impact evaluation in the appendixes of its April, July, and October 2014 Lending Growth Reports. Further, Treasury did not reference the results of the impact evaluation in the executive summary or body of the reports. GAO’s Designing Evaluations guide states that how evaluation results are communicated can dramatically affect how they are used. Our review of relevant resources on the use and presentation of program evaluations also indicates that rigorous evidence should be given prominence in comparison to less credible evidence so that the strongest evidence is used to inform policy and the allocation of resources. Further, SBLF’s authorizing legislation requires, in part, that Treasury provide transparency with respect to its use of funds provided by the SBLF program. As we concluded in our prior report, it is important that Congress and stakeholders have information about the extent to which the program has a meaningful impact on small business lending.

Treasury officials stated that the results of the SBLF impact evaluation (i.e., the propensity score analysis) were included as appendixes instead of in the body of the reports because the evaluation was intended to be a one-time analysis of the program. They explained that the purpose of the evaluation was to serve as another objective method for assessing the program and validating the results of its previous analyses. According to agency officials, including the results as an appendix allowed the report’s format to remain consistent with prior and future Lending Growth Report iterations, rather than revising the entire report for one quarterly issuance. However, as agency officials explained, the results of the propensity score matching approach are relatively comparable to their previous assessments because the propensity score matching approach also compares lending between SBLF banks and a selected control group of banks. As such, we determined it is unlikely that discussing the results of the impact evaluation in the body of its quarterly Lending Growth Reports would cause inconsistencies or major revisions.

Because Treasury presents the results of its simpler group comparison more prominently than the results of its impact evaluation in its Lending Growth Reports, readers may not be aware of the results of the more rigorous analysis or understand the differences between the various alternative control groups and what they might imply about the overall impact of the program. As previously discussed and shown in figure 6, Treasury’s less rigorous comparisons suggested that SBLF banks increased lending by 45 and 41 percentage points more than all small and medium-sized banks and peer banks, respectively. The propensity score matching approach—which is designed to isolate the impact of the program—produced the smallest relative median increase in lending at 23 percentage points. However, these results are not discussed in the body of Treasury’s Lending Growth Reports. As a result, readers—including Congress and other stakeholders—may not use the results of the more credible analysis to inform their conclusions about the program and the extent to which it has impacted small business lending.

In addition, Treasury has not updated the results of the impact evaluation. In our work on best practices that enhance the use of performance information, we found that communicating performance information frequently and routinely enhances the use of that information for making decisions that affect future strategies, planning and budgeting, identifying priorities, and allocating resources. Frequently communicated performance information typically includes performance measures, such as the changes in small business lending that Treasury reports each quarter for SBLF banks. Program evaluations, however, are typically conducted less frequently to assess how well a program works. Treasury officials stated that they have not updated the results of the impact evaluation because they intended for the evaluation be a one-time analysis that captured the effect of the program through September 30, 2013—the period during which participating banks were incentivized to increase small business lending over their baselines in order to reduce their dividend rates. Treasury published the same impact evaluation results in appendixes of its April, July, and October 2014 Lending Growth Reports and does not have plans to update the results in future reports. In contrast, Treasury updates the results of its simpler assessments in its quarterly Lending Growth Reports and anticipates continuing to update the results until 2016, when Treasury expects about 80 percent of participants to have exited the program as a result of the scheduled dividend rate increase to 9 percent for most participating banks.

Although program evaluations can be costly and time consuming, we determined that Treasury could adjust its existing propensity-matched control group to update the results of its impact evaluation without having to repeat the entire evaluation. Specifically, Treasury could take an approach similar to the one it uses to update the results of its peer group comparison. As SBLF banks exit the program, Treasury could remove similar non-SBLF banks from the control group and repeat its analyses using the updated groups of banks. Treasury officials acknowledged that they could update the results of their impact evaluation using this approach, but explained that they would need to obtain programming software to conduct the analysis and that considerable time and effort would be required to ensure the accuracy of the impact evaluation data and analyses.

By updating the results of its impact evaluation, Treasury could further enhance the performance information it provides on SBLF. As we have described in previous reports and summarize in appendix III, Treasury provides quarterly information on the performance of SBLF, including its simpler lending growth comparisons among SBLF and non-SBLF banks. Updated results from Treasury’s most rigorous and accurate assessment of SBLF could enhance the performance information that readers and decision makers consider. For example, updated information on the impact of the program could be useful for observing any changes in the impact of the program over time. In addition, because SBLF capital could still promote small business lending even in the absence of dividend- based incentives, updated information on the effect of the program after the lending incentives expired in September 2013 could be useful.

Treasury Is Taking Steps to Improve Its Efforts to Measure and Evaluate Performance for the SSBCI Program

In our prior work, we recommended that Treasury enhance its efforts to measure and evaluate the performance of SSBCI. Consistent with our previous recommendation that Treasury set targets for selected performance measures related to monitoring program performance, Treasury established a target for its performance measure related to the amount of private-sector leverage. In addition, Treasury developed additional performance measures and targets related to the disbursement of funds to states and the extent to which OCSPs target small businesses with an average size of 500 or fewer employees and target support towards loans with an average principal amount of $5 million or less (see table 1). Officials stated that because almost all SSBCI-funded programs became fully operational in 2014, they used the information from the 2013 annual reports to establish performance targets. Officials explained that they used the stated program objectives in the Small Business Jobs Act of 2010 to establish the additional measures and select the measures for which they established targets. OMB guidance defines performance indicators (or measures) as measurable values that agencies use to track progress toward set targets or goals within a time frame. The guidance defines indicators that do not require targets and time frames as “other indicators.” Consistent with this definition, Treasury officials stated that they intend to use the measures for which they did not establish targets as additional indicators of performance.

In response to our recommendation that Treasury seek input from Congress and other federal agencies on what information would be useful in assessing SSBCI’s effectiveness as Treasury designs its program evaluation, officials stated they received input from internal agency subject-matter experts and experts from OMB and have reached out to congressional staff for feedback. Specifically, Treasury conducted meetings with experts in December 2013 and January 2014 on how it will evaluate the performance of SSBCI in 2017. Officials stated that they reached out to staff of two congressional committees in October 2014 for feedback on the proposed evaluation design.

OMB guidance and our prior work have emphasized the importance of performance information and program evaluations in decision making. Specifically, OMB guidance instructs agencies to include a thorough discussion of evidence, both positive and negative, for major proposed policies in its budget submission, including performance indicators, performance goals, and evaluation results. In addition, our work on improving information to Congress found that information on how activities have been implemented and the extent to which the program reached the intended clients can be useful to inform reauthorization decisions. Further, our work assessing duplication, overlap, and fragmentation looked across 52 programs supporting entrepreneurs and emphasized the importance of performance information for decision making and highlights that program evaluations can provide additional information about program performance and help gauge program effectiveness. Specifically, we reported that several of these programs lacked adequate performance information and program evaluations, which resulted in limited information on program efficiency and effectiveness. We found that, without this type of information, Congress and the agencies may not be able to better ensure that scarce resources are being directed to the most effective programs and activities.

Treasury’s existing information on the performance of SSBCI, as well as information that will result from Treasury’s efforts to enhance performance measures and evaluation, could provide useful information for decision makers on the proposed reauthorization of the program. As previously discussed, Treasury officials said they considered information on the performance of SSBCI, such as participants’ progress in using funds, the speed at which funds were disbursed, and the impact of the program, to support the amount and structure of additional program funds under the proposed reauthorization. In addition, Treasury has published performance information on SSBCI that could be helpful for decisions related to the reauthorization of the program. Specifically, Treasury has published quarterly reports on participants’ progress in using funds and two annual reports that summarize key finding from participants’ 2012 and 2013 annual reports. Treasury also published two reports it requested from outside experts on lessons learned from SSBCI loan programs and venture capital programs. Treasury plans to publish its evaluation in 2017. Treasury officials said that they plan to use information from participants’ final program assessments, which are due to Treasury in early 2017, to help them evaluate SSBCI to help them determine whether the program model for SSBCI was successful. To determine the success of SSBCI, officials plan to focus on whether the program achieved its performance targets, whether state managers found the program to be effective for achieving their goals, and whether private lenders and investors found the program to be effective. While the evaluation will be useful for future decisions regarding programs that are intended to increase capital to small businesses in distressed credit markets, the information from Treasury’s evaluation of SSBCI will not be available to inform decisions on the current request for additional funding.

Conclusions

Treasury’s efforts to provide performance information on SBLF and SSBCI are positive developments that could help to ensure that agency decision makers and Congress have information to assist them in making programs more efficient and effective and allocating scarce resources. In response to our prior recommendations, Treasury conducted a rigorous impact evaluation, which is a distinct improvement over other analyses using comparisons because it uses statistical methods to compare SBLF banks to a carefully selected control group of banks that did not participate in the program but are as similar as possible to the participating SBLF banks along a broad set of characteristics. Treasury’s impact evaluation is important particularly because it provides a rigorous assessment of the program’s effects and indicates that SBLF increased small business lending by a smaller percentage than Treasury’s other simpler analyses. However, because Treasury presents its impact evaluation in appendixes in its Lending Growth Reports rather than in the body, users of Treasury’s lending reports may rely on less rigorous analyses to inform their conclusions.

Further, although Treasury intended for the impact evaluation to be a one-time analysis, the existing control group it used to estimate the impact of SBLF has the potential to be updated for future reports. Treasury officials stated that updating the impact evaluation results would require additional time and effort, and we acknowledge that costs and benefits would have to be considered in determining the frequency of the updates. However, Treasury has an opportunity to better utilize its impact evaluation to further enhance the performance information available on SBLF. Policymakers will likely face future constrained credit environments for small businesses and will seek options to address them. In such circumstances, the results of Treasury’s impact evaluation, and any updates to those results, could provide a useful assessment of the effectiveness of SBLF as a capital investment program.

Recommendations for Executive Action

To help ensure that Congress and stakeholders can easily access the best information available to inform their conclusions about the effectiveness of SBLF as a capital investment tool, we recommend that the Secretary of the Treasury take the following two actions: (1) make the results of its SBLF impact evaluation more prominent, such as by discussing the results in the executive summary and body of future Lending Growth Reports, and (2) update the results of its impact evaluation in future reports, taking into consideration the costs and benefits of doing so to determine the appropriate frequency of the updates.

Agency Comments and Our Evaluation

We provided a draft of this report to Treasury for their comment and review. Treasury provided written comments, which are reprinted in appendix III. In its written comments, Treasury agreed to implement our recommendations regarding its SBLF impact evaluation and described the actions it plans to take. Treasury stated that it would publish a stand- alone report with its own link on the Treasury website to better communicate the results of the SBLF impact evaluation, which analyzed lending growth performance as September 30, 2013. In its comments, Treasury suggested that all three of its methods of evaluating the performance of SBLF indicate the program has had a positive impact. However, as we stated in our report, only one of its methods—the impact evaluation using the propensity score matching approach—estimates the impact of the program. Treasury also stated that it would update its impact evaluation in the future, taking into consideration the costs and benefits of doing so as well as the lending incentive structure of the program.

We did not make any new recommendations regarding SSBCI in this report, but Treasury noted that it appreciated prior GAO recommendations. Treasury stated that it accepted GAO’s guidance on developing program evaluations and has sought to supplement this guidance with input from stakeholders for its planned SSBCI evaluation. Treasury also noted that its final assessment of SSBCI would include three sections: (1) a review of national program-wide outcomes, (2) review of the state-by-state variation in program outcomes, and (3) feedback from private sector lenders and investors. Treasury provided technical comments on the draft report, which we have incorporated in the final report, as appropriate.

We are sending copies of this report to appropriate congressional committees and Treasury. In addition, the report is also available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to examine (1) the status of the Small Business Lending Fund (SBLF), (2) the status of the State Small Business Credit Initiative (SSBCI), and (3) the extent to which Treasury’s efforts enhance performance measurement and program evaluation.

SBLF Program Status

To examine the status of SBLF, we reviewed Treasury program data from Treasury’s October 2014 Lending Growth Report and its June 2014 monthly transaction report. To show lending growth among SBLF participants, we analyzed Treasury data as of June 30, 2014. We reviewed Treasury’s monthly transaction report to determine the dividend, interest, and fee payment received by Treasury and the number of participants that exited the SBLF program as of June 30, 2014. We also used responses from SBLF respondents to an item in Treasury’s second annual SBLF survey to describe participants’ plans for exiting the program. To assess the reliability of SBLF data, we reviewed prior GAO work on the data and the systems that produced them, which included interviewing Treasury officials on how they assess the reliability of participants’ quarterly data and the information they maintain on SBLF. In addition, we inspected data for missing observations and outliers. Where there were questions or discrepancies we identified related to the data, we clarified them through communications with Treasury officials. We determined that the data were sufficiently reliable for the purposes of describing the status of the program.

SSBCI Program Status

To examine the status of SSBCI, we analyzed Treasury program data on participants’ allocation amounts and program types, and whether participants developed new programs to participate in SSBCI or expanded existing programs with SSBCI funds. In addition, we analyzed participants’ June 30, 2014, SSBCI quarterly report data—the most recent quarter available—on the total amount of funds used and fund usage by program type. To assess the reliability of SSBCI data, we reviewed prior GAO work on the data and the systems that produced them that included interviewing knowledgeable Treasury officials on how they assess the reliability of participants’ quarterly data and the information they maintain on SSBCI. In addition, we performed electronic testing for obvious errors in accuracy and completeness. Where there were questions or discrepancies we identified related to the data, we clarified them through discussions with Treasury officials. We determined that the data were sufficiently reliable for the purposes of describing the status of the program.

We also interviewed Treasury SSBCI officials and 10 purposively selected SSBCI participants on the use of funds, any challenges in using the allocations, and the president’s request for additional program funds. We interviewed officials from Connecticut; Idaho; Louisiana; Massachusetts; Minnesota; Nebraska; Oklahoma; Washington; West Virginia; and Laramie, Wyoming. We selected a non-random sample of 10 participants from the total of 57 participants—which included 47 states; Washington, D.C.; four municipalities; and five U.S. territories— using participants’ March 31, 2014, quarterly report data. We selected this nonrandom, purposive sample of participants because we assumed that the types of challenges related to using SSBCI funds would vary for participants characterized by a mix of fund usage histories and numbers of new and existing programs across all regions. We selected participants based on the following criteria: (1) the percentage of SSBCI funds participants had used as of March 31, 2014; (2) the percentage of SSBCI funds transferred to participants; (3) the percent of SSBCI funds recycled by participants; (4) the number of programs participants implemented and whether the programs were new or existing; (5) the number of program modifications; and (6) geographic dispersion. We purposively weighted the sample to approximate the distribution found in the population for percentage of funds used, percentage of funds transferred, and region to allow for a greater range of views that might also be found in the population. This sample does not represent the experience of the states that are not included in this sample.

To reduce respondent burden, we excluded from our selection (1) the four participants that the Treasury Office of the Inspector General told us it was in the process of auditing or was planning to audit at the time we selected the participants and (2) the nine participants we interviewed for our 2013 SSBCI review. When we selected participants to interview, the March 2014 data were the most recent available. We also reviewed testimonial evidence we obtained during our 2013 SSBCI report related to challenges SSBCI participants faced in using SSBCI funds. We also interviewed representatives from three trade associations involved with the SSBCI program: (1) the Council of Development Finance Agencies because of its role in supporting SSBCI participants through webinars and other resources; (2) the Opportunity Finance Network because it is a national network of community development financial institutions; and (3) the Community Development Venture Capital Alliance because of its role in supporting venture capital investing. To describe Treasury’s request for additional funds, we reviewed the 2015 President’s budget justification for SSBCI and interviewed Treasury officials about the budget request and lessons learned they would apply in a second round of funding.

Treasury’s Efforts to Enhance Performance Measures and Evaluations

To assess the extent to which Treasury’s efforts enhance performance measures and evaluations for SBLF and SSBCI, we interviewed agency officials on the activities they conducted to implement recommendations from our 2012 and 2013 reports and collected information related to these activities. For SBLF, we reviewed Treasury’s impact evaluation presented in its April and July 2014 Lending Growth Reports and interviewed agency officials for additional information on the methodology and results of the evaluation. In addition, to assess the extent to which Treasury effectively communicated the results of the impact evaluation, we reviewed its Lending Growth Reports to determine how other assessments are presented. We developed criteria and based our conclusions on professional judgment and consultation of a variety of relevant resources on the use and presentation of policy evaluations.SSBCI, we reviewed the new performance measures and targets Treasury adopted in October 2014.

We conducted this performance audit from April 2014 to December 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Summary of Treasury’s Efforts to Measure and Evaluate the Performance of the Small Business Lending Fund (SBLF) and State Small Business Credit Initiative (SSBCI)

Appendix II: Summary of Treasury’s Efforts to Measure and Evaluate the Performance of the Small Business Lending Fund (SBLF) and State Small Business Credit Initiative (SSBCI)

Treasury has undertaken several efforts to measure and evaluate the performance of SBLF and SSBCI, as shown in table 2.

Appendix III: Comments from the Department of the Treasury

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Daniel Garcia-Diaz, (202) 512-8678, garciadiazd@gao.gov.

Staff Acknowledgments

In addition to the individual named above, Kay Kuhlman (Assistant Director), Charlene Lindsay (Analyst-in-Charge), Bethany Benitez, Mark Braza, Pamela Davidson, Michael Hoffman, Robert Rieke, Jennifer Schwartz, Stephanie Shipman, and Jena Sinkfield made key contributions to this report.