October 2016

               Report on SBLF Program

Impact Evaluation
2

SBLF PROGRAM IMPACT EVALUATION  

The Small Business Jobs Act of 2010 directs the Secretary of the Treasury to provide a quarterly report on how

institutions participating in the SBLF program have used the funds they received under the program.  Each quarter

SBLF publishes a report providing information on changes in small business lending by participants relative to

baseline levels1

.  These reports have compared SBLF participant lending growth to that of a broad comparison group

as well as a representative peer group.  The results suggest that the SBLF program is supporting significantly

increased business lending by participants, consistent with the program’s objectives.

This report describes SBLF program results using a statistical methodology called propensity score analysis which

attempts to limit the potential reported effect of selection bias.  This analysis of participant lending growth was

performed for the quarter ending September 30, 2015.  Consistent with the findings in the Report on SBLF Program

Impact Evaluation dated January 2015, the results of this analysis suggest that the SBLF program is supporting

substantially increased business lending among program participants.

Program Impact Analysis Using Propensity Score Method

Propensity score methods seek to account for observable characteristics that may be predictive of a bank’s likelihood

of participating in the SBLF, extending beyond the conventional measures of market analysis used in the peer group

analysis.  In general, propensity score methods use balancing factors (“propensity scores”) to offset an unequal

distribution of observable characteristics between comparison groups.  In this way, they help mitigate selection bias

by controlling indirectly for a bank’s motivation to participate in the SBLF program.   

For this program impact analysis, Treasury implemented a “propensity score matching” approach” . This approach

calculates the predicted probability that a given institution would have received SBLF funding given the many

variables that may have influenced the institution’s ability or decision to participate in the SBLF program.  This

predicted probability of participation is codified as a “propensity score” for each SBLF bank participant as well as each

non‐participant (6,0282 banks).  To assess the effect of SBLF participation, the analysis matches each SBLF bank with a

non‐SBLF bank that has the closest propensity score.  This process has the effect of creating a propensity score‐

matched “control group” of non‐SBLF banks that is balanced with SBLF banks across all relevant variables.  

Assembling the two groups in this manner controls for the influence of these characteristic variables by largely

eliminating any imbalances across the distribution of these variables among the two groups of banks.  This approach

explores whether, when controlling for propensity to participate in SBLF, banks that received SBLF capital increased

their business lending balances more than other banks.

Similar to other retrospective statistical techniques, propensity score methodologies carry inherent limitations.  For

example, propensity score matching techniques presume that all relevant characteristic variables are subject to

measurement and inclusion when calculating an institution’s propensity to participate in one or more groups.  

Consequently, it remains possible that unmeasured factors may influence differences in outcomes in a comparison of

                                                            

1 As established in the Act, the baseline for measuring the change in small business lending is the average of the amounts that were reported

for each of the four calendar quarters ended June 30, 2010. 2 Of the 6,035 non‐SBLF banks, seven institutions were excluded from the matching analysis due to missing values associated with the data used

to calculate propensity scores.
3

SBLF banks and non‐participants.3  The use, however, of a large set of observable variables – a total of 71 in the

context of this analysis – helps mitigate this risk.

In addition, the scope of this analysis is limited to an assessment of whether SBLF participants have increased

business lending in comparison to similarly situated institutions.  This analysis does not seek to evaluate, for example,

whether increases in business lending by program participants have been offset by decreases by non‐participants or

slower growth in non‐business lending by participants than may otherwise have occurred.  Likewise, this analysis

does not seek to capture or calculate potential second‐order benefits to a community from increased business

lending, such as increased job creation, capital investment, tax revenues, or other effects.  As well, this analysis is not

intended to address questions of capital access or availability, such as whether banks that participated in SBLF could

have raised capital from alternative sources, nor does it seek to address potential limitations on the aggregate

amount of capital available to community banks.

Selection of Characteristic Variables

Treasury incorporated a broad set of 71 observable financial and market variables that could potentially affect a

bank’s likelihood of participating in the SBLF program.  These variables include balance sheet measures, financial

performance measures, capitalization, loan composition, deposit composition, asset quality, liquidity, yield to cost

ratios, corporate organization, and participation in the Capital Purchase Program, as well as characteristics of the

institution’s lending markets.  These variables were measured as of March 31, 2011, which was the quarter before

the first SBLF investment.

To obtain financial information on each bank, Treasury accessed information from SNL Financial (SNL), a private

database that contains publicly‐filed regulatory and financial reports.  To obtain market population, unemployment,

and income information on the bank’s lending markets, Treasury accessed U.S. Census information using SNL.  For

banks located within a Metropolitan Statistical Area (MSA), MSA‐level information was used, while county‐level data

was used for banks outside of an MSA.

The following is a list of the 71 characteristic variables selected, grouped by category.  These variables are based on

SNL’s array of summary financials for each institution.

• Balance sheet measures (6 continuous4 variables):  total assets, asset growth rate, loan growth rate, deposit

growth rate, gross loans to assets, and loans to deposits.

• Performance measures (5 continuous variables):  return on average assets (ROAA), return on average equity

(ROAE), net interest margin, yield to cost spread, and efficiency ratio.  

• Capitalization (8 continuous variables and 1 categorical5 variable): equity to assets, tangible equity to

tangible assets, tangible common equity to tangible assets, risk based capital ratio, tier 1 risk based ratio, tier

1 common risk based ratio, leverage ratio, change in common and preferred stock and capital surplus as a

                                                            

3 Randomized experiments are best suited to assessing program effectiveness where it is possible to maintain random assignment to minimize

the influence of external factors on program outcomes.  The structure of the SBLF program, however, did not permit random assignment, as

Treasury considered all applicants for funding and sought to provide consistent decisioning among applicants. 4 A continuous variable is a numeric variable that can take on an infinite number of values within a certain range. 5

A categorical variable is a variable that can take on one of a finite number of distinct values that assigns each institution to a particular group

or category based on some qualitative property.
4

percentage of total equity between Q1 2010 and Q1 2011 (0% or less, 0 to 5%, 5 to 10%, over 10%) and

common dividends declared to net income.  

• Loan composition (6 continuous variables):  construction land development loans to total loans, total 1‐4

family loans to total loans, multifamily loans to total loans, total real estate loans to total loans, commercial

and industrial loans to total loans, and consumer loans to total loans.

• Deposit composition (5 continuous variables):  non‐interest bearing deposits to total deposits, transaction

accounts to total deposits, money market deposit account (MMDA) savings to total deposits, retail time

deposits to deposits, and jumbo time deposits to total deposits.

• Asset quality (11 continuous variables):  non‐current loans to loans, nonperforming loans to loans, non‐

performing assets to total assets, non‐performing assets excluding restructured loans to total assets, non‐

performing assets to loans plus foreclosed real estate, loan loss reserves to gross loans, net charge‐offs to

average loans, non‐performing assets plus loans 90 days past due to tangible equity plus loan loss reserves,

adjusted non‐performing assets plus adjusted loans 90 days past due divided by tangible equity plus loan loss

reserves, adjusted non‐performing assets to total assets, and adjusted non‐performing loans to total loans.

• Liquidity (4 continuous variables):  liquidity ratio, interest earning assets to interest bearing liabilities,

brokered deposits to deposits, and jumbo deposits to total domestic deposits.

• Yield to cost ratios (4 continuous variables):  yield to loans and leases, cost of interest bearing deposits, cost

of interest bearing liabilities, and cost of funds.

• Government program participation (3 categorical variables):  current or former participation in CPP

(participant; non‐participant), former participation in Transaction Account Guarantee Program (TAGP)

(participant; non‐participant), former participation in Debt Guarantee Program (TDGP) (participant; non‐

participant).

• Corporate organization (6 categorical variables and 1 continuous variable):  de novo status (de novo; non‐

de novo), ownership (private; public), ownership structure (mutual; stock corporation), institution type

(savings bank; commercial bank), primary regulator (Federal Reserve Board (FRB); Office of the Comptroller

of the Currency (OCC); Federal Deposit Insurance Corporation (FDIC)), Community Reinvestment Act rating

(outstanding, needs to improve, satisfactory), and bank age in years.    

• Local market indicators (1 categorical variable and 10 continuous variables ):  region (Mid‐Atlantic,

Midwest, Northeast, Southeast, Southwest, West), unemployment rate, change in unemployment rate from

prior year, 2011 compound annual growth rate (CAGR) of households, 2011 CAGR of population, 2010 CAGR

of median household income, 2010 CAGR of per capita income),  projected 2017 CAGR of households,

projected 2017 CAGR of median household income, projected 2017 CAGR of per capita income, and

projected 2017 CAGR of population.
5

Propensity Score Matching Analysis Methodology

Propensity score matching follows five primary steps: (1) evaluation of the balance of characteristic variables

between SBLF participants and non‐participants, (2) calculation of a propensity score for each institution, (3)

selection of a control group of banks using propensity scores, and (4) testing of the control group to assess its

effectiveness in mitigating selection bias, and (5) estimation of net effect of SBLF participation.

Step 1:  Evaluation of the Balance of Characteristic Variables Between SBLF Participants and Non‐Participants  

Treasury measured the statistical differences between the SBLF banks and the broader market of community banks

across each of the 71 characteristic variables listed above to identify which characteristics were equally distributed

(i.e., balanced) among SBLF participants and non‐participants.  To evaluate balance, the statistical differences in each

variable between the two groups of banks were calculated using chi‐squared tests for categorical variables and

Kolmogorov–Smirnov tests (K‐S tests) for continuous variables.  The analysis found statistically significant imbalances

(p‐values < 0.05) between SBLF and non‐SBLF banks for 53 of the 71 characteristic variables.  This result suggests that

SBLF banks have observable characteristics that differ significantly from the broader market of community banks.

Step 2:  Calculation of a Propensity Score for Each SBLF Participant and Non‐Participant

Treasury calculated a “propensity score” ranging between 0 and 1 for each SBLF bank participant and each non‐

participant based on the 71 characteristic variables described above.  The calculation of the propensity score for each

bank follows a standard logistic regression.  The dependent variable is defined as program status (non‐participant = 0,

SBLF participant = 1) and the independent variables are the 71 characteristic variables.  A backward selection process

was used to iteratively eliminate highly insignificant variables (i.e., p‐values > 0.60).6

  Through the backward variable

selection process, 17 variables were eliminated.  The initial test of propensity score’s ability to balance across SBLF

and the control group (see Step 5) indicated remaining imbalances for the asset growth rate, loan growth rate, and

ROAA variables, possibly suggesting a non‐linear relationship for these variables not captured in the initial model.  To

correct for this imbalance, Treasury added squared terms for each of these factors (a common approach in

propensity score modeling) and cubed variables for the loan growth rate and ROAA variables, which yielded five

additional coefficients for the logistic regression.   

Step 3:  Selection of a Control Group of Non‐Participants Using Propensity Scores  

Treasury selected a control group of non‐participants for comparison to SBLF institutions by matching each SBLF

participant with the non‐participant that had the next closest propensity score (propensity score‐matched “control

group”).  For example, if an SBLF participant had a propensity score of 0.30, the institution selected for inclusion in

the control group was the non‐participant that received a propensity score closest to 0.30.

                                                            

6 The variables eliminated were found to be insignificant in the presence of other correlated variables; individually, however, they may be

correlated with lending growth.  The eliminated variables included: deposit growth rate, ownership = private, regulator = OCC or FRB, ROAE,

projected CAGR households, tier 1 common risk based capital ratio, NPAs to total assets, cost of interest bearing deposits, yield on loans and

leases, jumbo time deposits to total deposits, CAGR per capita income, MMDA savings to total deposits, leverage ratio, projected CAGR per

capita income, institution type = savings bank, and ownership structure = mutual.  
6

Step 4:    Evaluation of the Balance of Characteristic Variables Between SBLF Participants and the Control Group

Treasury performed statistical tests to assess whether the control group and the SBLF participants had comparable

distributions with respect to each of the 71 characteristic variables.  The statistical differences in each variable

between the two groups of banks were calculated using chi‐squared tests for categorical variables and Kolmogorov‐

Smirnov tests (K‐S tests) for continuous variables (see Table A for detailed test results).  These tests found no

statistically significant imbalances (p‐values < 0.05) between the group of SBLF banks and the control group across

any of the 71 characteristic variables (see Table A for additional detail).  This suggests that the propensity score

matching analysis reduced the possibility of selection bias by selecting a statistically balanced control group.

Step 5:  Estimation of Net Effect of SBLF Participation

To estimate the net effect of the SBLF participation using this methodology, Treasury compared the median increases

in lending growth at SBLF banks to the median increases in lending growth and the control group over baseline and

since investment.  Consistent with the findings from the peer and comparison group analysis published in the

quarterly SBLF Lending Growth Reports, the results of the propensity score matching analysis suggest that the SBLF

program is supporting increased business lending among program participants.  By convention, the output of these

methodologies reflects differentials between groups and are not directly comparable with the results of the peer and

comparison group analyses presented in the quarterly SBLF Lending Growth Reports.

As of September 30, 2015, SBLF banks have increased business loans outstanding by a median 91.6 percent over

baseline levels, versus a 56.4 percent median increase for the propensity score‐matched control group, for an

estimated net effect associated with SBLF participation of 35.2 percentage points.  The following table shows changes

in business lending and other lending by SBLF banks and the propensity score matched control group.

   

Results of Propensity Score Matching Analysis

Median Lending of

SBLF Participants

Median  Lending of

Control Group

Estimated Net Effect of

SBLF Participation

Number of institutions 184 184

Median Change in Lending Over Baseline

    Business lending 91.6% 56.4% 35.2%

    Other lending 74.3% 51.8% 22.5%

Median Change in Lending Since Investment

    Business lending 59.0% 34.1% 24.8%

    Other lending 55.9% 39.6% 16.3%
7

Table A:   Summary Statistics of Characteristic Variables: Propensity Score Matching

The following tables includes additional detail of the chi‐square test results for the 11 categorical variables and the K‐

S test results for the 60 continuous variables used in the propensity score matching analysis.  All variables were

measured at March 31, 2011, the quarter before the initial SBLF investment.  

Summary of Chi‐Square Test Results for Continuous Variables and KS‐Test Results for Categorical Variables

Corporate organization Asset quality

  Banks Age in Yrs. < 0.01 0.11   Noncurrent Loans to Loans < 0.01 0.49

Balance sheet measures   NPLs to Loans < 0.01 0.34

  Total Assets < 0.01 0.07   NPAs to Total Assets < 0.01 0.57

  Asset Growth Rate < 0.01 0.11   NPA Excl. Restruct. Lns to Total Assets < 0.01 0.95

  Loan Growth Rate < 0.01 0.09   NPAs to Loans+Frclsd RE < 0.01 0.49

  Deposit Growth Rate < 0.01 0.09   Loan Loss Reserves to Gross Loans 0.05 0.90

  Gross Loans to Assets < 0.01 0.34   NCOs to Avg Loans 0.23 0.66

  Loans to Deposits < 0.01 0.41   NPA+ Loans 90PD to Tang Equity + LLR < 0.01 0.57

Performance measures Adj NPA + Adj Lns 90PD / Tang Equity + LL < 0.01 0.57

  ROAA < 0.01 0.14   Adjusted NPA to Total Assets < 0.01 0.49

  ROAE < 0.01 0.34   Adjusted NPL to Total Loans < 0.01 0.34

  Net Interest Margin 0.05 0.23 Local market indicators

  Yield to Cost Spread 0.02 0.14   Unemployment Rate 0.06 0.95

  Efficiency Ratio 0.44 0.28   Change in Unempl Rate from Prior Year 0.12 0.57

Capitalization   CAGR Population < 0.01 0.83

  Equity to Assets 0.04 0.75   CAGR Households <0.01 0.75

  Tangible Equity to Tangible Assets 0.01 0.90   CAGR Per Capita Income < 0.01 0.75

  Tang Common Eqty to Tang Assts < 0.01 0.98   CAGR Median Household Income < 0.01 0.95

  Risk Based Capital Ratio < 0.01 0.95   Proj CAGR Population < 0.01 0.95

  Tier 1 Risk‐based Ratio < 0.01 0.95   Proj CAGR Households < 0.01 0.95

  Tier 1 Common Risk‐Based Ratio < 0.01 0.98   Proj CAGR Per Capita Income < 0.01 0.57

  Leverage Ratio 0.09 0.49   Proj CAGR Median Household Income < 0.01 0.75

  Com Div Decl to Net Income < 0.01 0.14

Loan composition Government Program Participation

  Const Land Dev Lns to Loans < 0.01 0.34   TARP < 0.01 0.40

  Tot 1‐4 Fam Loans to Loans < 0.01 0.57   TAGP Participant < 0.01 0.45

  Multifamily Loans to Loans < 0.01 0.57   TDGP Participant < 0.01 0.07

  Total Real Est Lns to Tot Loans 0.06 0.66 Ownership

  CI Loans to Loans < 0.01 0.41   Private < 0.01 0.83

  Consumer Loans to Loans < 0.01 0.07   Ownership Structure < 0.01 0.70

Deposit composition   Company Type 0.16 0.85

  Non Int Bear Dep to Total Deposits 0.15 0.49 Regulatory

  Trans Accts to Total Deposits < 0.01 0.14   Primary Regulator 0.15 0.48

  MMDA Savings to Total Deposits < 0.01 0.49   CRA Rating 0.23 0.60

  Retail Time Deposits to Dep < 0.01 0.83 De Novo Status

  Jumbo Time Deposit to Total Dep 0.02 0.41   De Novo Status < 0.01 0.09

Liquidity Region

  Liquidity Ratio < 0.01 0.66   Region < 0.01 0.70

  Int Earn Assets to Int Bear Liab 0.18 0.66 Capital Raising Activity

  Brokered Deposits to Deposits < 0.01 0.49   Capital Raise Observed < 0.01 0.88

  Jumbo Deposit to Total Dom Dep 0.02 0.41

Yield to cost ratios

  Yield on Loans and Leases <0.01 0.90

  Cost of Int Bearing Deposits 0.07 0.15

  Cost of Interest Bearing Liab 0.19 0.83

  Cost of Funds 0.07 0.41

p‐value

Before PS

Correction

p‐value

After PS

Correction

p‐value

Before PS

Correction

p‐value

After PS

Correction
8

References

This program impact analysis follows a methodology drawn from prior research on program evaluation.  To complete

this analysis, Treasury worked with a third‐party contractor with expertise in statistical methods.  In addition, the

following publications include information used in developing the methodological approach.

1. Augurzky, B., and C. Schmidt (2000): “The Propensity Score: A Means to An End," Working Paper, University

of Heidelberg.

2. Caliendo, M., Kopeinig, S. (2005): “Some practical guidance for the implementation of propensity score

matching”. Paper 1588. Institute for the Study of Labor, IZA.

3. Dehejia, R. H., and S. Wahba (1999): “Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation

of Training Programs," Journal of the American Statistical Association, 94(448), 1053‐1062.

4. Dehejia, R. H., and S. Wahba (2002): “Propensity Score Matching Methods for Nonexperimental Causal

Studies," The Review of Economics and Statistics, 84(1), 151‐161.

5. Rossman, S.B., Rempel, M., Roman, J.K., Zweig, J.M., Lindquist, C.H., Green, M., Downey, P.M., Bhati, A.S., &

Farole, D.J., Jr. (2011): The multi‐site adult drug court evaluation: The impact of drug courts. Washington,

D.C.: The Urban Institute.

6. Spreeuwenberg, M., Bartak, A., Croon, M., Hagenaars, J., Busschbach, J., Andrea, H., Twisk, J, and Stijnen, T.

(2010): “The Multiple Propensity Score as Control for Bias in the Comparison of More Than Two Treatment

Arms: An Introduction From a Case Study in Mental Health”. Medical Care, 48 (2), 166‐174.

7. Imbens, G.W. (2000): “The Role of the Propensity Score in Estimating Dose‐Response Functions.” Biometrika,

87(3), 706‐710.

8. Rosenbaum, P. and Rubin, D. (1983): “The Central Role of the Propensity Score in Observational Studies for

Causal Effects.” Biometrika, 70 (1), 41‐55.