GAO 

United States Government Accountability Office

Report to Congressional Requesters

SMALL BUSINESS 

ADMINISTRATION 

Actions Needed to 

Improve the 

Usefulness of the 

Agency’s Lender Risk 

Rating System 

November 2009 

GAO-10-53 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

November 2009

SMALL BUSINESS ADMINISTRATION

Actions Needed to Improve the Usefulness of the 

Agency's Lender Risk Rating System 

Highlights of GAO-10-53, a report to 

congressional requesters 

The Small Business Administration 

(SBA) guarantees individual loans 

that lenders originate. The agency 

uses its Loan and Lender 

Monitoring System (L/LMS) to 

assess the individual risk of each 

loan, and SBA’s contractor 

developed a lender risk rating 

system based on L/LMS data. 

However, questions have been 

raised about the extent to which 

SBA has used its lender risk rating 

system to improve its oversight of 

lenders. GAO was asked to 

examine (1) how SBA's risk rating 

system compares with those used 

by federal financial regulators and 

lenders and the system’s usefulness 

for predicting lender performance 

and (2) how SBA uses the lender 

risk rating system in its lender 

oversight activities. To meet these 

objectives, GAO reviewed SBA 

documents; interviewed officials 

from three federal financial 

regulators and 10 large SBA 

lenders; analyzed SBA loan data; 

and interviewed SBA officials. 

What GAO Recommends

GAO recommends that SBA ensure 

that its contractor, consistent with 

industry standards, follows sound 

model validation practices, use its 

own data to assess the lender risk 

rating system, develop a strategy 

for targeting lenders for on-site 

reviews that relies more on its 

lender risk ratings, and consider 

revising its on-site review policies 

and procedures. In responding to a 

draft of this report, SBA generally 

agreed with these 

recommendations and outlined 

some steps that it plans to take to 

address them. 

SBA’s lender risk rating system uses some of the same types of information that 

federal financial regulators and selected large lenders use to conduct off-site 

monitoring, but its usefulness has been limited because SBA has not followed 

common industry standards when validating the system—that is, assessing the 

system’s ability to accurately predict outcomes. Like the federal financial 

regulators and 10 large lenders GAO interviewed, SBA’s contractor developed 

lender risk ratings based on loan performance data and prospective, or forwardlooking, measures (such as credit scores). Using SBA data, GAO undertook a 

number of evaluative steps to test the lender risk rating system’s predictive ability. 

GAO found that the system was generally successful in distinguishing between 

higher- and lower-risk lenders, but it better predicted the performance of larger 

lenders. However, the system’s usefulness was limited because the contractor did 

not follow validation practices, such as independent and ongoing assessments of 

the system’s processes and results, consistent with those recommended by federal 

financial regulators and GAO’s internal control standards. For example, the agency 

did not require a party other than the one who developed the system to perform 

the validation, and SBA’s contractor did not routinely reassess the factors used in 

the system as part of its validations. Further, SBA does not use its own data to 

develop alternate measures of lender performance that could be used to 

independently assess or supplement the risk ratings, citing resource constraints. 

Because SBA does not follow sound validation practices or use its own data to 

independently assess the risk ratings, the effectiveness of its lender risk rating 

system—the primary system SBA relies on to monitor and predict lender 

performance—may deteriorate as economic conditions and industry trends 

change over time. 

Although SBA’s lender risk rating system has enabled the agency to conduct some 

off-site monitoring of lenders, the agency does not use the system to target lenders 

for on-site reviews or to inform the scope of the reviews. Unlike the Federal 

Deposit Insurance Corporation and the Federal Reserve, which use their off-site 

monitoring tools to target lenders for on-site reviews, SBA targets for review those 

lenders with the largest SBA-guaranteed loan portfolios. As a result of this 

approach, 97 percent of the lenders that SBA’s risk rating system identified as high 

risk in 2008 were not reviewed. Further, GAO found that the scope of the on-site 

reviews that SBA performs is not informed by the lenders’ risk ratings, and the 

reviews do not include an assessment of lenders’ credit decisions. The federal 

financial regulators use the results of off-site monitoring to identify which areas of 

a bank’s operations they should review more closely. Moreover, their reviews 

include an assessment of the quality of the lenders’ credit decisions. Federal 

financial regulators are able to use review results to update their off-site 

monitoring systems with data on emerging lending trends. Regardless of the 

lender’s risk rating, SBA relies on a standard on-site review form that includes an 

assessment of lenders’ compliance with SBA policies and procedures but not an 

assessment of lenders’ credit decisions. According to SBA officials, it is not the 

agency’s role to assess lenders’ credit decisions. Without targeting the most risky 

lenders for on-site reviews or gathering information related to lenders’ credit 

decisions, SBA cannot effectively assess the risk posed by lenders or ensure that 

its lender risk rating system incorporates updated information on emerging 

lending trends. 

View GAO-10-53 or key components.

For more information, contact William B. 

Shear at (202) 512-8678 or shearw@gao.gov. 
Page i GAO-10-53 

Contents 

Letter 1

Results in Brief 4 

Background 6 

SBA’s Lender Risk Rating System Is Similar to Those Used by 

Federal Financial Regulators but Is Limited by Insufficient 

Validation 10 

SBA Does Not Use Lender Risk Ratings to Target Lenders for OnSite Review or Tailor the Scope of the Reviews 26 

Conclusions 34 

Recommendations for Executive Action 35 

Agency Comments and Our Evaluation 36 

Appendix I Objectives, Scope, and Methodology 39 

Appendix II Comments from the Small Business Administration 44 

Appendix III Predictive Performance of the March 2007 and 

March 2008 Lender Risk Ratings 48 

Appendix IV Small Business Predictive Score 51 

Appendix V GAO Contact and Staff Acknowledgments 53 

Tables 

Table 1: Sources of Data Used to Calculate Lender Risk Ratings for 

7(a) Lenders 9 

Table 2: Comparison of Alternative Rankings and Rankings Based 

on 2007 Lender Risk Rating Raw Scores, 2007 Currency 

Rates, and 2008 Lender Risk Rating Raw Scores for 7(a) 

Lenders 49 

 SBA's Lender Risk Rating System 
Table 3: Comparison of Alternative Rankings and Rankings Based 

on 2007 Lender Risk Rating Raw Scores, 2007 Currency 

Rates, and 2008 Lender Risk Rating Raw Scores for 504 

Lenders 49 

Table 4: Results of Correlation Analysis 50 

Table 5: Predictive Ability of SBPS for Loans below and above 

$150,000 52 

Figures 

Figure 1: SBA’s Lender Risk Rating Process for 7(a) Lenders 12 

Figure 2: Data Used for Off-Site Monitoring 15 

Figure 3: Commonly Accepted Validation Practices and SBA’s 

Practices 18 

Figure 4: SBA On-Site Reviews, 2005 to 2008 30 

Abbreviations 

Basel Committee Basel Committee on Banking Supervision 

FDIC Federal Deposit Insurance Corporation 

Federal Reserve Board of Governors of the Federal Reserve System 

L/LMS Loan and Lender Monitoring System 

NAICS North American Industry Classification System 

OCC Office of the Comptroller of the Currency 

SBA Small Business Administration 

SBPS Small Business Predictive Score 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately. 

Page ii GAO-10-53 SBA's Lender Risk Rating System
Page 1 GAO-10-53 

 

United States Government Accountability Office

Washington, DC 20548 

 

November 6, 2009 

The Honorable Mary L. Landrieu 

Chair 

The Honorable Olympia J. Snowe 

Ranking Member 

Committee on Small Business and Entrepreneurship 

United States Senate 

The Honorable Richard J. Durbin 

Chairman 

The Honorable Susan M. Collins 

Ranking Member 

Subcommittee on Financial Services and General Government 

Committee on Appropriations 

United States Senate 

In April 2003, the Small Business Administration (SBA) obtained a loan 

monitoring service from Dun & Bradstreet to help manage and oversee the 

lending and risk management activities of lenders that extend 7(a) and 504 

loans to small businesses. The 7(a) and 504 loan programs, named after 

the sections of the acts that authorized them, are SBA’s two major 

business loan guarantee programs.1

 As of June 30, 2009, SBA had an 

outstanding portfolio of $67.6 billion in 7(a) and 504 loans. Because SBA 

guarantees the individual loans that lenders originate, it uses the Dun & 

Bradstreet service, now called the Loan and Lender Monitoring System 

(L/LMS), to monitor the individual risk that each loan poses to the agency 

in order to identify those lenders whose SBA loan operations and 

portfolios may require additional monitoring or other actions. In 2004, we 

reviewed the service and found that it was a positive and necessary step in 

improving SBA’s oversight of lenders but determined that the agency 

needed to develop policies and procedures to ensure that it used the 

service in a way that resulted in improved oversight of lenders.2

 Since we 

1

The proceeds of 7(a) loans may be used for working capital and other general business 

purposes, while the proceeds of 504 loans may be used for fixed capital. Section 7(a) of the 

Small Business Act, as amended, codified at 15 U.S.C. § 636(a); Section 504 of the Small 

Business Investment Act of 1958, as amended, codified at 15 U.S.C. § 696. 

2

GAO, Small Business Administration: New Service for Lender Oversight Reflects Some 

Best Practices, but Strategy for Use Lags Behind, GAO-04-610 (Washington, D.C.: June 8, 

2004). 

 SBA's Lender Risk Rating System 
 

issued our report in June 2004, SBA has made progress in developing 

policies for using L/LMS and expanding its use. For example, SBA hired a 

contractor to develop a lender risk rating system (that is, an off-site 

monitoring tool that produces a risk score for each lender) based on 

L/LMS data. This system enabled SBA for the first time to monitor the 

approximately 4,000 smaller lenders that it had not previously reviewed. 

However, questions have been raised about the extent to which SBA has 

used its lender risk rating system to improve its oversight of lenders—for 

example, to target lenders for on-site review. The SBA Inspector General 

reported in May 2008 that SBA had been unable to sufficiently mitigate the 

risk posed by lenders that it had identified as high risk and that SBA’s 7(a) 

program had incurred a cumulative net loss for four lenders of $329 

million as of September 2007.3

You asked us to review SBA’s lender risk rating system and its effect on 

the agency’s lender oversight program. Specifically, this report examines 

(1) how SBA’s risk rating system compares with the off-site monitoring 

tools used by federal financial regulators and lenders and the system’s 

usefulness for predicting lender performance and (2) how SBA uses the 

lender risk rating system in its lender oversight activities. 

To determine how SBA’s lender risk rating system compares with off-site 

monitoring tools used by federal financial regulators and lenders, we 

compared SBA’s system with common industry standards that we 

identified through interviews and document reviews. We interviewed 

officials from three federal financial regulators—the Office of the 

Comptroller of the Currency (OCC), the Board of Governors of the Federal 

Reserve System (Federal Reserve), and the Federal Deposit Insurance 

Corporation (FDIC)—five of the largest 7(a) lenders, and the five largest 

504 lenders.4

 We also reviewed relevant literature and analyzed procedural 

manuals and other related federal guidance to banks on loan portfolio 

monitoring. Although we interviewed federal financial regulators and 

reviewed agency documents explaining their off-site monitoring practices, 

 

3

SBA, Office of Inspector General, Oversight of SBA Supervised Lenders, Report no. 8-12 

(Washington, D.C.: May 9, 2008). 

4

The federal financial regulators we selected have policies and procedures for monitoring 

credit risk that are relevant to SBA. We focused on the largest lenders because they would 

be the most likely to have off-site monitoring tools similar to SBA’s lender risk rating 

system. According to SBA, there are approximately 5,000 SBA lenders. Although our 

sample of 10 large lenders is nongeneralizable, it offers perspectives on how some lenders 

conduct off-site monitoring. 

Page 2 GAO-10-53 SBA's Lender Risk Rating System
 

we did not evaluate their practices, such as by testing their models. In 

addition, we compared the techniques that SBA and its contractor used to 

develop and validate the lender risk rating system to our internal control 

standards.5

 To determine the usefulness of the lender risk ratings in 

predicting lender performance, we reviewed documents from SBA and its 

contractor that described the factors used in the risk rating system and the 

process for calculating the risk rating scores. We also obtained and 

analyzed the following SBA data: data on loans approved in 2003 through 

the end of 2007, the March 2007 and March 2008 lender risk ratings, and 

the currency rate for each lender.6

 We assessed the reliability of these data 

and found them to be sufficiently reliable for our purposes. Using these 

data, we undertook a number of evaluative steps to test SBA’s model. After 

we discussed SBA’s modeling approach in detail with SBA officials and the 

agency’s contractor to document the process used to develop the model, 

we developed statistical estimation techniques to assess how well SBA’s 

risk rating system predicts lender performance. In particular, we 

compared the scores from the lender risk rating system to lenders’ actual 

performance and alternate measures of lender performance that we 

developed using SBA data. To determine how SBA uses the lender risk 

rating system in its lender oversight activities, we compared SBA’s 

practices for assessing and monitoring the risk of lenders and loan 

portfolios against (1) the industry standards we identified through our 

interviews and document reviews and (2) our internal control standards. 

We also obtained and analyzed SBA data on risk ratings and on-site 

examinations from 2005 through 2008 to determine the characteristics of 

lenders that received on-site exams. 

We conducted this performance audit from August 2008 to November 2009 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

 

5

GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00.21.3.1 

(Washington, D.C.: November 1999) and Internal Control Management and Evaluation 

Tool, GAO-01-1008G (Washington, D.C.: August 2001). 

6

The currency rate is the sum of the dollar balance of guaranteed loans that are less than 30 

days past due divided by the dollar balance of the total portfolio of guaranteed loans 

outstanding. 

Page 3 GAO-10-53 SBA's Lender Risk Rating System
 

conclusions based on our audit objectives. Appendix I contains a full 

description of our objectives, scope, and methodology. 

SBA’s lender risk rating system uses some of the same types of 

information that federal financial regulators and selected large lenders use 

to conduct off-site monitoring. But the system’s usefulness has been 

limited because SBA has not followed common industry standards when 

validating the system—that is, assessing the system’s ability to accurately 

predict outcomes. Like the 3 federal financial regulators and 10 large 

lenders we interviewed, SBA’s contractor developed the lender risk rating 

system using loan performance data and prospective, or forward-looking, 

measures (such as credit scores). We independently assessed the lender 

risk rating system and found that it was generally successful in 

distinguishing between high- and low-risk lenders, but it better predicted 

the performance of larger lenders. However, the system’s usefulness was 

limited because the contractor did not follow validation practices, such as 

independent and ongoing assessments of the system’s processes and 

results, consistent with those recommended by federal financial regulators 

and our internal control standards. For example, the agency did not 

require a party other than the one who developed the system to perform 

the validation, and SBA’s contractor did not routinely reassess the factors 

used in the system as part of its validations. Further, SBA officials stated 

that resource constraints prevented them from using internally generated 

data to develop alternate measures of lender performance that could be 

used to independently assess or supplement the risk ratings. Federal 

financial regulator guidance and our internal control standards suggest 

that organizations should use their own data to assess the performance of 

risk rating systems developed by vendors. Because SBA does not follow 

sound validation practices or use its own data to independently assess the 

risk ratings, the effectiveness of its lender risk rating system—the primary 

system SBA relies on to monitor and predict lender performance—may 

deteriorate as economic conditions and industry trends change over time. 

According to SBA officials, the agency’s contractor is currently 

redeveloping the system because its performance has deteriorated in 

recent years. 

Results in Brief 

Although SBA’s lender risk rating system has enabled the agency to 

perform some off-site monitoring of lenders, the agency does not use the 

system to target lenders for on-site review or to inform the scope of those 

reviews. FDIC and the Federal Reserve use their off-site monitoring tools 

to target lenders for on-site reviews. SBA uses its risk rating system to 

monitor lenders and portfolio trends but does not rely on it to target the 

Page 4 GAO-10-53 SBA's Lender Risk Rating System
 

riskiest 7(a) and 504 lenders for on-site review. Instead, SBA focuses on 

what it thinks is the most important risk indicator—portfolio size—and 

targets for review those lenders with the largest SBA-guaranteed loan 

portfolios—that is, 7(a) lenders with at least $10 million in their 

guaranteed loan portfolio and 504 lenders with balances of at least $30 

million. Of the 477 reviews SBA conducted from 2005 through 2008, 380 

(80 percent) were of large lenders that, based on its lender risk rating 

system, posed limited risk to SBA. The remaining 97 reviews (20 percent) 

were of lenders that posed significant risk to the agency. As a result, the 

vast majority of high-risk lenders were not reviewed. For example, in 2008, 

97 percent of the 1,587 lenders identified as high risk were not reviewed. 

Of these lenders, 215 had an outstanding portfolio of at least $4 million. 

Because SBA relies on a lender’s size to target lenders for on-site reviews, 

smaller lenders with high-risk ratings that may still have significant 

portfolios of SBA loans have been allowed to participate in SBA’s loan 

programs with little or no oversight. In addition, SBA does not use the 

lender risk rating system to determine the scope of on-site reviews and 

does not assess lenders’ credit decisions during these reviews. Federal 

financial regulators we contacted use the results of off-site monitoring to 

identify which areas of a bank’s operations they should review more 

closely. Moreover, their reviews include an assessment of the quality of 

lenders’ credit decisions. These practices provide information on emerging 

trends in lending that regulators can use to update their off-site monitoring 

tools. Finally, internal control standards require that all federal agencies 

identify and analyze risks and determine the best way to manage or 

mitigate them. However, regardless of lenders’ risk ratings, SBA relies on a 

standard on-site review form that includes an assessment of lenders’ 

compliance with SBA policies and procedures but not an assessment of 

lenders’ credit decisions. For example, SBA examiners determine whether 

lenders have ensured that borrowers met eligibility requirements. SBA 

officials told us that it was not the agency’s role to assess lenders’ credit 

decisions. However, we believe that because SBA relies on lenders with 

delegated underwriting authority to make the majority of its loans, the 

agency should take a more active role in ensuring that these lenders are 

making sound credit decisions. Without targeting the riskiest lenders for 

on-site reviews or gathering information related to lenders’ credit 

decisions, SBA cannot effectively assess lenders’ risk or update its risk 

rating system based on emerging lending trends. 

This report contains four recommendations designed to improve SBA’s 

use of its lender risk rating system and oversight of its lenders. We are 

recommending that SBA ensure that its contractor follows sound model 

validation practices, including testing of the lender risk rating system data, 

Page 5 GAO-10-53 SBA's Lender Risk Rating System
 

processes, and results; utilizing an independent party to perform its 

validations; and maintaining complete documentation of the validation 

process and results. We also are recommending that SBA use its own data 

to assess the lender risk rating system, develop a strategy for targeting 

lenders for on-site reviews that relies more on its lender risk ratings, and 

consider revising its on-site review policies and procedures. We provided 

SBA with a draft of this report for its review and comment. In written 

comments, SBA stated that it generally agreed with our recommendations 

and outlined some steps that it plans to take to address them. For 

example, the agency noted that it is currently undertaking a 

redevelopment of its lender risk rating system and plans to ensure that 

best practices are incorporated into the redevelopment validation process. 

SBA’s comments are reprinted in appendix II. 

In pursuing its mission of aiding small businesses, SBA provides them with 

access to credit, primarily by guaranteeing loans through its 7(a) and 504 

loan programs. The 7(a) and 504 loan guarantee programs are intended to 

serve small business borrowers who could not otherwise obtain credit 

under reasonable terms and conditions from the private sector without an 

SBA guarantee. Under the 7(a) program, SBA generally provides 

guarantees of up to 85 percent on loans made by participating lenders that 

are subject to program oversight by SBA.7

 Many of these participating 

lenders are preferred lenders that have delegated underwriting authority. 

Loan proceeds can be used for most business purposes, including working 

capital, equipment, furniture and fixtures, land and buildings, leasehold 

improvements, and certain debt refinancing. The 504 program provides 

long-term, fixed-rate financing to small businesses for expansion or 

modernization, primarily of real estate. Financing for 504 loan programs is 

delivered through about 270 certified development companies, nonprofit 

corporations that were established to contribute to the economic 

development of their communities. For a typical 504 loan project, a thirdparty lender provides 50 percent or more of the financing pursuant to a 

first-lien mortgage, a certified development company provides up to 40 

percent of the financing through a debenture that is fully guaranteed by 

Background 

 

7

The American Recovery and Reinvestment Act of 2009 authorized SBA to temporarily 

increase the maximum 7(a) guarantee from 85 percent to 90 percent. SBA lenders consist 

of private banks, credit unions, and small business lending companies. Small business 

lending companies are nondepository institutions licensed by SBA that are not subject to 

state or federal supervision or examination other than oversight conducted by SBA. 

Page 6 GAO-10-53 SBA's Lender Risk Rating System
 

SBA, and a borrower contributes at least 10 percent of the financing.8

Although SBA’s 7(a) and 504 loan guarantee programs serve different 

needs, both programs rely on third parties to originate loans (participating 

lenders for 7(a) loans and certified development companies for 504 loans). 

Because SBA generally guarantees up to 85 percent of the 7(a) loans and 

up to 40 percent of the financing for 504 loan projects, SBA faces the same 

kind of risk as the lenders if the loans are not repaid. 

The Small Business Programs Improvement Act of 1996 required SBA to 

establish a risk management database that would provide timely and 

accurate information to identify loan underwriting, collections, recovery, 

and liquidation problems.9

 In 2003, SBA obtained a service from Dun & 

Bradstreet that would allow it to, among other things, predict the 

likelihood of a loan defaulting using a combination of SBA performance 

data and loan-level credit data. In 2004, we assessed the new service and 

found that the system was on par with industry best practices by providing 

a tool that could help SBA better assess the risk exposure of loans in its 

lenders’ portfolios.10 For example, we reported that the Small Business 

Predictive Score (SBPS), which is provided through the Dun & Bradstreet 

service, appeared to be consistent with private sector best practices 

because it was based on sound models.11 The models used to score the 

loans rely on data managed by Dun & Bradstreet and are commercial, offthe-shelf risk scoring models developed by Fair Isaac and validated to 

SBA’s 7(a) and 504 portfolios. We concluded that without the Dun & 

Bradstreet service, it was unlikely that SBA would be able to continue the 

same level of risk management of its overall portfolio, its individual 

lenders, and their portfolios. However, we also reported that SBA needed 

to make better use of the service in overseeing its lenders and 

recommended, among other things, that resources within SBA be devoted 

to developing policies for the use of the loan monitoring service. As a 

result, SBA contracted with Dun & Bradstreet to develop a system that 

would rate lenders based on risk. Dun & Bradstreet subcontracted with 

another company, TrueNorth, to develop the lender risk ratings—that is, 

 

8

A debenture is an unsecured debt backed only by the creditworthiness of the borrower. 

Debentures have no collateral, and SBA takes a junior lien position on the project property. 

The yields may vary from high to low, depending on who backs the debenture. 

9

Public Law No. 104-208, Div. D, § 102, 110 Stat. 3009-724, 3009-725, codified at 15 U.S.C. § 

633, as amended. 

10GAO-04-610. 

11The SBPS predicts the likelihood of a loan becoming severely delinquent. 

Page 7 GAO-10-53 SBA's Lender Risk Rating System
 

custom scores calculated using L/LMS data. Work on the lender risk rating 

system started in 2004. 

The purpose of the lender risk rating system is to improve the way SBA 

monitors lenders. The lender risk rating system uses the following factors 

for 7(a) lenders: 

• past 12 months’ actual purchase rate—a historical measure of SBA 

purchases from the lender in the preceding 12 months;12

• problem loan rate—the current delinquencies and liquidations in a lender’s 

SBA-guaranteed portfolio;13

• 3-month change in SBPS—a score that was developed to predict the 

likelihood of severe delinquency (61 or more days past terms) over the 

next 18 to 24 months, including bankruptcies and charge-offs;14 and 

• projected purchase rate—a measure of the amount of SBA guaranteed 

dollars in a lender’s portfolio that is likely to be purchased by SBA.15

Most of the data used to calculate these factors are loan and lender 

performance information that come from SBA. The remaining data are 

SBPSs or related scores provided by the Dun & Bradstreet service (see 

table 1). 

 

12When a loan defaults, the lender asks SBA to honor the guarantee (that is, purchase the 

loan). The 12 months’ actual purchase rate is calculated by dividing total gross dollars of 

the lender’s loans purchased during the past 12 months by the sum of total gross 

outstanding dollars of SBA loans at the end of the 12-month period and total gross dollars 

purchased during the past 12 months. 

13The problem loan rate is calculated by dividing the sum of total gross outstanding dollars 

of a lender’s loans that are 90 days or more delinquent and gross dollars in liquidation by 

total gross dollars outstanding. 

14According to SBA officials, the SBPS was validated to be predictive of loan purchases, as 

well as delinquencies. 

15The projected purchase rate is calculated by multiplying the amount of a lender’s 

guaranteed loan dollars outstanding by the probability of their purchase. This total is then 

divided by the lender’s total SBA-guaranteed dollars outstanding. 

Page 8 GAO-10-53 SBA's Lender Risk Rating System
 

Table 1: Sources of Data Used to Calculate Lender Risk Ratings for 7(a) Lenders 

Data sources 

Factor Lender data SBA 

Dun & 

Bradstreet Fair Isaac

Total gross dollars of the lender’s loans that were 

purchased during the past 12 months 

Past 12 months’ actual X 

purchase rate 

Total gross outstanding dollars of SBA loans at the end 

of 12-month period 

X 

Gross outstanding dollars of the lender’s loans that are 

90 days or more delinquent 

X 

Gross dollars in liquidation X 

Problem loan rate 

Gross dollars outstanding X 

3-month change in SBPS SBPS X X 

Probability of loan purchase X X 

Individual loans outstanding X 

Projected purchase rate 

SBA-guaranteed dollars outstanding X 

Source: GAO analysis of SBA data. 

For 504 lenders, the risk rating is based on three factors: (1) the past 12 

months’ actual purchase rate, (2) the problem loan rate, and (3) the 

average SBPS on loans in the 504 lender’s portfolio. The third factor 

replaced the third and fourth factors used for 7(a) lenders because it was 

found during the testing process to be more predictive of SBA purchases 

for 504 lenders. 

Some federal financial regulators and lenders rely on similar tools to 

conduct off-site monitoring. For example, FDIC relies on various off-site 

monitoring tools, including a system called the Statistical CAMELS Off-site 

Rating that helps the regulator identify institutions that have experienced 

noticeable financial deterioration since the last on-site exam. The Federal 

Reserve also relies on multiple tools to conduct off-site monitoring, 

including a system that enables the regulator to predict how the risk level 

of a bank likely will change in comparison to other banks that received 

similar ratings on on-site exams. OCC relies on a process called a core 

assessment that helps examiners assess the risk exposure for nine 

categories of risk, including quantity, quality, and direction of risk. 

Moreover, lenders frequently use models to summarize available relevant 

information about borrowers and reduce the information into a set of 

ordered categories, or scores, that estimate the borrower’s risk of 

delinquency or default at a given point in time. Such tools are playing a 

progressively more important role in the banking industry. In general, the 

Page 9 GAO-10-53 SBA's Lender Risk Rating System
 

goal of these models—whether they are generic or custom, developed 

internally or by third parties—is to obtain early indications of increasing 

risk. 

SBA’s Lender Risk 

Rating System Is 

Similar to Those Used 

by Federal Financial 

Regulators but Is 

Limited by 

Insufficient Validation 

Page 10 GAO-10-53 SBA's Lender Risk Rating System
 

SBA’s contractor takes four steps to assign lender risk ratings each 

quarter. First, the contractor separates lenders into peer groups based on 

the size of their SBA loan portfolios in order to compare similarly sized 

lenders. Second, for each lender, the contractor computes values for each 

of the factors. As discussed in more detail in the background, the four 

factors for 7(a) lenders are the (1) past 12 months’ actual purchase rate, 

(2) problem loan rate, (3) 3-month change in the SBPS, and (4) projected 

purchase rate. Third, the contractor inputs the value for each of the factors 

into an equation to compute a score for each lender. Fourth, the 

contractor uses the scores to place lenders into one of five risk rating 

categories (1 through 5, with 1 indicating the least risk).16 Figure 1 

illustrates this process for 7(a) lenders, and the shaded area represents a 

specific example. The process is generally the same for 504 lenders.17

SBA’s Contractor Uses a 

Multistep Process to 

Assign Lender Risk Ratings 

 

16According to SBA, lenders with a risk rating of 1 are considered strong in every respect 

and typically score well above their peer group averages for all or nearly all of the rating 

factors. The SBA operations of an SBA lender rated as a 2 are considered good and 

typically are above average for all or nearly all of the rating factors. Similar to lenders rated 

as a 2, lenders rated as a 3 are considered about average for all or nearly all of the rating 

factors but have room for improvement, should monitor their portfolios closely, and should 

consider methods to improve loan performance. Lenders rated as a 4 or 5 are considered 

below or well-below average, respectively, for all or nearly all rating factors that are used 

to calculate the lender risk ratings. 

17The process for assigning lender risk ratings to 504 lenders differs from the process for 

7(a) lenders in two ways. First, the 504 lender risk ratings are based on three factors: (1) 

past 12 months’ actual purchase rate, (2) problem loan rate, and (3) average SBPS of each 

lender’s portfolio. Second, the peer groups are sized differently. The 504 peer groups 

consist of lenders with portfolios of (1) $100,000,000 or more; (2) $30,000,000 to 

$99,999,999; (3) $10,000,000 to $29,999,999; (4) $5,000,000 to $9,999,999; and (5) less than 

$5,000,000. 

Page 11 GAO-10-53 SBA's Lender Risk Rating System
 

Figure 1: SBA’s Lender Risk Rating Process for 7(a) Lenders 

3

1

2 A B C D A B C D A B C D A B C D A B C D A B C D

Source: GAO.

Separates lenders into peer 

groups based on SBA loan 

portfolio size

Computes value of each 

factor for each 7(a) lender

Computes lenders’ scores

(1 - 999) by inputting each

lender’s value of each 

factor into an equation

Places lenders, based

on their scores, into 

five risk rating

categories

(1 through 5, 

with 1 indicating

the least risk)

$0 - $999,999

<1 loan

disbursed

$0 - $999,999

>1 loan

disbursed

$1M - $3.9M $4M - $9.9M $10M - $99.9M $100M or more

Peer

group

loan

performance

Lender

risk score

$0 - $999,999

<1 loan

disbursed

$0 - $999,999

>1 loan

disbursed

$1M - $3.9M $4M - $9.9M $10M - $99.9M $100M or more

Sample 

7(a) lender

SBA loan

portfolio: $7.8M

A

B

C

D

Projected purchase rate

Problem loan rate

3-month change in SBPS

Past 12 months’ actual purchase rate

250

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

2

1 2 3 4 5 1 2 3 4 5

Low High

4

Note: In step 2, the size of the symbols that represent each factor is illustrative and not necessarily to 

scale. 

Page 12 GAO-10-53 SBA's Lender Risk Rating System
 

According to SBA officials, this process for calculating lender risk ratings 

will likely change in the near future because its contractor is redeveloping 

the lender risk rating system. Several major changes are being 

contemplated. First, the contractor plans to use an updated version of the 

SBPS. Second, the contractor may use additional variables to calculate 

lender risk ratings. Finally, rather than varying the equation by peer group, 

SBA officials stated that they are considering a new variable that captures 

the size of the lender’s portfolio and the age of its loans. The contractor is 

still in the process of designing, testing, and documenting the new risk 

rating system. 

SBA rarely overrides risk ratings, but it may do so for several reasons. 

These include early loan default trends; abnormally high default or 

liquidation rates; lending concentrations; rapid growth in SBA lending; 

inadequate, incomplete, or untimely reporting to SBA; and nonpayment of 

required fees to SBA.18 In addition, SBA may override a lender risk rating 

due to issues identified during an on-site review. For the quarter ending 

September 30, 2008, SBA overrode the risk rating assigned by the 

contractor in 20 cases; in each case, the risk rating increased. 

SBA’s Lender Risk Rating 

System Uses Some of the 

Same Types of Data That 

Federal Financial 

Regulators and Selected 

Lenders Rely on to 

Conduct Off-Site 

Monitoring 

SBA’s lender risk rating system uses some of the same types of data that 

federal financial regulators and selected lenders rely on for off-site 

monitoring. The federal financial regulators we interviewed rely on lender 

information, performance data, and prospective measures to conduct offsite monitoring. Although the specific factors included in each regulator’s 

off-site monitoring tools can vary, each regulator uses these three types of 

data. Much of the lender and performance information they use are from 

the call reports that banks submit quarterly and include data on equity, 

loans past due, and charge-offs.19 Prospective measures include—when 

available—borrowers’ credit scores from lender files. One federal 

regulator is also working with a third party to obtain predictive scores, 

similar to the SBPS, to use as part of its off-site monitoring. The large 

lenders with whom we spoke also use performance data to rate loans, 

 

18SBA lenders are required to report monthly to SBA on the status of their SBA-guaranteed 

portfolio. To offset some of the costs of the 7(a) program, SBA assesses lenders two fees 

on each 7(a) loan, an up-front guarantee fee that may be passed on to the borrower and an 

annual servicing fee. 15 U.S.C. §§ 636(a)(23), (18). 

19Call reports are quarterly reports that collect basic financial data on commercial banks in 

the form of a balance sheet and income statement (formally known as Report of Condition 

and Income). 

Page 13 GAO-10-53 SBA's Lender Risk Rating System
 

focusing on factors such as portfolio performance, delinquencies, and 

trends by state and industry type in order to forecast future losses. 

Lenders also incorporate prospective measures, such as FICO scores and 

SBPSs.20

Like federal financial regulators and large lenders, SBA uses performance 

data and prospective measures to calculate lender risk ratings. As we have 

seen, to calculate risk ratings for 7(a) lenders, SBA relies on performance 

data (the past 12 months’ actual purchase rate and the problem loan rate) 

and prospective measures (the 3-month change in the SBPS and the 

projected purchase rate). The 3-month change in the SBPS is also a 

portfolio trend that has been incorporated into the rating system. 

However, unlike the federal financial regulators, SBA does not use lender 

information such as equity and loan concentrations as inputs into its 

lender risk rating system. Although the federal financial regulators and 

SBA both oversee lenders, their missions differ, and as a result they may 

choose to focus on different variables in conducting off-site monitoring. In 

general, the mission of the federal financial regulators is to maintain 

stability and public confidence in the nation’s financial system. In contrast, 

SBA’s mission is to aid, counsel, assist, and protect the interests of small 

business concerns, including guaranteeing loans to businesses in 

industries that lenders may avoid. Therefore, it is understandable that SBA 

might not include the same variables as federal financial regulators. In 

addition, while it is not an input into the lender risk rating system, SBA 

evaluates information such as equity and loan concentrations as part of 

other monitoring efforts. Figure 2 summarizes how the data that SBA uses 

in its lender risk rating system compare with the data included in the risk 

rating systems used by the federal financial regulators and lenders we 

interviewed. 

 

20A FICO score is a credit score derived from the credit model developed by the Fair Isaac 

Corporation. The FICO score is calculated by all three of the major credit bureaus from 

reported payment information. A higher FICO score indicates better credit, and a FICO 

score below 600 is considered poor. 

Page 14 GAO-10-53 SBA's Lender Risk Rating System
 

Figure 2: Data Used for Off-Site Monitoring 

Source: GAO.

OCC FDIC Federal

Reserve

Selected

lenders SBA

Lender information

Performance measures

Prospective measures

Loan concentrations

Income

Equity

Portfolio trends

Delinquency

Default

a

a

a

b

b

b

a

The lenders we interviewed do not collect other lenders’ information to rate their loans. 

b

SBA evaluates loan concentrations during on-site reviews of lenders and income and equity during 

performance-based reviews of lenders. These reviews are discussed in detail later in this report. 

SBA’s Lender Risk Rating 

System Better Predicted 

the Performance of Larger 

Lenders than Smaller 

Lenders 

When we performed our own independent assessments of the reliability of 

the lender risk ratings, we found that they were more reliable at predicting 

the performance of the largest lenders. To perform this independent 

assessment, we assessed how well the lender risk ratings predicted the 

actual performance of lenders (that is, lenders’ default rates).21 Because of 

data limitations, our analyses focused on lenders with larger SBAguaranteed portfolios.22 Overall, we found that SBA’s ratings were able to 

distinguish between high- and lower-risk lenders for a majority of the 7(a) 

and 504 lenders in our sample for 2007 and 2008.23 However, when we 

focused on the ratings’ ability to predict the performance of different-sized 

 

21Our measure of defaults is the purchase rate. 

22In order to estimate default rates, we needed a meaningful number of loans for each 

lender. Therefore, we excluded from our sample 7(a) and 504 lenders that had less than 100 

loans approved between January 2003 and December 2007. As a result, our sample of 

lenders does not generally include lenders with smaller guaranteed portfolios (such as 

portfolios of less than $10,000,000). 

23We identified 308 7(a) lenders in our sample that had at least 100 loans approved between 

January 2003 and December 2007. These 308 lenders’ loans represented about 79 percent of 

the total outstanding portfolio balance and about 85 percent of the total outstanding SBAguaranteed loans, based on the March 2008 lender performance report. For each of these 

lenders, we determined performance by estimating the relative odds of a loan in that 

portfolio being purchased (or defaulting), correcting for the age and current economic 

conditions. For more information on the method used, see appendix I. 

Page 15 GAO-10-53 SBA's Lender Risk Rating System
 

lenders, we found that the ratings were more effective at predicting the 

performance of lenders with the largest SBA-guaranteed portfolios (that is, 

lenders with SBA-guaranteed portfolios of at least $100 million). (See app. 

III for further discussion of how well the lender risk ratings predicted the 

performance of 7(a) and 504 lenders.) 

How the system was developed may have contributed to the lender risk 

ratings being more effective at predicting the performance of the largest 

lenders (that is, lenders with SBA-guaranteed portfolios of at least $100 

million). In order to determine how SBA developed the risk rating system, 

we reviewed the available documentation of the development process and 

discussed the process with SBA officials and the contractor. According to 

the contractor, it considered 32 variables to determine those that were the 

most predictive for each peer group. SBA then made a policy decision to 

use the same factors across all of the peer groups. Although the 

documentation did not provide the justification for this policy decision, 

SBA officials stated that the decision was made so that every lender’s risk 

rating was based on consistent information. Officials were concerned that 

lenders might be confused if the factors upon which the ratings were 

based varied by peer group, particularly since lenders do move between 

peer groups. The contractor ultimately selected four factors, each of which 

was a statistically significant predictor of lender performance for at least 

one of the peer groups. However, only for the largest peer group (those 

with guaranteed portfolios of at least $100 million) were all four factors 

statistically significant. According to SBA officials, in peer groups where a 

factor was statistically insignificant, it did not affect the lenders’ risk 

ratings—that is, for some peer groups, the ratings are determined by less 

than four factors. 

Usefulness of SBA’s Lender 

Risk Rating System Has 

Been Limited because SBA 

Does Not Ensure That Its 

Contractor Follows Sound 

Validation Techniques 

The effectiveness of SBA’s lender risk rating system has been limited 

because the agency’s contractor does not follow sound validation 

practices. According to one federal financial regulator, the ability of 

models to accurately predict outcomes can deteriorate over time. For 

example, changes in economic conditions and industry trends can affect 

model outcomes. Validation—the process of assessing whether ratings 

adequately identify risks by, for example, comparing predictions to actual 

results—helps to ensure that models remain reliable. Federal financial 

regulators (OCC, FDIC, and the Federal Reserve) and the Basel Committee 

on Banking Supervision (Basel Committee) have developed a number of 

Page 16 GAO-10-53 SBA's Lender Risk Rating System
 

common principles that financial institutions should follow in validating 

the models they use to manage risk, whether the models are purchased 

from a vendor or developed in-house.24 Validating some aspects of models 

developed by vendors may be difficult because of the proprietary nature of 

the information. But the guidance from federal financial regulators and the 

Basel Committee states that organizations have a responsibility to ensure 

that vendors follow good model validation practices. 

We identified four key elements of a sound validation policy that federal 

financial regulators and our internal control standards recommend and 

that some lenders we interviewed implemented. First, all three parts of a 

model—the data, processes, and results—should be validated using 

multiple techniques. Second, validation should be done by an independent 

party. Third, validation should include an ongoing assessment of the 

factors used in the model. Finally, the validation procedures should be 

documented. We found, however, that SBA had not adhered to the 

guidance in validating its lender risk rating system. First, SBA’s validation 

procedure does not include techniques to validate all parts of its model. 

Second, the model is not validated by an independent party. Third, SBA 

does not reassess which variables are the most predictive of lender 

performance on a routine basis. Finally, SBA’s documentation of the 

validation procedures and the results of the validation is not complete. 

Figure 3 shows how SBA’s practices align with commonly accepted 

practices. 

 

24The Basel Committee on Banking Supervision provides a forum for banking regulators to 

regularly cooperate on banking supervisory matters. Its objective is to enhance 

understanding of key supervisory issues and improve the quality of banking supervision 

worldwide. It seeks to do so by facilitating the exchange of information on national 

supervisory issues, approaches, and techniques with a view to promoting common 

understanding. At times, the committee develops guidelines and supervisory standards in 

various areas—for example, the Basel Committee’s Accord Implementation Group has 

developed guiding principles on the validation of rating systems. 

Page 17 GAO-10-53 SBA's Lender Risk Rating System
 

Figure 3: Commonly Accepted Validation Practices and SBA’s Practices 

Source: GAO.

OCC FDIC Federal

Reserve

Basel

Committee

GAO internal 

controls SBA

 Validation of the model’s data inputs

 Validation of the model’s processes

 Validation of the model’s results

Independent validation

Ongoing validation of factors used in the model 

Documentation of validation procedures

Model

validation

Included in guidance or practices

Partially included in guidance or practices

Not included in guidance or practices

Guidance from the federal financial regulators we interviewed and the 

Basel Committee states that each of the three parts of a model—the data, 

processes, and results—should be validated using a variety of techniques. 

According to FDIC guidance, validation should include ensuring that the 

data used in the model are accurate and complete, evaluating the model’s 

conceptual soundness, and analyzing the estimates the model produces 

against actual outcomes. The Basel Committee also states the importance 

of assessing all the components of a model. In addition, OCC guidance 

prescribes three generic procedures that could be used for validating each 

part of a model—a review of logical and conceptual soundness, 

comparison against other models, and comparison against subsequent 

actual events. Further, guidance from the Federal Reserve states that 

financial institutions should use a variety of techniques when validating 

their models. For example, some lenders we interviewed compared their 

internal rating systems with other commercially available models or 

compared model predictions against historical information to test the 

reliability of their models. In addition, GAO’s internal control standards 

specify that agencies should ensure the accuracy of data inputs and 

information system processing and results.25 For example, validation 

should be performed to verify that data are complete and to identify 

erroneous data. Furthermore, these standards state that management 

SBA’s Validation Procedure 

Does Not Include Techniques 

to Validate All Parts of Its 

Model 

 

25GAO/AIMD-00.21.3.1 and GAO-01-1008G. 

Page 18 GAO-10-53 SBA's Lender Risk Rating System
 

should establish controls over information processing and that output 

reports should be reviewed. 

Consistent with commonly accepted practices, SBA’s contractor has a 

documented process for validating the data used in the lender risk rating 

system. On the basis of previous reviews and recent interviews with 

contractor staff, we found that the contractor’s data quality control 

process, referred to as DUNSRight, appeared reasonable. In June 2004, we 

reported that the commercial data that Dun & Bradstreet collects go 

through a five-step quality assurance process that includes continuously 

updating databases and matching SBA records with Dun & Bradstreet 

records, with a 95 percent match of the data on critical pieces of 

information.26 In the same report, we also concluded that SBA’s controls 

over the 7(a) and 504 data used in the models helped to ensure that the 

data inputs were sufficiently reliable. Appendix IV provides information on 

Dun & Bradstreet’s procedures for ensuring the reliability of the SBPS and 

how well it predicts the likelihood that a loan will default. 

The contractor that developed the lender risk rating system also conducts 

periodic validations of the system that include using statistical tests to 

measure the model’s predictive ability and comparing the results of the 

model against lenders’ actual performance. For the years 2005 through 

2007, SBA’s contractor assessed whether the broad risk ratings were 

generally consistent with the actual performance of the lenders within 

each rating group. The contractor also determined whether each group of 

lenders (for example, those lenders rated as 1) performed better than 

other groups of lenders with lower risk ratings (that is, 2 through 5).27

However, we did not see evidence that the contractor validated the 

processes used to calculate the ratings. Specifically, neither SBA nor its 

contractor could provide documentation showing that the contractor had 

validated the theory behind the system or the logical and conceptual 

soundness of the model. For example, there was no documentation 

describing the processes followed or the link between the computer 

program and output that was used to produce the lender risk ratings. 

Therefore, we could not rerun the analysis to determine if we would have 

 

26GAO-04-610. 

27For example, the contractor determined whether those lenders that were rated as a 1 had 

lower rates of purchases than those groups of lenders that were rated as 2, 3, 4, or 5. The 

SBA contractor focused on two variables—purchase rates and cumulative net cash yields—

to assess how well the risk ratings rank ordered lenders by group. 

Page 19 GAO-10-53 SBA's Lender Risk Rating System
 

arrived at the same conclusion regarding the four factors used in the 

model. In addition, the contractor could not provide documentation 

showing that it had ensured that the mathematics and computer code were 

free of errors. According to officials from the contractor, they took steps 

to verify that the processes they followed were sound, including verifying 

the computer code they used; however, they did not document these steps. 

Further, the contractor’s validation of the model’s results was limited. 

Consistent with industry standards, SBA’s contractor has used a variety of 

statistical measures to validate the risk rating system’s results.28 But the 

documentation did not show that the contractor checked the model’s 

results against available benchmarks (such as the default rate or the 

currency rate) to validate whether the risk ratings reliably predicted 

individual lender performance. Rather, the documentation indicated that 

the contractor focused its validation on whether the broad risk ratings 

were generally consistent with the actual performance of the lenders 

within each rating group—groups that can be comprised of over 2,000 

lenders with a wide range of portfolio sizes and performance levels. 

Although this technique compares the model’s results to actual 

performance benchmarks, as suggested by industry standards, it is limited 

because it does not provide information on individual lender performance. 

According to SBA officials, the contractor tested how well individual 

scores produced by the lender rating system predicted individual lender 

performance; however, the results of this analysis were not included in the 

documentation we received and were not provided to SBA. Because lender 

performance can vary widely within the broad risk categories, the results 

of a more refined analysis would allow SBA to identify specific lenders 

placed in incorrect risk categories. 

Because SBA has never requested documentation from the contractor on 

its validation of the model’s processes, the agency cannot ensure that the 

processes used are sound. In addition, because the contractor does not 

document how well the lender risk ratings predict individual lenders’ 

performance, SBA may not be able to identify which lenders within the 

broad risk rating categories are not being rated accurately. As a result, 

 

28In particular, the contractor used the K-S statistic, which tests whether the distribution of 

a variable from a sample matches some other probability distribution. For example, the K-S 

statistic can test whether purchases follow a pattern based on a lender’s risk rating or 

whether they follow a random distribution. Guidance from federal financial regulators 

states that this statistic is commonly used in the banking industry. 

Page 20 GAO-10-53 SBA's Lender Risk Rating System
 

SBA may be relying on inaccurate ratings or missing out on opportunities 

to identify risky lenders and target them for closer monitoring. 

Each of the regulators we interviewed (OCC, FDIC, and the Federal 

Reserve) recommends in its guidance that validation include an 

independent review of the model. For example, OCC guidance states that 

model validation should be done by a party that is as independent as 

possible from the personnel who constructed the model. In addition, FDIC 

guidance states that validation should include competent and independent 

review by a reviewer who is as independent as practicable. Further, 

Federal Reserve and Basel Committee guidance notes that the validation 

process should be independent from the model development and 

implementation processes. Our internal control standards also emphasize 

the importance of independent review. They state that to reduce the risk 

of error, no one individual should control all key aspects of an activity.29

For example, an individual who is responsible for developing a model 

should not be responsible for validating it. An independent party can be 

either inside or outside the organization—for example, the internal audit 

staff, a risk management unit of the institution, an external auditor, or 

another contracted third party. Some lenders we interviewed that had 

internal risk rating systems have had them validated by a separate group 

within the institution, and others have invited independent auditors to 

review their systems. 

Validation Is Not Conducted by 

an Independent Party 

Contrary to common industry practices and internal control standards, the 

same contractor staff that developed and maintain the lender risk rating 

system are the officials who validate it. We have previously reported on 

SBA’s failure to ensure that independent parties routinely assess the 

reliability or integrity of its contractors’ models.30 Specifically, we reported 

in June 2004 that third parties did not validate the SBPS model that 

another contractor maintained because SBA believed that the model was 

stable and that clients would inform the company if the models were not 

reasonably predicting borrower behavior. Similarly, SBA and its 

contractor thought it was sufficient for someone to review the validation 

conducted by the staff who developed the model and for Dun & Bradstreet 

and SBA officials to review the contractor’s work. However, industry 

standards require that personnel other than those who developed the 

model validate it. Because SBA has not ensured that an independent party 

 

29GAO/AIMD-00.21.3.1 and GAO-01-1008G. 

30GAO-04-610. 

Page 21 GAO-10-53 SBA's Lender Risk Rating System
 

validates its lender risk ratings, certain systemic and structural issues with 

the design of the system may go undetected, and the predictive value of 

the risk ratings is more uncertain. 

Guidance from federal financial regulators and the Basel Committee states 

that validation of the factors used in the model should be ongoing and 

should take into consideration changes in the environment (such as 

changes in economic conditions or industry trends) or improvements in 

modelers’ understanding of the subject. For example, OCC guidance states 

that models are frequently altered in response to changes such as these. In 

addition, Federal Reserve guidance states that a model’s methodology 

should be validated periodically and modified to incorporate new events 

or findings as needed. Further, the Basel Committee notes that validation 

is an ongoing, iterative process. Failure to do so could cause the model to 

become less predictive and lose its ability to rank order risk over time. 

According to FDIC guidance, characteristics of a model need to be 

validated and refined when necessary because if management does not 

select and properly weight the best predictive variables, the model’s 

output will likely be less effective. Our internal control standards also 

specify that agencies that procure commercial software are responsible 

for ensuring that it meets the user’s needs and is operated properly.31

These standards state that controls should be in place to ensure that 

computer systems are modified safely by reviewing and testing them 

before placing them into operation. The standards also specify that 

management should ensure that ongoing monitoring is effective and will 

trigger separate evaluations where problems are identified. 

SBA Does Not Perform 

Ongoing Validation to Ensure 

That the Factors Used in the 

System Are the Most Predictive 

SBA’s contractor takes some steps to validate the lender risk rating 

system’s ability to reliably predict lender performance but does not ensure 

that the variables used to calculate the risk ratings are the most predictive 

of lender performance. We reviewed the validations of the risk rating 

system that the contractor conducted in 2005, 2006, and 2007. These 

validation efforts included testing of the statistical importance of each of 

the four factors used in the lender risk rating system. However, these 

validations did not routinely include testing of other factors to account for 

changes in economic conditions or industry trends. The 2005 validation 

effort was the only one that tested additional factors. SBA’s contractor 

tested three new variables to determine if they improved the model’s 

 

31GAO/AIMD-00.21.3.1 and GAO-01-1008G. 

Page 22 GAO-10-53 SBA's Lender Risk Rating System
 

ability to predict lender performance and found that they did not.32 Neither 

of the subsequent validations included assessments of additional variables, 

and SBA did not requested them. According to SBA officials, SBA and the 

contractor identified possible additional variables over the past several 

years that they did not test for use in the model because they wanted more 

experience with it and the data.33 They also noted that they always had 

plans to redevelop the model within 5 years but could not do so until the 

agency had signed a second contract with Dun & Bradstreet that provided 

funds for a redevelopment. However, if SBA had asked the contractor to 

test additional factors on a regular basis, the agency may have found that 

an earlier redevelopment effort or incremental adjustments could have 

improved the predictive ability of the model. Because new variables that 

might take into account economic changes or industry developments have 

not been routinely assessed, the ratings may not be as effective as they 

could be. 

In addition, according to the contractor’s validation reports, the lender 

risk rating system’s predictive ability for 7(a) lenders decreased from 2005 

to 2007.34 This decrease led the contractor to suggest in 2007 that SBA 

redevelop the model to improve its predictive ability and prevent further 

deterioration. SBA officials agreed, and the contractor is currently 

redeveloping the model, including testing new variables, to keep up with 

changing economic conditions and to reflect SBA’s and the contractor’s 

experiences working with the data and the model over the last several 

years. It will be important for SBA to ensure that the contractor conducts 

sound testing as part of its redevelopment. 

The federal financial regulators’ guidance states that a sound validation 

policy should include documentation of the validation. For example, FDIC 

and OCC guidance states that model validation documentation should 

describe the model, how it is used, and its limitations. Federal Reserve 

guidance also notes that the validation process should be documented. In 

addition, FDIC and OCC have said that the procedures used to validate the 

SBA’s Documentation of 

Validation Procedures and 

Results Is Incomplete 

 

32According to the 2005 validation report, the contractor performed a stepwise regression 

to determine if using last 24-month purchases, last 12-month charge-offs, or a modified 

problem loan rate would increase the model’s ability to predict future purchases among 

7(a) lenders. The contractor found that there would be no benefit to using these variables. 

33These additional variables included the age of the portfolio and type of loan product. 

34The K-S statistic for 7(a) lender ratings decreased from 36 in 2005 to a range of 27 to 29 in 

2007. 

Page 23 GAO-10-53 SBA's Lender Risk Rating System
 

model on an ongoing basis and the results of these validations should be 

documented, even if the institution uses a model developed by a vendor. 

For example, OCC guidance states that an institution should seek 

assurances that the vendor’s model is defensible and works as promised. 

Further, the Basel Committee guidance notes that even vendors that are 

not willing to reveal proprietary information should provide information 

on the validation techniques they use. Complete documentation of the 

results of ongoing validations assists users in understanding the model and 

facilitates independent reviewers’ assessments of the model’s validity. Our 

internal control standards also specify the importance of documenting 

information systems.35 For example, these standards state that all 

significant events in developing and maintaining computer systems should 

be clearly and completely documented. This documentation should 

describe the system, how the data used in the system are handled, and 

other controls in place to maintain the system. 

SBA did not ensure that the contractor provided complete documentation 

of the results of its validations or documented its validation procedures. 

SBA provided us with some documentation of the contractor’s process for 

validating the data used in the lender risk rating system, but 

documentation of the results of the validations was inconsistent and did 

not have information on the procedures for validating the model’s 

processes. For example: 

• The validation reports we reviewed (2005 to 2007) did not always include 

information on the statistical measure the contractor used to describe the 

model’s predictive abilities. The 2006 validation report did not contain this 

statistic for the 7(a) ratings, and only the 2007 report included it for 504 

lender risk ratings. 

• The validation reports did not describe the contractor’s validation 

procedures. As noted previously, SBA did not provide documentation 

showing that the contractor validated the mathematics and computer code 

used in the model. 

• The validation reports did not explain why in 2005 the contractor 

considered whether additional variables would improve the model’s ability 

to predict lender performance but did not consider additional variables in 

other years. 

 

35GAO/AIMD-00.21.3.1 and GAO-01-1008G. 

Page 24 GAO-10-53 SBA's Lender Risk Rating System
 

• The validation reports did not describe any limitations of the model that 

would have helped SBA to use the results accurately. 

Officials from the contractor explained that the documentation provided 

was typical of that seen in the private sector for such models, but stated 

that they would provide more detailed documentation in the future. 

Because SBA does not ensure that its contractor completely documents its 

validation procedures and results, it is difficult to assess the sufficiency of 

the validations performed. Further, as we noted previously, it is important 

for an independent party to validate a model’s reliability. Without clear 

documentation explaining the model’s limitations, the validation 

procedures, and the results of the validations, an independent reviewer 

would have difficulty conducting a thorough assessment of SBA’s model. 

SBA Does Not Use Its Own 

Data to Assess or 

Supplement the 

Contractor’s Validation of 

the Lender Risk Rating 

System 

In addition to not ensuring that its contractor follows sound validation 

techniques, SBA does not conduct its own analysis of data to supplement 

the contractor’s validation of the lender risk rating system. According to 

the Basel Committee guidance we reviewed, organizations must have 

clearly articulated strategies for regularly reviewing the results of vendor 

models and the integrity of the external data used in these systems. 

Further, OCC guidance states that vendor models should generally be held 

to the same minimum validation standards as internally developed models. 

When full and complete details concerning aspects of a vendor product are 

lacking, OCC and Basel Committee guidance states that organizations 

should rely more heavily on alternative validation techniques to 

compensate for the lack of access to full information. This guidance notes 

that in such cases, it is critical for organizations to test the results of the 

vendor’s model at least once a year using their own data on actual 

performance to assess the model’s predictive ability. This procedure helps 

to ensure that the models continue to function as intended and verifies the 

reliability and consistency of any external data used. Our internal control 

standards state that monitoring should be performed continually and that 

it should involve comparisons and reconciliations.36 For example, these 

standards specify that agencies should compare information generated 

from computer systems to actual records. Agencies should also analyze 

and reconcile any differences that might be found. 

 

36GAO/AIMD-00.21.3.1 and GAO-01-1008G. 

Page 25 GAO-10-53 SBA's Lender Risk Rating System
 

SBA does not use its own data to independently assess the lender risk 

rating system’s results. According to a 2007 SBA Inspector General report, 

SBA has previously rejected using its own data to develop lender 

performance benchmarks that could be used in lieu of or in conjunction 

with the risk ratings because doing so would be time-consuming and the 

benchmarks would have to be monitored and replaced as program and 

economic conditions changed.37 However, we found that SBA data could 

be useful for developing alternate measures of lender performance in 

order to independently validate the lender risk rating system’s results. For 

example, SBA could perform analyses similar to those we performed by 

using its own data to compare risk ratings with actual lender default rates. 

Further, SBA could use its own data to develop alternate measures, such 

as currency rates, as performance benchmarks. As we did in our analyses, 

SBA could compare how well lender risk ratings predicted actual 

performance to how well an alternate measure demonstrated lender’s 

actual performance. Because of data limitations, our analyses focused on 

lenders with larger SBA-guaranteed portfolios. As a result, we were unable 

to determine how well these alternate measures predict the performance 

of lenders with smaller portfolios, but SBA has more years of data 

available to facilitate such analyses. Without performing its own 

assessment, the agency may not be able to identify issues with the model’s 

ability to reasonably predict lender performance and notify the contractor. 

As a result, SBA may miss opportunities to identify risky lenders and 

mitigate the risks they pose to SBA’s portfolio. 

SBA Does Not Use 

Lender Risk Ratings 

to Target Lenders for 

On-Site Review or 

Tailor the Scope of 

the Reviews 

 

37SBA, Office of Inspector General, SBA’s Use of the Loan and Lender Monitoring System, 

Report no. 7-21 (Washington, D.C.: May 2, 2007). 

Page 26 GAO-10-53 SBA's Lender Risk Rating System
 

SBA uses its lender risk rating system to conduct off-site monitoring of 

lenders and their portfolios. In addition to routine on-site reviews, federal 

financial regulators and lenders use off-site tools to monitor lenders’ 

performance and portfolio trends. As part of a comprehensive risk 

management strategy, federal financial regulators use risk ratings to 

conduct portfolio analysis and identify problem trends. FDIC relies on a 

number of off-site monitoring tools to perform horizontal analyses (that is, 

compare similar lenders) and analyze emerging lending trends. For 

example, when subprime lending first began, the agency tracked the 

amount of subprime lending that each of its lenders did. The Federal 

Reserve uses various off-site monitoring tools that focus on asset quality 

and credit risk to identify banks whose ratings appear to have deteriorated 

since their most recent on-site reviews. For example, it analyzes 

information related to nonperforming and performing loans and the 

changing composition of loan concentrations. OCC uses its core 

assessment process to assess how much risk lenders have taken on and 

the quality of their risk management to determine aggregate risk. 

SBA Has Used the Lender 

Risk Rating System to 

Conduct Some Off-Site 

Monitoring of Lenders and 

Their Portfolios 

Lenders also use off-site monitoring tools to oversee loan portfolios. For 

example, one 7(a) lender we interviewed uses various scoring models to 

determine, among other things, how each loan’s risk rating has changed 

since the loan was originated. Other 7(a) lenders with whom we spoke use 

off-site monitoring tools that analyze factors such as geography, industry, 

management quality, company performance, and collateral to predict the 

risk of loans. Another 7(a) lender relies on several off-site monitoring 

systems to track portfolio performance—including delinquencies and 

trends by state, industry, and North American Industry Classification 

System (NAICS) code—and forecast losses.38 In addition, bank officials we 

interviewed stated that they reviewed all troubled loans on a monthly 

basis. 

Similarly, SBA uses its lender risk rating system to obtain quarterly 

performance information on all lenders and determine portfolio trends. 

SBA officials stated that before they had the risk rating system, they were 

not able to analyze the performance of all lenders, especially lenders with 

the smallest volume of SBA-guaranteed loans. SBA has formed a Portfolio 

 

38NAICS was developed as the standard for federal statistical agencies in classifying 

business establishments for the collection, analysis, and publication of statistical data 

related to the business economy of the United States. NAICS was developed under the 

auspices of the Office of Management and Budget and adopted in 1997 to replace the old 

Standard Industrial Classification system. 

Page 27 GAO-10-53 SBA's Lender Risk Rating System
 

Analysis Committee that meets monthly to discuss portfolio trends 

identified by analyzing loan and lender performance data. Comprised of 

top SBA officials, the committee typically discusses delinquencies, 

liquidations, charge-offs, and purchase rate trends by delivery method 

(that is, various SBA loan programs) for the 7(a) and 504 portfolios. The 

committee also discusses changes in loans’ SBPSs (from the end of the 

quarter in which the loan was disbursed to the most recent quarter) and 

the scores’ performance in ranking loans. To date, SBA has taken some 

actions as a result of these meetings. For example, SBA officials told us 

that as a result of discussions about portfolio performance during these 

meetings, they discontinued an SBA program that allowed borrowers to 

provide limited documentation. 

SBA officials told us that the agency also recently began using the results 

of the lender risk rating system to conduct “performance-based reviews.” 

According to SBA officials, the purpose of these reviews is to perform 

more in-depth, off-site monitoring that incorporates lenders’ information, 

such as lender financial ratios from call reports, that is currently not part 

of the lender risk rating system. Specifically, SBA financial analysts are 

assigned lenders that they will monitor over time. Each year, the analysts 

will focus on lenders with outstanding balances on their SBA portfolios of 

at least $10 million that are not scheduled for on-site reviews and on all 

other preferred lenders regardless of size. With the remaining resources, 

they will review small problem lenders—for instance, those with 

guaranteed portfolios that are less than $10 million but that received a 

lender risk rating of 4 or 5. SBA had conducted 517 of these reviews as of 

August 2009. 

SBA Has Not Effectively 

Integrated Its Lender Risk 

Rating System into the OnSite Examination Process 

Although SBA has begun some off-site monitoring using its risk rating 

system, it does not use the ratings to target lenders for on-site reviews. 

FDIC and the Federal Reserve use risk ratings as the primary tool for 

identifying lenders that need to be reviewed.39 For example, FDIC stated 

that they relied on off-site monitoring to determine the scope and 

frequency of on-site exams. Our internal control standards require that 

agencies assess and mitigate risks using quantitative and qualitative 

methods and then conduct a thorough and complete analysis of those 

risks. Although SBA identifies the risks that lenders pose, it does not 

mitigate these risks because it chooses not to target high-risk 7(a) and 504 

 

39According to OCC officials, they review all lenders on a regular schedule. 

Page 28 GAO-10-53 SBA's Lender Risk Rating System
 

lenders for on-site reviews. Instead, the agency targets lenders for reviews 

based on the size of their portfolios, focusing primarily on the largest 

lenders—that is, 7(a) lenders with at least $10 million in their guaranteed 

loan portfolio and 504 lenders with balances of at least $30 million. Only 

when prioritizing large lenders for review does SBA consider their risk 

ratings.40

We found that in calendar years 2005 to 2008, most of SBA’s 477 on-site 

reviews were of large 7(a) and 504 lenders that posed limited risk to SBA. 

Ninety-nine percent (472 of 477) of the lenders reviewed were large 

lenders, and 80 percent (380 of 477) posed limited risk to SBA (that is, 

were rated as a 1, 2, or 3 by the lender risk rating system). The agency has 

increased the number of on-site reviews performed (from 69 in 2005 to 188 

in 2008) because it can now charge lenders for them.41 However, SBA 

continues to conduct a limited number of reviews of high-risk lenders or 

those with a lender risk rating of 4 or 5 (see fig. 4). In 2005, 20 percent (14 

of 69) of SBA’s on-site reviews were of lenders that posed significant risk 

to the agency. In 2008, that proportion was 22 percent (42 of 188 reviews). 

As a result, a substantial number of high-risk lenders were not reviewed 

each year. For example, in 2008, only 3 percent of the 1,587 lenders that 

posed significant risk to SBA were reviewed. Because SBA relies on 

lenders’ size to target lenders for on-site reviews, smaller lenders that, 

based on their high-risk ratings, pose significant risk to SBA have not 

received oversight consistent with their risk levels. 

 

40Lender risk ratings are used to prioritize reviews for lenders within the same peer group. 

41According to SBA, it implemented fee-based reviews in late fiscal year 2007. 

Page 29 GAO-10-53 SBA's Lender Risk Rating System
 

Figure 4: SBA On-Site Reviews, 2005 to 2008 

0

20

40

60

80

100

120

140

160

180

200

2005 2006 2007 2008

Frequency

Year

Risk rating 5

Risk rating 4

Risk rating 3

Risk rating 2

Risk rating 1

Source: GAO analysis of SBA data.

Our findings are similar to those of SBA’s Inspector General. In a 2007 

report, the Inspector General concluded that SBA had made limited use of 

lender risk ratings to guide its oversight activities.42 It observed that the 

agency reviewed large lenders regardless of their risk ratings and did not 

do on-site reviews of smaller lenders with high-risk ratings. The report 

recognized that some of the smaller lenders might not have a sufficient 

number of loans in their portfolio to warrant an on-site review but noted 

that others could have a significant number of loans. The Inspector 

General recommended that SBA develop an on-site review plan or agreedupon procedures for all high-risk 7(a) lenders with guaranteed loan 

portfolios in excess of $4 million. We agree that although not all of the 

small lenders with high-risk ratings warrant more targeted monitoring, 

 

42SBA, Report no. 7-21. 

Page 30 GAO-10-53 SBA's Lender Risk Rating System
 

some do. Of the 1,545 high-risk lenders that we found were not reviewed in 

2008, 215 lenders had an outstanding portfolio of at least $4 million. 

According to SBA officials, the agency is developing agreed-upon 

procedures for conducting additional reviews of smaller lenders in 

response to the Inspector General’s recommendation. 

Lender Risk Ratings Do 

Not Inform the Scope of 

SBA’s On-Site Reviews, and 

Reviews Do Not Include an 

Assessment of Lenders’ 

Credit Decisions 

Unlike federal financial regulators, SBA does not rely on its lender risk 

ratings to help focus the scope of on-site reviews, and the reviews do not 

include an assessment of the lenders’ credit decisions. The federal 

financial regulators we interviewed rely on results from their off-site 

monitoring systems to identify which areas of a bank’s operations they 

should review more closely. Using the results of the off-site monitoring, 

they are able to tailor the scope of their on-site reviews to the specific 

areas of lenders’ operations that pose the most risk to the bank. In 

addition, during on-site reviews, the federal financial regulators often 

include an assessment of the quality of lenders’ credit decisions. They told 

us that the results of their on-site reviews helped not only to assess the 

risk that lenders posed, but also to identify emerging lending trends and 

areas of banking operations that may pose significant, new risk to banks in 

the future. They are then able to use the results to inform their off-site 

monitoring systems. For example, regulators stated that when their on-site 

reviews showed an increase in subprime lending, they incorporated 

subprime lending data into their off-site monitoring tools. Although SBA’s 

mission differs from the mission of the federal financial regulators, 

internal control standards require all federal agencies to identify and 

analyze risk, as well as to determine the best way to manage or mitigate it. 

According to SBA’s Standard Operating Procedure for on-site reviews, the 

agency assesses a lender’s (1) portfolio performance, (2) SBA management 

and operations, (3) credit administration practices, and (4) compliance 

with statutes and SBA regulations and policies. For the portfolio 

performance component, SBA uses L/LMS data to review the size, 

composition, performance, and credit quality of a lender’s SBA portfolio. 

When assessing a lender’s SBA operations, SBA evaluates, among other 

things, the lender’s internal policy and procedural guidance on SBA 

lending; the competence, leadership, and administrative ability of 

management and staff who have responsibility for the SBA loan portfolio; 

and the adequacy of the lender’s internal controls. For the credit 

administration component, SBA assesses the lender’s policies and 

procedures for originating, servicing, and liquidating SBA loans. An SBA 

contractor then uses this information during file reviews to determine the 

degree to which lending policies and procedures are followed. For the 

Page 31 GAO-10-53 SBA's Lender Risk Rating System
 

compliance component, SBA’s contractor performs file reviews that focus 

on the lender’s compliance with SBA-specific requirements. 

When performing file reviews, contractor staff do not rely on results from 

the lender risk rating system to tailor the scope of the reviews. Instead, 

contractor staff rely on a standard form—the lender review checklist—to 

conduct all file reviews, regardless of the lender risk rating or other 

information available to SBA about the lender’s portfolio. Moreover, these 

file reviews do not include an assessment of the quality of the credit 

decisions made by lenders. Rather, the lender review checklist focuses 

primarily on the lenders’ adherence to SBA policies, including those based 

on statutes or regulations, when making SBA-guaranteed loans. The 

checklist includes questions related to, among other things, the 

determination of borrower eligibility (including whether the borrower had 

any other outstanding SBA loans that are not current), the calculation of 

collateral value, and evidence that all required forms were obtained and 

reviewed. According to SBA officials, the file reviews focus on compliance 

with SBA policy because it is not SBA’s role to evaluate lenders’ credit 

decisions. The officials did not believe that the agency should be setting 

policy or underwriting standards for lenders. However, because SBA relies 

on lenders with delegated underwriting authority to make the majority of 

its loans, we believe that SBA should take a more active role in ensuring 

that these lenders are making sound credit decisions. 

We originally reported on SBA’s compliance-based reviews in 2002, when 

we found that SBA’s automated checklist lacked the substance to provide 

a meaningful assessment of lender performance.43 We reported that SBA’s 

on-site reviews were based on reviewers’ findings from a lender 

questionnaire and a review checklist in order to ensure objective scoring. 

The lender questionnaire addressed organizational structure, oversight 

policy, and controls. SBA officials said that prior to the implementation of 

the automated worksheet scoring process, on-site reviews were done in a 

narrative format, and reviewers’ assessments of lender performance were 

subjective. They noted that the worksheet format made the reviewers’ 

assessments of lenders more consistent and objective. As previously 

mentioned, SBA has since expanded the scope of its on-site reviews to 

include more than just a compliance component and revised the checklist 

 

43GAO, Small Business Administration: Progress Made but Improvements Needed in 

Lender Oversight, GAO-03-90 (Washington, D.C.: Dec. 9, 2002). 

Page 32 GAO-10-53 SBA's Lender Risk Rating System
 

used to conduct file reviews. But, as noted previously, the revised 

checklist still focuses on compliance with SBA policies and procedures. 

An example from our February 2009 report on compliance with the credit 

elsewhere requirement illustrates SBA’s emphasis on ensuring policy 

compliance rather than verifying lenders’ credit decisions during on-site 

reviews.44 Because the 7(a) and 504 programs are intended to serve 

borrowers who cannot obtain conventional credit at reasonable terms, 

lenders making 7(a) and 504 loans must ensure that borrowers meet the 

credit elsewhere requirement. This statutory requirement stipulates that to 

receive loans, borrowers must not be able to obtain financing under 

reasonable terms and conditions from conventional lenders. During an onsite review, the contractor is to determine whether lender policies and 

practices adhere to SBA’s credit elsewhere requirement. During the 

review, SBA’s contractor explained that it checks to see that the lender 

documented its credit elsewhere determination and cited one of the six 

factors that SBA has determined are acceptable reasons for concluding 

that a borrower could not obtain credit elsewhere. However, it does not 

routinely assess the information lenders provide to support credit 

elsewhere determinations. Contract staff answer “yes” or “no” on the 

checklist that “written evidence that credit is not otherwise available on 

terms not considered unreasonable without guarantee provided by SBA” 

was in the file. Contractor officials stated that when the documentation 

standard is not met, the examiner will sometimes look at the factual 

support in the file to independently determine whether the credit 

elsewhere requirement was actually met. 

Because SBA officials choose not to rely on lender risk ratings to inform 

file reviews conducted during on-site reviews or assess lenders’ credit 

decisions during the reviews, the agency does not have the type of 

information related to the quality of the underwriting standards and 

practices of lenders that is necessary to understand the risks that banks 

pose to SBA’s portfolio. Without this information, the agency cannot make 

informed improvements to the lender risk rating system that would enable 

it to take into account new emerging lending trends. 

 

44GAO, Small Business Administration: Additional Guidance on Documenting Credit 

Elsewhere Decisions Could Improve 7(a) Program Oversight, GAO-09-228 (Washington, 

D.C.: Feb. 12, 2009). 

Page 33 GAO-10-53 SBA's Lender Risk Rating System
 

Because SBA relies heavily on its lenders to determine if loans are eligible 

for an SBA guarantee and to underwrite the loans, lender oversight is of 

particular importance. By working with a contractor to develop a lender 

risk rating system, SBA has taken a positive step toward improving its 

oversight of lenders. The lender risk rating system enables SBA for the 

first time to systematically and routinely monitor the performance of all 

lenders, including lenders with the smallest loan portfolios, which SBA 

had not routinely monitored. However, SBA does not ensure that its 

contractor follows sound practices when validating the system. Guidance 

from the federal financial regulators we interviewed states, among other 

things, that validation should be performed by an independent party and 

should routinely reassess the factors used to determine risk, taking into 

consideration changes in the environment (such as changes in industry 

trends). SBA did not require its contractor to ensure that personnel other 

than the staff who developed the model validated it or to routinely 

reassess the factors used in the system as part of its validations. Unless 

SBA ensures that its contractor follows sound model validation practices, 

the agency’s ability to identify inaccurate ratings, detect systemic or 

structural issues with the design of the model, and determine whether the 

ratings are deteriorating over time as economic conditions change will be 

limited. SBA’s contractor is currently redeveloping the lender risk rating 

system to improve its predictive ability. However, the benefits that may be 

achieved through the redeveloped lender risk rating system will be limited 

if SBA continues the practice of not ensuring that its contractor adopts 

sound validation practices. In particular, testing to ensure that the system 

effectively evaluates risk is an important element to improve a risk rating 

system, regardless of whether such testing occurs during routine 

validation efforts or during model redevelopment. 

Conclusions 

In addition, contrary to federal financial regulator guidance and our 

internal control standards, SBA has not used its own data to conduct 

independent assessments of the risk rating system to help ensure the 

usefulness of the risk ratings. We found that SBA data could be useful for 

developing alternate measures of lender performance in order to 

independently validate the lender risk rating system’s results. Without 

performing its own assessment, the agency may not be able to identify 

issues with the model’s ability to reasonably predict lender behavior or to 

notify the contractor of any suspected deterioration. As a result, SBA may 

miss opportunities to identify risky lenders and mitigate the risks they 

pose to SBA’s portfolio. 

Page 34 GAO-10-53 SBA's Lender Risk Rating System
 

If SBA improves its validation of the lender risk ratings, the agency could 

rely more on them to determine which lenders need an on-site review. 

Currently, unlike FDIC and the Federal Reserve, SBA does not take full 

advantage of its risk ratings to set the schedules for on-site reviews. The 

agency targets lenders for on-site reviews based on size rather than risk 

level. As a result, we found that SBA conducted on-site reviews of only 3 

percent of the lenders that the lender risk rating system identified as high 

risk in 2008. Of these, 215 had an outstanding SBA portfolio of at least $4 

million. Relying more on the risk ratings to target lenders for review would 

enable the agency to focus on the lenders that pose the most risk to the 

agency. 

Although SBA has made improvements to its off-site monitoring of 

lenders, the agency will not be able to substantially improve its lender 

oversight efforts unless it improves its on-site review process. Federal 

financial regulators rely on results from their off-site monitoring to tailor 

the scope of their on-site reviews. SBA does not rely on its lender risk 

ratings to inform file reviews conducted during on-site reviews but rather 

consistently uses a checklist to examine lenders. In addition, federal 

financial regulators routinely assess the quality of lenders’ credit decisions 

as part of their on-site examination process. SBA fails to include this 

component but instead focuses more on compliance with SBA policies and 

procedures. For example, rather than assessing the quality of lender 

underwriting, contractor staff focus on whether lenders ensured that the 

borrowers met eligibility requirements, including whether borrowers had 

any other outstanding SBA loans that are not current. By including an 

assessment of lenders’ credit decisions as a routine part of their on-site 

review process, SBA would be able to determine the quality of the lenders’ 

underwriting standards and practices and make any necessary changes to 

its lender risk rating system to ensure that the tool is relevant and includes 

emerging lending trends. 

We recommend that the Administrator of the Small Business 

Administration take the following four actions: 

To ensure that the lender risk rating system effectively evaluates risk, 

when validating the system and undertaking any redevelopment efforts, 

the Administrator should 

Recommendations for 

Executive Action 

• ensure that SBA’s contractor follows sound model validation practices. 

These practices should include (1) testing of the lender risk rating system 

data, processes, and results, including a routine reassessment of which 

Page 35 GAO-10-53 SBA's Lender Risk Rating System
 

factors are the most predictive of lender performance; (2) utilizing an 

independent party to conduct validations; and (3) maintaining complete 

documentation of the validation process and results. 

• use SBA’s own data to assess how well the lender risk ratings predict 

individual lender performance. 

To make better use of the lender risk rating system in SBA’s oversight of 

lenders, the Administrator should 

• develop a strategy for targeting lenders for on-site reviews that relies more 

on SBA’s lender risk ratings. 

• consider revising SBA policies and procedures for conducting on-site 

reviews. These revised policies and procedures could require staff to (1) 

use lender risk ratings to tailor the scope of file reviews performed during 

on-site reviews to areas that pose the greatest risk, (2) incorporate an 

assessment of lenders’ credit decisions in file reviews, and (3) use the 

results of expanded file reviews to identify information, such as emerging 

lending trends, that could be incorporated into its lender risk rating 

system. 

We requested SBA’s comments on a draft of this report, and the Associate 

Administrator of the Office of Capital Access provided written comments 

that are presented in appendix II. SBA generally agreed with our 

recommendations and outlined some steps that it plans to take to address 

them. The agency also provided one technical comment, which we 

incorporated. 

Agency Comments 

and Our Evaluation 

SBA provided detailed comments on each of our four recommendations. 

In response to our recommendation to ensure that SBA’s contractor 

follows sound model validation techniques, SBA noted that the agency is 

currently undertaking a redevelopment of its lender risk rating system and 

plans to ensure that best practices are incorporated into the 

redevelopment validation process. According to the agency, the 

redevelopment contract will give SBA greater flexibility to reassess the 

predictiveness of the factors used in the model and to refine the model if 

necessary. SBA stated that it is also developing an independent review 

process as well as increasing the level of documentation of the validation 

process. 

Page 36 GAO-10-53 SBA's Lender Risk Rating System
 

Regarding our recommendation to use its own data to assess how well the 

lender risk ratings predict individual lender performance, SBA stated that 

although it remains confident that the lender risk ratings provide accurate 

predictions, the agency will determine whether alternative measures 

would be useful to supplement the lender risk ratings. 

In response to our recommendation to develop a strategy for targeting 

lenders for on-site review that relies more on the lender risk ratings, SBA 

stated that it agreed with our finding that between 2005 and 2008 on-site 

reviews had been limited and primarily focused on the largest lenders, but 

pointed out that the agency had significantly increased the number of 

lenders reviewed since it began charging for on-site reviews late in fiscal 

year 2007. The agency also noted that the largest lenders account for 

approximately 85 percent of SBA’s entire guaranteed portfolio, while the 

high-risk lenders that were not reviewed in 2008 represent 2 percent of 

SBA’s total 7(a) and 504 portfolios. In our report, we recognize that while 

not all of the small lenders with high risk ratings warrant more targeted 

monitoring, some do. Of the 1,545 high-risk lenders that we found were not 

reviewed in 2008, 215 lenders had significant portfolios—that is, portfolios 

of at least $4 million. While SBA indicated that it plans to continue to focus 

on-site reviews on the largest lenders that account for the majority of the 

guaranteed portfolio, it stated that it will consider revising its internal 

policies to make better use of the lender risk ratings to prioritize on-site 

reviews. 

Regarding our recommendation to consider revising policies and 

procedures for conducting on-site reviews, SBA stated that the agency is 

in the process of reprocuring its on-site review contract. According to the 

agency, SBA included the ability to conduct on-site reviews that can be 

better tailored to specific concerns about individual lender performance as 

part of the reprocurement process. SBA also stated that the agency is in 

the process of evaluating our recommendation to include an assessment of 

lender credit decisions in the on-site review process and will investigate 

ways to use the results of the on-site reviews to inform the lender risk 

rating system. 

As agreed with your offices, unless you publicly announce the contents of 

this report earlier, we plan no further distribution until 30 days from the 

report date. At that time, we will send copies of this report to interested 

congressional committees, the Administrator of the Small Business 

Administration, and other interested parties. In addition, the report will be 

available at no charge on the GAO Web site at http://www.gao.gov. 

Page 37 GAO-10-53 SBA's Lender Risk Rating System
 

If you or your staffs have any questions about this report, please contact 

me at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

William B. Shear 

of this report. Key contributors to this report are listed in appendix V. 

Director, Financial Markets and 

 Community Investment 

Page 38 GAO-10-53 SBA's Lender Risk Rating System
Appendix I: Objectives, Scope, and 

Methodology 

Page 39 GAO-10-53 

Appendix I: Objectives, Scope, and 

Methodology 

In this report, we examined (1) how the Small Business Administration’s 

(SBA) risk rating system compares with the off-site monitoring tools used 

by federal financial regulators and lenders and the system’s usefulness for 

predicting lender performance and (2) how SBA uses the lender risk rating 

system in its lender oversight activities. 

To determine how SBA’s lender risk rating system compares with off-site 

monitoring tools used by federal financial regulators and lenders, we 

conducted interviews and reviewed documents to identify common 

industry standards. We interviewed officials from three federal financial 

regulators—the Office of the Comptroller of the Currency (OCC), the 

Board of Governors of the Federal Reserve System (the Federal Reserve), 

and the Federal Deposit Insurance Corporation (FDIC)—five of the largest 

7(a) lenders, and the five largest 504 lenders.1

 We identified the largest 

lenders based on the size of their SBA-guaranteed portfolio in 2007, the 

most recent data available when we began our review.2

 The documents we 

reviewed included relevant literature, procedural manuals and other 

related federal guidance to banks on loan portfolio monitoring, and lender 

procedural manuals. We then obtained and analyzed documents from SBA 

on its lender risk rating system and conducted interviews with agency and 

contractor officials responsible for maintaining the system to determine 

how the system was developed and validated. We assessed SBA’s lender 

risk rating system against common industry standards and our internal 

control standards.3

 In addition, we reviewed our previous work on SBA 

and guidance on model validation from the Basel Committee on Banking 

Supervision, which provides a forum for banking regulators from around 

the world to regularly cooperate on banking supervisory matters and 

develop common guidelines.4

 

1

The federal financial regulators we selected have policies and procedures for monitoring 

credit risk that are relevant to SBA. We focused on the largest lenders because they would 

be most likely to use off-site monitoring tools similar to SBA’s lender risk rating system. 

2

According to SBA, there are approximately 5,000 SBA lenders. Although our sample of 10 

large lenders is nongeneralizable, it offers perspectives on how some lenders conduct offsite monitoring. 

3

GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00.21.3.1

(Washington, D.C.: November 1999) and Internal Control Management and Evaluation 

Tool, GAO-01-1008G (Washington, D.C.: August 2001). 

4

GAO, Small Business Administration: New Service for Lender Oversight Reflects Some 

Best Practices, but Strategy for Use Lags Behind, GAO-04-610 (Washington, D.C.: June 8, 

2004). 

 SBA's Lender Risk Rating System 
Appendix I: Objectives, Scope, and 

Methodology 

To assess the lender risk rating system’s usefulness for predicting lender 

performance, we performed independent statistical tests to determine how 

well it predicted individual lender performance. To perform these tests, 

we first obtained the following data from SBA: administrative data on 

loans approved in 2003 through the end of 2007 (including the date the 

loan was approved, the size of the loan, and whether and when the loan 

was purchased); the March 2007 and March 2008 lender performance 

reports containing risk ratings; and the currency rate for each lender.5

 We 

assessed the reliability of these data by reviewing information about the 

data and performing electronic data testing to detect errors in 

completeness and reasonableness. We found that the data were 

sufficiently reliable for the purposes of this report. 

Using SBA’s data, we undertook a number of evaluative steps to test the 

agency’s model. First, we assessed how well the lender risk ratings 

predicted lender default rates (our measure of actual lender performance). 

In order to test how well the lender risk ratings predicted lender 

performance, we estimated how well a lender performed during either the 

year or 6 months after the score was developed (depending on the amount 

of data available) using a logit regression. A logit regression is a statistical 

technique that estimates how the odds of an outcome changes with an 

attribute of the unit of analysis. In our case, we estimated how the odds of 

a loan being purchased by SBA varied by the lender that made the loan. 

Additionally, we controlled for the age of loans and how default rates for 

all loans changed over the year or 6 months. To control for the age and 

changing default rates over time, we employed a methodology called a 

discrete time hazard model. We restructured the data so that there was a 

separate observation for every quarter that a loan was at risk of being 

purchased. Then we estimated a logit regression and predicted whether 

the loan was purchased that quarter. In that regression, we included a 

dummy variable for each lender, a dummy variable for each quarter, and a 

dummy variable for each quarter since that loan was approved, to capture 

the age of the loan.6

 The following describes the regression equation we 

used: 

 

5

The currency rate is the sum of the dollar balance of guaranteed loans that are less than 30 

days past due divided by the dollar balance of the total portfolio of guaranteed loans 

outstanding. For comparison purposes, we subtracted the currency rate from 100, so that 

lower currency rates would be consistent with higher default rates. 

6

This regression was weighted by the guaranteed amount of the loan at the time of 

approval. 

Page 40 GAO-10-53 SBA's Lender Risk Rating System
Appendix I: Objectives, Scope, and 

Methodology 

P(loan i was purchased at time t) = logit(αl ,αt ,αd) 

where the parameters of interest, αl

, can be transformed to express the 

relative odds of a loan being purchased or defaulting for each lender, with 

one lender excluded as a reference. We used the coefficients αl

 as the 

measures of lender risk. In addition, the coefficients αt

 control for the 

differential rate of default by time period, and the coefficients αd control 

for the age of the loans. 

Once we estimated the performance for each lender, we matched it with 

each lender’s record in the lender performance report, which contained 

the risk rating. For 7(a) loans, we matched our performance measures 

with the lender risk rating using a “crosswalk” file obtained from SBA.7

Because the data we obtained from SBA only included loans that were 

approved from January 2003 to December 2007 and a lender had to have 

made at least 100 loans during that time period to make our analysis 

meaningful, we were only able to obtain measures for 308 of the 4,673 7(a) 

lenders in the March 2008 lender performance report. We were more likely 

to obtain measures for larger lenders.8

 For example, we were able to 

obtain measures for 56 of the 60 lenders with more than $100 million in 

outstanding SBA-guaranteed loan balances. In all, the 308 lenders, plus the 

lender excluded as the reference case, represented approximately 79 

percent of the outstanding balance and 85 percent of the outstanding loans 

reported in the March 2008 lender performance report. For 504 lenders, we 

were able to obtain measures for 86 of the 270 lenders. We were able to 

obtain 47 of the 48 lenders in the largest peer group—that is, those lenders 

with more than $100 million in outstanding SBA-guaranteed loan balances. 

To determine how SBA uses the lender risk rating system in its lender 

oversight activities, we reviewed agency documents and conducted 

interviews to document SBA’s practices for assessing and monitoring the 

risk of lenders and loan portfolios. We then compared these practices 

against (1) the industry standards we identified through our interviews 

 

7

We tested the crosswalk file obtained from SBA by comparing the outstanding balance in 

the March 2008 lender performance report to the amount disbursed by lenders in the 

administrative data for the lenders that we matched. The correlation was .95 for 7(a) 

lenders and .99 for 504 lenders. We also compared the number of loans in the lender 

performance report and the number of loans in the administrative data. The correlation 

was .99 for 7(a) lenders and .99 for 504 lenders. 

8

Note that for one 7(a) and one 504 lender, we did not obtain a ranking because that lender 

was the reference category to which the other lenders’ odds were relative. 

Page 41 GAO-10-53 SBA's Lender Risk Rating System
Appendix I: Objectives, Scope, and 

Methodology 

with federal financial regulators and lenders and reviews of their 

documents and (2) our internal control standards. We also obtained and 

analyzed SBA data on risk ratings and on-site examinations from 2005 

through 2008 to determine the role that the lender risk ratings played in 

identifying lenders for an on-site review. 

To analyze the data on risk ratings and on-site examinations, we had to 

make a number of assumptions because the risk ratings were reported by 

quarter and we planned on reporting them by year. First, we assigned 

lender risk ratings in two different ways. For those lenders that were 

reviewed, we assigned them the risk rating that they received during the 

quarter that immediately preceded the on-site review. For those lenders 

that were not reviewed, we assigned them the lowest risk rating that they 

received during that given year. Second, we assigned lenders to peer 

groups in two different ways.9

 For those lenders that were reviewed, we 

assigned them the peer group that they were in during the quarter that 

immediately preceded their on-site review. For those lenders that were not 

reviewed, we assigned them the peer group they were in when they 

received their lowest risk rating. Because lenders are assigned a risk rating 

four times in a given year, there were some instances when they received 

the same low-risk rating multiple times in a given year but were in 

different peer groups when these ratings were assigned. In these instances, 

we relied on the most recent, lowest-risk rating score. For example, a 

lender could have received a lender risk rating of 4 in the second, third, 

and fourth quarter of a given year. However, the lender was in the highest 

peer group during the second and third quarters and in the second highest 

peer group in the fourth quarter. We would rely on the most recent 

quarter’s information and assign this lender a risk rating of 4 and the 

second highest peer group. Third, we determined the on-site review date in 

two ways. For on-site reviews completed in 2005 and 2006, we relied on 

the date that the final report for the on-site review was issued to determine 

when an on-site review was completed. For on-site reviews completed in 

2007 and 2008, we were able to rely on an additional variable included in 

the data that identified the date the on-site review was completed to 

determine when the on-site review was completed. 

We conducted this performance audit from August 2008 to November 2009 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

 

9

SBA assigns lenders to different peer groups based on their portfolio size. 

Page 42 GAO-10-53 SBA's Lender Risk Rating System
Appendix I: Objectives, Scope, and 

Methodology 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives. 

Page 43 GAO-10-53 SBA's Lender Risk Rating System
Appendix II: Comments from the Small Appendix II: Comments from the Small Business Administration 

Business Administration 

Page 44 GAO-10-53 SBA's Lender Risk Rating System 
Appendix II: Comments from the Small 

Business Administration 

Page 45 GAO-10-53 SBA's Lender Risk Rating System
Appendix II: Comments from the Small 

Business Administration 

Page 46 GAO-10-53 SBA's Lender Risk Rating System
Appendix II: Comments from the Small 

Business Administration 

Page 47 GAO-10-53 SBA's Lender Risk Rating System
Appendix III: Predictive Performance of the 

March 2007 and March 2008 Lender Risk Appendix III: Predictive Performance of the Ratings 

March 2007 and March 2008 Lender Risk 

Ratings 

We performed two types of statistical tests to determine how well SBA’s 

lender risk ratings predicted individual lender performance.1

 For both 

tests, we focused on how well the March 2007 lender risk ratings predicted 

the performance of lenders for the following year and how well the March 

2008 lender risk ratings predicted the performance of lenders for the 

following 6 months. First, we compared raw scores from SBA’s lender risk 

rating system to actual default rates for 7(a) and 504 lenders to determine 

how well the lender risk ratings identified the best and worst performing 

lenders. We divided lenders into two groups—those with lender default 

rates in the top 50 percent of all lender default rates and those with default 

rates that were in the bottom 50 percent of all lender default rates. We 

found that SBA’s risk ratings were generally successful at distinguishing 

the performance of about two-thirds of the 7(a) and 504 lenders in our 

sample (see tables 2 and 3). For example, table 2 shows that 96 of the 

approximately 300 lenders in our sample were in the top 50 percent based 

on the March 2007 lender risk ratings and actual lender default rates, while 

another 99 lenders were in the bottom 50 percent based on both rankings. 

We also compared how well an alternate measure of lender 

performance—the currency rate—divided lenders into these same two 

performance groups and found that overall, it also correctly separated 

about two-thirds of the lenders in our sample. 

 

1

In order to estimate default rates, we needed a meaningful number of loans for each 

lender. Therefore, we excluded from our sample 7(a) and 504 lenders that had less than 100 

loans approved between January 2003 and December 2007. As a result, our sample of 

lenders does not generally include lenders with smaller guaranteed portfolios (such as 

portfolios of less than $10,000,000). 

Page 48 GAO-10-53 SBA's Lender Risk Rating System 
Appendix III: Predictive Performance of the 

March 2007 and March 2008 Lender Risk 

Ratings 

Table 2: Comparison of Alternative Rankings and Rankings Based on 2007 Lender Risk Rating Raw Scores, 2007 Currency 

Rates, and 2008 Lender Risk Rating Raw Scores for 7(a) Lenders 

Comparison of March 2007 lender 

risk rating and defaults between 

March 2007 and March 2008 

Comparison of March 2007 currency 

rate and defaults between March 

2007 and March 2008 

Comparison of March 2008 lender 

risk rating and defaults between 

March 2008 and September 2008 

Ranking based on March 2007 

lender risk rating raw score 

Ranking based on March 2007 

currency rate 

Ranking based on March 2008 

lender risk rating raw score 

Alternative 

ranking 

based on 

defaults 

Top 

50% 

Bottom 

50% 

Total Top

50%

Bottom 

50%

Total Top 

50% 

Bottom 

50%

Total

Top 50% 96 56 152 88 64 152 87 66 153

Bottom 

50% 

55 99 154 62 92 154 64 91 155

Total 151 155 306 150 156 306 151 157 308

Source: GAO analysis of SBA data. 

Note: The number of lenders in the March 2007 lender performance report that we were able to 

match with default rates we produced was two less than in the March 2008 lender performance 

report. 

Table 3: Comparison of Alternative Rankings and Rankings Based on 2007 Lender Risk Rating Raw Scores, 2007 Currency 

Rates, and 2008 Lender Risk Rating Raw Scores for 504 Lenders 

Comparison of March 2007 lender 

risk rating and defaults between 

March 2007 and March 2008 

Comparison of March 2007 currency 

rate and defaults between March 

2007 and March 2008 

Comparison of March 2008 lender 

risk rating and defaults between 

March 2008 and September 2008 

Ranking based on March 2007 

lender risk rating raw score 

Ranking based on March 2007 

currency rate 

Ranking based on March 2008 

lender risk rating raw score 

Alternative 

ranking 

based on 

defaults 

Top 

50% 

Bottom 

50% 

Total Top

50%

Bottom 

50%

Total Top 

50% 

Bottom 

50%

Total

Top 50% 23 19 42 28 14 42 24 18 42

Bottom 

50% 

14 30 44 13 31 44 17 27 44

Total 37 49 86 41 45 86 41 45 86

Source: GAO analysis of SBA data. 

We used the same data to perform the second statistical test: determining 

the correlation between the rankings based on lender default rates and (1) 

the lender risk ratings and (2) the alternate measure—currency rate. We 

found that for both 7(a) and 504 lenders, there was a positive correlation 

between actual performance (lender default rates) and the lender risk 

ratings and currency rate. For the largest 7(a) lenders (that is, those 

Page 49 GAO-10-53 SBA's Lender Risk Rating System
Appendix III: Predictive Performance of the 

March 2007 and March 2008 Lender Risk 

Ratings 

lenders with SBA-guaranteed portfolios of at least $100 million), the lender 

risk ratings were more correlated to the lender default rates than was the 

currency rate. For 504 lenders, we found that both measures—the lender 

risk rating and the currency rate—performed about the same (see table 4). 

Table 4: Results of Correlation Analysis 

7(a) 504 

Measure Comparison 

$100 million 

or more

Between $10 

million and 

$100 million Total

$100 million 

or more 

Between $30 

million and 

$100 million Total

1 Raw rating score 

from March 2007 

Lender’s relative odds of 

default from March 2007 

through March 2008 

.48

(50)

.34

(183)

.31

(308)

.42 

(39) 

.42

(47)

.40

(86)

2 Gross currency 

rate from March 

2007 

Lender’s relative odds of 

default from March 2007 

through March 2008 

.17

(50)

.37

(183)

.35

(308)

.48

(39) 

.42

(47)

.42

(86)

3 Raw rating score 

from March 2008

Lender’s relative odds of 

default from March 2008

through September 2008 

.54

(56)

.21

(187)

.23

(308)

.32 

(47) 

.44

(39)

.38

(86)

4 Gross currency 

rate from March 

2008 

Lender’s relative odds of 

default from March 2007 

through September 2008 

.30

(56)

.16

(187)

.16

(308)

.34 

(47) 

.37

(39)

.34

(86)

Source: GAO analysis of SBA data. 

Note: The numbers in parentheses represent the number of lenders in each category. 

Page 50 GAO-10-53 SBA's Lender Risk Rating System
Appendix IV: Small Business Predictive Score Appendix IV: Small Business Predictive Score

The Small Business Predictive Score (SBPS) predicts loan performance. 

Specifically, it predicts the likelihood of severe delinquency (61 or more 

days past terms) over the next 18 to 24 months, including bankruptcies 

and charge-offs.1

 It is an off-the-shelf product that was developed by Fair 

Isaac using consumer and business credit bureau data. The model is able 

to produce scores—ranging from 1 to 300, 1 being highest risk and 300 

being lowest risk—using either a mix of consumer and business data, only 

data from the consumer credit bureaus, or only business data from Dun & 

Bradstreet. According to SBA officials, approximately 74 percent of its 

7(a) loans and 83 percent of its 504 loans are scored using both consumer 

and business data. Approximately 17 percent of its 7(a) loans and 8 

percent of its 504 loans are scored using consumer data only, while 9 

percent of its 7(a) loans and 504 loans are scored with Dun & Bradstreet 

data only. 

As we reported in 2004, Dun & Bradstreet collects these data from various 

sources and processes them through a five-step quality assurance process.2

First, Dun & Bradstreet collects data from more than 150 million 

businesses globally and continuously updates its databases more than 1 

million times daily based on real-time business transactions. Second, it 

matches SBA records with its records and achieves at least a 95 percent 

match of the data on 11 critical pieces of information used to identify the 

borrower. Third, Dun & Bradstreet assigns a unique identifier to each 

company. Fourth, Dun & Bradstreet identifies the corporate linkage of a 

business’s branches or subsidiaries with their parent entity to help SBA 

understand their complete corporate exposure between borrowers and 

their parent entities. Finally, Dun & Bradstreet generates predictive 

indicators of a business’s potential inability to repay a loan. Dun & 

Bradstreet officials refer to this process as the DUNSRight process. 

We performed independent tests to determine how well the SBPS 

predicted the performance of 7(a) loans. Specifically, we used a logit 

regression to determine how well the SBPS at loan origination predicted 

the default of loans with disbursement amounts above and below 

 

1

According to SBA officials, the SBPS was validated to be predictive of loan purchases, as 

well as delinquencies. 

2

GAO, Small Business Administration: New Service for Lender Oversight Reflects Some 

Best Practices, but Strategy for Use Lags Behind, GAO-04-610 (Washington, D.C.: June 8, 

2004). 

Page 51 GAO-10-53 SBA's Lender Risk Rating System 
Appendix IV: Small Business Predictive Score 

$150,000.3 We examined loans that were approved between 2003 and 2007 

and default rates over the period of January 2007 to September 2008. 

We found that the origination SBPS was predictive for loans that were 

both less than $150,000 and more than $150,000. However, the SBPS was 

estimated to have a larger effect on the performance of loans that were 

less than $150,000. Table 5 shows the coefficients from the logistic 

regression we ran. The coefficient estimated for the sample of loans that 

were less than $150,000 is more negative than that for loans that were 

more than $150,000, indicating that an increase in the SBPS (which 

represents a decrease in the predicted risk of the loan) lowers the rate of 

default by a greater increment. Additionally, as shown in the last column, 

the difference in the coefficients between the two groups is statistically 

significant. 

Table 5: Predictive Ability of SBPS for Loans below and above $150,000 

Subset of data 

Below $150,000 Above $150,000 Difference between the effects 

SPBS score -0.0243

(0.000297)

-0.0189

(0.000760)

0.00555

(0.000815)

Source: GAO analysis of SBA data. 

Note: Standard errors of the logit regression are in parentheses. The logistic regressions corrected for 

the age of the loans and economic conditions. Expressed in terms of a change in odds, a one-point 

increase in the origination SBPS will lower the odds of default in a specific quarter by 2.4 percent for 

loans below $150,000 and 1.9 percent for loans above $150,000. 

 

3

Our measure of default is the purchase rate. 

Page 52 GAO-10-53 SBA's Lender Risk Rating System
Appendix V: 

A

GAO Contact and Staff 

cknowledgments 

Page 53 GAO-10-53 

Appendix V: GAO Contact and Staff 

Acknowledgments 

William B. Shear, (202) 512-8678 or shearw@gao.gov 

In addition to the contact named above, Paige Smith (Assistant Director), 

Triana Bash, Ben Bolitzer, Tania Calhoun, Emily Chalmers, Marc Molino, 

Jill Naamane, Anh Nguyen, Carl Ramirez, and Stacy Spence made key 

contributions to this report. 

 SBA's Lender Risk Rating System 

GAO Contact 

Staff 

Acknowledgments 

(250418) 
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday afternoon, GAO 

posts on its Web site newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.” 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s Web site, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information. 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Public Affairs 

Please Print on Recycled Paper