Small Business Administration: Actions Needed to Improve the Usefulness of the Agency's Lender Risk Rating System
Published: 2009-11-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Small Business Administration (SBA) guarantees individual loans that lenders originate. The agency uses its Loan and Lender Monitoring System (L/LMS) to assess the individual risk of each loan, and SBA's contractor developed a lender risk rating system based on L/LMS data. However, questions have been raised about the extent to which SBA has used its lender risk rating system to improve its oversight of lenders. The Government Accountability Office (GAO) was asked to examine (1) how SBA's risk rating system compares with those used by federal financial regulators and lenders and the system's usefulness for predicting lender performance and (2) how SBA uses the lender risk rating system in its lender oversight activities. To meet these objectives, GAO reviewed SBA documents; interviewed officials from three federal financial regulators and 10 large SBA lenders; analyzed SBA loan data; and interviewed SBA officials.
section_title: What GAO Found
paragraphs: SBA's lender risk rating system uses some of the same types of information that federal financial regulators and selected large lenders use to conduct off-site monitoring, but its usefulness has been limited because SBA has not followed common industry standards when validating the system--that is, assessing the system's ability to accurately predict outcomes. Like the federal financial regulators and 10 large lenders GAO interviewed, SBA's contractor developed lender risk ratings based on loan performance data and prospective, or forward-looking, measures (such as credit scores). Using SBA data, GAO undertook a number of evaluative steps to test the lender risk rating system's predictive ability. GAO found that the system was generally successful in distinguishing between higher- and lower-risk lenders, but it better predicted the performance of larger lenders. However, the system's usefulness was limited because the contractor did not follow validation practices, such as independent and ongoing assessments of the system's processes and results, consistent with those recommended by federal financial regulators and GAO's internal control standards. For example, the agency did not require a party other than the one who developed the system to perform the validation, and SBA's contractor did not routinely reassess the factors used in the system as part of its validations. Further, SBA does not use its own data to develop alternate measures of lender performance that could be used to independently assess or supplement the risk ratings, citing resource constraints. Because SBA does not follow sound validation practices or use its own data to independently assess the risk ratings, the effectiveness of its lender risk rating system--the primary system SBA relies on to monitor and predict lender performance--may deteriorate as economic conditions and industry trends change over time. Although SBA's lender risk rating system has enabled the agency to conduct some off-site monitoring of lenders, the agency does not use the system to target lenders for on-site reviews or to inform the scope of the reviews. Unlike the Federal Deposit Insurance Corporation and the Federal Reserve, which use their off-site monitoring tools to target lenders for on-site reviews, SBA targets for review those lenders with the largest SBA-guaranteed loan portfolios. As a result of this approach, 97 percent of the lenders that SBA's risk rating system identified as high risk in 2008 were not reviewed. Further, GAO found that the scope of the on-site reviews that SBA performs is not informed by the lenders' risk ratings, and the reviews do not include an assessment of lenders' credit decisions. The federal financial regulators use the results of off-site monitoring to identify which areas of a bank's operations they should review more closely. Moreover, their reviews include an assessment of the quality of the lenders' credit decisions. Federal financial regulators are able to use review results to update their off-site monitoring systems with data on emerging lending trends. Regardless of the lender's risk rating, SBA relies on a standard on-site review form that includes an assessment of lenders' compliance with SBA policies and procedures but not an assessment of lenders' credit decisions. According to SBA officials, it is not the agency's role to assess lenders' credit decisions. Without targeting the most risky lenders for on-site reviews or gathering information related to lenders' credit decisions, SBA cannot effectively assess the risk posed by lenders or ensure that its lender risk rating system incorporates updated information on emerging lending trends.

section_title: Letter
paragraphs: In April 2003, the Small Business Administration (SBA) obtained a loan  monitoring service from Dun & Bradstreet to help manage and oversee the  lending and risk management activities of lenders that extend 7(a) and 504  loans to small businesses. The 7(a) and 504 loan programs, named after  the sections of the acts that authorized them, are SBA’s two major  business loan guarantee programs. As of June 30, 2009, SBA had an  outstanding portfolio of $67.6 billion in 7(a) and 504 loans. Because SBA  guarantees the individual loans that lenders originate, it uses the Dun &  Bradstreet service, now called the Loan and Lender Monitoring System  (L/LMS), to monitor the individual risk that each loan poses to the agency  in order to identify those lenders whose SBA loan operations and  portfolios may require additional monitoring or other actions. In 2004, we  reviewed the service and found that it was a positive and necessary step in  improving SBA’s oversight of lenders but determined that the agency  needed to develop policies and procedures to ensure that it used the  service in a way that resulted in improved oversight of lenders. Since we  issued our report in June 2004, SBA has made progress in developing  policies for using L/LMS and expanding its use. For example, SBA hired a  contractor to develop a lender risk rating system (that is, an off-site  monitoring tool that produces a risk score for each lender) based on  L/LMS data. This system enabled SBA for the first time to monitor the  approximately 4,000 smaller lenders that it had not previously reviewed.  However, questions have been raised about the extent to which SBA has  used its lender risk rating system to improve its oversight of lenders—for  example, to target lenders for on-site review. The SBA Inspector General  reported in May 2008 that SBA had been unable to sufficiently mitigate the  risk posed by lenders that it had identified as high risk and that SBA’s 7(a)  program had incurred a cumulative net loss for four lenders of $329  million as of September 2007.
You asked us to review SBA’s lender risk rating system and its effect on  the agency’s lender oversight program. Specifically, this report examines  (1) how SBA’s risk rating system compares with the off-site monitoring  tools used by federal financial regulators and lenders and the system’s  usefulness for predicting lender performance and (2) how SBA uses the  lender risk rating system in its lender oversight activities.
To determine how SBA’s lender risk rating system compares with off-site  monitoring tools used by federal financial regulators and lenders, we  compared SBA’s system with common industry standards that we  identified through interviews and document reviews. We interviewed  officials from three federal financial regulators—the Office of the  Comptroller of the Currency (OCC), the Board of Governors of the Federal  Reserve System (Federal Reserve), and the Federal Deposit Insurance  Corporation (FDIC)—five of the largest 7(a) lenders, and the five largest  504 lenders. We also reviewed relevant literature and analyzed procedural  manuals and other related federal guidance to banks on loan portfolio  monitoring. Although we interviewed federal financial regulators and  reviewed agency documents explaining their off-site monitoring practices,  we did not evaluate their practices, such as by testing their models. In  addition, we compared the techniques that SBA and its contractor used to  develop and validate the lender risk rating system to our internal control  standards. To determine the usefulness of the lender risk ratings in  predicting lender performance, we reviewed documents from SBA and its  contractor that described the factors used in the risk rating system and the  process for calculating the risk rating scores. We also obtained and  analyzed the following SBA data: data on loans approved in 2003 through  the end of 2007, the March 2007 and March 2008 lender risk ratings, and  the currency rate for each lender. We assessed the reliability of these data  and found them to be sufficiently reliable for our purposes. Using these  data, we undertook a number of evaluative steps to test SBA’s model. After  we discussed SBA’s modeling approach in detail with SBA officials and the  agency’s contractor to document the process used to develop the model,  we developed statistical estimation techniques to assess how well SBA’s  risk rating system predicts lender performance. In particular, we  compared the scores from the lender risk rating system to lenders’ actual  performance and alternate measures of lender performance that we  developed using SBA data. To determine how SBA uses the lender risk  rating system in its lender oversight activities, we compared SBA’s  practices for assessing and monitoring the risk of lenders and loan  portfolios against (1) the industry standards we identified through our  interviews and document reviews and (2) our internal control standards.  We also obtained and analyzed SBA data on risk ratings and on-site  examinations from 2005 through 2008 to determine the characteristics of  lenders that received on-site exams.
We conducted this performance audit from August 2008 to November 2009  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I contains a full  description of our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: In pursuing its mission of aiding small businesses, SBA provides them with  access to credit, primarily by guaranteeing loans through its 7(a) and 504  loan programs. The 7(a) and 504 loan guarantee programs are intended to  serve small business borrowers who could not otherwise obtain credit  under reasonable terms and conditions from the private sector without an  SBA guarantee. Under the 7(a) program, SBA generally provides  guarantees of up to 85 percent on loans made by participating lenders that  are subject to program oversight by SBA. Many of these participating  lenders are preferred lenders that have delegated underwriting authority.  Loan proceeds can be used for most business purposes, including working  capital, equipment, furniture and fixtures, land and buildings, leasehold  improvements, and certain debt refinancing. The 504 program provides  long-term, fixed-rate financing to small businesses for expansion or  modernization, primarily of real estate. Financing for 504 loan programs is  delivered through about 270 certified development companies, nonprofit  corporations that were established to contribute to the economic  development of their communities. For a typical 504 loan project, a third- party lender provides 50 percent or more of the financing pursuant to a  first-lien mortgage, a certified development company provides up to 40  percent of the financing through a debenture that is fully guaranteed by  SBA, and a borrower contributes at least 10 percent of the financing.  Although SBA’s 7(a) and 504 loan guarantee programs serve different  needs, both programs rely on third parties to originate loans (participating  lenders for 7(a) loans and certified development companies for 504 loans).  Because SBA generally guarantees up to 85 percent of the 7(a) loans and  up to 40 percent of the financing for 504 loan projects, SBA faces the same  kind of risk as the lenders if the loans are not repaid.
The Small Business Programs Improvement Act of 1996 required SBA to  establish a risk management database that would provide timely and  accurate information to identify loan underwriting, collections, recovery,  and liquidation problems. In 2003, SBA obtained a service from Dun &  Bradstreet that would allow it to, among other things, predict the  likelihood of a loan defaulting using a combination of SBA performance  data and loan-level credit data. In 2004, we assessed the new service and  found that the system was on par with industry best practices by providing  a tool that could help SBA better assess the risk exposure of loans in its  lenders’ portfolios. For example, we reported that the Small Business  Predictive Score (SBPS), which is provided through the Dun & Bradstreet  service, appeared to be consistent with private sector best practices  because it was based on sound models. The models used to score the  loans rely on data managed by Dun & Bradstreet and are commercial, off- the-shelf risk scoring models developed by Fair Isaac and validated to  SBA’s 7(a) and 504 portfolios. We concluded that without the Dun &  Bradstreet service, it was unlikely that SBA would be able to continue the  same level of risk management of its overall portfolio, its individual  lenders, and their portfolios. However, we also reported that SBA needed  to make better use of the service in overseeing its lenders and  recommended, among other things, that resources within SBA be devoted  to developing policies for the use of the loan monitoring service. As a  result, SBA contracted with Dun & Bradstreet to develop a system that  would rate lenders based on risk. Dun & Bradstreet subcontracted with  another company, TrueNorth, to develop the lender risk ratings—that is,  custom scores calculated using L/LMS data. Work on the lender risk rating  system started in 2004.
The purpose of the lender risk rating system is to improve the way SBA  monitors lenders. The lender risk rating system uses the following factors  for 7(a) lenders:    past 12 months’ actual purchase rate—a historical measure of SBA  purchases from the lender in the preceding 12 months;    problem loan rate—the current delinquencies and liquidations in a lender’s    3-month change in SBPS—a score that was developed to predict the  likelihood of severe delinquency (61 or more days past terms) over the  next 18 to 24 months, including bankruptcies and charge-offs; and   projected purchase rate—a measure of the amount of SBA guaranteed  dollars in a lender’s portfolio that is likely to be purchased by SBA.
Most of the data used to calculate these factors are loan and lender  performance information that come from SBA. The remaining data are  SBPSs or related scores provided by the Dun & Bradstreet service (see  table 1).
For 504 lenders, the risk rating is based on three factors: (1) the past 12  months’ actual purchase rate, (2) the problem loan rate, and (3) the  average SBPS on loans in the 504 lender’s portfolio. The third factor  replaced the third and fourth factors used for 7(a) lenders because it was  found during the testing process to be more predictive of SBA purchases  for 504 lenders.
Some federal financial regulators and lenders rely on similar tools to  conduct off-site monitoring. For example, FDIC relies on various off-site  monitoring tools, including a system called the Statistical CAMELS Off-site  Rating that helps the regulator identify institutions that have experienced  noticeable financial deterioration since the last on-site exam. The Federal  Reserve also relies on multiple tools to conduct off-site monitoring,  including a system that enables the regulator to predict how the risk level  of a bank likely will change in comparison to other banks that received  similar ratings on on-site exams. OCC relies on a process called a core  assessment that helps examiners assess the risk exposure for nine  categories of risk, including quantity, quality, and direction of risk.  Moreover, lenders frequently use models to summarize available relevant  information about borrowers and reduce the information into a set of  ordered categories, or scores, that estimate the borrower’s risk of  delinquency or default at a given point in time. Such tools are playing a  progressively more important role in the banking industry. In general, the  goal of these models—whether they are generic or custom, developed  internally or by third parties—is to obtain early indications of increasing  risk.
section_title: SBA’s Lender Risk Rating System Is Similar to Those Used by Federal Financial Regulators but Is Limited by Insufficient Validation
subsections: section_title: SBA’s Contractor Uses a Multistep Process to Assign Lender Risk Ratings
paragraphs: SBA’s contractor takes four steps to assign lender risk ratings each  quarter. First, the contractor separates lenders into peer groups based on  the size of their SBA loan portfolios in order to compare similarly sized  lenders. Second, for each lender, the contractor computes values for each  of the factors. As discussed in more detail in the background, the four  factors for 7(a) lenders are the (1) past 12 months’ actual purchase rate,  (2) problem loan rate, (3) 3-month change in the SBPS, and (4) projected  purchase rate. Third, the contractor inputs the value for each of the factors  into an equation to compute a score for each lender. Fourth, the  contractor uses the scores to place lenders into one of five risk rating  categories (1 through 5, with 1 indicating the least risk). Figure 1  illustrates this process for 7(a) lenders, and the shaded area represents a  specific example. The process is generally the same for 504 lenders.
According to SBA officials, this process for calculating lender risk ratings  will likely change in the near future because its contractor is redeveloping  the lender risk rating system. Several major changes are being  contemplated. First, the contractor plans to use an updated version of the  SBPS. Second, the contractor may use additional variables to calculate  lender risk ratings. Finally, rather than varying the equation by peer group,  SBA officials stated that they are considering a new variable that captures  the size of the lender’s portfolio and the age of its loans. The contractor is  still in the process of designing, testing, and documenting the new risk  rating system.
SBA rarely overrides risk ratings, but it may do so for several reasons.  These include early loan default trends; abnormally high default or  liquidation rates; lending concentrations; rapid growth in SBA lending;  inadequate, incomplete, or untimely reporting to SBA; and nonpayment of  required fees to SBA. In addition, SBA may override a lender risk rating  due to issues identified during an on-site review. For the quarter ending  September 30, 2008, SBA overrode the risk rating assigned by the  contractor in 20 cases; in each case, the risk rating increased.
section_title: SBA’s Lender Risk Rating System Uses Some of the Same Types of Data That Federal Financial Regulators and Selected Lenders Rely on to Conduct Off-Site Monitoring
paragraphs: SBA’s lender risk rating system uses some of the same types of data that  federal financial regulators and selected lenders rely on for off-site  monitoring. The federal financial regulators we interviewed rely on lender  information, performance data, and prospective measures to conduct off- site monitoring. Although the specific factors included in each regulator’s  off-site monitoring tools can vary, each regulator uses these three types of  data. Much of the lender and performance information they use are from  the call reports that banks submit quarterly and include data on equity,  loans past due, and charge-offs. Prospective measures include—when  available—borrowers’ credit scores from lender files. One federal  regulator is also working with a third party to obtain predictive scores,  similar to the SBPS, to use as part of its off-site monitoring. The large  lenders with whom we spoke also use performance data to rate loans,  focusing on factors such as portfolio performance, delinquencies, and  trends by state and industry type in order to forecast future losses.  Lenders also incorporate prospective measures, such as FICO scores and  SBPSs.
Like federal financial regulators and large lenders, SBA uses performance  data and prospective measures to calculate lender risk ratings. As we have  seen, to calculate risk ratings for 7(a) lenders, SBA relies on performance  data (the past 12 months’ actual purchase rate and the problem loan rate)  and prospective measures (the 3-month change in the SBPS and the  projected purchase rate). The 3-month change in the SBPS is also a  portfolio trend that has been incorporated into the rating system.  However, unlike the federal financial regulators, SBA does not use lender  information such as equity and loan concentrations as inputs into its  lender risk rating system. Although the federal financial regulators and  SBA both oversee lenders, their missions differ, and as a result they may  choose to focus on different variables in conducting off-site monitoring. In  general, the mission of the federal financial regulators is to maintain  stability and public confidence in the nation’s financial system. In contrast,  SBA’s mission is to aid, counsel, assist, and protect the interests of small  business concerns, including guaranteeing loans to businesses in  industries that lenders may avoid. Therefore, it is understandable that SBA  might not include the same variables as federal financial regulators. In  addition, while it is not an input into the lender risk rating system, SBA  evaluates information such as equity and loan concentrations as part of  other monitoring efforts. Figure 2 summarizes how the data that SBA uses  in its lender risk rating system compare with the data included in the risk  rating systems used by the federal financial regulators and lenders we  interviewed.
section_title: SBA’s Lender Risk Rating System Better Predicted the Performance of Larger Lenders than Smaller Lenders
paragraphs: When we performed our own independent assessments of the reliability of  the lender risk ratings, we found that they were more reliable at predicting  the performance of the largest lenders. To perform this independent  assessment, we assessed how well the lender risk ratings predicted the  actual performance of lenders (that is, lenders’ default rates). Because of  data limitations, our analyses focused on lenders with larger SBA- guaranteed portfolios. Overall, we found that SBA’s ratings were able to  distinguish between high- and lower-risk lenders for a majority of the 7(a)  and 504 lenders in our sample for 2007 and 2008. However, when we  focused on the ratings’ ability to predict the performance of different-sized  lenders, we found that the ratings were more effective at predicting the  performance of lenders with the largest SBA-guaranteed portfolios (that is,  lenders with SBA-guaranteed portfolios of at least $100 million). (See app.  III for further discussion of how well the lender risk ratings predicted the  performance of 7(a) and 504 lenders.)
How the system was developed may have contributed to the lender risk  ratings being more effective at predicting the performance of the largest  lenders (that is, lenders with SBA-guaranteed portfolios of at least $100  million). In order to determine how SBA developed the risk rating system,  we reviewed the available documentation of the development process and  discussed the process with SBA officials and the contractor. According to  the contractor, it considered 32 variables to determine those that were the  most predictive for each peer group. SBA then made a policy decision to  use the same factors across all of the peer groups. Although the  documentation did not provide the justification for this policy decision,  SBA officials stated that the decision was made so that every lender’s risk  rating was based on consistent information. Officials were concerned that  lenders might be confused if the factors upon which the ratings were  based varied by peer group, particularly since lenders do move between  peer groups. The contractor ultimately selected four factors, each of which  was a statistically significant predictor of lender performance for at least  one of the peer groups. However, only for the largest peer group (those  with guaranteed portfolios of at least $100 million) were all four factors  statistically significant. According to SBA officials, in peer groups where a  factor was statistically insignificant, it did not affect the lenders’ risk  ratings—that is, for some peer groups, the ratings are determined by less  than four factors.
section_title: Usefulness of SBA’s Lender Risk Rating System Has Been Limited because SBA Does Not Ensure That Its Contractor Follows Sound Validation Techniques
paragraphs: The effectiveness of SBA’s lender risk rating system has been limited  because the agency’s contractor does not follow sound validation  practices. According to one federal financial regulator, the ability of  models to accurately predict outcomes can deteriorate over time. For  example, changes in economic conditions and industry trends can affect  model outcomes. Validation—the process of assessing whether ratings  adequately identify risks by, for example, comparing predictions to actual  results—helps to ensure that models remain reliable. Federal financial  regulators (OCC, FDIC, and the Federal Reserve) and the Basel Committee  on Banking Supervision (Basel Committee) have developed a number of  common principles that financial institutions should follow in validating  the models they use to manage risk, whether the models are purchased  from a vendor or developed in-house. Validating some aspects of models  developed by vendors may be difficult because of the proprietary nature of  the information. But the guidance from federal financial regulators and the  Basel Committee states that organizations have a responsibility to ensure  that vendors follow good model validation practices.
We identified four key elements of a sound validation policy that federal  financial regulators and our internal control standards recommend and  that some lenders we interviewed implemented. First, all three parts of a  model—the data, processes, and results—should be validated using  multiple techniques. Second, validation should be done by an independent  party. Third, validation should include an ongoing assessment of the  factors used in the model. Finally, the validation procedures should be  documented. We found, however, that SBA had not adhered to the  guidance in validating its lender risk rating system. First, SBA’s validation  procedure does not include techniques to validate all parts of its model.  Second, the model is not validated by an independent party. Third, SBA  does not reassess which variables are the most predictive of lender  performance on a routine basis. Finally, SBA’s documentation of the  validation procedures and the results of the validation is not complete.  Figure 3 shows how SBA’s practices align with commonly accepted  practices.
Guidance from the federal financial regulators we interviewed and the  Basel Committee states that each of the three parts of a model—the data,  processes, and results—should be validated using a variety of techniques.  According to FDIC guidance, validation should include ensuring that the  data used in the model are accurate and complete, evaluating the model’s  conceptual soundness, and analyzing the estimates the model produces  against actual outcomes. The Basel Committee also states the importance  of assessing all the components of a model. In addition, OCC guidance  prescribes three generic procedures that could be used for validating each  part of a model—a review of logical and conceptual soundness,  comparison against other models, and comparison against subsequent  actual events. Further, guidance from the Federal Reserve states that  financial institutions should use a variety of techniques when validating  their models. For example, some lenders we interviewed compared their  internal rating systems with other commercially available models or  compared model predictions against historical information to test the  reliability of their models. In addition, GAO’s internal control standards  specify that agencies should ensure the accuracy of data inputs and  information system processing and results. For example, validation  should be performed to verify that data are complete and to identify  erroneous data. Furthermore, these standards state that management  should establish controls over information processing and that output  reports should be reviewed.
Consistent with commonly accepted practices, SBA’s contractor has a  documented process for validating the data used in the lender risk rating  system. On the basis of previous reviews and recent interviews with  contractor staff, we found that the contractor’s data quality control  process, referred to as DUNSRight, appeared reasonable. In June 2004, we  reported that the commercial data that Dun & Bradstreet collects go  through a five-step quality assurance process that includes continuously  updating databases and matching SBA records with Dun & Bradstreet  records, with a 95 percent match of the data on critical pieces of  information. In the same report, we also concluded that SBA’s controls  over the 7(a) and 504 data used in the models helped to ensure that the  data inputs were sufficiently reliable. Appendix IV provides information on  Dun & Bradstreet’s procedures for ensuring the reliability of the SBPS and  how well it predicts the likelihood that a loan will default.
The contractor that developed the lender risk rating system also conducts  periodic validations of the system that include using statistical tests to  measure the model’s predictive ability and comparing the results of the  model against lenders’ actual performance. For the years 2005 through  2007, SBA’s contractor assessed whether the broad risk ratings were  generally consistent with the actual performance of the lenders within  each rating group. The contractor also determined whether each group of  lenders (for example, those lenders rated as 1) performed better than  other groups of lenders with lower risk ratings (that is, 2 through 5).  However, we did not see evidence that the contractor validated the  processes used to calculate the ratings. Specifically, neither SBA nor its  contractor could provide documentation showing that the contractor had  validated the theory behind the system or the logical and conceptual  soundness of the model. For example, there was no documentation  describing the processes followed or the link between the computer  program and output that was used to produce the lender risk ratings.  Therefore, we could not rerun the analysis to determine if we would have  arrived at the same conclusion regarding the four factors used in the  model. In addition, the contractor could not provide documentation  showing that it had ensured that the mathematics and computer code were  free of errors. According to officials from the contractor, they took steps  to verify that the processes they followed were sound, including verifying  the computer code they used; however, they did not document these steps.
Further, the contractor’s validation of the model’s results was limited.  Consistent with industry standards, SBA’s contractor has used a variety of  statistical measures to validate the risk rating system’s results. But the  documentation did not show that the contractor checked the model’s  results against available benchmarks (such as the default rate or the  currency rate) to validate whether the risk ratings reliably predicted  individual lender performance. Rather, the documentation indicated that  the contractor focused its validation on whether the broad risk ratings  were generally consistent with the actual performance of the lenders  within each rating group—groups that can be comprised of over 2,000  lenders with a wide range of portfolio sizes and performance levels.  Although this technique compares the model’s results to actual  performance benchmarks, as suggested by industry standards, it is limited  because it does not provide information on individual lender performance.  According to SBA officials, the contractor tested how well individual  scores produced by the lender rating system predicted individual lender  performance; however, the results of this analysis were not included in the  documentation we received and were not provided to SBA. Because lender  performance can vary widely within the broad risk categories, the results  of a more refined analysis would allow SBA to identify specific lenders  placed in incorrect risk categories.
Because SBA has never requested documentation from the contractor on  its validation of the model’s processes, the agency cannot ensure that the  processes used are sound. In addition, because the contractor does not  document how well the lender risk ratings predict individual lenders’  performance, SBA may not be able to identify which lenders within the  broad risk rating categories are not being rated accurately. As a result,  SBA may be relying on inaccurate ratings or missing out on opportunities  to identify risky lenders and target them for closer monitoring.
subsections: section_title: Validation Is Not Conducted by an Independent Party
paragraphs: Each of the regulators we interviewed (OCC, FDIC, and the Federal  Reserve) recommends in its guidance that validation include an  independent review of the model. For example, OCC guidance states that  model validation should be done by a party that is as independent as  possible from the personnel who constructed the model. In addition, FDIC  guidance states that validation should include competent and independent  review by a reviewer who is as independent as practicable. Further,  Federal Reserve and Basel Committee guidance notes that the validation  process should be independent from the model development and  implementation processes. Our internal control standards also emphasize  the importance of independent review. They state that to reduce the risk  of error, no one individual should control all key aspects of an activity.  For example, an individual who is responsible for developing a model  should not be responsible for validating it. An independent party can be  either inside or outside the organization—for example, the internal audit  staff, a risk management unit of the institution, an external auditor, or  another contracted third party. Some lenders we interviewed that had  internal risk rating systems have had them validated by a separate group  within the institution, and others have invited independent auditors to  review their systems.
Contrary to common industry practices and internal control standards, the  same contractor staff that developed and maintain the lender risk rating  system are the officials who validate it. We have previously reported on  SBA’s failure to ensure that independent parties routinely assess the  reliability or integrity of its contractors’ models. Specifically, we reported  in June 2004 that third parties did not validate the SBPS model that  another contractor maintained because SBA believed that the model was  stable and that clients would inform the company if the models were not  reasonably predicting borrower behavior. Similarly, SBA and its  contractor thought it was sufficient for someone to review the validation  conducted by the staff who developed the model and for Dun & Bradstreet  and SBA officials to review the contractor’s work. However, industry  standards require that personnel other than those who developed the  model validate it. Because SBA has not ensured that an independent party  validates its lender risk ratings, certain systemic and structural issues with  the design of the system may go undetected, and the predictive value of  the risk ratings is more uncertain.
section_title: SBA Does Not Perform Ongoing Validation to Ensure That the Factors Used in the System Are the Most Predictive
paragraphs: Guidance from federal financial regulators and the Basel Committee states  that validation of the factors used in the model should be ongoing and  should take into consideration changes in the environment (such as  changes in economic conditions or industry trends) or improvements in  modelers’ understanding of the subject. For example, OCC guidance states  that models are frequently altered in response to changes such as these. In  addition, Federal Reserve guidance states that a model’s methodology  should be validated periodically and modified to incorporate new events  or findings as needed. Further, the Basel Committee notes that validation  is an ongoing, iterative process. Failure to do so could cause the model to  become less predictive and lose its ability to rank order risk over time.  According to FDIC guidance, characteristics of a model need to be  validated and refined when necessary because if management does not  select and properly weight the best predictive variables, the model’s  output will likely be less effective. Our internal control standards also  specify that agencies that procure commercial software are responsible  for ensuring that it meets the user’s needs and is operated properly.  These standards state that controls should be in place to ensure that  computer systems are modified safely by reviewing and testing them  before placing them into operation. The standards also specify that  management should ensure that ongoing monitoring is effective and will  trigger separate evaluations where problems are identified.
SBA’s contractor takes some steps to validate the lender risk rating  system’s ability to reliably predict lender performance but does not ensure  that the variables used to calculate the risk ratings are the most predictive  of lender performance. We reviewed the validations of the risk rating  system that the contractor conducted in 2005, 2006, and 2007. These  validation efforts included testing of the statistical importance of each of  the four factors used in the lender risk rating system. However, these  validations did not routinely include testing of other factors to account for  changes in economic conditions or industry trends. The 2005 validation  effort was the only one that tested additional factors. SBA’s contractor  tested three new variables to determine if they improved the model’s  ability to predict lender performance and found that they did not. Neither  of the subsequent validations included assessments of additional variables,  and SBA did not requested them. According to SBA officials, SBA and the  contractor identified possible additional variables over the past several  years that they did not test for use in the model because they wanted more  experience with it and the data. They also noted that they always had  plans to redevelop the model within 5 years but could not do so until the  agency had signed a second contract with Dun & Bradstreet that provided  funds for a redevelopment. However, if SBA had asked the contractor to  test additional factors on a regular basis, the agency may have found that  an earlier redevelopment effort or incremental adjustments could have  improved the predictive ability of the model. Because new variables that  might take into account economic changes or industry developments have  not been routinely assessed, the ratings may not be as effective as they  could be.
In addition, according to the contractor’s validation reports, the lender  risk rating system’s predictive ability for 7(a) lenders decreased from 2005  to 2007. This decrease led the contractor to suggest in 2007 that SBA  redevelop the model to improve its predictive ability and prevent further  deterioration. SBA officials agreed, and the contractor is currently  redeveloping the model, including testing new variables, to keep up with  changing economic conditions and to reflect SBA’s and the contractor’s  experiences working with the data and the model over the last several  years. It will be important for SBA to ensure that the contractor conducts  sound testing as part of its redevelopment.
section_title: SBA’s Documentation of Validation Procedures and Results Is Incomplete
paragraphs: The federal financial regulators’ guidance states that a sound validation  policy should include documentation of the validation. For example, FDIC  and OCC guidance states that model validation documentation should  describe the model, how it is used, and its limitations. Federal Reserve  guidance also notes that the validation process should be documented. In  addition, FDIC and OCC have said that the procedures used to validate the  model on an ongoing basis and the results of these validations should be  documented, even if the institution uses a model developed by a vendor.  For example, OCC guidance states that an institution should seek  assurances that the vendor’s model is defensible and works as promised.  Further, the Basel Committee guidance notes that even vendors that are  not willing to reveal proprietary information should provide information  on the validation techniques they use. Complete documentation of the  results of ongoing validations assists users in understanding the model and  facilitates independent reviewers’ assessments of the model’s validity. Our  internal control standards also specify the importance of documenting  information systems. For example, these standards state that all  significant events in developing and maintaining computer systems should  be clearly and completely documented. This documentation should  describe the system, how the data used in the system are handled, and  other controls in place to maintain the system.
SBA did not ensure that the contractor provided complete documentation  of the results of its validations or documented its validation procedures.  SBA provided us with some documentation of the contractor’s process for  validating the data used in the lender risk rating system, but  documentation of the results of the validations was inconsistent and did  not have information on the procedures for validating the model’s  processes. For example:    The validation reports we reviewed (2005 to 2007) did not always include  information on the statistical measure the contractor used to describe the  model’s predictive abilities. The 2006 validation report did not contain this  statistic for the 7(a) ratings, and only the 2007 report included it for 504  lender risk ratings.
The validation reports did not describe the contractor’s validation  procedures. As noted previously, SBA did not provide documentation  showing that the contractor validated the mathematics and computer code  used in the model.
The validation reports did not explain why in 2005 the contractor  considered whether additional variables would improve the model’s ability  to predict lender performance but did not consider additional variables in  other years.
The validation reports did not describe any limitations of the model that  would have helped SBA to use the results accurately.
Officials from the contractor explained that the documentation provided  was typical of that seen in the private sector for such models, but stated  that they would provide more detailed documentation in the future.
Because SBA does not ensure that its contractor completely documents its  validation procedures and results, it is difficult to assess the sufficiency of  the validations performed. Further, as we noted previously, it is important  for an independent party to validate a model’s reliability. Without clear  documentation explaining the model’s limitations, the validation  procedures, and the results of the validations, an independent reviewer  would have difficulty conducting a thorough assessment of SBA’s model.
section_title: SBA Does Not Use Its Own Data to Assess or Supplement the Contractor’s Validation of the Lender Risk Rating System
paragraphs: In addition to not ensuring that its contractor follows sound validation  techniques, SBA does not conduct its own analysis of data to supplement  the contractor’s validation of the lender risk rating system. According to  the Basel Committee guidance we reviewed, organizations must have  clearly articulated strategies for regularly reviewing the results of vendor  models and the integrity of the external data used in these systems.  Further, OCC guidance states that vendor models should generally be held  to the same minimum validation standards as internally developed models.  When full and complete details concerning aspects of a vendor product are  lacking, OCC and Basel Committee guidance states that organizations  should rely more heavily on alternative validation techniques to  compensate for the lack of access to full information. This guidance notes  that in such cases, it is critical for organizations to test the results of the  vendor’s model at least once a year using their own data on actual  performance to assess the model’s predictive ability. This procedure helps  to ensure that the models continue to function as intended and verifies the  reliability and consistency of any external data used. Our internal control  standards state that monitoring should be performed continually and that  it should involve comparisons and reconciliations. For example, these  standards specify that agencies should compare information generated  from computer systems to actual records. Agencies should also analyze  and reconcile any differences that might be found.
SBA does not use its own data to independently assess the lender risk  rating system’s results. According to a 2007 SBA Inspector General report,  SBA has previously rejected using its own data to develop lender  performance benchmarks that could be used in lieu of or in conjunction  with the risk ratings because doing so would be time-consuming and the  benchmarks would have to be monitored and replaced as program and  economic conditions changed. However, we found that SBA data could  be useful for developing alternate measures of lender performance in  order to independently validate the lender risk rating system’s results. For  example, SBA could perform analyses similar to those we performed by  using its own data to compare risk ratings with actual lender default rates.  Further, SBA could use its own data to develop alternate measures, such  as currency rates, as performance benchmarks. As we did in our analyses,  SBA could compare how well lender risk ratings predicted actual  performance to how well an alternate measure demonstrated lender’s  actual performance. Because of data limitations, our analyses focused on  lenders with larger SBA-guaranteed portfolios. As a result, we were unable  to determine how well these alternate measures predict the performance  of lenders with smaller portfolios, but SBA has more years of data  available to facilitate such analyses. Without performing its own  assessment, the agency may not be able to identify issues with the model’s  ability to reasonably predict lender performance and notify the contractor.  As a result, SBA may miss opportunities to identify risky lenders and  mitigate the risks they pose to SBA’s portfolio.
section_title: SBA Does Not Use Lender Risk Ratings to Target Lenders for On-Site Review or Tailor the Scope of the Reviews
subsections: section_title: SBA Has Used the Lender Risk Rating System to Conduct Some Off-Site Monitoring of Lenders and Their Portfolios
paragraphs: SBA uses its lender risk rating system to conduct off-site monitoring of  lenders and their portfolios. In addition to routine on-site reviews, federal  financial regulators and lenders use off-site tools to monitor lenders’  performance and portfolio trends. As part of a comprehensive risk  management strategy, federal financial regulators use risk ratings to  conduct portfolio analysis and identify problem trends. FDIC relies on a  number of off-site monitoring tools to perform horizontal analyses (that is,  compare similar lenders) and analyze emerging lending trends. For  example, when subprime lending first began, the agency tracked the  amount of subprime lending that each of its lenders did. The Federal  Reserve uses various off-site monitoring tools that focus on asset quality  and credit risk to identify banks whose ratings appear to have deteriorated  since their most recent on-site reviews. For example, it analyzes  information related to nonperforming and performing loans and the  changing composition of loan concentrations. OCC uses its core  assessment process to assess how much risk lenders have taken on and  the quality of their risk management to determine aggregate risk.
Lenders also use off-site monitoring tools to oversee loan portfolios. For  example, one 7(a) lender we interviewed uses various scoring models to  determine, among other things, how each loan’s risk rating has changed  since the loan was originated. Other 7(a) lenders with whom we spoke use  off-site monitoring tools that analyze factors such as geography, industry,  management quality, company performance, and collateral to predict the  risk of loans. Another 7(a) lender relies on several off-site monitoring  systems to track portfolio performance—including delinquencies and  trends by state, industry, and North American Industry Classification  System (NAICS) code—and forecast losses. In addition, bank officials we  interviewed stated that they reviewed all troubled loans on a monthly  basis.
Similarly, SBA uses its lender risk rating system to obtain quarterly  performance information on all lenders and determine portfolio trends.  SBA officials stated that before they had the risk rating system, they were  not able to analyze the performance of all lenders, especially lenders with  the smallest volume of SBA-guaranteed loans. SBA has formed a Portfolio  Analysis Committee that meets monthly to discuss portfolio trends  identified by analyzing loan and lender performance data. Comprised of  top SBA officials, the committee typically discusses delinquencies,  liquidations, charge-offs, and purchase rate trends by delivery method  (that is, various SBA loan programs) for the 7(a) and 504 portfolios. The  committee also discusses changes in loans’ SBPSs (from the end of the  quarter in which the loan was disbursed to the most recent quarter) and  the scores’ performance in ranking loans. To date, SBA has taken some  actions as a result of these meetings. For example, SBA officials told us  that as a result of discussions about portfolio performance during these  meetings, they discontinued an SBA program that allowed borrowers to  provide limited documentation.
SBA officials told us that the agency also recently began using the results  of the lender risk rating system to conduct “performance-based reviews.”  According to SBA officials, the purpose of these reviews is to perform  more in-depth, off-site monitoring that incorporates lenders’ information,  such as lender financial ratios from call reports, that is currently not part  of the lender risk rating system. Specifically, SBA financial analysts are  assigned lenders that they will monitor over time. Each year, the analysts  will focus on lenders with outstanding balances on their SBA portfolios of  at least $10 million that are not scheduled for on-site reviews and on all  other preferred lenders regardless of size. With the remaining resources,  they will review small problem lenders—for instance, those with  guaranteed portfolios that are less than $10 million but that received a  lender risk rating of 4 or 5. SBA had conducted 517 of these reviews as of  August 2009.
section_title: SBA Has Not Effectively Integrated Its Lender Risk Rating System into the On- Site Examination Process
paragraphs: Although SBA has begun some off-site monitoring using its risk rating  system, it does not use the ratings to target lenders for on-site reviews.  FDIC and the Federal Reserve use risk ratings as the primary tool for  identifying lenders that need to be reviewed. For example, FDIC stated  that they relied on off-site monitoring to determine the scope and  frequency of on-site exams. Our internal control standards require that  agencies assess and mitigate risks using quantitative and qualitative  methods and then conduct a thorough and complete analysis of those  risks. Although SBA identifies the risks that lenders pose, it does not  mitigate these risks because it chooses not to target high-risk 7(a) and 504  lenders for on-site reviews. Instead, the agency targets lenders for reviews  based on the size of their portfolios, focusing primarily on the largest  lenders—that is, 7(a) lenders with at least $10 million in their guaranteed  loan portfolio and 504 lenders with balances of at least $30 million. Only  when prioritizing large lenders for review does SBA consider their risk  ratings.
We found that in calendar years 2005 to 2008, most of SBA’s 477 on-site  reviews were of large 7(a) and 504 lenders that posed limited risk to SBA.  Ninety-nine percent (472 of 477) of the lenders reviewed were large  lenders, and 80 percent (380 of 477) posed limited risk to SBA (that is,  were rated as a 1, 2, or 3 by the lender risk rating system). The agency has  increased the number of on-site reviews performed (from 69 in 2005 to 188  in 2008) because it can now charge lenders for them. However, SBA  continues to conduct a limited number of reviews of high-risk lenders or  those with a lender risk rating of 4 or 5 (see fig. 4). In 2005, 20 percent (14  of 69) of SBA’s on-site reviews were of lenders that posed significant risk  to the agency. In 2008, that proportion was 22 percent (42 of 188 reviews).  As a result, a substantial number of high-risk lenders were not reviewed  each year. For example, in 2008, only 3 percent of the 1,587 lenders that  posed significant risk to SBA were reviewed. Because SBA relies on  lenders’ size to target lenders for on-site reviews, smaller lenders that,  based on their high-risk ratings, pose significant risk to SBA have not  received oversight consistent with their risk levels.
Our findings are similar to those of SBA’s Inspector General. In a 2007  report, the Inspector General concluded that SBA had made limited use of  lender risk ratings to guide its oversight activities. It observed that the  agency reviewed large lenders regardless of their risk ratings and did not  do on-site reviews of smaller lenders with high-risk ratings. The report  recognized that some of the smaller lenders might not have a sufficient  number of loans in their portfolio to warrant an on-site review but noted  that others could have a significant number of loans. The Inspector  General recommended that SBA develop an on-site review plan or agreed- upon procedures for all high-risk 7(a) lenders with guaranteed loan  portfolios in excess of $4 million. We agree that although not all of the  small lenders with high-risk ratings warrant more targeted monitoring,  some do. Of the 1,545 high-risk lenders that we found were not reviewed in  2008, 215 lenders had an outstanding portfolio of at least $4 million.  According to SBA officials, the agency is developing agreed-upon  procedures for conducting additional reviews of smaller lenders in  response to the Inspector General’s recommendation.
section_title: Lender Risk Ratings Do Not Inform the Scope of SBA’s On-Site Reviews, and Reviews Do Not Include an Assessment of Lenders’ Credit Decisions
paragraphs: Unlike federal financial regulators, SBA does not rely on its lender risk  ratings to help focus the scope of on-site reviews, and the reviews do not  include an assessment of the lenders’ credit decisions. The federal  financial regulators we interviewed rely on results from their off-site  monitoring systems to identify which areas of a bank’s operations they  should review more closely. Using the results of the off-site monitoring,  they are able to tailor the scope of their on-site reviews to the specific  areas of lenders’ operations that pose the most risk to the bank. In  addition, during on-site reviews, the federal financial regulators often  include an assessment of the quality of lenders’ credit decisions. They told  us that the results of their on-site reviews helped not only to assess the  risk that lenders posed, but also to identify emerging lending trends and  areas of banking operations that may pose significant, new risk to banks in  the future. They are then able to use the results to inform their off-site  monitoring systems. For example, regulators stated that when their on-site  reviews showed an increase in subprime lending, they incorporated  subprime lending data into their off-site monitoring tools. Although SBA’s  mission differs from the mission of the federal financial regulators,  internal control standards require all federal agencies to identify and  analyze risk, as well as to determine the best way to manage or mitigate it.
According to SBA’s Standard Operating Procedure for on-site reviews, the  agency assesses a lender’s (1) portfolio performance, (2) SBA management  and operations, (3) credit administration practices, and (4) compliance  with statutes and SBA regulations and policies. For the portfolio  performance component, SBA uses L/LMS data to review the size,  composition, performance, and credit quality of a lender’s SBA portfolio.  When assessing a lender’s SBA operations, SBA evaluates, among other  things, the lender’s internal policy and procedural guidance on SBA  lending; the competence, leadership, and administrative ability of  management and staff who have responsibility for the SBA loan portfolio;  and the adequacy of the lender’s internal controls. For the credit  administration component, SBA assesses the lender’s policies and  procedures for originating, servicing, and liquidating SBA loans. An SBA  contractor then uses this information during file reviews to determine the  degree to which lending policies and procedures are followed. For the  compliance component, SBA’s contractor performs file reviews that focus  on the lender’s compliance with SBA-specific requirements.
When performing file reviews, contractor staff do not rely on results from  the lender risk rating system to tailor the scope of the reviews. Instead,  contractor staff rely on a standard form—the lender review checklist—to  conduct all file reviews, regardless of the lender risk rating or other  information available to SBA about the lender’s portfolio. Moreover, these  file reviews do not include an assessment of the quality of the credit  decisions made by lenders. Rather, the lender review checklist focuses  primarily on the lenders’ adherence to SBA policies, including those based  on statutes or regulations, when making SBA-guaranteed loans. The  checklist includes questions related to, among other things, the  determination of borrower eligibility (including whether the borrower had  any other outstanding SBA loans that are not current), the calculation of  collateral value, and evidence that all required forms were obtained and  reviewed. According to SBA officials, the file reviews focus on compliance  with SBA policy because it is not SBA’s role to evaluate lenders’ credit  decisions. The officials did not believe that the agency should be setting  policy or underwriting standards for lenders. However, because SBA relies  on lenders with delegated underwriting authority to make the majority of  its loans, we believe that SBA should take a more active role in ensuring  that these lenders are making sound credit decisions.
We originally reported on SBA’s compliance-based reviews in 2002, when  we found that SBA’s automated checklist lacked the substance to provide  a meaningful assessment of lender performance. We reported that SBA’s  on-site reviews were based on reviewers’ findings from a lender  questionnaire and a review checklist in order to ensure objective scoring.  The lender questionnaire addressed organizational structure, oversight  policy, and controls. SBA officials said that prior to the implementation of  the automated worksheet scoring process, on-site reviews were done in a  narrative format, and reviewers’ assessments of lender performance were  subjective. They noted that the worksheet format made the reviewers’  assessments of lenders more consistent and objective. As previously  mentioned, SBA has since expanded the scope of its on-site reviews to  include more than just a compliance component and revised the checklist  used to conduct file reviews. But, as noted previously, the revised  checklist still focuses on compliance with SBA policies and procedures.
An example from our February 2009 report on compliance with the credit  elsewhere requirement illustrates SBA’s emphasis on ensuring policy  compliance rather than verifying lenders’ credit decisions during on-site  reviews. Because the 7(a) and 504 programs are intended to serve  borrowers who cannot obtain conventional credit at reasonable terms,  lenders making 7(a) and 504 loans must ensure that borrowers meet the  credit elsewhere requirement. This statutory requirement stipulates that to  receive loans, borrowers must not be able to obtain financing under  reasonable terms and conditions from conventional lenders. During an on- site review, the contractor is to determine whether lender policies and  practices adhere to SBA’s credit elsewhere requirement. During the  review, SBA’s contractor explained that it checks to see that the lender  documented its credit elsewhere determination and cited one of the six  factors that SBA has determined are acceptable reasons for concluding  that a borrower could not obtain credit elsewhere. However, it does not  routinely assess the information lenders provide to support credit  elsewhere determinations. Contract staff answer “yes” or “no” on the  checklist that “written evidence that credit is not otherwise available on  terms not considered unreasonable without guarantee provided by SBA”  was in the file. Contractor officials stated that when the documentation  standard is not met, the examiner will sometimes look at the factual  support in the file to independently determine whether the credit  elsewhere requirement was actually met.
Because SBA officials choose not to rely on lender risk ratings to inform  file reviews conducted during on-site reviews or assess lenders’ credit  decisions during the reviews, the agency does not have the type of  information related to the quality of the underwriting standards and  practices of lenders that is necessary to understand the risks that banks  pose to SBA’s portfolio. Without this information, the agency cannot make  informed improvements to the lender risk rating system that would enable  it to take into account new emerging lending trends.
section_title: Conclusions
paragraphs: Because SBA relies heavily on its lenders to determine if loans are eligible  for an SBA guarantee and to underwrite the loans, lender oversight is of  particular importance. By working with a contractor to develop a lender  risk rating system, SBA has taken a positive step toward improving its  oversight of lenders. The lender risk rating system enables SBA for the  first time to systematically and routinely monitor the performance of all  lenders, including lenders with the smallest loan portfolios, which SBA  had not routinely monitored. However, SBA does not ensure that its  contractor follows sound practices when validating the system. Guidance  from the federal financial regulators we interviewed states, among other  things, that validation should be performed by an independent party and  should routinely reassess the factors used to determine risk, taking into  consideration changes in the environment (such as changes in industry  trends). SBA did not require its contractor to ensure that personnel other  than the staff who developed the model validated it or to routinely  reassess the factors used in the system as part of its validations. Unless  SBA ensures that its contractor follows sound model validation practices,  the agency’s ability to identify inaccurate ratings, detect systemic or  structural issues with the design of the model, and determine whether the  ratings are deteriorating over time as economic conditions change will be  limited. SBA’s contractor is currently redeveloping the lender risk rating  system to improve its predictive ability. However, the benefits that may be  achieved through the redeveloped lender risk rating system will be limited  if SBA continues the practice of not ensuring that its contractor adopts  sound validation practices. In particular, testing to ensure that the system  effectively evaluates risk is an important element to improve a risk rating  system, regardless of whether such testing occurs during routine  validation efforts or during model redevelopment.
In addition, contrary to federal financial regulator guidance and our  internal control standards, SBA has not used its own data to conduct  independent assessments of the risk rating system to help ensure the  usefulness of the risk ratings. We found that SBA data could be useful for  developing alternate measures of lender performance in order to  independently validate the lender risk rating system’s results. Without  performing its own assessment, the agency may not be able to identify  issues with the model’s ability to reasonably predict lender behavior or to  notify the contractor of any suspected deterioration. As a result, SBA may  miss opportunities to identify risky lenders and mitigate the risks they  pose to SBA’s portfolio.
If SBA improves its validation of the lender risk ratings, the agency could  rely more on them to determine which lenders need an on-site review.  Currently, unlike FDIC and the Federal Reserve, SBA does not take full  advantage of its risk ratings to set the schedules for on-site reviews. The  agency targets lenders for on-site reviews based on size rather than risk  level. As a result, we found that SBA conducted on-site reviews of only 3  percent of the lenders that the lender risk rating system identified as high  risk in 2008. Of these, 215 had an outstanding SBA portfolio of at least $4  million. Relying more on the risk ratings to target lenders for review would  enable the agency to focus on the lenders that pose the most risk to the  agency.
Although SBA has made improvements to its off-site monitoring of  lenders, the agency will not be able to substantially improve its lender  oversight efforts unless it improves its on-site review process. Federal  financial regulators rely on results from their off-site monitoring to tailor  the scope of their on-site reviews. SBA does not rely on its lender risk  ratings to inform file reviews conducted during on-site reviews but rather  consistently uses a checklist to examine lenders. In addition, federal  financial regulators routinely assess the quality of lenders’ credit decisions  as part of their on-site examination process. SBA fails to include this  component but instead focuses more on compliance with SBA policies and  procedures. For example, rather than assessing the quality of lender  underwriting, contractor staff focus on whether lenders ensured that the  borrowers met eligibility requirements, including whether borrowers had  any other outstanding SBA loans that are not current. By including an  assessment of lenders’ credit decisions as a routine part of their on-site  review process, SBA would be able to determine the quality of the lenders’  underwriting standards and practices and make any necessary changes to  its lender risk rating system to ensure that the tool is relevant and includes  emerging lending trends.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Administrator of the Small Business  Administration take the following four actions:  To ensure that the lender risk rating system effectively evaluates risk,  when validating the system and undertaking any redevelopment efforts,  the Administrator should    ensure that SBA’s contractor follows sound model validation practices.
These practices should include (1) testing of the lender risk rating system  data, processes, and results, including a routine reassessment of which  factors are the most predictive of lender performance; (2) utilizing an  independent party to conduct validations; and (3) maintaining complete  documentation of the validation process and results.   use SBA’s own data to assess how well the lender risk ratings predict  individual lender performance.
To make better use of the lender risk rating system in SBA’s oversight of  lenders, the Administrator should    develop a strategy for targeting lenders for on-site reviews that relies more  on SBA’s lender risk ratings.   consider revising SBA policies and procedures for conducting on-site  reviews. These revised policies and procedures could require staff to (1)  use lender risk ratings to tailor the scope of file reviews performed during  on-site reviews to areas that pose the greatest risk, (2) incorporate an  assessment of lenders’ credit decisions in file reviews, and (3) use the  results of expanded file reviews to identify information, such as emerging  lending trends, that could be incorporated into its lender risk rating  system.
section_title: Agency Comments and Our Evaluation
paragraphs: We requested SBA’s comments on a draft of this report, and the Associate  Administrator of the Office of Capital Access provided written comments  that are presented in appendix II. SBA generally agreed with our  recommendations and outlined some steps that it plans to take to address  them. The agency also provided one technical comment, which we  incorporated.
SBA provided detailed comments on each of our four recommendations.  In response to our recommendation to ensure that SBA’s contractor  follows sound model validation techniques, SBA noted that the agency is  currently undertaking a redevelopment of its lender risk rating system and  plans to ensure that best practices are incorporated into the  redevelopment validation process. According to the agency, the  redevelopment contract will give SBA greater flexibility to reassess the  predictiveness of the factors used in the model and to refine the model if  necessary. SBA stated that it is also developing an independent review  process as well as increasing the level of documentation of the validation  process.
Regarding our recommendation to use its own data to assess how well the  lender risk ratings predict individual lender performance, SBA stated that  although it remains confident that the lender risk ratings provide accurate  predictions, the agency will determine whether alternative measures  would be useful to supplement the lender risk ratings.
In response to our recommendation to develop a strategy for targeting  lenders for on-site review that relies more on the lender risk ratings, SBA  stated that it agreed with our finding that between 2005 and 2008 on-site  reviews had been limited and primarily focused on the largest lenders, but  pointed out that the agency had significantly increased the number of  lenders reviewed since it began charging for on-site reviews late in fiscal  year 2007. The agency also noted that the largest lenders account for  approximately 85 percent of SBA’s entire guaranteed portfolio, while the  high-risk lenders that were not reviewed in 2008 represent 2 percent of  SBA’s total 7(a) and 504 portfolios. In our report, we recognize that while  not all of the small lenders with high risk ratings warrant more targeted  monitoring, some do. Of the 1,545 high-risk lenders that we found were not  reviewed in 2008, 215 lenders had significant portfolios—that is, portfolios  of at least $4 million. While SBA indicated that it plans to continue to focus  on-site reviews on the largest lenders that account for the majority of the  guaranteed portfolio, it stated that it will consider revising its internal  policies to make better use of the lender risk ratings to prioritize on-site  reviews.
Regarding our recommendation to consider revising policies and  procedures for conducting on-site reviews, SBA stated that the agency is  in the process of reprocuring its on-site review contract. According to the  agency, SBA included the ability to conduct on-site reviews that can be  better tailored to specific concerns about individual lender performance as  part of the reprocurement process. SBA also stated that the agency is in  the process of evaluating our recommendation to include an assessment of  lender credit decisions in the on-site review process and will investigate  ways to use the results of the on-site reviews to inform the lender risk  rating system.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to interested  congressional committees, the Administrator of the Small Business  Administration, and other interested parties. In addition, the report will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In this report, we examined (1) how the Small Business Administration’s  (SBA) risk rating system compares with the off-site monitoring tools used  by federal financial regulators and lenders and the system’s usefulness for  predicting lender performance and (2) how SBA uses the lender risk rating  system in its lender oversight activities.
To determine how SBA’s lender risk rating system compares with off-site  monitoring tools used by federal financial regulators and lenders, we  conducted interviews and reviewed documents to identify common  industry standards. We interviewed officials from three federal financial  regulators—the Office of the Comptroller of the Currency (OCC), the  Board of Governors of the Federal Reserve System (the Federal Reserve),  and the Federal Deposit Insurance Corporation (FDIC)—five of the largest  7(a) lenders, and the five largest 504 lenders. We identified the largest  lenders based on the size of their SBA-guaranteed portfolio in 2007, the  most recent data available when we began our review. The documents we  reviewed included relevant literature, procedural manuals and other  related federal guidance to banks on loan portfolio monitoring, and lender  procedural manuals. We then obtained and analyzed documents from SBA  on its lender risk rating system and conducted interviews with agency and  contractor officials responsible for maintaining the system to determine  how the system was developed and validated. We assessed SBA’s lender  risk rating system against common industry standards and our internal  control standards. In addition, we reviewed our previous work on SBA  and guidance on model validation from the Basel Committee on Banking  Supervision, which provides a forum for banking regulators from around  the world to regularly cooperate on banking supervisory matters and  develop common guidelines.
To assess the lender risk rating system’s usefulness for predicting lender  performance, we performed independent statistical tests to determine how  well it predicted individual lender performance. To perform these tests,  we first obtained the following data from SBA: administrative data on  loans approved in 2003 through the end of 2007 (including the date the  loan was approved, the size of the loan, and whether and when the loan  was purchased); the March 2007 and March 2008 lender performance  reports containing risk ratings; and the currency rate for each lender. We  assessed the reliability of these data by reviewing information about the  data and performing electronic data testing to detect errors in  completeness and reasonableness. We found that the data were  sufficiently reliable for the purposes of this report.
Using SBA’s data, we undertook a number of evaluative steps to test the  agency’s model. First, we assessed how well the lender risk ratings  predicted lender default rates (our measure of actual lender performance).  In order to test how well the lender risk ratings predicted lender  performance, we estimated how well a lender performed during either the  year or 6 months after the score was developed (depending on the amount  of data available) using a logit regression. A logit regression is a statistical  technique that estimates how the odds of an outcome changes with an  attribute of the unit of analysis. In our case, we estimated how the odds of  a loan being purchased by SBA varied by the lender that made the loan.  Additionally, we controlled for the age of loans and how default rates for  all loans changed over the year or 6 months. To control for the age and  changing default rates over time, we employed a methodology called a  discrete time hazard model. We restructured the data so that there was a  separate observation for every quarter that a loan was at risk of being  purchased. Then we estimated a logit regression and predicted whether  the loan was purchased that quarter. In that regression, we included a  dummy variable for each lender, a dummy variable for each quarter, and a  dummy variable for each quarter since that loan was approved, to capture  the age of the loan. The following describes the regression equation we  used:  P(loan i was purchased at time t) = logit(α ,α  ,α )  where the parameters of interest, α, can be transformed to express the  relative odds of a loan being purchased or defaulting for each lender, with  one lender excluded as a reference. We used the coefficients α  as the  measures of lender risk. In addition, the coefficients α control for the  differential rate of default by time period, and the coefficients α control  for the age of the loans.
Once we estimated the performance for each lender, we matched it with  each lender’s record in the lender performance report, which contained  the risk rating. For 7(a) loans, we matched our performance measures  with the lender risk rating using a “crosswalk” file obtained from SBA.  Because the data we obtained from SBA only included loans that were  approved from January 2003 to December 2007 and a lender had to have  made at least 100 loans during that time period to make our analysis  meaningful, we were only able to obtain measures for 308 of the 4,673 7(a)  lenders in the March 2008 lender performance report. We were more likely  to obtain measures for larger lenders. For example, we were able to  obtain measures for 56 of the 60 lenders with more than $100 million in  outstanding SBA-guaranteed loan balances. In all, the 308 lenders, plus the  lender excluded as the reference case, represented approximately 79  percent of the outstanding balance and 85 percent of the outstanding loans  reported in the March 2008 lender performance report. For 504 lenders, we  were able to obtain measures for 86 of the 270 lenders. We were able to  obtain 47 of the 48 lenders in the largest peer group—that is, those lenders  with more than $100 million in outstanding SBA-guaranteed loan balances.
To determine how SBA uses the lender risk rating system in its lender  oversight activities, we reviewed agency documents and conducted  interviews to document SBA’s practices for assessing and monitoring the  risk of lenders and loan portfolios. We then compared these practices  against (1) the industry standards we identified through our interviews  with federal financial regulators and lenders and reviews of their  documents and (2) our internal control standards. We also obtained and  analyzed SBA data on risk ratings and on-site examinations from 2005  through 2008 to determine the role that the lender risk ratings played in  identifying lenders for an on-site review.
To analyze the data on risk ratings and on-site examinations, we had to  make a number of assumptions because the risk ratings were reported by  quarter and we planned on reporting them by year. First, we assigned  lender risk ratings in two different ways. For those lenders that were  reviewed, we assigned them the risk rating that they received during the  quarter that immediately preceded the on-site review. For those lenders  that were not reviewed, we assigned them the lowest risk rating that they  received during that given year. Second, we assigned lenders to peer  groups in two different ways. For those lenders that were reviewed, we  assigned them the peer group that they were in during the quarter that  immediately preceded their on-site review. For those lenders that were not  reviewed, we assigned them the peer group they were in when they  received their lowest risk rating. Because lenders are assigned a risk rating  four times in a given year, there were some instances when they received  the same low-risk rating multiple times in a given year but were in  different peer groups when these ratings were assigned. In these instances,  we relied on the most recent, lowest-risk rating score. For example, a  lender could have received a lender risk rating of 4 in the second, third,  and fourth quarter of a given year. However, the lender was in the highest  peer group during the second and third quarters and in the second highest  peer group in the fourth quarter. We would rely on the most recent  quarter’s information and assign this lender a risk rating of 4 and the  second highest peer group. Third, we determined the on-site review date in  two ways. For on-site reviews completed in 2005 and 2006, we relied on  the date that the final report for the on-site review was issued to determine  when an on-site review was completed. For on-site reviews completed in  2007 and 2008, we were able to rely on an additional variable included in  the data that identified the date the on-site review was completed to  determine when the on-site review was completed.
We conducted this performance audit from August 2008 to November 2009  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Small Business Administration
section_title: Appendix III: Predictive Performance of the March 2007 and March 2008 Lender Risk Ratings
paragraphs: We performed two types of statistical tests to determine how well SBA’s  lender risk ratings predicted individual lender performance. For both  tests, we focused on how well the March 2007 lender risk ratings predicted  the performance of lenders for the following year and how well the March  2008 lender risk ratings predicted the performance of lenders for the  following 6 months. First, we compared raw scores from SBA’s lender risk  rating system to actual default rates for 7(a) and 504 lenders to determine  how well the lender risk ratings identified the best and worst performing  lenders. We divided lenders into two groups—those with lender default  rates in the top 50 percent of all lender default rates and those with default  rates that were in the bottom 50 percent of all lender default rates. We  found that SBA’s risk ratings were generally successful at distinguishing  the performance of about two-thirds of the 7(a) and 504 lenders in our  sample (see tables 2 and 3). For example, table 2 shows that 96 of the  approximately 300 lenders in our sample were in the top 50 percent based  on the March 2007 lender risk ratings and actual lender default rates, while  another 99 lenders were in the bottom 50 percent based on both rankings.  We also compared how well an alternate measure of lender  performance—the currency rate—divided lenders into these same two  performance groups and found that overall, it also correctly separated  about two-thirds of the lenders in our sample.
We used the same data to perform the second statistical test: determining  the correlation between the rankings based on lender default rates and (1)  the lender risk ratings and (2) the alternate measure—currency rate. We  found that for both 7(a) and 504 lenders, there was a positive correlation  between actual performance (lender default rates) and the lender risk  ratings and currency rate. For the largest 7(a) lenders (that is, those  lenders with SBA-guaranteed portfolios of at least $100 million), the lender  risk ratings were more correlated to the lender default rates than was the  currency rate. For 504 lenders, we found that both measures—the lender  risk rating and the currency rate—performed about the same (see table 4).
section_title: Appendix IV: Small Business Predictive Score
paragraphs: The Small Business Predictive Score (SBPS) predicts loan performance.  Specifically, it predicts the likelihood of severe delinquency (61 or more  days past terms) over the next 18 to 24 months, including bankruptcies  and charge-offs. It is an off-the-shelf product that was developed by Fair  Isaac using consumer and business credit bureau data. The model is able  to produce scores—ranging from 1 to 300, 1 being highest risk and 300  being lowest risk—using either a mix of consumer and business data, only  data from the consumer credit bureaus, or only business data from Dun &  Bradstreet. According to SBA officials, approximately 74 percent of its  7(a) loans and 83 percent of its 504 loans are scored using both consumer  and business data. Approximately 17 percent of its 7(a) loans and 8  percent of its 504 loans are scored using consumer data only, while 9  percent of its 7(a) loans and 504 loans are scored with Dun & Bradstreet  data only.
As we reported in 2004, Dun & Bradstreet collects these data from various  sources and processes them through a five-step quality assurance process.  First, Dun & Bradstreet collects data from more than 150 million  businesses globally and continuously updates its databases more than 1  million times daily based on real-time business transactions. Second, it  matches SBA records with its records and achieves at least a 95 percent  match of the data on 11 critical pieces of information used to identify the  borrower. Third, Dun & Bradstreet assigns a unique identifier to each  company. Fourth, Dun & Bradstreet identifies the corporate linkage of a  business’s branches or subsidiaries with their parent entity to help SBA  understand their complete corporate exposure between borrowers and  their parent entities. Finally, Dun & Bradstreet generates predictive  indicators of a business’s potential inability to repay a loan. Dun &  Bradstreet officials refer to this process as the DUNSRight process.
We performed independent tests to determine how well the SBPS  predicted the performance of 7(a) loans. Specifically, we used a logit  regression to determine how well the SBPS at loan origination predicted  the default of loans with disbursement amounts above and below  $150,000. We examined loans that were approved between 2003 and 2007  and default rates over the period of January 2007 to September 2008.
We found that the origination SBPS was predictive for loans that were  both less than $150,000 and more than $150,000. However, the SBPS was  estimated to have a larger effect on the performance of loans that were  less than $150,000. Table 5 shows the coefficients from the logistic  regression we ran. The coefficient estimated for the sample of loans that  were less than $150,000 is more negative than that for loans that were  more than $150,000, indicating that an increase in the SBPS (which  represents a decrease in the predicted risk of the loan) lowers the rate of  default by a greater increment. Additionally, as shown in the last column,  the difference in the coefficients between the two groups is statistically  significant.
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Paige Smith (Assistant Director),  Triana Bash, Ben Bolitzer, Tania Calhoun, Emily Chalmers, Marc Molino,  Jill Naamane, Anh Nguyen, Carl Ramirez, and Stacy Spence made key  contributions to this report.