Performance Audit Report 

The Small Business Administration s Improper Payment Rate for 7(a) Guaranty Purchases Remains Significantly Underestimated 

For Official Use Only
U.S. Small Business Administration Office of Inspector General Washington, D.C. 20416 

R EPORT T RANSMITTAL 13-07 

D ATE : November 15, 2012 

T O : Jeanne A. Hulit, Associate Administrator, Office of Capital Access John A. Miller, Director, Office of Financial Program Operations 

S UBJECT : The Small Business Administration s Improper Payment Rate for 7(a) Guaranty Purchases Remains Significantly Underestimated 

This report presents the results of our audit of the Small Business Administration s (SBA) Improper Payment Rate for 7(a) Guaranty Purchases for Fiscal Year 2011. Our objective was to determine whether SBA accurately estimated its FY 2011 Improper Payment Rate for 7(a) Loan Program purchases. We conducted this audit in accordance with Generally Accepted Government Auditing Standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. 

We request that you provide your management decision for each recommendation on the attached SBA form 1824, Recommendation Action Sheet, by December 15, 2012. Your decision should identify the specific actions taken or planned for each recommendation and the target dates for completion. 

We appreciate the courtesies and cooperation of the SBA extended to the staff during this audit. Please direct any questions to me at (202) 205-6587 or Terry Settle, Director, Credit Programs Group at (703) 487-9940. 

*** 

/s/ John K. Needham Assistant Inspector General for Auditing
What OIG Audited The SBA is authorized under Section 7(a) of the Small Business Act to provide financial assistance to small businesses in the form of Government-guaranteed loans. When a loan goes into default and the lender requests guaranty payment, the SBA reviews loan documentation to evaluate the lender’s compliance with program rules and regulations. Each year, the SBA is required to estimate the amount of improper 7(a) guaranty purchases. We audited the improper payment review process that the Agency used to estimate its FY 2011 improper payment rate on purchased loans to determine the accuracy of the Agency s estimate. Specifically, we re-performed SBA s review of 30 guaranty purchases totaling approximately $8.9 million from the Agency s sample of 303 guaranty purchases totaling $129.5 million. All of these purchases were made between April 1, 2010 and March 31, 2011, the period that the SBA used to perform its improper payment review for FY 2011. SBA s 7(a) purchases totaled approximately $2.36 billion for FY 2011. 

Actions Taken Based on a previous OIG report issued in March 2012, the Agency has agreed to (1) reemphasize key loan review requirements, (2) provide credit analysis training to reviewers, (3) provide quality control feedback to reviewers, (4) enhance the identification of root causes of improper payments, and (5) work with the OIG to agree upon a consistent methodology for determining the estimated improper payment rate for the next reporting period. 

OIG Recommendations We made 12 recommendations to the SBA, which included: 

Create a more comprehensive improper payment detection checklist and monitor its effectiveness. Provide training to the three centers responsible for conducting improper payment reviews. Determine the appropriate amount of time needed for loan reviews to better identify erroneous payments. Confirm recovery for two loans. Seek recovery for four loans. 

What OIG Found The SBA did not detect all improper payments when conducting improper payment reviews to estimate its FY 2011 improper payment rate for 7(a) guaranty purchases. From a sample of 30 loan purchases totaling nearly $8.9 million, the SBA did not detect that 6 loans purchased for approximately $1.8 million were improper payments. 

This occurred because improper payment reviewers were unfamiliar with or misinterpreted Agency policies, or did not have adequate time to conduct reviews. As a result, the SBA reported an improper payment rate of 1.73 percent or $40.7 million in its FY 2011 Agency Financial Report, when the rate could have been as high as 20 percent, or about $472 million. In addition, the Congress and the public were misinformed of the level of improper payments made for 7(a) guaranty purchases. Furthermore, when the SBA did not identify improperly paid 7(a) guaranty purchases, it missed the opportunity to identify the related root causes and implement the necessary corrective actions. 

SBA Response The SBA agreed with eight of the twelve recommendations. The SBA did not agree with the OIGs estimation of the improper payment rate or that it had misinformed the Congress and the public. SBAs response stated that the 

OIG s results were incorrect because they were based on invalid and inconclusive statistical results. Additionally, the SBA stated that the OIG ignored the Agency s stewardship over taxpayer dollars. 

OIG Comments With the assistance of our statistician, we reexamined our statistical results and reaffirmed that our initial conclusions were statistically valid. The results of this review are provided in Appendix V. We determined that SBAs statistician made methodological and conceptual errors that led to a flawed SBA response. SBAs statistician (1) did not account for the complex sample design, and (2) assumed that the deficiencies identified by the OIG existed exclusively in the OIG s subsample of 30 loans, and nowhere else in SBA s full sample of 303 loans. It is extremely unlikely a one-in-1.72-million chance that the deficiencies would only exist in the subsample. As a result, we continue to support our conclusion that SBA significantly underestimated its improper payment rate. 

With regard to the Agency s stewardship over taxpayer dollars, this audit focused on only one aspect of SBAs responsibilities identifying improper payments. We agree that the purchase rate in comparison to the total outstanding 7(a) loan portfolio has recently declined. However, on a dollar basis, the amount of purchases has significantly increased since 2007, while the amount of denials and repairs decreased. Specifically, in 2007, the denial and repair rate was 1.3 percent of total purchases and in 2011, it was 0.5 percent. This decrease in the rate of denials and repairs supports our concern that SBA may not be identifying all improper payments.
Table of Contents 

Introduction ............................................................................................................................................ 5 

Objective, Scope and Methodology ......................................................................................................... 6 

Background ............................................................................................................................................. 7 

Nature of Limited or Omitted Information ............................................................................................... 9 

Review of Internal Controls ..................................................................................................................... 9 

Finding: The SBA Underestimated Its FY 2011 Improper Payment Rate for 7(a) Guaranty Purchases ... 9 

Deficient Repayment Ability and Equity Injection Determinations Made on Two Loans .................. 10 

Invalid Credit Model Used for Two Loans ....................................................................................... 11 

Business Plan Not Obtained from Borrower for One Loan .............................................................. 12 

IRS Verification Not Performed for One Loan ................................................................................. 12 

FY 2011 Improper Payment Rate Underestimated by more than $432 million. ............................... 13 

Conclusion ..................................................................................................................................... 14 

Other Matter ..................................................................................................................................... 14 

Recommendations ............................................................................................................................. 15 

Agency Comments and Office of Inspector General Response............................................................ 17 

Appendix I: Scope and Methodology..................................................................................................... 25 

Appendix II: Statistical Approach, Sampling Methodology and Projections ............................................ 27 

Appendix III: Sample of Purchased Loans .............................................................................................. 29 

Appendix IV: Management Comments.................................................................................................. 31 

Appendix V: OIG Statistician Response to SBA Comments ...................................................................... 40
5 

Introduction 

This report presents the results of our audit of the SBA Fiscal Year (FY) 2011 improper payment rate for 7(a) guaranty purchases of defaulted loans. The audit report focuses on improper payments that occurred during the purchase process and not the loan approval process. The objective of our audit was to determine whether the Agency accurately estimated and reported its annual improper payment rate to the Office of Management and Budget (OMB). 

In 2009, the SBA Office of Inspector General (OIG) issued a report on the SBAs FY 2008 Improper Payment Rate for 7(a) Guaranty Purchases. 1 In the report, the OIG made several recommendations to mitigate the risk of future improper payments and noncompliance with requirements prescribed by the OMB. One recommendation for the Agency to delegate final approval of guaranty purchase and improper payment disputes to the Office of Credit Risk Management remains open. 

The initial results of our current audit were used in a report recently issued by the OIG. In that report, the OIG assessed the Agency s compliance with provisions required by the Improper Payments Elimination and Recovery Act (IPERA) for several programs, including the 7(a) Guaranty Loan Program, and SBA procurement activities. 2 For the IPERA Report, the OIG performed limited testing and examined the SBAs overall efforts to prevent and reduce improper payments. The IPERA report contained a key finding that the SBA underestimated its improper payment rate for FY 2011 7(a) guaranty purchases by not properly weighing the OIG s results. This finding was based on preliminary results from our current audit. It was further reported in the IPERA report that if the SBA had used the OIG statistician s approach to properly weigh the five OIG-identified erroneous payments, which the Agency agreed were improper, the SBA should have reported an improper payment rate as high as 9.86 percent. The SBA reported a rate of 1.73 percent in its FY 2011 Agency Financial Report. The IPERA report made the following recommendations for 7(a) guaranty purchases: 

Require loan officers to thoroughly evaluate creditworthiness (including repayment ability) on early defaulted loans during both guaranty purchase and improper payment reviews; Determine and report an accurate and statistically valid estimate of improper 7(a) default purchases for FY 2012 in the next Agency Financial Report; Conduct a detailed and objective cost/benefit analysis for payment recapture audits of 7(a) purchases upon revising the improper payment rate projection for 7(a) purchases; and Revise the corrective action plan to identify all root causes of improper payments and appropriate actions for reduction upon revising the improper payment rate projection for 7(a) purchases. 

Although the IPERA report addressed payments made in several SBA programs and SBA procurement activities, the report was not intended to provide a detailed assessment of each function. In this report, we present a more detailed assessment of the SBAs efforts to estimate the rate of improper payments for its 7(a) guaranty purchases made in FY 2011. The Agency 

1 OIG Report 9-16, The Small Business Administration s Fiscal Year 2008 Improper Payment Rate for the 7(a) Guaranty Loan Program, July 10, 

2 2009. OIG Report 12-10, SBA Generally Meets IPERA Reporting Guidance but Immediate Attention is Needed to Prevent and Reduce Improper Payments, March 15, 2012.
6 

requested that we perform this audit concurrently with its improper payment reviews of 7(a) guaranty purchases. The Agency suggested that the reviews be completed simultaneously to ensure that the OIG s results and feedback could be considered before the SBA reported its FY 

2011 improper payment rate to the OMB. In this report, we provide details concerning the seven improper payments that we identified during our review. Of the seven, one improper payment was detected by both the SBA and the OIG. The Agency agreed that five of the other six OIG-identified improper payments were valid. The SBA did not agree the one remaining improper payment identified by the OIG was valid. For the reasons presented below, we continue to believe it was an improper payment and therefore, we have included it in our projection of the estimated improper payment rate. Additionally, in this report we explain the impact of the improper payments that the SBA did not identify and suggest improvements for the Agency s improper payment review process. 

The SBAs Office of the Chief Financial Officer, with support from the Office of Financial Program Operations, publishes the Agency s estimated rate of improper payments for 7(a) guaranty purchases. This estimate is based on an annual review of a sample of guaranty purchases. The FY 2011 improper payment reviews were performed by SBA review teams located at the National Guaranty Purchase Center, the Fresno Commercial Loan Servicing Center, and the Little Rock Commercial Loan Servicing Center. The Agency teams at the centers performed these reviews to determine if improper payments occurred when the Agency paid lenders to honor its guaranties. These teams reviewed a sample of 303 loans totaling $129.5 million, of the 7(a) loan guaranties purchased between April 1, 2010 and March 31, 2011, which numbered 28,237 and totaled approximately $2.36 billion. 

Objective, Scope and Methodology 

Our objective was to determine whether the SBA accurately estimated its FY 2011 improper payment rate for 7(a) guaranty purchases. To answer our objective, we obtained the Agency s universe of 28,237 purchases totaling approximately $2.36 billion made between April 1, 2010 and March 31, 2011. We statistically selected a sample of 30 guaranty purchases totaling nearly $8.9 million from the SBAs sample of 303 purchases totaling $129.5 million. Using a standard 

OIG review methodology that we prepared based on applicable standard operating procedures, we assessed whether the Agency s reviewers identified all improper payments made by the SBA during the guaranty purchase process. During the audit, we interviewed five reviewers and two supervisory reviewers from the Fresno, CA, and Little Rock, AR, Commercial Loan Servicing Centers, and the National Guaranty Purchase Center in Herndon, VA. These reviewers were responsible for evaluating the loans identified in our audit. Based on our review results with the assistance of a statistician, we developed an estimate of improper payments for the SBAs 7(a) guaranty purchases made in FY 2011. We met with the Agency on several occasions to reconcile improper payments identified by the OIG but not identified by the SBA. Our scope and methodology is explained in Appendix I. Our statistical approach and sampling methodology is explained in Appendix II. A listing of the sampled purchases is detailed in Appendix III.
7 

We conducted our fieldwork in Atlanta, GA, Washington, DC, and Herndon, VA between July 

2011 and June 2012, in accordance with Generally Accepted Government Auditing Standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. Staff turnover and limited resources created delays in completing fieldwork and issuing this audit report. 

Background 

The SBA is authorized under Section 7(a) of the Small Business Act to provide financial assistance to small businesses in the form of Government-guaranteed loans. The SBA’s 7(a) loans are made by participating lenders under an agreement to originate, service, and liquidate loans in accordance with SBA’s rules and regulations and prudent lending standards. The SBA is released from liability on its guaranty, in whole or in part, if the lender fails to comply materially with any SBA loan program requirement or does not make, close, service or liquidate the loan in a prudent manner. 

When a loan goes into default and the lender requests guaranty payment, the SBA reviews loan documentation to evaluate the lender’s compliance with program rules and regulations. The SBA reviews the lender’s actions on the loan to determine if the lender made, closed, serviced, and liquidated the loan in a commercially reasonable manner that was consistent with SBA requirements and prudent lending practices. Based on this review, the SBA determines whether it is appropriate to accept liability for payment of the guaranty (which the SBA refers to as a guaranty purchase"). The purchase review process is the SBA’s primary control for ensuring lender compliance and preventing improper payments. In the event of noncompliance, the SBA 

may be released from its liability on a loan guaranty, in full or in part. Since FY 2008, the SBA has paid more than $8 billion to 7(a) lenders to purchase its guaranteed share of defaulted loans. In FY 2011, the SBA paid approximately $2.36 billion to honor its guaranties. 3 

Table 1 SBA 7(a) Loan Guaranty Purchases 

Source 1 Generated by SBAs Office of Financial Program Operations 

3 For purposes of this audit, we used the FY 2011 universe of purchases totaling approximately $2.36 billion. As described in Appendix I, this universe total was provided by the Agency and was tested for completeness and accuracy during the audit. The amounts presented in Table 1 

were provided by the Office of Financial Program Operations and were not tested for accuracy and completeness. While there are slight 

4 differences, the figures presented in the table are for background information only and are not material to our audit results. Annual period is April 1 through March 31, the period SBA uses when estimating its improper payment reviews for 7(a) guaranty purchases. 

Year-End March 31 4 SBA 7(a) Loan Guaranty Purchases 

2007 $629,806,162 

2008 $874,699,335 

2009 $1,832,138,702 

2010 $2,388,255,032 

2011 $2,366,631,473 Total $8,091,530,707
8 

According to SBA procedures that were in effect from April 20, 2005 to November 14, 2010, 5 an early default is a default or business failure that occurs prior to final disbursement of the loan, or within 18 months from the date of final disbursement. A default includes an unremedied failure to make one or more payments required by the terms of the note, as well as events that would place a loan in liquidation status. An early problem loan is indicated when either prior to final disbursement or within the first 18 months after final disbursement, a borrower has a pattern of late payments. Monthly payments funded through the sale of collateral, or a lenders deferral of two or more consecutive scheduled payments are also indicators of an early problem loan. Revisions to SBA procedures, that became effective on November 15, 2010, 6 define an early default as a default that occurs within 18 months of the initial disbursement of the proceeds from an SBA loan. If the final loan disbursement occurs more than six months after the initial disbursement, the early default time frame is extended to 18 months after the final disbursement. 

For early-defaulted and early problem loans, the SBA must review the lender’s purchase package with the highest degree of scrutiny." Additionally, the recommending official 7 must determine whether the lender’s failure to make or close the loan in accordance with SBA requirements or prudent lending practices allowed or contributed to the early default. If the SBA purchase review indicates that the lender’s actions or omissions resulted in a material loss to the SBA, repair or denial of the guaranty is appropriate. If there was an early default, the lenders failure to verify and properly document a required equity injection by the borrower raises a rebuttable presumption that the default was caused by the lack of the injection and a full denial of liability may be justified. 8 

Congress enacted the Improper Payment Information Act (IPIA) on November 26, 2002. This legislation, in conjunction with implementing guidance from Appendix C of OMB Circular A-123 (Circular) requires executive branch agency heads to review their programs and activities annually and identify those that may be susceptible to significant improper payments. The guidance also requires these agency heads to estimate amounts improperly paid, and report on the amounts of improper payments and actions to reduce them. Furthermore, the Circular requires Inspectors General to review agencies performance to determine compliance with IPIA. 

For FY 2008, the SBA reported an improper payment rate of 0.53 percent for 7(a) guaranty purchases. In its audit of the FY 2008 improper payment estimate, the OIG determined that the Agency s improper payment rate should have been 27 percent for that year. 9 During the audit follow-up process, the Agency concurred with many of the OIG s improper payment findings. On September 21, 2011, the SBA responded to a congressional inquiry concerning the differences between the SBA and the OIG reported improper payment rates for FY 2008. In its response, the SBA stated, if the IG’s judgmental approach is applied to its 30-loan sample, the 7(a) guaranty purchase improper payment rate in that sample was 21 percent.

5 SBA Standard Operating Procedures (SOP) 50 51 2(b). 6 SBA SOP 50 51 3, page 16 7 

8 The term recommending official as used in SOP 50 51 3, refers to the loan specialist performing the purchase review. SBA SOP 50 51 3, page 16 and page 162 9 OIG Report 9-16, The Small Business Administration s Fiscal Year 2008 Improper Payment Rate for the 7(a) Guaranty Loan Program, July 10, 2009.
9 

The SBA reported improper payment rates of 3.81 percent for FY 2009 and 1.87 percent for FY 2010. Most recently, in the FY 2011 Agency Financial Report, the SBA estimated its improper payment rate for 7(a) guaranty purchases to be 1.73 percent, or $40.7 million, of the FY 2011 

program outlays for purchases of approximately $2.36 billion. 

Nature of Limited or Omitted Information 

We identified two instances of suspicious activity that we referred to our Investigations Division for further review, the details of which have not been included in this report. 

Review of Internal Controls 

The SBAs Standard Operating Procedure (SOP) for internal control systems 10 provides guidance on the implementation and maintenance of effective systems of internal control as required by OMB guidance. According to OMB Circular A-123, effective systems of internal control improve the accountability and effectiveness of Federal programs and operations by establishing, assessing, correcting, and reporting on internal controls. 

We identified internal control weaknesses in the Agency s improper payment review process for 7(a) guaranty purchases. Specifically, the SBA did not always detect when improper payments were made during the guaranty purchase process. As a result, the Agency significantly underestimated the rate of improper payments reported to the OMB. 

Implementing the recommendations in this report will address the identified internal control weaknesses, and improve the improper payment review process. We will provide a copy of the final report to the senior official responsible for internal controls in the Office of the Chief Financial Officer. 

Finding: The SBA Underestimated Its FY 2011 Improper Payment Rate for 7(a) Guaranty Purchases 

The SBA did not detect all improper payments when conducting improper payment reviews to estimate its FY 2011 improper payment rate for 7(a) guaranty purchases. From a sample of 30 loans, comprised of 29 unique transactions that totaled approximately $8.9 million, 11 the SBA did not detect that 6 loans purchased for approximately $1.8 million were improper payments. This occurred because improper payment reviewers responsible for reviewing the six purchases were unfamiliar with or misinterpreted Agency policies, or did not have adequate time to conduct their reviews. As a result, the SBA reported an improper payment rate of 1.73 percent or $40.7 million in its FY 2011 Agency Financial Report, when the rate could have been as high as 

10 SOP 00 02, Internal Control Systems. 11 The sample of 30 loans was comprised of 29 unique transactions because the statistician used a sample design that allowed loans to be sampled more than once.
10 

20 percent, or about $472 million. See Appendix II for our statistical approach, sampling methodology, and projections. 

According to the Circular, on an annual basis, for programs and activities that are susceptible to significant improper payments, agencies are required to obtain a statistically valid estimate of the amount of improper payments. Further, agencies must ensure that their sampling methodologies are approved by OMB prior to conducting their sampling measurement, after consulting with a statistician to ensure the validity of their sample design, sample size, and measurement methodology. In addition, agencies should also incorporate refinements to their improper payment rate methodologies based on recommendations from agency staff or auditors (such as their agency Inspector General, the Government Accountability Office, or private auditors). 

According to the Circular, an improper payment for a guaranteed loan program may include disbursements to intermediaries; third parties for defaults, delinquencies, interest and other subsidies; or other payments based on incomplete, inaccurate, or fraudulent information. They may also include duplicate disbursements, or any disbursements that do not comply with law, program regulations, or agency policies. 

Furthermore, SBA policy specifies that lenders must originate, close, service, and liquidate 7(a) loans in accordance with the SBAs rules and regulations including SBA standard operating procedures. However, for six loans in a sample of 30, SBAs improper payment reviewers did not detect that lenders failed to meet Agency requirements as detailed in the following examples. 

Deficient Repayment Ability and Equity Injection Determinations Made on Two Loans 

For one loan, a reviewer at the National Guaranty Purchase Center did not detect that the Agency erroneously paid $1,016,116 to a lender that did not comply with Agency policy when approving the $1,500,000 Preferred Lender Program (PLP) 12 loan. Specifically, the lender: 

Determined repayment ability was sufficient even though the borrower had unsupported and unrealistic revenue projections and no historical debt service coverage. Agency policy states that borrower projections must be based on reasonable assumptions and no loan can be guaranteed by the SBA unless there is a reasonable assurance of repayment in a timely manner. 13 

Determined required equity injection was sufficient although there was no evidence that the injection was made. The lender claimed that equity injection consisted of inventory purchases. However, the purchases were not made with injected funds. According to Agency policy, lenders must verify injections prior to disbursing loan proceeds and maintain evidence of such verification in their loan files. The policy further states that lenders are expected to use reasonable and prudent efforts to verify 

12 Preferred Lender Program loans are made by lenders using delegated authority and undergo very limited review by the SBA prior to loan disbursement. 13 SOP 50 10 5, Chapter 4, page 168
11 

that equity is injected and used as intended, and failure to do so may warrant a repair or partial/full denial. 14 

The loan was an early default 15 and the Agency should have denied its guaranty based on the issues mentioned above according to the SBAs policy. The reviewer stated that he needed additional time to perform a more rigorous review similar to the review methodology used by the OIG. 

For another PLP loan, a supervisory reviewer did not detect that the SBA improperly paid $714,444 to honor its guaranty on an early problem loan. This payment should not have been made because the sources of equity injection were not fully substantiated as required by Agency policy. In particular, the lender provided the borrower a $951,000 SBA 7(a) loan to finance the acquisition of an existing car wash. The transaction required an equity injection of $237,900. Of that amount, $134,659 could not be sourced. 

In addition to multiple deferments requested by the borrower during the first 18 months, the borrower only made five full payments on the loan and those were made with loan proceeds. This early problem made it critical for the Agency to ensure that all of the required equity injection was documented and from legitimate sources. According to SBA officials, reviewers used internal Quality Assurance guidance when reviewing for this issue. Further, these officials stated that this guidance was designed to supplement Agency policy in detecting improper payments during the review process. However, this internal guidance was not consistent with the SOP and resulted in the Agency not identifying the lenders equity injection deficiency for this loan. The SOP required equity injection greater than one-third of the loan amount or $200,000, whichever is less, to be sourced. Consequently, the lender was required to verify and document: (1) that the cash injection existed, and (2) the source of the cash used for the equity injection. Such verification and documentation allows for a reasonable presumption that the funds will be used for business purposes. 16 The Quality Assurance guidance, however, states, 

If the lender failed to verify the source when required to do so but satisfactorily verified the existence of the cash equity injection (using our criterion of verification of 95% or more the borrower/lender funding package) and there is no indication that the cash was removed from the business prior to default for inappropriate reasons, then the purchase may proceed and no other action is necessary. 

We believe that the equity injection should have been fully sourced in accordance with the SOP. The supervisory reviewer stated that for some loans, the requirements pertaining to the loan 

may be overlooked because the type of loan is rarely encountered by staff members. The supervisory reviewer also stated that this was a classic example of a loan with conditions that were not encountered very often. 

Invalid Credit Model Used for Two Loans 

For two of the six loan purchases, reviewers in Fresno and Little Rock did not detect that the Agency improperly paid $17,744 to honor its guaranties. The lender disbursed $10,000 for each 

14 SOP 50 10 5(A) 15 SOP 50 51 3, page 16 and page 162 16 SBA SOP 50 51 2(B) page 14-15
12 

of these two Community Express 17 working capital loans. However, the lenders credit scoring matrix had not been statistically validated for predictability as required by Agency policy. 18 

This issue was reported by the OIG in a prior audit. 19 The reviewers stated that they were not familiar with the OIG report that pertains to the lender validating its scoring matrix. A supervisory reviewer at the center echoed the same sentiment. Specifically, the supervisory reviewer commented that it was difficult to become familiar with the policies, procedures, and SOPs due to the variety of loan types serviced by Little Rock. The reviewer from Little Rock stated that methods for distributing updated policy information did not ensure that staff was aware of or familiar with policy changes and updates. 

Business Plan Not Obtained from Borrower for One Loan 

For another loan purchase, the reviewer in Little Rock did not detect that the Agency made an improper payment of $3,446. Specifically, the lender made a $15,000 Community Express loan to a startup business for working capital. The lender did not provide a copy of the applicant s business plan or document the rationale in the absence of a formal business plan to the SBA, as required by Agency policy. 20 The business plan is an essential document provided by the borrower to the lender to demonstrate how the loan proceeds will be used and how the business will operate. The plan also shows how the business will generate sufficient revenue to ensure adequate loan repayment. The reviewer stated that he was unfamiliar with the technical assistance requirement for a business plan for Community Express loans. 

IRS Verification Not Performed for One Loan 

For another loan purchase, the reviewer in Little Rock did not take exception to the SBAs payment of $10,626 to a lender who did not report the loan as an early default. For this loan, the lender also did not provide the required credit memorandum and IRS transcripts, as required by Agency policy. 21 The lender made a $30,000 SBAExpress 22 loan to the borrower for working capital. Only after the OIG detected the early default status and informed the SBA of the missing documents, did the Agency request the lender to provide the missing information. The Agency obtained the credit memorandum from the lender, but not the IRS transcripts to demonstrate the verification of financial information as required. 23 The reviewer stated that the high volume of loans and the limited time allotted for improper payment reviews was the cause for not detecting the deficiencies. 

17 Under Community Express borrowers receive management and technical assistance arranged and paid for by the lender. The SBAs standard 

18 75 to 85 percent loan guaranty is applied, which contrasts with the 50 percent guaranty for SBAExpress loans. SOP 50 10 5(A), Subpart B, Chapter 4, page 175 19 

20 SBA Office of Inspector General Report 10-19 

21 SBA Procedural Notice 5000-1013 

22 SBA Policy Notice 5000-725 

23 SBAExpress loans are made using streamlined and expedited loan procedures and have a 50 percent SBA loan guaranty. SBA SOP 50 10 4, Chapter 6, page 243
13 

FY 2011 Improper Payment Rate Underestimated by more than $432 million. 

The Agency s improper payment rate of 1.73 percent or $40.7 million was underestimated in its FY 2011 Agency Financial Report. The OIG determined that the rate could have been as high as 

20 percent or about $472 million of the nearly $2.36 billion in purchases. 

When calculating its FY 2011 improper payment rate, the SBA did not properly consider the 

OIG s findings, in that they should have used a methodology that properly weighed the findings such as the two-tier approach that the OIG recommended. 24 Initially, the SBA estimated a rate of 0.88 percent, prior to receiving the OIG s audit findings. After obtaining the OIG s preliminary findings, the SBA agreed that five of the six improper payments identified by the OIG were valid and included them in a revised estimate. The SBA, however, did not use the two-tiered statistical methodology recommended by the OIG when revising its improper payment rate; and therefore, the SBA understated its rate of 1.73 percent. Given the internal control weaknesses identified, it is reasonable to expect that there were additional improper payments in the 

tier remaining approach purchases recommended that the by OIG the did OIGnot s examine. statistician, Had based the on 

SBA the revised five OIG its rate improper using payment the twofindings that they agreed with, the Agency should have reported a rate as high as 9.86 percent. Had the Agency revised the rate based on all six of the OIG findings, its improper payment rate could have been as high as 20 percent. See Appendix II for the OIG s statistical approach, sampling methodology, and projections. 

Because the Agency did not use the proper methodology, the Congress and the public were misinformed of the level of improper payments made for 7(a) guaranty purchases and of the potential for financial loss. Furthermore, when the SBA did not identify improperly paid 7(a) guaranty purchases, it missed the opportunity to identify the related root causes, and impaired its ability to implement the necessary corrective actions to prevent and reduce improper payments. Initiatives aimed at aggressively reducing improper payments and the elimination of wasteful spending is a central policy focus of the Government, as reflected in congressional 

passage of the Improper Payments Elimination and Recovery Act of 2010 (P.L. 111-204). Ensuring that erroneous disbursements are detected during improper payment reviews for 7(a) guaranty purchases is consistent with this policy. 

Although the Agency did not use the statistical approach that the OIG proposed in FY 2011, it has taken steps to improve the improper payment review process. The SBA began drafting a revised methodology for sampling 7(a) guaranty purchases to ensure the Agency is better able to determine the precision of the estimated improper payment rate. The revised methodology includes a more complete checklist to detect 7(a) loan deficiencies during improper payment and guaranty purchase reviews. 

In consideration of the IPERA report recommendations, the SBA stated that it will: re-emphasize the requirement that loan officers thoroughly review repayment ability and creditworthiness on early default loans, provide credit analysis training, provide quality control feedback to reviewers, 

24 See explanation of two-tier approach in Appendix II, Statistical Approach, Sampling Methodology and Projections.
14 

enhance the corrective action plan so that it addresses the root causes and amount of improper payments, and work with the OIG to agree upon a consistent methodology for determining the estimated improper payment rate for the next reporting period. 

The SBA has recovered $14,082 for two of the six loans identified as improper payments by the 

OIG during the FY 2011 improper payment audit. The OIG is recommending recovery for the remaining four loans totaling $1,748,304, which have yet to be resolved. The SBA has not initiated recovery for three loan guaranties totaling $732,188. The remaining loan in the amount of $1,016,116 is still under review by the SBA. 

Conclusion 

The Agency s improper payment rate of 1.73 percent or $40.7 million was understated in its FY 2011 Agency Financial Report. The OIG determined that the rate could have been as high as 

20 percent or about $472 million of the nearly $2.36 billion in purchases. The SBAs improper payment review process can be improved by using a more robust improper payment detection checklist, providing training to the reviewers, and allowing for sufficient review time. The improper payment detection checklist should allow Agency reviewers to more comprehensively assess and identify deficiencies. 

Other Matter 

During the audit, we identified four additional improper payments totaling $1,456,588 from our sample of 30 loans. In each of these loans purchases, we determined that the Agency would have been unable to discern whether the payment was proper since the loan files contained insufficient documentation. 

Guidance from OMB states that in cases such as these, the payments must be considered improper payments. 25 However, in limited cases, this same guidance allows for the exclusion of improper payments that have been subsequently corrected. 

After completion of its improper payment reviews, the SBA obtained additional documentation from the lenders to resolve the deficiencies related to the four improper payments identified by the OIG. We are reporting the deficiencies to further illustrate the need for improvements to the Agency s improper payment review process, which we are recommending in this report. 

The SBA paid one lender $643,143 to purchase an early problem loan. The lender erroneously concluded that the borrower had repayment ability since the lender (1) computed cash flow with figures that did not agree with the borrower s tax transcripts and (2) presented totals that contained errors to support debt service coverage calculations. After the OIG discussed this with the Agency, the SBA requested additional information from the lender and re-calculated the debt service coverage. The Agency s reiteration showed that despite the fact that the lenders analysis appeared flawed and contained some errors, there was no harm to the Agency. 

25 OMB Circular 1-123, Appendix C, Memorandum M-11-16. Agencies that use a different method for reporting improper payments that result from documentation issues must present their proposal to OMB for review and approval.


17 

Agency Comments and Office of Inspector General Response 

Agency General Comments and Office of Inspector General Response 

On August 17, 2012, we provided a draft of this report to SBA management for comment. On September 21, 2012, SBA management submitted a formal response, which is included in its entirety in Appendix IV. A summary of the managements comments and our response follows. 

The SBA agreed with 8 of the 12 recommendations in the report and stated it has taken steps to address 

many of the recommendations. The Agency partially agreed with one recommendation and disagreed with the remaining three recommendations. Specifically, SBA management stated that it has (1) developed and finalized comprehensive checklists for conducting improper payment reviews, (2) developed a process for monitoring the effectiveness of the improper payment review checklist, (3) provided and will continue to provide adequate training to the National Guaranty Purchase Center (NGPC), the Little Rock Servicing Center (LRSC) and the Fresno Servicing Center (FSC), and (4) received recoveries for two loans and will continue to seek resolution or recapture funds, if deemed necessary, for one loan. Finally, SBA management stated it will analyze the time required to better identify erroneous payments when conducting improper payment reviews. 

SBA management did not concur with three of our recommendations. The SBA stated that it will not seek recovery for the two loans made by Superior Financial Group based on an opinion from SBAs Office of General Counsel (OGC). For the third recommendation, the SBA disagreed with the OIG s decision to include Splash and Dash as an improper payment. As a result, it stated that it will not seek 

recovery from the lender based on the Office of Credit Risk Managements review and concurrence with the Office of Financial Program Operations. 

Additionally, SBAs response to the draft report stated that the OIG relied on invalid and inconclusive statistical results when presenting their findings and ignored the Agency s stewardship over taxpayer dollars. SBA management also questioned the OIG s interpretation of a letter written to Congress regarding the difference between the FY 2008 improper payment rate reported by the SBA and the OIG. As a result, the OIG revised its final report to address the Agency s concern. A summary of the Agency s general comments and OIG s response is presented in Table 2 below.
18 

Table 2 Agency Comments and OIG Response 

Agency Comments OIG Response The Office of Inspector General (OIG) states that the Agency misinformed Congress and the public about its level of improper payments made for 7(a) guaranty purchases and the potential for financial loss by significantly underestimating its FY 2011 improper payment rate. This assertion is incorrect; it is based on invalid and inconclusive statistical results. 

SBA contends that the OIGs methodology is flawed. Specifically, utilizing the OIGs methodology, the resulting Standard Deviation for improper payments is 

$16 Billion. That amount is 6.6 times greater than SBAs total actual purchases proper and improper in FY 

2011 of $2.4 Billion. 

With the assistance of our statistician we reexamined our statistical results and reaffirmed that our conclusion was statistically valid. The results of this review are provided in Appendix V. 

SBAs statistician made methodological and conceptual errors in the calculations which were used by the SBA to respond to our audit report. This caused the SBA to reach a flawed conclusion about our reported results. Specifically, SBAs statistician did not account for the complex sample design in their projection of the double sample. Further, the statistician used an implausible assumption that the six OIG-identified deficiencies existed exclusively in the OIGs subsample of 30 loans and nowhere else in the Agency s sample of 303 transactions. million chanceIt is that extremely the deficiencies unlikely would a one-in-1.72- only exist in the subsample. As a result, we continue to support our conclusion that SBA significantly underestimated its improper payment rate and provided inaccurate information to the Congress and the public. The OIG ignores the Agency s stewardship over taxpayer dollars. 

The Agency is a faithful steward of taxpayer dollars. 

The current 7(a) 12-month purchase rate of 3.3 percent is 37 percent lower than FY 2011 and 55 percent lower than FY 2010. This speaks to the integrity of the SBAs data modeling and the Agency s commitment to risk management and account of taxpayer dollars. SBA believes its oversight of improper payments is equally well-managed and reported. 

The SBA has made significant and purposeful strides in improving its processes to reduce and prevent improper payments by implementing internal controls, including establishing a Quality Improvement Program in all of its centers and training its employees. 

With regard to the Agency s stewardship over taxpayer dollars, this audit focused on only one aspect of SBAs responsibilities identifying improper payments. As stated in the Agency s response, SBAs purchase rate in comparison to the total outstanding 7(a) loan portfolio has recently declined. However, we also noted that the dollar amount of purchases has significantly increased since 2007, while the amount of denials and repairs decreased. Specifically, in 2007, the denial and repair rate was 1.3 percent of total purchases and in 2011, it was .05 percent. This decrease in the rate of denials and repairs supports our concern that SBA may not be identifying all improper payments. 

We acknowledge the on-going efforts made by the Agency to improve the improper payment review process. We also acknowledge that the SBA has made significant and purposeful strides toward improving its processes to reduce and prevent improper payments by providing ongoing training to loan center staff and implementing internal controls, such as the Quality Improvement Program. 

The FY 2011 improper payment draft report stated that OFPO concurred with five of the six OIG-identified improper payments; however, after further We disagree with the Agency s decision to reverse its concurrence on the two Superior Financial Group loans. Although the OGC guidance may deter the Agency from

20 

Recommendation 1 

Create a more comprehensive improper payment detection checklist for reviewing 7(a) guaranty purchases to address the many requirements that reviewers must be familiar with when conducting improper payment reviews. The checklist should be consistent with the SOP and include details for assessing: eligibility, use of proceeds, debt refinancing, change of ownership, credit analysis, repayment ability, equity injections, standby debt, early default/ early problem, collateral, IRS tax verification, loan servicing, and liquidation. 

Management Comments 

SBA management concurred with this recommendation and has developed and implemented comprehensive checklists addressing the various requirements loan specialists must be familiar with when conducting improper payment reviews. 

OIG Response 

Managements comments were responsive to the recommendation. 

Recommendation 2 

Develop a procedure for monitoring the effectiveness of the checklist for detecting improper payments. 

Management Comments 

SBA management concurred with this recommendation and developed a process for monitoring the effectiveness of the checklist for detecting improper payments. 

OIG Response 

Managements comments were responsive to the recommendation. 

Recommendation 3 

Provide training to Little Rock improper payment reviewers on the Agency s policies concerning business plans for Community Express loans, using credit scoring, IRS Form 4506 and determining early default to ensure these reviewers are aware of and understand the 7(a) loan requirements that resulted in undetected improper payments. 

Management Comments 

SBA management concurred with this recommendation. Over the past few months, the LRSC has held several training sessions, including training on SBA Express loan purchases. The LRSC will continue to hold monthly training sessions with the loan specialists, and will include business plans for Community Express loans, using credit scoring, IRS Form 4506 and determining early default to ensure these reviewers are aware of and understand the 7(a) loan program requirements that resulted in undetected improper payments.
21 

OIG Response 

Managements comments were responsive to the recommendation. 

Recommendation 4 

Provide training to Fresno improper payment reviewers on the Agency s policy concerning the use of credit scoring to ensure these reviewers are aware of and understand the 7(a) loan requirements that resulted in undetected improper payment. 

Management Comments 

SBA management concurred with this recommendation and the FSC has held various training sessions over the past few months. The FSC will continue to hold monthly training sessions with the loan specialists and will include the use of credit scoring to ensure these reviewers are aware of and understand the 7(a) loan program requirements that resulted in undetected improper payments. 

OIG Response 

Managements comments were responsive to the recommendation. 

Recommendation 5 

Provide training to improper payment reviewers at the National Guaranty Purchase Center on the Agency s policies concerning assessing repayment ability, validating equity injection, and determining early default payments to ensure these reviewers are aware of and understand the 7(a) loan requirements that resulted in undetected improper payments. 

Management Comments 

SBA management concurred with this recommendation and recognizes the importance training plays in ensuring loan specialists possess the knowledge and skills necessary to perform improper payment reviews. The NGPC offers monthly training sessions to all loan specialists and provides both general and individualized feedback based on Quality Improvement Program reviews. The NGPC will continue to provide training to include assessing repayment ability, validating equity injections, and determining early default payments to ensure these reviewers are aware of and understand the 7(a) loan requirements that resulted in undetected improper payments. 

OIG Response 

Managements comments were responsive to the recommendation. 

Recommendation 6 

Determine the appropriate amount of time needed to better identify erroneous payments when conducting improper payment reviews.



25 

Appendix I: Scope and Methodology 

To answer our objective, we obtained the Agency s universe of 28,237 purchases totaling approximately $2.36 billion made between April 1, 2010 and March 31, 2011 and tested its completeness. We then assessed the Agency s methodology for selecting its sample of 303 guaranty purchases from the universe using OMB guidelines. 26 These guidelines provide specific instructions for estimating and reporting the rate of improper payments. We consulted a statistician to ensure the validity of the Agency s sample design, sample size, and measurement methodology. 

To determine whether the SBA identified all improper payments, we statistically selected a sample of 30 guaranty purchases from the SBAs sample of 303 purchases totaling $129.5 million. The sample was comprised of 29 unique transactions that totaled approximately $8.9 million. 27 A listing of the 29 purchases is detailed in Appendix III. Using a standard OIG review methodology that we prepared based on applicable SBA standard operating procedures, we assessed whether or not the Agency s reviewers identified all improper payments made by the SBA during the guaranty purchase process. Our sampling methodology is explained in Appendix II. 

To determine the SBAs process for identifying improper payments, we interviewed Agency personnel in Washington, D.C., Fresno, CA, and Little Rock, AR Commercial Loan Servicing Centers, and the National Guaranty Purchase Center in Herndon, VA. 

Based on our review results and the assistance of a statistician, we developed an estimate of improper payments for the SBAs 7(a) guaranty purchases made in FY 2011. During the course of the audit, we met with the Agency on several occasions to reconcile improper payments identified by the OIG but not identified by the SBA. 

Use of Computer Processed Data 

To test for completeness of the SBA universe of purchased loans, we compared the data provided by the Office of the Chief Information Officer (OCIO) to computer-processed data from the Agency s Mainframe/Partner Identification and Management System (PIMS) extract. We analyzed the universe of loans purchased between April 1, 2010 and March 31, 2011. 

To test for the accuracy of the purchased loan attribute in the data provided by the OCIO, we assessed each loan in the universe. The test ensured that each loan contained the purchase attributes of: purchase date, purchase principle gross amount, and loan type codes of 17 and 27, which indicates the loans were purchased and are being serviced by either the SBA or the lender. 

We tested the reliability of our sample of 7(a) guaranty purchases by comparing our sample against data in the SBAs Loan Accounting System (to source information found in the loan files of the sample loans to ensure the information matched). 

26 OMB Circular A-123, Appendix C 27 The sample of 30 loans was comprised of 29 unique transactions because the statistician used a sample design that allowed loans to be sampled more than once.
26 

Prior Coverage 

During the past 5 years, the Government Accountability Office (GAO) issued one report pertaining to the SBAs compliance with the Improper Payments Elimination and Recovery Act relative to 7(a) Loan Guaranty Purchases. In addition, the OIG has issued two reports. The public can access these reports on the Internet at http://www.gao.gov and http://www.sba.gov/office-of-inspector-general . 

GAO Reports and Testimony 

Report Number, GAO-08-438T, January 31, 2008. GAO (2008). Status of Agencies’ Efforts to Address Improper Payment and Recovery Auditing Requirements. Retrieved from: http://www.gao.gov/products/GAO-08-438T 

SBA OIG Reports and Memorandums 

Report Number, 9-16, July 10, 2009. SBA OIG (2009). The Small Business Administration s Fiscal Year 

2008 Improper Payment Rate for the 7(a) Guaranty Loan Program. Retrieved from: http://www.sba.gov/sites/default/files/oig_reptbydate_july9-16.pdf 

Report Number 12-10, March 15, 2012. SBA OIG (2012). SBA Generally Meets IPERA Reporting Guidance but Immediate Attention Is Needed to Prevent and Reduce Improper Payments. Retrieved from: http://www.sba.gov/office-of-inspector-general/874/129471
27 

Appendix II: Statistical Approach, Sampling Methodology and Projections 

Sample Selection The OCIO provided data on the universe of 7(a) loans purchased between April 1, 2010 and March 31, 2011. The universe consisted of 28,237 loans or approximately $2.36 billion in loans purchased during this review period. From this universe, the SBAs statistician generated a statistically random sample of 7(a) guaranty purchases that numbered 303 and totaled $129.5 million. The 303 purchased loans were then reviewed by the SBA for the detection of improper payments. 

The OIG consulted a statistician to produce a sample of 7(a) guaranty loan purchases from the 303 reviewed by the Agency. The OIG s sample numbered 30, but was comprised of 29 unique transactions that totaled approximately $8.9 million. 28 The methodology used by the OIG s statistician was a stratified random sample with selection probability proportional to size with the purchase amount as the unit of measure. The OIG s statistician tested the sampling methodology used by the SBA and the 

OIG to ensure it would produce a statistically valid estimate of the rate of improper payments. 

Projection of Sample Results (Two-Tier Approach) Upon reviewing the sampling plan produced by the Agency, the OIG s statistician concluded that the sampling methodology was consistent with industry practices. The serious shortcoming of the design was that 5 of the 11 strata had only 1 item included in the sample. As a result, this rendered the samples incapable of producing confidence intervals as the standard errors for these strata were undefined without resorting to extra assumptions. Of the five potentially affected strata, two of the single sampled item strata included exceptions. 

In validating the projections produced by the Agency, the statistician was able to replicate the projected results of the Agency while estimating the impact of including the exceptions detected by OIG when testing the subsample of the Agency s sample. 

The statistician used a two-tier approach to project the rate of improper payments for 7(a) guaranty purchases made in FY 2011. A two-tier approach allows for the projection of audit results from a full sample to be expanded based on the results of a sub-sample. Table 2, below, combines the first-tier and the second-tier approaches. Step one in the Table illustrates the first-tier and steps two through five illustrate the second-tier. 

When projecting the sample results that the Agency agreed with to the population, the OIG improper payment amount is approximately 11 times more than the improper payment amount produced using the full-sample results. When projecting the subsample results including additional improper payments identified by the OIG, but not agreed to by the Agency, to the population, the OIG improper payment amount is approximately 23 times more than the improper payment amount produced using the full sample results. This is illustrated in Table 2 below. 

28 The sample of 30 loans was comprised of 29 unique transactions because the statistician used a sample design that allowed loans to be sampled more than once.
28 

Appendix II: Statistical Approach, Sampling Methodology and Projections 

Table 3 Adjusted Estimated Improper Payment Rate 

Tier Step Description Population Estimated with All 7 Improper Payments Population Estimated with 6 Improper Payments One 1 Calculate SBA Improper Payment Rate 29 0.53% 0.53% 

Two 

2 Calculate OIG Improper Payment Rate 30 12.05% 5.92% 

3 Ratio Estimate of Step 2 / Step 1 22.87 11.24 

4 Result of Step 3 Multiplied by Agency s Original Improper Payment Rate of 0.877% 31 20.05% 9.86% 

5 Estimated Improper Payment Dollar Amount $472,398,689 $232,238,017 Source 2 Summit Consulting Statistician 

29 

30 This Improper Payment Rate is based on the number of improper payments SBA detected in the sample of 30 loans reviewed by the OIG. 

31 This Improper Payment Rate is based on the number of improper payments OIG detected when reviewing its sample of 30 loans. 0.877 percent is the initial improper payment rate calculated by the SBA after completing the FY 2011 improper payment review of 7(a) guaranty purchases. This would have been the rate reported in the annual Agency Financial Repot, had the OIG not conducted its own review and provided the Agency with the results.


31 

Appendix IV: Management Comments 

Date: September 21, 2012 

To: John K. Needham Assistant Inspector General for Auditing 

From: Jeanne Hulit Associate Administrator, Office of Capital Access 

John A. Miller Director, Office of Financial Program Operations 

Subject: Response to Draft Report on The Small Business Administration s Improper Payment Rate for 7(a) Guaranty Purchases Remains Significantly Underestimated, Project No. 11008 

Executive Summary The Office of Inspector General (OIG) states that the Agency misinformed Congress and the public about its level of improper payments made for 7(a) guaranty purchases and the potential for financial loss by significantly underestimating its FY2011 improper payment rate. This assertion is incorrect; it is based on invalid and inconclusive statistical results, and it ignores the Agency s stewardship over taxpayer dollars. The SBA s improper payment rate of 1.73 percent is a statistically valid estimated rate. That said, based upon SBA s work with the OIG subsequent to the OIG s initial draft report, the Agency believes that the improper payment rate could lie between the Agency s improper payment rate of 1.73 percent and the OIG s reported rate of 9.86 percent for all 7(a) guaranteed loans. 33 

As such, SBA would like to communicate the following points: 

OIG s report is based on a flawed statistical approach. OIG s report ignores the Agency s stewardship of taxpayer dollars. The Agency has addressed the OIG s recommendations related to improper payments and has taken appropriate action. Going forward, the Agency will continue to reduce and prevent improper payments. 

33 OIG Report 12-10, SBA Generally Meets IPERA Reporting Guidance but Immediate Attention is Needed to Prevent and Reduce Improper Payments, March 15, 2012.
32 

Appendix IV: Management Comments 

The Agency Is a Faithful Steward of Taxpayer Dollars Agency data shows that the 7(a) 12 Month Purchase Rate for FY2011 was 4.4 percent. 34 

Purchase rates rose in 2009 (6.6 percent) and 2010 (7.3 percent) due, in part, to the recession. The current 7(a) 12 Month Purchase Rate of 3.3 percent is 37 percent lower than FY2011 and 55 percent lower than FY2010. This same trend is true for both the 12 Month Charge-Off and Delinquency Rates which peaked in late 2010 and since declined. 

SBA strives to ensure proper payments on guaranty purchases consistent with SBA regulations and policies. The FY2011 draft OIG report references the OIG s FY2008 improper payment audit and its recommendation to mitigate the risk of future improper payments. The Office of Financial Program Operations (OFPO), which oversees all of SBA s guaranty purchase centers, addressed the FY2008 OIG recommendations. OFPO has made significant and purposeful strides in improving its processes to reduce and prevent improper payments by implementing internal controls. Among the controls instituted, OFPO established Quality Improvement Programs (QIP) in all of its centers. The QIP Team reviews actions posing the highest level of improper payment risk to the Agency and uses that data to analyze and improve center processes. SBA complies by performing the annual improper payment audit and, in addition, conducts random guaranty purchase reviews prior to guaranty payment on a daily basis through the QIP. 

OFPO recognizes the importance training plays in ensuring SBA employees possess the knowledge and skills necessary to perform robust improper payment reviews. The National Guaranty Purchase Center (NGPC), Fresno Commercial Loan Servicing Center (FSC), and Little Rock Commercial Loan Servicing Center (LRSC), collectively known as the Centers for the 

purposes of this document, offer monthly training sessions to their employees and provide both general and individualized feedback based on the reviews conducted by the QIP. Additionally, 

OFPO updates policies and procedures to align with the Agency s goal of reducing and preventing improper payments. 

34 12 Month Purchase Rate Sum of gross dollars purchased over the previous 12 months, divided by the sum of total gross outstanding unpurchased dollars at the end of the period plus the sum of gross dollars purchased over the previous 12 months. 

0.00% 1.00% 2.00% 3.00% 4.00% 5.00% 6.00% 7.00% 8.00% 7(a) Rates (FY05 - Present) 

Delinquency Rate 

12 Month Purchase Rate 

12 Month Charge-Off Rate

34 

Appendix IV: Management Comments 

Wiley, but no chapter or page references. Upon review of the text, SBA s statistician determined that the approach the OIG took was likely the double sampling approach described in chapter 14 of Thompson. To date, the OIG has yet to provide SBA with a calculation of the variance or confidence interval around their estimate. On page 141 of Thompson, the formulas to determine an estimate of the level of improperly paid dollars and the variance of that estimate are described. SBA s statistician calculated a variance and confidence interval according to Thompson using loan-level results from the OIG s review of the subsample of 30 loans from Appendix III of their draft report. The column in Appendix III titled All OIG Improper Payments Plus One Identified by SBA & OIG is treated as the variable of interest (y-values) and the column titled Improper Payments Not Agreed to by SBA is used to determine the auxiliary variable of the subsample (x-values). 

It is important to note that the OIG indicated that when drawing the subsample from the SBA s sample of 303 loans that they followed the sampling technique that SBA describes in its FY2011 sampling methodology, i.e., stratified probability proportional to size (PPS) sampling with replacement. Therefore, SBA s statistician calculated the variance of the improper payments in 

OIG s subsample (y-values) according to the approach in SBA s FY2011 sampling methodology document. 

This calculation is provided by SBA to the OIG via Appendix I, which includes the data and calculations for the ratio estimator, variance, and confidence intervals for an estimate of improperly paid dollars using the Double Sampling approach described in chapter 14 of Thompson. Table 1 below shows summary results of the Excel calculations. Tau x represents the population estimate of improperly paid dollars based on the auxiliary variable, SBA s determination of improper payments in the sample of 303 loans. Tau r represents the ratio estimator of the population total of the variable of interest, or the SBA s estimate of OIG s determination of improper payments based on the Double Sampling approach. SBA s statistician estimates Tau r to be $248 Million. 

Table 1: Results of OIG Double Sampling Approach, Estimated by SBA s Statistician 

Variable Value tau x $151,578,749 tau r $248,411,045 Variance $259,920,039,143,587,000,000 Standard Deviation $16,122,035,825 

90 Percent CI Lower Bound -$26,272,337,887 90 Percent CI Upper Bound $26,769,159,977
35 

Appendix IV: Management Comments 

As Table 1 shows, the variance around the ratio estimator yields 90 percent confidence interval that is not materially differentiated from positive or negative infinity, rendering OIG s estimator devoid of reasonable operational meaning. More specifically, OIG s sample yields an Expected Improper Payment of $248 Million, and a Standard Deviation of $16 Billion. These statistics indicate that a 90 Percent confidence interval yields a Lower Bound Estimate of minus $26 Billion, and an Upper Bound Estimate of positive $27 Billion. The actual annual purchases for FY2011 - proper and improper was $2.4 Billion. The OIG s Standard Deviation for improper payments is $16 Billion. That is 6.6 times greater than all of SBA s actual FY2011 purchases. This difference evidences the inherent flaw in the OIG s methodology. 

Given that all the 7(a) purchases (proper and improper) for FY2011 were $2.4 Billion, this range indicates that there is a 90 percent chance that the actual 7(a) improper payments for FY2011 were anywhere from zero percent to 100 percent of total purchases. In essence, these statistics state that there is a 90 percent chance that all of the SBA 7(a) purchases were either proper or improper. 

By contrast, the SBA findings yield an Estimated Improper Payment of $41 Million, or 1.7 percent of all the 7(a) purchases for FY2011, and a Standard Deviation of $14 Million. A similar 

90 percent confidence interval yields a Lower Bound Estimate of $18 Million, or 0.8 percent of all the FY2011 7(a) purchases, and an Upper Bound Estimate of $64 Million, or 2.7 percent of all the FY2011 7(a) purchases. Therefore, SBA does not believe that the Double Sampling approach used by the OIG provides improved estimates of the population total of improper payments. 

Conclusion The SBA seeks to ensure accurate reporting to Congress and the public on the risks and returns of the Agency s loan programs. SBA is committed to good stewardship of taxpayer dollars. Due to the deterioration in the economy, recent 7(a) cohorts have required additional subsidies. More specifically, the projected macroeconomic performance projected by OMB and the SBA s assumption based on actual historical loan performance data in similar conditions have increased the Agency s projected default rate so as to require subsidy appropriations to cover projected losses in the 7(a) loan programs. 36 Using this model for the 7(a) Guaranty program, the Agency estimated a Cumulative Purchase Rate of 13.5 percent, Cumulative Recovery Rate of 25 percent, and Cumulative Loss Rate of 10.1 percent for the 2002-2011 cohorts. 37 However, in actuality, the Cumulative Purchase Rate was 8.1 percent; the Cumulative Recovery Rate was 11.9 percent; and the Cumulative Loss Rate was 7.1 percent. This speaks to the integrity of the SBA s data 

36 The Office of Financial Analysis and Modeling (OFAM) executes econometric and other models for SBA s loan 

programs that project the subsidy required to operate the SBA s loan programs. These models incorporate cash flow assumptions based on historical loan performance; cohort composition assumptions in terms of characteristics such 

as loan size, geographic region, industry, etc.; and projected macroeconomic performance. 37 The Cumulative Purchase Rate is different than the Annual Purchase Rate. Broadly speaking the Cumulative Purchase Rate for a specific loan guarantee cohorts approximates the summation of the Annual Purchase Rates for that cohort.
36 

Appendix IV: Management Comments 

modeling and the Agency s commitment to risk management and accountability of taxpayer dollars. SBA believes its oversight of improper payments is equally well-managed and reported. 

OFPO Management s response to the recommendations in the draft report is noted as follows: 1. Create a more comprehensive improper payment detection checklist for reviewing 7(a) guaranty purchase to address the many requirements that reviewers must be familiar with when conducting improper payment reviews. The checklist should be consistent with the SOP and include details for assessing: eligibility, use of proceeds, debt refinancing, change of ownership, credit analysis, repayment ability, equity injections, standby debt, early default/early problem, collateral, IRS tax verification, loan servicing, and liquidation. 

OFPO concurs with this recommendation and has taken the following actions: 

The Centers have developed comprehensive checklists addressing the various requirements loan specialists must be familiar with when conducting improper payment reviews. Versions of these checklists have been shared with the OIG, with feedback incorporated into the documents. These documents encompass the pertinent purchase review areas and are designed to ensure lender compliance with all 7(a) loan program requirements. The centers have finalized this revised document and have since implemented its use when conducting reviews. 2. Develop a procedure for monitoring the effectiveness of the checklist for detecting improper payments. 

OFPO concurs with this recommendation and has taken the following actions: 

OFPO has developed a process for monitoring the effectiveness of the checklist for detecting improper payments and will share the process with the OIG. OFPO is open to the OIG s suggestions for improvement on the process. 3. Provide training to Little Rock improper payment reviews on the Agency s policies concerning business plans for Community Express loans, using credit scoring, IRS Form 

4506 and determining early default to ensure these reviewers are aware of and understand the 7(a) loan program requirements that resulted in undetected improper payments. 

OFPO concurs with this recommendation. Over the past few months, LRSC has held several training sessions discussing general servicing guidance, portfolio servicing actions, 504 Department loan servicing, and SBA Express loan purchases. LRSC will continue to hold monthly training sessions with the loan specialists, and will include business plans for Community Express loans, using credit scoring, IRS Form 4506 and determining early default to ensure these reviewers are aware of and understand the 7(a) loan program requirements that resulted in undetected improper payments.
37 

Appendix IV: Management Comments 4. Provide training to Fresno improper payment reviewers on the Agency s policy concerning the use of credit scoring to ensure these reviewers are aware of and understand the 7(a) loan program requirements that resulted in undetected improper payments. 

OFPO concurs with this recommendation and has taken the following actions: 

FSC and OFPO appreciate the value of the continuing education of their employees. Over the past few months, FSC has held various training sessions including, general loan servicing guidance, workout training, SBA Express liquidation training, E-Tran training, and ARC training. FSC will continue to hold monthly training sessions with the loan specialists and will include the use of credit scoring to ensure these reviewers are aware of and understand the 7(a) loan program requirements that resulted in undetected improper payments. 5. Provide training to improper payment reviewers at the National Guaranty Purchase Center on the Agency s policies concerning assessing repayment ability, validating equity injections, and determining early default payments to ensure these reviewers are aware of and understand the 7(a) loan requirements that resulted in undetected improper payments. 

OFPO concurs with this recommendation and plans the following actions: 

NGPC and OFPO recognize the importance training plays in ensuring loan specialists possess the knowledge and skills necessary to perform improper payment reviews. NGPC offers monthly training sessions to all loan specialists and provides both general and individualized feedback based on QIP reviews. NGPC will continue to provide training to include assessing repayment ability, validating equity injections, and determining early default payments to ensure these reviewers are aware of and understand the 7(a) loan requirements that resulted in undetected improper payments. 6. Determine the appropriate amount of time needed to better identify erroneous payments when conducting improper payment reviews. 

OFPO concurs with this recommendation and plans to take the following actions: 

OFPO and its Centers are committed to improving the Agency s improper payment review 

process and ensuring loan specialists have the appropriate resources to efficiently conduct such reviews, consistent with Agency policies, and detect improper payments while maintaining high production standards. OFPO will conduct an analysis to determine the time needed to better identify erroneous payments when conducting improper payment reviews by March 31, 2013 which is prior to the FY2013 IPERA review.


40 

Appendix V: OIG Statistician Response to SBA Comments 

SummitConsulting LLC October 18,2012 

To: Jeff Brindle, John Needham, Betty Norwood, Christopher Scott, Terry Settle, and Karmel Smith, SBA- OIG 

From: Albert Lee and Corey West, Summit Consulting, LLC (Summit) 

Date: October 18, 2012 

Subject: SBA 7(a) Purchase Report Response 

Memo Objectives 

This memo serves to: 1. Explain and correct the methodological and conceptual errors in the Agency s September 21, 2012 memo. 2. Illustrate the implausibility of the Agency s assumption that the six OIG-identified discrepancies existed exclusively in the subsample, and nowhere else in the sample of 303 transactions. 

Background 

For the 7(a) Purchase IPIA project, OIG tasked Summit Consulting to: 1. Assess the sampling methodology implemented for calculating the improper payment rate and improper payment dollar amount for SBAs 7(a) FY2011 portfolio. 2. Replicate the calculation of the improper payment rate and improper payment dollar amount point estimates produced by the Agency. 3. Determine the appropriate improper payment rate and improper payment dollar amount estimates based on a sub-sample of 30 transactions from the sample of 303 selected by the Agency. 38 

At the conclusion of the initial phase of the project, Summit Consulting concluded the following: 1. The Agency had adopted a sampling methodology consistent with best practices, namely, a stratified random sample with selection probability proportional to size (PPS) with replacement. 2. Summit Consulting successfully replicated the Agency s point estimates based on the sample design. 

38 The documents reviewed to perform these tasks were: SBA_IPIA_Methodology_FY2011_20110707.docx and Summary_Estimated_Improper_Payments_FY2011_20111107.xlxs
41 

Appendix V: OIG Statistician Response to SBA Comments 3. Based on the result of the sub-sample of 30 transactions, Summit Consulting determined the Agency s improper payment rate to be approximately 20% and not the 1.7% calculated by the Agency. This translated to an underestimation of the improper payment dollar amount figure of approximately $41 million, whereas the findings of implementing the double sampling approach produced an improper payment dollar amount of approximately $472 million. 39 4. Summit Consulting further noted that the Agency s sample design included five strata that contained only one sampled transaction. Without additional assumptions, standard errors were undefined for these five of eleven strata. As of our review in September 2012, the Agency had not provided any confidence intervals with its point estimates. 

SBA s September 21, 2012 Memo 

The Agency issued a memo dated September 21, 2012 responding to the OIG s analysis and conclusions. The Agency stated that SBA-OIG s conclusion is based on invalid and inconclusive statistical results, and its estimated improper payment rate of 1.73 percent is a statistically valid estimated rate. 40 The 

Agency reached this conclusion after an attempt to replicate the OIG s calculation based on the Thompson (1992) formula on double sampling. According to its calculation, a point estimate of $248.4 Million was provided. This compares to the Agency s finding of $41 Million, or 1.7 percent of all the 

7(a) purchases for FY2011, and a standard deviation of $14 Million. Consequently, SBA does not believe that the Double Sampling approach&improved estimates of the population total of improper payments. 41 

Summary of Findings 

Upon reviewing the Agency s September 21, 2012 memo, Summit identified methodological and conceptual errors, namely: 1. Methodologically, the Agency failed to account for the complex sample design (e.g., a stratified random sample design with PPS) in their projection of the double sample. 2. Conceptually, the Agency erroneously extrapolated one unresolved disagreement with the OIG, and not the six discrepancies identified by the OIG. 3. In its extrapolation, the Agency made an unsubstantiated assumption that the six OIG-identified discrepancies existed in the subsample exclusively, and nowhere else in the sample of 303 transactions. This assumption led to an underestimation of the Agency s improper payment rate and amount. After correcting for the aforementioned errors, the OIG concluded the Agency s estimated improper 

payment dollar amount to be approximately $472 million. 

40 

39 SBA_11008_Final_20120131_v2.docx 

41 SBA Memo to John Needham, date September 21, 2012. Page 1, Executive Summary. SBA Memo, Page 5, third paragraph.
42 

Appendix V: OIG Statistician Response to SBA Comments 

Methodologically, the Agency failed to account for the stratified random sample design in their projection of the double sample 

The full sample and subsample were selected using a stratified random PPS with replacement design. Consequently, the Agency s extrapolation needed to take this complex sample design into consideration. Instead, the Excel spreadsheet provided by the Agency applied formulas that were applicable for a simple random sample design for the double sample. As a result, the Agency s extrapolations were inconsistent with the complex sample design. 

Conceptually, the Agency erroneously extrapolated one unresolved disagreement with OIG, and not 

the six discrepancies identified by the OIG. 

In the present context, double sampling seeks to determine whether and the extent to which discrepancies exist between the Agency s sample evaluation and that of the OIG. To achieve this, the 

OIG subsampled 30 transactions from the Agency s sample of 303 transactions. The OIG performed its independent evaluation to these 30 transactions and compared its finding with those of the Agency, and noted the discrepancies, if any. 

These discrepancies are important because if they exist in the subsample, it is likely that other discrepancies also exist in the Agency s sample beyond the subsample of 30. Double sampling allows 

an extrapolation of the subsample discrepancies to the entire Agency sample, which could further extrapolate to the entire population from which the Agency s sample was selected. Therefore, the 

proper understanding of what constitutes discrepancies is necessary to double sampling. The relevant discrepancies are the differences between the OIG s findings and the Agency s initial findings in the subsample. 

Below is a table tabulating the 29 distinct subsampled transactions, comparing the OIG and Agency s audit findings. 42 Among the 29 distinct subsampled transactions, six distinct discrepancies were found. The total amount of these six distinct discrepancies is about $1,763,205. For most of the discrepancies, the Agency failed to identify them as improper payments whereas the OIG identified them as such. 

42 Because we subsample with replacement, one transaction was selected twice in the subsample.