![](media/image1.png){width="5.166666666666667in"
height="1.5930555555555554in"}

May 11, 2001

Honorable Dan Burton

Chairman

Committee on Government Reform

U.S. House of Representatives

2157 Rayburn House Office Building

Washington, DC 20515-6143

Dear Mr. Chairman:

In response to your request of April 5, 2001, I am pleased to submit the
Office of Inspector General\'s (OIG) assessment of what we consider to
be the Small Business Administration's (SBA) ten most significant
performance measures and the data underlying those measures. Because of
the importance of performance-based management, OIG has expended
substantial resources in reviewing, and reporting on, the Agency's
implementation of the Government Performance and Results Act (GPRA). We
identified managing for results as a top ten management challenge for FY
2001.

As indicated in the enclosed report, OIG has conducted five audits and
two inspections that assess GPRA implementation, and the validity and
reliability of underlying performance data of selected major Agency
programs. These reports can be accessed on SBA\'s website at
[www.sba.gov/IG/reports.html]{.underline}.

For the assessment you requested, OIG has used the measures in SBA\'s FY
2002 Performance Plan because they improve substantially on the measures
in the FY 2000 Performance Report. In addition to identifying the ten
most significant performance measures for achieving SBA\'s mission, OIG
has indicated (1) the extent to which the performance measure is a
useful indicator of performance, (2) whether the data or information
underlying the measure is valid and accurate, and (3) what actions SBA
is taking, or plans to take, to resolve measure and data problems.

If you or your staff have any questions, please do not hesitate to call
me or Peter McClintock, Deputy Inspector General, at (202) 205-6586.

Sincerely,

> ![](media/image2.png){width="2.1368055555555556in"
> height="0.7097222222222223in"}

Phyllis K. Fong

Inspector General

Enclosure

![](media/image3.png){width="6.4847222222222225in"
height="9.854861111111111in"}

# SMALL BUSINESS ADMINISTRATION OFFICE OF INSPECTOR GENERAL 

# ASSESSMENT OF 

# THE TEN MOST SIGNIFICANT AGENCY PERFORMANCE MEASURES

**Summary.** In the past two years, the Office of Inspector General
(OIG) has conducted five audits and two inspections of selected major
Small Business Administration (SBA) programs to assess the
implementation of the Government Performance and Results Act (GPRA), and
the validity and reliability of the performance data used. In general,
this work has concluded that improvements are needed in implementing
GPRA. We found that some major SBA programs still needed: (1) outcome
measures, (2) additional cost or service quality measures, (3) an
alignment of the measures with the program\'s statutory mission, and (4)
valid and reliable data. The OIG\'s reports can be accessed on SBA\'s
website at [www.sba.gov/IG/reports.html]{.underline}. For FY 2001, one
of SBA\'s ten most significant management challenges is managing for
results.

As requested, OIG has selected what we consider to be SBA\'s ten most
significant performance measures. SBA\'s FY 2000 Performance Report
contains 35 measures of which almost all are output measures. Because
the Agency\'s FY 2002 Performance Plan significantly improved the
quality of the measures, we have chosen the ten most significant
measures from the FY 2002 plan. SBA now has two general program goals:

\(1\) help small businesses succeed, and (2) help families and business
recover from disasters. The Agency has reduced the number of program
measures to 13 and made substantial improvements in the quality of the
measures. SBA lists one operational goal---improve SBA management. Under
this goal, the Agency lists six initiatives: (1) manage for results, (2)
transform the workforce, (3) modernize our information systems,

\(4\) improve financial management, (5) improve credit program
management, and (6) create electronic government. These initiatives have
no measures associated with them. SBA has indicated that the initiatives
will be published after a review by the incoming Administrator. Five of
these initiatives relate specifically to the ten FY 2001 management
challenges.

In the following table, OIG provides the ten most significant
performance measures and our assessment of (1) the extent to which the
performance measure is a useful indicator of performance, (2) whether
the data or information underlying the measure is valid and accurate,
and (3) what actions SBA is taking, or plans to take, to resolve measure
and data problems. OIG believes that, of the ten measures, four are
useful in indicating program performance, four are somewhat useful, and
two are not useful. In responding to OIG\'s challenge to improve the
Agency\'s \"managing for results process,\" SBA agreed to develop
specific guidance concerning the preparation of organizational
performance goals and indicators.

In audits and inspections relating to the ten measures identified in
this report, OIG found data validity and accuracy problems. To address
these problems, in its FY 2000 Performance Report, SBA stated that it
will pursue four strategies for data validation and verification: (1)
foster organizational commitment and capacity for data quality, (2)
assess the quality of existing data, (3) respond to data limitations,
and (4) build quality into the development of performance data.

+-----------+----------------+--------------+----------+--------------+
| # FY 2    | #              | **Data       | #        | # SBA Act    |
| 002 Signi |  Usefulness as | Validity and | OIG Audi | ions to Reso |
| ficant Pe |  an Indicator  | Accuracy**   | t and In | lve Problems |
| rformance | of Performance |              | spection |              |
|  Measures |                |              |  Reports |              |
|           |                |              |          |              |
| #         |                |              |          |              |
+===========+================+==============+==========+==============+
| Goal      |                |              |          |              |
| 1--Help   |                |              |          |              |
| Small     |                |              |          |              |
| Business  |                |              |          |              |
| Succeed   |                |              |          |              |
+-----------+----------------+--------------+----------+--------------+
| 1\.       | Useful. By     | OIG has not  | None     |              |
| R         | determining    | reviewed the |          |              |
| egulatory | the cost       | underlying   |          |              |
| cost      | savings of     | data for     |          |              |
| savings   | reducing or    | this         |          |              |
| to small  | minimizing the | measure. We  |          |              |
| b         | impact of      | anticipate   |          |              |
| usinesses | regulations on | including it |          |              |
|           | small          | in our audit |          |              |
|           | businesses,    | plan for FY  |          |              |
|           | SBA can        | 2002.        |          |              |
|           | provide some   |              |          |              |
|           | indication of  |              |          |              |
|           | the success of |              |          |              |
|           | the efforts of |              |          |              |
|           | the Office of  |              |          |              |
|           | Advocacy in    |              |          |              |
|           | reviewing and  |              |          |              |
|           | monitoring     |              |          |              |
|           | regulatory     |              |          |              |
|           | proposals for  |              |          |              |
|           | small business |              |          |              |
|           | impact.        |              |          |              |
|           | Nevertheless,  |              |          |              |
|           | it will not be |              |          |              |
|           | possible to    |              |          |              |
|           | draw a direct  |              |          |              |
|           | causal         |              |          |              |
|           | relationship.  |              |          |              |
+-----------+----------------+--------------+----------+--------------+

+-----------+---------------+--------------+-----------+--------------+
| # FY 2    | # Us          | **Data       | # OIG A   | # SBA Act    |
| 002 Signi | efulness as a | Validity and | udit and  | ions to Reso |
| ficant Pe | n Indicator o | Accuracy**   | Inspectio | lve Problems |
| rformance | f Performance |              | n Reports |              |
|  Measures |               |              |           |              |
|           |               |              |           |              |
| #         |               |              |           |              |
+-----------+---------------+--------------+-----------+--------------+
| 2\. Jobs  | Useful. The   | A 7(a)       | *Results  | SBA is       |
| Created   | number of     | business     | Act       | working with |
| by SBA    | jobs created  | loan program | Pe        | the          |
| Borrowers | by SBA        | audit of the | rformance | Department   |
|           | borrowers     | information  | Me        | of Labor and |
|           | directly ties | reported for | asurement | the Internal |
|           | to the        | this measure | for the   | Revenue      |
|           | legislative   | in FY 1999   | 7(a)      | Service to   |
|           | purpose of    | showed that  | Business  | obtain       |
|           | the 504 loan  | the data     | Loan      | reliable     |
|           | program and   | were based   | Program*, | data on jobs |
|           | is a useful   | on estimates | Audit     | created. SBA |
|           | outcome       | that have    | Report    | acknowledges |
|           | indicator for | not been     | #1-01,    | the          |
|           | SBA's loan    | validated    | December  | challenge it |
|           | programs.     | and          | 4, 2000.  | faces in     |
|           |               | verified.    |           | quantifying  |
|           |               | SBA is still |           | the causal   |
|           |               | using the    |           | relationship |
|           |               | estimates.   |           | between      |
|           |               | Thus they    |           | outputs and  |
|           |               | are not      |           | measurable   |
|           |               | reliable.    |           | growth in    |
|           |               |              |           | the small    |
|           |               |              |           | business     |
|           |               |              |           | sector.      |
+-----------+---------------+--------------+-----------+--------------+
| 3\.       | Not useful.   | An SBIC      | *Results  | For FY 2002, |
| Improved  | This dollar   | program      | Act       | the number   |
| access to | volume        | audit found  | Pe        | of actual    |
| capital   | measure is    | that some of | rformance | SBIC         |
| and       | not useful in | the data     | Me        | financings   |
| credit:   | measuring the | relating to  | asurement | will be      |
| 7(a),     | success of    | the FY 1999  | for the   | reported by  |
| 504,      | the various   | o            | Small     | fiscal year  |
| M         | loan          | utputs---the | Business  | and,         |
| icroloan, | programs. It  | number of    | I         | therefore,   |
| & SBIC    | is an output  | finan        | nvestment | should be    |
| programs  | and does not  | cings---were | Company*, | more         |
|           | provide a     | not          | Audit     | reliable.    |
|           | meaningful    | reliable. A  | Report    |              |
|           | indicator of  | 7(a)         | #0-25,    | SBA is       |
|           | whether small | business     | September | developing   |
|           | businesses    | loan audit   | 6, 2000.  | specific     |
|           | have been     | found that   |           | guidance     |
|           | helped to     | the program  | *Results  | concerning   |
|           | succeed.      | was          | Act       | the          |
|           |               | reporting    | Pe        | preparation  |
|           |               | loans        | rformance | of           |
|           |               | approved     | Me        | or           |
|           |               | rather than  | asurement | ganizational |
|           |               | loans        | for the   | performance  |
|           |               | disbursed.   | 7(a)      | goals and    |
|           |               | For FY 2000  | Business  | indicators.  |
|           |               | the data     | Loan      | SBA is also  |
|           |               | were         | Program*, | developing   |
|           |               | reported in  | Audit     | standards    |
|           |               | the same     | Report    | and          |
|           |               | way.         | #1-01,    | procedures   |
|           |               |              | December  | for data     |
|           |               |              | 4, 2000.  | verification |
|           |               |              |           | and          |
|           |               |              |           | validation.  |
+-----------+---------------+--------------+-----------+--------------+

+-----------+---------------+--------------+-----------+--------------+
| # FY 2    | # Us          | **Data       | # OIG A   | # SBA Act    |
| 002 Signi | efulness as a | Validity and | udit and  | ions to Reso |
| ficant Pe | n Indicator o | Accuracy**   | Inspectio | lve Problems |
| rformance | f Performance |              | n Reports |              |
|  Measures |               |              |           |              |
|           |               |              |           |              |
| #         |               |              |           |              |
+-----------+---------------+--------------+-----------+--------------+
| 4\.       | Useful. This  | The 7(a) and | *Results  | For FY 2002  |
| Increase  | is a useful   | SBIC audits  | Act       | random       |
| diversity | output        | found that   | Pe        | sampling and |
| (number   | indicator     | the data     | rformance | projections  |
| of loans  | that measures | reported for | Me        | will be used |
| to        | the level of  | this measure | asurement | to identify  |
| targeted  | activity to   | for FY 1999  | for the   | financings   |
| groups)   | specific      | were not     | 7(a)      | to women and |
|           | targeted      | reliable or  | Business  | minorities.  |
|           | borrowers and | supported by | Loan      |              |
|           | is linked to  | source       | Program*, | SBA is       |
|           | the           | records. For | Audit     | developing   |
|           | legislative   | FY 2000 the  | Report    | standards    |
|           | purpose of    | data were    | #1-01,    | and          |
|           | selected SBA  | reported in  | December  | procedures   |
|           | programs.     | the same     | 4, 2000.  | for data     |
|           |               | way.         |           | verification |
|           |               |              | *Results  | and          |
|           |               | An           | Act       | validation.  |
|           |               | inspection   | Pe        |              |
|           |               | of data from | rformance |              |
|           |               | SBA loan     | Me        |              |
|           |               | processing   | asurement |              |
|           |               | centers      | for the   |              |
|           |               | found        | Small     |              |
|           |               | uncertainty  | Business  |              |
|           |               | regarding    | I         |              |
|           |               | what         | nvestment |              |
|           |               | constitutes  | Company*, |              |
|           |               | adequate     | Audit     |              |
|           |               | data         | Report    |              |
|           |               | v            | #0-25,    |              |
|           |               | erification. | September |              |
|           |               |              | 6, 2000.  |              |
|           |               |              |           |              |
|           |               |              | *Advisory |              |
|           |               |              | Me        |              |
|           |               |              | morandum: |              |
|           |               |              | Data      |              |
|           |               |              | Issues    |              |
|           |               |              | Regarding |              |
|           |               |              | the       |              |
|           |               |              | P         |              |
|           |               |              | rocessing |              |
|           |               |              | Centers,* |              |
|           |               |              | I         |              |
|           |               |              | nspection |              |
|           |               |              | Report    |              |
|           |               |              | #         |              |
|           |               |              | 00-09-01, |              |
|           |               |              | September |              |
|           |               |              | 28 2000.  |              |
+-----------+---------------+--------------+-----------+--------------+
| 5\.       | Useful. This  | OIG has not  |           |              |
| Number of | is an output  | reviewed the |           |              |
| start-up  | indicator for | data for     |           |              |
| firms     | measuring SBA | this         |           |              |
| financed  | activity in   | measure.     |           |              |
| by 7(a),  | reaching      |              |           |              |
| 504, and  | small         |              |           |              |
| SBIC      | business      |              |           |              |
| programs  | start-ups.    |              |           |              |
+-----------+---------------+--------------+-----------+--------------+

+-----------+----------------+--------------+----------+--------------+
| # FY 2    | #              | **Data       | #        | # SBA Act    |
| 002 Signi |  Usefulness as | Validity and | OIG Audi | ions to Reso |
| ficant Pe |  an Indicator  | Accuracy**   | t and In | lve Problems |
| rformance | of Performance |              | spection |              |
|  Measures |                |              |  Reports |              |
|           |                |              |          |              |
| #         |                |              |          |              |
+-----------+----------------+--------------+----------+--------------+
| 6\.       | Somewhat       | OIG has not  |          |              |
| Mandated  | useful. It is  | reviewed the |          |              |
| fair      | a useful       | data         |          |              |
| share of  | indicator of   | underlying   |          |              |
| Federal   | G              | this         |          |              |
| prime     | overnment-wide | measurement  |          |              |
| contract  | success but    | because they |          |              |
| dollars   | not of SBA     | are provided |          |              |
|           | success        | by other     |          |              |
|           | because SBA    | Federal      |          |              |
|           | does not       | agencies.    |          |              |
|           | control        |              |          |              |
|           | procurement in |              |          |              |
|           | other Federal  |              |          |              |
|           | agencies.      |              |          |              |
+-----------+----------------+--------------+----------+--------------+
| 7\. 8(a)  | Not useful.    | SBA did not  | *Per     | Management   |
| program   | This outcome   | report data  | formance | took no      |
| client    | measure does   | for this     | Mea      | exception to |
| success   | not address    | measure in   | surement | our          |
| rate 3    | success in     | its FY 2000  | for the  | re           |
| years     | terms of       | Performance  | Minority | commendation |
| after     | applicable     | Report.      | Small    | that SBA     |
| gr        | legislation.   |              | Business | measure the  |
| aduation. | Public Law     |              | and      | success of   |
| Success   | (P.L.) 100-656 |              | Capital  | the 8(a) BD  |
| means     | states that    |              | O        | program in   |
| that the  | \"\...the      |              | wnership | accordance   |
| client is | measure of     |              | Dev      | with the     |
| still in  | success ...    |              | elopment | measure of   |
| business. | shall be the   |              | P        | success      |
|           | number of      |              | rogram*, | prescribed   |
|           | competitive    |              | Audit    | in P.L.      |
|           | firms that     |              | Report   | 100-656.     |
|           | exit the       |              | #1-11,   | After the    |
|           | program        |              | March    | report was   |
|           | without being  |              | 27,      | issued,      |
|           | unreasonably   |              | 2001.    | management   |
|           | reliant on     |              |          | stated that  |
|           | section 8(a)   |              |          | it would     |
|           | contracts and  |              |          | make         |
|           | that can       |              |          | programmatic |
|           | compete in the |              |          | changes so   |
|           | mainstream of  |              |          | it would be  |
|           | the American   |              |          | possible to  |
|           | economy\...\"  |              |          | track the    |
|           | Section        |              |          | indicators   |
|           | 7(j)(10)(H) of |              |          | as stated in |
|           | the Small      |              |          | the law.     |
|           | Business Act   |              |          |              |
|           | provides that  |              |          |              |
|           | an 8(a) firm   |              |          |              |
|           | demonstrates   |              |          |              |
|           | its ability to |              |          |              |
|           | compete in the |              |          |              |
|           | marketplace    |              |          |              |
|           | without        |              |          |              |
|           | assistance     |              |          |              |
|           | from the 8(a)  |              |          |              |
|           | business       |              |          |              |
|           | development    |              |          |              |
|           | (BD) program   |              |          |              |
|           | by             |              |          |              |
|           | substantially  |              |          |              |
|           | achieving the  |              |          |              |
|           | targets,       |              |          |              |
|           | objectives,    |              |          |              |
|           | and goals      |              |          |              |
|           | contained in   |              |          |              |
|           | that firm\'s   |              |          |              |
|           | business plan. |              |          |              |
+-----------+----------------+--------------+----------+--------------+

+-----------+---------------+--------------+-----------+--------------+
| # FY 2    | # Us          | **Data       | # OIG A   | # SBA Act    |
| 002 Signi | efulness as a | Validity and | udit and  | ions to Reso |
| ficant Pe | n Indicator o | Accuracy**   | Inspectio | lve Problems |
| rformance | f Performance |              | n Reports |              |
|  Measures |               |              |           |              |
|           |               |              |           |              |
| #         |               |              |           |              |
+-----------+---------------+--------------+-----------+--------------+
| 8\.       | Somewhat      | No data have | *Coo      | To obtain    |
| Customer  | useful. This  | been         | rdination | information  |
| sat       | intermediate  | reported.    | and       | on the final |
| isfaction | outcome       |              | Pe        | *outcome*,   |
| rate with | measure shows |              | rformance | OIG          |
| Entrep    | the quality   |              | Me        | recommended  |
| reneurial | of services   |              | asurement | that SBA     |
| De        | delivered as  |              | in SBA\'s | develop a    |
| velopment | perceived by  |              | Entrep    | client       |
| (ED)      | SBA clients.  |              | reneurial | survey. SBA  |
| programs  | Although more |              | De        | is surveying |
|           | refined than  |              | velopment | clients in   |
|           | an output     |              | P         | some of the  |
|           | measure       |              | rograms*, | ED programs. |
|           | focussing on  |              | I         | It is not    |
|           | quantity of   |              | nspection | yet clear    |
|           | assistance,   |              | Report    | whether      |
|           | this measure  |              | #         | outcomes,    |
|           | does not show |              | 00-09-02, | such as jobs |
|           | how the       |              | September | created,     |
|           | clients       |              | 28, 2000. | will be      |
|           | benefit. The  |              |           | reported.    |
|           | measure is    |              |           | SBA has      |
|           | useful for    |              |           | stated that  |
|           | indicating    |              |           | its program  |
|           | whether       |              |           | review of    |
|           | clients       |              |           | the Small    |
|           | believed the  |              |           | Business     |
|           | service to be |              |           | Development  |
|           | successful    |              |           | Center       |
|           | and, thus,    |              |           | network      |
|           | provides      |              |           | includes     |
|           | information   |              |           | outcomes.    |
|           | on how        |              |           |              |
|           | program       |              |           |              |
|           | service       |              |           |              |
|           | delivery can  |              |           |              |
|           | be improved.  |              |           |              |
+-----------+---------------+--------------+-----------+--------------+

+-----------+---------------+--------------+-----------+--------------+
| # FY 2    | # Us          | **Data       | # OIG A   | # SBA Act    |
| 002 Signi | efulness as a | Validity and | udit and  | ions to Reso |
| ficant Pe | n Indicator o | Accuracy**   | Inspectio | lve Problems |
| rformance | f Performance |              | n Reports |              |
|  Measures |               |              |           |              |
|           |               |              |           |              |
| #         |               |              |           |              |
+-----------+---------------+--------------+-----------+--------------+
| Goal      |               |              |           |              |
| 2         |               |              |           |              |
| --Helping |               |              |           |              |
| B         |               |              |           |              |
| usinesses |               |              |           |              |
| and       |               |              |           |              |
|           |               |              |           |              |
| Families  |               |              |           |              |
| Recover   |               |              |           |              |
| from      |               |              |           |              |
| Disasters |               |              |           |              |
+-----------+---------------+--------------+-----------+--------------+
| 9\. Field | Somewhat      | An audit of  | *Pe       | The Office   |
| presence  | useful. For a | the disaster | rformance | of Disaster  |
| within 3  | program where | program      | Me        | Assistance   |
| days of a | timeliness is | showed that  | asurement | has stated   |
| disaster  | of the        | this measure | for the   | it is taking |
| de        | essence, this | is not       | Disaster  | steps to     |
| claration | measure is    | reliable     | A         | establish    |
|           | useful.       | because      | ssistance | specific     |
|           | However, a    | disaster     | Program*, | performance  |
|           | field         | field office | Audit     | indicators   |
|           | presence      | locations do | Report    | to measure   |
|           | indicator     | not report   | #1-06,    | program      |
|           | addresses     | on it        | February  | outcomes.    |
|           | only the      | c            | 15, 2001. | They have    |
|           | efficiency of | onsistently. |           | also         |
|           | providing     |              |           | developed a  |
|           | on-site       |              |           | draft        |
|           | assistance.   |              |           | definition   |
|           | SBA does not  |              |           | which they   |
|           | have          |              |           | will share   |
|           | performance   |              |           | with         |
|           | indicators    |              |           | Disaster     |
|           | that measure  |              |           | Area         |
|           | whether the   |              |           | Directors at |
|           | disaster      |              |           | their next   |
|           | assistance    |              |           | meeting.     |
|           | program helps |              |           |              |
|           | businesses    |              |           |              |
|           | and families  |              |           |              |
|           | recover from  |              |           |              |
|           | disasters.    |              |           |              |
|           | Thus, there   |              |           |              |
|           | is no outcome |              |           |              |
|           | measure for   |              |           |              |
|           | this goal.    |              |           |              |
+-----------+---------------+--------------+-----------+--------------+

+-----------+---------------+--------------+-----------+--------------+
| # FY 2    | # Us          | **Data       | # OIG A   | # SBA Act    |
| 002 Signi | efulness as a | Validity and | udit and  | ions to Reso |
| ficant Pe | n Indicator o | Accuracy**   | Inspectio | lve Problems |
| rformance | f Performance |              | n Reports |              |
|  Measures |               |              |           |              |
|           |               |              |           |              |
| #         |               |              |           |              |
+-----------+---------------+--------------+-----------+--------------+
| 10\.      | Somewhat      | While OIG    | *Pe       | SBA is       |
| Customer  | useful. While | has not      | rformance | surveying    |
| sat       | this          | assessed the | Me        | borrowers    |
| isfaction | intermediate  | data, an     | asurement | who have     |
| rate      | outcome       | audit found  | for the   | received     |
|           | measure does  | that SBA     | Disaster  | loans.       |
|           | not measure   | plans to     | A         |              |
|           | the success   | survey only  | ssistance |              |
|           | of the        | borrowers    | Program*, |              |
|           | program in    | who have     | Audit     |              |
|           | helping       | received     | Report    |              |
|           | businesses    | loans.       | #1-06,    |              |
|           | and families  |              | February  |              |
|           | recover from  |              | 15, 2001. |              |
|           | disasters, it |              |           |              |
|           | is useful for |              |           |              |
|           | indicating    |              |           |              |
|           | whether loan  |              |           |              |
|           | recipients    |              |           |              |
|           | believe the   |              |           |              |
|           | service to be |              |           |              |
|           | successful.   |              |           |              |
|           | Thus, it can  |              |           |              |
|           | provide       |              |           |              |
|           | information   |              |           |              |
|           | on how        |              |           |              |
|           | delivery of   |              |           |              |
|           | program       |              |           |              |
|           | services can  |              |           |              |
|           | be improved.  |              |           |              |
+-----------+---------------+--------------+-----------+--------------+