GAO

United States General Accounting Office

Report to the Administrator, 

Small Business Administration

April 2000 SBA LOAN 

MONITORING 

SYSTEM

Substantial Progress 

Yet Key Risks and 

Challenges Remain

GAO/AIMD-00-124

Page 1 GAO/AIMD-00-124 SBA Loan Monitoring System

Contents

Letter 3

Appendixes Appendix I: GAO’s February 29, 2000, Testimony 8

Appendix II: Comments From the Small Business Administration 27

Related GAO Products 35

Abbreviations

BPR business process reengineering

COTS commercial-off-the-shelf software

IEEE Institute of Electrical and Electronics Engineers, Inc.

SBA Small Business Administration
Page 2 GAO/AIMD-00-124 SBA Loan Monitoring System
Page 3 GAO/AIMD-00-124 SBA Loan Monitoring System

United States General Accounting Office

Washington, D.C. 20548

Accounting and Information

Management Division

B-284931 Letr

April 25, 2000

The Honorable Aida Alvarez

Administrator, Small Business Administration

Dear Madam Administrator:

On February 29, 2000, we testified before the Subcommittee on 

Government Programs and Oversight, House Committee on Small 

Business, on the Small Business Administration’s (SBA) progress in 

performing planning actions for its loan monitoring system.1

 We testified 

that SBA had made substantial progress in completing the eight planning 

actions mandated by the Small Business Reauthorization Act of 1997 but 

still had to complete work for some actions and implement key functions to 

effectively manage the development of the loan monitoring system.

This report transmits our recommendations to assist SBA in completing 

work for the eight planning actions and in strengthening its information 

technology management practices. We are sending this report to you 

because of the significant investment that SBA plans to make in developing 

the loan monitoring system, and because of the key role of the loan 

monitoring system in SBA’s modernization. Our testimony is reprinted in 

appendix I. We performed our work from August 1999 through February 28, 

2000, in accordance with generally accepted government auditing 

standards.

SBA’s Chief Operating Officer provided us with comments on a draft of this 

report. These comments are discussed in the “Agency Comments” section 

and reprinted in appendix II.

Recommendations We recommend that the Administrator, Small Business Administration, 

take a series of actions to ensure that SBA completes the steps mandated 

by the act, as stated in our February 29, 2000, testimony. Specifically, we 

recommend that the Administrator

1

SBA Loan Monitoring System: Substantial Progress Yet Key Risks and Challenges Remain

(GAO/T-AIMD-00-113, February 29, 2000).
B-284931

Page 4 GAO/AIMD-00-124 SBA Loan Monitoring System

• complete the analyses of benefits and costs of alternatives being 

considered for each business process identified through SBA’s business 

reengineering effort, and complete an implementation plan for the new 

business processes; 

• perform benefit-cost analyses for systems alternatives to increase the 

probability that SBA will obtain a system that meets its needs at the 

lowest cost;

• develop the information architecture’s rules and standards for 

interoperability and maintainability of interrelated systems;

• identify high-level requirements for all internal reports and define 

detailed input and output data elements necessary for the timely 

generation of reports;

• complete the definition of specific data quality standards, develop a 

schedule of planned actions to improve data quality in the current 

systems, and implement data quality measures;

• define system capacity and performance requirements for the systems 

requirements documentation; 

• develop an acquisition strategy that ensures that a sound justification 

exists for pursuing custom-developed functions; and 

• continue to refine the cost-to-completion estimate following the 

completion of the benefit-cost analysis of alternatives and selection of 

the best alternatives for implementation, and develop a lifecycle cost 

estimate for the system and its components.

As also stated in our February 29, 2000, testimony, the implementation of 

project management processes and capabilities is essential for a major 

systems development acquisition. We therefore recommend that the 

Administrator ensure the implementation of key functions to effectively 

manage the development of the loan monitoring system. Specifically, to 

strengthen project management processes and controls, we recommend 

that the Administrator

• implement project tracking and oversight capabilities; 

• finalize and implement configuration management processes;

• acquire independent verification and validation for the loan monitoring 

system project;

• establish an internal quality assurance function; and 

• address the security challenge posed by Internet-based access to loan 

monitoring system functions and data, complete the security 

architecture, and update the security operating procedure.
B-284931

Page 5 GAO/AIMD-00-124 SBA Loan Monitoring System

As you know, the head of a federal agency is required by 31 U.S.C. 720 to 

submit a written statement on the actions taken on these recommendations 

to the Senate Committee on Governmental Affairs and the House 

Committee on Government Reform, not later than 60 days after the date of 

this report. A written statement must also be sent to the House and Senate 

Committees on Appropriations with the agency’s first request for 

appropriations made more than 60 days after the date of this report.

Agency Comments In commenting on a draft of this report, the Chief Operating Officer stated 

that SBA generally agreed with the recommendations. She added that SBA 

action had now been completed in part on five of our recommendations 

with the revision of the analysis of benefits-costs to address each business 

process, completion of the benefit-cost analysis of system alternatives, 

completion of the information technology architecture, revision of the 

systems requirements to include high-level system capacity and 

performance requirements, and update of the security operating procedure. 

She also noted that SBA had initiated or planned actions for the remaining 

recommendations.

We are sending copies of this report to Senator Christopher S. Bond, 

Chairman, and Senator John F. Kerry, Ranking Minority Member, Senate 

Committee on Small Business; Representative James M. Talent, Chairman, 

and Representative Nydia M. Velazquez, Ranking Minority Member, House 

Committee on Small Business; and Representative Roscoe G. Bartlett, 

Chairman, and Danny K. Davis, Ranking Minority Member, Subcommittee 

on Government Programs and Oversight, House Committee on Small 

Business. We are also sending copies of this report to The Honorable 

Jacob J. Lew, Director, Office of Management and Budget. Copies will also 

be made available to others upon request.
Page 6 GAO/AIMD-00-124 SBA Loan Monitoring System

Should you or your staff have any questions concerning this report, please 

contact me at (202) 512-6253. I can also be reached by e-mail at 

willemssenj.aimd@gao.gov.

Sincerely yours,

Joel C. Willemssen

Director, Civil Agencies Information Systems
Page 7 GAO/AIMD-00-124 SBA Loan Monitoring System
Page 8 GAO/AIMD-00-124 SBA Loan Monitoring System

Appendix I

GAO’s February 29, 2000, Testimony ApApenednxidexiIs

For Release on Delivery

Expected at

10 a.m. EST

Tuesday,

February 29, 2000

GAO/T-AIMD-00-113

SBA LOAN MONITORING

SYSTEM

Substantial Progress Yet

Key Risks and Challenges

Remain

Statement of Joel C. Willemssen

Director, Civil Agencies Information Systems

Accounting and Information Management Division

Testimony

Before the Subcommittee on Government Programs and

Oversight, Committee on Small Business, House of

Representatives

United States General Accounting Office

GAO
Appendix I

GAO’s February 29, 2000, Testimony

Page 9 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 1 GAO/T-AIMD-00-113

Mr. Chairman and Members of the Subcommittee:

Thank you for inviting us to discuss the progress of the Small Business

Administration (SBA) in performing the planning actions for its loan

monitoring system, as mandated by the Small Business Reauthorization

Act of 1997. After providing brief background information, my testimony

today will discuss SBA’s progress in completing the mandated actions, our

evaluation of SBA’s products completed thus far, the processes used to

develop these products and manage key activities, and actions the agency

needs to take to manage risks.

SBA has made substantial progress in completing the eight mandated

planning actions, but must still complete work for some actions and

implement key functions to effectively manage the development of the

loan monitoring system. SBA has benchmarked its business processes

against those of leading organizations and has conducted a reengineering

study to identify and select new processes to improve its operations. Using

the results of these efforts, SBA has also started identifying the data

needed for the proposed loan monitoring system, defining data quality

standards, developing the information architecture, determining an

acquisition strategy, defining systems requirements, and estimating the

costs to complete the project. SBA has reported that all of the eight

mandated planning actions are complete, except for two concerning the

information architecture and systems requirements.

Our analyses of SBA products for the planning actions have shown that

the agency has made substantial progress. At the same time, some of the

products lack one or more important elements, and there are critical steps

that SBA has not performed. Several key functions—such as configuration

management, quality assurance, and system security—need to be

established and implemented to effectively manage the project.

Before beginning systems design and development, SBA will need to

complete key planning actions—such as performing benefit and cost

analyses of business process and system alternatives—for the mandated

planning actions. It should also implement critical project management

controls—such as those needed to ensure that system design addresses

the security challenge posed by Internet-based access. Actions will be

needed in such areas as these if SBA is to effectively manage the risks it

will encounter in the systems development process.

In commenting on a draft of this testimony, the deputy administrator and

other SBA officials told us that they recognize the benefit of the actions we

suggest to improve project management. However, they said, the risks
Appendix I

GAO’s February 29, 2000, Testimony

Page 10 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 2 GAO/T-AIMD-00-113

from not fully completing such actions before system development should

be weighed against the risks and opportunity costs associated with

delaying the implementation of a system that would help oversee SBA’s

guaranteed loan portfolio. They added that the first system increment they

plan to develop will assist them in further defining the requirements for

the entire system, and therefore they need to proceed with it

expeditiously.

SBA’s need to monitor the activities of lenders who help deliver its

programs has increased significantly in recent years. Annual loan

approvals for the 7(a) General Business Loan Guarantee Program and the

section 504 Certified Development Company Debenture Program have

almost doubled since 1992, and the loan portfolio for all its programs now

exceeds $40 billion. During that same time, SBA has decreased its staff by

20 percent and shifted to lenders the responsibility for key loan

origination, servicing, and liquidation functions. Lenders now originate

about 75 percent of new loans with little or no involvement by SBA in the

eligibility and credit approval processes.

To enhance its capabilities for loan and lender monitoring, SBA has

proposed improvements to its automated systems, lender oversight, and

risk management infrastructure. The purpose of SBA’s proposed loan

monitoring system is to use technology and new processes to manage its

loan portfolios, identify and effectively mitigate risks incurred through

loans guaranteed by SBA, implement oversight of internal and external

operations, and calculate subsidy rates.

After reviewing SBA’s basis for this request, we reported in June 1997 that

the agency had not undertaken the essential planning needed to develop

the proposed loan monitoring system.1 The Congress subsequently

enacted provisions in the Small Business Reauthorization Act of 1997 that

required the SBA Administrator to perform and complete eight planning

actions to serve as the basis for funding the development and

implementation of the computerized loan monitoring system. The act also

required SBA to report by June 2, 1998, on its progress in completing the

planning actions.

1Small Business Administration: Better Planning and Controls Needed For Information Systems (GAO/

AIMD-97-94, June 27, 1997).

Background
Appendix I

GAO’s February 29, 2000, Testimony

Page 11 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 3 GAO/T-AIMD-00-113

As required by the act, in June 1998 we commented on SBA’s report.2 We

reported that while SBA had formed a team for the loan monitoring system

in December 1997, it had not yet completed any of the eight mandated

actions. SBA’s report included a project plan, laying out its approach for

addressing these actions. Work on the first of the required planning

actions was begun in May 1998 and, according to the project plan, SBA

was to complete work on the last of the eight mandated actions in August

1999.

In July 1998 we testified that SBA’s project plan delineated the project’s

goals and objectives, resource requirements, quality standards and control

systems, assumptions, methodologies, work breakdown structure with

timetable for completion of tasks, and estimated costs.3 The plan

estimated that a staff of 18 would be needed for the first phase of the

project, which was to address the eight mandated planning actions, and

scheduled completion of the mandated actions by the end of August 1999.

While development of the project plan was a good start, we also testified

that SBA faced formidable technical and management challenges and risks

in executing the plan, including

• establishing software project management capability while undertaking its

largest information technology project ever;

• using methodologies and practices for the first time while conducting a

large, complex project; and

• implementing the loan monitoring system without having an information

technology architecture in place.

SBA acknowledged these challenges and committed to providing the loan

monitoring system project with the necessary management support.

To perform the planning for the loan monitoring system and conduct

related modernization activities, SBA was appropriated $8 million annually

for fiscal years 1998, 1999, and 2000. According to loan monitoring system

project data, SBA used about $1 million in fiscal year 1998 and $0.7 million

in fiscal year 1999. For fiscal year 2000, SBA plans to use about $2 million

2The act required us to evaluate and report on SBA’s compliance within 28 days of receipt of SBA’s

report. Accordingly, we issued Small Business Administration: Mandated Planning for Loan Monitoring

System Is Not Complete (GAO/AIMD-98-214R, June 30, 1998).

3Small Business Administration: Planning for Loan Monitoring System Has Many Positive Features But

Still Carries Implementation Challenges (GAO/T-AIMD-98-233, July 16, 1998).
Appendix I

GAO’s February 29, 2000, Testimony

Page 12 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 4 GAO/T-AIMD-00-113

for contractor project support, SBA staff, and travel costs; and about $8.5

million for infrastructure acquisition and system development activities.

As you requested, our overall objective was to evaluate SBA’s efforts to

complete actions required by the Small Business Reauthorization Act of

1997 in accordance with required and generally accepted systems

development practices.

We conducted our work at SBA’s headquarters in Washington, D.C., from

August 1999 through February 2000, in accordance with generally

accepted government auditing standards. In our analyses of SBA’s

products for the mandated actions, we used the methodologies and

criteria that SBA officials said they used in performing their work and

preparing the products, as well as guidance issued by the Office of

Management and Budget, the General Services Administration, the

Institute of Electrical and Electronics Engineers, Inc. (IEEE), and our

office that are applicable to the mandated planning actions. Our analyses

of the products were performed to assess the structure, general content,

and processes used in the planning actions. Detailed analyses could not be

performed on all SBA products because many were only recently provided

to us and time constraints precluded an opportunity to discuss these

products and the processes used to produce them with cognizant SBA

officials. We provided a copy of our draft testimony to SBA officials; we

received comments from them, and made changes as appropriate.

SBA has completed 17 products for the eight mandated actions and has

prepared 3 additional draft products. Table 1 summarizes the status of

SBA’s products for the eight mandated actions.

Objective, Scope, and

Methodology

SBA Has Made

Substantial Progress

in Completing

Mandated Actions
Appendix I

GAO’s February 29, 2000, Testimony

Page 13 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 5 GAO/T-AIMD-00-113

Table 1: Status of Products for Mandated Actions as Reported by SBA, as

of February 23, 2000

Mandated Action SBA Product

Status of

Product

Benchmark study Final

Business process reengineering

study Final

Feasibility analysis of

recommendations Final

Analysis of “as is” baseline cost

and time Final

Concept of operations Final

Benchmark loan monitoring business

processes and systems against

comparable industry processes and, if

appropriate, simplify or redefine work

processes based on these benchmarks

Implementation strategy Final

Analyze the benefits and costs of

alternatives and use them to

demonstrate the advantages of final

project Business case for reengineering Final

“As is” information technology

architecture

Final

Technology policy statement Final

Enterprise information technology

architecture report Draft

Gap analysis, migration strategy,

and transition plan Draft

Ensure that proposed information

system is consistent with agency’s

information architecture

Plan to synchronize loan

monitoring system with information

technology architecture Final

Identify all data inputs and outputs

necessary for timely report generation

Needs statement (logical data

model) Final

Data quality guidance Final

Data quality issues Final

Determine data quality standards and

control systems for ensuring information

accuracy Data quality management plan Final

Statement of need for the loan

monitoring system Final

Fully define the requirements for the

system that uses on-line, automated

capabilities to the extent feasible Systems requirements Draft

Identify acquisition strategy and work

increments to completion Acquisition strategy Final

Estimate cost to system completion,

identifying essential cost elements

Needs statement (total cost to

completion) Final

SBA officials advised us on February 23 that they expected to complete

the remaining actions by March 2000 and then proceed to design and

develop the first increment of the proposed system.
Appendix I

GAO’s February 29, 2000, Testimony

Page 14 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 6 GAO/T-AIMD-00-113

Our analyses of SBA products for the planning actions reveal that the

agency has made substantial progress. At the same time, SBA had not

completed some critical steps in preparing the products, and key products

did not contain one or more important elements. For many of these

missing or incomplete items, SBA plans to contract for their completion.

SBA conducted a benchmark study and reported its results in December

1998. Benchmarking is the comparison of core process performance with

other components of the agency or organization (internal benchmarking)

and/or with leading agencies or organizations (external benchmarking).

Best practices include the processes, practices, and systems that perform

exceptionally well in specific areas of public and private organizations.

Benchmarking provides a means of establishing a compelling business

case for change. It should identify more efficient and effective processes

for achieving intended results, and suggest goals for program output,

product and service quality, and process improvement.

SBA’s contractor used a seven-step benchmarking process to evaluate SBA

business gaps with similar organizations for five loan management

functions. The functions benchmarked were risk management, lender

oversight, guaranty procedures, subsidy rate calculation, and asset sales.

These functions were benchmarked against the practices of 11 federal and

private-sector organizations.

The benchmark report identified standard industry or “good” practices and

showed a significant gap between SBA and benchmark partners’ practices

for each of the management functions. The report also contained

suggestions that senior management needed to “buy in” to the

reengineering process, communications plans needed to be developed,

systems requirements needed to be preliminarily defined, and training

plans needed to be examined.

In evaluating SBA’s benchmark effort, in May 19994 we reported that it was

an important first step in SBA’s actions to develop a loan monitoring

system. In general, the benchmarking methods used were consistent with

accepted practices and the benchmarking methodology was followed at a

high level. However, the study had a number of weaknesses, the most

significant being that it did not produce cost and performance

measurement data for SBA and the benchmark partners’ processes. SBA

agreed with our analysis and stated that it planned to collect additional

4Small Business Administration: Enhancements Needed for Loan Monitoring System Benchmark Study

(GAO/AIMD-99-165, May 14, 1999).

Actions Still Needed

for Key Items

Benchmarking and

Business Process

Reengineering Are

Complete Except for Costs

and Performance

Measurement Data,

Analysis of Alternatives,

and Implementation Plans
Appendix I

GAO’s February 29, 2000, Testimony

Page 15 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 7 GAO/T-AIMD-00-113

benchmarking information during its business process reengineering

activities.

Because the benchmark study identified wide gaps between SBA’s

business processes and the best practices of the benchmark partners’

practices for each of the management functions, SBA decided to pursue

business process reengineering for each of the five SBA areas that were

included in the study. Business process reengineering is an approach for

redesigning the way work is done to better support the organization’s

mission and reduce costs. Reengineering identifies, analyzes, and

redesigns an organization’s core business processes with the aim of

achieving dramatic improvement in critical performance measures such as

cost, quality, service, and speed.

The purpose of SBA’s business process reengineering (BPR) study was to

analyze the current business practices within five functional areas of the

organization, and develop new, more effective processes, supported by

modernized, state of the art, information technology systems.

The five functional areas addressed in the study were the following:

Guaranty Procedures encompass the full life cycle of a loan, from

application through payment in full or liquidation, with three major

subprocesses:

• Processing: encompasses application, approval, and closing

• Servicing: includes all loan actions handled through payment in full

• Liquidation: includes the process of recovering value from defaulted loans

Lender Oversight is composed of three main functions:

(1) communicating to lenders about policies, procedures, and standards of

performance; (2) monitoring of lender performance; and (3) taking

enforcement action when lender behavior and/or performance deviate

from accepted standards.

Risk Management is the process by which SBA monitors its loan

portfolio, tracks lenders and borrows and oversees the management of the

portfolio to keep losses to an acceptable level.

Subsidy Rate is an estimate of the subsidy cost of SBA’s guaranteed and

direct loan programs as a percentage of the total level of commitment.
Appendix I

GAO’s February 29, 2000, Testimony

Page 16 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 8 GAO/T-AIMD-00-113

Asset Sales is composed of the processes used to sell SBA loan assets,

including direct loans and repurchased guaranteed loans, to private

investors.

The SBA BPR team, with facilitation support from contractor staff,

analyzed the best practices of the industry and made recommendations for

SBA’s systems modernization primarily from the standpoint of maximizing

efficiency with the highest degree of automation. As a result of this

analysis, the BPR report contained 38 recommendations for new elements

or characteristics for SBA’s business processes. A few of the more

significant recommendations were that SBA’s new business processes

include

• one set of core data elements for all loan programs, and one standard

electronic channel for submitting all applications;

• centralized processing of all guaranty applications;

• lenders’ ability to directly access the SBA system to submit a servicing

action request or report a unilateral action;

• lenders, direct borrowers, and designated SBA personnel being able to

view the real-time status of all loans;

• liquidations being centralized in the servicing centers to achieve

economies of scale in labor and technology;

• continuous capturing of lender performance information and electronic

analysis for early warning of potential changes in lender performance; and

• performance information collected through the new lender monitoring

system and lender reviews providing the necessary base of information to

facilitate informed decisionmaking.

To decide which of the recommendations would be adopted in whole or in

part, SBA formed a team that analyzed the risks and barriers associated

with their implementation. Based on this analysis, the SBA Administrator

fully adopted 30 of the 38 recommendations and adopted the remaining

eight with modifications. For example, the recommendation to centralize

the processing of all applications was modified to centralize the

processing for programs that represent about 75 percent of all guaranteed

loans.

SBA’s contractor followed a methodology that conforms with generally

accepted practices. However, as acknowledged in the report, key cost and
Appendix I

GAO’s February 29, 2000, Testimony

Page 17 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 9 GAO/T-AIMD-00-113

performance measurement data—needed to compare and analyze

proposed processes against current—were not collected during the BPR

study. According to generally accepted practices, a performance-based

and risk-adjusted benefit and cost analysis of alternatives being

considered for each business process is needed to support the final

selection of processes to implement.5 Accordingly, the BPR report

recommended that SBA perform an activity-based cost analysis to provide

critical data in evaluating current practices. SBA officials subsequently

told us that they would produce a business case that would support their

selection of new business processes. SBA did prepare this business case

but it did not include benefit and cost analyses of alternatives being

considered for each business process. Without analyzing benefits and

costs, SBA increases the risk that the most effective and efficient business

processes will not be selected.

SBA also has not yet developed an implementation plan for the new

business processes as required by generally accepted BPR practices. It

developed an overall strategy for implementing the new business

processes, but did not develop a detailed plan that lays out the critical

elements and milestones for implementing them. SBA should consider

formulating such a plan before it starts developing the first segment of the

new system to ensure that the development and implementation of

supporting information systems will be synchronized with the

implementation of new business processes.

Based on the results of its BPR study, SBA has developed a general

description of the new loan monitoring system. The system is to be used

by program managers and staff in headquarters, loan processing and

service centers, field offices, financial operations, lenders, and external

service providers under contract to perform specific portfolio support

tasks. The system is expected to be “on-line to all users around the clock.”

Internally, SBA staff are to have all necessary data available through a loan

system that provides access to records from anywhere in the agency, while

externally the system is expected to allow lenders to view their own

portfolios.

SBA plans to have the loan monitoring system include a “virtual private

network” using high-speed communications based on the Internet and dialup access for smaller lenders, a security system that requires prior

identification and approval of users, and high-level encryption of all

messages. Because the Internet is a public network, SBA states that it will

5The generally accepted BPR practices cited here are taken from our Business Process Reengineering

Assessment Guide (GAO/AIMD-10.1.15, April 1997).

Loan Monitoring System Is

Intended to Provide Electronic

Data Collection and Ready

Access to a Comprehensive

Data Repository
Appendix I

GAO’s February 29, 2000, Testimony

Page 18 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 10 GAO/T-AIMD-00-113

require authentication of lenders and SBA staff as they try to initiate

access to the system.

In addition to the network, the system is expected to integrate a secure

Web site and a technologically advanced system of data, applications, and

processes. Requests for loan guarantees are to be submitted electronically,

either through a formatted file transfer or on-line entry into the system.

The loan monitoring system is also expected to have a comprehensive

central data repository to support early warning systems, exception

reporting, management reporting, decision support, ad hoc reporting,

operational reporting, and financial management reconciliation. The

central data repository is also considered to be the key to providing early

warning systems for lender oversight and risk management functions.

SBA has decided to design and develop the loan monitoring system in

increments. According to the agency, the first increment will include the

establishment of a standard set of data elements for loan guarantee

applications and the electronic processing of applications for part of its

loan guarantee programs. However, SBA has not yet provided us with key

documents related to this, such as a description of the system design,

documentation on the make or buy decision, proposed acceptance criteria

for contract deliverables, and project plans.

A benefit-cost analysis is a generally accepted method for comparison of

alternative means of meeting a specific objective. In its simplest form,

benefit-cost analyses should identify alternatives, determine the benefits

and costs of each alternative, and recommend the most cost-effective

alternative.

SBA’s business case analysis6 describes the current system, discusses

proposed system changes, identifies alternatives for the proposed loan

monitoring system, and presents a benefit-cost analysis showing that the

benefits associated with the new loan monitoring system are greater that

the increases in costs for investment, maintenance, ongoing operations,

and related items. SBA estimated that the new system would produce, by

the end of fiscal year 2006, cumulative cash savings of $147 million. In its

analysis, SBA considered—but dismissed—alternatives such as the

privatization and outsourcing of loan monitoring functions to the private

sector, noting that “SBA already has accomplished most of what can be

done in terms of privatization.”

6The Business Case for the Reengineered Loan Monitoring System, LMS.V1.1.006, February 2000.

Benefit-Cost Analyses

Have Not Yet Been

Performed
Appendix I

GAO’s February 29, 2000, Testimony

Page 19 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 11 GAO/T-AIMD-00-113

SBA considered five system alternatives. It concluded that two of the

alternatives—continued use of the current system and making

improvements to the current system—were not viable because the current

system is completely outmoded in both functionality and technical design.

SBA noted that the remaining three alternatives—using standard

commercial-off-the-shelf (COTS) software, standard COTS software with

custom-made software, and custom-made software alone—will be

analyzed at a later point.

Before beginning system design, SBA should perform benefit-cost analyses

of all identified alternatives, determine the benefits and costs of each

alternative, evaluate alternatives by comparing their benefits and costs,

and select the best alternative for implementation. This will increase the

probability that SBA will obtain a system that meets its needs at the lowest

cost.

An information technology architecture is a blueprint—consisting of

logical and technical components—to guide and constrain the

development and evolution of a collection of related systems. At the

logical level, the architecture provides a high-level description of an

organization’s mission, the business functions being performed and the

relationships among the functions, the information needed to perform the

functions, and the flow of information among functions. At the technical

level, the architecture provides the rules and standards needed to ensure

that the interrelated systems are built to be interoperable and

maintainable. These include specifications of critical aspects of

component systems’ hardware, software, communication, data, security,

and performance characteristics.

SBA has analyzed and documented7 its existing architecture, defined the

future—or target—architecture,8 and analyzed the gaps between the two.

The gap analysis9 forms the basis for development of a migration strategy

to move from the current systems to the new system. However, SBA has

not fully documented the current systems in the existing architecture, and

has not completed its target information technology architecture.

7SBA Existing Information Technology Review, Capability, and Cost Analysis, Version 1.3,

November 9, 1998.

8SBA Information Technology Architecture (draft), Final Version 1.0, October 1999.

9SBA Target ITA Gap Analysis Assessment and Action/Migration Plan (undated draft).

SBA’s Information

Architecture Is Incomplete
Appendix I

GAO’s February 29, 2000, Testimony

Page 20 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 12 GAO/T-AIMD-00-113

To deal with the incomplete architecture, SBA has developed an

approach10 to maintaining consistency between the SBA information

technology architecture and proposed loan monitoring system. This

approach—which requires that the system under construction be mapped

to the partially defined target architecture—increases the risk that the loan

monitoring system would not be seamlessly integrated with the SBA target

architecture. To address this increased risk, before beginning system

design SBA should consider developing and including the rules and

standards needed to ensure that the interrelated systems are built to be

interoperable and maintainable in its information technology architecture.

These include specifications of critical aspects of component systems’

hardware, software, communication, data, security, and performance

characteristics.

Requirements definition begins with high-level requirements and ends with

detailed requirements prior to system design. High-level requirements for

reports need to be consistent, and validated to a level of detail sufficient

for defining acceptance criteria. System outputs, which consist of reports

and files to satisfy the organization’s information needs, must be precisely

defined prior to design. The inputs needed to produce these specific files,

and reports are then defined in detail.

SBA identified a sample of current reports, including reports now being

produced by local systems and several reports being produced by the

mainframe system, but has not identified high-level requirements for all

internal reports. Before initiating system design for each increment, SBA

should consider identifying the high-level requirements for all internal

reports. In addition, it should define detailed input and output data

elements necessary for the timely generation of reports.

Plans for improving data quality should include the definition of data

quality standards, the development of quality measures, and the

assessment impact of inaccurate data on business units. The plan for

improving data quality should also have a schedule of activities and

resources that are identified and allocated.

SBA developed a data quality plan and a conceptual data model that

includes data quality information. The data quality plan is a strategy paper

and, as such, provides a framework for pursuing data quality goals and

10SBA Plan to Reconcile On-going Loan Monitoring System Development with In-process Information

Technology Architecture Development, September 1999.

Data Inputs and Outputs

Are Necessary for Timely

Report Generation

Data Quality Standards

Still Need Schedule,

Resource Allocation, and

Business Process

Measures
Appendix I

GAO’s February 29, 2000, Testimony

Page 21 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 13 GAO/T-AIMD-00-113

contains guidelines for developing and maintaining data quality. For

example, it discusses data migration actions to cleanse data in current

systems. However, it does not identify the business priorities with respect

to near-term and long-term requirements for data quality improvement, or

provide a schedule of planned actions to improve data quality.

Before beginning design, SBA should consider completing the definition of

specific data quality standards, developing a schedule of planned actions

to improve data quality in the current systems, and implementing data

quality measures for the new loan monitoring system.

In general, industry and federal practices for systems requirements include

system life expectancy; functions and capabilities of the system; business,

organizational, and user requirements; computer resource requirements;

design constraints; security requirements; operations and maintenance

requirements; human-factors engineering requirements; and compliance

standards and procedures. The systems requirements should be

documented and specify the methods used to ensure that each

requirement has been met. It is essential for a requirement to have

characteristics that can be verified and assessed throughout the system

development life cycle. Methods of verification include inspection, testing,

demonstration, and analysis. All practices involving the creation, changing,

or verification of requirements must maintain requirements traceability.11

SBA has drafted a systems requirements document that defines

requirements for each function in the loan monitoring system, crossreferences data bases to loan monitoring system business processes,

identifies some of the reports by user and purpose, and includes sections

required by the SBA methodology. However, some areas of systems

requirements are not complete. For example, the systems requirements

document does not specify capacity and performance requirements.

Accordingly, before proceeding with system development, SBA should

define its system capacity and performance requirements.

An acquisition strategy allocates risk between the government and

contractor, effectively uses competition, ties contract payments to

accomplishments, and takes maximum advantage of commercial

technology. SBA’s acquisition strategy explains the approach for

developing the loan monitoring system, identifies project increments, and

11A requirement is traceable if its origin is clear and if it facilitates the referencing of each requirement

in future development.

Some System

Requirements Are Yet to

Be Defined

Acquisition Strategy

Awaits Final Selection of

Implementation

Alternative
Appendix I

GAO’s February 29, 2000, Testimony

Page 22 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 14 GAO/T-AIMD-00-113

establishes a risk management approach. In line with the model just

described, it allocates risk between the government and the contractor,

uses competition, ties contract payments to accomplishments, and takes

maximum advantage of commercial technology.

In addition, SBA has taken action to mitigate acquisition risk, by selecting

an incremental approach to systems development. Under this approach,

cost and schedule risks will be managed by revisiting cost, schedule, and

project objectives after the first increment. For each business function,

SBA has identified whether automated solutions are available from

vendors or government sources, whether business functions can be

outsourced, and whether business functions can be developed as

customized applications. However, in documenting this information, SBA

does not identify sources for each approach, nor does it explain why it

believes that 40 percent of the functions must be custom-developed and

therefore cannot be outsourced or purchased. Because the risk and cost

generally increase as the proportion of customized components increases,

it is important that SBA have a sound, justified basis explaining its

rationale for this.

SBA has substantially complied with this requirement. In documenting its

compliance with the requirement to estimate the cost to completion, SBA

provided us with its cost-to-completion document. The cost estimate was

derived from its business case analysis.

Cost to Completion

Substantially Set
Appendix I

GAO’s February 29, 2000, Testimony

Page 23 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 15 GAO/T-AIMD-00-113

Table 2: Loan Monitoring System Estimated Cost to Completion ($000)

Cost Element FY 1998 FY 1999 FY 2000 FY 2001 FY 2002 Total

Project startup $ 375 $ 300 $ 675

Initiate project 150 $650 1,160 $ 950 $ 600 3,510

Definition 325 50 150 525

System design 150 350 500

Build system:

Data scrub, Integrator,

IV&V 1,150 1,200 650 3,000

Iteration 1 843 843

Iteration 2 2,468 800 250 3,518

Iteration 3 93 675 768

Iteration 4 928 1,930 200 3,058

Iteration 5 218 970 1,188

Iteration 6 898 1,530 700 3,128

Data migration 950 1,150 200 2,300

Infrastructure 950 1,900 1,100 3,950

Evaluate 84 120 120 324

Operate 50 150 200 400

Total cost to completion $1,000 $700 $10,592 $11,375 $4,020 $27,687

Source: The SBA Loan Monitoring System Estimated Cost to Completion, Project Plan

(Attachment) LMS.V1.0.004, February 14, 2000.

The total $27.7 million estimate includes about $4 million for

infrastructure, $9.3 million for software, $8.8 million for services,

$2.5 million for support, $3 million for internal labor, and $0.3 million for

other costs.12 SBA has awarded a contract to refine the cost-to-completion

estimate, including costs of work increments.

SBA should continue to refine its cost-to-completion estimate following

the completion of the benefit-cost analysis of alternatives and the selection

of the best alternative for implementation. In addition, once SBA

determines the life expectancy of the loan monitoring system, it also

should develop and maintain a lifecycle cost estimate for the system and

its components.

12Numbers do not add due to rounding.
Appendix I

GAO’s February 29, 2000, Testimony

Page 24 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 16 GAO/T-AIMD-00-113

As stated earlier, we testified in July 1998 that SBA faced formidable

technical and management challenges and risks in executing the plan,

including establishing software project management capability, using

methodologies and practices, and implementing the loan monitoring

system project without having an information technology architecture in

place. In following up on SBA’s actions to deal with these challenges, we

found that the agency is just beginning to adopt and implement basic

policies, processes, and tools needed to control and manage a major

system development and acquisition project. These include project

tracking and oversight, configuration management, quality assurance, and

security requirements.

Project tracking and oversight involves tracking and reviewing project

accomplishments and results against documented estimates, schedules,

and performance plans. A documented plan for the project is used as the

basis for tracking systems development activities. Actual results and

performance are tracked against the project schedule, and the project

follows organizational policy for managing systems development.

SBA has adopted an agencywide systems development methodology that

suggests that projects should use this type of project tracking and review.

According to SBA officials, they intend to use this for the loan monitoring

system.

Configuration management plans, policies, and procedures are a set of

management controls over the composition of and changes to computer

and network systems components and documentation, including software

code documentation and project planning documents. Configuration

management is essential to successfully managing complex information

systems and ensuring their integrity throughout their life cycles.

SBA has started to formulate and implement configuration management

plans, policies, and processes for the loan monitoring system project.

Finalizing these will provide SBA with further assurance of the success of

the project.

Quality assurance involves reviewing and auditing systems development

activities to verify that they comply with applicable procedures and

standards. A software quality assurance group reviews project activities

and audits software work products throughout the life cycle, and provides

management with visibility as to whether the software project is adhering

to its established plans, standards, and procedures. Compliance issues are

Key Management

Controls and

Processes Need to Be

Implemented

Project Tracking and

Oversight Is Planned

Configuration Management

Plans, Policies, and

Procedures Need to be

Finalized

Quality Assurance

Activities Are Planned
Appendix I

GAO’s February 29, 2000, Testimony

Page 25 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 17 GAO/T-AIMD-00-113

first addressed and resolved on the project level. For issues not resolvable

within the project, the quality assurance group elevates the issue to an

appropriate level of management for resolution.

SBA has not yet established a quality assurance process to ensure that the

loan monitoring system project and its activities comply with SBA policies,

procedures, and systems development methodologies. However, it is

planning to establish a technical review group whose purpose will be to

review loan monitoring system project adherence to SBA standards

outlined in the SBA systems development methodology. In addition, SBA

is planning to contract for independent verification and validation to

provide oversight of its systems development efforts.

Security focuses on the ability to ensure the confidentiality, integrity, and

availability of stored and processed data. Unsecured or poorly secured

systems are highly vulnerable to external and internal attacks and

unauthorized use. Security planning includes the identification of highlevel security requirements, including mission, management, and technical

security requirements; functional security requirements that cover users’

security and privacy needs; data-sensitivity analysis to identify data

requiring special protection; and a security architecture that describes the

security controls and relationships among the various system components.

While SBA’s proposed Internet-based virtual private network may reduce

telecommunications costs and provide easy nationwide access to loan

monitoring system, the reliance on the Internet as a key component of the

system’s architecture brings unique security challenges that must be

addressed early in the project’s life. However, SBA has not yet developed a

security architecture for its target environment, updated its security

operating procedures, or defined security and privacy requirements for the

loan monitoring system.13 Because security is a critical feature for the loan

monitoring system, SBA should complete its security architecture and

update its security operating procedures before it begins the design and

development phase of the loan monitoring system.

SBA has made substantial progress in planning for the loan monitoring

system; however, the agency needs to take additional actions to manage

the project’s risks. The issues I have outlined today pose considerable

challenge, both in the area of timely completion of key planning activities

and in the strengthening of project management processes and controls.

13Automated Information Security Program (undated), SOP 90-47.

Loan Monitoring System

Security and Privacy

Requirements Are Not

Fully Defined

Summary of Actions

Needed
Appendix I

GAO’s February 29, 2000, Testimony

Page 26 GAO/AIMD-00-124 SBA Loan Monitoring System

Page 18 GAO/T-AIMD-00-113

In the area of planning actions, SBA should consider taking the following

actions: completing the analyses of benefits and costs for alternative

business processes identified through SBA’s business reengineering effort;

performing benefit-cost analyses for systems alternatives; completing the

part of its information architecture that specifies the rules and standards

for interoperability and maintainability of interrelated systems; identifying

requirements and data elements for reports; completing the definition of

specific data quality standards; ensuring that systems requirements

document include capacity and performance requirements; ensuring that

sound justification exists for pursuing custom-developed functions; and

estimating the cost to completion that are based on an analysis of the

benefits and costs of system alternatives.

In the project management area, SBA should strengthen its project

management process and controls. These include putting in place project

tracking and oversight capabilities; implementing configuration

management processes; acquiring independent verification and validation

for the loan monitoring system project and establishing an internal quality

assurance function; and addressing the security challenge posed by

Internet-based access to loan monitoring system functions and data. These

processes and capabilities are essential to a major systems development

and acquisition.

Mr. Chairman, this concludes my statement. I would be pleased to respond

to any questions that you or other members of the Subcommittee may

have at this time.

For information about this testimony, please contact Joel C. Willemssen at

(202) 512-6253 or by e-mail at willemssenj.aimd@gao.gov. Individuals

making key contributions to this testimony include Mirko J. Dolak,

James R. Hamilton, E. Randolph Tekeley, Hai V. Tran, and Daniel K.

Wexler.

(511761)

Contact and

Acknowledgments
Page 27 GAO/AIMD-00-124 SBA Loan Monitoring System

Appendix II

Comments From the Small Business 

Administration ApendxiI

Note: GAO comment 

supplementing those in the 

report text appears at the 

end of this appendix.
Appendix II

Comments From the Small Business 

Administration

Page 28 GAO/AIMD-00-124 SBA Loan Monitoring System
Appendix II

Comments From the Small Business 

Administration

Page 29 GAO/AIMD-00-124 SBA Loan Monitoring System

See comment.
Appendix II

Comments From the Small Business 

Administration

Page 30 GAO/AIMD-00-124 SBA Loan Monitoring System

See comment.

See comment.
Appendix II

Comments From the Small Business 

Administration

Page 31 GAO/AIMD-00-124 SBA Loan Monitoring System
Appendix II

Comments From the Small Business 

Administration

Page 32 GAO/AIMD-00-124 SBA Loan Monitoring System
Appendix II

Comments From the Small Business 

Administration

Page 33 GAO/AIMD-00-124 SBA Loan Monitoring System
Appendix II

Comments From the Small Business 

Administration

Page 34 GAO/AIMD-00-124 SBA Loan Monitoring System

The following is GAO’s comment on the Small Business Administration’s 

March 29, 2000, letter.

GAO Comment Our testimony considered the products cited by SBA. The testimony, 

included in appendix I to this report, lists status of products for mandated 

actions that were analyzed as part of our evaluation of SBA’s planning 

actions for the loan monitoring system.
Page 35 GAO/AIMD-00-124 SBA Loan Monitoring System

Related GAO Products

Small Business Administration: Enhancements Needed for Loan 

Monitoring System Benchmark Study (GAO/AIMD-99-165, May 14, 1999).

Small Business Administration: Planning for Loan Monitoring System Has 

Many Positive Features But Still Carries Implementation Challenges

(GAO/T-AIMD-98-233, July 16, 1998).

Small Business Administration: Mandated Planning For Loan Monitoring 

System Is Not Complete (GAO/AIMD-98-214R, June 30, 1998).

Small Business Administration: Better Planning and Controls Needed For 

Information Systems (GAO/AIMD-97-94, June 27, 1997).

(511841) Letr
Ordering Information The first copy of each GAO report is free. Additional copies of 

reports are $2 each. A check or money order should be made out to 

the Superintendent of Documents. VISA and MasterCard credit 

cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and 

testimony. To receive facsimile copies of the daily list or any list 

from the past 30 days, please call (202) 512-6000 using a touchtone 

phone. A recorded menu will provide information on how to obtain 

these lists.

Orders by Internet:

For information on how to access GAO reports on the Internet, 

send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web home page at: 

http://www.gao.gov

To Report Fraud, 

Waste, or Abuse in 

Federal Programs

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• e-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. GI00