ADEQUACY OF QUALITY ASSURANCE 


OVERSIGHT OF THE LOAN MANAGEMENT 


AND ACCOUNTING SYSTEM PROJECT 


Report Number: 10-14 


Date Issued: September 13,2010 


Prepared by the 


Office of Inspector General 


U. S. Small Business Administration 

u.s. 

Office 

Small 

Inspector 

Business 

General 

Administration Memorandum

Date: To: 	 David B. Robbins September 13, 2010 

Director, LMAS Modernization Program 

Paul T. Christy 


Acting Chief Information Officer 


lsi Original Signed 


From: 	 Debra S. Ritt 


Assistant Inspector General for Auditing 


Subject: 	 Report on the Adequacy of Quality Assurance Oversight of the Loan Management and 

Accounting System Project 

Report No. 10-14 

This report presents the results of our audit ofthe Small Business Administration's 

(SBA) Quality Assurance (QA) Oversight of the Loan Management and 

Accounting System (LMAS) Project. To guide QA activities, SBA developed a 

QA Plan identifying steps that it will take to evaluate overall performance of the 

LMAS project on a regular basis to ensure that it conforms to the Agency's quality 

standards and satisfies user needs. A major component of a QA plan is the 

Independent Verification and Validation (IV & V) of development products to 

ensure that design specifications are correctly implemented and that deliverables 

accomplish their intended purpose. SBA contracted with TestPros to develop and 

implement the QA plan, including conducting all IV & V activities. 

The objectives of the audit were to determine the: (1) adequacy of SBA's QA 

Plan; (2) extent to which the QA Plan was being implemented; and (3) quality of 

IV& V activities. The audit was performed in response to language in the OIG's 

Fiscal Year (FY) 2010 appropriation, which directed the OIG to provide routine 

analysis and reporting on SBA' s modernization of its loan management and 

accounting systems. 

The OIG has issued three reports on the LMAS project since it was initiated in 

2005 and, due to umesolved project risks, designated oversight of the LMAS 

project as a new management challenge in FY 2010. 1 Two ofthe most recent 

1 Management Challenge 9 - SBA Needs to Modernize its Loan Accounting System and Migrate it offthe Mainjinme. 
2 

reports2 found that the Agency had not established either an effective enterprisewide or project-level QA function to ensure that LMAS project deliverables meet 

SBA requirements and quality standards. The most recent audit found that the 

project-level QA process was not independent from project management staff and 

that the project lacked a defined process for accepting deliverables. 

To assess the adequacy ofLMAS QA Plans, we compared the LMAS QA Plan to 

SEA's Systems Development Methodology (SDM) issued by the Office of the 

Chief Information Officer and to the Federal Acquisition Regulation. We also 

reviewed the Quality Assurance Surveillance Plans (QASP) for each project task 

order to determine whether components of the QA plan had been incorporated. 

To determine the extent to which the QA plan had been implemented, we 

reviewed all activities of TestPros, the QAlIV &V contractor, that were 

documented in project status reports and correspondence between TestPros and the 

LMAS project management staff. We also interviewed selected personnel from 

SBA's Office of the Chief Information Officer (OCIO), LMAS project 

management staff, and contractor personnel to gain an understanding of the QA 

process, staff roles and responsibilities for implementing the QA plan, and the 

project's status. Finally, to determine the adequacy of IV & V activities, we 

reviewed the solicitation and other documents related to SBA' s contract for IV & V 

and compared the activities to work performed. 

The OIG contracted with the Loch Harbour Group to assist the OIG in its audit of 

the LMAS QA process. We performed the audit work between October 2009 and 

April 2010 in accordance with Government Auditing Standards prescribed by the 

Comptroller General of the United States. 

The audit disclosed that the QA plan does not address all key areas prescribed by 

the Agency's System Development Methodology (SDM), and that the QA 

contractor has not performed all of the QA activities stipulated in its contract, 

including the performance audits of the other LMAS contractors. Further, none of 

the LMAS task orders issued to date have undergone IV & V testing. These lapses 

in QA oversight of the project have made the Agency unduly reliant on the other 

LMAS contractors to ensure the quality of their products and increased the risk 

that the project will not perform as intended. 

We recommended that SBA: revise the LMAS QA plan to incorporate all the 

components required by the enterprise-wide QA plan; take steps to hold TestPros 

2 OIG Report No. 08-13, Planningfor the Loan Management and Accounting System Modernization and Development 

Effort, May 14,2008; OIG Report No. 09-17, Review ofAllegations Concerning How the Loan Management and 

Accounting System Modernization Project is Being Managed, July 30, 2009. 
3 

accountable for performing all the activities specified in its contract; and revise 

TestPros' statement of work to include the IV & V responsibilities previously 

deleted. Further, we recommended that SBA evaluate and make necessary 

adjustments to the QA manager's workload to ensure that he can devote adequate 

time to oversee complete implementation of the enterprise QA oversight function. 

SBA management expressed concurrence with our recommendations. 

BACKGROUND 

The LMAS project is one in a series of attempts by SBA during the past several 

years to upgrade existing financial software and application modules and to 

migrate them off the mainframe environment. LMAS remained in the planning 

phase until September 2008 when SBA awarded three blanket purchase 

agreements, which included a: (1) $7.5 million award to TestPros to establish 

QAlIV&V monitoring and oversight; (2) $5 million award to Data Networks 

Corporation (DNC) for project management support; and (3) $250 million award 

to Systems Research Applications (SRA) for systems integration services. 

SBA requires a QA plan, which incorporates all Federal requirements for system 

development efforts, for all of its major IT projects. The purpose ofthe QA plan 

is to define the approach that will be used to monitor and evaluate the performance 

of the project and the quality ofthe deliverables. A major component of the QA 

process is the QASP. A QASP is developed for each task order and specifies all 

work requiring surveillance and the methode s) of surveillance that will be used. 

Another key component of the QA plan is IV & V testing to evaluate whether the 

system being developed complies with the project's specifications and meets 

performance standards. 

SBA contracted with TestPros for both the development and performance of the 

QA plan as well as the conduct of IV & V activities. As part of its IV & V 

responsibilities, TestPros is responsible for reviewing the quality of deliverables 

from both DNC and SRA. Originally, TestPros reported solely to the LMAS 

Program Manager and the Contracting Officer's Technical Representative. 

However, based on a prior IG report that disclosed the reporting arrangement did 

not ensure that QAlIV & V was independent of project management, SBA assigned 

a QA manager from the OCIO to provide independent project oversight and to 

review TestPros' work activities. While the LMAS Program Manager is 

responsible for ensuring that TestPros completes all activities identified in the 

LMAS QA plan, the QA manager is responsible for approving the QA plan and 

ultimately ensuring that the project meets the OCIO's quality standards. 

In March 2010, SBA began revising its plans for the LMAS project and elevated 

governance of the project. The LMAS Program Manager position was eliminated 
4 

and replaced with a LMAS Project Director that directly reports to SBA's Chief 

Operating Officer. Also, at the request of the Office of Management and Budget, 

the Agency is reviewing the project's roll-out strategy with a focus on providing a 

defined return on investment and an incremental approach for building the system. 

An effective QA process will be integral to SBA in meeting these goals. 

RESULTS 

The QA Plan for the LMAS Project Has Significant Weaknesses 

The LMAS QA Plan developed by TestPros, and accepted by SBA on December 

2009, does not address all of the key areas required by SBA's SDM. The SDM 

requires that project QA plans identify the types of activities that will be 

performed in key areas to ensure that all IT projects are aligned with SBA's 

enterprise architecture3 and follow a consistent development process. 

Despite this requirement, the QA plan developed by TestPros does not identify 

planned activities that would address SDM requirements for security planning, 

configuration management, 4 

user acceptance testing, risk analyses, and earned 

value management reporting. For example, the QA plan does not: 

• 	 Identify the security requirements for LMAS and controls to be 

implemented based on the level(s) of security needed. Federal regulations 

and SBA guidance require that systems be categorized as to their 

importance to the overall mission of the agency and that security controls 

be implemented based on their security categorization. 5 However, the 

LMAS QA plan does not monitor whether an adequate level of security was 

established, or needed security controls were identified. Also, a system 

security plan was not developed for the project. To date, OCIO and LMAS 

project management have focused their oversight activities on the 

certification and accreditation process, rather than initiating security 

planning. 

3 Enterprise architecture is a management practice to maximize the contribution of an agency's resources, IT 

investments, and system development activities to achieve its performance goals. 

4 Configuration management is the means by which the content, change, or status of shared information within a project 

is managed and controlled. 

5 NIST SP 800-64, Security Considerations in the System Development Lifecycle, requires that security categorization 

begin during the initiation phase of the system development life cycle, and that it identify the information that 

supports each government line of business, as defined by the enterprise architecture. It further requires that security 

controls be selected and documented in a system security plan during the acquisition/development phase ofthe system 

development life cycle. In addition, SBA's SDM requires that a system security plan be completed at the Define 

System phase of the system development life cycle. 
5 

• 	 Address whether configuration audits will be performed to ensure that the 

system successfully fulfills its requirements and that technical 

documentation is complete, consistent, and accurate, as required by SBA's 

SDM. The SDM requires that these audits be performed and any 

deviations addressed prior to putting systems into production, performing 

periodic maintenance, and placing reliance on the technical documentation. 

• 	 Explain the steps that will be taken to ensure that changes made in the 

system hardware and software throughout the development effort will be 

tested, documented, and controlled. 

• 	 Contain steps to review the Program Management Office's compliance 

with the LMAS risk management plan to ensure that it is assessing and 

managing internal and external risks throughout the project's life cycle. 

• 	 Include steps to assess the completeness and accuracy of Earned Value 

Management reporting on the project. OMB Circular A-II, Preparation, 

Submission and Execution ofthe Budget, requires that agencies submit 

Earned Value Management reports for all major investments to ensure that 

project performance is being measured and that corrective actions are 

implemented to address any cost and schedule variances. 

While a complete QA plan does not guarantee that a project will be successful, it 

does minimize a project's exposure to risk, and thus the likelihood that it will fall 

short of expectations. As a result of these gaps, management has no assurance that 

LMAS project deliverables will meet security requirements and the Agency's 

development standards. 

The LMAS QA Plan Has Not Been Fully Implemented 

The LMAS Program Manager did not ensure that TestPros performed all of the 

QA activities stipulated in its contract. For example, TestPros did not conduct any 

QA audits of the activities and work products of SRA and DNC to ensure that they 

were being implemented as specified, and that the resulting work products 

conformed to requisite standards, satisfied specified user requirements, and 

fulfilled their intended use. Further, while TestPros reviewed LMAS deliverables 

from SRA, it did not independently validate that they were created in accordance 

with SRA's standard processes or Agency requirements stipulated in the LMAS 

QA Plan. Rather, TestPros' reviews primarily addressed grammar and formatting 

of the work products and not the substance or quality of what was provided. 

According to representatives from TestPros, it was unable to fully implement the 

QA plan because SRA, the systems integration contractor, would not grant 
6 

TestPros access to its development processes. TestPros reported to SBA that it 

tried for months, but was unable to obtain the required documentation from SRA 

and DNC needed to perform the audits. The Program Manager stated that the 

delay was due, in part, to SRA replacing the project manager with an individual 

who was unfamiliar with the contract provisions that allowed TestPros access. 

However, SBA was unable to make the two contractors comply with TestPros' 

requests until April 2010. 

IV& V Activities Were Terminated 

Finally, TestPros was originally required to conduct all IV & V of critical LMAS 

development processes and deliverables. IV & V is the process of checking that a 

system meets specifications and fulfills its intended purpose, and often involves 

user acceptance testing. IV & V is especially critical for complex projects, such as 

LMAS, which is expected to encompass 165 updated or new interfaces, 46 data 

conversions, and 74 extensions of functionality. Due to these complexities, SBA 

project management officials recognized the importance of IV & V and made it a 

critical component of the Agency's testing strategy throughout the system's 

development lifecycle. For example, plans for LMAS IV&V were outlined in the 

project's Office of Management and Budget (OMB) Exhibit 3006 and in 

solicitation and contract documents for the QA/IV & V contract. 7 IV & V was also 

specified as a requirement in the QASPs for SRA's task orders. 

However, as of April 2010, none of the task orders have undergone IV & V testing 

of the results by TestPros. For example, hosting of the Joint Accounting and 

Administrative Management System was migrated from IBM facilities to SA VVIS 

without completing IV & V testing. While SBA will perform user acceptance 

testing, it may not fully assess operational requirements. Without IV & V, 

management has no independent assurance that complex interfaces and extensions 

to commercial off-the-shelf software are performing in accordance with standards. 

Further, in May 2009, SBA modified the contract with TestPros to eliminate 

IV & V activities. However, we noted that no adjustments were made to the cost of 

the contract, which was originally negotiated as a fix-priced contract. OCIO's 

Chief Technology Officer, who served as OCIO's technical advisor to the LMAS 

project, advised us that he was unaware that IV & V had been removed from the 

6 The Exhibit 300 is a capital investment report to OMB that is designed to be used as a one-stop document for many IT 

management issues, such as the business case for investment, Clinger-Cohen Act implementation, E-Gov Act 

implementation, Government Paperwork Elimination Act implementation, an agency's modernization efforts, and the 

overall project (investment) management. 

7 Since 2006, SBA has included IV & V in the LMAS project lifecycle and listed it in reports to OMB. In 2008, SBA 

contracted with a small business to perform this function. The request for proposal, contract, and technical proposal, 

all stated that the contractor would perform QAlIV & V. 
7 

project's QA activities. This occurred without the advisor's knowledge because at 

the time of the modification, OCIO had not assigned an independent Quality 

Manager to the LMAS project, and allowed the LMAS Program Manager to act in 

that capacity. 

By eliminating IV & V testing for the LMAS project, SBA is at risk of deploying 

this mission-critical system with undetected errors and with limited assurance that 

all requirements are fully satisfied. Further, had SBA originally solicited only for 

QA services, more bidders may have submitted proposals at more competitive 

prices, since QA is less intensive and requires less technical expertise than IV & V._ 

OCIO's Delay in Designating a QA Manager Contributed to Quality 

Assurance Weaknesses 

TestPros was allowed to proceed with an incomplete QA plan and without 

fulfilling all of its QA responsibilities, because OCIO had not acted timely in 

establishing a QA oversight function to ensure that the LMAS QA Plan was 

properly developed and fully implemented. The OIG had recommended in May 

2008 that a QA function be established that fully addresses the risk and scope of 

the LMAS project. However, it was not until October 2009 that OCIO identified a 

QA manager for the project. 

Further, although the QA manager was added to the project, he was unable to 

dedicate a sufficient amount of time to the project due to his other workload 

demands. In addition to performing QA oversight ofLMAS, the manager is also 

the Acting Enterprise Architect, which requires him to support the Agency's 

governance structure and to promote the use of common technologies, standards 

and services. He also told us that he has not been given resources that would help 

him fulfill his oversight responsibilities to the LMAS project. As a result, the QA 

Manager was unable to monitor execution of the QA plan and his involvement in 

the project was limited. 

The lack of a robust QA process has placed SBA in a position where it is unduly 

reliant on DNC and SRA to ensure the quality of their products. While we are 

encouraged that the governance structure and plans for the project itself are 

undergoing significant changes that will reduce project risks, SBA will still need 

to have an effective QA function to ensure that it is meeting its responsibilities to 

provide independent QA and oversight of its LMAS investment. 

RECOMMENDATIONS 

We recommend that the LMAS Project Director: 
8 

1. 	 Revise the LMAS QA plan to incorporate all the components required by 

the enterprise-wide QA plan. 

2. 	 Take steps to hold TestPros accountable for performing the activities 


specified in its contract. 


3. 	 Revise the TestPros Performance Work Statement to include the IV&V 

responsibilities previously deleted. 

We recommend that the Chief Information Officer: 

4. 	 Evaluate and make necessary adjustments to the QA Manager's workload 

to ensure that he can devote adequate time to oversee complete 

implementation of the enterprise QA oversight function. 

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL 

RESPONSE 

On August 6, 2010, we provided the LMAS Project Director and Acting Chief 

Information Officer with the draft report for comment. On September 7, 2010, the 

Associate Administrator for Management and Administration (formerly the LMAS 

Project Director) submitted a formal response, which is contained in its entirety in 

Appendix I. The response, which incorporated comments from the Acting CIO, 

expressed concurrence with all of the recommendations presented in this report. 

Management agreed that the LMAS QA plan should be revised to conform 

completely to the requirements of SBA's enterprise-wide QA plan as stated in 

recommendation 1, and commented that a revised QA plan has been drafted that 

conforms to the April 2010 OCIO QA Plan template. In response to 

recommendations 2 and 3, management agreed that TestPros' IV&Vactivities 

should be robust and meaningful. Management stated that the TestPros contract 

has been modified to reflect that the OCIO QA Manager is now responsible for 

oversight of the TestPros activities. The LMAS Project Director is also working 

with the OCIO QA Manager to strengthen LMAS IV&V activities. Management 

stated that significant progress has been made in implementing the QA processes 

of the Agency as demonstrated by the QAlIV & V audits and reviews that have 

already been completed. Further, management stated it did not believe that 

further revisions to the TestPros Performance Work Statement are necessary to 

accomplish meaningful IV & V activities on behalf of LMAS, but to the extent that 

any revisions are necessary, it would be fully supportive of them. 

Management concurred with recommendation 4 and stated that the Chief 

Information Officer has assigned a full-time IT QA manager, who is responsible 
9 

for the overall IT QA program operations, including establishing the relevant 

policy, processes and deliverables to ensure implementation of the IT QA function 

within SBA. Further, the QA manager has additional resources including contract 

support staff specific to the LMAS QA function. 

Finally, management stated that the CIO has established an IT QA program to 

validate that information technology investments meet planned requirements and 

to ensure that LMAS managers and other internal SBA managers utilize consistent 

practices to manage the successful delivery of new or on-going IT initiatives 

across Agency programs. 

ACTIONS REQUIRED 

Please provide your management decision for each recommendation on the 

attached SBA forms 1824, Recommendation Action Sheet, within 30 days from 

the date of this report. Your decision should identify the specific action( s) taken or 

planned for each recommendation and the target date( s) for completion. 

We appreciate the courtesies and cooperation of the OCIO, LMAS project staff, 

and LMAS contractor personnel during this audit. If you have any questions 

concerning this report, please call me at (202) 205JFOIAex2]or Jeffrey Brindle, the 

Director, Information Technology and Financial Management Group at (202) 205­

[FOIAex.2] 
10 APPENDIX I. MANAGEMENT COMMENTS 

u.s. SMALL BUSINESS ADMINISTRATION 

WASHINGTON, D.C. 20416 

DATE: September 7,2010 

TO: Debra S. Ritt, Assistant Inspector General for Auditing 

Jeffrey R. Brindle, Director, IT and Financial Management Group, OIG 

FROM: David B. Robbins, Associate Administrator for Management and Administration 

SUBJECT: Response to the Draft Report on the Adequacy of Quality Assurance Oversight 

ofthe Loan Management And Accounting System Project, Project No. 10002 

CC: Paul T. Christy, Acting ChiefInformation Officer 

Thank you for the opportunity to comment on the draft report. My comments and those ofthe 

CIO follow the recitation of each recommendation, below. 

Recommendation 1: Revise the !MAS QA plan to incorporate all the components required by 

the enterprise-wide QA plan. 

Comment: I agree that the LMAS QA plan should be revised to conform completely to the 

requirements ofthe SBA enterprise-wide QA plan. 

The SBA ChiefInformation Officer (CIO) 

The IT QA function is documented and in the process of getting formal approval for the IT QA 

Standard Operating Procedures (SOP). This SOP, based on the IT QA Concept of Operations, 

will serve as required practice for LMAS and all SBA IT investments and projects. 

The SBA CIO has assigned a full-time IT QA manager who is responsible for the overall IT QA 

program operations, including relevant policy, processes and deliverables to ensure 

implementation ofthe IT QA function within SBA. Additional resources include contracted 

support specific to the LMAS QA function. 

The overarching goal ofthe SBA's IT QA program is for the systematic monitoring and 

evaluation ofthe various aspects of a project, service, or facility to ensure that standards of 

quality are being met. Quality is determined by the intended users and stakeholders. The IT QA 

program has been added to the Audit Visibility Tracking System. This system is being deployed 

throughout SBA to ensure consistent tracking and monitoring of audit-related activities. 
11 

LMAS and any succcssor projccts will implemcnt QA proccsscs that conform to SBA's 

Enterprise Quality Assurance Plan. LMAS already has drafted a revised QA plan that conforms 

to the April 2010 OCIO QA Plan template. 

Recommendation 2: Take steps to hold TestPros accountable for performing the activities 

specified in its contract. 

Comment: I agree that TestPros should be held accountable for performing the activities in its 

contract. 

The TestPros contract has been modified such that TestPros activities are the responsibility ofthe 

OCIO QA Manager. That said, I am committed to seeing that the activities of TestPros are 

appropriately focused and managed and intend to cooperate fully with the OCIO QA Manager 

towards that end. Significant progress has been made in implementing the quality assurance 

processes ofthe agency as demonstrated by the QAJIV &V audits and reviews that have already 

been completed, as well as the initial follow-up and corrective actions that have begun as a result 

ofthose audits and reviews. Further, LMAS QA activities will conform to the IT QA Standard 

Operating Procedures in the process of being formally approved for SBA-wide use when that 

approval is given and the procedures become effective. 

Recommendation 3: Revise the TestPros Performance Work Statement to include the IV&V 

responsibilities previously deleted. 

Comment: I agree that TestPros activities should include robust and meaningful IV&V 

responsibilities. We have been working with the OCIO QA Manager to strengthen the LMAS 

IV&Vactivities. I am not aware that further revisions to the TestPros Performance Work 

Statement are necessary to accomplish meaningful IV & V activities on behalf of LMAS, although 

to the extent that any revisions are necessary, we will be fully supportive ofthem. LMAS will 

conform to the IT QA Standard Operating Procedures in the process of being formally approved 

for SBA-wide use when that approval is given and the procedures become effective. 

Recommendation 4: We recommend that the ChiefInformation Officer evaluate and make 

necessary adjustments to the QA Manager's workload to ensure that he can devote adequate time 

to oversee complete implementation ofthe enterprise QA oversight function. 

Comment: The SBA ChiefInformation Officer (CIO) concurs with this recommendation and 

has assigned a full-time IT QA manager who is responsible for the overall IT QA program 

operations, including relevant policy, processes and deliverables to ensure implementation ofthe 

IT QA function within SBA. Additional resources to the QA manager include contracted support 

specific to the LMAS QA function. 

The SBA CIO has also established the Information Technology Quality Assurance (IT QA) 

program to serve as a management control to validate that information technology investments 

meet planned requirements and to ensure that LMAS managers, and other internal SBA managers 

utilize consistent practices to manage the successful delivery of new or on-going Information 

Technology (IT) initiatives across the agency programs. 

The IT QA function is documented and in the process of getting formal approval for the IT QA 

Standard Operating Procedures (SOP). This SOP, based on the IT QA Concept of Operations, 

will serve as required practice for LMAS and all SBA IT investments and projects. 
12 

Thc ovcrarching goal of thc SBA's IT QA program is for thc systcmatic monitoring and 

evaluation of the various aspects of a project, service, or facility to ensure that standards of 

quality are being met. Quality is determined by the intended users and stakeholders. The IT QA 

program has been added to the Audit Visibility Tracking System. This system is being deployed 

throughout SBA to ensure consistent tracking and monitoring of audit-related activities. 

SIGNED - DAVID B. ROBBINS 

David B. Robbins