United States General Accounting Office 

Washington, DC 20548 

December 21, 2001


The Honorable Stephen A. Perry


Administrator


General Services Administration


1800 F Street NW


Washington, DC 20405


Subject: GSA On-line Procurement Programs Lack Documentation and Reliability Testing


Dear Mr. Perry: 

1 Earlier this year, we conducted a review of small business participation in selected on-line 

procurement programs in fiscal years 1999 and 2000. During the course of this review, we 

compared the small business share of procurement dollars awarded by two GSA on-line 

procurement programs, GSA Advantage! and Information Technology Solutions Shop (ITSS), 

to the small business share of all federal contract dollars awarded and we assessed the 2 reliability of the data needed for our report. We evaluated the data by (1) performing 

electronic tests of relevant fields and (2) requesting and reviewing, if available, related 

program and system design documentation, audit and system reviews, and reports. We 

determined that the data were reliable enough for the purposes of that report. However, we 

did identify several errors that GSA needed to correct to more accurately state total sales and 

small business participation. We also identified serious management control weaknesses 

related to data reliability as well as GSA’s ability to ensure the continuity of the operations 

for the on-line programs. This letter discusses our concerns and recommends improvement 

actions. 

BACKGROUND 

The federal government is taking steps to increase its use of electronic commerce, 

particularly in the area of conducting on-line procurement. For example, as part of an effort 

to improve the functioning of the government and achieve efficiencies of operations, the 

President has requested agencies to expand their use of on-line procurement in fiscal 

year 2002.3 

1

 Electronic Commerce: Small Business Participation in Selected On-line Procurement Programs (GAO-02-1, Oct. 29, 2001).
 2

 Generally accepted government auditing standards require that we assess the reliability of any data used to support our


findings, conclusions, and recommendations.
 3

 OMB Memorandum No. 01-11, Feb. 14, 2001.


GAO-02-229R GSA Data Documentation Issues 
GSA Advantage! and ITSS are two systems agencies can turn to for electronic procurements. 

GSA Advantage! serves as an “electronic mall” in which government buyers can search 

listings, compare prices, and purchase items on-line. ITSS on-line procurement program is 

used for larger, complex purchases. Customers enter requirements on-line, GSA staff solicit 

quotes, issue purchase orders, and authorize payment electronically. GSA administers both 

Advantage! and ITSS on-line procurement programs. Sales through these two on-line 

programs have been small but growing. The two on-line programs grew as a percentage of 

total federal procurement dollars from about one-half percent in fiscal year 1999 to about 

1 percent in fiscal year 2000. The table below provides a description of each of the programs. 

Table 1: GSA’s Advantage! and ITSS Programs: Services, Products, and Sales 

Dollars in thousands 

Fiscal year sales 

Program Services Products (Actual) 1999 (Actual) 2000 

Advantage! Web-based catalog 

ordering system 

Over 2 million commercial 

products and services, many 

of which are office supplies 

and information technology 

products and services 

$85,687 $124,892 

ITSS Web-based 

contracting services 

Information technology 

products and services, from 

computer monitors to 

complete network installations 

977,472 2,258,709 

Source: GAO, based on GSA documentation 

It is important that the data in both systems be as accurate and reliable as possible. 

Government agencies, for example, need to know that pricing data is accurate to make their 

purchases. Agencies also rely on these systems for summary data on government contracting 

opportunities, solicitations, awards, etc. The Congress and other oversight entities need to 

know that data on such things as total sales, agency use, and small business participation is 

reliable in order to assess the effectiveness of government contracting programs. 

LACK OF SYSTEM DOCUMENTATION 

GSA program officials told us that they did not have system documentation for either GSA 

Advantage! or ITSS. Documentation would normally include the data processing or 

automated aspects of the system (such as application program coding), operating 

instructions, and system policies, processes, and procedures (such as procedures on 

“configuration control” or how changes to the system are to be made). Documenting systems 

is a common management practice and is required of GSA under the Joint Financial 

Page 2 GAO-02-229R GSA Data Documentation Issues 
Management Improvement Program’s

4

 Federal Financial System Requirements to completely 

document both management systems. 

According to GSA Advantage! officials, there is no formal documentation of the system 

because the system evolved over many years in a piecemeal fashion among many developers 

–both government and contractors. The Chief Technology Officer for ITSS also said that 

ITSS was developed without extensive documentation and evolved over time, based on rapid 

prototyping and deployment. GSA officials told us that they have just started efforts to 

contract for services to document the systems. 

It is important for GSA to take time now to document the overall system. Documenting 

systems helps organizations ensure the data is reliably collected and entered into a system 

and that changes made are appropriate and completed in a timely and effective manner. 

Moreover, by documenting such things as coding and system operations, organizations are 

better positioned to ensure that they can continue operations in the event of a disaster or 

emergency. This is especially important for GSA because it relies heavily on contractors to 

operate and maintain the two systems. 

LACK OF DATA ENTRY CONTROLS AND ROUTINE DATA VALIDATION 

As part of our reliability assessment, we electronically analyzed data in both GSA Advantage! 

and ITSS and found significant errors. For example: 

• In the case of Advantage!, the database included $32.2 million that represented 16 “test 

orders”, which were used to train and provide demonstrations to users, according to GSA 

officials. These data were not marked as test cases and their inclusion would have 

skewed future reports, including small business participation and data extracts. 

Currently, there is no separate training environment to ensure that test orders are not 

included with real orders. 

• In the case of ITSS, we found that GSA was using an incorrect formula to calculate total 

sales. As a result, initially fiscal year sales totaled $3.1 billion; as corrected, they summed 

to $2.3 billion. The ITSS Chief Technology Officer told us that ITSS was not originally 

designed to provide strong reporting capability. Consequently, those trying to generate 

reports from the system need a thorough understanding of the data and data queries to 

obtain correct information. 

4

 The Joint Financial Management Improvement Program (JFMIP) is a joint undertaking of the US Department of Treasury, the 

General Accounting Office, the Office of Management and Budget and the Office of Personnel Management, working in 

cooperation with each other and other agencies to improve financial management practices in Government. JFMIP’s guidance 

covers acquisition systems such as Advantage! and ITSS. 

Page 3 GAO-02-229R GSA Data Documentation Issues 
In each case, contract and program officials agreed with our analysis and subsequently 

corrected the databases and/or reported numbers to reflect more accurate numbers. 

The data problems we identified may be attributable to the fact that GSA lacks data entry or 

edit controls to catch this type of errors and does not periodically validate data. Such 

controls are considered to be common industry practice since they help ensure that data are 

accurate and reliable. Although contractors told us that the system was designed to “police 

itself”—that is, its customers, contracting officials and other agencies would point out 

errors—the errors we discovered had not previously been brought to GSA’s attention. 

Moreover, GSA officials did not affirm that they would be implementing routine data 

validation and other safeguards to ensure data accuracy and integrity. 

CONCLUSION 

GSA’s two online procurement systems—GSA Advantage! and ITSS—lack basic management 

controls to ensure the systems produce reliable data, can not be inappropriately modified, 

and provide continuous service. These controls are increasingly important since government 

agencies are expected to increase their use of these systems and GSA relies heavily on 

contractors for systems operations and maintenance. Moreover, the Congress and other 

oversight entities need adequate assurance that these systems are reliable and accurate to 

assess the effectiveness of government contracting programs. 

RECOMMENDATIONS FOR AGENCY ACTION 

We recommend that the GSA Administrator direct managers responsible for oversight of both 

GSA Advantage! and ITSS to: 

• Fully document the procurement systems in accordance with guidelines for government 

financial management systems, such as those in JFMIP; 

• Establish procedures and deadlines for documenting any future changes to either system; 

• Establish procedures to prevent entry of test data into production systems; and 

• Develop procedures for periodic examination, validation, and timely correction of GSA 

Advantage! and ITSS data. 

Page 4 GAO-02-229R GSA Data Documentation Issues 
- - - -

AGENCY COMMENTS 

We provided GSA with a draft of this report for review and comments. In oral comments, 

GSA officials agreed with our recommendations and provided technical comments, which we 

have incorporated as appropriate. GSA officials also said that even though both systems are 

being reengineered, the issues we raised continue to be relevant, and that they are initiating 

actions to address our recommendations. The officials could not estimate when 

implementation of the recommendations would be complete. 

To evaluate the reliability of data provided by contractors and officials responsible for 

Advantage! and ITSS, we: 

• reviewed governmentwide guidance for developing and maintaining management 

systems; 

• reviewed existing agency information such as audit reports by the GSA Inspector 

General, independent reviews and system documentation, if available; 

• performed electronic data testing of relevant fields such as checking for missing data, the 

relationship of one field to another, values outside of a designated range, and dates 

outside valid time frames; and 

• discussed control weaknesses with officials and contractors. 

We conducted our review from April 2001 to September 2001 in accordance with generally 

accepted government auditing standards. In conducting our review we met with officials at 

the General Services Administration, and contractors for both Advantage! and ITSS on-line 

procurement programs. 

This report contains recommendations to you. The head of a federal agency is required 

under 31 U.S.C. 720 to submit a written statement of actions taken on our recommendations 

to the Senate Committee on Governmental Affairs and the House Committee on Government 

Reform not later than 60 days after the date of this letter and to the Senate and House 

Committees on Appropriations with the agency’s first request for appropriations made more 

than 60 days after the date of this letter. 

Page 5 GAO-02-229R GSA Data Documentation Issues 
We are sending copies of this letter to interested congressional committees. This letter will 

also be available on GAO’s home page at http://www.gao.gov. Please contact me at 

(202) 512-4841 or Hilary Sullivan at (214) 777-5652 if you or your staff have questions. Major 

contributors to this report were Barbara Johnson, Rosa M. Johnson, and Beverly Ross. 

Sincerely yours, 

David E. Cooper 

Director, Acquisition 

and Sourcing Management 

(120106)


Page 6 GAO-02-229R GSA Data Documentation Issues