Financial Systems Integration Office

Financial Management Systems

for U.S. Government Agencies

Core Financial System 

Product Qualification Test Policy 

March 2007
 
Contents 

Abbreviations vi

Foreword 1

Chapter 1 Introduction 3

1.1 Purpose of This Document 3 

1.2 Audience 3 

1.3 Supplemental Testing Materials 4 

1.4 Document Organization 4 

Chapter 2 Overview of Qualification Testing 5

2.1 Scope of Qualification Testing 5

2.2 Limitations of Qualification Testing 6

2.3 Relationship to Agency Procurement Activities 6

2.4 The FSIO Test Team 7

2.5 Test Types 7

2.6 Timing of the Test 8

2.7 Duration of the Test 10

Chapter 3 Developing a Qualification Test 11

3.1 Agency Outreach 11

3.2 Requirements Identification 11

3.3 Test Development 11

Chapter 4 Qualifying a Product 13

4.1 Applying for the Test 13

4.2 Exchanging Pretest Information 17

4.3 Conducting the Test 21

4.4 Grading the Test 25

4.5 Conducting a Retest 26

4.6 Performing Posttest Activities 27

Appendix—Test Activities and Time Frames 

 iii 
Contents

 iv 
Abbreviations 

CFOC Chief Financial Officers Council 

FAS Federal Acquisition Service (GSA) 

FFMIA Federal Financial Management Improvement Act of 1996 

FMS Financial Management Service (Treasury) 

FSIO Financial Systems Integration Office 

GSA General Services Administration 

OMB Office of Management and Budget 

Treasury Department of the Treasury 

U.S. SGL U.S. Standard General Ledger 

 v 
Contents

 vi 
Foreword 

This document specifies the core financial system qualification test policy and procedures that will be followed by the government starting in 2007. This policy replaces 

the one previously in place, which was last updated in September 2002. The following key policy changes are reflected in the sections indicated, and highlighted in the 

text with a side bar: 

• Eliminates the policy to fully retest qualified products on a 3-year cycle. Certifications issued to qualified software products will continue to be active as 

long as subsequent full and incremental tests are taken and passed. (Section 

4.6.1.1, Passed the Test) 

• Revises the criterion for passing the test. Previously, the criterion was successful execution of 100 percent of the test steps. Products must now show 

they can meet 100 percent of the mandatory requirements tested, as evidenced 

by successful test step execution. Note that alternate test procedures are still 

allowed. (Section 4.4.1, FSIO Validates Observed Processing Results) 

• Changes the requirement for vendor demonstrations of product functionality. 

Demonstrations at the start of a test are no longer optional. These demonstrations will be used as part of the first test readiness assessment of the software’s ability to meet the current mandatory requirements. These demonstrations also familiarize FSIO testers with the software product, increasing testing efficiency. (Section 4.2.3, FSIO Conducts Setup Interview)

• Requires vendors to demonstrate test readiness and establishes test readiness 

criteria. Vendors that are determined not ready will be rescheduled. (Section 

4.2.4, FSIO Conducts First Readiness Assessment; Section 4.3.4, FSIO Conducts Second Readiness Assessment; Section 4.5.4, FSIO Conducts Retest 

Readiness Assessment) 

• Expands incremental test coverage to include any functional capability of 

agency interest, supported by an existing mandatory requirement or newly 

mandated requirement, driven by external agency needs. Previously, only 

new mandated requirements were tested, such as the Central Contractor Registration. (Section 2.5.2, Incremental Test) 

• Adds a policy requiring full testing of version upgrades when version demonstrations are determined by FSIO to be insufficient evidence of continued 

compliance with all mandatory core financial system requirements. (Section 

2.6.3, Timing of a Version Test; Section 4.1.6, FSIO Notifies Vendor of Need 

to Test) 

• Reduces the time from 1 year to 3 months that vendors must wait before applying to retest a nonqualifying product in a full retest. (Section 2.6.5, Timing 

of a Test of a Nonqualifying Software Product; Section 4.6.6, Vendor Applies 

to Test New Version after Failure) 

 1 
Foreword 

•

est) 

procedure in the execution of a test; 

FSIO may grant permission under conditions similar to those previously 

•

ation for testing. Instead, documentation, updated to reflect 

software changes made to pass the tests, must be submitted within 30 days of 

t 

•

 government customers within 6 months of notification of 

successful test results. (Section 4.6.4, Vendor Provides Software to the Gov-

•

 as configured for the test. (Chapter 4 Qualifying a 

•

e. These assurance statements assert that all of the changes 

• t 

, Application Screen-

•

 from third-party report writer tools which are specified in the 

•

websites, with links provided on the FSIO website. (Section 

This do nized around the test process. 

Eliminates the requirement to submit a separate test application for a retest. 

(Section 4.5.1, FSIO Schedules Ret

• Changes the term “variance” to “alternate test procedure.” Vendors may request permission to use an alternate test 

granted as variances. (Section 4.2.1, Vendor Submits Alternate Test Procedure Requests) 

Eliminates the requirement to submit user documentation for software products upon applic

notification of successful test results. (Section 4.6.2, Vendor Submits Produc

Documentation) 

Requires vendors to make the newly qualified software and related documentation available to

ernment) 

Requires vendors to save the configuration used to pass the test and make information about it available at agency request for subsequent demonstration 

of software functionality

Product) 

Changes the timing of vendors’ submissions of assurance statements to be at 

the time of test application, instead of 2 weeks after notification of a successful test outcom

made to a product during the test process will be incorporated into the product

version to be named on the certificate and that the product version will be 

available for agency acquisition. (Section 4.1.1, Vendor Submits Test Application; Section 4.1.2.1, Application Screening Criteria) 

Requires vendors to provide an additional assurance that the software produc

has been fully tested (e.g., unit test, integration test, regression test). (Section

4.1.1, Vendor Submits Test Application; Section 4.1.2.1

ing Criteria) 

Eliminates the collection and publication of Report Sources information, 

which documented how each test output report was generated, since most are 

now produced

application. 

Changes the timing for when information is collected on the value-added features provided by tested software products. The information will be posted on

the vendors’ 

4.6.2, Vendor Submits Product Documentation and Information on ValueAdded Features) 

cument has been restructured to improve readability. Policies are now orga

 2 
Chapter 1 Introduction 

The Federal Financial Management Improvement Act of 1996 (FFMIA) mandates 

that agencies implement financial system software that substantially complies with 

the following: 

• Federal Accounting Standards Advisory Board statements of Federal financial accounting concepts and standards 

• Established government-wide system requirements 

• U.S. Standard General Ledger (U.S. SGL) at the transaction level. 

Office of Management and Budget (OMB) Circular A-127, Financial Management 

Systems, mandates that Federal agencies acquire only financial system software and 

services that comply with Core Financial System Requirements. The requirements, 

composed and published by the Financial Systems Integration Office (FSIO), are the 

established government-wide system requirements referred to in the FFMIA. 

Circular A-127 further assigns responsibility for requirements compliance testing to 

the Chief Financial Officers Council (CFOC). Actual product testing is carried out on 

behalf of the CFOC by FSIO. 

FSIO develops policy for and manages the core financial system qualification process. As part of the process, FSIO develops and administers a qualification test to prospective vendors. FSIO is also responsible for publications that document core financial system functional and technical requirements. 

Final determination that a product fully complies with established government-wide 

requirements is made by the FSIO Executive. This determination is based solely on 

the results of the formal software testing program administered by the FSIO staff. 

1.1 PURPOSE OF THIS DOCUMENT

This document specifies the FSIO qualification test policies and practices. The policies govern how FSIO tests the products offered by commercial vendors. Detailed 

information about each test, such as schedule, scope, and requirements to be tested, is 

provided in the test plan specific to that test. 

1.2 AUDIENCE

The audience for this document includes two primary parties: 

• Federal agencies, on whose behalf the test policy is established and the testing is conducted by FSIO

 3 
Introduction 

• Commercial vendors, who are required to adhere to these policies in order to 

obtain and maintain certification of their core financial system software products. 

1.3 SUPPLEMENTAL TESTING MATERIALS

This test policy applies to other FSIO testing materials mentioned in this policy and 

published separately on the FSIO website (http://www.fsio.gov). These materials include the 

• test application, 

• test plans, 

• format for alternate test procedure requests, and 

• format for test questions. 

1.4 DOCUMENT ORGANIZATION

The remainder of this document is organized in sections that further describe qualification testing. The major sections are as follows: 

• Overview of qualification testing 

• Developing a qualification test 

• Qualifying a product, including conducting the test, evaluating the results, 

and publishing the resulting certifications.

 4 
Chapter 2 Overview of Qualification Testing 

FSIO tests commercially available core financial system software products to certif

them for federal acquisition. FSIO testing is designed to generate objective evidence

that core financ

y 

ial system products comply with mandatory government-wide core 

financial system requirements. 

The obj e agencies to acquire new commercial 

core financial systems with the assurance that selected products can be configured to 

han c business transactions. 

This reduces the risk that agencies will acquire noncompliant or otherwise ineffective 

core n

-

ve (incorporating all mandatory requirements), traceable to core requirements, and 

d provide examples to help agencies and vendors understand published requirements and deliver better solutions. 

e test for software implemented in 

their specific environment. 

s 

nts 

as quickl with minimal cost to the government. 

esources in the test of core financial 

2.1 SCO

FSIO tests compliance with requirements published in Core Financial System Reas supplemented by incremental test requirements. The test includes only 

pabilities; FSIO-administered tests do not address performance or prodective of this testing program is to enabl

dle common federal accounting, reporting, and electroni

 fi ancial system software. 

Other testing objectives include the following: 

• Promote consistency of core financial systems across the government. It iden

tifies and requires a set of functional core financial system requirements used 

by all stakeholders. The testing process is designed to be comprehensi

transparent. 

• Clarify requirements an

• Reduce the need for agencies to independently develop a test for common 

core financial system requirements, allowing them to focus on their own 

unique requirements. Agencies can reuse th

• Incrementally test commercial off-the-shelf software products over time. Thi

helps ensure that the software offerings are adapted to meet new requireme

y as possible, 

FSIO serves as a focal point to leverage federal r

systems for the benefit of all agencies. 

PE OF QUALIFICATION TESTING

quirements, 

functional ca

uct suitability for use at a specific agency. 

 5 
Overview of Qualification Testing 

2.2 LIMITA O

The qualification test process is the entry point to the Federal financial management 

ough the test comm requirements, it does 

not address a number of other important factors in the selection of core financial syssess whether a product is suitable for a specific agency’s operating environment. Agencies differ dramatically in size, budgetary resources, and busin agency may have unique agency-specific requirements in 

addition to the core financial system requirements. 

• Assess the product’s ease of use. Each product, while meeting the mandatory 

own strengths and weaknesses, such as number of screens accessed to complete a transaction and 

 may impact ultimate success in day-to-day operation at a specific agency. 

rformance. Performance depends on such 

ta. 

 installed and configured appropriately for 

ent. Although each vendor configures its 

 test, this configuration is not necessarily the 

2.3 RELATIONSHIP TO AGENCY PROCUREMENT 

ACTIVITIES

Qualification testing is independent of any agency procurement process. FSIO 

certifies only that a named product has passed all FSIO-required qualification tests. It 

is the vendor’s responsibility to maintain this certification for products offered to 

agencies over time. It is the acquiring agency’s responsibility to confirm that 

certifications of proposed products are in good standing.1

FSIO recognizes that agency contracts or orders for core financial system products 

may require the successful bidder to submit a FSIO-issued certificate as evidence of 

qualification. Vendors are responsible for ensuring that the name and version of the 

 

TI NS OF QUALIFICATION TESTING

system market for commercially available software products. Th

prehensively checks for compliance with core financial syste

tem software. The qualification test does not: 

• As

ness practices. A

FSIO core financial system requirements, has its 

“user friendliness,” which

• Assess the product’s computing pe

things as compatibility with the enterprise architecture, response time, and 

transaction throughput capacity, which can be especially important when an 

agency has large volumes of transaction da

• Ensure that delivered products are

an agency’s operating environm

product to pass the qualification

same one an agency uses. It is the agency’s responsibility to test the product 

as implemented. An improperly configured system has the potential to give

noncompliant test results. 

1

 Section 39.101(c) of the Federal Acquisition Regulation requires agencies to acquire core financial management software that has been certified. 

 6 
Overview of Qualification Testing 

software product tested, and for which a certificate of compliance is issued, 

sistent with the name and version identified in contract vehicles. 

are conA complete list of all product certifications can be found on the FSIO website. The 

2.4 THE FSIO TEST TEAM

 vendor proprietary information. The team works 

 on-site representative. The team 

, and advise the FSIO on-site team 

leader. 

The tea

testing 

with the

To m ke the 

app r

Manage determine their 

accuracy. 

2.5 TEST TY

FSIO uses three types of qualification tests to qualify new products and to enable 

ven rs

•

 the first try, the product 

may be retested if FSIO determines that it is eligible for retesting (that is, if the prod2.5.1 Full T

Full testing is used by FSIO to qualify new products and to retest qualified products 

after significant requirement updates. A full test allows FSIO to determine product 

compliance with mandatory requirements in all functional areas as published in the 

requirements that form the basis of testing, the test materials, and the results of the 

FSIO testing process are also posted on this website. FSIO notifies the GSA’s Federal

Acquisition Service (FAS) of any changes to the list of product certifications. 

The FSIO staff directs the test team. Team members may consist of either government employees (including FSIO personnel) or qualified contract support personnel. 

The government takes reasonable steps to prevent contract support personnel conflicts of interest and to safeguard

under the direction of FSIO and a FSIO-designated

members make observations, gather information

m members interact with software vendor personnel for routine planning and 

activities. The FSIO on-site representative resolves issues after consultation 

 vendor’s designated senior representative. 

a test more operational/effective, other stakeholders are involved when 

rop iate. As an example, the Department of the Treasury (Treasury) Financial 

ment Service (FMS) has assisted with testing interface files to

PES

do to maintain their product certifications over time: 

Full test 

• Incremental test 

• Version test. 

When a product does not pass a full or incremental test on

uct passes at least 90 percent of the tested requirements on the first attempt). 

est 

 7 
Overview of Qualification Testing 

current Core Financial System Requirements and related FSIO-issu

The functional areas are as follows: 

ed supplements. 

•

• Fund Balance with Treasury management 

•

2.5.2 Increment

FSIO uses an incremental testing process to assess product compliance with new 

• System management 

• General ledger management 

• Budgetary resource management 

• Payment management 

Receivable management 

Cost management. 

al Test 

requ m ire ents as they are issued. In planning an incremental test, FSIO may 

incorporate new central-agency-mandated requirements and any other mandatory 

requirements for which expanded testing is desired based on input from agencies. 

All certified products are subject to ongoing incremental testing in order to maintain 

thei u

2.5.3 Version Te

Version

are rele . A version test allows FSIO to assess product 

changes for potential imp ousl . New 

 may be sub2.6 TIMING OF THE T 

2.6.1 Timin

begin making software changes based on the draft requirements to maximize the time 

available to 

r q alification. 

st 

 testing is used by FSIO to qualify new versions of qualified products as they

ased (issued) by the vendor

acts on previ y certified product functionality

product versions will continue to be evaluated prior to their release and

ject to full testing at FSIO’s option. 

EST

The timing of each test type is triggered by differing events, as described below. Appendix A contains the time frames for developing the test and qualifying software.

g of a Full Test 

The need for a full test of qualified software products is communicated based on the 

publication of substantial requirement changes. FSIO gives advance notice of testing

when draft requirement updates are issued for comment. Vendors are encouraged to 

prepare for the test. Formal notice of a full test may be given at any time 

after the final requirements are published. 

 8 
Overview of Qualification Testing 

The vendor is responsible for product engineering and for the timing of new product 

introductions. Before offering a new product to a Federal agency, the vendor must 

apply for and successfully complete the full test process. 

2.6.2 

remental test is in the best interest of the govern-

 to 

ental test plan and the test materials that will accompany the 

2.6.3 Timin

e responsible for defining the mechanism and timing for releasing new 

product versions. Whenever a vendor makes changes that result in a uniquely identi-

 qualified product (e.g., v8.3) that can be acquired by an agency, eiaintenance upgrade process or as a new product installation, the ven-

 the information provided 

A full test, including any retest, must be completed within 12 months of submitting 

the test application. 

Timing of an Incremental Test 

When FSIO determines that an inc

ment, FSIO drafts any new requirements and requirement revisions that are needed

describe the required functionality. The vendor is responsible for updating its products to comply with all new mandatory requirements. Vendors are encouraged to begin making software changes based on the draft requirements to maximize the time

available to prepare for the test. Considering comments received on the draft requirements, FSIO updates the requirements and issues them in final form. FSIO then

issues a notice of intent to conduct an incremental test. Following this announcement, 

FSIO develops an increm

plan. 

An incremental test, including any retest, must be completed within 6 months of submitting the test application. 

g of a Version Test 

Vendors ar

fied version of a

ther through a m

dor must apply for a version test. 

Upon receipt of a version test application, FSIO will review

and the release notes related to the version update. If more information is needed, 

FSIO will request a product demonstration. FSIO may qualify the new version based 

on a review of the release notes alone or after the additional product demonstration. If 

the changes to a software product are extensive, the vendor may be required to apply 

for u a f ll test. 

2.6.4 Timin

 a list of requirements to be tested. Retests are to be scheduled within 30 calays of receipt of notice from FSIO. 

g of a Retest 

Upon completion of the validation of test results for full and incremental tests, FSIO 

will notify the vendor if a retest is needed to complete a test. The notification will 

include

endar d

 9 
Overview of Qualification Testing 

2.6.5 Timin

Vendors m of 3 months after notification of unsuccessful test reg of a Test of a Nonqualifying Software Product ust wait a minimum

sult e s b fore applying for a new test of a nonqualifying software product. 

2.7 DURATION OF THE EST

Time frames for vendor completion of each test process, from test application 

thro h specified in Appendix A. Time frames for vendor 

completion of unobserved and observed processing are specified in the individual test 

pla rt of a full test is defined as the start of the execution 

of unobserved test steps. The time frames allowed for test execution are determined 

by O n specifics on test phases and applicable test execution time frames. 

T 

ug any required retest, are

ns published by FSIO. The sta

FSI for each test version. The test plans contai

 10 
Chapter 3 Developing a Qualification Test 

The FSIO testing program starts with agency outreach to identify or update core financial system requirements and ends with formal system testing of commercial 

3.1 AGE

aterials are published by FSIO. 

3.2 REQ TIFICATION

 on FSIO-published requirements as the official reference describing the essential characteristics of their automated systems. Auditors of 

fFSIO publishes new and revised requirements before the test date to allow enough 

 new editions of Core Financial System Requirements or supplemental 

updates. 

3.3 TEST D

ith an unimpeachable basis for determining whether a product is qualified for federal government use. 

software products. 

NCY OUTREACH

Agencies are integral to the success of the testing process. Common agency needs 

drive the requirements identification process on the front end. Agency representatives

help establish test development priorities and are the primary audience for published 

test results. 

To help agencies stay engaged in the testing process, FSIO actively solicits their input using a variety of special study groups, topical workshops, open houses, surveys 

and key personnel interviews. The resulting reports, requirements, and other testingrelated m

UIREMENTS IDEN

Federal financial managers rely

federal financial management systems rely on these requirements as their official re

erence for required core financial system functionality, and vendors rely on them to 

help them understand agency-processing needs, enabling them to develop improved 

system solutions. 

time for software vendors to update products to perform newly required functions. 

The basic process followed by FSIO to update a requirements document has three 

steps: (1) assemble a draft set of updated requirements, (2) distribute a draft to stakeholders for comment, and (3) publish resulting requirements in a final form as approved by the CFOC. New and revised requirements for incremental tests are published either as

EVELOPMENT

FSIO translates mandatory functional requirements into a series of test steps that are 

suitable for execution in a wide variety of operating environments. The goal is to develop a test that provides the government w

 11 
Developing a Qualification Test 

For each test, FSIO publishes a test plan, which contains instructions on ho

cute the test. The test materials, including setup data, test scripts, and expec

w to exeted re-

. 

 on the 

ay be needed to test each mandacted 

assults, are published as appendixes to the test plan and are posted on the FSIO website

The test script contains test cases, each composed of several test steps, based

mandatory requirements. One or more test steps m

tory requirement. 

The test is organized by accounting cycles, and includes a complete set of expe

results, including financial statements, transaction registers, trial balances, and Tre

ury e-commerce files. Agencies and vendors are invited to review the draft test materials before they are finalized. 

 12 
Chapter 4 Qualifying a Product 

Qualification testing starts with a vendor application that identifies the subject product. The process typically ends with the issuance of a certificate of qualification. 

Vendors set up for and execute the test; FSIO observes the test and reviews the actual 

results against expected results. 

onfigure their products to meet core financial system 

s at key points in the process and requires a threshold for passing the ini-

. If this threshold is not met, vendors must 

invest in further development before proceeding with a test. 

sion number) for all software 

In taking a test, vendors must c

requirements, enter the data, and generate the results. Vendors provide the computing 

facilities and the software subject to qualification. 

To conserve government resources on products that lack functionality, FSIO assesses 

test readines

tial test. A product is not certified unless it meets 100% of the requirements. Vendors 

are initially required to pass at least 90 percent of the requirements tested. The software is then given a chance to retest the 10% of failed requirements that will qualify 

it for certification. Software is considered to be nonqualifying if it fails to meet more 

than 10 percent of the tested requirements

Product specifics (vendor name, software name, ver

products that passed the test are posted on the FSIO website. Vendors are required to 

save the configuration used to pass the test and make information about it available at

agency request for subsequent demonstration of software functionality as configured 

for the test. 

Execution of the test includes the following processes: 

• Applying for the test 

• Exchanging pretest information 

• Conducting the test 

• Grading the test 

• Conducting a retest 

vities. 

4.1 APP

This section explains the purpose, uses, and processes associated with applying for 

the core financial system qualification test. The actual test application with 

instructions is published separately on the FSIO website. The application form should 

be used for all tests. 

• Performing posttest acti

The activities and related policies for each process are described below. 

LYING FOR THE TEST

 13 
Qualifying a Product 

4.1.1 Vend

t 

o provides the following information: 

• Vendor assurance of the software’s capability to meet mandatory requirements 

• Vendor assurance that the software has been fully tested (e.g., unit test, inteor Submits Test Application 

The qualification test and resulting certificate of compliance pertain to a specific 

software product version (and not to a vendor). Software vendors applying for the tes

must designate the software product name and version in the application. 

The application als

• Administrative information used to schedule and conduct the test 

• Information for screening applications

gration test, regression test) 

• Vendor assurance that all of the changes made to a product during the test 

process will be incorporated into the product version named on the certificate 

and that the product version will be available for agency acquisition. 

When software changes have been made beyond those needed for an incremental test, 

the vendor must submit separate applications for the version and incremental tests. 

This will allow FSIO to evaluate the changes separately. The version test will be conducted separately and before the incremental test. 

Applications for full tests of new software products and for version tests of new software versions may be submitted at any time. Applications for full and incremental 

tests of previously qualified software products must be submitted within 30 calendar 

days of the publication of final test materials. 

4.1.2 FSIO Notifies Vendor of Application Acceptance or 

Rejection 

Upon receipt, FSIO reviews each application for completeness. If additional information is needed, FSIO will contact the vendor. 

FSIO also reviews the application in accordance with the screening criteria described 

above. Within 30 calendar days of receipt of the application or of additional information, whichever is later, FSIO will notify the vendor whether the application has been 

accepted or rejected. 

 14 
Qualifying a Product 

4.1.2.1 Application Screening Criteria 

FSIO uses portions of the application to assess the capability of the applicant software and determines with reasonable assurance that it is potentially capable of satiscent of the mandatory requirements, whether or not 

fying the FSIO requirements. These portions include the following: 

1. Mandatory requirements assurance. The vendor asserts that the candidate 

software can meet each mandatory requirement. The vendor’s software is 

required to meet 100 per

FSIO actually tests that the software meets the requirement. 

2. Testing assurance. The vendor must provide a written assurance that the 

software product has been fully tested (e.g., unit test, integration test, 

regression test). 

3. Updated software assurance. The vendor must provide a written assurance to 

FSIO that all changes (e.g., fixes) made to a product during the test process 

will be incorporated into the product version named on the certificate and that

the product version will be available for agency acquisition. 

4. References. FSIO uses the references as supplementary information to help 

determine whether the candidate system software has the capability to pass 

ion. 

ucts (e.g., release notes). 

4.1.2.2 Acceptance of Application 

If the application is accepted, FSIO states this event in writing and schedules a mutually c ant. 

4.1.2.3 Rejection

App a testing may be rejected for one or more of the following 

reasons: 

eets all mandatory requirements 

tion in accordance with the instructions in this 

ation2

• Absence of an officer’s signature on the application. 

the test. Lack of references does not prejudice consideration of an applicat

FSIO may contact the references as part of the evaluation. 

5. Release notes. As part of a version or incremental test application, the vendor 

is required to fully disclose all material changes made to previously qualified 

prod

 ac eptable test period with the applic

 of Application 

lic tions for software

• Failure to attest that the software product m

• Failure to complete the applica

document and those on the application 

• Provision of inaccurate or false inform

 2

 For example, FSIO may observe during testing that a module or software product other than 

that listed in the application was used to pass the test. 

 15 
Qualifying a Product 

FSIO will notify the vendor, in writing, of th

and will return the application. 

e reason for an application’s rejection 

4.1.3 FSIO R

For r IO will review the release notes to 

assess the extent to which new functional capabilities have been added and previous 

fun n

4.1.4 Vendor Provides Version Demonstration 

For ver ay request a version demonstration (prior to 

qualification 

nee -

quested

tions do

4.1.5 FSIO

n demonstration. If the changes to 

4.1.6 FSIO st 

version does 

ill replace the 

 version number on its website with the latest certified version. 

A vendor whose application is rejected by FSIO may reapply at any time. A vendor 

resubmitting an application must complete all parts. The vendor should attach a letter 

to the resubmitted application describing the actions taken to correct the issues that 

resulted in the previous rejection. 

eviews Release Notes 

 ve sion or incremental test applications, FS

ctio ality has been affected. 

sion test applications, FSIO m

testing) as further evidence that the product can perform the functions 

ded to generally satisfy mandatory core financial system requirements. When re

, version demonstrations are to be scripted by the vendor. Version demonstra-

 not take the place of the test. 

 Evaluates Version Demonstration 

For version test applications, FSIO may qualify the new version based on a review of 

the release notes alone or after the additional versio

a software product are extensive, the vendor may be required to apply for a full test. 

FSIO reserves the right to make the final determination in this matter. 

 Notifies Vendor of Need to Te

For version test applications, if FSIO determines that the new software 

not require testing, it will issue a certificate for the new version. FSIO w

certified

Full testing of version upgrades is required when the product demonstration does not 

provide sufficient proof that the software meets all the mandatory core financial system requirements tested. If FSIO determines that the new version requires testing, the 

vendor will be required to take a full test to qualify the new software version. 

4.1.7 FSIO

FSIO will schedule the observed test based on the dates requested by the vendor, or at 

FSIO’s earliest convenience. The scheduled date for observed testing is subject to 

successful completion of the readiness assessments (as described in the Conducting 

the Test process). Unobserved testing typically occurs before observed testing. 

 Schedules Observed Testing 

 16 
Qualifying a Product 

4.2 EXCHANGING PRETEST INFORMATION

est 

t-

• a report that pre-

• ion format. Once the test has been scheduled, FSIO will 

d 

•

 alternative 

s request for use of the alterior to the test setup interview. The standard format that 

equesting an alternate test procedure is provided on the 

4.2.1 Vendor Submits Alternate Test Procedure Requests 

To maintain the integrity of the test process and ensure that the expected results are 

The vendor and FSIO will exchange information before the test is conducted. FSIO 

will provide the vendors with the test materials discussed below: 

• Test plan. Test plans contain live testing procedures for each subject test. T

plan appendixes are used to deliver setup data, test scripts, and expected ou

puts. 

Requirements trace matrix. The requirements trace matrix is 

sents the linkage between test steps and the core financial system requirements. 

Vendor test quest

work with the vendor to resolve any questions the vendor may have on the 

test materials. It is the vendor’s responsibility to understand the mandatory 

requirements and the test steps. FSIO will not advise the vendor on how to 

operate the software to obtain the desired results. However, before the test, 

FSIO will answer reasonable vendor questions about the test execution. The 

standard format that vendors must use in submitting test questions is provide

on the FSIO website. 

Alternate test procedure request format. FSIO will consider requests for exceptions to the test scripts individually, reviewing the proposed

and seeking clarification from the vendor, if necessary. The revised test script 

processing must be adequate for FSIO to perform regression testing necessary 

to ensure that the changes do not affect the integrity of the FSIO qualification 

test. FSIO will respond in writing to the vendor’

nate test procedure pr

vendors must use in r

FSIO website. 

achieved, the vendor is not allowed to combine, separate, resequence, skip, or modify 

individual test steps, posting effects, or other expected results without prior consent. 

If a tested core financial system cannot process a step as specified, the vendor must 

requ t es an alternate test procedure. Alternate test procedure requests identify the affected steps, the execution problem, and proposed alternative. A single request can 

cover multiple steps, if the execution problem is exactly repeated. In this case, the 

vendor would make a single alternate test procedure request but is obligated to itemize all steps to which the alternate test procedure applies. Table 1 contains types of 

alte t rna e test procedures and guidelines for how FSIO will view the proposal. 

 17 
Qualifying a Product 

Table 1. Alternate Test Procedure Types 

Type of alternate procedure Guideline 

Setup. An alternate setup procedure occurs when the vendor uses different configuration or assumption data than what 

were provided in the test plan. 

Minor setup differences that can be directly 

cross-walked to expected results are generally 

allowed. The use of account code suffixes to 

establish subaccounts would be considered 

an acceptable alternate provided these 

subaccounts roll up as intended and are not 

used for other unexpected posting results. 

Wholesale substitutions of specified fund 

codes, object classes, U.S. SGL accounts, 

programs, and projects are not permitted. 

Posting. An alternate posting procedure 

occurs when a subject product is deby FSIO. (The test rules governing the 

If the core financial system can be configured 

to use the prescribed posting rules, FSIO exway (without an alter-

. An alternative posting model 

may be allowed, if the net effect of the alternasigned to post transactions in a manner 

different from the posting model specified 

pects it to be tested that 

nate procedure)

use of alternate document numbers to 

reference U.S. SGL posting results are 

not considered an alternate procedure.) 

tive model matches the specified result. The 

use of journal vouchers to accomplish posting 

results is prohibited. 

Execution. An alternate execution procedure occurs when a tested product completes a test step using multiple, sepaAlternate execution procedures may be acceptable if their initiation is automatic or sysrately initiated documents or batch processes. 

sults, and they

funds availabil

nancial positio

tem controlled, they achieve the expected re-

 do not cause a misstatement of 

ity (real-time) or period-end fin. 

Function. An alternate functional procedure occurs when a tested product was 

not designed to support a stated requirement (e.g., produce a report, produce a 

query result, or perform process control). 

All tested functionality is considered mandatory. 

Reporting. An alternate reporting procedure occurs when a product does not 

generate a required output report, query 

Treasury- a

be changed

example form

result, or transaction file according to form 

and content rules cited in the test step. 

approved provided all requested information is 

presented comprehensively. 

nd OMB-defined formats cannot 

. When the test is based on an 

at, alternate procedures may be 

Process. An alternate process procedure 

occurs when the way a product handles a 

FSIO expects differences in the way a core 

financial system handles complex po

multistep test case differs from the testspecified approach (e.g., starting balances, year-end closing, application programming interface, cost distribution). processes. The basis for allowing an alternate 

process is whether the intended result is 

achieved and whether the end-to-end process 

is automatic. 

sting 

If the vendor proposes using an alternate postin

script, or alternate document numbering for any

g model, a deviation from the test 

 test step or accounting event (alternate test procedure), the vendor must request permission from FSIO before the test. 

The alternate test procedure request format available on the FSIO website must be 

used for this purpose. Alternate test procedures are subject to FSIO evaluation and 

approval. All alternate test procedure requests must be submitted at least 1 week before the test setup interview. 

 18 
Qualifying a Product 

Vendors who wish to use an alternate test procedure must do the following: 

• Specify the alternate document numbering, alternate processing, and detailed 

• e all changes to the test materials that result from use of the alternate 

test procedure, including those to the setup data, test scripts, expected results, 

ility to provide a cross-walk from the test script and expected results, 

respectively, to the corresponding vendor’s system information. 

Reques

mance w

dures w y 

test resu

test pro

FSIO. 

FSIO’s prior review of these exceptions should not be construed as determination of 

the tc

based o re 

that the

FSIO re

the ven

4.2.2 Vendor Co

ent, 

establish 

aterials 

-

-

fication by FSIO. The test question format provided on the FSIO website 

must be used for this purpose. All questions on the test materials must be 

submitted prior to the setup interview. 

• Explain why the alternate test procedure is needed, and what the benefit is to 

the government. 

ledger postings associated with each affected test step, as applicable. 

Provid

system queries, and reports. If the vendor uses an alternate document numbering system or an alternate general ledger posting model, it is the vendor’s responsib

ts will be reviewed case-by-case for their impact on the test and for conforith the underlying requirements being tested. Allowed alternate test proceill be confirmed during the setup interview and used during the test to qualif

lts. Additional alternate test procedures and revisions to approved alternate 

cedures may be requested during the test to document processing observed by

ou ome of the execution of the related test steps. All test step scoring will occur

n the execution of the test steps themselves during and after the test, to ensu

 alternative processing still meets the applicable FSIO requirements. 

serves the right to publish descriptions of all alternate test procedures used by 

dor to pass the test. 

nducts Setup and Other Pretest Activities 

The vendor is responsible for reviewing all test steps, establishing a test environm

loading FSIO-prescribed setup data, and recording transactions necessary to 

any beginning balances. Vendors are required to provide FSIO with the test m

listed below: 

• Vendor test participants list. Vendors must submit a list of all employees, 

contractors, and consultants assisting or present during the test, including 

those who may be at another location. For each person, the list must include 

name, system user identification, company affiliation, title, and role (even if

an observer) during the test. The vendor will provide this at the test setup in

terview session, and will update it when the test is executed, including any re

test. 

• Test questions. Vendors may submit questions on the test materials for clari-

 19 
Qualifying a Product 

• Table config the configuration of 

tables will be provided in the test plan. 

mes. 

4.2.3 FSIO w

During the test setup interview, the FSIO

• Verify that the vendor’s system a on 

provided in the application, notin

am

, co

 u nding

ur ve 

r’s roissin e 

 be

tion provided might be incomple to 

configure each software product. sponsible for assessing the 

ptim diuce

e vendor supplied earlier, such as approved alare

 will stem 

pd

he s rther evidence 

uration. Requests for information related to 

See the Appendix for related time fra

 Conducts Setup Intervie

 test team will do the following: 

nd architecture conform to the descripti

g any differences. 

• Verify the software product n

• As part of the test application

the required infrastructure to be

dor in the application), inclu

software products. 

e and version. 

nfirm the required application software and

sed during the test (as attested to by the ve

 all applicable modules and any third-party 

• Validate that complete and acc

been established in the vendo

vided by the vendor. Any m

unobserved test execution can

ate setup data and beginning balances ha

test environment, on the basis of reports p

g or incorrect data must be corrected befor

gin. FSIO recognizes that the setup informate in terms of all the information required

 The vendor is re

test scripts to determine the o

cated transactions and to prod

• Confirm the information that th

al configuration needed to process the in

 the expected results. 

ternate test procedures, hardw

tion. (During the test, FSIO

operation and request vendor u

• Observe a demonstration of t

 components, and system table configuracompare this information to observed sy

ates, as appropriate.) 

oftware by the vendor, for fu

that operational system software exists and can perform the functions that satisfy mandatory core financial system requirements. This demonstration will 

also familiarize FSIO with the operation of the software and thus facilitate efficient use of testing time. Software demonstrations are to be scripted by the

vendor. 

The vendor scripts the demonstration using the site, architecture, and platform of its 

choosing. To show the operational system, the vendor should script the demonstrati

to display the software’s ability to accomplish several mandatory requirements such

as the software’s ability to meet the funds control requirements. To familiarize FSIO

with the system, the vendor should also demonstrate user interaction with the applic

tion, data input screens, system configuration, maintenance and setup, security, and 

reporting capabilities. FSIO realizes that the demonstration is time constrained and 

not comprehensive. 

on 

aThis demonstration of software capability does not take the place of a qualification 

test. 

 20 
Qualifying a Product 

4.2.4 FSIO

At t 

 Conducts First Readiness Assessment 

his point, FSIO also performs the first of two test readiness assessments based on 

the information collected at the setup interview and observations during the software 

demonstration. Test readiness criteria are published as part of the test materials for 

eac s h te t. Readiness assessments are conducted to evaluate whether the vendor is 

prepared to start the test and to successfully meet at least 90 percent of the tested requirements on the first attempt. Vendors that are determined to not be ready will be 

resc d he uled, at which time test readiness will be reevaluated. 

4.3 CONDUC

FSIO w endorprovided hardware and software at a time scheduled by the vendor and FSIO. The 

The test plan is a separate document that provides specifics about the conduct and 

The vendor is responsible for determining how to meet the expected results, in accorst team will note the processing of each mandatory test 

step, but completion of processing of the test step does not imply that the vendor has 

ediately notify the vendor that a requirement has 

not e

proce s d, 

the test 

execute

requirem

4.3.1 General Test Requirements 

Ven r

•

•

TING THE TEST

ill conduct the test at a vendor-provided site and location using v

vendor is expected to provide the test team with workspace to accommodate a team

of up to four people. The office space should provide privacy and allow the test team

adequate table or desk space to set up computers and review documents. A highspeed Internet connection and a printer should also be provided for use by the FSIO 

test team. 

execution of the test. FSIO will provide data for the test in the form of appendixes to 

the test plans. The vendor is responsible for data entry into the system as well as its 

total operation. 

dance with the published mandatory requirements, and for executing each of the test 

steps. During the test, the te

passed that step. 

FSIO is the final authority on the interpretation of all aspects of the test, including 

mandatory requirements, the test application, the test policy, the test plan, and all related test products. 

FSIO is under no obligation to imm

be n met during the execution of the test. If the test team discovers, after the 

s ing of a test step is completed, that the expected result has not been achieve

team may notify the vendor during the test that the test step was improperly 

d. This information is provided to the vendor to help the vendor avoid failed 

ents in subsequent steps. 

do s are required to meet certain general requirements in the conduct of the test: 

Do not use journal vouchers 

Provide access log 

 21 
Qualifying a Product 

• Produce and provide report outputs immediately 

• Execute test steps to demonstrate compliance with requirements. 

These requirements are described further in the sections below. 

 system 

pro s

4.3.1.2 Access Log 

The s ndor’s initial system setup 

thro h

the v n

ment, in

people will be called “test participants”; substitutions can be m

notifies

Onl

base an

execute

tivity d

4.3.1.3 Report Prod

During is required 

to b r

FSIO T

complia

FSIO w

same as m prescreening or 

cha n

requ t

4.3.1.4 Test Step E

While F

keying ce a test step has been executed, the result achieved is the basis for 

4.3.2 Vend

ng involves test steps that are not subject to FSIO observation 

(i.e., unobserved testing) but are still subject to FSIO evaluation against expected 

4.3.1.1 Journal Vouchers 

FSIO expects that the software will meet expected results through standard

ces ing, without the use of journal vouchers. 

 te t period is defined as the time period from the ve

ug test completion, including any retest. As part of the setup interview session, 

e dor is required to identify all people who will have access to the test environcluding their applicable system identification. For purposes of the test, these 

ade after the vendor 

 FSIO. 

y authorized individuals are allowed to access the application, including its datad configuration parameters, for the duration of the test period. The vendor will

 necessary programs to produce an automatic log of all system access and acuring the test period in a manner consistent with the core requirements. 

uction 

the observed processing phases of the test, when a report or query

e p oduced, the vendor must deliver all specified outputs immediately, directly to 

. he vendor may then choose to validate the output before FSIO evaluates it for 

nce with the tested requirements. If a second report is run by the vendor, 

ill verify that any differences between the first and second reports are the 

 those found by the vendor. The vendor is prohibited fro

ngi g queries or reports prior to FSIO receiving them. FSIO reserves the right to 

es additional queries or reports at any time during the test. 

xecution 

SIO will hold harmless the vendor for correction of simple mistakes, such as 

errors, on

FSIO evaluation and scoring. Unless FSIO determines that the vendor made a simple

mistake, results not in conformance with that expected count as an executed test step. 

FSIO will make the official pass/fail decision for the associated requirements based 

on the test observations and the posttest detailed evaluation of results. 

or Completes Unobserved Processing 

Unobserved processi

 22 
Qualifying a Product 

results. The time frames allowed for test execution are determined by FSIO for each

test version. Applicable test execution time frames are published in the test plan. 

sually not present at 

ng the test and how to 

s earlier. When the test 

processing portion of the test is finished, even if 

t expired. 

4.3.3 FSIO

ults 

y FSIO. Other expected results are specified in the individual test steps. 

4.3.4 FSIO Conducts Second Readiness Assessment 

During the unobserved processing portion of the test, FSIO is u

the vendor site, but is available to respond to questions regardi

proceed if the vendor experiences problems or perceived test step failures. 

Unobserved processing often consists of multiple cycles, corresponding to accounting

periods. Each cycle comprises test steps in which financial transactions, queries, and

reports are processed by the vendor. 

Unobserved processing ends either at the end of the days allowed for in the test plan 

or when the vendor has executed the last test step, whichever i

steps are completed, the unobserved 

the days allowed for the test have no

 Validates Unobserved Processing Results 

As each cycle is completed, the vendor submits the test step results to FSIO for validation against the expected results. These results include the reports, query results, 

screen prints, and other data specified by FSIO in the test script. The expected res

for cycle end reports, such as trial balances and transaction registers, are published 

separately b

Discrepancies between the vendor’s general ledger results and FSIO’s expected results, other than those allowed by approved alternate test procedures, are to be resolved before the vendor begins unobserved processing for the next cycle. 

Before starting observed testing, FSIO will conduct the second of two test readiness 

assessments. Based on the results of the unobserved testing, FSIO will determine 

whether the vendor is ready to proceed with the test. Vendors that are determined to 

not be ready will be rescheduled. The vendor’s readiness to start observed testing will

be reassessed at the start of the rescheduled test. 

4.3.5 Vend

Observed testing involves test steps that are subject to FSIO observation. The time 

In addition to scripted test steps, FSIO may use the following approaches in conductng 

or Starts Observed Processing 

frames allowed for test execution are determined by FSIO for each test. Applicable 

test execution time frames are published in the test plan. 

FSIO observes the entering of the observed transaction processing test steps and receives the expected results at the end of each cycle. 

ing the observed portion of the test: 

• Use of changed data in test cases 

• Reserved transaction processi

 23 
Qualifying a Product 

• Test restart. 

The a t follow. 

4.3.5.1 Use of C

e vendor’s core financial system software is expected to produce results that match the expected results 

e FSIO website. However, either prior to or during the test, the government reserves the right to change data values (such as fund codes, object classes, 

e vendor’s software will change 

appropriately. (For example, if a payee name is changed from Jones to Smith or a 

dger account code is changed, the system should reflect the accurate information based on the latest available test data.) 

4.3.5.2 Reser

ts to process. These scripts will be similar to those the vendor has already processed, but with different document numbers and data values. The vendor 

If this cycle occurs mid-test, vendors will restore the system to the backup before 

 processing. This way, the cumulative expected results (e.g., general ledger reports and document balances) will not be affected. 

4.3.5.3 Test R

SIO 

c-

 a 

se a different starting 

point than that requested by FSIO because of the application’s programming logic. 

 must explain why the different starting point is needed. The decision on whether to accept the proposal will be made by FSIO. FSIO may also re4.3.6 Vend

e test plan or 

when the vendor has executed the last test step, whichever is earlier. When the test 

steps are completed, the observed processing portion of the test is finished, even if 

the days allowed for the test have not expired. 

se pproaches are described further in the sections tha

hanged Data in Test Cases 

FSIO will use the test steps as published in the FSIO website. Th

as published in th

general ledger accounts, posting model, document numbers or amounts) from those 

published, and expects that results produced by th

general le

ved Transaction Processing 

At FSIO’s discretion, one cycle of the test will be designated for reserved transaction 

processing. At the beginning of this cycle, FSIO will provide the vendor with additional test scrip

will perform a system backup, produce selected queries or reports, enter the test 

scripts, and provide further queries or reports. FSIO will evaluate the results as part 

of the overall scoring of the test. 

continuing regular

estart 

If a test step is improperly executed, or if the test must be halted for any reason, F

will coordinate with the vendor to determine at what point in the test execution pro

ess that the test may resume. However, FSIO reserves the right to resume the test at

test step different from the one in which the test was stopped, including at a “system 

backup” step as specified in the test plan. The vendor may propo

However, the vendor

quest that the vendor produce system reports to verify the condition of the data at the 

point where execution was halted. 

or Completes Observed Processing 

Observed processing ends either at the end of the days allowed for in th

 24 
Qualifying a Product 

The execution of test will end when (1) all test steps have be

quired outputs are delivered to FSIO, or (2) the maximum tim

en executed and the ree frame allowed for test 

4.4 GRA

ecific 

4.4.1 FSIO Validates Observed Processing Results 

 passed the requirement. Complete 

al ledger effects, 

y Treasury), and 

-

rating environment, including software, hardware, and data, is isolated and not modified or updated. 

of the mandatory core financial sysstep execution is reached, whichever is earlier. FSIO will not accept test results generated after test execution ends.

DING THE TEST

Once the vendor has executed all test steps and delivered all outputs specified in the 

test plan, FSIO will validate the results and will determine a pass/fail outcome by 

tested requirement. FSIO will also notify the vendor of the need to retest sp

functionality, if the software has not passed one or more of the mandatory requirements tested. 

Each mandatory requirement will be graded as either pass or fail, as evidenced by 

successful test step execution. The software passes the test when the software passes 

every mandatory requirement tested. During the test, FSIO does not formally acknowledge whether or not the software actually

evaluation of test step results requires detailed examination of gener

subsidiary records, external file processing (such as payment files b

other test outputs such as reports and queries. After the completion of the test, the 

FSIO test team performs this thorough evaluation, which may take a considerable 

amount of time (sometimes in excess of 30 days). Without exception, the test team 

will make the final pass/fail determination for each requirement based on its posttest 

detailed examination of results. 

During the evaluation period, FSIO may need clarification of test results. As appropriate, FSIO will contact the vendor’s team lead to present any questions for resolu

tion. 

During the evaluation period, the vendor should ensure that the ope

This will allow the vendor and FSIO to check the system when there are questions 

about test results and also provide the appropriate environment to retest any test 

steps. 

Tested products are required to meet 100 percent 

tem requirements, whether or not FSIO actually tests that the product meets the requirement. 

4.4.1.1 Gradi

but may provide feedback for consideration by the vendor. All final pass/fail 

d 

test examination of results. 

Information provided by FSIO to the vendor either prior to or during the test may not 

be construed as determination of the outcome of the execution of the related test 

steps. 

ng Full and Incremental Tests 

During the test, FSIO will not direct the vendor’s proposed method to meet a test step 

determinations will be made solely on the results of the execution of the test steps an

the results produced, as reviewed during FSIO’s post

 25 
Qualifying a Product 

To pass a full test and receive a certificate of compliance, or to pass an incremental 

test and maintain the certificate, the tested product must pass 100 percent of the mandatory requirements tested. If the product does not pass the test, FSIO will issue a 

noti t age of requirements that were successfully 

met. 

If the product passes at least 90 percent of the tested requirements on the first attempt, 

it is ig -time retest. Retesting generally involves reexecuting selected 

portions of the test. Refer to the section on conducting a retest for further information 

 of the tested requirements, the test outcome 

will be considered to be a failure, and the product will be designated nonqualifying. 

4.4.1.2 Gradi

n 

4.4.2 FSIO

r full and incremental tests, FSIO 

ed to complete a test. The notification will 

4.5 CON

ed when a vendor has been notified by FSIO of the 

need to retest one or more requirements: 

•

endor completes pretest activities 

ce o the vendor indicating the percent

 el ible for a one

on this test process. 

If the product passes less than 90 percent

ng a Version Test 

If FSIO determines that the documented changes do not adversely affect prior-versio

functionality, the new version will be deemed qualified, and a certificate for that version will be issued. 

 Notifies Vendor of Need to Retest 

Upon completion of the validation of test results fo

will notify the vendor if a retest is need

include a list of requirements and the specific test steps in which failure was evidenced. FSIO will also indicate the percentage of requirements that were successfully 

met. 

DUCTING A RETEST

The following activities are requir

• FSIO schedules retest 

Vendor submits retest plan 

• V

• FSIO conducts retest readiness assessment 

• Vendor completes the retest 

• FSIO validates the retest results. 

The retest activities are described further in the sections below. 

 26 
Qualifying a Product 

4.5.1 FSIO Schedules Retest 

The vendor will propose to FSIO a date for the retest. FSIO will schedule the retest 

based on the dates requested by the vendor, or at FSIO’s earliest convenience. A 

separate test application is not required. 

4.5.2 Vendor Submits Retest Plan 

est plan. The retest plan must also identify all required setup data, test steps to be executed, and expected results. FSIO will work with 

st approach and content. 

re 

4.5.3 Vendor Completes Pretest Activities 

g the product to corta. 

4.5.4 FSIO

In preparing for a retest, vendors must submit a recommended approach (retest plan). 

This approach must fully test all requirements subject to retesting, in a manner consistent with FSIO’s original full t

the vendor to finalize the rete

Retests are typically conducted in a single phase that includes test steps subject to 

FSIO observation (i.e., observed testing). The time frames allowed for test execution

are predetermined by FSIO for each retest. Applicable test execution time frames a

published in the final retest plan. 

Before executing a retest, the vendor is responsible for upgradin

rect identified deficiencies, preparing the test environment, and loading all setup da

Vendors may also submit alternate test procedure requests for FSIO’s approval. 

 Conducts Retest Readiness Assessment 

Before starting a retest, FSIO will confirm that the vendor is ready to proceed. Vendors that are determined to not be ready will be rescheduled. 

4.5.5 Vend

oved test plan and provide FSIO with all expected 

results. Retests may be conducted using web-based software demonstration tools. 

4.5.6 FSIO

O 

4.6 PERFORMING POSTTEST ACTIVITIES

If the test is successfully passed, the following activities also occur: 

• Vendor submits product documentation 

or Completes the Retest 

The vendor will execute the appr

 Validates Retest Results 

Once the vendor has executed all test steps and delivered all specified outputs, FSI

will validate the results and will determine a pass/fail outcome by tested requirement. 

To pass the retest, the product must meet 100 percent of the requirements subject to 

retesting. 

After FSIO grades the test, FSIO notifies the vendor of the test results. 

 27 
Qualifying a Product 

• FSIO publishes successful test results 

4.6.1 FSIO Notifies Vendor of Test Results 

 and the reason the results were judged noncompliant. 

4.6.1.1 Passe

 calendar days following the completion of test grading or the 

submission of other required documentation, whichever is later. The certificate of 

For software products that pass the incremental test, the certificate of compliance reFSIO ISSUANC NS

s that pass a full test or a ver-

• FSIO publishes alternate test procedures and supplemental information. 

If the test is failed, the vendor may apply to test a new version. 

These posttest activities are described further in the sections below. 

After review and reconciliation of test output and processing results, FSIO will provide an official pass/fail determination to the vendor in writing. If the official determination is fail, an attachment to the letter will identify the requirements that were 

failed, the FSIO test team’s observations for those requirements and associated test 

steps,

d the Test 

For software products that pass the full or version test, FSIO will issue a certificate of 

compliance within 30

compliance qualifies the software as meeting FSIO requirements for a core financial 

system. However, the government may require incremental testing to maintain the 

certificate. 

mains in effect. 

E OF CERTIFICATIO

FSIO will issue a certificate of compliance for product

sion test. FSIO-issued certificates are specific to the product version tested and qualify the named product as meeting all mandatory requirements in force at the time of 

the test. Certificates will remain in effect as long as the vendor continues to successfully complete all subsequent full and incremental tests required by FSIO. Certificates will be revoked for products that fail or do not take required full and incremental tests. 

A FSIO-issued certificate that results

ject product version meets all manda

 from passing these tests establishes that the subtory core financial system requirements in force 

at the time of a given test. 

CERTIFICATION

A v one core financial system software product certified at 

any given time. 

4.6.1.2 Failed the Test 

If, after completing the test process (including retesting), the product does not 

suc s ents, or if the vendor fails to 

 OF MULTIPLE PRODUCTS

endor may have more than 

ces fully meet 100 percent of the tested requirem

 28 
Qualifying a Product 

com e equired time frame (within 12 months of 

submitting the application for a full test or within 6 months of submitting the 

app a st outcome will be considered to be a 

failure, and the product will be designated nonqualifying. Any existing certificate will 

 Documentation and 

Inform

pass 

pl te the test process within the r

lic tion for an incremental test), the te

be revoked. 

4.6.2 Vendor Submits Product

ation on Value-Added Features 

A full set of product documentation, updated to reflect software changes made to 

the test, is required within 30 calendar days of notification of successful test results. 

This software documentation includes user guides, implementation guides, and system setup guides for all software modules that the vendor used to take the test. It may 

be provided in electronic format or made available online (with any necessary passwords provided to FSIO). 

If the vendor chooses to provide information on the software product’s value-added 

features, FSIO will provide a link on its website to the vendor’s website for this information. The Value-Added Features form is available on the FSIO website. FSIO 

does not verify the accuracy of the information posted by the vendor. 

4.6.3 FSIO

rmation on its website for currently certified 

• Vendor name 

 its website. When a 

certificate has been issued for a version update, the certificate for the prior version is 

 website to reflect the new version that is cer4.6.4 Vend

d software and related documentation avail-

 Publishes Successful Test Results 

FSIO publishes the following info

software products: 

• Product name 

• Version tested 

• Recent test results. 

FSIO will maintain the history of all software certifications on

no longer in effect. FSIO will update its

tified. 

or Provides Software to the Government 

The vendor will make the newly qualifie

able to government customers within 6 months of notification of successful test results. 

 29 
Qualifying a Product 

4.6.5 FSIO Publishes Alternate Test Procedures 

and Supplemental Information 

Over the course of testing, FSIO collects product and test execution information 

IO may elect to 

make this information available to the public. The following are the main categories 

of o r

stricted dissemination of system documentation and other proprietary information to which FSIO test-

 that do not receive a certificate of 

4.6.5.1 Suppl

ct description information: 

se history 

prepare a Supplemental Information report to 

4.6.5.2 Vendo

als. 

with FSIO internal observations and information dissemina-

(other product information) that it considers to be nonproprietary. FS

the product information: 

• Alternate test procedures 

• Supplemental information. 

FSIO will generally respect written requests by the vendor for re

ers may be exposed during the qualification process. 

FSIO will not publish any information on products

compliance, nor will FSIO publish information on applications received, applications 

rejected, or test scheduling. 

emental Information 

During a test, FSIO will collect the following types of produ

• Product relea

• Software used for the test 

• Hardware and network infrastructure used for the test 

• Documentation sources 

• Product architecture 

• Functional observations. 

At the conclusion of a test, FSIO will 

document findings and observations in each of these categories. 

r Review of Product Information 

FSIO will provide the vendor with advance notice about product information intended for publication and the opportunity to review and comment on these materi

FSIO will attempt to incorporate all vendor-provided comments to the extent such 

comments are consistent 

tion standards. 

 30 
Qualifying a Product 

4.6.6 Vendor Applies to Test New Version after Failure 

To protect the government from a continued commitment of resources to test nonqualifying core financial systems, vendors must wait a minimum of 3 months before 

applying to retest a nonqualifying product, and demonstrate that the software is ready. 

FSIO may reduce the waiting period, at its discretion. To continue the test process at 

that time, the vendor must submit an application to test a new or updated version of 

the product and retake the full test in its entirety. 

 31 

Appendix

Table A-1 provides the time frames and responsibilities for testing activities. 

Test Activities and Time frames

 33 
Qualifying a Product 

 34 
Table A-1. Time Frames for Testing Activities

Time Frame by Test Type 

Activity 

Responsible 

Entity Full Test Incremental Test Version Test 

Test Development 

1) Start the test development process. FSIO Upon issuance of new and revised requirements Upon issuance of new and revised requirements NA 2) Notify vendors of 

need to test. 

FSIO At any time after issuance of 

new requirements 

At any time after issuance of new requirements 

NA 

3) Publish test plan exposure draft. FSIO Depends on extent of testing Depends on extent of testing NA 4) Comment on test 

plan exposure draft. 

Vendors and Agencies 

60 days after publication of test plan exposure draft 

30 days after publication of test plan exposure draft 

NA 

5) Publish final test 

plan. 

FSIO Depends on extent of comments Depends on extent of com- ments NA 

Test Execution 

6) Submit test application. Vendor Within 30 calendar days of publication of final test plan 

or at any time for new software products 

Within 30 calendar days of publication of final test plan for qualified software products 

At any time 

7) Notify vendor of application acceptance 

or rejection. 

FSIO Within 30 calendar days of 

receipt of the application or receipt of additional information, whichever is later 

Within 30 calendar days of receipt of the application or receipt of additional information, whichever is later 

Within 30 calendar days of receipt of the application or receipt of additional information, whichever is later 

8) Review release 

notes. Request version demonstration, if needed. 

FSIO NA NA Within 30 calendar days of 

receipt of test application 

9) Provide version demonstration. Vendor NA NA Within 30 calendar days of FSIO request 

 35 
Table A-1. Time Frames for Testing Activities

Time Frame by Test Type 

Activity 

Responsible 

Entity Full Test Incremental Test Version Test 

10) Evaluate version 

demonstration, if needed. 

FSIO NA NA Within 30 calendar days of 

end of version demonstratio

11) Notify vendors of 

need to test. 

FSIO NA NA Upon completion of the review of the release notes, o

evaluation of a version demonstration, whichever is late

12) Schedule observed 

testing. 

FSIO Within 30 days of application 

acceptance 

Within 30 days of application acceptance 

Same as Full Test 

13) Submit alternate test 

procedure requests. 

Vendor At least 1 week prior to setup 

interview 

At least 1 week prior to the setup interview 

Same as Full Test 

14) Conduct setup and 

other pretest activities. 

Vendor Prior to setup interview Prior to setup interview Same as Full Test 

15) Conduct setup interview. FSIO At least 2 business days prior to receipt of first unobserved 

test results 

At least 2 business days prior to receipt of first unobserved test results 

Same as Full Test 

16) Conduct first readiness assessment. FSIO At least 2 business days prior to receipt of first unobserved 

test results 

At least 2 business days prior to receipt of first unobserved test results 

Same as Full Test 

17) Complete unobserved processing. Vendor At least 6 business days prior to start of observed processing At least 6 business days prior to start of observed process- ing Same as Full Test 18) Validate unobserved 

processing results. 

FSIO Prior to start of observed 

processing 

Prior to start of observed processing 

Same as Full Test 

19) Conduct second 

readiness assessment. 

FSIO After validation of unobserved testing results and 

before starting observed processing 

After validation of unobserved testing results and before starting observed processing 

Same as Full Test 

 36 
Table A-1. Time Frames for Testing Activities

Time Frame by Test Type 

Activity 

Responsible 

Entity Full Test Incremental Test Version Test 

20) Start observed pr

essing. 

oc- Vendor Within 6 months of publication of test plan Within 3 months of publica- tion of test plan Same as Full Test 

21) Complete observe

processing. 

d Vendor As specified in test plan As specified in test plan Same as Full Test 

22) Validate observed 

processing results. 

FSIO As needed As needed Same as Full Test 

23) Notify vendor of need 

to retest. 

FSIO Upon completion of test results validation Upon completion of test re- sults validation Same as Full Test 

24) Schedule retest. FSIO Within 30 calendar days of Same as Full Test 

notice to retest 

Within 30 calendar days of notice to retest 

25) Submit retest plan. Vendor Within 30 calendar days of 

notice to retest notice to retest 

Same as Full Test 

Within 30 calendar days of 

26) Complete pre-retest Vendor At least 1 business day be- At least 1 business day be- Same as Full Test 

activities. fore starting retest fore starting retest 

27) Conduct retest readiness assessment. FSIO At least 1 business day be- ame as Full Test fore starting retest At least 1 business day be- fore starting retest S28) Complete retest. Vendor 

Within 60 calendar days of 

notice to retest and within 12months from submission of the test application 

Within 60 calendar days of notice to retest and within 6 months from submission of the test application 

Same as Full Test 

29) Validate retest results. FSIO As needed As needed Same as Full Test 30) FSIO Upon completion of test results validation Upon completion of test re- sults validation Same as Full Test 

 Notify vendor of test results. 

 37 
Table A-1. Time Frames for Testing Activities

Time Frame by Test Type 

Activity 

Responsible 

Entity Full Test Incremental Test Version Test 

31) Submit product 

documentation aninformation on valadded features. 

d 

uer in 30 days of notification of successful test results in 30 days of notification of successful test results 

Vendo With With Same as Full Test 

32) Publish successful 

test results. 

FSIO After notification of test results After notification of test re- sults Same as Full Test 

33) Provide software to 

government customers. 

Vendor Within 6 months of notification of successful test results l test results Within 6 months of notifica- tion of successfu Same as Full Test 

34) st 

dures and supFSIO After publication of success- ful test results After publication of success- ful test results Publish alternate te Same as Full Test proce

plemental information. 

35) Apply to test new Vendor of 3 months after of 3 months after Same as Full Test 

version after failure. 

A minimum notification of unsuccessful 

test results 

A minimum notification of unsuccessful test results 

 38