![](media/image1.wmf){width="8.05in" height="10.630555555555556in"}

#  {#section .unnumbered}

# Contents {#contents .unnumbered}

**Abbreviations** vi

**Foreword** 1

**Chapter 1 Introduction** 3

> **1.1** **Purpose of This Document** 3
>
> **1.2** **Audience** 3
>
> **1.3** **Supplemental Testing Materials** 4
>
> **1.4** **Document Organization** 4

**Chapter 2 Overview of Qualification Testing 5**

> **2.1** **Scope of Qualification Testing 5**
>
> **2.2** **Limitations of Qualification Testing 6**
>
> **2.3** **Relationship to Agency Procurement Activities 6**
>
> **2.4** **The FSIO Test Team 7**
>
> **2.5** **Test Types 7**
>
> **2.6** **Timing of the Test 8**
>
> **2.7** **Duration of the Test 10**

**Chapter 3 Developing a Qualification Test 11**

> **3.1** **Agency Outreach 11**
>
> **3.2** **Requirements Identification 11**
>
> **3.3** **Test Development** 11

**Chapter 4 Qualifying a Product 13**

> **4.1** **Applying for the Test 13**
>
> **4.2** **Exchanging Pretest Information 17**
>
> **4.3** **Conducting the Test 21**
>
> **4.4** **Grading the Test 25**
>
> **4.5** **Conducting a Retest 26**
>
> **4.6** **Performing Posttest Activities** **27**

**Appendix---Test Activities and Time Frames**

# Abbreviations {#abbreviations .unnumbered}

CFOC Chief Financial Officers Council

FAS Federal Acquisition Service (GSA)

FFMIA Federal Financial Management Improvement Act of 1996

FMS Financial Management Service (Treasury)

FSIO Financial Systems Integration Office

GSA General Services Administration

OMB Office of Management and Budget

Treasury Department of the Treasury

U.S. SGL U.S. Standard General Ledger

# Foreword {#foreword .unnumbered}

This document specifies the core financial system qualification test
policy and procedures that will be followed by the government starting
in 2007. This policy replaces the one previously in place, which was
last updated in September 2002. The following key policy changes are
reflected in the sections indicated, and highlighted in the text with a
side bar:

-   Eliminates the policy to fully retest qualified products on a 3-year
    cycle. Certifications issued to qualified software products will
    continue to be active as long as subsequent full and incremental
    tests are taken and passed. (Section 4.6.1.1, Passed the Test)

-   Revises the criterion for passing the test. Previously, the
    criterion was successful execution of 100 percent of the test steps.
    Products must now show they can meet 100 percent of the mandatory
    requirements tested, as evidenced by successful test step execution.
    Note that alternate test procedures are still allowed. (Section
    4.4.1, FSIO Validates Observed Processing Results)

-   Changes the requirement for vendor demonstrations of product
    functionality. Demonstrations at the start of a test are no longer
    optional. These demonstrations will be used as part of the first
    test readiness assessment of the software's ability to meet the
    current mandatory requirements. These demonstrations also
    familiarize FSIO testers with the software product, increasing
    testing efficiency. (Section 4.2.3, FSIO Conducts Setup Interview)

-   Requires vendors to demonstrate test readiness and establishes test
    readiness criteria. Vendors that are determined not ready will be
    rescheduled. (Section 4.2.4, FSIO Conducts First Readiness
    Assessment; Section 4.3.4, FSIO Conducts Second Readiness
    Assessment; Section 4.5.4, FSIO Conducts Retest Readiness
    Assessment)

-   Expands incremental test coverage to include any functional
    capability of agency interest, supported by an existing mandatory
    requirement or newly mandated requirement, driven by external agency
    needs. Previously, only new mandated requirements were tested, such
    as the Central Contractor Registration. (Section 2.5.2, Incremental
    Test)

-   Adds a policy requiring full testing of version upgrades when
    version demonstrations are determined by FSIO to be insufficient
    evidence of continued compliance with all mandatory core financial
    system requirements. (Section 2.6.3, Timing of a Version Test;
    Section 4.1.6, FSIO Notifies Vendor of Need to Test)

-   Reduces the time from 1 year to 3 months that vendors must wait
    before applying to retest a nonqualifying product in a full retest.
    (Section 2.6.5, Timing of a Test of a Nonqualifying Software
    Product; Section 4.6.6, Vendor Applies to Test New Version after
    Failure)

-   Eliminates the requirement to submit a separate test application for
    a retest. (Section 4.5.1, FSIO Schedules Retest)

-   Changes the term "variance" to "alternate test procedure." Vendors
    may request permission to use an alternate test procedure in the
    execution of a test; FSIO may grant permission under conditions
    similar to those previously granted as variances. (Section 4.2.1,
    Vendor Submits Alternate Test Procedure Requests)

-   Eliminates the requirement to submit user documentation for software
    products upon application for testing. Instead, documentation,
    updated to reflect software changes made to pass the tests, must be
    submitted within 30 days of notification of successful test results.
    (Section 4.6.2, Vendor Submits Product Documentation)

-   Requires vendors to make the newly qualified software and related
    documentation available to government customers within 6 months of
    notification of successful test results. (Section 4.6.4, Vendor
    Provides Software to the Government)

-   Requires vendors to save the configuration used to pass the test and
    make information about it available at agency request for subsequent
    demonstration of software functionality as configured for the test.
    (Chapter 4 Qualifying a Product)

-   Changes the timing of vendors' submissions of assurance statements
    to be at the time of test application, instead of 2 weeks after
    notification of a successful test outcome. These assurance
    statements assert that all of the changes made to a product during
    the test process will be incorporated into the product version to be
    named on the certificate and that the product version will be
    available for agency acquisition. (Section 4.1.1, Vendor Submits
    Test Application; Section 4.1.2.1, Application Screening Criteria)

-   Requires vendors to provide an additional assurance that the
    software product has been fully tested (e.g., unit test, integration
    test, regression test). (Section 4.1.1, Vendor Submits Test
    Application; Section 4.1.2.1, Application Screening Criteria)

-   Eliminates the collection and publication of Report Sources
    information, which documented how each test output report was
    generated, since most are now produced from third-party report
    writer tools which are specified in the application.

-   Changes the timing for when information is collected on the
    value-added features provided by tested software products. The
    information will be posted on the vendors' websites, with links
    provided on the FSIO website. (Section 4.6.2, Vendor Submits Product
    Documentation and Information on Value-Added Features)

This document has been restructured to improve readability. Policies are
now organized around the test process.

# Introduction

The Federal Financial Management Improvement Act of 1996 (FFMIA)
mandates that agencies implement financial system software that
substantially complies with the following:

-   Federal Accounting Standards Advisory Board statements of Federal
    financial accounting concepts and standards

-   Established government-wide system requirements

-   U.S. Standard General Ledger (U.S. SGL) at the transaction level.

Office of Management and Budget (OMB) Circular A-127, *Financial
Management Systems*, mandates that Federal agencies acquire only
financial system software and services that comply with *Core Financial
System Requirements.* The requirements, composed and published by the
Financial Systems Integration Office (FSIO), are the established
government-wide system requirements referred to in the FFMIA.

Circular A-127 further assigns responsibility for requirements
compliance testing to the Chief Financial Officers Council (CFOC).
Actual product testing is carried out on behalf of the CFOC by FSIO.

FSIO develops policy for and manages the core financial system
qualification process. As part of the process, FSIO develops and
administers a qualification test to prospective vendors. FSIO is also
responsible for publications that document core financial system
functional and technical requirements.

Final determination that a product fully complies with established
government-wide requirements is made by the FSIO Executive. This
determination is based solely on the results of the formal software
testing program administered by the FSIO staff.

## Purpose of This Document

This document specifies the FSIO qualification test policies and
practices. The policies govern how FSIO tests the products offered by
commercial vendors. Detailed information about each test, such as
schedule, scope, and requirements to be tested, is provided in the test
plan specific to that test.

## Audience

The audience for this document includes two primary parties:

-   *Federal agencies*, on whose behalf the test policy is established
    and the testing is conducted by FSIO

-   *Commercial vendors*, who are required to adhere to these policies
    in order to obtain and maintain certification of their core
    financial system software products.

## Supplemental Testing Materials

This test policy applies to other FSIO testing materials mentioned in
this policy and published separately on the FSIO website
([http://www.fsio.gov](http://www.fsio.gov/)). These materials include
the

-   test application,

-   test plans,

-   format for alternate test procedure requests, and

-   format for test questions.

## Document Organization

The remainder of this document is organized in sections that further
describe qualification testing. The major sections are as follows:

-   Overview of qualification testing

-   Developing a qualification test

-   Qualifying a product, including conducting the test, evaluating the
    results, and publishing the resulting certifications.

# Overview of Qualification Testing

FSIO tests commercially available core financial system software
products to certify them for federal acquisition. FSIO testing is
designed to generate objective evidence that core financial system
products comply with mandatory government-wide core financial system
requirements.

The objective of this testing program is to enable agencies to acquire
new commercial core financial systems with the assurance that selected
products can be configured to handle common federal accounting,
reporting, and electronic business transactions. This reduces the risk
that agencies will acquire noncompliant or otherwise ineffective core
financial system software.

Other testing objectives include the following:

-   Promote consistency of core financial systems across the government.
    It identifies and requires a set of functional core financial system
    requirements used by all stakeholders. The testing process is
    designed to be comprehensive (incorporating all mandatory
    requirements), traceable to core requirements, and transparent.

-   Clarify requirements and provide examples to help agencies and
    vendors understand published requirements and deliver better
    solutions.

```{=html}
<!-- -->
```
-   Reduce the need for agencies to independently develop a test for
    common core financial system requirements, allowing them to focus on
    their own unique requirements. Agencies can reuse the test for
    software implemented in their specific environment.

-   Incrementally test commercial off-the-shelf software products over
    time. This helps ensure that the software offerings are adapted to
    meet new requirements as quickly as possible, with minimal cost to
    the government.

FSIO serves as a focal point to leverage federal resources in the test
of core financial systems for the benefit of all agencies.

## Scope of Qualification Testing

FSIO tests compliance with requirements published in *Core Financial
System Requirements*, as supplemented by incremental test requirements.
The test includes only functional capabilities; FSIO-administered tests
do not address performance or product suitability for use at a specific
agency.

## Limitations of Qualification Testing

The qualification test process is the entry point to the Federal
financial management system market for commercially available software
products. Though the test comprehensively checks for compliance with
core financial system requirements, it does not address a number of
other important factors in the selection of core financial system
software. The qualification test does not:

-   Assess whether a product is suitable for a specific agency's
    operating environment. Agencies differ dramatically in size,
    budgetary resources, and business practices. An agency may have
    unique agency-specific requirements in addition to the core
    financial system requirements.

-   Assess the product's ease of use. Each product, while meeting the
    mandatory FSIO core financial system requirements, has its own
    strengths and weaknesses, such as number of screens accessed to
    complete a transaction and "user friendliness," which may impact
    ultimate success in day-to-day operation at a specific agency.

-   Assess the product's computing performance. Performance depends on
    such things as compatibility with the enterprise architecture,
    response time, and transaction throughput capacity, which can be
    especially important when an agency has large volumes of transaction
    data.

-   Ensure that delivered products are installed and configured
    appropriately for an agency's operating environment. Although each
    vendor configures its product to pass the qualification test, this
    configuration is not necessarily the same one an agency uses. It is
    the agency's responsibility to test the product as implemented. An
    improperly configured system has the potential to give noncompliant
    test results.

## Relationship to Agency Procurement Activities

Qualification testing is independent of any agency procurement process.
FSIO certifies only that a named product has passed all FSIO-required
qualification tests. It is the vendor's responsibility to maintain this
certification for products offered to agencies over time. It is the
acquiring agency's responsibility to confirm that certifications of
proposed products are in good standing.[^1]

FSIO recognizes that agency contracts or orders for core financial
system products may require the successful bidder to submit a
FSIO-issued certificate as evidence of qualification. Vendors are
responsible for ensuring that the name and version of the software
product tested, and for which a certificate of compliance is issued, are
consistent with the name and version identified in contract vehicles.

A complete list of all product certifications can be found on the FSIO
website. The requirements that form the basis of testing, the test
materials, and the results of the FSIO testing process are also posted
on this website. FSIO notifies the GSA's Federal Acquisition Service
(FAS) of any changes to the list of product certifications.

## The FSIO Test Team

The FSIO staff directs the test team. Team members may consist of either
government employees (including FSIO personnel) or qualified contract
support personnel. The government takes reasonable steps to prevent
contract support personnel conflicts of interest and to safeguard vendor
proprietary information. The team works under the direction of FSIO and
a FSIO-designated on-site representative. The team members make
observations, gather information, and advise the FSIO on-site team
leader.

The team members interact with software vendor personnel for routine
planning and testing activities. The FSIO on-site representative
resolves issues after consultation with the vendor's designated senior
representative.

To make the test more operational/effective, other stakeholders are
involved when appropriate. As an example, the Department of the Treasury
(Treasury) Financial Management Service (FMS) has assisted with testing
interface files to determine their accuracy.

## Test Types

FSIO uses three types of qualification tests to qualify new products and
to enable vendors to maintain their product certifications over time:

-   Full test

-   Incremental test

-   Version test.

When a product does not pass a full or incremental test on the first
try, the product may be retested if FSIO determines that it is eligible
for retesting (that is, if the product passes at least 90 percent of the
tested requirements on the first attempt).

### Full Test

Full testing is used by FSIO to qualify new products and to retest
qualified products after significant requirement updates. A full test
allows FSIO to determine product compliance with mandatory requirements
in all functional areas as published in the current *Core Financial
System Requirements* and related FSIO-issued supplements. The functional
areas are as follows:

-   System management

-   General ledger management

-   Budgetary resource management

-   Payment management

-   Receivable management

-   Fund Balance with Treasury management

-   Cost management.

### Incremental Test

FSIO uses an incremental testing process to assess product compliance
with new requirements as they are issued. In planning an incremental
test, FSIO may incorporate new central-agency-mandated requirements and
any other mandatory requirements for which expanded testing is desired
based on input from agencies.

All certified products are subject to ongoing incremental testing in
order to maintain their qualification.

### Version Test

Version testing is used by FSIO to qualify new versions of qualified
products as they are released (issued) by the vendor. A version test
allows FSIO to assess product changes for potential impacts on
previously certified product functionality. New product versions will
continue to be evaluated prior to their release and may be subject to
full testing at FSIO's option.

## Timing of the Test

The timing of each test type is triggered by differing events, as
described below. Appendix A contains the time frames for developing the
test and qualifying software.

### Timing of a Full Test

The need for a full test of qualified software products is communicated
based on the publication of substantial requirement changes. FSIO gives
advance notice of testing when draft requirement updates are issued for
comment. Vendors are encouraged to begin making software changes based
on the draft requirements to maximize the time available to prepare for
the test. Formal notice of a full test may be given at any time after
the final requirements are published.

The vendor is responsible for product engineering and for the timing of
new product introductions. Before offering a new product to a Federal
agency, the vendor must apply for and successfully complete the full
test process.

A full test, including any retest, must be completed within 12 months of
submitting the test application.

### Timing of an Incremental Test

When FSIO determines that an incremental test is in the best interest of
the government, FSIO drafts any new requirements and requirement
revisions that are needed to describe the required functionality. The
vendor is responsible for updating its products to comply with all new
mandatory requirements. Vendors are encouraged to begin making software
changes based on the draft requirements to maximize the time available
to prepare for the test. Considering comments received on the draft
requirements, FSIO updates the requirements and issues them in final
form. FSIO then issues a notice of intent to conduct an incremental
test. Following this announcement, FSIO develops an incremental test
plan and the test materials that will accompany the plan.

An incremental test, including any retest, must be completed within 6
months of submitting the test application.

### Timing of a Version Test

Vendors are responsible for defining the mechanism and timing for
releasing new product versions. Whenever a vendor makes changes that
result in a uniquely identified version of a qualified product (e.g.,
v8.3) that can be acquired by an agency, either through a maintenance
upgrade process or as a new product installation, the vendor must apply
for a version test.

Upon receipt of a version test application, FSIO will review the
information provided and the release notes related to the version
update. If more information is needed, FSIO will request a product
demonstration. FSIO may qualify the new version based on a review of the
release notes alone or after the additional product demonstration. If
the changes to a software product are extensive, the vendor may be
required to apply for a full test.

### Timing of a Retest

Upon completion of the validation of test results for full and
incremental tests, FSIO will notify the vendor if a retest is needed to
complete a test. The notification will include a list of requirements to
be tested. Retests are to be scheduled within 30 calendar days of
receipt of notice from FSIO.

### Timing of a Test of a Nonqualifying Software Product

Vendors must wait a minimum of 3 months after notification of
unsuccessful test results before applying for a new test of a
nonqualifying software product.

## Duration of the Test

Time frames for vendor completion of each test process, from test
application through any required retest, are specified in Appendix A.
Time frames for vendor completion of unobserved and observed processing
are specified in the individual test plans published by FSIO. The start
of a full test is defined as the start of the execution of unobserved
test steps. The time frames allowed for test execution are determined by
FSIO for each test version. The test plans contain specifics on test
phases and applicable test execution time frames.

# Developing a Qualification Test

The FSIO testing program starts with agency outreach to identify or
update core financial system requirements and ends with formal system
testing of commercial software products.

## Agency Outreach

Agencies are integral to the success of the testing process. Common
agency needs drive the requirements identification process on the front
end. Agency representatives help establish test development priorities
and are the primary audience for published test results.

To help agencies stay engaged in the testing process, FSIO actively
solicits their input using a variety of special study groups, topical
workshops, open houses, surveys and key personnel interviews. The
resulting reports, requirements, and other testing-related materials are
published by FSIO.

## Requirements Identification

Federal financial managers rely on FSIO-published requirements as the
official reference describing the essential characteristics of their
automated systems. Auditors of federal financial management systems rely
on these requirements as their official reference for required core
financial system functionality, and vendors rely on them to help them
understand agency-processing needs, enabling them to develop improved
system solutions.

FSIO publishes new and revised requirements before the test date to
allow enough time for software vendors to update products to perform
newly required functions. The basic process followed by FSIO to update a
requirements document has three steps: (1) assemble a draft set of
updated requirements, (2) distribute a draft to stakeholders for
comment, and (3) publish resulting requirements in a final form as
approved by the CFOC. New and revised requirements for incremental tests
are published either as new editions of *Core Financial System
Requirements* or supplemental updates.

## Test Development

FSIO translates mandatory functional requirements into a series of test
steps that are suitable for execution in a wide variety of operating
environments. The goal is to develop a test that provides the government
with an unimpeachable basis for determining whether a product is
qualified for federal government use.

For each test, FSIO publishes a test plan, which contains instructions
on how to execute the test. The test materials, including setup data,
test scripts, and expected results, are published as appendixes to the
test plan and are posted on the FSIO website.

The test script contains test cases, each composed of several test
steps, based on the mandatory requirements. One or more test steps may
be needed to test each mandatory requirement.

The test is organized by accounting cycles, and includes a complete set
of expected results, including financial statements, transaction
registers, trial balances, and Treasury e-commerce files. Agencies and
vendors are invited to review the draft test materials before they are
finalized.

# Qualifying a Product

Qualification testing starts with a vendor application that identifies
the subject product. The process typically ends with the issuance of a
certificate of qualification. Vendors set up for and execute the test;
FSIO observes the test and reviews the actual results against expected
results.

In taking a test, vendors must configure their products to meet core
financial system requirements, enter the data, and generate the results.
Vendors provide the computing facilities and the software subject to
qualification.

To conserve government resources on products that lack functionality,
FSIO assesses test readiness at key points in the process and requires a
threshold for passing the initial test. A product is not certified
unless it meets 100% of the requirements. Vendors are initially required
to pass at least 90 percent of the requirements tested. The software is
then given a chance to retest the 10% of failed requirements that will
qualify it for certification. Software is considered to be nonqualifying
if it fails to meet more than 10 percent of the tested requirements. If
this threshold is not met, vendors must invest in further development
before proceeding with a test.

Product specifics (vendor name, software name, version number) for all
software products that passed the test are posted on the FSIO website.
Vendors are required to save the configuration used to pass the test and
make information about it available at agency request for subsequent
demonstration of software functionality as configured for the test.

Execution of the test includes the following processes:

-   Applying for the test

-   Exchanging pretest information

-   Conducting the test

-   Grading the test

-   Conducting a retest

-   Performing posttest activities.

The activities and related policies for each process are described
below.

## Applying for the Test

This section explains the purpose, uses, and processes associated with
applying for the core financial system qualification test. The actual
test application with instructions is published separately on the FSIO
website. The application form should be used for all tests.

### Vendor Submits Test Application

The qualification test and resulting certificate of compliance pertain
to a specific software product version (and not to a vendor). Software
vendors applying for the test must designate the software product name
and version in the application.

The application also provides the following information:

-   Administrative information used to schedule and conduct the test

-   Information for screening applications

-   Vendor assurance of the software's capability to meet mandatory
    requirements

-   Vendor assurance that the software has been fully tested (e.g., unit
    test, integration test, regression test)

-   Vendor assurance that all of the changes made to a product during
    the test process will be incorporated into the product version named
    on the certificate and that the product version will be available
    for agency acquisition.

When software changes have been made beyond those needed for an
incremental test, the vendor must submit separate applications for the
version and incremental tests. This will allow FSIO to evaluate the
changes separately. The version test will be conducted separately and
before the incremental test.

Applications for full tests of new software products and for version
tests of new software versions may be submitted at any time.
Applications for full and incremental tests of previously qualified
software products must be submitted within 30 calendar days of the
publication of final test materials.

### FSIO Notifies Vendor of Application Acceptance or Rejection

Upon receipt, FSIO reviews each application for completeness. If
additional information is needed, FSIO will contact the vendor.

FSIO also reviews the application in accordance with the screening
criteria described above. Within 30 calendar days of receipt of the
application or of additional information, whichever is later, FSIO will
notify the vendor whether the application has been accepted or rejected.

#### Application Screening Criteria

FSIO uses portions of the application to assess the capability of the
applicant software and determines with reasonable assurance that it is
potentially capable of satisfying the FSIO requirements. These portions
include the following:

1.  *Mandatory requirements assurance.* The vendor asserts that the
    candidate software can meet each mandatory requirement. The vendor's
    software is required to meet 100 percent of the mandatory
    requirements, whether or not FSIO actually tests that the software
    meets the requirement.

2.  *Testing assurance.* The vendor must provide a written assurance
    that the software product has been fully tested (e.g., unit test,
    integration test, regression test).

3.  *Updated software assurance.* The vendor must provide a written
    assurance to FSIO that all changes (e.g., fixes) made to a product
    during the test process will be incorporated into the product
    version named on the certificate and that the product version will
    be available for agency acquisition.

4.  *References.* FSIO uses the references as supplementary information
    to help determine whether the candidate system software has the
    capability to pass the test. Lack of references does not prejudice
    consideration of an application. FSIO may contact the references as
    part of the evaluation.

5.  *Release notes.* As part of a version or incremental test
    application, the vendor is required to fully disclose all material
    changes made to previously qualified products (e.g., release notes).

#### Acceptance of Application

If the application is accepted, FSIO states this event in writing and
schedules a mutually acceptable test period with the applicant.

#### Rejection of Application

Applications for software testing may be rejected for one or more of the
following reasons:

-   Failure to attest that the software product meets all mandatory
    requirements

-   Failure to complete the application in accordance with the
    instructions in this document and those on the application

-   Provision of inaccurate or false information[^2]

-   Absence of an officer's signature on the application.

FSIO will notify the vendor, in writing, of the reason for an
application's rejection and will return the application.

A vendor whose application is rejected by FSIO may reapply at any time.
A vendor resubmitting an application must complete all parts. The vendor
should attach a letter to the resubmitted application describing the
actions taken to correct the issues that resulted in the previous
rejection.

### FSIO Reviews Release Notes

For version or incremental test applications, FSIO will review the
release notes to assess the extent to which new functional capabilities
have been added and previous functionality has been affected.

### Vendor Provides Version Demonstration

For version test applications, FSIO may request a version demonstration
(prior to qualification testing) as further evidence that the product
can perform the functions needed to generally satisfy mandatory core
financial system requirements. When requested, version demonstrations
are to be scripted by the vendor. Version demonstrations *do not* take
the place of the test.

### FSIO Evaluates Version Demonstration

For version test applications, FSIO may qualify the new version based on
a review of the release notes alone or after the additional version
demonstration. If the changes to a software product are extensive, the
vendor may be required to apply for a full test. FSIO reserves the right
to make the final determination in this matter.

### FSIO Notifies Vendor of Need to Test

For version test applications, if FSIO determines that the new software
version does not require testing, it will issue a certificate for the
new version. FSIO will replace the certified version number on its
website with the latest certified version.

Full testing of version upgrades is required when the product
demonstration does not provide sufficient proof that the software meets
all the mandatory core financial system requirements tested. If FSIO
determines that the new version requires testing, the vendor will be
required to take a full test to qualify the new software version.

### FSIO Schedules Observed Testing

FSIO will schedule the observed test based on the dates requested by the
vendor, or at FSIO's earliest convenience. The scheduled date for
observed testing is subject to successful completion of the readiness
assessments (as described in the Conducting the Test process).
Unobserved testing typically occurs before observed testing.

## Exchanging Pretest Information

The vendor and FSIO will exchange information before the test is
conducted. FSIO will provide the vendors with the test materials
discussed below:

-   *Test plan.* Test plans contain live testing procedures for each
    subject test. Test plan appendixes are used to deliver setup data,
    test scripts, and expected outputs.

-   *Requirements trace matrix.* The requirements trace matrix is a
    report that presents the linkage between test steps and the core
    financial system requirements.

-   *Vendor test question format.* Once the test has been scheduled,
    FSIO will work with the vendor to resolve any questions the vendor
    may have on the test materials. It is the vendor's responsibility to
    understand the mandatory requirements and the test steps. FSIO will
    not advise the vendor on how to operate the software to obtain the
    desired results. However, before the test, FSIO will answer
    reasonable vendor questions about the test execution. The standard
    format that vendors must use in submitting test questions is
    provided on the FSIO website.

-   *Alternate test procedure request format.* FSIO will consider
    requests for exceptions to the test scripts individually, reviewing
    the proposed alternative and seeking clarification from the vendor,
    if necessary. The revised test script processing must be adequate
    for FSIO to perform regression testing necessary to ensure that the
    changes do not affect the integrity of the FSIO qualification test.
    FSIO will respond in writing to the vendor's request for use of the
    alternate test procedure prior to the test setup interview. The
    standard format that vendors must use in requesting an alternate
    test procedure is provided on the FSIO website.

### Vendor Submits Alternate Test Procedure Requests

To maintain the integrity of the test process and ensure that the
expected results are achieved, the vendor is not allowed to combine,
separate, resequence, skip, or modify individual test steps, posting
effects, or other expected results without prior consent. If a tested
core financial system cannot process a step as specified, the vendor
must request an alternate test procedure. Alternate test procedure
requests identify the affected steps, the execution problem, and
proposed alternative. A single request can cover multiple steps, if the
execution problem is exactly repeated. In this case, the vendor would
make a single alternate test procedure request but is obligated to
itemize all steps to which the alternate test procedure applies. Table 1
contains types of alternate test procedures and guidelines for how FSIO
will view the proposal.

  -----------------------------------------------------------------------
  Table 1. Alternate Test Procedure 
  Types                             
  --------------------------------- -------------------------------------
  Type of alternate procedure       Guideline

  *Setup.* An alternate setup       Minor setup differences that can be
  procedure occurs when the vendor  directly cross-walked to expected
  uses different configuration or   results are generally allowed. The
  assumption data than what were    use of account code suffixes to
  provided in the test plan.        establish subaccounts would be
                                    considered an acceptable alternate
                                    provided these subaccounts roll up as
                                    intended and are not used for other
                                    unexpected posting results. Wholesale
                                    substitutions of specified fund
                                    codes, object classes, U.S. SGL
                                    accounts, programs, and proj­ects are
                                    not permitted.

  *Posting.* An alternate posting   If the core financial system can be
  procedure occurs when a subject   configured to use the prescribed
  product is designed to post       posting rules, FSIO expects it to be
  transactions in a manner          tested that way (without an alternate
  different from the posting model  procedure). An alternative posting
  specified by FSIO. (The test      model may be allowed, if the net
  rules governing the use of        effect of the alternative model
  alternate document numbers to     matches the specified result. The use
  reference U.S. SGL posting        of journal vouchers to accomplish
  results are not considered an     posting results is prohibited.
  alternate procedure.)             

  *Execution.* An alternate         Alternate execution procedures may be
  execution procedure occurs when a acceptable if their initiation is
  tested product completes a test   automatic or system controlled, they
  step using multiple, separately   achieve the expected results, and
  initiated documents or batch      they do not cause a misstatement of
  processes.                        funds availability (real-time) or
                                    period-end financial position.

  *Function.* An alternate          All tested functionality is
  functional procedure occurs when  considered mandatory.
  a tested product was not designed 
  to support a stated requirement   
  (e.g., produce a report, produce  
  a query result, or perform        
  process control).                 

  *Reporting.* An alternate         Treasury- and OMB-defined formats
  reporting procedure occurs when a cannot be changed. When the test is
  product does not generate a       based on an example format, alternate
  required output report, query     procedures may be approved provided
  result, or transaction file       all requested information is
  according to form and content     presented comprehensively.
  rules cited in the test step.     

  *Process.* An alternate process   FSIO expects differences in the way a
  procedure occurs when the way a   core financial system handles complex
  product handles a multistep test  posting processes. The basis for
  case differs from the             allowing an alternate process is
  test-specified approach (e.g.,    whether the intended result is
  starting balances, year-end       achieved and whether the end-to-end
  closing, application programming  process is automatic.
  interface, cost distribution).    

                                    
  -----------------------------------------------------------------------

If the vendor proposes using an alternate posting model, a deviation
from the test script, or alternate document numbering for any test step
or accounting event (alternate test procedure), the vendor must request
permission from FSIO before the test. The alternate test procedure
request format available on the FSIO website must be used for this
purpose. Alternate test procedures are subject to FSIO evaluation and
approval. All alternate test procedure requests must be submitted at
least 1 week before the test setup interview.

Vendors who wish to use an alternate test procedure must do the
following:

-   Explain why the alternate test procedure is needed, and what the
    benefit is to the government.

-   Specify the alternate document numbering, alternate processing, and
    detailed ledger postings associated with each affected test step, as
    applicable.

-   Provide all changes to the test materials that result from use of
    the alternate test procedure, including those to the setup data,
    test scripts, expected results, system queries, and reports. If the
    vendor uses an alternate document numbering system or an alternate
    general ledger posting model, it is the vendor's responsibility to
    provide a cross-walk from the test script and expected results,
    respectively, to the corresponding vendor's system information.

Requests will be reviewed case-by-case for their impact on the test and
for conformance with the underlying requirements being tested. Allowed
alternate test procedures will be confirmed during the setup interview
and used during the test to qualify test results. Additional alternate
test procedures and revisions to approved alternate test procedures may
be requested during the test to document processing observed by FSIO.

FSIO's prior review of these exceptions should not be construed as
determination of the outcome of the execution of the related test steps.
All test step scoring will occur based on the execution of the test
steps themselves during and after the test, to ensure that the
alternative processing still meets the applicable FSIO requirements.

FSIO reserves the right to publish descriptions of all alternate test
procedures used by the vendor to pass the test.

### Vendor Conducts Setup and Other Pretest Activities

The vendor is responsible for reviewing all test steps, establishing a
test environment, loading FSIO-prescribed setup data, and recording
transactions necessary to establish any beginning balances. Vendors are
required to provide FSIO with the test materials listed below:

-   *Vendor test participants list.* Vendors must submit a list of all
    employees, contractors, and consultants assisting or present during
    the test, including those who may be at another location. For each
    person, the list must include name, system user identification,
    company affiliation, title, and role (even if an observer) during
    the test. The vendor will provide this at the test setup interview
    session, and will update it when the test is executed, including any
    retest.

-   *Test questions.* Vendors may submit questions on the test materials
    for clarification by FSIO. The test question format provided on the
    FSIO website must be used for this purpose. All questions on the
    test materials must be submitted prior to the setup interview.

-   *Table configuration.* Requests for information related to the
    configuration of tables will be provided in the test plan.

See the Appendix for related time frames.

### FSIO Conducts Setup Interview

During the test setup interview, the FSIO test team will do the
following:

-   Verify that the vendor's system and architecture conform to the
    description provided in the application, noting any differences.

-   Verify the software product name and version.

-   As part of the test application, confirm the required application
    software and the required infrastructure to be used during the test
    (as attested to by the vendor in the application), including all
    applicable modules and any third-party software products.

-   Validate that complete and accurate setup data and beginning
    balances have been established in the vendor's test environment, on
    the basis of reports provided by the vendor. Any missing or
    incorrect data must be corrected before unobserved test execution
    can begin. FSIO recognizes that the setup information provided might
    be incomplete in terms of all the information required to configure
    each software product. The vendor is responsible for assessing the
    test scripts to determine the optimal configuration needed to
    process the indicated transactions and to produce the expected
    results.

-   Confirm the information that the vendor supplied earlier, such as
    approved alternate test procedures, hardware components, and system
    table configuration. (During the test, FSIO will compare this
    information to observed system operation and request vendor updates,
    as appropriate.)

-   Observe a demonstration of the software by the vendor, for further
    evidence that operational system software exists and can perform the
    functions that satisfy mandatory core financial system requirements.
    This demonstration will also familiarize FSIO with the operation of
    the software and thus facilitate efficient use of testing time.
    Software demonstrations are to be scripted by the vendor.

The vendor scripts the demonstration using the site, architecture, and
platform of its choosing. To show the operational system, the vendor
should script the demonstration to display the software's ability to
accomplish several mandatory requirements such as the software's ability
to meet the funds control requirements. To familiarize FSIO with the
system, the vendor should also demonstrate user interaction with the
application, data input screens, system configuration, maintenance and
setup, security, and reporting capabilities. FSIO realizes that the
demonstration is time constrained and not comprehensive.

This demonstration of software capability *does not* take the place of a
qualification test.

### FSIO Conducts First Readiness Assessment

At this point, FSIO also performs the first of two test readiness
assessments based on the information collected at the setup interview
and observations during the software demonstration. Test readiness
criteria are published as part of the test materials for each test.
Readiness assessments are conducted to evaluate whether the vendor is
prepared to start the test and to successfully meet at least 90 percent
of the tested requirements on the first attempt. Vendors that are
determined to not be ready will be rescheduled, at which time test
readiness will be reevaluated.

## Conducting the Test

FSIO will conduct the test at a vendor-provided site and location using
vendor-provided hardware and software at a time scheduled by the vendor
and FSIO. The vendor is expected to provide the test team with workspace
to accommodate a team of up to four people. The office space should
provide privacy and allow the test team adequate table or desk space to
set up computers and review documents. A high-speed Internet connection
and a printer should also be provided for use by the FSIO test team.

The test plan is a separate document that provides specifics about the
conduct and execution of the test. FSIO will provide data for the test
in the form of appendixes to the test plans. The vendor is responsible
for data entry into the system as well as its total operation.

The vendor is responsible for determining how to meet the expected
results, in accordance with the published mandatory requirements, and
for executing each of the test steps. During the test, the test team
will note the processing of each mandatory test step, but completion of
processing of the test step does not imply that the vendor has passed
that step.

FSIO is the final authority on the interpretation of all aspects of the
test, including mandatory requirements, the test application, the test
policy, the test plan, and all related test products.

FSIO is under no obligation to immediately notify the vendor that a
requirement has not been met during the execution of the test. If the
test team discovers, after the processing of a test step is completed,
that the expected result has not been achieved, the test team may notify
the vendor during the test that the test step was improperly executed.
This information is provided to the vendor to help the vendor avoid
failed requirements in subsequent steps.

### General Test Requirements

Vendors are required to meet certain general requirements in the conduct
of the test:

-   Do not use journal vouchers

-   Provide access log

-   Produce and provide report outputs immediately

-   Execute test steps to demonstrate compliance with requirements.

These requirements are described further in the sections below.

#### Journal Vouchers

FSIO expects that the software will meet expected results through
standard system processing, without the use of journal vouchers.

#### Access Log

The test period is defined as the time period from the vendor's initial
system setup through test completion, including any retest. As part of
the setup interview session, the vendor is required to identify all
people who will have access to the test environment, including their
applicable system identification. For purposes of the test, these people
will be called "test participants"; substitutions can be made after the
vendor notifies FSIO.

Only authorized individuals are allowed to access the application,
including its database and configuration parameters, for the duration of
the test period. The vendor will execute necessary programs to produce
an automatic log of all system access and activity during the test
period in a manner consistent with the core requirements.

#### Report Production

During the observed processing phases of the test, when a report or
query is required to be produced, the vendor must deliver all specified
outputs immediately, directly to FSIO. The vendor may then choose to
validate the output before FSIO evaluates it for compliance with the
tested requirements. If a second report is run by the vendor, FSIO will
verify that any differences between the first and second reports are the
same as those found by the vendor. The vendor is prohibited from
prescreening or changing queries or reports prior to FSIO receiving
them. FSIO reserves the right to request additional queries or reports
at any time during the test.

#### Test Step Execution

While FSIO will hold harmless the vendor for correction of simple
mistakes, such as keying errors, once a test step has been executed, the
result achieved is the basis for FSIO evaluation and scoring. Unless
FSIO determines that the vendor made a simple mistake, results not in
conformance with that expected count as an executed test step. FSIO will
make the official pass/fail decision for the associated requirements
based on the test observations and the posttest detailed evaluation of
results.

### Vendor Completes Unobserved Processing

Unobserved processing involves test steps that are not subject to FSIO
observation (i.e., unobserved testing) but are still subject to FSIO
evaluation against expected results. The time frames allowed for test
execution are determined by FSIO for each test version. Applicable test
execution time frames are published in the test plan.

During the unobserved processing portion of the test, FSIO is usually
not present at the vendor site, but is available to respond to questions
regarding the test and how to proceed if the vendor experiences problems
or perceived test step failures.

Unobserved processing often consists of multiple cycles, corresponding
to accounting periods. Each cycle comprises test steps in which
financial transactions, queries, and reports are processed by the
vendor.

Unobserved processing ends either at the end of the days allowed for in
the test plan or when the vendor has executed the last test step,
whichever is earlier. When the test steps are completed, the unobserved
processing portion of the test is finished, even if the days allowed for
the test have not expired.

### FSIO Validates Unobserved Processing Results

As each cycle is completed, the vendor submits the test step results to
FSIO for validation against the expected results. These results include
the reports, query results, screen prints, and other data specified by
FSIO in the test script. The expected results for cycle end reports,
such as trial balances and transaction registers, are published
separately by FSIO. Other expected results are specified in the
individual test steps. Discrepancies between the vendor's general ledger
results and FSIO's expected results, other than those allowed by
approved alternate test procedures, are to be resolved before the vendor
begins unobserved processing for the next cycle.

### FSIO Conducts Second Readiness Assessment

Before starting observed testing, FSIO will conduct the second of two
test readiness assessments. Based on the results of the unobserved
testing, FSIO will determine whether the vendor is ready to proceed with
the test. Vendors that are determined to not be ready will be
rescheduled. The vendor's readiness to start observed testing will be
reassessed at the start of the rescheduled test.

### Vendor Starts Observed Processing

Observed testing involves test steps that are subject to FSIO
observation. The time frames allowed for test execution are determined
by FSIO for each test. Applicable test execution time frames are
published in the test plan.

FSIO observes the entering of the observed transaction processing test
steps and receives the expected results at the end of each cycle.

In addition to scripted test steps, FSIO may use the following
approaches in conducting the observed portion of the test:

-   Use of changed data in test cases

-   Reserved transaction processing

-   Test restart.

These approaches are described further in the sections that follow.

#### Use of Changed Data in Test Cases

FSIO will use the test steps as published in the FSIO website. The
vendor's core financial system software is expected to produce results
that match the expected results as published in the FSIO website.
However, either prior to or during the test, the government reserves the
right to change data values (such as fund codes, object classes, general
ledger accounts, posting model, document numbers or amounts) from those
published, and expects that results produced by the vendor's software
will change appropriately. (For example, if a payee name is changed from
Jones to Smith or a general ledger account code is changed, the system
should reflect the accurate information based on the latest available
test data.)

#### Reserved Transaction Processing

At FSIO's discretion, one cycle of the test will be designated for
reserved transaction processing. At the beginning of this cycle, FSIO
will provide the vendor with additional test scripts to process. These
scripts will be similar to those the vendor has already processed, but
with different document numbers and data values. The vendor will perform
a system backup, produce selected queries or reports, enter the test
scripts, and provide further queries or reports. FSIO will evaluate the
results as part of the overall scoring of the test.

If this cycle occurs mid-test, vendors will restore the system to the
backup before continuing regular processing. This way, the cumulative
expected results (e.g., general ledger reports and document balances)
will not be affected.

#### Test Restart

If a test step is improperly executed, or if the test must be halted for
any reason, FSIO will coordinate with the vendor to determine at what
point in the test execution process that the test may resume. However,
FSIO reserves the right to resume the test at a test step different from
the one in which the test was stopped, including at a "system backup"
step as specified in the test plan. The vendor may propose a different
starting point than that requested by FSIO because of the application's
programming logic. However, the vendor must explain why the different
starting point is needed. The decision on whether to accept the proposal
will be made by FSIO. FSIO may also request that the vendor produce
system reports to verify the condition of the data at the point where
execution was halted.

### Vendor Completes Observed Processing

Observed processing ends either at the end of the days allowed for in
the test plan or when the vendor has executed the last test step,
whichever is earlier. When the test steps are completed, the observed
processing portion of the test is finished, even if the days allowed for
the test have not expired.

The execution of test will end when (1) all test steps have been
executed and the required outputs are delivered to FSIO, or (2) the
maximum time frame allowed for test step execution is reached, whichever
is earlier. FSIO will not accept test results generated after test
execution ends.

## Grading the Test

Once the vendor has executed all test steps and delivered all outputs
specified in the test plan, FSIO will validate the results and will
determine a pass/fail outcome by tested requirement. FSIO will also
notify the vendor of the need to retest specific functionality, if the
software has not passed one or more of the mandatory requirements
tested.

### FSIO Validates Observed Processing Results

Each mandatory requirement will be graded as either pass or fail, as
evidenced by successful test step execution. The software passes the
test when the software passes every mandatory requirement tested. During
the test, FSIO does not formally acknowledge whether or not the software
actually passed the requirement. Complete evaluation of test step
results requires detailed examination of general ledger effects,
subsidiary records, external file processing (such as payment files by
Treasury), and other test outputs such as reports and queries. After the
completion of the test, the FSIO test team performs this thorough
evaluation, which may take a considerable amount of time (sometimes in
excess of 30 days). Without exception, the test team will make the final
pass/fail determination for each requirement based on its posttest
detailed examination of results.

During the evaluation period, FSIO may need clarification of test
results. As appropriate, FSIO will contact the vendor's team lead to
present any questions for resolution.

During the evaluation period, the vendor should ensure that the
operating environment, including software, hardware, and data, is
isolated and not modified or updated. This will allow the vendor and
FSIO to check the system when there are questions about test results and
also provide the appropriate environment to retest any test steps.

Tested products are required to meet 100 percent of the mandatory core
financial system requirements, whether or not FSIO actually tests that
the product meets the requirement.

#### Grading Full and Incremental Tests

During the test, FSIO will not direct the vendor's proposed method to
meet a test step but may provide feedback for consideration by the
vendor. All final pass/fail determinations will be made solely on the
results of the execution of the test steps and the results produced, as
reviewed during FSIO's posttest examination of results. Information
provided by FSIO to the vendor either prior to or during the test may
not be construed as determination of the outcome of the execution of the
related test steps.

To pass a full test and receive a certificate of compliance, or to pass
an incremental test and maintain the certificate, the tested product
must pass 100 percent of the mandatory requirements tested. If the
product does not pass the test, FSIO will issue a notice to the vendor
indicating the percentage of requirements that were successfully met.

If the product passes at least 90 percent of the tested requirements on
the first attempt, it is eligible for a one-time retest. Retesting
generally involves reexecuting selected portions of the test. Refer to
the section on conducting a retest for further information on this test
process.

If the product passes less than 90 percent of the tested requirements,
the test outcome will be considered to be a failure, and the product
will be designated nonqualifying.

#### Grading a Version Test

If FSIO determines that the documented changes do not adversely affect
prior-version functionality, the new version will be deemed qualified,
and a certificate for that version will be issued.

### FSIO Notifies Vendor of Need to Retest

Upon completion of the validation of test results for full and
incremental tests, FSIO will notify the vendor if a retest is needed to
complete a test. The notification will include a list of requirements
and the specific test steps in which failure was evidenced. FSIO will
also indicate the percentage of requirements that were successfully met.

## Conducting a Retest

The following activities are required when a vendor has been notified by
FSIO of the need to retest one or more requirements:

-   FSIO schedules retest

-   Vendor submits retest plan

-   Vendor completes pretest activities

-   FSIO conducts retest readiness assessment

-   Vendor completes the retest

-   FSIO validates the retest results.

The retest activities are described further in the sections below.

### FSIO Schedules Retest

The vendor will propose to FSIO a date for the retest. FSIO will
schedule the retest based on the dates requested by the vendor, or at
FSIO's earliest convenience. A separate test application is not
required.

### Vendor Submits Retest Plan

In preparing for a retest, vendors must submit a recommended approach
(retest plan). This approach must fully test all requirements subject to
retesting, in a manner consistent with FSIO's original full test plan.
The retest plan must also identify all required setup data, test steps
to be executed, and expected results. FSIO will work with the vendor to
finalize the retest approach and content.

Retests are typically conducted in a single phase that includes test
steps subject to FSIO observation (i.e., observed testing). The time
frames allowed for test execution are predetermined by FSIO for each
retest. Applicable test execution time frames are published in the final
retest plan.

### Vendor Completes Pretest Activities

Before executing a retest, the vendor is responsible for upgrading the
product to correct identified deficiencies, preparing the test
environment, and loading all setup data. Vendors may also submit
alternate test procedure requests for FSIO's approval.

### FSIO Conducts Retest Readiness Assessment

Before starting a retest, FSIO will confirm that the vendor is ready to
proceed. Vendors that are determined to not be ready will be
rescheduled.

### Vendor Completes the Retest

The vendor will execute the approved test plan and provide FSIO with all
expected results. Retests may be conducted using web-based software
demonstration tools.

### FSIO Validates Retest Results

Once the vendor has executed all test steps and delivered all specified
outputs, FSIO will validate the results and will determine a pass/fail
outcome by tested requirement. To pass the retest, the product must meet
100 percent of the requirements subject to retesting.

## Performing Posttest Activities

After FSIO grades the test, FSIO notifies the vendor of the test
results.

If the test is successfully passed, the following activities also occur:

-   Vendor submits product documentation

-   FSIO publishes successful test results

-   FSIO publishes alternate test procedures and supplemental
    information.

If the test is failed, the vendor may apply to test a new version.

These posttest activities are described further in the sections below.

### FSIO Notifies Vendor of Test Results

After review and reconciliation of test output and processing results,
FSIO will provide an official pass/fail determination to the vendor in
writing. If the official determination is fail, an attachment to the
letter will identify the requirements that were failed, the FSIO test
team's observations for those requirements and associated test steps,
and the reason the results were judged noncompliant.

#### Passed the Test

For software products that pass the full or version test, FSIO will
issue a certificate of compliance within 30 calendar days following the
completion of test grading or the submission of other required
documentation, whichever is later. The certificate of compliance
qualifies the software as meeting FSIO requirements for a core financial
system. However, the government may require incremental testing to
maintain the certificate.

For software products that pass the incremental test, the certificate of
compliance remains in effect.

##### FSIO Issuance of Certifications

FSIO will issue a certificate of compliance for products that pass a
full test or a version test. FSIO-issued certificates are specific to
the product version tested and qualify the named product as meeting all
mandatory requirements in force at the time of the test. Certificates
will remain in effect as long as the vendor continues to successfully
complete all subsequent full and incremental tests required by FSIO.
Certificates will be revoked for products that fail or do not take
required full and incremental tests.

A FSIO-issued certificate that results from passing these tests
establishes that the subject product version meets all mandatory core
financial system requirements in force at the time of a given test.

##### Certification of Multiple Products

A vendor may have more than one core financial system software product
certified at any given time.

#### Failed the Test

If, after completing the test process (including retesting), the product
does not successfully meet 100 percent of the tested requirements, or if
the vendor fails to complete the test process within the required time
frame (within 12 months of submitting the application for a full test or
within 6 months of submitting the application for an incremental test),
the test outcome will be considered to be a failure, and the product
will be designated nonqualifying. Any existing certificate will be
revoked.

### Vendor Submits Product Documentation and Information on Value-Added Features

A full set of product documentation, updated to reflect software changes
made to pass the test, is required within 30 calendar days of
notification of successful test results. This software documentation
includes user guides, implementation guides, and system setup guides for
all software modules that the vendor used to take the test. It may be
provided in electronic format or made available online (with any
necessary passwords provided to FSIO).

If the vendor chooses to provide information on the software product's
value-added features, FSIO will provide a link on its website to the
vendor's website for this information. The Value-Added Features form is
available on the FSIO website. FSIO does not verify the accuracy of the
information posted by the vendor.

### FSIO Publishes Successful Test Results

FSIO publishes the following information on its website for currently
certified software products:

-   Vendor name

-   Product name

-   Version tested

-   Recent test results.

FSIO will maintain the history of all software certifications on its
website. When a certificate has been issued for a version update, the
certificate for the prior version is no longer in effect. FSIO will
update its website to reflect the new version that is certified.

### Vendor Provides Software to the Government

The vendor will make the newly qualified software and related
documentation available to government customers within 6 months of
notification of successful test results.

### FSIO Publishes Alternate Test Procedures and Supplemental Information

Over the course of testing, FSIO collects product and test execution
information (other product information) that it considers to be
nonproprietary. FSIO may elect to make this information available to the
public. The following are the main categories of other product
information:

-   Alternate test procedures

-   Supplemental information.

FSIO will generally respect written requests by the vendor for
restricted dissemination of system documentation and other proprietary
information to which FSIO testers may be exposed during the
qualification process.

FSIO will not publish any information on products that do not receive a
certificate of compliance, nor will FSIO publish information on
applications received, applications rejected, or test scheduling.

#### Supplemental Information

During a test, FSIO will collect the following types of product
description information:

-   Product release history

-   Software used for the test

-   Hardware and network infrastructure used for the test

-   Documentation sources

-   Product architecture

-   Functional observations.

At the conclusion of a test, FSIO will prepare a Supplemental
Information report to document findings and observations in each of
these categories.

#### Vendor Review of Product Information

FSIO will provide the vendor with advance notice about product
information intended for publication and the opportunity to review and
comment on these materials. FSIO will attempt to incorporate all
vendor-provided comments to the extent such comments are consistent with
FSIO internal observations and information dissemination standards.

### Vendor Applies to Test New Version after Failure

To protect the government from a continued commitment of resources to
test nonqualifying core financial systems, vendors must wait a minimum
of 3 months before applying to retest a nonqualifying product, and
demonstrate that the software is ready. FSIO may reduce the waiting
period, at its discretion. To continue the test process at that time,
the vendor must submit an application to test a new or updated version
of the product and retake the full test in its entirety.

Appendix Test Activities and Time frames

Table A-1 provides the time frames and responsibilities for testing
activities.

+-----------+-----+------------+------------+------------+-----------+
| *Table    |     |            |            |            |           |
| A‑2. Time |     |            |            |            |           |
| Frames    |     |            |            |            |           |
| for       |     |            |            |            |           |
| Testing   |     |            |            |            |           |
| Ac        |     |            |            |            |           |
| tivities* |     |            |            |            |           |
+===========+=====+============+============+============+===========+
| **A       | *   | **Time     |            |            | **Notes** |
| ctivity** | *Re | Frame by   |            |            |           |
|           | spo | Test       |            |            |           |
|           | nsi | Type**     |            |            |           |
|           | ble |            |            |            |           |
|           | En  |            |            |            |           |
|           | tit |            |            |            |           |
|           | y** |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
|           |     | **Full     | **Incremen | **Version  |           |
|           |     | Test**     | tal Test** | Test**     |           |
+-----------+-----+------------+------------+------------+-----------+
| Test      |     |            |            |            |           |
| De        |     |            |            |            |           |
| velopment |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 1)  Start | F   | Upon       | Upon       | NA         |           |
|     the   | SIO | issuance   | issuance   |            |           |
|     test  |     | of new and | of new and |            |           |
|     de    |     | revised    | revised    |            |           |
| velopment |     | re         | re         |            |           |
|           |     | quirements | quirements |            |           |
|  process. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 2         | F   | At any     | At any     | NA         |           |
| )  Notify | SIO | time after | time after |            |           |
|           |     | issuance   | issuance   |            |           |
|   vendors |     | of new     | of new     |            |           |
|     of    |     | re         | re         |            |           |
|     need  |     | quirements | quirements |            |           |
|     to    |     |            |            |            |           |
|     test. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 3)        | F   | Depends on | Depends on | NA         | Test plan |
|   Publish | SIO | extent of  | extent of  |            | a         |
|     test  |     | testing    | testing    |            | ppendices |
|     plan  |     |            |            |            | provide   |
|           |     |            |            |            | setup     |
|  exposure |     |            |            |            | data,     |
|           |     |            |            |            | test      |
|    draft. |     |            |            |            | scripts,  |
|           |     |            |            |            | and       |
|           |     |            |            |            | expected  |
|           |     |            |            |            | results.  |
+-----------+-----+------------+------------+------------+-----------+
| 4)        | V   | 60 days    | 30 days    | NA         |           |
|   Comment | end | after      | after      |            |           |
|     on    | ors | p          | p          |            |           |
|     test  | and | ublication | ublication |            |           |
|     plan  | Ag  | of test    | of test    |            |           |
|           | enc | plan       | plan       |            |           |
|  exposure | ies | exposure   | exposure   |            |           |
|           |     | draft      | draft      |            |           |
|    draft. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 5)        | F   | Depends on | Depends on | NA         |           |
|   Publish | SIO | extent of  | extent of  |            |           |
|     final |     | comments   | comments   |            |           |
|     test  |     |            |            |            |           |
|     plan. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| Test      |     |            |            |            |           |
| Execution |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 6         | Ven | Within 30  | Within 30  | At any     | Release   |
| )  Submit | dor | calendar   | calendar   | time       | notes     |
|     test  |     | days of    | days of    |            | must      |
|     app   |     | p          | p          | []{.mark}  | accompany |
| lication. |     | ublication | ublication |            | version   |
|           |     | of final   | of final   |            | test      |
|           |     | test plan  | test plan  |            | appl      |
|           |     | or at any  | for        |            | ications. |
|           |     | time for   | qualified  |            |           |
|           |     | new        | software   |            |           |
|           |     | software   | products   |            |           |
|           |     | products   |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 7         | F   | Within 30  | Within 30  | Within 30  |           |
| )  Notify | SIO | calendar   | calendar   | calendar   |           |
|           |     | days of    | days of    | days of    |           |
|    vendor |     | receipt of | receipt of | receipt of |           |
|     of    |     | the        | the        | the        |           |
|     ap    |     | a          | a          | a          |           |
| plication |     | pplication | pplication | pplication |           |
|     a     |     | or receipt | or receipt | or receipt |           |
| cceptance |     | of         | of         | of         |           |
|     or    |     | additional | additional | additional |           |
|     r     |     | in         | in         | in         |           |
| ejection. |     | formation, | formation, | formation, |           |
|           |     | whichever  | whichever  | whichever  |           |
|           |     | is later   | is later   | is later   |           |
+-----------+-----+------------+------------+------------+-----------+
| 8         | F   | NA         | NA         | Within 30  | .         |
| )  Review | SIO |            |            | calendar   |           |
|           |     |            |            | days of    |           |
|   release |     |            |            | receipt of |           |
|           |     |            |            | test       |           |
|    notes. |     |            |            | a          |           |
|           |     |            |            | pplication |           |
|   Request |     |            |            |            |           |
|           |     |            |            |            |           |
|   version |     |            |            |            |           |
|     demon |     |            |            |            |           |
| stration, |     |            |            |            |           |
|     if    |     |            |            |            |           |
|           |     |            |            |            |           |
|   needed. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 9)        | Ven | NA         | NA         | Within 30  | When      |
|   Provide | dor |            |            | calendar   | r         |
|           |     |            |            | days of    | equested, |
|   version |     |            |            | FSIO       | version   |
|     demon |     |            |            | request    | demon     |
| stration. |     |            |            |            | strations |
|           |     |            |            |            | are to be |
|           |     |            |            |            | scripted  |
|           |     |            |            |            | by the    |
|           |     |            |            |            | vendor.   |
+-----------+-----+------------+------------+------------+-----------+
| 10)       | F   | NA         | NA         | Within 30  | If        |
|  Evaluate | SIO |            |            | calendar   | testing   |
|           |     |            |            | days of    | is        |
|   version |     |            |            | end of     | needed,   |
|     demon |     |            |            | version    | go to     |
| stration, |     |            |            | dem        | step 11.  |
|     if    |     |            |            | onstration |           |
|           |     |            |            |            | If no     |
|   needed. |     |            |            |            | testing   |
|           |     |            |            |            | is        |
|           |     |            |            |            | needed,   |
|           |     |            |            |            | go to     |
|           |     |            |            |            | step 31.  |
+-----------+-----+------------+------------+------------+-----------+
| 1         | F   | NA         | NA         | Upon       |           |
| 1) Notify | SIO |            |            | completion |           |
|           |     |            |            | of the     |           |
|   vendors |     |            |            | review of  |           |
|     of    |     |            |            | the        |           |
|     need  |     |            |            | release    |           |
|     to    |     |            |            | notes, or  |           |
|     test. |     |            |            | evaluation |           |
|           |     |            |            | of a       |           |
|           |     |            |            | version    |           |
|           |     |            |            | demo       |           |
|           |     |            |            | nstration, |           |
|           |     |            |            | whichever  |           |
|           |     |            |            | is later   |           |
+-----------+-----+------------+------------+------------+-----------+
| 12)       | F   | Within 30  | Within 30  | Same as    | Decide on |
|  Schedule | SIO | days of    | days of    | Full Test  | test      |
|           |     | a          | a          |            | dates.    |
|  observed |     | pplication | pplication |            |           |
|           |     | acceptance | acceptance |            |           |
|  testing. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 1         | Ven | At least 1 | At least 1 | Same as    |           |
| 3) Submit | dor | week prior | week prior | Full Test  |           |
|           |     | to setup   | to the     |            |           |
| alternate |     | interview  | setup      |            |           |
|     test  |     |            | interview  |            |           |
|           |     |            |            |            |           |
| procedure |     |            |            |            |           |
|           |     |            |            |            |           |
| requests. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 14        | Ven | Prior to   | Prior to   | Same as    |           |
| ) Conduct | dor | setup      | setup      | Full Test  |           |
|     setup |     | interview  | interview  |            |           |
|     and   |     |            |            |            |           |
|     other |     |            |            |            |           |
|           |     |            |            |            |           |
|   pretest |     |            |            |            |           |
|     ac    |     |            |            |            |           |
| tivities. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 15        | F   | At least 2 | At least 2 | Same as    |           |
| ) Conduct | SIO | business   | business   | Full Test  |           |
|     setup |     | days prior | days prior |            |           |
|     i     |     | to receipt | to receipt |            |           |
| nterview. |     | of first   | of first   |            |           |
|           |     | unobserved | unobserved |            |           |
|           |     | test       | test       |            |           |
|           |     | results    | results    |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 16        | F   | At least 2 | At least 2 | Same as    |           |
| ) Conduct | SIO | business   | business   | Full Test  |           |
|     first |     | days prior | days prior |            |           |
|           |     | to receipt | to receipt |            |           |
| readiness |     | of first   | of first   |            |           |
|     as    |     | unobserved | unobserved |            |           |
| sessment. |     | test       | test       |            |           |
|           |     | results    | results    |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 17)       | Ven | At least 6 | At least 6 | Same as    |           |
|  Complete | dor | business   | business   | Full Test  |           |
|     u     |     | days prior | days prior |            |           |
| nobserved |     | to start   | to start   |            |           |
|     pr    |     | of         | of         |            |           |
| ocessing. |     | observed   | observed   |            |           |
|           |     | processing | processing |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 18)       | F   | Prior to   | Prior to   | Same as    |           |
|  Validate | SIO | start of   | start of   | Full Test  |           |
|     u     |     | observed   | observed   |            |           |
| nobserved |     | processing | processing |            |           |
|     p     |     |            |            |            |           |
| rocessing |     |            |            |            |           |
|           |     |            |            |            |           |
|  results. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 19        | F   | After      | After      | Same as    |           |
| ) Conduct | SIO | validation | validation | Full Test  |           |
|           |     | of         | of         |            |           |
|    second |     | unobserved | unobserved |            |           |
|           |     | testing    | testing    |            |           |
| readiness |     | results    | results    |            |           |
|     as    |     | and before | and before |            |           |
| sessment. |     | starting   | starting   |            |           |
|           |     | observed   | observed   |            |           |
|           |     | processing | processing |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 20) Start | Ven | Within 6   | Within 3   | Same as    | FSIO may  |
|           | dor | months of  | months of  | Full Test  | extend    |
|  observed |     | p          | p          |            | the       |
|     pr    |     | ublication | ublication |            | deadline, |
| ocessing. |     | of test    | of test    |            | if        |
|           |     | plan       | plan       |            | necessary |
|           |     |            |            |            | to        |
|           |     |            |            |            | ac        |
|           |     |            |            |            | commodate |
|           |     |            |            |            | g         |
|           |     |            |            |            | overnment |
|           |     |            |            |            | testing   |
|           |     |            |            |            | partners. |
+-----------+-----+------------+------------+------------+-----------+
| 21)       | Ven | As         | As         | Same as    |           |
|  Complete | dor | specified  | specified  | Full Test  |           |
|           |     | in test    | in test    |            |           |
|  observed |     | plan       | plan       |            |           |
|     pr    |     |            |            |            |           |
| ocessing. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 22)       | F   | As needed  | As needed  | Same as    |           |
|  Validate | SIO |            |            | Full Test  |           |
|           |     |            |            |            |           |
|  observed |     |            |            |            |           |
|     p     |     |            |            |            |           |
| rocessing |     |            |            |            |           |
|           |     |            |            |            |           |
|  results. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 2         | F   | Upon       | Upon       | Same as    | 10%       |
| 3) Notify | SIO | completion | completion | Full Test  | retest.   |
|           |     | of test    | of test    |            |           |
|    vendor |     | results    | results    |            | If        |
|     of    |     | validation | validation |            | retesting |
|     need  |     |            |            |            | is not    |
|     to    |     |            |            |            | ap        |
|           |     |            |            |            | plicable, |
|   retest. |     |            |            |            | go to     |
|           |     |            |            |            | step 29.  |
+-----------+-----+------------+------------+------------+-----------+
| 24)       | F   | Within 30  | Within 30  | Same as    | Decide on |
|  Schedule | SIO | calendar   | calendar   | Full Test  | test      |
|           |     | days of    | days of    |            | dates.    |
|   retest. |     | notice to  | notice to  |            |           |
|           |     | retest     | retest     |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 2         | Ven | Within 30  | Within 30  | Same as    | Must be   |
| 5) Submit | dor | calendar   | calendar   | Full Test  | approved  |
|           |     | days of    | days of    |            | by FSIO.  |
|    retest |     | notice to  | notice to  |            |           |
|     plan. |     | retest     | retest     |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 26)       | Ven | At least 1 | At least 1 | Same as    |           |
|  Complete | dor | business   | business   | Full Test  |           |
|     p     |     | day before | day before |            |           |
| re-retest |     | starting   | starting   |            |           |
|     ac    |     | retest     | retest     |            |           |
| tivities. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 27        | F   | At least 1 | At least 1 | Same as    |           |
| ) Conduct | SIO | business   | business   | Full Test  |           |
|           |     | day before | day before |            |           |
|    retest |     | starting   | starting   |            |           |
|           |     | retest     | retest     |            |           |
| readiness |     |            |            |            |           |
|     as    |     |            |            |            |           |
| sessment. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 28)       | Ven | Within 60  | Within 60  | Same as    | If vendor |
|  Complete | dor | calendar   | calendar   | Full Test  | is        |
|           |     | days of    | days of    |            | assessed  |
|   retest. |     | notice to  | notice to  |            | as being  |
|           |     | retest and | retest and |            | ready to  |
|           |     | within 12  | within 6   |            | retest.   |
|           |     | months     | months     |            |           |
|           |     | from       | from       |            |           |
|           |     | submission | submission |            |           |
|           |     | of the     | of the     |            |           |
|           |     | test       | test       |            |           |
|           |     | a          | a          |            |           |
|           |     | pplication | pplication |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 29)       | F   | As needed  | As needed  | Same as    |           |
|  Validate | SIO |            |            | Full Test  |           |
|           |     |            |            |            |           |
|    retest |     |            |            |            |           |
|           |     |            |            |            |           |
|  results. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 3         | F   | Upon       | Upon       | Same as    | Ce        |
| 0) Notify | SIO | completion | completion | Full Test  | rtificate |
|           |     | of test    | of test    |            | is        |
|    vendor |     | results    | results    |            | issued,   |
|     of    |     | validation | validation |            | extended, |
|     test  |     |            |            |            | or        |
|           |     |            |            |            | revoked   |
|  results. |     |            |            |            | as        |
|           |     |            |            |            | indicated |
|           |     |            |            |            | in test   |
|           |     |            |            |            | results   |
|           |     |            |            |            | letter.   |
+-----------+-----+------------+------------+------------+-----------+
| 3         | Ven | Within 30  | Within 30  | Same as    | Full      |
| 1) Submit | dor | days of    | days of    | Full Test  | docum     |
|           |     | no         | no         |            | entation, |
|   product |     | tification | tification |            | updated   |
|     docu  |     | of         | of         |            | to        |
| mentation |     | successful | successful |            | reflect   |
|     and   |     | test       | test       |            | all       |
|     in    |     | results    | results    |            | software  |
| formation |     |            |            |            | changes   |
|     on    |     |            |            |            | made to   |
|     va    |     |            |            |            | pass the  |
| lue-added |     |            |            |            | test.     |
|           |     |            |            |            |           |
| features. |     |            |            |            | The       |
|           |     |            |            |            | vendor    |
|           |     |            |            |            | may       |
|           |     |            |            |            | choose to |
|           |     |            |            |            | submit    |
|           |     |            |            |            | in        |
|           |     |            |            |            | formation |
|           |     |            |            |            | on its    |
|           |     |            |            |            | va        |
|           |     |            |            |            | lue-added |
|           |     |            |            |            | features, |
|           |     |            |            |            | using the |
|           |     |            |            |            | form      |
|           |     |            |            |            | provided  |
|           |     |            |            |            | on the    |
|           |     |            |            |            | FSIO      |
|           |     |            |            |            | website.  |
+-----------+-----+------------+------------+------------+-----------+
| 32        | F   | After      | After      | Same as    |           |
| ) Publish | SIO | no         | no         | Full Test  |           |
|     s     |     | tification | tification |            |           |
| uccessful |     | of test    | of test    |            |           |
|     test  |     | results    | results    |            |           |
|           |     |            |            |            |           |
|  results. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 33        | Ven | Within 6   | Within 6   | Same as    |           |
| ) Provide | dor | months of  | months of  | Full Test  |           |
|           |     | no         | no         |            |           |
|  software |     | tification | tification |            |           |
|     to    |     | of         | of         |            |           |
|     g     |     | successful | successful |            |           |
| overnment |     | test       | test       |            |           |
|     c     |     | results    | results    |            |           |
| ustomers. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 34        | F   | After      | After      | Same as    |           |
| ) Publish | SIO | p          | p          | Full Test  |           |
|           |     | ublication | ublication |            |           |
| alternate |     | of         | of         |            |           |
|     test  |     | successful | successful |            |           |
|     p     |     | test       | test       |            |           |
| rocedures |     | results    | results    |            |           |
|     and   |     |            |            |            |           |
|     sup   |     |            |            |            |           |
| plemental |     |            |            |            |           |
|     inf   |     |            |            |            |           |
| ormation. |     |            |            |            |           |
+-----------+-----+------------+------------+------------+-----------+
| 35) Apply | Ven | A minimum  | A minimum  | Same as    | FSIO may  |
|     to    | dor | of 3       | of 3       | Full Test  | reduce    |
|     test  |     | months     | months     |            | the       |
|     new   |     | after      | after      |            | waiting   |
|           |     | no         | no         |            | period,   |
|   version |     | tification | tification |            | at its    |
|     after |     | of         | of         |            | di        |
|           |     | un         | un         |            | scretion. |
|  failure. |     | successful | successful |            |           |
|           |     | test       | test       |            |           |
|           |     | results    | results    |            |           |
+-----------+-----+------------+------------+------------+-----------+

[^1]: Section 39.101(c) of the Federal Acquisition Regulation requires
    agencies to acquire core financial management software that has been
    certified.

[^2]: For example, FSIO may observe during testing that a module or
    software product other than that listed in the application was used
    to pass the test.