***Department of Labor***

***International Price System***

***Initiation Subsystem***

***(INIT)***

##### **Year 2000 Testing Quality Assurance** 

**TQA SUMMARY REPORT ADDENDUM**

######### [TABLE OF CONTENTS]{.underline}

INITIATION Subsystem tqa DETAIL RESULTS \-- ADDENDUM

I. INTRODUCTION A-1

II\. INITIATION TQA SUMMARY TABLES A-2

III\. INITIATION TQA DETAIL RESULTS A-3

> A. CRITICAL SUCCESS FACTORS \-- REVIEW RESULTS A-3
>
> 1.0 Achieved DOL Year 2000 Objectives A-4
>
> 2.0 Staffed Test Team Appropriately A-7
>
> 3.0 Used Compliant Year 2000 Test Environment ..A-8
>
> 4.0 Identified Key Processes and Date Impacted Business Functions A-10
>
> 5.0 Created Testing Documentation that is Logically Linked A-12
>
> 6.0 Identified and Tested System Interfaces A-13
>
> 7.0 Executed Tests According to Plan A-14
>
> 8.0 Completed Tests with Satisfactory Results A-16
>
> 9.0 Followed Quality Assurance Procedures A-17

B. BASELINE TESTING DOCUMENTATION \-- REVIEW RESULTS A-18

> Test Plan A-19
>
> Test Cases A-20
>
> Test Results A-20
>
> Problem Tracking A-21
>
> Management Issue Log A-21

**I. INTRODUCTION**

This addendum to the International Price System Testing Quality
Assurance Review Summary Report contains the individual observations
made by the Testing Quality Assurance Review team for the Initiation
Subsystem (INIT) Year 2000 testing activities. We compared the Year 2000
testing activities performed for the Initiation Subsystem to
requirements issued by the Department of Labor (DOL) and the General
Accounting Office (GAO). We also compared testing efforts to generally
accepted systems testing standards. In this addendum, we describe the
results of our review in terms of nine [Critical Success
Factors]{.underline} and five categories of [Baseline Testing
Documentation]{.underline}. These Critical Success Factors and Baseline
Testing Documentation categories were used to structure our review of
the Initiation Subsystem Year 2000 test activities and to develop our
observations. These observations and the results of our review were
derived from documentation reviews, interviews and discussions conducted
by the TQA team. Critical Success Factors represent key segments of the
Year 2000 testing process and Baseline Testing Documentation categories
represent a set of documents that help to provide the application team
with assurances that critical steps of the Year 2000 testing process
were conducted and documented. We describe each of the nine Critical
Success Factors and Baseline Testing Documentation areas by using one of
three indicators; 1) Meets Baseline Practices, 2) Management Awareness
Advised, or 3) Management Review Advised. The indicator is developed
from the analysis of baseline testing practices that comprise a critical
success factor or the Baseline Testing Documentation. The actual rating
is substantiated by the results of reviewing supporting baseline testing
practices and observations that support each overall rating. The
definition for each of the ratings is:

1)  **Meets Baseline Practices** **(MBP)** -- The TQA Team found that
    the Critical Success Factor or the Testing Documentation contents
    that were reviewed **adequately** **satisfied Baseline Practices.**

2)  **Management Awareness Advised** **(MAA)** -- The TQA Team found
    that the Critical Success Factor or the Testing Documentation
    Contents **DID NOT satisfy one or more of the Baseline Practices and
    may warrant further management review.**

**3) Management Review Advised (MRA) --** Significant observations
support the conclusion that **Baseline Practices WERE NOT followed**
**or NOT supported** in the testing documentation. **Further management
review is advised.**

  ----------------------------------- -----------------------------------
  **MBP = Meets Baseline Practices**  **ADEQUATELY SATISFIED baseline
                                      practices and documentation
                                      elements.**

  **MAA = Management Awareness        **DID NOT SATISFY one or more of
  Advised**                           the baseline practices and/or
                                      documentation elements.**

  **MRA = Management Review Advised** **Baseline practices and/or
                                      documentation elements WERE NOT
                                      FOLLOWED or NOT SUPPORTED.**
  ----------------------------------- -----------------------------------

**II. INITIATION SUBSYSTEM REVIEW TQA SUMMARY TABLES**

The following two summary tables are the result of consolidating the
supporting detail evaluated by the TQA team for each Critical Success
Factor and each Baseline Testing Documentation item. To fully understand
how the TQA team arrived at the summary of observations, one should also
carefully review the supporting detail that is presented in the
Initiation Subsystem TQA Detail Results sections that follow the Summary
Tables.

+--------------------------------------+---------+---------+---------+
| ######## Critical Success Factors    | **Meets | **Man   | **Man   |
|                                      | B       | agement | agement |
|                                      | aseline | Aw      | Review  |
|                                      | Prac    | areness | Ad      |
|                                      | tices** | Ad      | vised** |
|                                      |         | vised** |         |
+--------------------------------------+---------+---------+---------+
| 1.  **Achieved DOL Year 2000         |         |         |         |
|     Objectives**                     |         |         |         |
+--------------------------------------+---------+---------+---------+
| 2.  **Staffed Test Team              |         |         |         |
|     Appropriately**                  |         |         |         |
+--------------------------------------+---------+---------+---------+
| 3.  **Used Compliant Year 2000 Test  |         |         |         |
|     Environment**                    |         |         |         |
+--------------------------------------+---------+---------+---------+
| 4.  **Identified Key Processes &     |         |         |         |
|     Date Impacted Business           |         |         |         |
|     Functions**                      |         |         |         |
+--------------------------------------+---------+---------+---------+
| 5.  **Created Testing Documentation  |         |         |         |
|     that is Logically Linked**       |         |         |         |
+--------------------------------------+---------+---------+---------+
| 6.  **Identified & System Tested     |         |         |         |
|     Interfaces** [^1]                |         |         |         |
+--------------------------------------+---------+---------+---------+
| 7.  **Executed Tests According to    |         |         |         |
|     Plan**                           |         |         |         |
+--------------------------------------+---------+---------+---------+
| 8.  **Completed Tests with           |         |         |         |
|     Satisfactory Results**           |         |         |         |
+--------------------------------------+---------+---------+---------+
| 9.  **Followed Quality Assurance     |         |         |         |
|     Procedures**                     |         |         |         |
+--------------------------------------+---------+---------+---------+

+--------------------------------------+---------+---------+---------+
| ######## Bas                         | **Meets | **Man   | **Man   |
| eline Testing Documentation Contents | B       | agement | agement |
|                                      | aseline | Aw      | Review  |
|                                      | Prac    | areness | Ad      |
|                                      | tices** | Ad      | vised** |
|                                      |         | vised** |         |
+--------------------------------------+---------+---------+---------+
| #### **Test Plan**                   |         |         |         |
+--------------------------------------+---------+---------+---------+
| **Test Cases**                       |         |         |         |
+--------------------------------------+---------+---------+---------+
| **Test Results**                     |         |         |         |
+--------------------------------------+---------+---------+---------+
| **Problem Tracking Log**             |         |         |         |
+--------------------------------------+---------+---------+---------+
| **Management Issue Log**             |         |         |         |
+--------------------------------------+---------+---------+---------+

The TQA Review process analyzed the testing practices used by the
Initiation Subsystem testing team in the Year 2000 testing processes of
preparation, execution, and validation. This was accomplished by 1)
reviewing testing and 2) conducting interviews to help the TQA Review
teams understand the actual practices used by the Initiation Subsystem
test team. The Initiation Subsystem Year 2000 testing activities were
assessed against Year 2000 requirements set forth by the Department of
Labor and the General Accounting Office (GAO). In addition, we assessed
the Initiation Subsystem Year 2000 testing procedures by evaluating them
against processes that are recognized as generally accepted software
configuration management and testing practices.

## III. Initiation Subsystem TQA - DETAIL RESULTS

# A. CRITICAL SUCCESS FACTORS -- DETAIL TQA RESULTS

In our Testing Quality Assurance Review process, Critical Success
Factors are comprised of several individual baseline testing practices.
An "Overall Indicator" for each Critical Success Factor, along with the
supporting observations, are presented in the detailed results section,
starting on the next page.

**Overall Indicator \--** The overall rating for each Critical Success
Factor is based on the combined result of: 1) reviewing underlying
baseline practices, 2) understanding the application and the
documentation context, and 3) recording the consensus evaluation of the
TQA Review team. The TQA Review team assessed these combined factors as
a group of actual practices that support (or do not support) the desired
(Critical Success Factor) outcome.

**Detail Comparisons --** Each of the Critical Success Factors is
followed by supporting detail comparisons derived from the results of
the TQA Review. Each detailed comparison consists of an individually
numbered baseline testing practice, followed by the supporting (actual
practice) detail recorded by the TQA team. The result of each comparison
between baseline practice and actual practice of the Initiation
Subsystem test team, is indicated as **["YES"]{.underline},
["NO"]{.underline}, or ["N/A"]{.underline}**:

> **[YES]{.underline}** - The TQA Review team found this baseline
> practice to be met based on the documentation sample(s) reviewed
> and/or actual practice information provided in test team interviews.
>
> **[NO]{.underline}** - The TQA Review team found this baseline
> practice was not met based on the documentation sample(s) reviewed
> and/or actual practice information provided in test team interviews.
>
> **[N/A]{.underline}** - The Application team indicated that this
> baseline practice was not applicable to the application under review.

# CRITICAL SUCCESS FACTORS - REVIEW RESULTS 

+--------------------------------------+---------+----------+---------+
|                                      | **Meets | **Ma     | **Man   |
|                                      | B       | nagement | agement |
|                                      | aseline | A        | Review  |
|                                      | Prac    | wareness | Ad      |
|                                      | tices** | A        | vised** |
|                                      |         | dvised** |         |
+--------------------------------------+---------+----------+---------+
| ## 1                                 |         |          |         |
| .0 Achieved DOL Year 2000 Objectives |         |          |         |
+--------------------------------------+---------+----------+---------+

**OVERALL OBSERVATION:** While Year 2000 test plans did generally met
DOL, Federal, private sector and other professional Year 2000 testing
guidelines, we could not identify separate levels of testing for
baseline, unit, integration, and end to end test procedures.

**DETAIL COMPARISONS:** Not Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **An application component         |        |         |         |
|     compliance checklist was           |        |         |         |
|     prepared**                         |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Non-compliant components were    |        |         |         |
|     assessed for Year 2000 impact**    |        |         |         |
+----------------------------------------+--------+---------+---------+
| See section 3.1 for further discussion |        |         |         |
| of non-compliant components in Year    |        |         |         |
| 2000 test environment                  |        |         |         |
+----------------------------------------+--------+---------+---------+
| 3.  **External data feeds identified   |        |         |         |
|     and owners contacted**             |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **Test plans are within DOL Year   |        |         |         |
|     2000 guidelines**                  |        |         |         |
+----------------------------------------+--------+---------+---------+
| Test plans generally are within DOL    |        |         |         |
| Year 2000 guidelines.                  |        |         |         |
+----------------------------------------+--------+---------+---------+
| 5.  **Test plans are within Federal,   |        |         |         |
|     private sector and other           |        |         |         |
|     professional standard Year 2000    |        |         |         |
|     Testing**                          |        |         |         |
|                                        |        |         |         |
| > **Guidelines**                       |        |         |         |
+----------------------------------------+--------+---------+---------+
| While test plans generally met         |        |         |         |
| Federal, private sector and other      |        |         |         |
| professional Year 2000 testing         |        |         |         |
| guidelines, they did not identify      |        |         |         |
| separate levels of testing for         |        |         |         |
| baseline, unit, integration, and end   |        |         |         |
| to end test procedures.                |        |         |         |
+----------------------------------------+--------+---------+---------+
| 6.  **Testing methodologies selected   |        |         |         |
|     were appropriate for Remediation   |        |         |         |
|     methodology used**                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| See 1.5 above.                         |        |         |         |
+----------------------------------------+--------+---------+---------+
| 7.  **Unique application high risk     |        |         |         |
|     dates were documented and tested** |        |         |         |
+----------------------------------------+--------+---------+---------+
| Subsystem level date processing        |        |         |         |
| features (e.g., date sorting, date     |        |         |         |
| display and date outputs to reports)   |        |         |         |
| were identified by system developers   |        |         |         |
| using the BLS Year 2000 Subsystem      |        |         |         |
| Cross-reference Worksheet and          |        |         |         |
| Compliance Checklist and were          |        |         |         |
| identified as "points of concern"      |        |         |         |
| (POCs). Dates identified were limited  |        |         |         |
| to point of concern guidance and       |        |         |         |
| unique application high risk dates     |        |         |         |
| beyond those were not evidenced.       |        |         |         |
+----------------------------------------+--------+---------+---------+
| 8.  **Defined system acceptance        |        |         |         |
|     testing requirements:**            |        |         |         |
|                                        |        |         |         |
| > **Functional, performance and        |        |         |         |
| > security**                           |        |         |         |
+----------------------------------------+--------+---------+---------+
| Acceptance Test Standards have been    |        |         |         |
| defined for the IPS system which       |        |         |         |
| address the need to establish these    |        |         |         |
| requirements during acceptance testing |        |         |         |
| efforts. During Year 2000 related      |        |         |         |
| acceptance testing, subsystem level    |        |         |         |
| acceptance testing requirements were   |        |         |         |
| not developed for performance and      |        |         |         |
| security.                              |        |         |         |
+----------------------------------------+--------+---------+---------+
| 9.  **Strategy was developed for       |        |         |         |
|     baseline testing**                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| Year 2000 related baseline testing was |        |         |         |
| not performed.                         |        |         |         |
+----------------------------------------+--------+---------+---------+
| 10. **Strategy was developed for       |        |         |         |
|     current date testing**             |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 11. **Strategy was developed for       |        |         |         |
|     future date testing**              |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 12. **Strategy was developed for       |        |         |         |
|     simulated interface testing**      |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| **1.13 Strategy was developed for      |        |         |         |
| testing of bridges**                   |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 14. **Metrics defined for categories   |        |         |         |
|     of defect severity**               |        |         |         |
+----------------------------------------+--------+---------+---------+
| Metrics for "quantity of errors found" |        |         |         |
| and "quantity of errors resolved" were |        |         |         |
| briefly described in the IPS Test      |        |         |         |
| Plan. No evidence was provided of      |        |         |         |
| metrics maintained for categories of   |        |         |         |
| defect severity.                       |        |         |         |
+----------------------------------------+--------+---------+---------+
| 15. **End-to-end test procedures and   |        |         |         |
|     data prepared**                    |        |         |         |
+----------------------------------------+--------+---------+---------+
| The purpose of end-to-end testing is   |        |         |         |
| described by GAO to verify that a      |        |         |         |
| defined set of interrelated systems    |        |         |         |
| which collectively support an          |        |         |         |
| organizational core function           |        |         |         |
| interoperate as intended in an         |        |         |         |
| operational environment, either actual |        |         |         |
| or simulated. Those interrelated       |        |         |         |
| system include not only those owned    |        |         |         |
| and managed by the organization, but   |        |         |         |
| also the external systems with which   |        |         |         |
| they interface. A separate end to end  |        |         |         |
| testing process was not performed for  |        |         |         |
| the IPS subsystems to include all      |        |         |         |
| internal and/or external interfaces,   |        |         |         |
| infrastructure and telecommunications, |        |         |         |
| along with the fully integrated        |        |         |         |
| application. According to BLS          |        |         |         |
| Management, reliance for end to end    |        |         |         |
| test coverage was placed upon test     |        |         |         |
| scripts and test cases executed during |        |         |         |
| point of concern testing and           |        |         |         |
| acceptance testing which were expected |        |         |         |
| to address subsystem interoperations.  |        |         |         |
+----------------------------------------+--------+---------+---------+
| 16. **Project status reports include   |        |         |         |
|     progress tracking**                |        |         |         |
|                                        |        |         |         |
| ### Mechanisms                         |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 17. **Change control procedures and    |        |         |         |
|     software**                         |        |         |         |
|                                        |        |         |         |
| #######  Configur                      |        |         |         |
| ation management processes are defined |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 18. **Processes and metrics for        |        |         |         |
|     reporting test activity and        |        |         |         |
|     Progress are established**         |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| **1.19 Results of initial baseline     |        |         |         |
| testing, test plans, and a**           |        |         |         |
|                                        |        |         |         |
| ### D                                  |        |         |         |
| escription of the test data maintained |        |         |         |
+----------------------------------------+--------+---------+---------+
| Year 2000 baseline testing was not     |        |         |         |
| performed.                             |        |         |         |
+----------------------------------------+--------+---------+---------+

+--------------------------------------+---------+---------+---------+
|                                      | **Meets | **Man   | **Man   |
|                                      | B       | agement | agement |
|                                      | aseline | Aw      | Review  |
|                                      | Prac    | areness | Ad      |
|                                      | tices** | Ad      | vised** |
|                                      |         | vised** |         |
+--------------------------------------+---------+---------+---------+
| ##                                   |         |         |         |
|  2.0 Staffed Test Team Appropriately |         |         |         |
+--------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** The INIT Year 2000 testing did not include a
separate and distinct end to end testing process involving
interorganizational end-to-end test team (s).

**DETAIL COMPARISONS:**

Not

Yes No Applicable

+----------------------------------------+--------+----------+--------+
| 1.  **Required resources for testing   |        |          |        |
|     were identified**                  |        |          |        |
+----------------------------------------+--------+----------+--------+
|                                        |        |          |        |
+----------------------------------------+--------+----------+--------+
| 2.  **Subject Matter Experts were      |        |          |        |
|     involved in creation and execution |        |          |        |
|     of tests**                         |        |          |        |
+----------------------------------------+--------+----------+--------+
|                                        |        |          |        |
+----------------------------------------+--------+----------+--------+
| 3.  **Test team displays general       |        |          |        |
|     experience and understanding of    |        |          |        |
|     standard testing methods and       |        |          |        |
|     procedures**                       |        |          |        |
+----------------------------------------+--------+----------+--------+
|                                        |        |          |        |
+----------------------------------------+--------+----------+--------+
| #### **Interorganizational             |        |          |        |
| end-to-end test team (s) established** |        |          |        |
+----------------------------------------+--------+----------+--------+
| Year 2000 end-to-end testing was not   |        |          |        |
| performed, see 1.15.                   |        |          |        |
+----------------------------------------+--------+----------+--------+
| #### **Plan                            |        |          |        |
| ned resources were available when requ |        |          |        |
| ired for the duration of the project** |        |          |        |
+----------------------------------------+--------+----------+--------+
|                                        |        |          |        |
+----------------------------------------+--------+----------+--------+

+--------------------------------------+---------+---------+---------+
|                                      | **Meets | **Man   | **Man   |
|                                      | B       | agement | agement |
|                                      | aseline | Aw      | Review  |
|                                      | Prac    | areness | Ad      |
|                                      | tices** | Ad      | vised** |
|                                      |         | vised** |         |
+--------------------------------------+---------+---------+---------+
| ## 3.0 Used                          |         |         |         |
| Compliant Year 2000 Test Environment |         |         |         |
+--------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** During Year 2000 testing efforts, IPP
identified components that were not reported as fully compliant by the
related vendors. The IPS system staff have identified these
non-compliant components and developed an analysis of the non-compliant
aspects. Many of these components have either been upgraded since Year
2000 testing was performed or will be upgraded in the near future. It
was not evident that additional Year 2000 testing activities were
scheduled.

**DETAIL COMPARISONS:**

> Not

Yes No Applicable

+---------------------------------------+--------+----------+---------+
| 1.  **Current and future date system  |        |          |         |
|     tests performed in a Year 2000    |        |          |         |
|     test environment**                |        |          |         |
+---------------------------------------+--------+----------+---------+
| The INIT subsystem testing was        |        |          |         |
| performed in a simulated Year 2000    |        |          |         |
| testing environment. During Year 2000 |        |          |         |
| testing efforts, this environment     |        |          |         |
| contained components that were not    |        |          |         |
| reported as fully compliant by the    |        |          |         |
| related vendors. The IPS system staff |        |          |         |
| have identified these non-compliant   |        |          |         |
| components and developed an analysis  |        |          |         |
| of the non-compliant aspects. Many of |        |          |         |
| these components have either been     |        |          |         |
| upgraded since Year 2000 testing was  |        |          |         |
| performed or will be upgraded in the  |        |          |         |
| near future. It was not evident that  |        |          |         |
| additional Year 2000 testing          |        |          |         |
| activities were scheduled. IPS        |        |          |         |
| management has also stated that no    |        |          |         |
| problems were evidenced during the    |        |          |         |
| year 2000 testing process relating to |        |          |         |
| those non-compliant products.         |        |          |         |
|                                       |        |          |         |
| Guidance issued by the GAO states     |        |          |         |
| that, "organizations should implement |        |          |         |
| plans for mitigating the impact of    |        |          |         |
| vendor products and services not      |        |          |         |
| being compliant on time." The GAO     |        |          |         |
| Year 2000 Testing Guide states that   |        |          |         |
| "In order to execute system           |        |          |         |
| acceptance tests and ensure system    |        |          |         |
| Year 2000 compliance, all system      |        |          |         |
| components (application software,     |        |          |         |
| systems software, hardware, firmware, |        |          |         |
| and communications networks) must be  |        |          |         |
| compliant and ready for testing as an |        |          |         |
| integrated system"                    |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Change management              |        |          |         |
| procedures support promotion of appli |        |          |         |
| cation components into testing area** |        |          |         |
+---------------------------------------+--------+----------+---------+
|                                       |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Chan                           |        |          |         |
| ge management procedures utilized for |        |          |         |
|  retention and promotion of the code  |        |          |         |
| from test environment to production** |        |          |         |
+---------------------------------------+--------+----------+---------+
|                                       |        |          |         |
+---------------------------------------+--------+----------+---------+
| 3.  **End-to-end test(s) were         |        |          |         |
|     executed**                        |        |          |         |
+---------------------------------------+--------+----------+---------+
| No end-to-end tests were performed.   |        |          |         |
| (see also Critical Success Factor     |        |          |         |
| 1.15) The IPS/INIT system manager is  |        |          |         |
| relying upon vendor certification and |        |          |         |
| the BLS Certification Lab to ensure   |        |          |         |
| compliance of the IPS/INIT supporting |        |          |         |
| infrastructure. IPS support staff did |        |          |         |
| not provide documentation confirming  |        |          |         |
| compliance of infrastructure          |        |          |         |
| components.                           |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Minimum                        |        |          |         |
| Year 2000 system compliant date sensi |        |          |         |
| tive requirements established (e.g.,  |        |          |         |
| Special Dates, Date Intervals, Days o |        |          |         |
| f the Year and Miscellaneous Dates)** |        |          |         |
+---------------------------------------+--------+----------+---------+
| Minimum Year 2000 system compliant    |        |          |         |
| date sensitive requirements were      |        |          |         |
| established by the INIT application   |        |          |         |
| test team. (See Critical Success      |        |          |         |
| Factor 7.0)                           |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Test envi                      |        |          |         |
| ronment enables future date testing** |        |          |         |
+---------------------------------------+--------+----------+---------+
| The INIT test environment included    |        |          |         |
| future date testing a date that the   |        |          |         |
| IPS application and system modules    |        |          |         |
| used to perform future date testing.  |        |          |         |
+---------------------------------------+--------+----------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 4.0 Identified Key Processes       |         |         |         |
|  and Date Impacted Business Functions |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION**: The INIT Year 2000 testing did not include
separate and distinct software integration and end to end testing
process to aid in identifying key processes and date impacted business
functions.

**DETAIL COMPARISONS:** Not

Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **Application was decomposed to    |        |         |         |
|     identify business functions,       |        |         |         |
|     files, databases, and processing   |        |         |         |
|     cycles that are affected by date   |        |         |         |
|     processing**                       |        |         |         |
+----------------------------------------+--------+---------+---------+
| System developers used point of        |        |         |         |
| concern guidance which required them   |        |         |         |
| to identify potential Year 2000 issues |        |         |         |
| by business functions of the           |        |         |         |
| subsystem.                             |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Test scripts were developed      |        |         |         |
|     using the application              |        |         |         |
|     decomposition inventory**          |        |         |         |
+----------------------------------------+--------+---------+---------+
| System developers used point of        |        |         |         |
| concern guidance which required them   |        |         |         |
| to identify potential Year 2000 issues |        |         |         |
| by business functions of the           |        |         |         |
| subsystem.                             |        |         |         |
+----------------------------------------+--------+---------+---------+
| 3.  **Business critical archiving and  |        |         |         |
|     history functionality has been     |        |         |         |
|     included in future date tests**    |        |         |         |
+----------------------------------------+--------+---------+---------+
| Subsystem descriptions did not include |        |         |         |
| archiving and history functionality.   |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **Unique points of failure that    |        |         |         |
|     could occur prior to the Year 2000 |        |         |         |
|     have been considered in the        |        |         |         |
|     planning process**                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| Unique points of failure prior to the  |        |         |         |
| Year 2000 were not identified as       |        |         |         |
| points of concern. Dates identified    |        |         |         |
| were limited to point of concern       |        |         |         |
| guidance and unique application points |        |         |         |
| of failure prior to the Year 2000 were |        |         |         |
| not evidenced.                         |        |         |         |
+----------------------------------------+--------+---------+---------+
| 5.  **Test planning had representation |        |         |         |
|     from subject matter experts**      |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 6.  **System acceptance test criteria, |        |         |         |
|     procedures and data are defined**  |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 7.  **Software integration test        |        |         |         |
|     criteria defined**                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| Management relied upon integration     |        |         |         |
| testing through the Unified Database   |        |         |         |
| during POC and AT testing. However,    |        |         |         |
| separate integration test criteria was |        |         |         |
| not defined.                           |        |         |         |
+----------------------------------------+--------+---------+---------+
| 8.  **System boundaries of end-to-end  |        |         |         |
|     test(s) have been defined**        |        |         |         |
+----------------------------------------+--------+---------+---------+
| Documentation of defined system        |        |         |         |
| boundaries was provided, although no   |        |         |         |
| end-to-end testing was performed. (See |        |         |         |
| Critical Success Factor 1.15.)         |        |         |         |
+----------------------------------------+--------+---------+---------+
| 9.  **Edits for invalid date           |        |         |         |
|     processing were included in test   |        |         |         |
|     cases**                            |        |         |         |
+----------------------------------------+--------+---------+---------+
| Edits for invalid date processing were |        |         |         |
| not evident in the identification of   |        |         |         |
| points of concern or in test cases.    |        |         |         |
+----------------------------------------+--------+---------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 5.0 Created Testing Do             |         |         |         |
| cumentation that is Logically Linked  |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION**: The INIT Year 2000 test results could not be
logically linked to the test cases, points of concern or BLS Year 2000
compliance criteria. In addition, we could not identify if Year 2000
test execution logs were prepared to track the INIT test results.

**DETAIL COMPARISONS:**

> Not
>
> Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **Defined conditions or            |        |         |         |
|     requirements for Unit test         |        |         |         |
|     procedures, and data to test       |        |         |         |
|     correct date handling.**           |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Test plans have associated       |        |         |         |
|     scripts, cases, execution logs,    |        |         |         |
|     and test results that are          |        |         |         |
|     logically linked.**                |        |         |         |
+----------------------------------------+--------+---------+---------+
| Test plans contain test scripts and    |        |         |         |
| test cases. No execution logs were     |        |         |         |
| available to track testing as it was   |        |         |         |
| performed. Test results could not be   |        |         |         |
| logically linked to test cases, points |        |         |         |
| of concern and BLS Year 2000           |        |         |         |
| compliance criteria.                   |        |         |         |
+----------------------------------------+--------+---------+---------+
| **5.3 Test results and data have been  |        |         |         |
| archived for re- execution and         |        |         |         |
| comparison**                           |        |         |         |
+----------------------------------------+--------+---------+---------+
| Only some system test results were     |        |         |         |
| retained. The IPP test team stated     |        |         |         |
| that the aged Volume_Test database     |        |         |         |
| used for testing on the Y2K mini-net   |        |         |         |
| had been archived.                     |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **Test plan identifies how test    |        |         |         |
|     data will be obtained or created** |        |         |         |
+----------------------------------------+--------+---------+---------+
| **5.5 Test plan identifies how defects |        |         |         |
| are to be corrected**                  |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 6.  **The test plan identifies how the |        |         |         |
|     data will be aged**                |        |         |         |
+----------------------------------------+--------+---------+---------+
| Systemic data aging was performed by   |        |         |         |
| aging the dates in the unified         |        |         |         |
| database. SQL scripts were developed   |        |         |         |
| to changed dates in the database to    |        |         |         |
| reflect dates that would be in the     |        |         |         |
| database when future date testing was  |        |         |         |
| performed. Aging was based on the      |        |         |         |
| knowledge of the test team.            |        |         |         |
+----------------------------------------+--------+---------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 6.0 Id                             |         |         |         |
| entified and Tested System Interfaces |         |         |         |
+---------------------------------------+---------+---------+---------+

**Overall Observation:** There is no overall rating given for this
critical success factor, due to the fact all baseline testing practices
are not applicable for this subsystem.

**Detail comparisons:** Not

Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **All critical interfaces could be |        |         |         |
|     simulated in future date tests**   |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Integration testing includes     |        |         |         |
|     interface tests to ensure that     |        |         |         |
|     when sub-systems, when combined,   |        |         |         |
|     work together as intended.**       |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 3.  **Simulated interface data         |        |         |         |
|     represents all possible            |        |         |         |
|     date-impacted business functions   |        |         |         |
|     affected by this interface.**      |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **Interfaces with two digit years  |        |         |         |
|     have been checked for previously   |        |         |         |
|     communicated pivot point values**  |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+

## 

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ##                                    |         |         |         |
|  7.0 Executed Tests According to Plan |         |         |         |
+---------------------------------------+---------+---------+---------+

**Overall Observation**: The INIT Year 2000 testing process did not
include baseline testing, issue logs, problem logs nor were test results
logically linked to the test cases.

**Detail Comparisons:**

> Not

**Yes No Applicable**

+---------------------------------------------+------+-------+---------+
| 1.  **Baseline tests were executed prior to |      |       |         |
|     19xx and 20xx tests**                   |      |       |         |
+---------------------------------------------+------+-------+---------+
| Baseline testing for Year 2000 tests were   |      |       |         |
| not conducted.                              |      |       |         |
+---------------------------------------------+------+-------+---------+
| 2.  **Test cases were developed to test     |      |       |         |
|     according to the type of renovation     |      |       |         |
|     method used**                           |      |       |         |
+---------------------------------------------+------+-------+---------+
| Test cases were developed to address points |      |       |         |
| of concern identified by                    |      |       |         |
| developers/programmers and BLS Year 2000    |      |       |         |
| Compliance Criteria, not the specific       |      |       |         |
| renovation method. Test cases developed or  |      |       |         |
| executed to test the renovated features     |      |       |         |
| were not uniquely identified.               |      |       |         |
+---------------------------------------------+------+-------+---------+
| 3.  **Test summary log shows when tests     |      |       |         |
|     were executed and verified**            |      |       |         |
+---------------------------------------------+------+-------+---------+
| No test summary logs are documented.        |      |       |         |
+---------------------------------------------+------+-------+---------+
| 4.  **An issue log was maintained which     |      |       |         |
|     includes status and resolutions**       |      |       |         |
+---------------------------------------------+------+-------+---------+
| It was not evident that testers maintained  |      |       |         |
| a log that tracked the status and           |      |       |         |
| resolution of issues that arose during the  |      |       |         |
| testing process.                            |      |       |         |
+---------------------------------------------+------+-------+---------+
| 5.  **Test progress reporting identifies    |      |       |         |
|     deviation from plans and appropriate    |      |       |         |
|     action is taken to address any          |      |       |         |
|     deviation**                             |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+
| 6.  **A problem log was maintained which    |      |       |         |
|     includes status and resolutions**       |      |       |         |
+---------------------------------------------+------+-------+---------+
| It was not evident that management          |      |       |         |
| maintained a log to track the status and    |      |       |         |
| resolutions of problems that arose during   |      |       |         |
| the testing process.                        |      |       |         |
+---------------------------------------------+------+-------+---------+
| 7.  **All date related business functions   |      |       |         |
|     have associated test data**             |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+
| 8.  **Test cases, scripts, and results met  |      |       |         |
|     test plan objectives**                  |      |       |         |
+---------------------------------------------+------+-------+---------+
| Generally, it appears that test plan        |      |       |         |
| objectives were met, however, test results  |      |       |         |
| could not be logically linked to test       |      |       |         |
| cases, points of concern and BLS Year 2000  |      |       |         |
| compliance criteria.                        |      |       |         |
+---------------------------------------------+------+-------+---------+
| 7.9 Test cases and scripts were executed    |      |       |         |
| according to the documented test plans      |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+
| 10. **Execute test according to DOL         |      |       |         |
|     compliance criteria**                   |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+

##  

## 

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 8.0 Comp                           |         |         |         |
| leted Tests with Satisfactory Results |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION**: The INIT Year 2000 testing process did not
include baseline testing.

**DETAILED COMPARISONS:**

> Not Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **19xx and 20xx test results were  |        |         |         |
|     validated against the results of   |        |         |         |
|     the baseline test**                |        |         |         |
+----------------------------------------+--------+---------+---------+
| Baseline testing for Year 2000 was not |        |         |         |
| performed and documentation for        |        |         |         |
| baseline testing was not provided.     |        |         |         |
| (see section 7.1 and 1.9)              |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **The acceptance criteria for the  |        |         |         |
|     testing process has been           |        |         |         |
|     determined**                       |        |         |         |
+----------------------------------------+--------+---------+---------+
| Acceptance criteria was identified for |        |         |         |
| the overall testing process, not       |        |         |         |
| identified at the individual test case |        |         |         |
| level.                                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| 3.  **The approach for validating the  |        |         |         |
|     test results has been identified\  |        |         |         |
|     **                                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **The expected results are         |        |         |         |
|     documented in the scripts**        |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 5.  **Differences in actual and        |        |         |         |
|     expected results were analyzed and |        |         |         |
|     reconciled**                       |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 6.  **Specified test conditions        |        |         |         |
|     (logical scenarios) are met and    |        |         |         |
|     sign-offs from IV + V/QA agent,    |        |         |         |
|     configuration management and       |        |         |         |
|     functional management have been    |        |         |         |
|     obtained**                         |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 7.  **Test results were validated by   |        |         |         |
|     Subject Matter Experts**           |        |         |         |
+----------------------------------------+--------+---------+---------+
| ##                                     |        |         |         |
+----------------------------------------+--------+---------+---------+

## 

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 9.0                                |         |         |         |
| Followed Quality Assurance Procedures |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** We concluded that the IPP Year 2000 test team
followed quality assurance procedures.

**DETAIL COMPARISONS:**

Not

Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **The test plans, test scripts and |        |         |         |
|     test cases were validated by       |        |         |         |
|     subject matter experts**           |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| #### **Y                               |        |         |         |
| ear 2000-related issues and problems w |        |         |         |
| ere all resolved in a timely manner**  |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| #### **Peer reviews were cond          |        |         |         |
| ucted on test plans and test results** |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Test team held regular meetings  |        |         |         |
|     to resolve technical and           |        |         |         |
|     procedural problems**              |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+

# B. BASELINE TESTING DOCUMENTATION -- DETAIL TQA RESULTS 

In addition to the comparison of the Year 2000 testing practices to the
baseline practices, the TQA Review process also involves assessing the
application testing team's documentation against a set of generally
accepted baseline documentation content standards (elements that should
be represented in those documents). These methods and procedures have
been converted to a list of key testing documents for the purposes of
identifying whether documentation standards for Year 2000 testing
efforts were achieved.

Each of the individual document's contents was evaluated by the TQA
Review team and described by one of the following indicators:

> **[YES]{.underline}** - The TQA Review team found this element was
> documented to satisfy generally accepted Year 2000 test documentation
> guidelines, based on the sample reviewed.
>
> **[NO]{.underline}** - The TQA Review team found this element was not
> documented to satisfy generally accepted Year 2000 test documentation
> guidelines, based on the sample reviewed.
>
> **[Evidenced Elsewhere]{.underline} -** The TQA Review team found that
> the element required for this individual document existed but was
> supported elsewhere as part of the testing of this application.

The indication of an individual element within a Baseline document
category as *NO* or *Evidenced Elsewhere* does not necessarily
correspond to the entire document category receiving a rating of
Awareness or Review Advised. The rating for the entire category is based
on how the TQA Review team assessed the overall effectiveness of the
group of elements against Year 2000 guidelines.

# BASELINE TESTING DOCUMENTATION - REVIEW RESULTS 

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| #### **Test Plan**                    |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** The INIT Test Plan met Year 2000 Testing
Quality Assurance Baseline Documentation Practices.

> Evidenced

Yes No Elsewhere

+---------------------------------------+---------+---------+---------+
| #### **Scope**                        |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Test Objectives**              |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Test Approach                     |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Dependencies and Constraints** |         |         |         |
+---------------------------------------+---------+---------+---------+
| #                                     |         |         |         |
| ### **Test Environment Requirements** |         |         |         |
|                                       |         |         |         |
| ####                                  |         |         |         |
|                                       |         |         |         |
| ####                                  |         |         |         |
|                                       |         |         |         |
| #### Test Environment Requirements    |         |         |         |
+---------------------------------------+---------+---------+---------+
| ###                                   |         |         |         |
|  Problem and Status Reporting Methods |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Test Deliverables**            |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Staffing and Training Needs       |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **                               |         |         |         |
| Test Schedules and Responsibilities** |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Open Issues**                  |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Test Sc                           |         |         |         |
| enarios, Test Cases, and Test Scripts |         |         |         |
+---------------------------------------+---------+---------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| #### **Test Cases**                   |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** The INIT Test Cases met Year 2000 Testing
Quality Assurance Baseline Documentation Practices, with the exception
that INIT Test Case instructions were not documented, but were relied
upon by the knowledge of the test team.

**DETAIL COMPARISONS** Evidenced

Yes No Elsewhere

+---------------------------------------+---------+---------+---------+
| #### **Test Case Identification**     |         |         |         |
+---------------------------------------+---------+---------+---------+
| **Focuses on Components identified in |         |         |         |
| the Decomposition Inventory**         |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Status of Test Case**          |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Busine                            |         |         |         |
| ss Function/Test Scenario Identified  |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Purpose of Test**              |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Instructions**                 |         |         |         |
+---------------------------------------+---------+---------+---------+
| Test Case instructions were not       |         |         |         |
| documented, but relied, instead, on   |         |         |         |
| the knowledge of the testers.         |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Expected Results                  |         |         |         |
+---------------------------------------+---------+---------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ###### Test Results                   |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** The INIT Test Results met Year 2000 Testing
Quality Assurance Baseline Documentation Practices.

**DETAIL COMPARISONS**

> Evidenced
>
> Yes No Elsewhere

+---------------------------------------+---------+---------+---------+
| #### **Test Case Identification**     |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Test Date**                    |         |         |         |
+---------------------------------------+---------+---------+---------+
| Test dates were reflected in the      |         |         |         |
| subsystem test plan and Revisions to  |         |         |         |
| the overall test plan.                |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Actual Results**               |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Test Summary**                 |         |         |         |
+---------------------------------------+---------+---------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ###### Problem Tracking               |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** There was not an established, formal method
used to track problems that arose during testing activities.

**DETAIL COMPARISONS**

> Evidenced

Yes No Elsewhere

+---------------------------------------+---------+---------+---------+
| #### **Problem Identification**       |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Problem Description               |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Submit Date**                  |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Problem Status                    |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Resolve Date**                 |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Resolution**                   |         |         |         |
+---------------------------------------+---------+---------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
|                                       |         |         |         |
+---------------------------------------+---------+---------+---------+
| ###### Management Issue Log           |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** No management issue log was evidenced, although
management issues were tracked via regular discussions/meetings between
testers and system management.

**DETAIL COMPARISONS**

> Evidenced

Yes No Elsewhere

+---------------------------------------+---------+---------+---------+
| #### **Issue Identification**         |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Issue Description                 |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Submit Date**                  |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Severity Level                    |         |         |         |
+---------------------------------------+---------+---------+---------+
| ### Escalated To                      |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Issue Status**                 |         |         |         |
+---------------------------------------+---------+---------+---------+
| #### **Resolve Date**                 |         |         |         |
+---------------------------------------+---------+---------+---------+

[^1]: There is no overall rating given for this critical success factor,
    due to the fact all baseline testing practices are not applicable
    for this subsystem.