***Department of Labor***

***International Price System***

***Index Review Subsystem***

***(Merger)***

##### **Year 2000 Testing Quality Assurance** 

**TQA SUMMARY REPORT ADDENDUM**

######### [TABLE OF CONTENTS]{.underline}

INDEX REVIEW SUBSYSTEM REVIEW tqa DETAIL RESULTS \-- ADDENDUM

I. INTRODUCTION A-1

II\. INDEX REVIEW SUBSYSTEM TQA REVIEW SUMMARY TABLES A-2

III\. INDEX REVIEW SUBSYSTEM TQA REVIEW DETAIL RESULTS A-3

> A. CRITICAL SUCCESS FACTORS \-- REVIEW RESULTS A-3
>
> 1.0 Achieved DOL Year 2000 Objectives A-4
>
> 2.0 Staffed Test Team Appropriately A-7
>
> 3.0 Used Compliant Year 2000 Test Environment ..A-8
>
> 4.0 Identified Key Processes and Date Impacted Business Functions A-10
>
> 5.0 Created Testing Documentation that is Logically Linked A-12
>
> 6.0 Identified and Tested System Interfaces A-13
>
> 7.0 Executed Tests According to Plan A-14
>
> 8.0 Completed Tests with Satisfactory Results A-16
>
> 9.0 Followed Quality Assurance Procedures A-17

B. BASELINE TESTING DOCUMENTATION \-- REVIEW RESULTS A-18

> Test Plan A-19
>
> Test Cases A-20
>
> Test Results A-20
>
> Problem Tracking A-21
>
> Management Issue Log A-21

**I. INTRODUCTION**

This addendum to the International Price System Testing Quality
Assurance Review Summary Report contains the individual observations
made by the Testing Quality Assurance Review team for the Index Review
Subsystem Year 2000 testing activities. We compared the Year 2000
testing activities performed for the Index Review Subsystem to
requirements issued by the Department of Labor (DOL) and the General
Accounting Office (GAO). We also compared testing efforts to generally
accepted systems testing standards. In this addendum, we describe the
results of our review in terms of nine [Critical Success
Factors]{.underline} and five categories of [Baseline Testing
Documentation]{.underline}. These Critical Success Factors and Baseline
Testing Documentation categories were used to structure our review of
the Index Review Subsystem Year 2000 test activities and to develop our
observations. These observations and the results of our review were
derived from documentation reviews, interviews and discussions conducted
by the TQA team. Critical Success Factors represent key segments of the
Year 2000 testing process and Baseline Testing Documentation categories
represent a set of documents that help to provide the application team
with assurances that critical steps of the Year 2000 testing process
were conducted and documented. We describe each of the nine Critical
Success Factors and Baseline Testing Documentation areas by using one of
three indicators; 1) Meets Baseline Practices, 2) Management Awareness
Advised, or 3) Management Review Advised. The indicator is developed
from the analysis of baseline testing practices that comprise a critical
success factor or the Baseline Testing Documentation. The actual rating
is substantiated by the results of reviewing supporting baseline testing
practices and observations that support each overall rating. The
definition for each of the ratings is:

> **1) Meets Baseline Practices** **(MBP)** -- The TQA Team found that
> the Critical Success Factor or the Testing Documentation contents that
> were reviewed **adequately** **satisfied Baseline Practices.**
>
> **2) Management Awareness Advised** **(MAA)** -- The TQA Team found
> that the Critical Success Factor or the Testing Documentation Contents
> **DID NOT satisfy one or more of the Baseline Practices and may
> warrant further management review.**

**3) Management Review Advised (MRA) --** Significant observations
support the conclusion that **Baseline Practices WERE NOT followed**
**or NOT supported** in the testing documentation. **Further management
review is advised.**

  ----------------------------------- -----------------------------------
  **MBP = Meets Baseline Practices**  **ADEQUATELY SATISFIED baseline
                                      practices and documentation
                                      elements.**

  **MAA = Management Awareness        **DID NOT SATISFY one or more of
  Advised**                           the baseline practices and/or
                                      documentation elements.**

  **MRA = Management Review Advised** **Baseline practices and/or
                                      documentation elements WERE NOT
                                      FOLLOWED or NOT SUPPORTED.**
  ----------------------------------- -----------------------------------

**II. INDEX REVIEW SUBSYSTEM REVIEW TQA SUMMARY TABLES**

The following two summary tables are the result of consolidating the
supporting detail evaluated by the TQA team for each Critical Success
Factor and each Baseline Testing Documentation item. To fully understand
how the TQA team arrived at the summary of observations, one should also
carefully review the supporting detail that is presented in the Index
Review Subsystem TQA Detail Results sections that follow the Summary
Tables.

+--------------------------------------+---------+---------+---------+
| ######## Critical Success Factors    | **Meets | **Man   | **Man   |
|                                      | B       | agement | agement |
|                                      | aseline | Aw      | Review  |
|                                      | Prac    | areness | Ad      |
|                                      | tices** | Ad      | vised** |
|                                      |         | vised** |         |
+--------------------------------------+---------+---------+---------+
| 1.  **Achieved DOL Year 2000         |         |         |         |
|     Objectives**                     |         |         |         |
+--------------------------------------+---------+---------+---------+
| 2.  **Staffed Test Team              |         |         |         |
|     Appropriately**                  |         |         |         |
+--------------------------------------+---------+---------+---------+
| 3.  **Used Compliant Year 2000 Test  |         |         |         |
|     Environment**                    |         |         |         |
+--------------------------------------+---------+---------+---------+
| 4.  **Identified Key Processes &     |         |         |         |
|     Date Impacted Business           |         |         |         |
|     Functions**                      |         |         |         |
+--------------------------------------+---------+---------+---------+
| 5.  **Created Testing Documentation  |         |         |         |
|     that is Logically Linked**       |         |         |         |
+--------------------------------------+---------+---------+---------+
| 6.  **Identified & System Tested     |         |         |         |
|     Interfaces** [^1]                |         |         |         |
+--------------------------------------+---------+---------+---------+
| 7.  **Executed Tests According to    |         |         |         |
|     Plan**                           |         |         |         |
+--------------------------------------+---------+---------+---------+
| 8.  **Completed Tests with           |         |         |         |
|     Satisfactory Results**           |         |         |         |
+--------------------------------------+---------+---------+---------+
| 9.  **Followed Quality Assurance     |         |         |         |
|     Procedures**                     |         |         |         |
+--------------------------------------+---------+---------+---------+

+--------------------------------------+---------+---------+---------+
| ######## Bas                         | **Meets | **Man   | **Man   |
| eline Testing Documentation Contents | B       | agement | agement |
|                                      | aseline | Aw      | Review  |
|                                      | Prac    | areness | Ad      |
|                                      | tices** | Ad      | vised** |
|                                      |         | vised** |         |
+--------------------------------------+---------+---------+---------+
| #### **Test Plan**                   |         |         |         |
+--------------------------------------+---------+---------+---------+
| **Test Cases**                       |         |         |         |
+--------------------------------------+---------+---------+---------+
| **Test Results**                     |         |         |         |
+--------------------------------------+---------+---------+---------+
| **Problem Tracking Log**             |         |         |         |
+--------------------------------------+---------+---------+---------+
| **Management Issue Log**             |         |         |         |
+--------------------------------------+---------+---------+---------+

The TQA Review process analyzed the testing practices used by the Index
Review Subsystem testing team in the Year 2000 testing processes of
preparation, execution, and validation. This was accomplished by 1)
reviewing testing and 2) conducting interviews to help the TQA Review
teams understand the actual practices used by the Index Review Subsystem
test team. The Index Review Subsystem Year 2000 testing activities were
assessed against Year 2000 requirements set forth by the Department of
Labor and the General Accounting Office (GAO). In addition, we assessed
the Index Review Subsystem Year 2000 testing procedures by evaluating
them against processes that are recognized as generally accepted
software configuration management and testing practices.

## III. Index Review Subsystem TQA - DETAIL RESULTS

# A. CRITICAL SUCCESS FACTORS -- DETAIL TQA RESULTS

In our Testing Quality Assurance Review process, Critical Success
Factors are comprised of several individual baseline testing practices.
An "Overall Indicator" for each Critical Success Factor, along with the
supporting observations, are presented in the detailed results section,
starting on the next page.

**Overall Indicator \--** The overall rating for each Critical Success
Factor is based on the combined result of: 1) reviewing underlying
baseline practices, 2) understanding the application and the
documentation context, and 3) recording the consensus evaluation of the
TQA Review team. The TQA Review team assessed these combined factors as
a group of actual practices that support (or do not support) the desired
(Critical Success Factor) outcome.

**Detail Comparisons --** Each of the Critical Success Factors is
followed by supporting detail comparisons derived from the results of
the TQA Review. Each detailed comparison consists of an individually
numbered baseline testing practice, followed by the supporting (actual
practice) detail recorded by the TQA team. The result of each comparison
between baseline practice and actual practice of the Index Review
Subsystem test team, is indicated as **["YES"]{.underline},
["NO"]{.underline}, or ["N/A"]{.underline}**:

> **[YES]{.underline}** - The TQA Review team found this baseline
> practice to be met based on the documentation sample(s) reviewed
> and/or actual practice information provided in test team interviews.
>
> **[NO]{.underline}** - The TQA Review team found this baseline
> practice was not met based on the documentation sample(s) reviewed
> and/or actual practice information provided in test team interviews.
>
> **[N/A]{.underline}** - The Application team indicated that this
> baseline practice was not applicable to the application under review.

# CRITICAL SUCCESS FACTORS - REVIEW RESULTS 

+--------------------------------------+---------+----------+---------+
|                                      | **Meets | **Ma     | **Man   |
|                                      | B       | nagement | agement |
|                                      | aseline | A        | Review  |
|                                      | Prac    | wareness | Ad      |
|                                      | tices** | A        | vised** |
|                                      |         | dvised** |         |
+--------------------------------------+---------+----------+---------+
| ## 1                                 |         |          |         |
| .0 Achieved DOL Year 2000 Objectives |         |          |         |
+--------------------------------------+---------+----------+---------+

**OVERALL OBSERVATION:**

**DETAIL COMPARISONS:**

Yes No Not Applicable

+----------------------------------------+-------+---+---------+---------+
| 1.  **An application component         |       |   |         |         |
|     compliance checklist was           |       |   |         |         |
|     prepared**                         |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 2.  **Non-compliant components were    |       |   |         |         |
|     assessed for Year 2000 impact**    |       |   |         |         |
|                                        |       |   |         |         |
| **See section 3.1 for further          |       |   |         |         |
| discussion of non-compliant components |       |   |         |         |
| in Year 2000 test environment**        |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 3.  **External data feeds identified   |       |   |         |         |
|     and owners contacted**             |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 4.  **Test plans are within DOL/BLS    |       |   |         |         |
|     Year 2000 guidelines**             |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| Test plans generally are within DOL    |       |   |         |         |
| Y2K guidelines.                        |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 5.  **Test plans are within Federal,   |       |   |         |         |
|     private sector and other           |       |   |         |         |
|     professional standard Year 2000    |       |   |         |         |
|     Testing**                          |       |   |         |         |
|                                        |       |   |         |         |
| > **Guidelines**                       |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| While test plans generally met         |       |   |         |         |
| Federal, private sector and other      |       |   |         |         |
| professional Year 2000 testing         |       |   |         |         |
| guidelines, they did not identify      |       |   |         |         |
| separate levels of testing for         |       |   |         |         |
| baseline, unit, integration,           |       |   |         |         |
| independent user acceptance and end to |       |   |         |         |
| end test procedures. Independent Year  |       |   |         |         |
| 2000 acceptance testing was not        |       |   |         |         |
| performed for the Merger subsystem.    |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 6.  **Testing methodologies selected   |       |   |         |         |
|     were appropriate for Remediation   |       |   |         |         |
|     methodology used**                 |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| See 1.5 above.                         |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 7.  **Unique application high risk     |       |   |         |         |
|     dates were documented and tested** |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| Subsystem level date processing        |       |   |         |         |
| features (e.g., date sorting, date     |       |   |         |         |
| display and date outputs to reports)   |       |   |         |         |
| were identified by system developers   |       |   |         |         |
| using the BLS Year 2000 Subsystem      |       |   |         |         |
| Cross-reference Worksheet and          |       |   |         |         |
| Compliance Checklist and were          |       |   |         |         |
| identified as "points of concern"      |       |   |         |         |
| (POCs). Dates identified were limited  |       |   |         |         |
| to point of concern guidance and       |       |   |         |         |
| unique application high risk dates     |       |   |         |         |
| beyond those were not evidenced.       |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 8.  **Defined system acceptance        |       |   |         |         |
|     testing requirements:**            |       |   |         |         |
|                                        |       |   |         |         |
| > **Functional, performance and        |       |   |         |         |
| > security**                           |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| Acceptance Test Standards have been    |       |   |         |         |
| defined for the IPP system which       |       |   |         |         |
| address the need to establish these    |       |   |         |         |
| requirements during acceptance testing |       |   |         |         |
| efforts. During Year 2000 related      |       |   |         |         |
| acceptance testing, subsystem level    |       |   |         |         |
| acceptance testing requirements were   |       |   |         |         |
| not developed for performance and      |       |   |         |         |
| security. Additionally, no independent |       |   |         |         |
| user acceptance testing was conducted. |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 9.  **Strategy was developed for       |       |   |         |         |
|     baseline testing**                 |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| Y2K related baseline testing was not   |       |   |         |         |
| performed.                             |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| **1.10 Strategy was developed for      |       |   |         |         |
| current date testing**                 |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
|                                        |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 11. **Strategy was developed for       |       |   |         |         |
|     future date testing**              |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| **1.12 Strategy was developed for      |       |   |         |         |
| simulated interface testing**          |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
|                                        |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| **1.13 Strategy was developed for      |       |   |         |         |
| testing of bridges**                   |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
|                                        |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 14. **Metrics defined for categories   |       |   |         |         |
|     of defect severity**               |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| Metrics for "quantity of errors found" |       |   |         |         |
| and "quantity of errors resolved" were |       |   |         |         |
| briefly described in the IPS Test      |       |   |         |         |
| Plan. No evidence was provided of      |       |   |         |         |
| metrics maintained for categories of   |       |   |         |         |
| defect severity.                       |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 15. **End-to-end test procedures and   |       |   |         |         |
|     data prepared**                    |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| The purpose of end-to-end testing is   |       |   |         |         |
| described by GAO to verify that a      |       |   |         |         |
| defined set of interrelated systems    |       |   |         |         |
| which collectively support an          |       |   |         |         |
| organizational core function           |       |   |         |         |
| interoperate as intended in an         |       |   |         |         |
| operational environment, either actual |       |   |         |         |
| or simulated. Those interrelated       |       |   |         |         |
| systems include not only those owned   |       |   |         |         |
| and managed by the organization, but   |       |   |         |         |
| also the external systems with which   |       |   |         |         |
| they interface. A separate end to end  |       |   |         |         |
| testing process was not performed for  |       |   |         |         |
| the IPS subsystems to include all      |       |   |         |         |
| internal and/or external interfaces,   |       |   |         |         |
| infrastructure and telecommunications, |       |   |         |         |
| along with the fully integrated        |       |   |         |         |
| application. According to BLS          |       |   |         |         |
| Management, reliance for end to end    |       |   |         |         |
| test coverage was placed upon test     |       |   |         |         |
| scripts and test cases executed during |       |   |         |         |
| point of concern testing and           |       |   |         |         |
| acceptance testing which were expected |       |   |         |         |
| to address subsystem interoperations.  |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 16. **Project status reports include   |       |   |         |         |
|     progress tracking**                |       |   |         |         |
|                                        |       |   |         |         |
| ### Mechanisms                         |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| .                                      |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 17. **Change control procedures and    |       |   |         |         |
|     software**                         |       |   |         |         |
|                                        |       |   |         |         |
| #######  Configur                      |       |   |         |         |
| ation management processes are defined |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
|                                        |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| 18. **Processes and metrics for        |       |   |         |         |
|     reporting test activity and        |       |   |         |         |
|     Progress are established**         |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| **1.19 Results of initial baseline     |       |   |         |         |
| testing, test plans, and a**           |       |   |         |         |
|                                        |       |   |         |         |
| ### D                                  |       |   |         |         |
| escription of the test data maintained |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+
| Year 2000 baseline testing was not     |       |   |         |         |
| performed.                             |       |   |         |         |
+----------------------------------------+-------+---+---------+---------+

+--------------------------------------+---------+---------+---------+
|                                      | **Meets | **Man   | **Man   |
|                                      | B       | agement | agement |
|                                      | aseline | Aw      | Review  |
|                                      | Prac    | areness | Ad      |
|                                      | tices** | Ad      | vised** |
|                                      |         | vised** |         |
+--------------------------------------+---------+---------+---------+
| ##                                   |         |         |         |
|  2.0 Staffed Test Team Appropriately |         |         |         |
+--------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:**

**DETAIL COMPARISONS:**

Not

Yes No Applicable

+----------------------------------------+--------+----------+--------+
| 1.  **Required resources for testing   |        |          |        |
|     were identified**                  |        |          |        |
+----------------------------------------+--------+----------+--------+
| 2.  **Subject Matter Experts were      |        |          |        |
|     involved in creation and execution |        |          |        |
|     of tests**                         |        |          |        |
+----------------------------------------+--------+----------+--------+
| Independent user acceptance testing    |        |          |        |
| was not performed. However, management |        |          |        |
| stated that users were involved during |        |          |        |
| the test planning and execution        |        |          |        |
| process for the Merger subsystem.      |        |          |        |
+----------------------------------------+--------+----------+--------+
| 3.  **Test team displays general       |        |          |        |
|     experience and understanding of    |        |          |        |
|     standard testing methods and       |        |          |        |
|     procedures**                       |        |          |        |
+----------------------------------------+--------+----------+--------+
|                                        |        |          |        |
+----------------------------------------+--------+----------+--------+
| #### **Interorganizational             |        |          |        |
| end-to-end test team (s) established** |        |          |        |
+----------------------------------------+--------+----------+--------+
| Year 2000 end-to-end testing was not   |        |          |        |
| performed, see 1.15.                   |        |          |        |
+----------------------------------------+--------+----------+--------+
| #### **Plan                            |        |          |        |
| ned resources were available when requ |        |          |        |
| ired for the duration of the project** |        |          |        |
+----------------------------------------+--------+----------+--------+

+--------------------------------------+---------+---------+---------+
|                                      | **Meets | **Man   | **Man   |
|                                      | B       | agement | agement |
|                                      | aseline | Aw      | Review  |
|                                      | Prac    | areness | Ad      |
|                                      | tices** | Ad      | vised** |
|                                      |         | vised** |         |
+--------------------------------------+---------+---------+---------+
| ## 3.0 Used                          |         |         |         |
| Compliant Year 2000 Test Environment |         |         |         |
+--------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:**

**DETAIL COMPARISONS:**

> Not

Yes No Applicable

+---------------------------------------+--------+----------+---------+
| 1.  **Current and future date system  |        |          |         |
|     tests performed in a Year 2000    |        |          |         |
|     test environment**                |        |          |         |
+---------------------------------------+--------+----------+---------+
| The IPS subsystem testing was         |        |          |         |
| performed in a simulated Year 2000    |        |          |         |
| testing environment. During Year 2000 |        |          |         |
| testing efforts, this environment     |        |          |         |
| contained components that were not    |        |          |         |
| reported as fully compliant by the    |        |          |         |
| related vendors. BLS management       |        |          |         |
| performed an analysis of the majority |        |          |         |
| of these components and is relying on |        |          |         |
| the ability of these products to      |        |          |         |
| operate in the Year 2000 with the     |        |          |         |
| non-compliant features of the         |        |          |         |
| specific components.                  |        |          |         |
|                                       |        |          |         |
| Guidance issued by the GAO states     |        |          |         |
| that, "organizations should implement |        |          |         |
| plans for mitigating the impact of    |        |          |         |
| vendor products and services not      |        |          |         |
| being compliant on time." The GAO     |        |          |         |
| Year 2000 Testing Guide states that   |        |          |         |
| "In order to execute system           |        |          |         |
| acceptance tests and ensure system    |        |          |         |
| Year 2000 compliance, all system      |        |          |         |
| components (application software,     |        |          |         |
| systems software, hardware, firmware, |        |          |         |
| and communications networks) must be  |        |          |         |
| compliant and ready for testing as an |        |          |         |
| integrated system"                    |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Change management p            |        |          |         |
| rocedures support promotion of applic |        |          |         |
| ation components into testing area**  |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Chan                           |        |          |         |
| ge management procedures utilized for |        |          |         |
|  retention and promotion of the code  |        |          |         |
| from test environment to production** |        |          |         |
+---------------------------------------+--------+----------+---------+
|                                       |        |          |         |
+---------------------------------------+--------+----------+---------+
| **3.4 End-to-end test(s) were         |        |          |         |
| executed**                            |        |          |         |
+---------------------------------------+--------+----------+---------+
| No end-to-end tests were performed.   |        |          |         |
| (see also Critical Success Factor     |        |          |         |
| 1.15) The IPS system manager is       |        |          |         |
| relying upon vendor certification and |        |          |         |
| the BLS Certification Lab to ensure   |        |          |         |
| compliance of the IPS supporting      |        |          |         |
| infrastructure.                       |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Minimum                        |        |          |         |
| Year 2000 system compliant date sensi |        |          |         |
| tive requirements established (e.g.,  |        |          |         |
| Special Dates, Date Intervals, Days o |        |          |         |
| f the Year and Miscellaneous Dates)** |        |          |         |
+---------------------------------------+--------+----------+---------+
| Minimum Year 2000 system compliant    |        |          |         |
| date sensitive requirements were      |        |          |         |
| established by the Merger application |        |          |         |
| test team. (See Critical Success      |        |          |         |
| Factor 7.0)                           |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Test envi                      |        |          |         |
| ronment enables future date testing** |        |          |         |
+---------------------------------------+--------+----------+---------+
| The Merger test environment included  |        |          |         |
| future date testing                   |        |          |         |
+---------------------------------------+--------+----------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 4.0 Identified Key Processes       |         |         |         |
|  and Date Impacted Business Functions |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION**:

**DETAIL COMPARISONS:**

Not

Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **Application was decomposed to    |        |         |         |
|     identify business functions,       |        |         |         |
|     files, databases, and processing   |        |         |         |
|     cycles that are affected by date   |        |         |         |
|     processing**                       |        |         |         |
+----------------------------------------+--------+---------+---------+
| System developers used point of        |        |         |         |
| concern guidance which required them   |        |         |         |
| to identify potential Year 2000 issues |        |         |         |
| by business functions of the           |        |         |         |
| subsystem.                             |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Test scripts were developed      |        |         |         |
|     using the application              |        |         |         |
|     decomposition inventory**          |        |         |         |
+----------------------------------------+--------+---------+---------+
| Test scripts were derived from the     |        |         |         |
| application decomposition performed by |        |         |         |
| system developers, in identifying      |        |         |         |
| "critical subsystems" and "points of   |        |         |         |
| concern".                              |        |         |         |
+----------------------------------------+--------+---------+---------+
| 3.  **Business critical archiving and  |        |         |         |
|     history functionality has been     |        |         |         |
|     included in future date tests**    |        |         |         |
+----------------------------------------+--------+---------+---------+
| Subsystem descriptions did not include |        |         |         |
| archiving and history functionality    |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **Unique points of failure that    |        |         |         |
|     could occur prior to the Year 2000 |        |         |         |
|     have been considered in the        |        |         |         |
|     planning process**                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| Unique points of failure prior to the  |        |         |         |
| Year 2000 were not identified as       |        |         |         |
| points of concern. Dates identified    |        |         |         |
| were limited to point of concern       |        |         |         |
| guidance and unique application points |        |         |         |
| of failure prior to the Year 2000 were |        |         |         |
| not evidenced                          |        |         |         |
+----------------------------------------+--------+---------+---------+
| 5.  **Test planning had representation |        |         |         |
|     from subject matter experts**      |        |         |         |
+----------------------------------------+--------+---------+---------+
| While no independent user acceptance   |        |         |         |
| testing was performed, management      |        |         |         |
| stated that user involvement did occur |        |         |         |
| during the test planning process for   |        |         |         |
| the Merger subsystem.                  |        |         |         |
+----------------------------------------+--------+---------+---------+
| 6.  **System acceptance test criteria, |        |         |         |
|     procedures and data are defined**  |        |         |         |
+----------------------------------------+--------+---------+---------+
| No independent acceptance testing was  |        |         |         |
| conducted for the Merger subsystem.    |        |         |         |
+----------------------------------------+--------+---------+---------+
| 7.  **Software integration test        |        |         |         |
|     criteria defined**                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| Management relied upon integration     |        |         |         |
| testing through the Unified Database   |        |         |         |
| during POC and AT testing. However,    |        |         |         |
| separate integration test criteria was |        |         |         |
| not defined.                           |        |         |         |
+----------------------------------------+--------+---------+---------+
| 8.  **System boundaries of end-to-end  |        |         |         |
|     test(s) have been defined**        |        |         |         |
+----------------------------------------+--------+---------+---------+
| Documentation of defined system        |        |         |         |
| boundaries was provided, although no   |        |         |         |
| end-to-end testing was performed. (See |        |         |         |
| Critical Success Factor 1.15.)         |        |         |         |
+----------------------------------------+--------+---------+---------+
| 9.  **Edits for invalid date           |        |         |         |
|     processing were included in test   |        |         |         |
|     cases**                            |        |         |         |
+----------------------------------------+--------+---------+---------+
| Edits for invalid date processing were |        |         |         |
| not evident in the identification of   |        |         |         |
| points of concern or in test cases.    |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 5.0 Created Testing Do             |         |         |         |
| cumentation that is Logically Linked  |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION**:

**DETAIL COMPARISONS:**

> Not
>
> Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **Defined conditions or            |        |         |         |
|     requirements for Unit test         |        |         |         |
|     procedures, and data to test       |        |         |         |
|     correct date handling.**           |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Test plans have associated       |        |         |         |
|     scripts, cases, execution logs,    |        |         |         |
|     and test results that are          |        |         |         |
|     logically linked.**                |        |         |         |
+----------------------------------------+--------+---------+---------+
| Test plans contain test scripts and    |        |         |         |
| test cases. No execution logs were     |        |         |         |
| available to track testing as it was   |        |         |         |
| performed. Test results could not be   |        |         |         |
| logically linked to test cases, points |        |         |         |
| of concern and BLS Year 2000           |        |         |         |
| compliance criteria.                   |        |         |         |
+----------------------------------------+--------+---------+---------+
| **5.3 Test results and data have been  |        |         |         |
| archived for re-execution and          |        |         |         |
| comparison**                           |        |         |         |
+----------------------------------------+--------+---------+---------+
| Only some system test results were     |        |         |         |
| retained. Although the IPP test team   |        |         |         |
| stated that data had been archived, it |        |         |         |
| was not evident.                       |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **Test plan identifies how test    |        |         |         |
|     data will be obtained or created** |        |         |         |
+----------------------------------------+--------+---------+---------+
| **5.5 Test plan identifies how defects |        |         |         |
| are to be corrected**                  |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 6.  **The test plan identifies how the |        |         |         |
|     data will be aged**                |        |         |         |
+----------------------------------------+--------+---------+---------+
| Systemic data aging was performed by   |        |         |         |
| aging the dates in the unified         |        |         |         |
| database. SQL scripts were developed   |        |         |         |
| to changed dates in the database to    |        |         |         |
| reflect dates that would be in the     |        |         |         |
| database when future date testing was  |        |         |         |
| performed. Aging was based on the      |        |         |         |
| knowledge of the test team.            |        |         |         |
+----------------------------------------+--------+---------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 6.0 Id                             |         |         |         |
| entified and Tested System Interfaces |         |         |         |
+---------------------------------------+---------+---------+---------+

**Overall Observation:**

**Detail comparisons:**

Not

Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **All critical interfaces were     |        |         |         |
|     simulated in future date tests**   |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Integration testing includes     |        |         |         |
|     interface tests to ensure that     |        |         |         |
|     when sub-systems, when combined,   |        |         |         |
|     work together as intended.**       |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 3.  **Simulated interface data         |        |         |         |
|     represents all possible            |        |         |         |
|     date-impacted business functions   |        |         |         |
|     affected by this interface.**      |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **Interfaces with two digit years  |        |         |         |
|     have been checked for previously   |        |         |         |
|     communicated pivot point values**  |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+

## 

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ##                                    |         |         |         |
|  7.0 Executed Tests According to Plan |         |         |         |
+---------------------------------------+---------+---------+---------+

**Overall Observation**:

**Detail Comparisons:**

> Not

Yes No Applicable

+---------------------------------------------+------+-------+---------+
| 1.  **Baseline tests were executed prior to |      |       |         |
|     19xx and 20xx tests**                   |      |       |         |
+---------------------------------------------+------+-------+---------+
| Baseline testing for Year 2000 tests were   |      |       |         |
| not conducted                               |      |       |         |
+---------------------------------------------+------+-------+---------+
| 2.  **Test cases were developed to test     |      |       |         |
|     according to the type of renovation     |      |       |         |
|     method used**                           |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+
| 3.  **Test summary log shows when tests     |      |       |         |
|     were executed and verified**            |      |       |         |
+---------------------------------------------+------+-------+---------+
| No test summary logs are documented.        |      |       |         |
+---------------------------------------------+------+-------+---------+
| 4.  **An issue log was maintained which     |      |       |         |
|     includes status and resolutions**       |      |       |         |
+---------------------------------------------+------+-------+---------+
| It was not evident that testers maintained  |      |       |         |
| a log that tracked the status and           |      |       |         |
| resolution of issues that arose during the  |      |       |         |
| testing process.                            |      |       |         |
+---------------------------------------------+------+-------+---------+
| 5.  **Test progress reporting identifies    |      |       |         |
|     deviation from plans and appropriate    |      |       |         |
|     action is taken to address any          |      |       |         |
|     deviation**                             |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+
| 6.  **A problem log was maintained which    |      |       |         |
|     includes status and resolutions**       |      |       |         |
+---------------------------------------------+------+-------+---------+
| It was not evident that management          |      |       |         |
| maintained a log to track the status and    |      |       |         |
| resolutions of problems that arose during   |      |       |         |
| the testing process.                        |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+
| 7.  **All date related business functions   |      |       |         |
|     have associated test data**             |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+
| 8.  **Test cases, scripts, and results met  |      |       |         |
|     test plan objectives**                  |      |       |         |
+---------------------------------------------+------+-------+---------+
| Generally, it appears that test plan        |      |       |         |
| objectives were met, however, test results  |      |       |         |
| could not be logically linked to test       |      |       |         |
| cases, points of concern and BLS Year 2000  |      |       |         |
| compliance criteria.                        |      |       |         |
+---------------------------------------------+------+-------+---------+
| 7.9 Test cases and scripts were executed    |      |       |         |
| according to the documented test plans      |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+
| **7.10 Execute test according to DOL        |      |       |         |
| compliance criteria**                       |      |       |         |
+---------------------------------------------+------+-------+---------+
|                                             |      |       |         |
+---------------------------------------------+------+-------+---------+

##  

## 

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 8.0 Comp                           |         |         |         |
| leted Tests with Satisfactory Results |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION**:

**DETAILED COMPARISONS:**

> Not Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **19xx and 20xx test results were  |        |         |         |
|     validated against the results of   |        |         |         |
|     the baseline test**                |        |         |         |
+----------------------------------------+--------+---------+---------+
| Baseline testing for Year 2000 was not |        |         |         |
| performed and documentation for        |        |         |         |
| baseline testing was not provided.     |        |         |         |
| (see section 7.1 and 1.9)              |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **The acceptance criteria for the  |        |         |         |
|     testing process has been           |        |         |         |
|     determined**                       |        |         |         |
+----------------------------------------+--------+---------+---------+
| Acceptance criteria was identified for |        |         |         |
| the overall testing process, not       |        |         |         |
| identified at the individual test case |        |         |         |
| level.                                 |        |         |         |
+----------------------------------------+--------+---------+---------+
| 3.  **The approach for validating the  |        |         |         |
|     test results has been identified\  |        |         |         |
|     **                                 |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 4.  **The expected results are         |        |         |         |
|     documented in the scripts**        |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 5.  **Differences in actual and        |        |         |         |
|     expected results were analyzed and |        |         |         |
|     reconciled**                       |        |         |         |
+----------------------------------------+--------+---------+---------+
| Pending                                |        |         |         |
+----------------------------------------+--------+---------+---------+
| 6.  **Specified test conditions        |        |         |         |
|     (logical scenarios) are met and    |        |         |         |
|     sign-offs from IV + V/QA agent,    |        |         |         |
|     configuration management and       |        |         |         |
|     functional management have been    |        |         |         |
|     obtained**                         |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 7.  **Test results were validated by   |        |         |         |
|     Subject Matter Experts**           |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+

## 

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ## 9.0                                |         |         |         |
| Followed Quality Assurance Procedures |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:**

**DETAIL COMPARISONS:**

Not

Yes No Applicable

+----------------------------------------+--------+---------+---------+
| 1.  **The test plans, test scripts and |        |         |         |
|     test cases were validated by       |        |         |         |
|     subject matter experts**           |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| #### **Y                               |        |         |         |
| ear 2000-related issues and problems w |        |         |         |
| ere all resolved in a timely manner**  |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| #### Peer reviews were co              |        |         |         |
| nducted on test plans and test results |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+
| 2.  **Test team held regular meetings  |        |         |         |
|     to resolve technical and           |        |         |         |
|     procedural problems**              |        |         |         |
+----------------------------------------+--------+---------+---------+
|                                        |        |         |         |
+----------------------------------------+--------+---------+---------+

# B. BASELINE TESTING DOCUMENTATION -- DETAIL TQA RESULTS 

In addition to the comparison of the Year 2000 testing practices to the
baseline practices, the TQA Review process also involves assessing the
application testing team's documentation against a set of generally
accepted baseline documentation content standards (elements that should
be represented in those documents). These methods and procedures have
been converted to a list of key testing documents for the purposes of
identifying whether documentation standards for Year 2000 testing
efforts were achieved.

Each of the individual document's contents was evaluated by the TQA
Review team and described by one of the following indicators:

> **[YES]{.underline}** - The TQA Review team found this element was
> documented to satisfy generally accepted Year 2000 test documentation
> guidelines, based on the sample reviewed.
>
> **[NO]{.underline}** - The TQA Review team found this element was not
> documented to satisfy generally accepted Year 2000 test documentation
> guidelines, based on the sample reviewed.
>
> **[Evidenced Elsewhere]{.underline} -** The TQA Review team found that
> the element required for this individual document existed but was
> supported elsewhere as part of the testing of this application.

The indication of an individual element within a Baseline document
category as *NO* or *Evidenced Elsewhere* does not necessarily
correspond to the entire document category receiving a rating of
Awareness or Review Advised. The rating for the entire category is based
on how the TQA Review team assessed the overall effectiveness of the
group of elements against Year 2000 guidelines.

# BASELINE TESTING DOCUMENTATION - REVIEW RESULTS 

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| #### **Test Plan**                    |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:**

**DETAIL COMPARISONS:**

> Evidenced Yes No Elsewhere

+---------------------------------------+--------+----------+---------+
| #### **Scope**                        |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Test Objectives**              |        |          |         |
+---------------------------------------+--------+----------+---------+
| ### Test Approach                     |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Dependencies and Constraints** |        |          |         |
+---------------------------------------+--------+----------+---------+
| #                                     |        |          |         |
| ### **Test Environment Requirements** |        |          |         |
|                                       |        |          |         |
| ####                                  |        |          |         |
|                                       |        |          |         |
| ####                                  |        |          |         |
|                                       |        |          |         |
| #### Test Environment Requirements    |        |          |         |
+---------------------------------------+--------+----------+---------+
| ###                                   |        |          |         |
|  Problem and Status Reporting Methods |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Test Deliverables**            |        |          |         |
|                                       |        |          |         |
| ####                                  |        |          |         |
|                                       |        |          |         |
| ####                                  |        |          |         |
|                                       |        |          |         |
| #### Test Environment Requirements    |        |          |         |
+---------------------------------------+--------+----------+---------+
| ### Staffing and Training Needs       |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **                               |        |          |         |
| Test Schedules and Responsibilities** |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Open Issues**                  |        |          |         |
+---------------------------------------+--------+----------+---------+
| ### Test Sc                           |        |          |         |
| enarios, Test Cases, and Test Scripts |        |          |         |
+---------------------------------------+--------+----------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| #### **Test Cases**                   |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:**

**DETAIL COMPARISONS:** Evidenced

Yes No Elsewhere

+---------------------------------------+--------+----------+---------+
| #### **Test Case Identification**     |        |          |         |
+---------------------------------------+--------+----------+---------+
| **Focuses on Components identified in |        |          |         |
| the Decomposition Inventory**         |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Status of Test Case**          |        |          |         |
+---------------------------------------+--------+----------+---------+
| ### Busine                            |        |          |         |
| ss Function/Test Scenario Identified  |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Purpose of Test**              |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Instructions**                 |        |          |         |
+---------------------------------------+--------+----------+---------+
| Test Case instructions were not       |        |          |         |
| documented, but relied, instead, on   |        |          |         |
| the knowledge of the testers.         |        |          |         |
+---------------------------------------+--------+----------+---------+
| ### Expected Results                  |        |          |         |
+---------------------------------------+--------+----------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ###### Test Results                   |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:**

**DETAIL COMPARISONS**

> Evidenced
>
> Yes No Elsewhere

+---------------------------------------+--------+----------+---------+
| #### **Test Case Identification**     |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Test Date**                    |        |          |         |
+---------------------------------------+--------+----------+---------+
| Test dates were reflected in the      |        |          |         |
| subsystem test plan and Revisions to  |        |          |         |
| the overall test plan.                |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Actual Results \[              |        |          |         |
| see subsystem test summary sheets\]** |        |          |         |
+---------------------------------------+--------+----------+---------+
| #### **Test Summary \[ see            |        |          |         |
|  subsystem annotated test results\]** |        |          |         |
+---------------------------------------+--------+----------+---------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
| ###### Problem Tracking               |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:**

**DETAIL COMPARISONS:**

> Evidenced

Yes No Elsewhere

+---------------------------------------+-------+----------+----------+
| #### **Problem Identification**       |       |          |          |
+---------------------------------------+-------+----------+----------+
| ### Problem Description               |       |          |          |
+---------------------------------------+-------+----------+----------+
| #### **Submit Date**                  |       |          |          |
+---------------------------------------+-------+----------+----------+
| ### Problem Status                    |       |          |          |
+---------------------------------------+-------+----------+----------+
| #### **Resolve Date**                 |       |          |          |
+---------------------------------------+-------+----------+----------+
| #### **Resolution**                   |       |          |          |
+---------------------------------------+-------+----------+----------+

+---------------------------------------+---------+---------+---------+
|                                       | **Meets | **Man   | **Man   |
|                                       | B       | agement | agement |
|                                       | aseline | Aw      | Review  |
|                                       | Prac    | areness | Ad      |
|                                       | tices** | Ad      | vised** |
|                                       |         | vised** |         |
+---------------------------------------+---------+---------+---------+
|                                       |         |         |         |
+---------------------------------------+---------+---------+---------+
| ###### Management Issue Log           |         |         |         |
+---------------------------------------+---------+---------+---------+

**OVERALL OBSERVATION:** No management issue log was evidenced, although
management issues were tracked via regular discussions/meetings between
testers and system management.

**DETAIL COMPARISONS**

> Evidenced

Yes No Elsewhere

+---------------------------------------+--------+---------+----------+
| #### **Issue Identification**         |        |         |          |
+---------------------------------------+--------+---------+----------+
| ### Issue Description                 |        |         |          |
+---------------------------------------+--------+---------+----------+
| #### **Submit Date**                  |        |         |          |
+---------------------------------------+--------+---------+----------+
| ### Severity Level                    |        |         |          |
+---------------------------------------+--------+---------+----------+
| ### Escalated To                      |        |         |          |
+---------------------------------------+--------+---------+----------+
| #### **Issue Status**                 |        |         |          |
+---------------------------------------+--------+---------+----------+
| #### **Resolve Date**                 |        |         |          |
+---------------------------------------+--------+---------+----------+

# 

[^1]: There is no overall rating given for this critical success factor,
    due to the fact all baseline testing practices are not applicable
    for this subsystem.