![](media/image1.png){width="1.4986111111111111in"
height="1.5020833333333334in"}

![](media/image2.png){width="1.5in" height="1.4590277777777778in"}

**SYSTEM TEST PHASE II**

**PLAN**

**for the**

**Radiosonde Replacement System (RRS)**

**April 2005**

**U.S. DEPARTMENT OF COMMERCE**

**National Oceanic and Atmospheric Administration**

**National Weather Service/Office of Operational Systems**

**Field Systems Operations Center/Observing Systems Branch**

(This Page Left Blank)

**Table of Contents** Page

Acronyms iv

1 Introduction 1

1.1 ST Phases 1

1.2 Test Plan Organization 1

1.3 RRS Subsystems 2

1.4 Purpose of Test 2

1.5 System Test Phase II Strategy 3

1.6 Corollary Documents 4

2 Test Methodology 4

2.1 Prerequisites, Assumptions, and Risks 4

2.2 Test Plan 5

2.2.1 Test Location 5

2.2.2 Pre-Test Activities 6

2.2.3 Documentation 6

> 2.2.3.1 Required Documentation for Test 6
>
> 2.2.3.2 Forms 8

2.2.4 Evaluation Areas 8

> 2.2.4.1 Workstation Training 9
>
> 2.2.4.2 Supplemental Operator Familiarization 9
>
> 2.2.4.3 Upper Air Operations 9
>
> 2.2.4.4 Flight Characteristics 10
>
> 2.2.4.5 Security and Software Updates 11

2.3 Analyses 11

2.3.1 NCEP Analyses 11

2.3.2 NCDC Analyses 12

2.3.3 Flight Performance 12

2.3.4 RRS/MicroART Meteorological Comparisons 14

2.3.5 GPS-IPW Comparisons 15

2.3.6 Reliability, Maintainability, and Availability (RMA) Data Analyses
15

2.4 Communications Tests 15

2.4.1 Back-up Communications 16

2.4.2 Communications Headers 16

2.4.3 NWS Customer Involvement 16

2.5 Post-ST II Recommendations, and Report 17

2.5.1 Evaluation of Objectives 17

2.5.2 Test Result Analysis 18

2.5.3 Test Recommendations and Report 19

3 Test Schedules 20

3.1 Test Objectives by Week 20

3.2 OFS Test Objectives 20

3.2.1 Week 1 Test Objectives 20

3.2.2 Week 2 Objectives 21

3.2.3 Week 3 Objectives 22

3.2.4 Week 4 Test Objectives 22

3.3 Test Milestones 23

4 Test Roles and Responsibilities 26

4.1 Test Review Group (TRG) 26

4.2 Test Membership 26

**Tables**

Table 1-1. Major Activities 4

Table 2-1. RRS Test and Operations Documentation 7

Table 2-2. RRS Maintenance Documentation 7

Table 2-3. Forms 8

Table 2-4. Objectives and Evaluation Methods 17

Table 3-1. Activities Prior to Start of OFS 23

Table 3-2. Activities Prior to Start of OFS 23

Table 3-3. Activities Just Prior to Start of OFS 24

Table 3-4. Activities During OFS 24

Table 3-5. Activities After OFS 25

Table 4-1. Test Personnel 26

**Figures**

**Figure 1**. WFO-KCAR. 5

**Figure 2**. Example of RRS comparison with first-guess. 12

**Figure 3**. Example of IPW plot for GPS, MicroART, and RRS. 15

**Attachments**

Appendix A - Legacy System Performance Measures A-1

Appendix B - Operational Flight Series Test Schedule B-1

**Acronyms**

  --------------- -------------------------------------------------------
  A~s~            System Availability

  AWIPS           Advanced Weather Interactive Processing System

  BILS            Balloon Inflation and Launch Shelter

  BIT             Built-In-Test

  BWWFO           Baltimore, MD/Washington, DC Weather Forecast Office

  CCB             Configuration Control Board

  CLIMAT          NWS Monthly Site Upper Air Summary Product

  COTR            Contracting Officer=s Technical Representative

  COTS            Commercial Off-The-Shelf

  CDU             Control Display Unit

  ERH             NWS Eastern Region Headquarters

  EMRS            Engineering Management Reporting System

  ESA             Electronics System Administrator

  FTP             File Transfer Protocol

  GPS             Global Positioning System

  GUI             Graphical User Interface

  hPa             Hecto-Pascal

  km              Kilometer

  kts             Knots

  LDAD            Local Data Acquisition Device

  LRIP            Low-Rate Initial Production

  LRU             Line Replaceable Unit

  LWX             National Weather Service identifier for the BWWFO.

  MIC             Meteorologist-In-Charge

  NCDC            National Climatic Data Center

  NCEP            National Centers for Environmental Prediction

  NCF             (AWIPS) National Control Facility

  NHDA            [N]{.underline}WS [H]{.underline}eadquarters
                  [D]{.underline}evelopment [A]{.underline}WIPS

  NLSC            National Logistics Support Center

  NRC             National Reconditioning Center

  NMTW            [N]{.underline}WS Headquarters
                  [M]{.underline}odernization [T]{.underline}est and
                  Integration AWIPS, [W]{.underline}FO System

  NWS             National Weather Service

  NWSO            National Weather Service Observing

  NWSTC           NWS Training Center

  NWSTG           NWS Telecommunication Gateway

  OAT             Operational Acceptance Test

  OBIT            Offline Built In Test utility suite

  OMS             Offline Maintenance Suite

  OPS11           NWS Engineering and Acquisition Branch

  OPS12           NWS Maintenance Branch

  OPS13           NWS Configuration Branch

  OPS22           NWS Observing Systems Branch

  OPS23           NWS Software Branch

  OPS24           NWS Test and Evaluation Branch

  OS7             NWS Observing Services Division

  OST11           NWS Program Management Branch

  OST31           NWS Analysis Branch

  PAMS            Product Availability Monitoring System

  PCA             Physical Configuration Audit

  PDB             Precision Digital Barometer

  RH              Relative Humidity

  RMA             Reliability, Maintainability, and Availability

  RRS             Radiosonde Replacement System

  RSIM            RRS System Integration Manager

  RSOIS           Radiosonde Surface Observation Instrumentation System

  RWS             RRS Workstation Subsystem

  SIR             System Issue Report

  SIT             System Integration Test

  SRS             System Requirement Specification

  SPS             Signal Processing System

  SR&DC           Sterling Research and Development Center

  ST              System Test

  S/W             Software

  TRG             Test Review Group

  TRS             Telemetry Receiving System

  UPS             Uninterruptible Power Supply

  UTC             Universal Time Coordinated

  WFO             Weather Forecast Office

  WMO             World Meteorological Organization

  WSH             National Weather Service Headquarters

  WSOM            Weather Service Operations Manual

  XDP             External Data Pump
  --------------- -------------------------------------------------------

23764. Introduction

The National Weather Service (NWS) will transition from the current
radiosonde system into the new Radiosonde Replacement System (RRS) to
collect and process upper air data using modern technology. The RRS will
be nationally deployed after the government successfully completes a
series of tests, the last two of which are the System Test (ST) and the
Operational Acceptance Test (OAT).

This plan describes the government tests performed during ST Phase II.
The ST Phase II will verify the RRS:

a.  Meets NWS upper air operational usability for use at field sites.
    Field staff will conduct all functions associated with the
    operations phase.

b.  Meets field performance metrics and Concept of Operation performance
    standards.

c.  Can transmit upper air products through various telecommunications
    networks to NWS internal users and external customers.

The ST II will be performed at the National Weather Forecast Office
located in Caribou, Maine. Assistance with data analysis will be
provided as requested by the National Centers for Environmental
Prediction (NCEP) located in Camp Springs, Maryland; and the National
Climatic Data Center (NCDC) located in Asheville, North Carolina.

1.  ST Phases

The ST will be conducted in two phases. The following sections discuss
the requirements for each phase of testing. ST Phase I will verify the
functional software requirements from the RWS Software Requirements
Specification have been met to the satisfaction of NWS Management. ST
Phase II will simulate operational use of the RRS and will commence
after successful completion of Phase I and prior to OAT. Thus, ST Phase
II is the capstone test ensuring the RRS is ready for deployment to
field sites and can perform in a similar manner to the legacy system it
is replacing.

2.  Test Plan Organization

This plan is a subsidiary of the *System Test Plan for the RRS*, dated
July 2003. It is composed of four sections. Section 1 contains
introductory material on the RRS major components and presents the test
strategy, test objectives, and test prerequisites for this phase of
testing. Section 2 describes the methodology for conducting the ST Phase
II, the facilities employed, and pre- and post-test activities. Section
3 describes the critical milestones during the test and Section 4
contains information on test personnel and their responsibilities. The
following appendices are included:

> Appendix A - Legacy System Performance Measures
>
> Appendix B - Operational Flight Series Test Schedule

3.  RRS Subsystems

The RRS consists of the following major subsystems:

d.  Radiosonde Surface Observation Instrumentation System (RSOIS) and
    Precision Digital Barometer (PDB) B collects pre-release surface
    data. RSOIS will be tested with both the radio and the optical cable
    connected. There will also be a portion of the test where the RSOIS
    will not be interfaced to the RWS, thus, requiring manual entry of
    the surface data.

e.  Telemetry Receiving System (TRS) and associated Uninterruptible
    Power Supply (UPS) B tracks the Global Positioning System (GPS)
    radiosonde and receives radiosonde telemetry signal; the UPS allows
    radiosonde tracking and data collection to continue in the event of
    local power failure.

f.  Signal Processing System (SPS) B translates the telemetry signal
    into displayable upper air parameters, both received and processed.
    The SPS maintenance port is activated for use with the OBIT software
    for advanced diagnostics.

g.  RWS B allows RRS operators to receive and process data from the TRS,
    SPS, RSOIS, and PDB. The RWS also controls the TRS, displays
    received and processed data, and disseminates formatted products to
    the Advanced Weather Interactive Processing System (AWIPS). The
    primary software under test will be RWS Build 1.0.4.0. After the ST,
    the approved build delivered to the OAT sites will be designated as
    1.1.

h.  GPS radiosondes - allow upper air wind direction and speed to be
    determined from differential triangulation data transmitted by the
    24-satellite GPS constellation. (Note, the initial ST Phase II will
    only use Sippican GPS radiosondes.)

```{=html}
<!-- -->
```
4.  Purpose of Test

The purpose of the ST Phase II is to verify the RRS is fully functional
in an operational sense, reliable when compared to the legacy system,
products can be transmitted reliably to NWS internal users and external
customers. This phase of ST must meet the following objectives under
operational field conditions:

i.  Validate RRS operational usability during a quasi-operational
    environment.

j.  Exercise all RRS subsystem interfaces to other NWS systems \[e.g.,
    the Advanced Weather Interactive Processing System (AWIPS)/Local
    Data Acquisition Device (LDAD)\] and back-up telecommunications.

k.  Collect performance data for evaluating meteorological algorithms
    including the solar radiation correction and generation of coded
    messages.

l.  Collect data for use during the Reliability, Maintainability, and
    Availability (RMA) analysis using the Engineering Management
    Reporting System (EMRS).

m.  Validate simulated end-to-end functionality to NCEP and NCDC and
    other NWS customers for their product verification of decoders and
    data bases.

n.  Assess the operational usability of RRS-generated data and products
    by a limited set of operational forecast applications and through
    comparisons with MicroART using RAOB[^1], a commercial upper air
    analysis product.

o.  Evaluate the installation of security and software updates without
    impacting operations.

p.  Assess operator/maintenance training and supporting documentation.

q.  Limited assessment of the logistic process.

Refer to Section 2.2.4 for a discussion of the areas to be evaluated:

5.  System Test Phase II Strategy

The basic strategy for conducting ST II is to operate the RRS in a
Aquasi-operational mode and evaluate its performance in various areas as
shown in Table 1-1. In the first column, pre-test activities are listed,
which will be completed before the actual test begins.

This will be followed by actual use of RRS in operations as conducted at
a real upper air site using field staff to conduct the operations (refer
to the second column in Table 1-1). Core aspects such as flying
radiosondes coincident with MicroART flights at synoptic and asynoptic
times will be performed and the results tabulated exactly like a field
site. To not cause problems for the field site, a unique test header
(KCRT, station identifier 69992) has been established on all messages
transmitted over AWIPS. In this way, there shouldn=t be any confusion
with official products issued.

From this data, performance metrics will be generated to determine if
the RRS is functioning at sufficient levels to warrant fielding the
system. Comparisons will be made with MicroART, the Forecast Systems
Laboratory=s GPS-Integrated Precipitable Water (IPW) sensor, and
assessed by both NCEP for their numerical weather models, and NCDC for
its quality control procedures.

In addition to the above, a detailed end-to-end telecommunications test
will be conducted, coordinated with both NWS and non-NWS users. The
purpose of this portion of the test will be to validate product
throughput from KCAR via AWIPS/NOAAPORT to the NWS Telecommunications
Gateway (NWSTG) for distribution to users. There are a small number of
product differences between the legacy products and the RRS products
which need to be tested with customer decoders and data bases.

Two Key Decision Points (KDP-1 and KDP-2) will occur during the test to
determine the state of the testing. The KDPs are specified times during
the test where the Test Review Group (TRG) (see Section 4.1 of this plan
and the ST Plan for further details about the TRG) will review the test
results and make any necessary decisions about the state of the test. At
the completion of the test, preliminary results and a final report will
be developed to ascertain the state of the system and its suitability
for use in an operational environment.

6.  Corollary Documents

The reader of this plan is expected to be familiar with the *System Test
Plan for the RRS*, dated July 2003. In addition, this document has been
prepared for those with an understanding of upper air systems and the
Radiosonde Replacement Program. A review of the *Radiosonde Replacement
System (RRS) Deployment Plan*, June 2003, V9.4 is considered critical
for understanding the major steps in the deployment of this system.
Documents required in support of the ST Phase II are found in Section
2.2.3.

+-----------------------+--------------------------+-------------------+
| **Table 1-1. Major    |                          |                   |
| Activities**          |                          |                   |
+=======================+==========================+===================+
| **Pre-Test            | **Operational Testing**  | **Post-Test       |
| Activities**          |                          | Activities**      |
+-----------------------+--------------------------+-------------------+
| Ground system         | Conduct                  | Analysis of test  |
| certified             | synoptic/asynoptic       | results           |
|                       | flights                  |                   |
+-----------------------+--------------------------+-------------------+
| Radiosondes in stock  | Products sent over NCEP  | Final Test Review |
| at NLSC               | and NWS users            | Group meeting     |
|                       |                          | conducted         |
+-----------------------+--------------------------+-------------------+
| AWIPS configured      | NCDC receives archive    | Complete test     |
|                       | files                    | reports           |
+-----------------------+--------------------------+-------------------+
| Documentation         | Security updates         |                   |
| available             | completed                |                   |
+-----------------------+--------------------------+-------------------+
| Training completed    | New software version     |                   |
|                       | updated                  |                   |
+-----------------------+--------------------------+-------------------+
| Software installed &  | Data analyses performed  |                   |
| switched to           | by NCEP & NCDC           |                   |
| operations mode       |                          |                   |
+-----------------------+--------------------------+-------------------+
| Supplemental Observer | Flight performance       |                   |
| Familiarization       | statistics generated     |                   |
| completed             |                          |                   |
+-----------------------+--------------------------+-------------------+
|                       | Meteorological           |                   |
|                       | comparisons with legacy  |                   |
|                       | system conducted         |                   |
|                       |                          |                   |
|                       | Reliability,             |                   |
|                       | Maintainability,         |                   |
|                       | Availability data        |                   |
|                       | analysis performed       |                   |
+-----------------------+--------------------------+-------------------+
|                       | Training survey          |                   |
|                       | completed                |                   |
+-----------------------+--------------------------+-------------------+

1.  Test Methodology

The following sections describe the test methodology, which will be
employed during ST Phase II. It is intended to provide a logical
progression of the major activities and responsibilities for each aspect
of the test.

24928. Prerequisites, Assumptions, and Risks

The prerequisites, assumptions, and risks associated with this phase of
testing can be found in the *System Test Plan for the RRS*, dated July
2003.

1.  Test Plan

The sections below delineate the general plan for the conduct of the ST
Phase II test.

1.  Test Location

The ST Phase II will be conducted in Caribou, Maine (referred to as
KCAR, herein). This site makes for an ideal test facility, because the
RRS is configured exactly the way other field upper air sites will be
done and it affords a high degree of operational assessment without
interfering with the commissioned system, MicroART. Figure 1shows a
picture of the WFO environment.

![](media/image3.png){width="4.5472222222222225in"
height="3.6104166666666666in"}

Factors which were considered in selecting Caribou as the ST II test
site include:

-   Office has a separate inflation building housing a TRS under its
    radome and installed just like field sites will be during the
    deployment phase.

-   Office has historically been the NWS upper air winter weather test
    site.

-   Comparisons with the legacy system, MicroART, are possible.

-   As a predetermined winter test site, many of the training,
    installation and checkout aspects along with field operations have
    been evaluated in preparation for ST Phase II.

-   Site has been pre-configured and tested to be ready for RRS
    comminations tests.

    25580. Pre-Test Activities

The following pre-test activities have been or must be completed before
the test begins.

+-----------------------+-----------------------------------------------+
| > Ground Equipment    | \- TRS already installed at KCAR              |
+-----------------------+-----------------------------------------------+
|                       | \- SPS/Base GPS installed at KCAR             |
+-----------------------+-----------------------------------------------+
|                       | \- RSOIS with optical cable & base station    |
|                       | already installed at KCAR                     |
+-----------------------+-----------------------------------------------+
|                       | \- PDB already installed at KCAR              |
+-----------------------+-----------------------------------------------+
|                       | \- GPS Repeater already installed             |
+-----------------------+-----------------------------------------------+
|                       | \- RWS XP workstation configured and          |
|                       | connected to the office LAN                   |
+-----------------------+-----------------------------------------------+
|                       | \- Ghosting CD, RWS and OMS s/w should be     |
|                       | delivered for the test with RWS Version       |
|                       | Description Document (VDD) and OMS VDD        |
+-----------------------+-----------------------------------------------+
|                       | \- Tool kit                                   |
+-----------------------+-----------------------------------------------+
| > System              | \- Engineering and software staff complete    |
| > Certifications      | certifications at KCAR, like was done for ST  |
|                       | Phase 1A B checklists                         |
+-----------------------+-----------------------------------------------+
| > Radiosondes         | \- 3 boxes delivered from NLSC from LRIP      |
+-----------------------+-----------------------------------------------+
|                       | \- Power supply (part of tool kit), if        |
|                       | available                                     |
+-----------------------+-----------------------------------------------+
| > AWIPS               | \- LDAD port configured for RRS per EHB-13,   |
|                       | AWIPS RWS-LDAD System Administration Note 15  |
+-----------------------+-----------------------------------------------+
|                       | \- New comms line interfaced to LDAD port     |
+-----------------------+-----------------------------------------------+
|                       | \- ISP interfaced for security updates        |
+-----------------------+-----------------------------------------------+
|                       | \- AWIPS set up to ingest and display RRS     |
|                       | products with test headers                    |
+-----------------------+-----------------------------------------------+
| > Documentation       | \- Operator training materials shipped to     |
|                       | field staff                                   |
+-----------------------+-----------------------------------------------+
|                       | \- Maintenance materials available            |
+-----------------------+-----------------------------------------------+
|                       | \- System administration-related procedures   |
|                       | available                                     |
+-----------------------+-----------------------------------------------+
|                       | \- Surveys for training completed             |
+-----------------------+-----------------------------------------------+
| > Training            | \- 1 ET/RMS Trained via NWSTC                 |
+-----------------------+-----------------------------------------------+
|                       | \- 3-6 HMTs complete training via distance    |
|                       | learning and proficiency test                 |
+-----------------------+-----------------------------------------------+

1.  Documentation

Two sets of documents are required for ST Phase II: required
documentation for conducting the test and a series of forms as part of
the test paperwork. These can be found in the subsections below.

1.  Required Documentation for Test

RRS support documentation such as hardware/software installation and
operational instructions, will be used in the ST Phase II. Reference to
these documents will be made as required throughout the test. The list
of test and operational documentation and procedures includes, but is
not limited to, those items contained in Table 2-1. Table 2-2 contains
the maintenance-related and system administration documentation.

**Table 2-1. RRS Test and Operations Documentation**

  -----------------------------------------------------------------------
  System Test Phase II Procedures for the RRS (final)

  RWS Build 1.1 VDD and OMS VDD

  The RRS home page: http://www.ua.nws.noaa.gov/RRS.htm

  RRS Operator Training Guide (final draft)

  Weather Service Operations Manual (WSOM) 30-2109 (EHB-9) Update (final
  draft)

  WSOM 30-2101 (EHB-1) Update (final draft)

  WSOM 30-2104 (EHB-4) Update (final draft)

  National Weather Service Observing (NWSO) Handbook 10 Update (final
  draft)
  -----------------------------------------------------------------------

**Table 2-2. RRS Maintenance Documentation**

+-------------------+-------------+----------+-----------------------+
| **Description**   | **Manual    | *        | **Status**            |
|                   | Number**    | *Current |                       |
|                   |             | Draft    |                       |
|                   |             | Date**   |                       |
+===================+=============+==========+=======================+
| RSOIS O & M       | NWS EHB     | 13 April | Revision B submitted  |
| Manual, Rev B     | 9-201       | 2004     | on 3/22/04 for Final  |
|                   |             |          | Publication Process.  |
+-------------------+-------------+----------+-----------------------+
| TRS O & M Manual, | NWS EHB     | 06 May   | Submitted on 3/30/04  |
|                   | 9-753       | 2004     | for Final Publication |
| w/Attachments A   |             |          | Process               |
| thru E            |             |          |                       |
+-------------------+-------------+----------+-----------------------+
| SIPPICAN SPS &    | NWS EHB     | 18 March | Submitted on 3/30/04  |
| Radiosonde        | 9-801       | 2004     | for Final Publication |
|                   |             |          | Process.              |
| O & M Manual      |             |          |                       |
|                   |             |          |                       |
| &                 |             |          |                       |
|                   |             |          |                       |
| SPS Software      |             |          |                       |
| Operations        |             |          |                       |
+-------------------+-------------+----------+-----------------------+
| Radiosonde        | NWS EHB     | 05 May   | Document distributed  |
| Workstation (RWS) | 9-901       | 2004     | on 13 April 2004 for  |
| O & M Manual,     |             |          | technical review.     |
|                   |             |          |                       |
| w/Attachments A & |             |          |                       |
| B                 |             |          |                       |
+-------------------+-------------+----------+-----------------------+
| SIPPICAN SPS      | NWS EHB     | 18 March | Submitted on 3/30/04  |
| Users Manual for  | 9-902       | 2004     | for Final Publication |
| SPS Software used |             |          | Process.              |
| on the RWS        |             |          |                       |
+-------------------+-------------+----------+-----------------------+
| RRS Fault         | NWS EHB     | 05 May   | Fault Analysis is     |
| Analysis Manual   | 9-903       | 2004     | **available in        |
|                   |             |          | individual O&M        |
|                   |             |          | manuals**. (This      |
|                   |             |          | manual is under       |
|                   |             |          | development).         |
+-------------------+-------------+----------+-----------------------+
| RRS Systems       | NWS EHB     | 05 May   | Document distributed  |
| Administration    | 9-904       | 2004     | on 13 April 2004 for  |
| Manual            |             |          | technical review.     |
+-------------------+-------------+----------+-----------------------+

2.  Forms

During the test, a series of forms including surveys,
operations-related, EMRS, and checklists will be completed during and
after the test. The complete list can be found in Table 2-3.

**Table 2-3. Forms**

  ---------------------------------------------------------------------------
  **Form Type**     **Form Name**     **Purpose**           **Responsible
                                                            organization**
  ----------------- ----------------- --------------------- -----------------
  Operational Form  B-85              **Enter logistics     **OPS22/OS7**
                                      information**         

  Operational Form  B-29              **Flight              **OPS22/OS7**
                                      Performance**         

  Operational Form  B-47              **NCDC Report**       **OPS22/OS7**

  Operational Form  H-6               **Reject              **OPS22/OS7**
                                      radiosondes**         

  Survey            Proficiency Test  **Evaluate the RWS    **OS7/OPS24**
                                      training and          
                                      proficiency**         

  Maintenance Form  EMRS              **Entering            **OPS21**
                                      maintenance-related   
                                      information**         
  ---------------------------------------------------------------------------

2.  Evaluation Areas

The following major areas will be evaluated during the ST Phase II:

1.  There will be an assessment of the training process and materials.
    This will include an assessment of the proficiency exam and the
    activities associated with familiarizing the operations staff with
    the RRS. Note: field staff from non-KCAR locations will be involved
    in this test.

2.  Verifying that RRS can operate in a 24/7 mode with system/flight
    performance similar to or better than MicroART.

3.  Demonstrate, using available system administration documentation, to
    perform security updates, and the installation of a new software
    load without having an adverse impact on operations.

4.  Although there will not be any specific evaluation of the
    maintenance aspects of the system, there may be a need for NLSC to
    deliver a replacement part to the site and the site ET/RMS staff
    replace the failed component. If problems or issues exist with
    either the logistics support, maintenance documentation, or level of
    training during the test these will be relayed to the appropriate
    engineering staff.

The sections below discuss each of these in detail.

1.  Workstation Training

Since the workstation, application software, training materials, and the
proficiency test, will be shipped to the sites where the operators will
be trained in advance, this will be the first area to be evaluated. The
following is a list of activities that will be completed by the
operators participating in the test.

> \(1\) Reviewing and commenting on the new RRS Handbook WSOH#10 and RRS
> Operator Training Guide
>
> \(2\) Loading the RRS software (and sample flights on a CD for
> reworking) on a PC in your office for training purposes
>
> \(3\) Reviewing a RRS training DVD
>
> \(4\) Taking an open book examination on observations with RRS and
> pass the exam.

2.  Supplemental Operator Familiarization

One of the key deployment elements is familiarizing the office staff
with the various components of the RRS. The approach is for the
deployment team to provide hands on familiarization training to site
personnel using the RRS equipment after completion of the RRS INCO. This
process is called the Supplemental Operator Familiarization (SOF). For
this to take place, it is assumed that site personnel will have
completed the required training material and have passed the RRS
certification exam prior to system deployment. It is also assumed that
since the RRS workstation and software will be delivered to the sites
well in advance of deployment, the field staff will already be familiar
with the basic software functions. The concept of the SOF is to value
add the training material with actual hands on experience. The SOF will
consist of , but not limited to, operating the TRS, activating the
radiosonde and performing the baseline, acquiring GPS signals, and
launching the radiosondes. Since the deployment team will only have a
few days to conduct this activity, it is imperative to maximize the
limited time in terms of providing this familiarization to the office
staff in an efficient manner. For the purposes of this test, the SOF
will be provided to both the KCAR and non-KCAR staff to ensure a good
assessment of this activity. A survey will be provided to the operations
staff to identify strengths and weaknesses with this approach.

3.  Upper Air Operations

The next phase of testing will be a series of synoptic and non-synoptic
(Special) flights conducted over a contiguous 30-day period. This is
referred to as the Operational Flight Series (OFS). The OFS will consist
of approximately 85 upper air observations being taken for the purpose
of developing performance characteristics for RRS. One other important
feature of the OFS is that it will facilitate the RWS comparing current
flight data with flights conducted 12-hrs and 24-hrs ago. Unlike
previous tests where a system/radiosonde failure caused a System Issue
Report (SIR) to be written; in this test, this type of failure will
result in a second or third release.

Since this is a performance-based test, the number of second and third
releases for RRS will be compared with the average number of
similar-type failures encountered with MicroART. These are delineated in
Appendix A as the Legacy System Performance Measures (LSPM) Tables.

4.  Flight Characteristics

Flights will be conducted exactly as if it were the actual KCAR field
site, with one-person releases, messages transmitted via AWIPS
(including back-ups), and the archive data sent to NCDC after each
flight. Additional tests such as transmitting the archive data, say
after 2 flights, or multiple releases, will also be performed. NCEP will
receive all these products for assessing any issues with the data and
will also generate their standard performance measures. The Test
Director will coordinate with NCEP and NCDC on the particulars of each
test.

The ascension number assigned to the MicroART flight and the associated
first Aofficial@ synoptic RRS flight will be documented as part of the
record. Note, the ascension numbers do not have to match throughout the
test, just have a record of both. For the first three weeks of the test,
synoptic flights will be conducted without inducing any special
situations during the synoptic flights. In this way, actual performance
statistics can be acquired. However, different configurations such as
changing the RSOIS connection from optical cable to radio will be
conducted during this period.

During asynoptic times a host of special situations can be performed
without interfering with the synoptic flight performance. On several
occasions during [asynoptic]{.underline} flights, dummy failures will be
induced to verify that, for example, second and third release conditions
can be met without problems. When the first flight fails, then a second
release will be taken. If that one also fails, a third one will also be
taken. Two flights might have the first flight fail due to an induced
radiosonde failure, e.g., cut temperature sensor, and then a second
flight will be attempted as normal. Note, any induced failures will not
be counted against the RRS performance statistics (Appendix A) and
additional flights will have the 1st & 2^nd^ flights fail and the third
flight will be normal. If second and third flights are required during
[synoptic]{.underline} flight conditions, then these could replace the
Adummy@ situations.

Additional special flights will be conducted if the Test Director
believes a special condition is warranted, say, a night-time flight. All
special flights will be transmitted via AWIPS. BUFR files will be sent
to NCDC after every flight and operational forms as specified in Section
2.2.3.2 will be completed.

If any abnormalities (other than those induced for test purposes) or
indications of noncompliant operations are observed during the ST Phase
II, SIRs will be written to document the problem and called to the
immediate attention of the Test Director. The Test Director will present
the discrepancies to the TRG for classification and recommendation.

5.  Security and RWS Software Updates

At least once during the OFS, a download of one or more security updates
must be attempted, using EHB9-904. The field ESA or ITO staff will
perform this update using the System Administrator=s manual for
guidance. The update will be performed by downloading the update via an
Internet Service Provider connection directly into the RRS workstation.
After the RWS has been updated and checked out, flight operations will
continue with no adverse effects. A report on the effectiveness of this
activity to support field operations will be included in the final
report. At about half way through the test the RWS software will be
updated using the appropriate manuals.

26784. Analyses

The following sections delineate the types of analyses required during
the test.

27436. NCEP Analyses

NCEP personnel will import the RRS test products into their
meteorological models and perform quality control of the data. They will
also compare the RRS products with official products from WFO-KCAR. This
comparison will occur during the ST Phase II using the WFO-KCAR legacy
upper air system and RRS launched balloons. This will be accomplished by
launching two balloons within 15 minutes of one another. NCEP will add
the ST II system WMO identifier (69992) to their ATHKS@ web page
([http://www.ncep.noaa.gov/NCO/DMQAB/QAP/thanks/)]{.underline} and
display the status of the data along with the data from the WFO-KCAR.

In addition, NCEP will determine if upper air data from the 69992 site
is:

Being correctly ingested into models, analyses, and can determine
estimated height/temperature biases

Identifying the freezing levels correctly

Not rejecting significant amounts of data from QC algorithms

An example of one of these charts is shown in Figure 2.

1.  NCDC Analyses

The RWS archive flight data will be submitted to the NCDC over the
Internet by File Transfer Protocol (FTP) after every flight. OPS22 and
NCDC staff will evaluate the files for format and accuracy of content.
SIRs will be generated if problems are identified. This analysis will be
completed for products generated from live flights and products
generated after performing a rework.

NCDC will ingest these data into their archival system and generate
their normal quality-control reports. OPS22 and other staff will compare
the RRS results with the MicroART from the KCAR site to determine if any
performance issues exist.

2.  Flight Performance

The objective of this portion of the test is to verify RRS flight
characteristics for approximately 70 consecutive flights and its ability
to produce flight performance similar to the legacy system. These must
be in good agreement with the legacy system with none categorized as
poor by the end of the assessment.

A baseline set of performance measures have been derived from the legacy
systems and are referred to as the Legacy System Performance Measures
(LSPM). Each is supported from data collected and processed from current
parameters over the years. These will then serve as a comparison set and
form a part of the report. The specific criteria are listed in Appendix
A. Statistics for 2nd/3rd releases, unsuccessful releases, missed fights
and rejected radiosondes, etc., will also be computed comparing MicroART
with RRS. To collect this information, upper air forms (Section 2.2.3.2,
Table 2-3) will be filled out as is done at the field site using the
Management Information Retrieval System (MIRS). Statistics and graphics
will also be generated by OPS22 for the RRS to compare with MicroART.

**[Possible outcomes:]{.underline}** The following outcomes are possible
for each LSPM when compared to the RRS for the duration of the test:

*Excellent Performance* - the RRS performance measure far exceeds the
minimum criteria for acceptable field use when compared to the LSPM

*Good Performance* - the RRS performance measure exceeds or meets the
minimum criteria for acceptable field use when compared to the LSPM

*Marginal Performance* - the RRS performance measure just meets the
minimum criteria for acceptable field use when compared to the LSPM

*Poor Performance -* the RRS performance does not meet the minimum
criteria for acceptable field use when compared to the LSPM

By the end of the test period, all performance measures must be in the
Marginal Performance or better categories, i.e., for this aspect of the
test to be deemed successful, none can be in the poor category.

**[Test Conditions:]{.underline}** After each flight, data related to
each performance measure will be collected and documented. The TRG will
assess the RRS results against the LSPMs as shown in the Tables in
Appendix A. When problems occur, a determination will be made as to
whether they will impact the operation of the system or a work-around
can be established. If a performance measure status is moving from
*marginal* to *poor*, then various test conditions are possible:

**Condition 1. Continue testing and monitor.**

**Condition 2. Pause and fix problem, then continue.**

**Condition 3. Halt test and restart from beginning.**

A review of the test status will be conducted each day. Condition 3 will
only be recommended if any of the RRS performance measures (RRS PM) are
categorized as poor (see below) and the prognosis also appears to be
poor. For example, if the software/hardware were very unstable and not
able to process flight data in order to meet the LSPM, then it makes
sense to assign Condition 3. If there are bugs, glitches, etc. needing
corrective action, then Condition 2 would probably be valid. A condition
considered more of a nuisance would probably call for a Condition 1
state. The full operational assessment should not be conducted until
there is confidence Condition 3 situations will be of low probability,
as a result of ST Phase I testing.

3.  RRS/MicroART Meteorological Comparisons

Because of the impact RRS will have on field operations, it is necessary
to produce comparison charts of upper air meteorological profiles with
MicroART. Perfect agreement is not critical here, rather the
temperature, pressure/height, relative humidity, and wind plots must
provide a certain degree of agreement, since both are sensing the same
atmosphere. The solar radiation correction will be activated for the
RRS, and therefore, the temperature profiles are expected to be cooler
than the MicroART ones. Plots as produced by the RAOB program will be
collected for surface-to-700 hPa, surface-to-400 hPa, surface-to-100
hPa, and surface-to-flight termination. Pressure/height curves for both
systems will be acquired to compare profiles and calculated parameters
from the compared flights will also be evaluated to determine if any
problems exist. Areas, which will be specifically evaluated using the
RAOB program, are as follows:

Comparison of surface values between RRS and MicroART for the early part
of the sounding.

Ekman spiral in the first two kilometers of wind data.

Low-level and mid-to-high tropospheric jets along with max winds and
shears.

Cloud formation including areas of high and low RH, and integrated
precipitable water.

Identification of freezing level crossings.

Thickness values.

Selection of Tropopause(s).

Stratospheric temperature profiles with RRS solar radiation corrections
applied.

OPS22 with support from the KCAR forecast office and OS7 staff will
review the comparison plots. If they find any inherent problems, they
will inform the Test Director for resolution of the problem. To simplify
the meteorological analysis and to save on resources, some of the
asynoptic flights may be conducted as a dual with one RRS and one
MicroART radiosonde on the same balloon.

4.  GPS-IPW Comparisons

Comparisons of GPS-IPW values produced from FSL=s differential GPS
system will be used to determine if there are biases with the total
precipitable water values from the RRS observation as obtained from the
RAOB program.

5.  Reliability, Maintainability, and Availability (RMA) Data Analyses

RMA information will be collected during the ST Phase II using the EMRS
(see Attachment C of the RRS ST plan). The collected data will be used
to assess the potential reliability, maintainability, and availability
only during this phase. The EMRS data will be provided to the
appropriate staff for assessing the system reliability. This reliability
value will not be a prediction of future RRS reliability, but only an
initial estimation.

1.  Communications Tests

The WFO-KCAR AWIPS will be used to receive coded messages from the
RWS/RRS and to send upper air products to the NCEP and other NWS
customers via the NWSTG using the current communications path. In
general, products are issued from the local site through AWIPS to the
Network Control Facility for satellite broadcast over NOAAPORT and to
the NWSTG for world-wide distribution.

28032. Back-up Communications

At least 2 synoptic flights will transmit all of its products over the
1^st^ backup, which is the ERH LDAD located in Long Island, NY, instead
of the primary. At least two additional flights will use the 2^nd^
backup at SRH in Dallas/Fort Worth in a similar manner. The Product
Availability Monitoring System (PAMS) will be used to collect the
frequency/timeliness of upper air products transmitted from the WFO-KCAR
AWIPS to the NCF and then back to the WFO. NCEP may also provide
information on the receipt times to their NCEP computer, especially if
long delays exist. Performance measurement requirements for timeliness
and frequency can be found in Appendix A.

1.  Communications Headers

Products issued over the telecommunications networks will require
message headers to be appended to the products. This facilitates the
transmission of these products over AWIPS and through the back-up
communications line using the following headers:

  -----------------------------------------------------------------------
  **AWIPS PIL**  **Individual     **Individual    **Collective**
                 Test WMO         WMO Header**    
                 Header**                         
  -------------- ---------------- --------------- -----------------------
  CRTMANCRT      USUS97 KCRT      USUSii KCRT     USUS05 KWBC

  CRTSGLCRT      UMUS97 KCRT      UMUSii KCRT     UKUS50 KWBC and UGUS50
                                                  KWBC

  CRTABVCRT      UFUS97 KCRT      UFUSii KCRT     ULUS05 KWBC, UEUS05
                                                  KWBC, ULUS50 KWBC and
                                                  UEUS50 KWBC

  CRTFZLCRT      UXUS97 KCRT      UXUSii KCRT     UXUS70 KWBC

  CRTULGCRT      NXUS97 KCRT      NXUSii KCRT     (Note, ULG product not
                                                  required for this test)
  -----------------------------------------------------------------------

BUFR Collectives: IUST41 KWBC, IUST42 KWBC and IUST46 KWBC

The NWSTG will verify these products are being routed correctly to NWS
customers and also verify they are being sent to NCEP. Having the
*ii*-portion of the header denoted as *97* prevents the products from
being official, even though the contents will be actual data collected
from the RRS.

2.  NWS Customer Involvement

Although there are only a few changes with respect to the actual coded
messages, they can become a problem if these changes are not handled
correctly by decoders and data bases. NWS will be issuing a series of
Public Notification Statements (PNS) to NWS customers indicating the
availability of test messages and where to find information about the
coded message changes. The necessary product header information as
described in Section 2.4.2 will be included. NWS customers will be
invited to participate in the test through the PNS as either passive or
active participants. Passive participants can use the information
provided to check out their decoders and other processing functions and
do not need to inform the NWS of their results unless they need some
assistance. Active participants are those which the NWS would like to
hear from, to ensure at least some of the participants exercised their
processing systems during the test and verified no adverse effects. Some
potential NWS customers who could become active participants include:

FAA, AFWA, and FNMOC B government agencies

Unidata B universities

WSI B commercial

The actual active participants will be selected after coordination
within NWS is completed on potential candidates and a request is made to
them.

2.  Post-ST II Recommendations, and Report

The following sections describe how the data collected during the ST II
will be analyzed and how a recommendation to proceed is determined.

28628. Evaluation of Objectives

The ST Phase II Objectives identified in Section 1.4 will be evaluated
by the methods specified in Table 2-4.

**Table 2-4. Objectives and Evaluation Methods**

+----------------------------------+-----------------------------------+
| **Objective**                    | **Evaluation Method**             |
+==================================+===================================+
| Validate RRS operational         | Field staff will exercise RRS in  |
| usability and maintainability.   | Aquasi-operational mode and       |
|                                  | verify system is ready for field  |
|                                  | use. Maintenance staff can        |
|                                  | maintain the system in accordance |
|                                  | with established documentation.   |
|                                  | Problems needing corrective       |
|                                  | action before field deployment    |
|                                  | will be captured in SIRs.         |
+----------------------------------+-----------------------------------+
| Exercise all RRS subsystem       | Successful completion of flight   |
| interfaces to other NWS systems  | and transmission of RRS           |
| \[e.g., the AWIPS/Local Data     | observations, followed by         |
| Acquisition Device (LDAD)\].     | transmission of archive data.     |
+----------------------------------+-----------------------------------+
| Collect performance data for     | Analysis by OPS22 and OS7 subject |
| evaluating meteorological        | matter experts verifies RRS is    |
| algorithms including solar       | operating nominally.              |
| radiation correction and         |                                   |
| generation of coded messages.    |                                   |
+----------------------------------+-----------------------------------+
| Collect data for use during the  | OPS11/OPS12/OPS13 to analyze data |
| Reliability, Maintainability,    | collected using a modified        |
| and Availability (RMA) analysis  | Engineering Management Reporting  |
| using the Engineering Management | System A-26 form.                 |
| Reporting System.                |                                   |
+----------------------------------+-----------------------------------+
| Validate end-to-end              | **NWSTG** will evaluate the       |
| functionality at an operational  | products by generating their      |
| site. This will include RRS      | monthly CLIMAT report and         |
| product format and data          | checking throughput to NWS        |
| evaluation.                      | customers.                        |
|                                  |                                   |
|                                  | **NCEP** will import RRS products |
|                                  | into their meteorological models  |
|                                  | and perform quality control of    |
|                                  | the data.                         |
|                                  |                                   |
|                                  | **NCDC** will evaluate archive    |
|                                  | generation and data quality for   |
|                                  | their products.                   |
|                                  |                                   |
|                                  | **NWS Customers** can             |
|                                  | successfully decode and integrate |
|                                  | the products.                     |
+----------------------------------+-----------------------------------+
| Assess the operational usability | OPS24 and OPS22 will examine RRS  |
| of RRS-generated data and        | products distributed by the AWIPS |
| products by a limited set of     | NCF and their use in AWIPS UA     |
| operational forecast             | applications.                     |
| applications.                    |                                   |
+----------------------------------+-----------------------------------+
| Assess Maintenance Actions/Fault | Evaluated by the field            |
| Isolation using the OMS and the  | electronics staff with support    |
| Offline Built-In-Test (OBIT)     | from OPS11/OPS12 as faults arise. |
| applications.                    |                                   |
+----------------------------------+-----------------------------------+
| Verify the Radiosonde            | Field staff will collect data for |
| Replacement System Project       | analysis as follows:              |
| Agreement document minimum RRS   |                                   |
| performance requirements.        | 1\. Percentage of all naturally   |
|                                  | terminating test flights          |
|                                  | processing data above the 400     |
|                                  | hecto-Pascal (hPa) level is equal |
|                                  | to or greater than 95%.           |
|                                  |                                   |
|                                  | 2\. Percentage of all naturally   |
|                                  | terminating test flights          |
|                                  | processing data above the 10 hPa  |
|                                  | level is equal to or greater than |
|                                  | 60%.                              |
|                                  |                                   |
|                                  | 3\. Percentage of all test        |
|                                  | flights with 90% of wind and      |
|                                  | temperature values passing NCEP   |
|                                  | quality control tests will be 97% |
|                                  | or greater.                       |
|                                  |                                   |
|                                  | 4\. Percentage of all test        |
|                                  | flights transmitting messages     |
|                                  | within three hours of balloon     |
|                                  | release is 95% or greater.        |
+----------------------------------+-----------------------------------+
| Verify the Legacy System         | Data collected by the field staff |
| Performance Measures (LSPM) are  | and OPS                           |
| being met.                       |                                   |
|                                  | 22 and analyzed by OPS22/OS7.     |
+----------------------------------+-----------------------------------+

1.  Test Result Analysis

During the real time analysis of the test results, the criteria
presented below will be monitored and presented by the test team during
the weekly TRG. If at any time during the test this criteria is not met,
the test director will notify the TRG for further analysis of the
problem and to determine if the problem is serious enough to suspend
testing. The TRG may suspend the test, if at any time a problem
prohibits the following criteria from being met.

a.  Successful completion of the test procedure for each test case will
    be used as one of the pass/fail criteria. If any abnormalities are
    observed during the test procedures, SIRs will be written to
    document the problems and will be provided to the TRG for
    adjudication. The TRG will prioritize all SIRs and determine if they
    warrant being submitted to the RRS CCB. The TRG will
    [not]{.underline} submit a recommendation to the RRS Program Manager
    to proceed if there are any unresolved Impact 1 or 2 SIRs.

b.  The second result analysis will be an examination of RRS upper air
    observation products output for content correctness as well as
    initial operational performance characteristics. This analysis will
    be performed by OPS22 and OS7 subject matter experts coordinating
    data analysis with NCEP and NCDC.

c.  The Legacy System Performance Measures are deemed minimally
    acceptable. By the end of the test period, all performance measures
    must be in the Marginal Performance or better categories. (i.e., for
    this aspect of the test to be deemed [successful]{.underline}, none
    should be in the *poor* category.)

d.  The Radiosonde Replacement System Project Agreement document states
    the minimum RRS performance requirements in terms of operational
    use. The following minimum performance requirements are extrapolated
    from this document and tailored to the ST:

    1.  Percentage of all naturally terminating test flights processing
        data above the 400 hecto-Pascal (hPa) level is equal to or
        greater than 95%.

    2.  Percentage of all naturally terminating test flights processing
        data above the 10 hPa level is equal to or greater than 60%.

    3.  Percentage of all test flights with 90% of wind and temperature
        values passing NCEP quality control tests will be 97% or
        greater.

    4.  Percentage of all test flights transmitting messages within
        three hours of balloon release is 95% or greater.

The statistics for the above requirements will be calculated for each
system independently and as a group for all systems used during the
ST-II. The statistics will be adjusted to remove, where and when
necessary, inappropriate data such as defective balloons or an operator
induced failure to test a specific requirement.

1.  Test Recommendations and Report

Following completion of the ST Phase II, the Test Director will conduct
a AWrap-up@ meeting with the TRG describing what was tested, a summary
of any discrepancies found, major findings, and recommendations. The TRG
will review the materials presented by the Test Director and make a
recommendation to the RRS Program Manager whether to proceed with OAT.
After the AWrap-up@ meeting, the Test Director will prepare a formal
report of all test activities, including details of any deficiencies.
The report will also include findings and recommendations.

8.  Test Schedules

The following sections contain the schedules for various aspects of the
test.

1.  Test Objectives by Week

Specific test objectives will be met each week of the test. Even though
testing during ST Phase II will be a 24/7 operation to mimic actual
field operations, a *test week* will be defined as starting with the
Friday 00 UTC sounding, and concluding on Thursday of the following week
at 12 UTC. In this way, a number of test-related activities can be
concluded throughout the week leading up to the start of the new week.

2.  OFS Test Objectives

In order to meet the evaluation areas of Section 2.2.4 above, the types
of following test objectives will be adhered to during each week of the
test. However, the Test Director has the flexibility to modify this plan
as conditions warrant and terminate the test at any time if there is a
determination that continuance of the test would be counter-productive.
Documentation found in Section 2.2.3 will be used to support the
operational use, system administration, and maintenance of the system.

1.  Week 1 Test Objectives

The following test objectives will be conducted, and the results
evaluated, during this week of testing:

[Synoptic Flight Objectives]{.underline}

Perform 00 &12 UTC flights exactly like would occur at a field site:

-   1-person releases and different configurations, e.g., each flight
    will have a different frequency.

-   All 00 UTC flights use primary AWIPS & all 12 UTC flights will use
    1^st^ back-up dial to ERH AWIPS. A select set will also be
    transmitted to the second back-up at SRH.

-   If sonde/system failure requires 2nd/3rd release, then appropriate
    action will take place.

-   Some flights will have launches move towards the end of the synoptic
    window.

-   Comparisons with MicroART using RAOB software will be conducted and
    results transmitted to OPS22/OS7 for review.

Updated security profile from ISP download, per regional procedures.

Primary objective is to collect flight performance data to compare
against Legacy System Performance Measures tables (Appendix A) & ensure
number of rejected radiosondes is within specifications, based on H-6
Form.

-   Evaluate RRS product throughput using PAMS.

-   Determine if RRS is stable in terms of day-in/day-out
    operational/system availability.

-   Flight/System performance must be at least marginal, while goal is
    good-to-excellent range.

-   NCEP /NCDC reports/graphics are deemed positive and NCDC verifies
    BUFR messages correspond with RWS files produced.

-   All flights will reside in RWS to verify system performance.

-   EMRS updated as required.

Forms will be filled out just like regular site and transmitted to OPS22
for assessment, when requested.

Local applications will be checked out, if possible, e.g., AWIPS Skew-T,
Log P diagram.

RSOIS optical cable used to receive surface data.

[Asynoptic Flight Objectives]{.underline}

Test special cases not found during Anormal@ flights.

-   TRS off-target launches, i.e., 180 degrees off target at launch.

-   Special meteorological conditions cases. (e.g., launch near
    thunderstorm)

-   multiple releases situations. (i.e., 2nd/3rd release situations)

    1.  Week 2 Objectives

The following test objectives will be conducted, and the results
evaluated during this week of testing:

[Synoptic Flight Objectives]{.underline}

Same as week 1, continue compiling performance results.

-   Skip transmission of archive data to NCDC for one flight and then
    transmit both synoptic BUFR messages after next synoptic flight.
    (Note, NCDC must inform site that BUFR message was not received
    before next synoptic flight.)

-   RSOIS spread spectrum radio used instead of optical cable.

-   KDP-1 on results to-date.

-   All flights will reside in RWS to verify system performance.

[Asynoptic Flight Objectives]{.underline}

Test Special Observation cases similar to week 1.

1.  Week 3 Objectives

The following test objectives will be conducted, and the results
evaluated, during this week of testing:

[Synoptic Flight Objectives]{.underline}

Same as week 1, continue compiling performance results.

-   Skip entire flight to demonstrate ascension numbers are correct
    after no release case and coded messages transmitted Awithout data@
    from RWS to NCEP.

-   RSOIS not used, instead surface values entered via ASOS.

-   KDP-2 on results to-date.

-   All flights will reside in RWS to verify system performance.

[Asynoptic Flight Objectives]{.underline}

Same as week 1, except:

-   Replace RWS with a replacement computer, then verify no impact on
    operations.

    1.  Week 4 Test Objectives

The following test objectives will be conducted, and the results
evaluated, during this week of testing:

[Synoptic Flight Objectives]{.underline}

Same as week-1, continue compiling performance results.

[Asynoptic Flight Objectives]{.underline}

Any remaining situations not tested.

-   Install new software & use backup drive to restore data using admin.

-   Download security from Internet.

-   Retransmit messages from rework.

-   Test 2nd/3rd releases if none leave occurred to-date.

-   All flights will reside in RWS to verify system performance.

-   Test Power failure cases.

    1.  Test Milestones

The following tables delineate the major time steps leading to
commencement of the OFS. For the purposes of this test, T=0 is when the
entire RRS is connected within the office environment and ready for
certification. This means T=0 may require several days to accomplish,
and so it is not considered a single day, but rather a period of time
leading to the certification of the RRS.

Table 3-1 delineates those activities required 45 days or more
[prior]{.underline} to beginning the test (T \> - 45). Table 3-2 lists
those actions required for completion [within]{.underline} 45 days of
the start of the test and Table 3-3 defines those activities that must
be completed successfully [within]{.underline} a week around T=0. At
this point, Table 3-4 activities support the evaluations as delineated
under Section 2.2.4 and subsections. Test reports and final coordination
activities are listed in Table 3-5.

**Table 3-1. Activities Prior to Start of OFS**

  -----------------------------------------------------------------------
  **Timeframe**           **Areas Tested**        **Responsible Org.**
  ----------------------- ----------------------- -----------------------
  T \> - 45 days          Radiosondes shipped     KCAR via CLS
                          from NLSC to KCAR (3    
                          boxes)                  

                          Operations              OS7
                          documentation/          
                          software/1-radiosonde   
                          shipped to KCAR &       
                          volunteers              

                          At least 1-ET/RMS       ERH
                          trained @ NWSTC         

                          Master station file for OPS22/Region
                          KCAR established        
  -----------------------------------------------------------------------

**Table 3-2. Activities Prior to Start of OFS**

+-----------------------+-----------------------+-----------------------+
| **Timeframe**         | **Areas Tested**      | **Responsible Org.**  |
+=======================+=======================+=======================+
| 0 \< T \< (-45days)   | Field/Test operations | Test staff            |
|                       | staff train on RWS    |                       |
+-----------------------+-----------------------+-----------------------+
|                       | GPS repeater          | RMS @ KCAR            |
|                       | calibrated @ KCAR     |                       |
+-----------------------+-----------------------+-----------------------+
|                       | System certifications | OPS11/OPS12/OPS23     |
|                       | completed             |                       |
|                       | (checklists)          |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Remaining             | OPS12                 |
|                       | documentation         |                       |
|                       |                       |                       |
|                       | available to KCAR via |                       |
|                       | web                   |                       |
+-----------------------+-----------------------+-----------------------+
|                       | FTP to NCDC link      | OPS22/KCAR staff      |
|                       | tested                |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Field Test staff      | KCAR staff & field    |
|                       | complete proficiency  | Volunteers/ OS7       |
|                       | testing and survey    |                       |
+-----------------------+-----------------------+-----------------------+

**Table 3-3. Activities Just Prior to Start of OFS**

  -----------------------------------------------------------------------
  **Timeframe**           **Areas Tested**        **Responsible Org.**
  ----------------------- ----------------------- -----------------------
  T = 0                   TRS connected to RWS &  OPS22/OS&T DM
                          RSOIS optical cable,    
                          then radio, then        
                          neither, i.e., use ASOS 

                          2-5 test flights        OPS22/KCAR staff
                          conducted               

                          AWIPS/Internet/Backup   KCAR/ESA/ITO
                          comms lines connected   
                          and security patches    
                          downloaded              

                          AWIPS/LDAD setup for    KCAR staff/OPS22/OPS24
                          ingest and display RRS  
                          test products on D2D.   

                          Test mode switch to     OPS22/ESA/ITO
                          Operations mode via     
                          re-install of s/w,      
                          i.e., data base         
                          wiped-out & ASC No. set 
                          to correct one.         

                          System certifications   OPS11/OPS12/OPS23
                          completed (checklists)  

                          Test obs sent to NCEP   KCAR staff/OPS22
                          and BUFR files to NCDC  
                          via FTP                 

                          Begin Familiarization   OPS22/KCAR staff
                          on RRS conducted with   
                          staff                   
  -----------------------------------------------------------------------

**Table 3-4. Activities During OFS**

+-----------------------+-----------------------+-----------------------+
| **Timeframe**         | **Areas Tested**      | **Responsible Org.**  |
+=======================+=======================+=======================+
| 0 \< T \< 30 days     | Begin operational     | KCAR staff/OPS22      |
|                       | flights:              |                       |
|                       |                       |                       |
|                       | (2 synoptic/day and   |                       |
|                       | 1-2 Asynoptic/day     |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Obs sent to NCEP &    | KCAR staff            |
|                       | NCDC                  |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Complete              |                       |
|                       | familiarization on    |                       |
|                       | RRS conducted with    |                       |
|                       | staff                 |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Collect performance   | OPS22                 |
|                       | statistics            |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Test special          | KCAR STAFF/OPS22      |
|                       | situations            |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Key Decisions PT-1    | PM/TRG                |
|                       | for                   |                       |
|                       |                       |                       |
|                       | @ T = 15 days KDP-2   |                       |
|                       | \@T = 30 days         |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Test local AWIPS      | OPS22/KCAR staff      |
|                       | applications          |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Checks AWIPS          | KCAR staff            |
|                       | Applications          |                       |
+-----------------------+-----------------------+-----------------------+
|                       | Security patch Update | KCAR ESA/ITO          |
+-----------------------+-----------------------+-----------------------+
|                       | Install new s/w load  | KCAR ESA/ITO          |
+-----------------------+-----------------------+-----------------------+

**Table 3-5. Activities After OFS**

  -----------------------------------------------------------------------
  **Timeframe**           **Areas Tested**        **Responsible Org.**
  ----------------------- ----------------------- -----------------------
  45 \< T \< 60 days      Preliminary Report      OPS22

                          NCEP Report sent to     NCEP
                          NWSH                    

                          NCDC Report sent to     NCDC
                          NWSH                    

                          Final Report Completed  OPS22
                          & issued                
  -----------------------------------------------------------------------

8.  Test Roles and Responsibilities

The following sections delineate the test roles and responsibilities.

1.  Test Review Group (TRG)

The TRG is an assembly of subject-matter experts and is chaired by the
Chief, Test and Evaluation Branch (OPS24). The role of the TRG is to
evaluate each observed deficiency (both repeatable and intermittent
problems) as documented by a SIR, on its impact on daily field service
operations and to make recommendations to the RRS Program Manager on SIR
criticality. Membership in the TRG can be found in the *System Test Plan
for the RRS*, dated July 2003.

2.  Test Membership

There will be two test teams on site. Each team will consist of one or
two volunteer field personnel and one test monitor from the OPS 22 staff
in Sterling. Volunteer field personnel can be from any site, but in the
selection, priority will be given to personnel from the OAT sites.
Volunteers will be on site for approximately two weeks . At the end of
the first two week period, field personnel on site will depart and new
field personnel will arrive. Personnel stationed in Caribou will be
utilized as appropriate if conditions permit.

One team works a day shift starting with the 12Z sounding, the other
team will work evening/night covering the 00Z sounding. Each shift in
most cases will also conduct one special or asynoptic sounding. OPS22
will provide the specific scenario for each flight conducted by field
personnel.

The test director will coordinate all activities with the OPS22 test
monitor.

**Table 4-1. Test Personnel**

+----------------+-------------+-------------+-----------------------+
| **Name/        | *           | **Phone**   | **E-mail**            |
| Organization** | *Function** |             |                       |
+================+=============+=============+=======================+
| James          | RRS ST II   | 7           | jame                  |
| Fitzgibbon     | Test        | 03-661-1243 | s.fitzgibbon@noaa.gov |
| (OPS22)        | Director    |             |                       |
+----------------+-------------+-------------+-----------------------+
| Jae Lee/       | RRS Test    | 3           | jae.lee@noaa.gov      |
| (OPS24)        | Advisor     | 01-713-0326 |                       |
|                |             | x158        |                       |
+----------------+-------------+-------------+-----------------------+
| Bill Blackmore | OPS22       | 3           | willi                 |
| (OPS22)        | Analytical  | 01-713-2091 | am.blackmore@noaa.gov |
|                | Support     | x107        |                       |
+----------------+-------------+-------------+-----------------------+
| Fred Branski   | NWSTG Test  | 3           | fred.branski@noaa.gov |
| (CIO11)        | Support     | 01-713-0864 |                       |
|                |             | x146        |                       |
+----------------+-------------+-------------+-----------------------+
| William Ryman  | NWSTC Test  | 8           | w                     |
|                | Support     | 16-880‑9368 | illiam.ryman@noaa.gov |
| (OS612)        |             | x242        |                       |
+----------------+-------------+-------------+-----------------------+
| Carl Bower     | Analytical  | 3           | carl.bower@noaa.gov   |
| (OS7)          | Support     | 01-713-0722 |                       |
|                | (Data       | x145        |                       |
|                | Processing  |             |                       |
|                | and         |             |                       |
|                | Algorithm   |             |                       |
|                | Analysis)   |             |                       |
+----------------+-------------+-------------+-----------------------+
| Bob Thomas     | Analytical  | 3           | r                     |
| (OS7)          | Support     | 01-713-0722 | obert.thomas@noaa.gov |
|                |             | x127        |                       |
+----------------+-------------+-------------+-----------------------+
| Ken Bashford   | Survey      | 3           | kenn                  |
|                | Support     | 01-713-0326 | eth.bashford@noaa.gov |
|                |             |             |                       |
|                |             | x113        |                       |
+----------------+-------------+-------------+-----------------------+
| Field          | RRS Test    | N/A         | N/A                   |
| Personnel      | Support @   |             |                       |
| (TBD)          | KCAR        |             |                       |
+----------------+-------------+-------------+-----------------------+
| Jeff Ator      | NCEP        | 3           | jeff.ator@noaa.gov    |
|                | Analytical  | 01-763-8000 |                       |
| (NP12)         | Support     | x7104       |                       |
+----------------+-------------+-------------+-----------------------+
| Richard        | NWSTG       | 3           | rich                  |
| Robinson (CIO) | Support     | 01-713-0864 | ard.robinson@noaa.gov |
|                |             |             |                       |
|                |             | x179        |                       |
+----------------+-------------+-------------+-----------------------+
| Dr. Bradley    | NCEP        | 3           | bra                   |
| Ballish (NCEP) | Analytical  | 01-763-8000 | dley.ballish@noaa.gov |
|                | Support     |             |                       |
|                |             | x7159       |                       |
+----------------+-------------+-------------+-----------------------+
| Wayne Martin   | AWIPS       | 3           | wayne.martin@noaa.gov |
| (OPS21)        | Support     | 01-713-1724 |                       |
|                |             |             |                       |
|                |             | x166        |                       |
+----------------+-------------+-------------+-----------------------+
| Eddie Roberts  | RWS Support | 3           | ed                    |
| (OPS23)        |             | 01-713-0191 | ward.roberts@noaa.gov |
|                |             |             |                       |
|                |             | x154        |                       |
+----------------+-------------+-------------+-----------------------+
| Ivan Navarro   | Radiosonde  | 3           | ivan.navarro@noaa.gov |
| (OPS11)        | Support     | 01-713-0844 |                       |
|                |             |             |                       |
|                |             | x134        |                       |
+----------------+-------------+-------------+-----------------------+
| Darryl         | TRS Support | 3           | dar                   |
| Modracek       |             | 01-713-1842 | ryl.modracek@noaa.gov |
| (OPS11)        |             |             |                       |
|                |             | x111        |                       |
+----------------+-------------+-------------+-----------------------+
| Kenneth Clark  | Maintenance | 3           | k                     |
| (OPS12)        | Support     | 01-713-1833 | enneth.clark@noaa.gov |
|                |             |             |                       |
|                |             | x190        |                       |
+----------------+-------------+-------------+-----------------------+
| John Monte     | RRS Support | 3           | john.monte@noaa.gov   |
|                |             | 01-713-1845 |                       |
| (OPS12)        |             |             |                       |
|                |             | x117        |                       |
+----------------+-------------+-------------+-----------------------+
| Eric Parr      | Logistics   | 3           | eric.parr@noaa.gov    |
|                | Support     | 01-713-1830 |                       |
| (OPS14)        |             |             |                       |
|                |             | x111        |                       |
+----------------+-------------+-------------+-----------------------+
| Stuart         | NCDC        | 8           | [Stuart.Hinson@       |
| Hinson/Larry   | Support     | 28-271-4437 | noaa.gov]{.underline} |
| Griffin        |             |             | /                     |
|                |             |             |                       |
| (CC11)         |             |             | [larry.j.griffin@     |
|                |             |             | noaa.gov]{.underline} |
+----------------+-------------+-------------+-----------------------+

The following describes the major roles and responsibilities of the test
personnel.

**Test Director** - Ensures all tests defined for the ST Phase II are
completed and the results properly documented in the ST II report.
Responsible for collecting and presenting all test trouble reports to
the TRG for classification and convening emergency TRG meeting, if
necessary. Following completion of the ST Phase II, the Test Director
will conduct a Awrap-up@ meeting of the TRG, provide details to the RRS
Program Manager what was tested, report the ST II conclusions, and
recommend whether to proceed with the next phase. This person ensures
all test trouble reports documented and classified during the ST II are
forwarded to the proper WSH organization or board for adjudication;
writes/issues the ST II preliminary and final reports to document the
test results and recommendations. The Test Director also serves as a
member of the test team.

**Test Team** - Responsible for performing individual test procedures as
assigned; documents the results of each test in test logs;
electronically completes trouble report forms when
problems/discrepancies are observed; provides the Test Director with all
completed forms (if not entered electronically). Informs the Test
Director with comprehensive technical information on how the tests were
conducted and any problems encountered.

**Test Support** - Responsible for providing technical support and
information as required when RRS questions arise; forwards RRS hardware
and software discrepancies to the respective contractor for resolution.

**NWSTG Test Support** - Responsible for evaluating RRS product
throughput and the ability of the NWSTG to generate monthly upper air
CLIMAT products using RRS products.

**NWSTC Test Support** - Responsible for evaluating RRS maintainability;
conducting RRS maintenance training; performing hardware evaluations as
time permits.

**NCEP Test Support** - Responsible for evaluating RRS products for
format and usability in meteorological models and performing quality
control checks.

**NCDC Test Support** - Responsible for determining if archived RRS
atmospheric data from radiosonde flights is in the proper format for use
in producing NCDC climatic products.

**Field Personnel Test Support** - Responsible for evaluating system
functionality and usability; participates in documentation reviews;
prepares radiosonde during pre-flight and assists in both synoptic and
special balloon launches. Responsible for assisting with product quality
control and collecting performance data.

**Analytical Support** - NCEP and NCDC personnel are responsible for
evaluating synoptic products for accuracy, usability, timeliness, and
the ability to be archived and retrieved. OPS22 and OS7 personnel are
responsible for evaluating all RRS algorithms and products for accuracy
and maintaining ST flight logs.

**Appendix A - Legacy System Performance Measures**

**General System**

+----------+----------+----------+----------+-------------+--------+
| **Per    | **E      | **Good   | **       | **Poor      | **RRS  |
| formance | xcellent | Perfo    | Marginal | Pe          | P      |
| M        | Perfo    | rmance** | Perfo    | rformance** | erform |
| easure** | rmance** |          | rmance** |             | ance** |
+==========+==========+==========+==========+=============+========+
| **Hardw  | **No     | **No     | **No     | **More than |        |
| are:\*** | missed   | more     | more     | 3 missed    |        |
|          | fligh    | than 1   | than 3   | flights**   |        |
|          | ts\*\*** | missed   | missed   |             |        |
|          |          | flight** | f        |             |        |
|          |          |          | lights** |             |        |
+----------+----------+----------+----------+-------------+--------+
| **System | **No     | **No     | **No     | **More than |        |
| Failures | missed   | more     | more     | 3 missed    |        |
| \*\*\*** | f        | than 1   | than 3   | flights**   |        |
|          | lights** | missed   | missed   |             |        |
|          |          | flight** | f        |             |        |
|          |          |          | lights** |             |        |
+----------+----------+----------+----------+-------------+--------+
| **St     | **System | **System | **System | **System    |        |
| art-up** | is       | is       | is       | isn=t       |        |
|          | fu       | fu       | fu       | functional  |        |
|          | nctional | nctional | nctional | within 20   |        |
|          | within   | within   | within   | minutes of  |        |
|          | 10       | 15       | 20       | a           |        |
|          | minutes  | minutes  | minutes  | ctivation** |        |
|          | of       | of       | of       |             |        |
|          | acti     | acti     | acti     |             |        |
|          | vation** | vation** | vation** |             |        |
+----------+----------+----------+----------+-------------+--------+
| **In-    | **No     | **No     | **No     | **More than |        |
| flight** | loss of  | more     | more     | 10 min loss |        |
|          | PTU or   | than 5   | than 10  | of          |        |
|          | winds    | min loss | min loss | PTU/winds   |        |
|          | due to   | of       | of       | due to      |        |
|          | system   | P        | P        | system      |        |
|          | malfun   | TU/winds | TU/winds | mal         |        |
|          | ctions** | due to   | due to   | functions** |        |
|          |          | system   | system   |             |        |
|          |          | malfun   | malfun   |             |        |
|          |          | ctions** | ctions** |             |        |
+----------+----------+----------+----------+-------------+--------+
| **R      | **No     | **No     | **No     | **More than |        |
| anging** | loss of  | more     | more     | 3 flights   |        |
|          | winds**  | than 1   | than 3   | have loss   |        |
|          |          | flight   | flight   | of winds    |        |
|          | **at     | having   | have     | less than   |        |
|          | less     | loss of  | loss of  | 250km (see  |        |
|          | than 250 | winds    | winds    | i           |        |
|          | km from  | less     | less     | n-flight)** |        |
|          | launch   | than     | than     |             |        |
|          | point**  | 250km    | 250km    |             |        |
|          |          | (see     | (see     |             |        |
|          |          | in-f     | in-f     |             |        |
|          |          | light)** | light)** |             |        |
+----------+----------+----------+----------+-------------+--------+
| **Shut   | **No     | **No     | **No     | **More than |        |
| Down**   | system   | loss of  | more     | 1 flight    |        |
|          | malfu    | PTU or   | than 1   | having loss |        |
|          | nction** | winds    | flight   | of PTU or   |        |
|          |          | and no   | having   | winds and   |        |
|          |          | more     | loss of  | more than 3 |        |
|          |          | than 1   | PTU or   | system      |        |
|          |          | system   | winds    | mal         |        |
|          |          | malfu    | and no   | functions** |        |
|          |          | nction** | more     |             |        |
|          |          |          | than 3   |             |        |
|          |          |          | system   |             |        |
|          |          |          | malfun   |             |        |
|          |          |          | ctions** |             |        |
+----------+----------+----------+----------+-------------+--------+

**\*Hardware includes: TRS, SPS, RWS, Multiplexer, DGPS Antenna,
interfaces, Cables, RSOIS**

**\*\* A Missed flight is defined as a failure causing the sounding not
to be taken during its observational window.**

**\*\*\* System failures include hardware and software supporting the
hardware, but not the radiosonde.**

**COMMUNICATIONS**

+-----------+----------+----------+-----------+-------------+--------+
| **Pe      | **Exc    | **Good** | **M       | **Poor**    | **RRS  |
| rformance | ellent** |          | arginal** |             | P      |
| Measure** |          | **Perfo  |           | **Pe        | erform |
|           | **Perfo  | rmance** | **Perf    | rformance** | ance** |
|           | rmance** |          | ormance** |             |        |
+-----------+----------+----------+-----------+-------------+--------+
| **From    |          |          |           |             |        |
| RRS to    |          |          |           |             |        |
| NCF/      |          |          |           |             |        |
| NOA       |          |          |           |             |        |
| APORT\*** |          |          |           |             |        |
+-----------+----------+----------+-----------+-------------+--------+
| **Fr      | **99.    | **98.9 - | **98.     | **Less than |        |
| equency** | 0-100%** | 98.4%**  | 3-98.0%** | 98%**       |        |
+-----------+----------+----------+-----------+-------------+--------+
| **Tim     | **0.5    | **0.5 -  | **1.0 -   | **greater   |        |
| eliness** | min or   | 1.0      | 1.25      | than 1.25   |        |
|           | less for | mins**   | min.**    | min**       |        |
|           | all      |          |           |             |        |
|           | U/A**    |          |           |             |        |
|           |          |          |           |             |        |
|           | **pr     |          |           |             |        |
|           | oducts** |          |           |             |        |
+-----------+----------+----------+-----------+-------------+--------+
| **From    |          |          |           |             |        |
| RRS to    |          |          |           |             |        |
| WFO\***   |          |          |           |             |        |
+-----------+----------+----------+-----------+-------------+--------+
| **Fr      | **98.5   | **98.4 - | **98.0 -  | **Less than |        |
| equency** | %-100%** | 98.1%**  | 97.0%**   | 97%**       |        |
+-----------+----------+----------+-----------+-------------+--------+
| **Tim     | **3 min  | **3.1 to | **5.1 to  | **Greater   |        |
| eliness** | for all  | 5 min.** | 7 min.**  | than 7      |        |
|           | U/A      |          |           | min.**      |        |
|           | pr       |          |           |             |        |
|           | oducts** |          |           |             |        |
+-----------+----------+----------+-----------+-------------+--------+
| **From    |          |          |           |             |        |
| RRS to    |          |          |           |             |        |
| NCEP**    |          |          |           |             |        |
+-----------+----------+----------+-----------+-------------+--------+
| **Fr      | **98.    | **97.9 - | **97.     | **Less than |        |
| equency** | 0-100%** | 97.4%**  | 3-97.0%** | 97%**       |        |
+-----------+----------+----------+-----------+-------------+--------+
| **Tim     | **5 min  | **5.1 to | **7.1 to  | **greater   |        |
| eliness** | for all  | 7 min.** | 15 min.** | than 15     |        |
|           | U/A      |          |           | min.**      |        |
|           | pr       |          |           |             |        |
|           | oducts** |          |           |             |        |
+-----------+----------+----------+-----------+-------------+--------+
| **From    | **No     | **1      | **2-3     | **1 or more |        |
| site to   | BUFR     | missed   | tra       | missed      |        |
| NCDC      | files    | tran     | nsmission | tr          |        |
| (FTP)**   | missing  | smission | missing & | ansmissions |        |
|           | from     | & NCDC   | NCDC      | at NCDC     |        |
|           | NCDC=s   | notifies | notifies  | after NCDC  |        |
|           | HDSS**   | NWS to   | to        | notifies    |        |
|           |          | ret      | re        | NWS to      |        |
|           |          | ransmit, | transmit, | r           |        |
|           |          | which is | which are | etransmit** |        |
|           |          | succ     | succ      |             |        |
|           |          | essful** | essful.** |             |        |
+-----------+----------+----------+-----------+-------------+--------+
| **From    | **N/A**  | **100%** | **N/A**   | **0%**      |        |
| RRS to    |          |          |           |             |        |
| TG**      |          |          |           |             |        |
|           |          |          |           |             |        |
| **(CLIMA  |          |          |           |             |        |
| T-TEMP)** |          |          |           |             |        |
+-----------+----------+----------+-----------+-------------+--------+

**\*Includes transmissions over back up communications paths as well.**

**Software Performance**

+----------+----------+----------+----------+----------+-----------+
| **Per    | **Exc    | **Good** | **Ma     | **Poor** | **RRS     |
| formance | ellent** |          | rginal** |          | Perf      |
| M        |          | **Perfo  |          | **Perfo  | ormance** |
| easure** | **Perfo  | rmance** | **Perfo  | rmance** |           |
|          | rmance** |          | rmance** |          |           |
+----------+----------+----------+----------+----------+-----------+
| **       | **None** | **No     | **No     | **Nore   |           |
| Software |          | more     | more     | than 3   |           |
| Halts**  |          | than 1   | than 3   | missed   |           |
|          |          | missed   | missed   | f        |           |
|          |          | flight** | flight** | lights** |           |
+----------+----------+----------+----------+----------+-----------+
| **I      | **All    | **No     | **No     | **One or |           |
| n-flight | function | more     | more     | more     |           |
| s/w      | nom      | than 1   | than 3   | f        |           |
| fun      | inally** | s/w      | s/w      | unctions |           |
| ctions** |          | function | f        | do not   |           |
|          |          | does not | unctions | operate  |           |
|          |          | operate  | do not   | c        |           |
|          |          | co       | operate  | orrectly |           |
|          |          | rrectly, | co       | and      |           |
|          |          | but does | rrectly, | impair   |           |
|          |          | not      | but none | the      |           |
|          |          | impair   | impair   | co       |           |
|          |          | f        | f        | llection |           |
|          |          | light.** | light.** | or       |           |
|          |          |          |          | pr       |           |
|          |          |          |          | ocessing |           |
|          |          |          |          | of       |           |
|          |          |          |          | flight   |           |
|          |          |          |          | data or  |           |
|          |          |          |          | coded    |           |
|          |          |          |          | me       |           |
|          |          |          |          | ssages** |           |
+----------+----------+----------+----------+----------+-----------+
| **Per    | **No     | **Minor  | **Some   | **Major  |           |
| formance | System   | System** | s        | System** |           |
| in       | Slow     |          | lowdown, |          |           |
| response | -downs** | **Slo    | but but  | **S      |           |
| to       |          | wdowns** | none     | lowdowns |           |
| operator |          |          | impair   | a        |           |
| co       |          |          | flight   | ffecting |           |
| mmands** |          |          | pr       | overall  |           |
|          |          |          | ocessing | perf     |           |
|          |          |          | of coded | ormance, |           |
|          |          |          | me       | e.g., 15 |           |
|          |          |          | ssages** | minutes  |           |
|          |          |          |          | to       |           |
|          |          |          |          | display  |           |
|          |          |          |          | coded    |           |
|          |          |          |          | me       |           |
|          |          |          |          | ssages** |           |
+----------+----------+----------+----------+----------+-----------+
| **No     | **All    | **No     | **No     | **OBIT   |           |
| n-flight | function | more     | more     | is not   |           |
| s/w      | nom      | than 1   | than 3   | fun      |           |
| fu       | inally** | s/w      | s/w      | ctioning |           |
| nctions, |          | function | function | or       |           |
| e.g.,    |          | does not | do not   | another  |           |
| off-line |          | operate  | operate  | utility  |           |
| uti      |          | co       | co       | is not   |           |
| lities** |          | rrectly, | rrectly, | fun      |           |
|          |          | but does | but does | ctioning |           |
|          |          | not      | not      | and      |           |
|          |          | impair   | impair   | impairs  |           |
|          |          | the      | the      | the      |           |
|          |          | flight   | flight   | flight** |           |
|          |          | and OBIT | and OBIT |          |           |
|          |          | is       | is       |          |           |
|          |          | functi   | functi   |          |           |
|          |          | oning.** | oning.** |          |           |
+----------+----------+----------+----------+----------+-----------+

**Radiosonde Performance**

+----------+----------+----------+----------+----------+-----------+
| **Per    | **Exc    | **Good** | **Ma     | **Poor** | **RRS     |
| formance | ellent** |          | rginal** |          | Perf      |
| M        |          | **Perfo  |          | **Perfo  | ormance** |
| easure** | **Perfo  | rmance** | **Perfo  | rmance** |           |
|          | rmance** |          | rmance** |          |           |
+----------+----------+----------+----------+----------+-----------+
| **Data   |          |          |          |          |           |
| Qu       |          |          |          |          |           |
| ality:** |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **       | **1 to   | **3 to   | **       |           |
| *Percent | \<1.0%** | 2.9%**   | 3.9%**   | \>4.0%** |           |
| of RAOBS |          |          |          |          |           |
| with 10% |          |          |          |          |           |
| or more  |          |          |          |          |           |
| tem      |          |          |          |          |           |
| perature |          |          |          |          |           |
| levels   |          |          |          |          |           |
| rejected |          |          |          |          |           |
| by       |          |          |          |          |           |
| NCEP**   |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **       | **0.3 to | **2.0 to | **       |           |
| *Percent | \<0.3%** | 1.9%**   | 3.9%**   | \>4.0%** |           |
| of total |          |          |          |          |           |
| tem      |          |          |          |          |           |
| perature |          |          |          |          |           |
| errors   |          |          |          |          |           |
| detected |          |          |          |          |           |
| by       |          |          |          |          |           |
| NCDC**   |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **0%**   | **0%**   | **0.05%  | **       |           |
| *Percent |          |          | to       | \>0.1%** |           |
| of total |          |          | 0.1%**   |          |           |
| height   |          |          |          |          |           |
| errors   |          |          |          |          |           |
| detected |          |          |          |          |           |
| by       |          |          |          |          |           |
| NCDC**   |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **0.5%** | **0.5 to | **1.3%   | **       |           |
| *Percent |          | 1.2%**   | to       | \>2.0%** |           |
| of total |          |          | 1.9%**   |          |           |
| pressure |          |          |          |          |           |
| errors   |          |          |          |          |           |
| detected |          |          |          |          |           |
| by       |          |          |          |          |           |
| NCDC**   |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **0%**   | **0%**   | *        | *        |           |
| *Percent |          |          | *0.05%** | *\>0.1** |           |
| of       |          |          |          |          |           |
| relative |          |          |          |          |           |
| Humidity |          |          |          |          |           |
| errors   |          |          |          |          |           |
| detected |          |          |          |          |           |
| by       |          |          |          |          |           |
| NCDC**   |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **\<1%** | **1-3%** | **       | **\>     |           |
| *Percent |          |          | 3.1-5%** | 5%**     |           |
| of wind  |          |          |          |          |           |
| errors   |          |          |          |          |           |
| detected |          |          |          |          |           |
| by       |          |          |          |          |           |
| NCDC**   |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| **Flight |          |          |          |          |           |
| Perfor   |          |          |          |          |           |
| mance:** |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| **No.    | **None** | **Not    | **Not    | **More   |           |
| Flights  |          | more     | more     | than 2** |           |
| not      |          | than 1** | than 2** |          |           |
| reaching |          |          |          |          |           |
| 400      |          |          |          |          |           |
| hPa**    |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **100%** | **90     | **85     | **Less   |           |
| *Percent |          | -99.9%** | -89.9%** | than     |           |
| higher   |          |          |          | 85%**    |           |
| than**   |          |          |          |          |           |
|          |          |          |          |          |           |
| **100    |          |          |          |          |           |
| HPA**    |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **9      | **90     | **85     | **Less   |           |
| *Percent | 5-100%** | -94.9%** | -89.9%** | than     |           |
| higher   |          |          |          | 85%**    |           |
| than 20  |          |          |          |          |           |
| HPA**    |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **8      | **70     | **60     | **Less   |           |
| *Percent | 0-100%** | -79.9%** | -69.9%** | than     |           |
| higher   |          |          |          | 60%**    |           |
| than 10  |          |          |          |          |           |
| HPA**    |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| **Second | **None** | **No     | **No     | **More   |           |
| rele     |          | more     | more     | than 3** |           |
| ases\*** |          | than 2** | than 3** |          |           |
+----------+----------+----------+----------+----------+-----------+
| **Third  | **None** | **No     | **No     | **More   |           |
| rele     |          | more     | more     | than 2** |           |
| ases\*** |          | than 1** | than 1** |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **Less   | **1      | **16-20  | *        |           |
| *Average | than 10  | 0.1-15.9 | hPa**    | *Greater |           |
| ter      | hPa**    | hPa**    |          | than 20  |           |
| mination |          |          |          | hPa**    |           |
| pressu   |          |          |          |          |           |
| re\*\*** |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| *        | **None** | **No     | **No     | **More   |           |
| *Rejects |          | more     | more     | than 5** |           |
| during   |          | than 2** | than 5** |          |           |
| baseline |          |          |          |          |           |
| \*\*\*** |          |          |          |          |           |
+----------+----------+----------+----------+----------+-----------+
| **Missed | **None** | **No     | **No     | **More   |           |
| F        |          | more     | more     | than 2** |           |
| lights** |          | than 1** | than 2** |          |           |
+----------+----------+----------+----------+----------+-----------+

**\* These only include non-induced failures of any type causing a
second or third release.**

**\*\* It is recognized these are also a function of balloon, therefore
early balloon burst will not be counted against RRS POPM.**

**\*\*\* Note, radiosondes rejected during baseline will be retested in
the next scheduled observation window as is currently done.**

**Aggregate Table**

+----------+----------+----------+----------+----------+-----------+
| **Per    | **Exc    | **Good** | **Ma     | **Poor** | **RRS     |
| formance | ellent** |          | rginal** |          | Perf      |
| M        |          | **Perfo  |          | **Perfo  | ormance** |
| easure** | **Perfo  | rmance** | **Perfo  | rmance** |           |
|          | rmance** |          | rmance** |          |           |
+----------+----------+----------+----------+----------+-----------+
| **System | **0      | **no     | **2-3    | *        |           |
| and      | missed   | more     | missed   | *Greater |           |
| software | f        | than 1   | f        | than 3   |           |
| fai      | lights** | missed   | lights** | missed   |           |
| lures:** |          | flight** |          | f        |           |
|          |          |          |          | lights** |           |
+----------+----------+----------+----------+----------+-----------+
| **Ra     | **0 R/S  | **1-2    | **3 R/S  | *        |           |
| diosonde | i        | r/s      | i        | *Greater |           |
| fail     | n-flight | i        | n-flight | than 3   |           |
| ures\*** | fa       | n-flight | failures | R/S      |           |
|          | ilures** | failures | causing  | i        |           |
|          |          | causing  | 2nd/3rd  | n-flight |           |
|          |          | 2nd/3rd  | r        | failures |           |
|          |          | r        | elease** | causing  |           |
|          |          | elease** |          | 2nd/3rd  |           |
|          |          |          |          | r        |           |
|          |          |          |          | elease** |           |
+----------+----------+----------+----------+----------+-----------+
| **Comms  | **All    | **1. FLT | **2-3    | **More   |           |
| f        | flights  | doesnt   | flts     | than 3   |           |
| ailure** | arrive** | make**   | don=t    | flts**   |           |
|          |          |          | make it  |          |           |
| **(And   | **@      | **it to  | to       | **don=t  |           |
| ba       | destin   | e        | e        | make it  |           |
| ckups)** | ation:** | ither:** | ither:** | to       |           |
|          |          |          |          | e        |           |
|          | **       | **       | **       | ither:** |           |
|          | a.NCF/NO | a.NCF/NO | a.NCF/NO |          |           |
|          | AAPORT** | AAPORT** | AAPORT** | **       |           |
|          |          |          |          | a.NCF/NO |           |
|          | **b.     | **b.     | **b.     | AAPORT** |           |
|          | NCEP**   | NCEP**   | NCEP**   |          |           |
|          |          |          |          | **b.     |           |
|          | **c.     | **c.     | **c.     | NCEP**   |           |
|          | NCDC**   | NCDC**   | NCDC**   |          |           |
|          |          |          |          | **c.     |           |
|          |          |          |          | NCDC**   |           |
+----------+----------+----------+----------+----------+-----------+

**Note, for comms failures, a running total for NCF/NOAAPORT, NCEP, and
NCDC will be used to determine which category should be assigned.**

**\* Radiosonde failure reason is determined from Flight Summary data
and the failure occurs at pressures greater than 400 hPa.**

**Applications/Model/Analyses**

+-----------+----------+----------+----------+----------+-----------+
| **Pe      | **Exc    | **Good** | **Ma     | **Poor** | **RRS     |
| rformance | ellent** |          | rginal** |          | Perf      |
| Measure** |          | **Perfo  |          | **Perfo  | ormance** |
|           | **Perfo  | rmance** | **Perfo  | rmance** |           |
|           | rmance** |          | rmance** |          |           |
+-----------+----------+----------+----------+----------+-----------+
| *         |          |          |          |          |           |
| *AWIPS:** |          |          |          |          |           |
+-----------+----------+----------+----------+----------+-----------+
| **U/A     | **All    | **no     | **No     | **more   |           |
| Profile   | work     | more     | more     | than 2** |           |
| S/W**     | cur      | than 1** | than 2** |          |           |
|           | rently** |          |          |          |           |
+-----------+----------+----------+----------+----------+-----------+
| **U/A     | **All    | **No     | **No     | **more   |           |
| Indice    | work     | more     | more     | than 2** |           |
| Calc.**   | cur      | than 1** | than 2** |          |           |
|           | rently** |          |          |          |           |
+-----------+----------+----------+----------+----------+-----------+
| **Decoder | **       | **No     | **No     | **more   |           |
| /database | Messages | more     | more     | than 2   |           |
| coded     | de       | than 1   | than 2   | fail to  |           |
| MSGS**    | coded/db | fails to | fail to  | reach    |           |
|           | without  | reach    | reach    | data     |           |
|           | p        | data     | data     | base**   |           |
|           | roblem** | base**   | base**   |          |           |
+-----------+----------+----------+----------+----------+-----------+
| **Other   | **All    | **No     | **No     | **more   |           |
| Appli     | work     | more     | more     | than 2** |           |
| cations** | cor      | than 1** | than 2** |          |           |
|           | rectly** |          |          |          |           |
+-----------+----------+----------+----------+----------+-----------+
| **NWSTG   |          |          |          |          |           |
| CLIMAT**  |          |          |          |          |           |
+-----------+----------+----------+----------+----------+-----------+
| **NCEP:** | **All    | **No     | **No     | **more   |           |
|           | work     | more     | more     | than 2** |           |
| **A       | cor      | than 1   | than 2** |          |           |
| nalyses** | rectly** | fails    |          |          |           |
|           |          | to**     |          |          |           |
| *         |          |          |          |          |           |
| *Models** |          |          |          |          |           |
|           |          |          |          |          |           |
| *         |          |          |          |          |           |
| *Freezing |          |          |          |          |           |
| Level**   |          |          |          |          |           |
|           |          |          |          |          |           |
| **QC      |          |          |          |          |           |
| r         |          |          |          |          |           |
| outines** |          |          |          |          |           |
+-----------+----------+----------+----------+----------+-----------+

**Appendix B - Operational Flight Series Test Schedule**

  ---------- ---------- ------------- ------------------------------------------
  **Date**   **Time**   **Ascension   **Special Flight Charactaristics**
                        No.**         

             **Any      **None**      **Test mode with in-flight simulator**
             time**                   

             **Any      **None**      **Test mode with in-flight simulator**
             time**                   

             **Any      **None**      **Test mode with in-flight simulator**
             time**                   

             **Any      **None**      **Test mode with in-flight simulator,
             time**                   Reload software and set to ops mode Asc
                                      No. = current ascension number for
                                      MicroART + 1**

  **Day 1**  **12**     **702**       **Nominal Sippican Only**

  **Day 1**  **0**      **703**       **Nominal Sippican Only; Use Rework to
                                      retransmit corrected TTBB message**

  **Day**                             

  **Day**                             

  **Day**                             

  **Day**    **0**      **707**       **Nominal Sippican Only**

  **Day**    **12**     **708**       **Nominal Sippican Only; Leave RH cap on
                                      instrument**

  **Day**    **0**      **1**         **Nominal Sippican Only,**

  **Day**    **12**     **2**         **Nominal Sippican Only; Reject 1 sonde
                                      during baseline and document**

  **Day**    **0**      **3**         **Nominal Sippican Only,**

  **Day**    **12**     **4**         **2^nd^ Back up comms path; Primary and
                                      1^st^ backup fail**

  **Day**    **0**      **5**         **Nominal Sippican Only,**

  **Day**    **0**      **6**         **Nominal Sippican Only,**

  **Day**    **12**     **7**         **2^nd^ release flight\*\***

  **Day**    **0**      **8**         **Nominal Sippican Only,**

  **Day**    **12**     **9**         **Nominal Sippican Only,**

  **Day**    **0**      **10**        **Nominal Sippican Only,**

  **Day**    **12**     **11**        **1^st^ Back up comms path; Use Rework to
                                      transmit**

  **Day**    **0**      **12**        **Nominal Sippican Only,**

  **Day**    **12**     **13**        **Nominal Sippican Only,**

  **Day**    **0**      **14**        **Nominal Sippican Only,**

  **Day**    **12**     **15**        **Nominal Sippican Only,**

  **Day**    **0**      **16**        **1^st^ & 2^nd^ release fail, 3^rd^
                                      release flight is nominal\*\***

  **Day**    **12**     **17**        **Nominal Sippican Only,**

  **Day**    **0**      **18**        **Nominal Sippican Only,**

  **Day**    **12**     **19**        **Power Failure near end of flight; Rework
                                      to transmit ABV message**

  **Day**    **0**      **20**        **Nominal Sippican Only,**

  **Day**    **12**     **21**        **Nominal Sippican Only; Reject 2
                                      radiosondes during baseline\*\***

  **Day**    **0**      **22**        **Nominal Sippican Only,**

  **Day**    **12**     **23**        **2^nd^ release flight:**

  **Day**    **0**      **24**        **Nominal Sippican Only,**

  **Day**    **12**     **25**        **Nominal Sippican Only,**

  **Day**    **0**      **26**        **Nominal Sippican Only,**

  **Day**    **12**     **27**        **1^st^ Back up comms path**

  **Day**    **0**      **28**        **Nominal Sippican Only,**

  **Day**    **6**      **29**        **Special; Change pressure after flight
                                      using rework and then retransmit
                                      messages**

  **Day**    **12**     **30**        **Nominal Sippican Only,**

  **Day**    **18**     **31**        **Special, Use CD backup to retransmit all
                                      messages**

  **Day**    **0**      **32**        **2^nd^ Back up comms path**

  **Day**    **6**      **33**        **Special; Use rework to retransmit all
                                      messages**

  **Day**    **12**     **34**        **Nominal Sippican Only,**

  **Day**    **18**     **35**        **Special**

  **Day**    **0**      **36**        **Nominal Sippican Only,**

  **Day**    **6**      **37**        **Special**

  **Day**    **12**     **38**        **Nominal Sippican Only,**

  **Day**    **18**     **39**        **Special, 3 releases, all flights fail,
                                      reprocess best flight and transmit\*\***

  **Day**    **0**      **40**        **Nominal Sippican Only,**

  **Day**    **12**     **41**        **Nominal Sippican Only,**

  **Day**    **0**      **42**        **Nominal Sippican Only,**

  **Day**    **12**     **43**        **Nominal Sippican Only,**

  **Day**    **0**      **\-\-\--**   **No obs taken; Issue Ano data@ message**

  **Day**    **12**     **44**        **Nominal Sippican Only,**

  **Day**    **0**      **45**        **Nominal Sippican Only,**

  **Day**    **12**     **46**        **Nominal Sippican Only,**

  **Day**    **0**      **47**        **Nominal Sippican Only,**

  **Day**    **12**     **48**        **Nominal Sippican Only,**

  **Day**    **0**      **49**        **Nominal Sippican Only,**

  **Day**    **12**     **50**        **Nominal Sippican Only,**

  **Day**    **0**      **51**        **Nominal Sippican Only,**

  **Day**    **12**     **52**        **Nominal Sippican Only,**

  **Day**    **0**      **53**        **Nominal Sippican Only,**

  **Day**    **12**     **54**        **Nominal Sippican Only,**

  **Day**    **0**      **55**        **Nominal Sippican Only,**

  **Day**    **12**     **56**        **Nominal Sippican Only,**

  **Day**    **0**      **57**        **Nominal Sippican Only,**

  **Day**    **12**     **58**        **Nominal Sippican Only,**

  **Day**    **0**      **59**        **Nominal Sippican Only,**

  **Day**    **12**     **60**        **Nominal Sippican Only,**

  **Day**    **0**      **61**        **Nominal Sippican Only,**

  **Day**    **12**     **62**        **Nominal Sippican Only,**

  **Day**    **0**      **63**        **Nominal Sippican Only,**

  **Day**    **12**     **64**        **Nominal Sippican Only,**

  **Day**    **0**      **65**        **Nominal Sippican Only,**

  **Day**    **12**     **66**        **Nominal Sippican Only,**
  ---------- ---------- ------------- ------------------------------------------

**\* If a second or third release should or before the scheduled fight,
then the actual occurrence can replace one-for-one the scheduled
condition.**

**\*\* Any situations induced to create a special situation will not be
used for performance measure calculations.**

[^1]: RAOB, the complete rawinsonde observation program, is produced by
    Environmental Research Services, 8, 1994-2002.