Number range CHAPTER

![](media/image1.wmf){width="1.61875in" height="1.625in"}

**Valid Time Event Code (VTEC)/**

**Watch-by-County (WBC)**

**Phase II**

**Operational Test and Evaluation (OT&E) Assessment Report**

**December 2004**

**U.S. DEPARTMENT OF COMMERCE**

**National Oceanic and Atmospheric Administration's**

**National Weather Service**

**Office of Science and Technology**

**Office of Climate Water and Weather Services**

**Office of Operational Systems**

**National Centers for Environmental Prediction**

**Valid Time Event Code (VTEC)/**

**Watch-by-County (WBC)**

**Phase II**

**Operational Test and Evaluation (OT&E) Assessment Report**

**December 2004**

**Approved:**

**\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_\_\_**

Gregory A. Mandt, Director Date

Office of Science and Technology

**\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_\_\_**

Dennis McCarthy, Acting Director Date

Office of Climate, Water, and Weather Services

**\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_\_\_**

Louis W. Uccellini, Director Date

National Centers for Environmental Prediction

**\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_\_\_**

John McNulty, Director Date

Office of Operational Systems

**Valid Time Event Code (VTEC)/Watch-by-County (WBC) Operational Test
and Evaluation (OT&E) Assessment Report**

**1. Introduction** [3](#__RefHeading___Toc93280855)

**2. Recommendations** [3](#__RefHeading___Toc93280856)

**3. Purpose** [4](#__RefHeading___Toc93280857)

**4. Success Criteria Results** [4](#__RefHeading___Toc93280858)

**5. Conclusions for the OT&E Objectives**
[9](#__RefHeading___Toc93280859)

**6. Lessons Learned** [18](#__RefHeading___Toc93280860)

**7. Attachment A -- VTEC Product Statistics**
[24](#__RefHeading___Toc93280861)

**8. Attachment B - VTEC Product Statistics by Service Area**
[25](#__RefHeading___Toc93280862)

**9. Attachment C -- VTEC Statistics for WarnGen**
[26](#__RefHeading___Toc93280863)

**10. Attachment D -- VTEC Statistics for WWA**
[27](#__RefHeading___Toc93280864)

**11. Attachment E -- Test Trouble Report VTEC Summary**
[28](#__RefHeading___Toc93280865)

**12. Attachment F -- WBC Statistics** [30](#__RefHeading___Toc93280866)

**14. Attachment G - Test Trouble Report WBC Summary**
[31](#__RefHeading___Toc93280867)

**15. Attachment H -- GHG Commentary** [33](#__RefHeading___Toc93280868)

**16. Attachment I - TTR Analysis**

**1. Introduction**[]{#__RefHeading___Toc93280855 .anchor}

Two Operational Test and Evaluations (OT&E) were performed concurrently
for the Valid Time Event Coder (VTEC) and Watch-By-County (WBC) from
August 27 to October 15, 2004. Both OT&Es were conducted in accordance
with the *Valid Time Event Code-Watch By County Operational Test and
Evaluation Test Plan*, dated, August, 13, 2004.

**2. Recommendations**[]{#__RefHeading___Toc93280856 .anchor}

Based on the results of the OT&Es described in this report, these
recommendations should be undertaken:

**VTEC**

> The following recommendations were discussed during the OT&E VTEC/WBC
> Summit Meeting held last October 29, 2004:

1.  Go forward with VTEC in five WarnGen Products on February 8, 2005
    (SVR, TOR, SVS, SMW and MWS). The statistical criteria were met; the
    SVS and MWS were met in the final two weeks.

    2.  Do not go operational with VTEC in WWA products on February
        8, 2005. The statistical criteria were not met. The service
        backup did not pass. The OT&E sites retain experimental VTEC
        code "x".

    3.  Do not go operational with WCN on February 8, 2005. The
        statistical criteria for WCN products issued (82%) was not met.
        The WWA software was deemed not stable.

    4.  Do not go operational with VTEC in any Hydrology products (WWA,
        WarnGen, and RiverPro) on February 8, 2005.

    5.  Limit the scope of the VTEC and WBC requirements while phasing
        in the system capabilities.

        **WBC**

        Additional WBC related recommendations include:

```{=html}
<!-- -->
```
1.  Implement operational Valid Time Event Code in initial and final
    Watch Outline Update Message (WOU) on February 8, 2005.

2.  Discontinue legacy Watch County List (SEV) and State Areal Outline
    Statement (SLS) on February 22, 2005.

3.  Continue Watch County Notification Message (WCN) and update WOU as
    experimental products until fall 2005.

4.  Switch WCN creation and dissemination software from WWA to Graphical
    Hazards Generation (GHG) text formatter in GFE.

**3. Purpose**[]{#__RefHeading___Toc93280857 .anchor}

The VTEC/WBC Operational Test and Evaluation (OT&E) was used to
determine the operational effectiveness and suitability of a system
under realistic conditions, and to determine if specified minimum
acceptable operational performance requirements have been satisfied. The
OT&E of VTEC functionality demonstrated and validated the NWS' readiness
to release operational messages with the VTEC string implemented. The
OT&E of WBC functionality demonstrated and validated the NWS' readiness
to release the WBC suite of operational products. This report documents
the OT&E results; specifically whether the test objectives and success
criteria were met and what conclusions can be formulated after analyzing
the results. The Attachments A-I are included to document the VTEC/WBC
statistical data that were gathered during the OT&E.

**4. Success Criteria Results**[]{#__RefHeading___Toc93280858 .anchor}

Evaluation criteria in each of the test scenarios allowed the service
program focal points and test personnel to validate the creation of the
VTEC coding string. Evaluation criteria are established to ensure the
test objectives are met and are classified as follows:

1.  *[Training will be assessed by survey of OT&E participants who
    receive the VTEC and WBC training provided by OS6 personnel. Care
    will be taken to ensure software and policy issues are not
    considered training failures. Training evaluation will be used to
    improve training to be offered in Fall 2004 to all forecasters.
    Training effectiveness will be evaluated mid-August to enable time
    to fold improvements into the training plan/materials.]{.underline}*

    a.  Success criteria: Survey responses from forecasters rating
        effectiveness of various aspects of training to enable them to
        issue VTEC and Watch by County products.

    b.  Success: greater than or equal to 3 average on scale of 1-5.

    c.  Failure: less than 3 average rating on scale of 1-5. Failure
        results in providing significant improvement to training prior
        to offering training prior to VTEC going operational.

        **RESULTS: PASS**

        **For VTEC training, the average rating was a successful 3.5 on
        a 5 point scale.**

```{=html}
<!-- -->
```
1.  *[VTEC code compliance will be evaluated by inspection of VTEC
    strings in products issued by OT&E offices during the test period.
    Every field of the VTEC strings will be checked. One error or
    multiple errors in one product's VTEC string will be counted as one
    product error. Multiple lines of VTEC will count
    separately.]{.underline}*

    a.  Success criteria:

        i.  Format of VTEC String matches NWSI 10-1701 and NWSI 10-1703

        ii. Product type (O, T, E, X) and Mass News Dissemination (MND)
            header are consistent.

    b.  Success: Product VTECs accurate 95% of the time (tolerance, 5%).

    c.  Failure: aggregate of erroneous products is more than 20% of
        total products issued during the period.

        **RESULTS:**

+--------------------------------------+--------------------+----------+
| **Severe Weather (TOR, SVR, SVS)**   | **2a (i) and       | **PASS** |
|                                      | (ii):**            |          |
+--------------------------------------+--------------------+----------+
|                                      | **2b**             | **PASS** |
+--------------------------------------+--------------------+----------+
|                                      | **2c**             | **PASS** |
+--------------------------------------+--------------------+----------+
|                                      |                    |          |
+--------------------------------------+--------------------+----------+
| **Fire Weather (RFW)**               | **2a (i) and       | **PASS** |
|                                      | (ii):**            |          |
+--------------------------------------+--------------------+----------+
|                                      | **2b**             | **FAIL** |
+--------------------------------------+--------------------+----------+
|                                      | **2c**             | **FAIL** |
+--------------------------------------+--------------------+----------+
|                                      |                    |          |
+--------------------------------------+--------------------+----------+
| **Public Weather (NPW, WSW)**        | **2a (i) and       | **PASS** |
|                                      | (ii):**            |          |
+--------------------------------------+--------------------+----------+
|                                      | **2b**             | **FAIL** |
+--------------------------------------+--------------------+----------+
|                                      | **2c**             | **FAIL** |
+--------------------------------------+--------------------+----------+
|                                      |                    |          |
+--------------------------------------+--------------------+----------+
| **Marine Severe Weather**            | **2a (i) and       | **PASS** |
|                                      | (ii)**             |          |
| **(SMW, MWS as MA.W)**               |                    |          |
+--------------------------------------+--------------------+----------+
|                                      | **2b**             | **PASS** |
+--------------------------------------+--------------------+----------+
|                                      | **2c**             | **PASS** |
+--------------------------------------+--------------------+----------+
|                                      |                    |          |
+--------------------------------------+--------------------+----------+
| **Coastal/Lakeshore Flood and High   | **2a (i) and       | **FAIL** |
| Surf (CFW)**                         | (ii)**             |          |
+--------------------------------------+--------------------+----------+
|                                      | **2b**             | **FAIL** |
+--------------------------------------+--------------------+----------+
|                                      | **2c**             | **FAIL** |
+--------------------------------------+--------------------+----------+
|                                      |                    |          |
+--------------------------------------+--------------------+----------+
| **Hydrology (FFW, FFS)**             | **2a (i) and       | **PASS** |
|                                      | (ii)**             |          |
+--------------------------------------+--------------------+----------+
|                                      | **2b**             | **PASS** |
+--------------------------------------+--------------------+----------+
|                                      | **2c**             | **PASS** |
+--------------------------------------+--------------------+----------+
|                                      |                    |          |
+--------------------------------------+--------------------+----------+
| **Hydrology (FFA, FLW, FLS)**        | **2a (i) and       | **FAIL** |
|                                      | (ii)**             |          |
+--------------------------------------+--------------------+----------+
|                                      | **2b**             | **FAIL** |
+--------------------------------------+--------------------+----------+
|                                      | **2c**             | **FAIL** |
+--------------------------------------+--------------------+----------+

2.  *[Watch By County products compliance was evaluated during the first
    OT&E in 2003. Will verify products were not degraded when usability
    was improved.]{.underline}*

    a.  Success Criteria:

        i.  WCNs compliant with 10-511.

        ii. WOUs compliant with 10-512.

    b.  Success:

        i.  Single watch products 95% accurate (tolerance 5%).

        ii. Multiple watch products 85% accurate (tolerance 5%).

    c.  Failure: aggregate of erroneous products is more than 10%
        (single) or 20% (multiple).

        **RESULTS:**

  --------------------------------------- -------------------------------
  **3a (i):**                             **PASS**

  **3a (ii):**                            **PASS**

  **3b (i) WOU:**                         **PASS**

  **3b (i) WCN:**                         **FAIL**

  **3b (ii) WOU:**                        **N/A**

  **3b (ii) WCN:**                        **FAIL**
  --------------------------------------- -------------------------------

3.  *[Product automated Quality Control will be evaluated by performing
    internal consistency check of the products.]{.underline}*

    a.  Success criteria:

        i.  Event Tracking Numbers (ETNs) are continuous in normal
            operations

        ii. ETNs are continuous in service backup mode

        iii. Corrections are accurately reflected in the BBB field

        iv. Correctly ordered segments in multiple watch convective
            watches (WCNs)

    b.  Success: Minimum of 85% of products/product segments found to be
        internally consistent (tolerance, 5%).

    c.  Failure: inconsistency of either type found within 20% of the
        OT&E products.

        **RESULTS:**

  ----------------------------------- --------------------------- ----------
  **Severe Weather:**                 **WOU, SVR, TOR, SVS**      **PASS**

                                      **WCN**                     **FAIL**

                                                                  

  **Fire Weather:**                   **RFW**                     **FAIL**

                                                                  

  **Public Weather:**                 **NPW, WSW**                **FAIL**

                                                                  

  **Marine Severe Weather:**          **SMW, MWS (as MA.W)**      **PASS**

                                                                  

  **Coastal/Lakeshore Flood: and High **CFW**                     **PASS**
  Surf products**                                                 

                                                                  

  **Hydrology**                       **FFW, FFS**                **PASS**

                                      **FFA, FLW, FLS**           **FAIL**
  ----------------------------------- --------------------------- ----------

4.  *[Concept of operations will be evaluated by verifying the intended
    action of the VTEC and WBC products was accurately communicated.
    Will require same day review of the products.]{.underline}*

```{=html}
<!-- -->
```
a.  Success criteria

```{=html}
<!-- -->
```
1.  Any county receiving a watch or warning was initiated via NEW or EXA
    action.

\(ii\) Updates to products are properly sequenced.

> \(1\) Upgrade/downgrade, Extensions, and continued products.
>
> \(iii\) Cancellations are issued at appropriate times,
>
> \(iv\) Counties accurately warned in service Backup,
>
> \(v\) VTEC and headline are internally consistent.

b.  Success will be determined by 85% accurate products (tolerance, 5%).
    Each segment will be evaluated and will be counted. Reviewer must
    verify action codes, phenomenon codes and significance match the
    verbs and adjectives in the headlines. And also verify Time is
    valid.

c.  Failure: More than 20% of the products are in error.

    **RESULTS:**

  ---------------------------- ---------------------------------- ----------
  **Severe Weather:**          **WOU, SVR, TOR, SVS, WCN**        **PASS**

  **Fire Weather:**            **RFW**                            **FAIL**

  **Public Weather:**          **NPW, WSW**                       **FAIL**

  **Marine Severe Weather:**   **SMW, MWS (as MA.W)**             **PASS**

  **Coastal/Lakeshore Flood    **CFW**                            **PASS**
  and High Surf products**                                        
  ---------------------------- ---------------------------------- ----------

6.  *[NWS Partners will verify the products are complying with the
    advertised specifications.]{.underline}*

    a.  Success: greater than or equal to 3 average on scale of 1-5

    b.  Failure: less than 3 average rating on scale of 1-5. Failure
        results in providing significant improvement to training prior
        to offering training prior to VTEC going operational.

        **RESULT: PASS**

        The original evaluation items were initially sent to the
        Partners as written, with the evaluation scales ranging from 1
        to 5. However, during a November 16, 2004 conference call, the
        Partners asked to subdivide the evaluation items into two
        sub-items, as the Partners thought that the wording was too
        broad. They asked to have the first item scored via a Yes/No
        rather than a 1 to 5 scale. Since the two items were gauging
        Partner opinions of the OT&E, their parameters for evaluation
        were agreed to accurately reflect their assessments. Votes and
        comments were transcribed from the conference call.

The Partners divided this criterion into two sections.

> 1\. First, the format of the VTEC string itself was rated with a
> *Response: Yes or No, referring to whether the strings complied with
> the advertised specifications.*
>
> The consensus of the Partners was: [\"No\" for Hydrologic
> products]{.underline}, and [\"Mostly Yes\" for Non-hydrologic
> products]{.underline}.
>
> 2\. Then, the Partners rated whether the usage of VTEC accurately
> represents the forecaster\'s intent. *Response: 1 (does not represent)
> through 5 (accurately represents). An example of not representing
> intent is using an EXP action code well before the actual expiration
> of a product rather than a CON action code*.

**The average score was 3.25.**

> The Partners that provided the input were M. McInnis (First Alert), K.
> Strebe (TWC), C. Keller (NYSEMO), and B. Callahan (WSI).

**5. Conclusions for the OT&E Objectives**[]{#__RefHeading___Toc93280859
.anchor}

The conclusions for the following NDFD OT&E objectives are:

**VTEC (See Attachments A-E)**

All event driven watch and warning products issued by the NWS must
contain an accurate VTEC string. Assessment of the operational
effectiveness of VTEC will be accomplished through validation of the
following test objectives and associated evaluation criteria:

1.  *[Demonstrate that accurate VTEC codes are generated in operational
    products in an operational environment under real world weather
    situations and scenarios.]{.underline}*

    a.  VTEC codes conform to the specification defined in NWSI 10-1703.

        Discussion:

        [Severe Weather:]{.underline} 337 of 343 Severe Thunderstorm and
        Tornado Warnings contained proper VTEC (98% pass rate). A
        WarnGen software bug accounted for the errors of 15% of Severe
        Weather Statements issued. This bug was fixed by Week 5 of the
        OT&E.

        **Conclusion:** *[OT&E objective was met]{.underline}*

        **The success rate was 100% after this fix was implemented in
        AWIPS.**

        Discussion:

        [Fire and Public Weather:]{.underline} WWA "Test Mode" did not
        have full functionality for the first four weeks of the OT&E.
        The failures in this suite of products resulted from two
        significant software bugs:

        1.  Upgrade and downgrade of products (e.g. from watch to
            warning) did not work in TEST mode.

        2.  The Event Tracking Number falsely incremented during random
            product actions throughout the OT&E.

            **Conclusion:** *[OT&E objective was NOT met.]{.underline}*

            **Success rate did not meet the evaluation threshold.**

            Discussion:

            [Marine Severe Weather:]{.underline} 42 of 45 Special Marine
            Warnings contained proper VTEC (93% pass rate). A WarnGen
            software bug accounted for the errors of 17% of Marine
            Weather Statements (as MA.W) issued. This bug was fixed by
            Week 5 of the OT&E.

            **Conclusion:** *[OT&E objective was met]{.underline}*

            **The success rate was 100% after this fix was implemented
            in AWIPS.**

            Discussion:

            Coastal/Lakeshore Flood and High Surf products (CFW): 47 of
            71 CFW products contained proper VTEC.

            **Conclusion:** *[OT&E objective was NOT met.]{.underline}*

            **There was 66% accuracy which is below the success rate.**

Discussion:

> [WarnGen:]{.underline} All Flash Flood Warnings contained proper VTEC
> except a few which were manually edited in such a way that violated
> VTEC policy in 10-1703.
>
> **Conclusion:** *[OT&E objective was met]{.underline}*
>
> **Although the OT&E objective was met, WarnGen needs to be adapted to
> meet the special requirements of hydrology - particularly in the area
> of short-fuse product extensions.**
>
> Discussion:
>
> [RiverPro:]{.underline} Several problems with VTEC accuracy occurred,
> including (1) incorrect incrementing of ETNs within the application,
> (2) incorrect time assignments in the Flood Begin Date/Time group, (3)
> incorrect VTEC elements and ETNs in point-specific flood advisories,
> and (4) major product format errors. Also, use of more than one
> application to produce products under a given identifier resulted in
> ETN conflicts. This brought about a realization that further
> refinements were needed to WFO hydrologic applications to ensure both
> national product consistency and correct inclusion of VTEC.
>
> **Conclusion:** *[OT&E objective was NOT met.]{.underline}*
>
> **Products generated by RiverPro failed to meet OT&E pass goals due to
> several problems with the hydrologic applications and insufficient WFO
> understanding of hydrologic product policy at the field office
> level.**

b.  VTEC strings can be accurately generated when WFOs are in service
    backup.

    **Conclusions:**

    **[WarnGen:]{.underline} WFOs Cheyenne and Riverton, WY successfully
    demonstrated service backup using WarnGen on November 8, 2004.**
    *[OT&E objective was met]{.underline}*

    **[WWA:]{.underline} This capability was not successfully
    demonstrated in WWA during the OT&E.** *[OT&E objective was NOT
    met]{.underline}*

    **[RiverPro:]{.underline} This capability was successfully
    demonstrated at WFOs Corpus Christi and San Antonio, TX in WarnGen
    during the OT&E**. *[OT&E objective was met]{.underline}*

```{=html}
<!-- -->
```
2.  *[Demonstrate the VTEC concept of operations is
    viable.]{.underline}*

    a.  Forecasters understand how to accomplish forecast functions such
        that automated VTEC features yield correct interpretation by
        customers.

        Discussion:

> 91% of forecasters believe WarnGen products pose little problem as
> they fully understand how to use the software in operations.
>
> While 86% believe RiverPro did not hinder product creation, comments
> received revealed manual editing of the VTEC was occurring and
> software streamlining is necessary. Only a small percentage of offices
> needed to use RiverPro during the OT&E.
>
> Forecasters were split on the effectiveness of WWA. 56% judged WWA to
> have no discernable impact on operations while 44% felt a negative
> impact. Comments were particularly focused on the non-intuitive nature
> of the software which caused forecasters to change their approach to
> long-fuse hazard creation

**Conclusion:** *[OT&E objective was PARTIALLY met]{.underline}*

**WarnGen easily works in WFO VTEC operations while WWA posed
significant problems. A larger sample of WFO RiverPro use is needed
before a determination can be made.**

b.  The AWIPS software is usable for reliable issuance of watch,
    warning, and advisory products and system performance is not
    degraded during hazardous weather or service backup operations.

    Discussion:

> WarnGen only had one problem during OT&E and that was during a complex
> tropical situation where the AWIPS system was being taxed overall.
> RiverPro had significant configuration issues early which adversely
> impacted both product creation and customer ability to receive
> products. WWA experienced severe slowdowns at several offices, saw
> many configuration issues and had a general inability to work as
> expected.
>
> Service backup tests were successful for both RiverPro and WarnGen,
> but the WWA backup test failed.
>
> **Conclusion:** *[OT&E objective was PARTIALLY met]{.underline}*

**WarnGen is ready for VTEC implementation while RiverPro and WWA have
issues to be addressed.**

c.  Implementation of VTEC causes no significant increase in forecaster
    workload.

    Discussion:

> 42% believe there was no impact whatsoever on their forecast
> operations while 24% believe there was a somewhat negative impact,
> such as longer product creation times, with the implementation of
> VTEC. The vast majority of the negative impact can be attributed to
> many forecasters having to switch to WWA during OT&E to create their
> long-fuse products. Other forecasters noted they felt compelled to
> manually QC the VTEC before issuance, a step they previously did not
> do.
>
> **Conclusion:** *[OT&E objective was PARTIALLY met]{.underline}*

**Excluding WWA, VTEC posed little additional work for the
forecasters.**

3.  *[Demonstrate training was provided to meet the objectives defined
    for the VTEC program.]{.underline}*

```{=html}
<!-- -->
```
a.  Training addressed the key components necessary to successfully
    issue watches, warnings, advisories and follow-up statements with
    VTEC strings.

    Discussion:

    Live teletraining, presentations with talking points and web pages
    were created to assist forecasters in the transition to VTEC.

    **Conclusion:** *[OT&E objective was PARTIALLY met]{.underline}*

    **The key components were addressed. However comments received,
    during training and again during OT&E, revealed a lack of
    understanding in the use of WWA for product creation. Hands-on
    training was requested to assist in the transition.**

```{=html}
<!-- -->
```
4.  *[Demonstrate that VTEC quality control functionality is implemented
    effectively.]{.underline}*

```{=html}
<!-- -->
```
a.  Products are standardized in accordance with the requirements of the
    VTEC specification.

    Discussion:

    [WarnGen:]{.underline} WarnGen utilizes a passive quality control
    software application which checks a warning or follow up statement
    prior to dissemination from the AWIPS text editor. The application
    checks the VTEC content for compliance with NWSI 10-1703 and the
    content of the product.

    **Conclusion:** *[OT&E objective was PARTIALLY met]{.underline}*

    **WWA: The WWA quality control application permitted incorrect
    combinations of VTEC lines in the segments of some WSW products. For
    example, a WSW was issued with cancellation (CAN) and continuation
    (CON) action codes in one segment for the same group of zones.
    Obviously, an event cannot be both cancelled and remain in effect.**

    **Headline errors detected in products were attributed to local edit
    requirements. These were corrected by the sites when identified.**

    Discussion:

    [RiverPro:]{.underline} There was insufficient quality control of
    ETN integrity in products issued by RiverPro. Some ETNs were
    assigned erroneous values while others did not increment correctly.
    Use of other applications to generate products under the same AWIPS
    identifier as RiverPro caused more than one product to share the
    same ETN.

    **Conclusion:** *[OT&E objective was NOT met]{.underline}*

    **ETN errors were not detected by the RiverPro application**.

```{=html}
<!-- -->
```
5.  *[Demonstrate VTEC can be read and processed by NWS customers and
    partners.]{.underline}*

```{=html}
<!-- -->
```
a.  The intended watch or warning could be ingested and decoded.

    Discussion:

> The Partners divided this criterion into two sections. First, they
> rated the format of the VTEC string itself (i.e., if they were able to
> accurately read, decode, and process the VTEC). *Response: 1 (was not
> able to be decoded) through 5 (was easily decoded).*

**Conclusion:** *[OT&E objective was met]{.underline}*

**The average score they gave was 3.875.**

> Discussion:
>
> Then, the Partners rated whether the VTEC in a product was consistent
> with the text in that product and with previous and later products in
> that ETN series). *Response: 1 (was never consistent) through 5 (was
> always consistent)*

**Conclusion:** *[OT&E objective was met]{.underline}*

**The average score they gave was 3.25.**

> The Partners also noted the following:

-   VTEC has to be highly reliable to be usable by the customers.

-   Some of the more egregious software errors were corrected during the
    OT&E.

-   There was a significant improvement toward the end of the OT&E.

-   There was tremendous variability in error rates from product to
    product (i.e., short duration warnings vs. hydrologic products) and
    from office to office (i.e., diligent configuration, preparedness,
    and forecast training paid off in fewer VTEC errors).

-   VTEC can succeed, and if it does, it will mean a substantial
    improvement in the services that can be provided to customers.

-   For those offices that continue to use VTEC after the OT&E, the
    primary error seen is orphaned events. Products with a continuing
    VTEC event are not being followed up before product expiration, even
    though the VTEC event ending time may be well in the future.

-   One partner added, "It is incumbent on the NWS to push forward with
    timely implementation of other products not included in the initial
    rollout. This is particularly true because most of the issues VTEC
    will substantially help with are associated with long fuse events
    (event ending times, upgrades, downgrades), which are not part of
    the February 2005 release".

    **WBC (See Attachments F-I)**

1.  *[Demonstrate that WBC products (WOU and WCN) are accurately
    generated in an operational environment under real world weather
    situations and scenarios.]{.underline}*

    a.  WBC products conform to the specification defined in NWSI
        10-511, 10-512, and 10-1703

        **Conclusion:** *[OT&E objective was met]{.underline}*

        **Products generated by WWA conformed to NWSI specifications.**

    b.  WBC products can be accurately generated when WFOs and/or the
        Storm Prediction Center (SPC) are/is in service backup

        i.  WFOs can provide effective service backup for convective
            watches using AWIPS software.

            **Conclusion:** *[OT&E objective was met]{.underline}*

            **WWA demonstrated service backup capability in multiple
            static tests and for a real time watch situation. WWA
            exhibited difficulty with the "k" code in the VTEC string
            when the primary WFO resumed normal operations ("O" mode
            rather than "E" or "T" modes).**

        ii. Air Force Global Weather Center can provide effective
            service backup for SPC in convective watch situations using
            Watch By County products.

            **Conclusion:** *[OT&E objective was met]{.underline}*

            **Air Force Global Weather Center (AFGWC) successfully
            generated all specified Watch-by-County Products indicated
            in the NWS Memorandum of Agreement (SEL, SAW, and WOU) for
            SPC backup operations during a series of OT&E static tests.
            WOU products created at AFWGC contained appropriate MND
            headers and VTEC codes. WOU updates were also successfully
            created.**

```{=html}
<!-- -->
```
2.  [*Demonstrate the WBC concept of operations is viable*.]{.underline}

    a.  Forecasters at both the WFO and SPC understand how to generate
        WBC products in terms of segmentation and ordering

        **Conclusion:** *[OT&E objective was met]{.underline}*

        **WFO forecasters understand how to generate WCN with proper
        segmentation and ordering after a 4 to 6 week familiarization
        period. Forecasters at WFOs Pittsburgh, Charleston SC, and
        Pleasant Hill have issued numerous single watch WCNs since June
        2004 with proper segmentation and ordering.**

        i.  SPC can cancel the entire convective watch area using a WOU

> **Conclusion:** *[OT&E objective was met]{.underline}*
>
> **SPC successfully issued final WOU products canceling the entire
> watch area. Success was demonstrated for watches canceled by properly
> formatted WCNs prior to watch expiration and for watches that expired.
> Proper VTEC action codes appeared in these products.**

b.  Forecasters understand how to generate WBC action codes for
    real-time service operations.

    i.  WFOs can update, cancel, and extend in time watches using WCNs
        for their geographic area of responsibility. WCNs contain proper
        Universal Geographic Codes, Valid Time Event Code (VTEC), and
        list of counties, independent cities, and marine zones for each
        segment of the product.

        **Conclusion:** *[OT&E objective was met]{.underline}*

        **WFOs consistently issued proper WCNs for one watch situations.
        There were three significant software "bugs" in WWA which caused
        WCN creation and dissemination failures.**

    ```{=html}
    <!-- -->
    ```
    1.  **Watches issued for "TATTNALL COUNTY" Georgia.**

    2.  **Multiple watches in effect at the same time.**

    3.  **Watches issued during the 8^th^ or 9^th^ month of the year.**

        i.  WFOs can combine more than one active convective watch into
            a WCN product.

            **Conclusion:** *[OT&E objective was NOT met]{.underline}*

            **WFOs had limited success issuing WCNs for multiple watch
            situations using WWA. A WWA software "bug" prevented several
            WFOs from issuing multiple watch WCNs during static drills.
            The offices discovered during testing that the deletion of
            the proposed watch allowed this task to be accomplished
            (October 13 in Central Region and October 14 in Eastern
            Region).**

        ii. Affected WFOs can "clear" all counties and independent
            cities from the convective watch for their geographic area
            of responsibility using the WCN.

> **Conclusion:** *[OT&E objective was met]{.underline}*

**WWA contained a software "bug" which caused the Event Tracking Number
(ETN) in VTEC to switch from a national to local number when the
remaining counties were "cleared" from a watch area.**

c.  The AWIPS software is usable for reliable issuance of watch products
    and system performance is not degraded during hazardous weather or
    service backup operations.

    Discussion:

    The AWIPS software, specifically the Watch, Warning, and Advisory
    (WWA) application, demonstrated its reliability and operational
    viability for issuing and tracking the life cycle of a single watch
    (for either severe thunderstorm of tornado). The same could not be
    said of WWA while fielding multiple watch scenarios. While
    confronting weather scenarios, both live and scripted, that included
    the managing of two or more watches, application
    glitches/deficiencies and operator errors contributed to the
    generation of enough erroneous products to fall below the
    established error free product threshold, of 85%, set forth in
    VTEC-WBC OT&E Plan. Similar problems were present during service
    backup operations.

    **Conclusion:** *[OT&E objective was met]{.underline}*

    **AWIPS system performance was not adversely impacted while using
    the WWA application to ingest, create, and/or distribute WBC related
    products. The current state of WWA functionality can not be used to
    reliably issue WBC related products.**

```{=html}
<!-- -->
```
d.  Implementation of WBC causes no significant increase in forecaster
    workload.

    **Conclusion:** *[OT&E objective was met]{.underline}*

    **Full Watch By County implementation involves implementing two new
    convective watch products (WOU, WCN) and the discontinuation of
    three legacy watch products (SEV, SLS, SPS). There is a noticeable
    workload decrease for 52 State Liaison Offices. As of February 22,
    2005, they are no longer required to issue convective watch products
    for counties outside their geographic area of responsibility.**

```{=html}
<!-- -->
```
3.  *[Demonstrate adequate training was provided to meet the objectives
    defined for the WBC program.]{.underline}*

    a.  Training addressed the key components necessary to successfully
        issue WBC products.

        Discussion:

        Training was provided to all WFOs on basic WBC procedures. More
        individual instruction is needed on watch replacement scenarios
        and service backup situations. This could be in the form of
        exercises using an off-line system (WES) and application with
        national WBC tests. The best way to ensure WBC proficiency for
        all staff is for NWS upper management to require all MICs to put
        WBC training exercises in their local office training plans.

        **Conclusion:** *[OT&E objective was met]{.underline}*

        **In conclusion, the training did address the key components
        necessary to successfully issue WBC products.**

4.  *[Demonstrate that quality control functionality for WOU and WCN
    products is implemented effectively.]{.underline}*

    a.  Products are standardized in accordance with the requirements of
        the WBC product and operational specifications.

        **Conclusion:** *[OT&E objective was met]{.underline}*

        **The text portion of the WOU and WCN has been of high quality.
        The VTEC in the WOU has also been of high quality. WCNs created
        and disseminated in WWA have contained erroneous VTEC Event
        Tracking Numbers (local rather than national).**

5.  *[Demonstrate WBC products can be read and processed by NWS
    customers and partners.]{.underline}*

    a.  The intended WOUs and WCNs could be ingested and decoded.

        **Conclusion:** *[OT&E objective was met]{.underline}*

        **National partners are processing WOUs in their normal business
        operations. These partners realize the benefit of adding marine
        zones to convective watch products and are processing VTEC
        information off line for both the WCN and WOU. Local customers
        in South Carolina and Missouri are using WCNs for guidance
        purposes.**

**6. Lessons Learned**[]{#__RefHeading___Toc93280860 .anchor}

Configuration

1\. [*Good configuration is the foundation for success!*]{.underline}

> This is by far the most important lesson of the OT&E. Having all of
> your templates properly configured and the various applications set to
> the needs of your office will pave the way for successful use of all
> of the applications.

2\. [*Test all of your templates before using
operationally*.]{.underline}

> This goes for WWA, WarnGen and RiverPro. Numerous VTEC and format
> errors were the result of changes to baseline templates. If you change
> the baseline, test your changes out before going operational. If you
> are using older templates, update to the latest version to ensure
> complicity with VTEC.
>
> 3\. *[Templates should never be sent to the field that don\'t match
> NWS policy, nor should they ever contain punctuation or grammatical
> errors.]{.underline}*
>
> When national corrections are needed, they must be made once
> nationally and shared with the field, not made separately 122 times.
> Anytime there is a template change, a baseline template should be
> distributed.

4\. [*WarnGen: Check your templates to ensure they comply with VTEC
standards*.]{.underline}

> You can either create in test mode (set the AWIPS clock back), or
> create and do not send, to make sure the VTEC code is properly
> generated.

5\. [*WWA: Make sure your office is properly configured.*]{.underline}

> A properly configured WWA makes things much easier for the users. If
> the focal point is unsure of how to do this, ask a neighboring WFO.
> Many WFOs across the country have correctly configured their WWA and
> are willing to guide others. MDL is also available to help where
> needed.

6\. *[RiverPro: Make sure your templates are correct and are
segmented.]{.underline}*

> If you are unsure how to do this, the Hydrologic Services Division
> Support Brach can assist your office in accomplishing the task.

Become comfortable with the new VTEC Settings Interface before going
operational.

This will help you make the correct decisions when creating a river
flood product.

7\. *[Any configuration script needs to include purging the
database.]{.underline}*

OT&E Process

> 1\. If a TTR is generated by a TRG member at NWS HQ for a problem
> observed at a WFO, then the local WFO contact should be contacted so
> that multiple trouble tickets are not generated for the same problem.
> This occurred several times here at CHS.
>
> OS was honoring commitment not to contact sites directly to keep the
> regions in the loop. Perhaps during OT&E, regions and service areas
> should establish a protocol for frequent site interaction to enable
> such coordination.
>
> 2\. Continue the practice of keeping all HQ internal activities
> (discussions, decisions, etc) off the TRG calls to help facilitate
> shorter calls. Regions/field representatives should specifically
> inquire status of items that were handled or closed outside of the TRG
> forum if they are interested in the outcome of a specific action item
> or TTR.
>
> 3\. Be sure a developer for each software package (WarnGen, etc) is
> available on each call to ensure that local WFO or HQ questions can be
> answered promptly. [It is good practice to have the developer on hand
> to address issues in real time.]{.underline}
>
> 4\. A fully completed test and/or implementation plan must be in place
> at least three months before the scheduled test/implementation date.
> All participants must fully understand the scope of the test and their
> responsibilities in the test before it begins.
>
> 5\. All necessary software must be sent to the field at least a month
> before the expected implementation date (due to shift schedules,
> leave, weather, etc.). There is no way software should be expected to
> be installed in a matter of days before a major implementation. A
> hurried installation greatly increases the chances of error.
>
> 6\. All instructions must be prepared, tested, tested, and distributed
> to the field long before the implementation date. Configurations
> should be verified and locally tested long before the date, to allow
> for corrections to be made before the date. Errors in the instructions
> destroy confidence in all future instructions coming from the same
> party.
>
> Instructions for turning on vital parts of the test software, such as
> VTEC, should be in place before the test begins.
>
> 7\. To reduce the number of duplicate TTRs generated, email the
> service programs or the sites before opening a TTR. If the site
> already opened a TTR, then the service program manager won't have to
> open one and if the service program manager already opened the TTR,
> then the site won't have to open one (TestTrack Pro may have email
> capabilities).
>
> 8\. Consider the following objective milestone activities for running
> an AWIPS OT&E of this nature:

a.  Software available 3 months before start of OT&E (minimum of 1
    month)

b.  Test plan needs to be available 1 month before the software is
    distributed

c.  A conference call should be held with the sites, not just the
    regions one month before the test starts.

9\. Automate as many instructions as possible (installation,
configuration, etc.)

10\. IOT&E should have a written test plan.

11\. Need accurate estimates of costs for travel

> 12\. More realistic scenarios needed for test cases. Field should be
> able to provide outlines of test scenarios or hazard progressions
> which are based on real field experiences.
>
> 13\. During OT&E, test messages should not be transmitted over NWR or
> via any Emergency Alerting system. Instructions should be provided to
> configure sites to ensure this.

Software Development

> 1\. The method in which our policy makers feed requirements to
> software developers must be improved. There must also be a method for
> developers to check and see if their software is on the right track
> with what policy makers have in mind.
>
> 2\. The software should be tested in both \"live\" and
> \"test/practice\" modes before it is sent to the field. Finding
> differences in them reduces the confidence level in all of them.
>
> 3\. A fully functioning practice mode (mirroring the live mode) must
> be a basic part of any NWS warning software. Without this, test
> products that get erroneously transmitted serve to reduce the
> confidence level for all forecasters as they attempt to be become
> proficient in using the software.
>
> 4\. Service area lead should witness retest to verify closure of
> deficiency reports or trouble reports before deployment to field for
> installation.
>
> 5\. Service area leads need a working proficiency in operation of the
> NWS baseline software pertinent to their service area. This will
> enhance their understanding of the impact of policy changes on
> operations and how implementation affects the execution of the policy.
>
> 6\. A test drive of the software with field site participation during
> development enabled identification and correction of problems early in
> the cycle.

Recommendations

1\. [Practice, Practice, Practice!]{.underline}

> Familiarity with WWA makes warning operations run smoother. This was
> seen during OT&E as the percentage of successful products
> significantly increased by the last 2 weeks of the test. Job sheets
> are available in the OB4 WWA User\'s Guide and the OB3 WWA training
> discusses everything from the basics of WWA to more advanced
> operations.

2\. [Full WWA functionality exists while in practice mode.]{.underline}

> Just be sure to change the mode for each hazard you create
> (Options\--\> Mode\--\> Practice from the Composer window). You can
> note the mode you are in from the top of the composer window. It is
> best to not practice when there are active hazards in the area.
>
> 3\. All WFOs should issue VTEC products with a \"X\" action code for
> the first six months after the February turn key date so that WFOs can
> become familiar with dealing with VTEC in real time.
>
> 4\. [Use the refresh button for the WOU ingester before rebooting WWA
> to clear possible ingest problems]{.underline}.
>
> 5\. Test and practice watches and warnings should not be transmitted
> on NWR unless for a specific drill involving emergency management
> communities. A procedure developed during OT&E disables CRS
> transmissions on a product by product basis, which seems to work well.
>
> 6\. [Cancelled products are dead as far as the partners are
> concerned.]{.underline} Generally no changes are needed to cancel
> products (corrections) unless the change affects the way it will be
> voiced on NWR and a correction will substantively improve cancelled
> product until its retransmit cycle ends.
>
> 7\. [VTEC products affect backup operations, as discovered when an RFC
> was unavailable during the OT&E.]{.underline} The WFO was looking for
> updates on a flood condition, and the RFC was not willing to enter
> backup operations. OCWWS and Hydrologic Services division can discuss
> with regions and help lead the group to a decision.
>
> 8\. Consider having OT&E after all software changes are completed.
> Require development scheduling to be more accurate and no further
> development once OT&E was initiated.

Hydrology

> 1\. Fully functional applications need to be ready which all field
> offices will use on a nation-wide basis for issuance of products.
>
> 2\. We need to improve communication with WFOs and provide more
> specific, advance training to ensure forecasters are familiar with
> hydrologic product policy and formats well ahead of time. Product
> templates need to be ready at WFOs well ahead of time. We need to
> improve communication between regions and HSD to ensure directives are
> being interpreted correctly.
>
> 3\. To ensure there are no more ETN conflicts in products, we need to
> develop a way in VTEC to distinguish between areal flood products and
> point-specific flood products. The quality control functions of
> RiverPro need to be enhanced to prevent the common VTEC errors in
> river flood products before they can occur.
>
> 4\. More thorough test scenarios should be run before the OT&E to
> ensure the applications are ready to handle all possible situations.
> We need to improve the hydrologic product specification directive to
> ensure it reflects all products field offices need to issue.
>
> VTEC
>
> STATISTICS

**ATTACHMENT A -- VTEC Product
Statistics**[]{#__RefHeading___Toc93280861 .anchor}

  -------------- --------- ------- --------- ------- --------- ------- --------- ------- --------- ------- --------- ------- --------- ------- --------- -------
  **VTEC                                                                                                                                                 
  Statistics**                                                                                                                                           
  Does not                                                                                                                                               
  include Hydro,                                                                                                                                         
  WBC or OCONUS                                                                                                                                          

                 Week 1            Week 2            Week 3            Week 4            Week 5            Week 6            Week 7            Entire    
                                                                                                                                               OT&E      

                 pass      total   pass      total   pass      total   pass      total   pass      total   pass      Total   pass      total   pass      total

  **NPW**        17        26      41        59      29 (63%)  46      66        78      31        44      37        41      16        17      237       311
                 (65.3%)           (69.5%)                             (84.6%)           (70.4%)           (90.2%)           (94.1%)           (76.2%)   

  **WSW**        3 (100%)  3       30        36      20        24      9 (52.9%) 17      51 (75%)  68      4 (100%)  4       4 (100%)  4       121       156
                                   (83.3%)           (83.3%)                                                                                   (77.6%)   

  **CFW (no      9 (60%)   15      10        17      1 (20%)   5       8 (61.5%) 13      19        21                                          47        71
  MA.S)**                          (58.8%)                                               (90.4%)                                               (66.2%)   

  **RFW**        8 (47%)   17                        5 (45.4%) 11      12 (75%)  16                        6 (60%)   10                        31        54
                                                                                                                                               (57.4%)   

  **MA.S**       14        16      12        14      1 (100%)  1       11 (100%) 11      6 (75%)   8       5 (62.5%) 8       7 (100%)  7       56        65
                 (87.5%)           (85.7%)                                                                                                     (86.2%)   

  **WWA**        51        77      93        126     56        87      106       135     107       141     52        63      27        28      492       657
                 (66.2%)           (73.8%)           (64.4%)           (78.5%)           (75.9%)           (82.5%)           (96.4%)           (74.9%)   

  **SVR**        18        19      24 (100%) 24      36        37      14 (100%) 14      21        23      11        12      6 (100%)  6       130       135
                 (94.7%)                             (97.2%)                             (91.3%)           (91.6%)                             (96.3%)   

  **TOR**        1 (100%)  1       113       113     57 (100%) 57      1 (50%)   2       35 (100%) 35                                          207       208
                                   (100%)                                                                                                      (99.5%)   

  **SVS**        14        16      137       176     78        97      10        12      63        73      7 (100%)  7       9 (100%)  9       318       390
                 (87.5%)           (77.8%)           (80.4%)           (83.3%)           (86.3%)                                               (81.5%)   

  **SMW**        7 (77.7%) 9       18 (100%) 18      6 (100%)  6       2 (100%)  2       1 (100%)  1       6 (85.7%) 7       2 (100%)  2       42        45
                                                                                                                                               (93.3%)   

  **MWS (MA.W)** 5 (71.4%) 7       8 (80%)   10      5 (100%)  5       1 (50%)   2                                                             19        24
                                                                                                                                               (79.2%)   

  **WarnGen**    45        52      300 (88%) 341     182       202     28        32      120       132     24        26      17 (100%) 17      716       802
                 (86.5%)                             (90.1%)           (87.5%)           (90.9%)           (92.3%)                             (89.3%)   

  **BOI**        4 (66.6%) 6       9 (81.8%) 11      8 (72.7%) 11      7 (58.3%) 12      21        23      5 (100%)  5                         54        68
                                                                                         (91.3%)                                               (79.4%)   

  **CHS**        25        29      144       165     16        17      25        30      67        78      19        22      9 (100%)  9       305       350
                 (86.2%)           (87.3%)           (94.1%)           (83.3%)           (85.9%)           (86.4%)                             (87.1%)   

  **CRP**                                                                                                  17 (85%)  20      2 (100%)  2       19        22
                                                                                                                                               (86.4%)   

  **CYS**        3 (100%)  3       13        17      27        28      8 (53.3%) 15      12        27      1 (100%)  1       6 (85.7%) 7       70        98
                                   (76.4%)           (96.4%)                             (44.4%)                                               (71.4%)   

  **DMX**                          58 (92%)  63      54        56      14 (100%) 14      9 (69.2%) 13      4 (100%)  4                         139       150
                                                     (96.4%)                                                                                   (92.7%)   

  **DTX**        9 (90%)   10      4 (57.1%) 7       1 (20%)   5       5 (62.5%) 8       13 (100%) 13      4 (100%)  4       3 (100%)  3       39 (78%)  50

  **EWX**                                                                                7 (100%)  7       1 (50%)   2       13 (100%) 13      21        22
                                                                                                                                               (95.5%)   

  **FGZ**        3 (100%)  3       1 (100%)  1       5 (41.6%) 12      18 (90%)  20      8 (80%)   10      8 (100%)  8                         43        54
                                                                                                                                               (79.6%)   

  **JAX**        21        34      111       140                       11        17      44 (88%)  50      8 (100%)  8                         195       249
                 (61.8%)           (79.3%)                             (64.7%)                                                                 (78.3%)   

  **PHI**                          1 (100%)  1       4 (50%)   8                         11 (100%) 11      1 (50%)   2                         17        22
                                                                                                                                               (77.3%)   

  **PSR**        16        18      10 (100%) 10      14        18      24        26      3 (60%)   5                         2 (100%)  2       69        79
                 (88.8%)                             (77.8%)           (92.3%)                                                                 (87.3%)   

  **RIW**        3 (100%)  3       6 (66.6%) 9       12        19      2 (100%)  2       18        19                        2 (100%)  2       43        54
                                                     (63.1%)                             (94.7%)                                               (79.6%)   

  **SGX**        2 (40%)   5       4 (44.4%) 9       5 (71.4%) 7       11        13      10        12      7 (87.5%) 8       7 (100%)  7       46        61
                                                                       (84.6%)           (83.3%)                                               (75.4%)   

  **TAE**        3 (75%)   4                         81 (89%)  91      1 (100%)  1                                                             85        96
                                                                                                                                               (88.5%)   

  **VEF**        7 (50%)   14      32        34      11        17      8 (88.9%) 9       4 (80%)   5       1 (20%)   5                         63 (75%)  84
                                   (94.1%)           (64.7%)                                                                                             
  -------------- --------- ------- --------- ------- --------- ------- --------- ------- --------- ------- --------- ------- --------- ------- --------- -------

**ATTACHMENT B -- VTEC PRODUCT STATISTICS by SERVICE
AREA**[]{#__RefHeading___Toc93280862 .anchor}

+---------------+---------------+---------------+--------------------+
| **Service     | **Total**     | **Passed**    | **Failed**         |
| area**        |               |               |                    |
|               | **Evaluated** | **\# - (%)**  | **\# - (%)**       |
+---------------+---------------+---------------+--------------------+
| **Hydro**     | **745**       | **437**       | **308**            |
|               |               |               |                    |
|               |               | **(59%)**     | **(41%)**          |
+---------------+---------------+---------------+--------------------+
| **Marine**    | **199**       | **164**       | **35**             |
|               |               |               |                    |
|               |               | **(82%)**     | **(18%)**          |
+---------------+---------------+---------------+--------------------+
| **Severe**    | **733**       | **657**       | **76**             |
|               |               |               |                    |
|               |               | **(90%)**     | **(10%)**          |
+---------------+---------------+---------------+--------------------+
| **Fire**      | **54**        | **31**        | **22**             |
|               |               |               |                    |
|               |               | **(57%)**     | **(43%)**          |
+---------------+---------------+---------------+--------------------+
| **Public**    | **469**       | **358**       | **111**            |
|               |               |               |                    |
|               |               | **(76%)**     | **(24%)**          |
+---------------+---------------+---------------+--------------------+
| **TOTALS**    | **2200**      | **1647**      | **553**            |
|               |               |               |                    |
|               |               | **(75%)**     | **(25%)**          |
+---------------+---------------+---------------+--------------------+

**ATTACHMENT C -- VTEC STATISTICS for
WARNGEN**[]{#__RefHeading___Toc93280863 .anchor}

![](media/image3.wmf){width="6.686805555555556in"
height="5.0368055555555555in"}

In general, the pass rate improved as the test progressed. Of particular
note, the SVS made a sharp jump after a software fix was implemented to
correct the VTEC end time problem on cancellations. The 2 50% pass rates
were due to the limited sample size. That week there were only 2 TOR and
MWS follow-ups issued.

**ATTACHMENT D -- VTEC STATISTICS for
WWA**[]{#__RefHeading___Toc93280864 .anchor}

![](media/image4.wmf){width="6.502083333333333in"
height="4.8902777777777775in"}

For the WWA-issued products, the trend was not quite as distinctive, but
still in a generally positive direction. Of particular note were the NPW
and WSW products which were the majority of the data set. The notable
dips were due in large part to scenarios being run in test mode where
the upgrade/downgrade functionality was not working. Once this problem
was fixed, the pass rates soared. The exception to this was fire weather
products. There continues to be issues with maps and the way WWA handles
CWAs where the public and fire weather areas differ.

MA.S products were actually not to be included in the record keeping,
but in general were handled well. But their use brought up policy
questions of how to handle High Surf advisories during Tropical events.

**ATTACHMENT E -- TEST TROUBLE REPORT VTEC
SUMMARY**[]{#__RefHeading___Toc93280865 .anchor}

![](media/image5.wmf){width="6.375in" height="3.9506944444444443in"}

  ----------------- ------------- ------------- -------------- --------------
  ** **             **WWA**       **WARNGEN**   **RIVERPRO**   **TOTAL**

  **Routine**       **9**         **4**         **0**          **13**

  **Enhancement**   **6**         **1**         **0**          **7**

  **Watch**         **19**        **6**         **2**          **27**

  **Urgent**        **28**        **14**        **9**          **51**

  **Critical**      **23**        **10**        **13**         **46**

  **No Priority**   **2**         **0**         **0**          **2**

  **TOTALS**        **87**        **35**        **24**         **146**
  ----------------- ------------- ------------- -------------- --------------

WBC

STATISTICS

**ATTACHMENT F -- WBC STATISTICS**[]{#__RefHeading___Toc93280866
.anchor}

![](media/image6.wmf){width="6.75in" height="3.7625in"}

\*Does not count products that were attempted but were not issued due to
software issues

**ATTACHMENT G -- TEST TROUBLE REPORT WBC
SUMMARY**[]{#__RefHeading___Toc93280867 .anchor}

![](media/image7.wmf){width="6.375in" height="2.9833333333333334in"}

  -------------------------------------- --------------------------------
                                         **WWA**

  **Routine**                            3

  **Enhancement**                        0

  **Watch**                              1

  **Urgent**                             10

  **Critical**                           23

  **No Priority**                        1

  **TOTALS**                             38
  -------------------------------------- --------------------------------

Attachment H

GHG

Commentary

**ATTACHMENT H -- GHG COMMENTARY**[]{#__RefHeading___Toc93280868
.anchor}

WBC

During OT&E, another software application, GHG was used to issue WBC
products from a WFO. An informal assessment of the performance follows:

1.  *[Demonstrate that WBC products (WOU and WCN) are accurately
    generated in an operational environment under real world weather
    situations and scenarios.]{.underline}*

    a.  WBC products conform to the specification defined in NWSI
        10-511, 10-512, and 10-1703

        **Conclusion:**

        **Products generated by GHG conformed to NWSI specifications.
        GHG demonstrated the capability of creating more readable text
        phrases for watch cancellations and replacements.**

    b.  WBC products can be accurately generated when WFOs and/or the
        SPC are/is in service backup

        i.  WFOs can provide effective service backup for convective
            watches using AWIPS software.

            **Conclusion:**

            **GHG demonstrated service backup capability in multiple
            static tests.**

        ii. Air Force Global Weather Center can provide effective
            service backup for SPC in convective watch situations using
            Watch By County products.

            **Conclusion:**

            **A test with the Air Force Global Weather Center was not
            conducted with the GHG test site.**

```{=html}
<!-- -->
```
2.  [*Demonstrate the WBC concept of operations is viable*.]{.underline}

    a.  Forecasters at both the WFO and SPC understand how to generate
        WBC products in terms of segmentation and ordering

        **Conclusion:**

        **Not evaluated for GHG.**

        i.  SPC can cancel the entire convective watch area using a WOU

> **Conclusion:**
>
> **Successfully evaluated.**

b.  Forecasters understand how to generate WBC action codes for
    real-time service operations.

    i.  WFOs can update, cancel, and extend in time watches using WCNs
        for their geographic area of responsibility. WCNs contain proper
        Universal Geographic Codes, Valid Time Event Code (VTEC), and
        list of counties, independent cities, and marine zones for each
        segment of the product.

        **Conclusion:**

        **GHG successfully created and disseminated WCN for single watch
        situations in both test and experimental modes in two field
        tests.**

    ii. WFOs can combine more than one active convective watch into a
        WCN product.

        **Conclusion:**

        **GHG created and disseminated several multiple watch WCNs
        successfully during a field test on October 27.**

    iii. Affected WFOs can "clear" all counties and independent cities
         from the convective watch for their geographic area of
         responsibility using the WCN.

> **Conclusion:**

**GHG successfully accomplished this test objective in two field tests
with the Storm Prediction Center (September 15 and October 27).**

c.  The AWIPS software is usable for reliable issuance of watch products
    and system performance is not degraded during hazardous weather or
    service backup operations.

    **Conclusion:**

    **Not evaluated for GHG.**

```{=html}
<!-- -->
```
d.  Implementation of WBC causes no significant increase in forecaster
    workload.

    **Conclusion:**

    **Not evaluated for GHG.**

```{=html}
<!-- -->
```
3.  *[Demonstrate adequate training was provided to meet the objectives
    defined for the WBC program.]{.underline}*

```{=html}
<!-- -->
```
a.  Training addressed the key components necessary to successfully
    issue WBC products.

    **Conclusion:**

    **Not evaluated for GHG.**

```{=html}
<!-- -->
```
4.  *[Demonstrate that quality control functionality for WOU and WCN
    products is implemented effectively.]{.underline}*

```{=html}
<!-- -->
```
a.  Products are standardized in accordance with the requirements of the
    WBC product and operational specifications.

    **Conclusion:**

    **In limited testing, GHG-created WCNs have not contained erroneous
    information.**

```{=html}
<!-- -->
```
5.  *[Demonstrate WBC products can be read and processed by NWS
    customers and partners.]{.underline}*

```{=html}
<!-- -->
```
a.  The intended WOUs and WCNs could be ingested and decoded.

    **Conclusion:**

    **Successfully processed.**

    Attachment I

    TTR Analysis

    RiverPro

    WarnGen

    WWA -- VTEC

    WWA - WBC

  ------------------------------------------------------------------------------------------------------------------------
  **TTR   **Same   **Action   **Program**   **Application**   **Service   **Problem         **Root cause**  **Priority**
  \#**    as TTR   Item**                                     Area**      description**                     
          #?**                                                                                              
  ------- -------- ---------- ------------- ----------------- ----------- ----------------- --------------- --------------
  124     168                 Both          All               Public      Headline for      Template        PE - Potential
                                                                          Expiring NPWs                     Enhancement
                                                                          incorrect                         

  346                         VTEC          RiverPro          Hydro       FLW action code                   C - Critical
                                                                          was CON instead                   
                                                                          of NEW                            

  97                          VTEC          RiverPro          Hydro       River Flood       SW              C - Critical
                                                                          Warning VTEC                      
                                                                          Corrupt on                        
                                                                          Followup                          

  104                         VTEC          RiverPro          Hydro       Incorrect format  Template        C - Critical
                                                                          for Flood                         
                                                                          Statement                         
                                                                          (forecast point)                  

  114     122                 VTEC          RiverPro          Hydro       FLW cancellation  SW              C - Critical
                                                                          error - FLS VTEC  Configuration   
                                                                          Resets to \"NEW\"                 

  122     114                 VTEC          RiverPro          Hydro                         SW              C - Critical

  131                         VTEC          RiverPro          Hydro       Incorrect format  Template        C - Critical
                                                                          for flood warning                 
                                                                          (small errors)                    

  163                         VTEC          RiverPro          Hydro       Same ETN in       SW              C - Critical
                                                                          segments for two                  
                                                                          separate FLW                      
                                                                          products                          

  202     250      68         VTEC          RiverPro          Hydro       Bad ETN                           C - Critical

  212              71         VTEC          RiverPro          Hydro       RiverPro produced                 C - Critical
                                                                          wrong Crest time                  
                                                                          in HVTEC line of                  
                                                                          FLS                               

  230              91         VTEC          RiverPro          Hydro       Downgrade from                    C - Critical
                                                                          Warning to                        
                                                                          Advisory -- Event                 
                                                                          begin time                        
                                                                          incorrect                         

  231              92,93      VTEC          RiverPro          Hydro       ETN not assigned                  C - Critical
                                                                          correctly.                        

  239              98         VTEC          RiverPro          Hydro       Date and time                     C - Critical
                                                                          data on P and H                   
                                                                          VTEC lines all                    
                                                                          zeros.                            

  240              98         VTEC          RiverPro          Hydro       Ending time after                 C - Critical
                                                                          beginning time on                 
                                                                          P VTEC line.                      

  245              98         VTEC          RiverPro          Hydro       Cancelled product                 C - Critical
                                                                          showed up as NEW                  
                                                                          in VTEC line.                     

  254              108,79     VTEC          RiverPro          Hydro       Change Riverpro                   C - Critical
                                                                          product                           
                                                                          recommendation                    
                                                                          method.                           

  171                         VTEC          RiverPro          Hydro       NEW segments in   SW              U - Urgent
                                                                          Flood statement                   
                                                                          from TLH                          

  175                         VTEC          RiverPro          Hydro       Two different     SW              U - Urgent
                                                                          ETNs and                          
                                                                          Significant Codes                 
                                                                          for Same Fcst                     
                                                                          Point                             

  187                         VTEC          RiverPro          Hydro       Product class     Configuration   U - Urgent
                                                                          incorrect                         

  207                         VTEC          RiverPro          Hydro       \"O\" Instead of                  U - Urgent
                                                                          \"X\" in RiverPro                 

  241              98         VTEC          RiverPro          Hydro       Two P VTEC lines                  U - Urgent
                                                                          with only One H                   
                                                                          VTEC line.                        

  244              98         VTEC          RiverPro          Hydro       VTEC line does                    U - Urgent
                                                                          not agree with                    
                                                                          text.                             

  248              100        VTEC          RiverPro          Hydro       Failure Sending                   U - Urgent
                                                                          Info to Backup                    
                                                                          Sites in RiverPro                 

  249              101        VTEC          RiverPro          Hydro       Action code of                    U - Urgent
                                                                          CON for a                         
                                                                          cancelled                         
                                                                          product.                          

  259              117        VTEC          RiverPro          Hydro       Missing                           U - Urgent
                                                                          Significance code                 

  105                         VTEC          RiverPro          Hydro       HVTEC string has  SW              W - Watch Item
                                                                          incorrect Flood                   
                                                                          Begin Date/Time                   
  ------------------------------------------------------------------------------------------------------------------------

  -------------------------------------------------------------------------------------------------------------------------
  **TTR   **Same   **Action   **Program**   **Application**   **Service   **Problem          **Root cause**  **Priority**
  \#**    as TTR   Item**                                     Area**      description**                      
          #?**                                                                                               
  ------- -------- ---------- ------------- ----------------- ----------- ------------------ --------------- --------------
  147                         VTEC          WARNGEN           Hydro       TEST Warning and   Operator error  C - Critical
                                                                          Follow Up                          
                                                                          Statements in                      
                                                                          \"X\" VTEC mode                    

  150                         VTEC          WARNGEN           Hydro       Missing VTEC in    Template        C - Critical
                                                                          FFS from CHS                       

  198              52, 53,66  VTEC          WARNGEN           Hydro       MND Error for Test Template        C - Critical
                                                                          Dam Break Warning                  

  210              70         VTEC          WARNGEN           Hydro       .X in H-VTEC code  Configuration   C - Critical

  96                          VTEC          WARNGEN           Hydro       Incorrect format   Template        U - Urgent
                                                                          for FLS - advisory                 
                                                                          from WarnGen                       

  117                         VTEC          WARNGEN           Hydro       Incorrect Format   Template        U - Urgent
                                                                          for FFA from                       
                                                                          WarnGen                            

  123                         VTEC          WARNGEN           Hydro       Missing H-VTEC     Operator error  U - Urgent
                                                                          string in FFW                      

  129     178                 VTEC          WARNGEN           Hydro       Unable to COR      SW              U - Urgent
                                                                          WarnGen Statements                 

  174                         VTEC          WARNGEN           Hydro       Wording error in   Template        U - Urgent
                                                                          FFS headline from                  
                                                                          CHS                                

  184                         VTEC          WARNGEN           Hydro       Flood Severity in  Configuration   U - Urgent
                                                                          HVTEC line                         
                                                                          incorrect.                         

  320                         VTEC          WARNGEN           Hydro       EXT not available  Operator error  W - Watch Item
                                                                          in WARNGEN                         

  142                         VTEC          WARNGEN           Hydro       FFW SECOND LINE OF Operator error  
                                                                          VTEC CODE ALL                      
                                                                          ZEROS. FOLLOWUP                    
                                                                          FLS HAD NO VTEC                    
                                                                          CODES.                             

  162     115                 VTEC          WARNGEN           Marine      MWSCHS, KCHS       SW              C - Critical
                                                                          302329                             

  164     115                 VTEC          WARNGEN           Marine      MWSCHS, KCHS       SW              C - Critical
                                                                          062356                             

  173     115                 VTEC          WARNGEN           Marine      MWSDTX, KDTX       SW              C - Critical
                                                                          271811                             

  327                         VTEC          WARNGEN           Marine      MWS Headlines did                  U - Urgent
                                                                          not Match Scenario                 
                                                                          Examples                           

  107                         VTEC          WARNGEN           Marine      Warngen VTEC Codes Operator error  
                                                                          for Follow-Up                      

  121                         VTEC          WARNGEN           Marine                         Operator error  

  124     168                 Both          All               Public      Headline for       Template        PE - Potential
                                                                          Expiring NPWs                      Enhancement
                                                                          incorrect                          

  205              73         VTEC          WARNGEN           Public      Warngen QC error   Configuration   U - Urgent
                                                                          on Segmented SVS                   

  115                         VTEC          WARNGEN           Public      Incorrect VTEC     SW              C - Critical
                                                              Marine      Event Ending Time                  
                                                              Hydro       with CAN in                        
                                                                          WarnGen                            

  251              105        VTEC          WARNGEN                       No H VTEC line.    Template        C - Critical

  317                         VTEC          WARNGEN                       Warngen Problem    Operator error  R - Routine

  289              145        VTEC          WARNGEN           Marine      MWSJAX, KJAX       SW              W - Watch Item
                                                                          242255                             

  328              174        VTEC          WARNGEN           Marine      Warngen Crash      SW              C- Critical
                                                                          after MWS issued                   

  133                         VTEC          WARNGEN           Hydro       Incorrect format   Operator error  \-
                                                                          for areal FLW from                 
                                                                          JAX using WarnGen                  

  311              159        VTEC          WARNGEN           Hydro       Flash Flood        Operator error  C - Critical
                                                                          Warning correction                 
                                                                          and followup                       
                                                                          problem                            

  256              116        VTEC          WARNGEN           Hydro       Correction showed                  U - Urgent
                                                                          up with a NEW                      
                                                                          Action code.                       

  125                         VTEC          WARNGEN           Hydro       ETN not            SW - Policy     W - Watch Item
                                                                          incrementing in                    
                                                                          FLS- advisory                      
                                                                          product from DTX                   

  246              99         VTEC          WARNGEN           Hydro       Advisory                           W - Watch Item
                                                                          significance shown                 
                                                                          as S instead of Y.                 

  308                         VTEC          WARNGEN           Hydro       VTEC WarnGen       Operator error  W - Watch Item
                                                                          Creation of COR to                 
                                                                          original warning                   
                                                                          Fail                               

  281              139        VTEC          WARNGEN           Hydro       no VTEC string in                  C - Critical
                                                              Public      followup messages                  
                                                                          using WarnGen                      

  169                         VTEC          WARNGEN           Marine      SMWJAX, KJAX       Configuration   C - Critical
                                                                          292028                             

  326                         VTEC          WARNGEN           Marine      SMW Correction -   SW              U - Urgent
                                                                          Event Beg                          
                                                                          Date/Time updating                 
                                                                          with Current Time                  

  178     34, 129             VTEC          WARNGEN           Marine      MWSCHS, KCHS 10243 SW              W - Watch Item

  285     297      147        VTEC          WARNGEN           Public      issuance times     SW              C - Critical
                                                                          advances with CORs                 
                                                                          in WarnGen                         

  253              107        VTEC          WARNGEN           Public      UGCs manually                      R - Routine
                                                                          edited out of SVS                  

  232                         VTEC          WARNGEN           Public      Incorrect EXP                      W - Watch Item
                                                                          and/or CAN with                    
                                                                          ending SVS\'s                      
                                                                          using WARNGEN                      

  153     92                  VTEC          WARNGEN           Severe                         Operator error  W - Watch Item

                                                                                                             
  -------------------------------------------------------------------------------------------------------------------------

  -----------------------------------------------------------------------------------------------------------------------------
  **TTR   **Same as **Action   **Program**   **Application**   **Service   **Problem          **Root cause**     **Priority**
  \#**    TTR #?**  Item**                                     Area**      description**                         
  ------- --------- ---------- ------------- ----------------- ----------- ------------------ ------------------ --------------
  102                          WBC           WWA               Severe      Add MND header     SW                 C - Critical
                                                                           tags to WWA                           
                                                                           generated products                    

  108     113, 118,            WBC           WWA               Severe      WOU Ingester Crash SW                 C - Critical
          119, 120                                                         WW 802                                

  113                          WBC           WWA               Severe                         SW                 C - Critical

  118     116                  WBC           WWA               Severe      Wrong coding for   SW                 C - Critical
                                                                           CFW                                   

  119                          WBC           WWA               Severe                         SW                 C - Critical

  120                          WBC           WWA               Severe                         SW                 C - Critical

  140                          WBC           WWA               Severe      WBC WCN product    SW                 C - Critical
                                                                           would not format                      

  148     214, 220             WBC           WWA               Severe      Getting the error  SW                 C - Critical
                                                                           box \"Can\'t mix                      
                                                                           text /                                
                                                                           experimental                          
                                                                           hazards with                          
                                                                           operational                           
                                                                           hazards\" when the                    
                                                                           VTEC product class                    
                                                                           is not mixed.                         

  149                          WBC           WWA               Severe      While in test mode SW                 C - Critical
                                                                           the automated WCN                     
                                                                           features do not                       
                                                                           work.                                 

  213     203                  WBC           WWA               Severe      WOU Ingester       SW                 C - Critical
                                                                           Failure                               

  214     148                  WBC           WWA               Severe      Multiple watch WCN SW                 C - Critical

  219     203                  WBC           WWA               Severe      WOU Ingester GUI   SW                 C - Critical
                                                                           would not open for                    
                                                                           TOR Watch 835                         

  233     257, 277, 56         WBC           WWA               Severe      ETN Problem with   SW                 C - Critical
          243, 315                                                         Test Svr Tstorm                       
                                                                           Watch 9999                            

  242     203                  WBC           WWA               Severe      WOU Ingester crash SW                 C - Critical
                                                                           (2 WOUs in 1                          
                                                                           minute)                               

  243     233, 243,            WBC           WWA               Severe      Final WCN          SW                 C - Critical
          257, 258                                                         Cancellation                          
                                                                           Product                               

  257     233, 243,            WBC           WWA               Severe      Final WCN          SW                 C - Critical
          257, 258                                                         Cancellation                          
                                                                           Product                               

  258     233, 243,            WBC           WWA               Severe      Final WCN          SW                 C - Critical
          257, 258                                                         Cancellation                          
                                                                           Product                               

  284               86         WBC           WWA               Severe      WOU ingestor       SW                 C - Critical
                                                                           failure                               

  314               89         WBC           WWA               Severe      WCNs from          SW                 C - Critical
                                                                           surrounding                           
                                                                           offices in WWA                        
                                                                           GeoViewer                             

  340                          WBC           WWA               ?           Problem with ATAN                     C - Critical
                                                                           653 and OB 4                          
                                                                           installation at a                     
                                                                           WBC (not VTEC)                        
                                                                           test site                             

  203     206, 204,            WBC           WWA               Severe      WOU Ingester                          C - Critical
          242, 213,                                                        Failure                               
          282, 283                                                                                               

  267                          WBC           WWA               Severe      WOU Ingestor       SW/Documentation   R - Routine
                                                                           Crashed with                          
                                                                           Simulated WOU                         

  342                          WBC           WWA               Severe      \"Experimental\"                      R - Routine
                                                                           in Header                             

  188                          WBC           WWA               Severe      Commas observed in Map                U - Urgent
                                                                           Marine Products                       
                                                                           produced by                           
                                                                           WarnGen and WWA                       

  291               87         WBC           WWA               Severe      WCN didn\'t        Operator error     U - Urgent
                                                                           generate                              
                                                                           automatically                         

  182     180                  WBC           WWA               Severe                         SW                 U - Urgent

  183     180                  WBC           WWA               Severe                         SW                 U - Urgent

  247                          WBC           WWA               Severe      WOU ingester crash SW/Documentation   U - Urgent

  116                          WBC           WWA               Severe      Test WCNs - Mass   Template           
                                                                           News Dissemination                    
                                                                           Header Incorrect                      

  226                          WBC           WWA               Severe      Marine Zone not in SW                 C - Critical
                                                                           cleared WCN                           

  343     344, 345  106        WBC           WWA               Severe      Failure to ingest                     C - Critical
                                                                           WOU w/proposed                        
                                                                           showing                               

  208               37         WBC           WWA               Marine      Marine Zone Name   Shapefile          U - Urgent
                                                                           Incorrect in WOU                      

  227                          WBC           WWA               Severe      ILM did not issue  SW                 U - Urgent
                                                                           final WCN; 15                         
                                                                           minute expiration                     

  211                          WBC           WWA               Marine      ILM Coastal Waters SW                 W - Watch Item
                                                                           segments out of                       
                                                                           order                                 

  286                          WBC           WWA               Severe      ILM Unable to                         R - Routine
                                                                           update WCN from                       
                                                                           Watch 844 to 845                      

  276     287, 295, 72         WBC           WWA               Severe      Unable to issue    SW                 C - Critical
          313                                                              correct WCN upon                      
                                                                           return from                           
                                                                           Service Backup                        

  278               74         WBC           WWA               Severe      WWA failed to      SW                 C - Critical
                                                                           segment replaced                      
                                                                           counties                              

  279               75         WBC           WWA               Severe      WOU ingest problem SW                 C - Critical

  294               88         WBC           WWA               Severe      Three problems:    SW                 C - Critical
                                                                           WOU, WCN, Watch                       

  300     302       90         WBC           WWA               Severe      WOU ingest problem SW                 C - Critical

  318                          WBC           WWA               Severe      Independent cities SW                 C - Critical
                                                                           missing from WCN                      

  296               89         WBC           WWA               Severe      WWA not Refreshing SW                 U - Urgent

  180     182,183              WBC           WWA               Severe      Poor wording in    Template           U - Urgent
                                                                           WCN product                           

  312               87         WBC           WWA               Severe      Test Tornado Watch                    U - Urgent
                                                                           in D2D                                

  303                          WBC           WWA               Severe      WWA Viewer did not                    W - Watch Item
                                                                           allow me to look                      
                                                                           at backup sites                       
                                                                           Tornado Watch                         
  -----------------------------------------------------------------------------------------------------------------------------

  --------------------------------------------------------------------------------------------------------------------------
  **TTR   **Same as **Action   **Program**   **Application**   **Service   **Problem          **Root cause**  **Priority**
  \#**    TTR #?**  Item**                                     Area**      description**                      
  ------- --------- ---------- ------------- ----------------- ----------- ------------------ --------------- --------------
  319                          VTEC          WWA               ?           CAN errors w/BOI   SW              C - Critical
                                                                           FG.Y                               

  111     109                  VTEC          WWA               All                            SW              C - Critical

  332               179, 180   VTEC          WWA               All         Attribute line                     PE - Potential
                                                                           needed with                        Enhancement
                                                                           Upgrades                           

  93                           VTEC          WWA               All         Unable to Delete   SW              PE - Potential
                                                                           Proposed Products                  Enhancement
                                                                           in WWA                             

  304               152        VTEC          WWA               All         WWA Backup for                     U - Urgent
                                                                           Brownsville not                    
                                                                           available                          

  310     89        158        VTEC          WWA               All         EXT coded as CON   SW              U - Urgent

  309                          VTEC          WWA               All         Multi-code and     Operator error  W - Watch Item
                                                                           segment COR issues                 

  221                          VTEC          WWA               All         VTEC Code                          W - Watch Item
                                                                           Appearing in CAFE                  
                                                                           Products                           

  100     103                  VTEC          WWA               Fire        WWA problem with   Configuration   C - Critical
                                                               Weather     issuing RFWs                       

  99                           VTEC          WWA               Fire        WWA advancing ETN  SW              C - Critical
                                                               Weather     on CON/EXA                         

  197               47,48      VTEC          WWA               Fire        VEF FWAW test      SW              C - Critical
                                                               Weather     problems                           

  338                          VTEC          WWA               Fire        ETN rollback on                    C - Critical
                                                               Weather     operational RFW                    

  98                           VTEC          WWA               Fire        Problem with WWA   SW              R - Routine
                                                               Weather     issuing Red Flag                   
                                                                           Warning                            

  191     190                  VTEC          WWA               Fire                           Template        U - Urgent
                                                               Weather                                        

  339     341       190        VTEC          WWA               Fire        Red Flag would not                 W - Watch item
                                                               Weather     EXP                                

  127     126                  VTEC          WWA               Hydro                          SW              \-

  94                           VTEC          WWA               Hydro       Incorrect format   Template        C - Critical
                                                                           for FFA products                   

  95                           VTEC          WWA               Hydro       Incorrect format   Template        C - Critical
                                                                           for FLS - advisory                 
                                                                           from WWA                           

  141     126                  VTEC          WWA               Hydro                          SW              C - Critical

  143                          VTEC          WWA               Hydro       WWA generated FFA  Template        C - Critical
                                                                           has incorrect                      
                                                                           Template                           

  145                          VTEC          WWA               Hydro                          SW              C - Critical

  146                          VTEC          WWA               Hydro                          SW Template     C - Critical

  177                          VTEC          WWA               Hydro       Unable to select   SW              C - Critical
                                                                           zones for WWA                      
                                                                           products                           

  200               67         VTEC          WWA               Hydro       Headline states    SW - Policy     C - Critical
                                                                           termination while                  
                                                                           VTEC action states                 
                                                                           CON.                               

  234                          VTEC          WWA               Hydro       Possible MND                       PE - Potential
                                                                           Enhancement                        Enhancement
                                                                           Request for                        
                                                                           CFW/RFW products                   

  333               181        VTEC          WWA               Hydro       ETN Error in FLS                   R - Routine
                                                                           Advisory Product                   

  347                          VTEC          WWA               Hydro       Can\'t Cancel a                    R - Routine
                                                                           Flash Flood Watch                  
                                                                           in WWA                             

  201                          VTEC          WWA               Hydro       Cancelled product                  R - Routine
                                                                           twice.                             

  126     127, 141             VTEC          WWA               Hydro       No second VTEC     SW              U - Urgent
                                                                           line in FFA                        
                                                                           created thru WWA                   

  196                          VTEC          WWA               Hydro       VTEC office ID     Configuration   U - Urgent
                                                                           does not match                     
                                                                           either WMO or                      
                                                                           AWIPS ID headers                   

  199               63         VTEC          WWA               Hydro                          SW Training     U - Urgent

  261               120,121    VTEC          WWA               Hydro       The word TEST in                   U - Urgent
                                                                           VTEC string.                       

  262               122        VTEC          WWA               Hydro       Word CORRECTED in                  U - Urgent
                                                                           H VTEC line.                       

  263               123        VTEC          WWA               Hydro       County names                       U - Urgent
                                                                           instead of zone                    
                                                                           names.                             

  264     265       124        VTEC          WWA               Hydro       Did not segment                    U - Urgent
                                                                           extension in area.                 

  266               126        VTEC          WWA               Hydro       Cancel failed in                   U - Urgent
                                                                           FFA test.                          

  186                          VTEC          WWA               Hydro       AWIPS ID and VTEC  SW              W - Watch Item
                                                                           phenomena code not                 
                                                                           correlated                         

  152                          VTEC          WWA               Hydro       Erroneous VTEC     Operator error  W - Watch Item
                                                                           action code in FLS                 
                                                                           from JAX                           

  154                          VTEC          WWA               Hydro                          Operator error  W - Watch Item

  92      153                  VTEC          WWA               Hydro       VTEC strings cause SW              C
                                                               Public      NWR CAFE errors                    

  128                          VTEC          WWA               Mari        Informational      SW              \-
                                                                           (Non-Severe)                       
                                                                           Marine Weather                     
                                                                           Statement                          

  168     124                  VTEC          WWA               Marine                         \-              \-

  176                          VTEC          WWA               Marine      Short Term         SW              \-
                                                                           Forecast Expansion                 
                                                                           in area                            

  155                          VTEC          WWA               Marine      CFWJAX, KJAX       SW              C - Critical
                                                                           022139                             

  156     307,252              VTEC          WWA               Marine      ETN increments on  SW              C - Critical
                                                                           CAN                                

  157                          VTEC          WWA               Marine      CFWJAX, KJAX       Operator error  C - Critical
                                                                           040907                             

  159                          VTEC          WWA               Marine      MWSJAX, KJAX       SW              C - Critical
                                                                           021833                             

  165                          VTEC          WWA               Marine      CFWCHS, KCHS       SW              C - Critical
                                                                           030959                             

  166     165                  VTEC          WWA               Marine      CFWCHS, KCHS       SW              C - Critical
                                                                           042034                             

  167     ?                    VTEC          WWA               Marine      CFWCHS, KCHS       SW              C - Critical
                                                                           062031                             

  172                          VTEC          WWA               Marine      MWSCHS, KCHS       SW              C - Critical
                                                                           071133                             

  272               132        VTEC          WWA               Marine      CFWDTX, KDTX                       C - Critical
                                                                           231725                             

  273     274, 275  133        VTEC          WWA               Marine      CFWDTX, KDTX                       C - Critical
                                                                           231734                             

  110                          VTEC          WWA               Marine      High Surf Adv      SW              PE - Potential
                                                                           ProdExp freezes                    Enhancement
                                                                           aft 100 hours                      

  170                          VTEC          WWA               Marine      MWSCHS, KCHS       Configuration   U - Urgent
                                                                           310130                             

  161                          VTEC          WWA               Marine      CFWSGX, KSGX       SW              U - Urgent
                                                                           051700                             

  158                          VTEC          WWA               Marine      CFWJAX, KJAX       Template        U - Urgent
                                                                           070857                             

  217                          VTEC          WWA               Marine      Missing \"TEST\"                   U - Urgent
                                                                           wording in MND for                 
                                                                           Lakeshore Flood                    
                                                                           CFWs and other WWA                 
                                                                           Products                           

  292               148        VTEC          WWA               Marine      CFWCHS, KCHS                       U - Urgent
                                                                           271825                             

  316               160        VTEC          WWA               Marine      MWSCHS, KCHS       SW              U - Urgent
                                                                           281438                             

  179                          VTEC          WWA               Marine                         Operator error  W - Watch Item

  216               74         VTEC          WWA               Marine      Lakeshore Flood                    W - Watch Item
                                                                           Headline non                       
                                                                           conformance                        

  218               74         VTEC          WWA               Marine      Product name in                    W - Watch Item
                                                                           the MND for                        
                                                                           Lakeshore Flood                    
                                                                           when both watch                    
                                                                           and warning are in                 
                                                                           effect                             

  223                          VTEC          WWA               Marine      CFWDTX, KDTX                       W - Watch Item
                                                                           171741                             

  224     111, 222             VTEC          WWA               Marine      CFWDTX, KDTX                       W - Watch Item
                                                                           171747                             

  225     111, 222             VTEC          WWA               Marine      CFWDTX, KDTX                       W - Watch Item
                                                                           171752                             

  238                          VTEC          WWA               Marine      MWSCHS, KCHS                       W - Watch Item
                                                                           191914                             

  271               130        VTEC          WWA               Marine      CFWJAX, KJAX                       W - Watch Item
                                                                           030841                             

  288                          VTEC          WWA               Marine      CFWJAX, KJAX                       W - Watch Item
                                                                           231925                             

  293                          VTEC          WWA               Marine      SMWTLH, KTAE                       W - Watch Item
                                                                           271500                             

  322                          VTEC          WWA               Marine      CFWSGX, KSGX                       W - Watch Item
                                                                           012357                             

  323                          VTEC          WWA               Marine      CFWSGX, KSGX                       W - Watch Item
                                                                           020001                             

  324                          VTEC          WWA               Marine      MWSCHS, KCHS                       W - Watch Item
                                                                           040040                             

  109     111, 130,            VTEC          WWA               Public      Upgrade in test    Template        C - Critical
          132, 222,                                                        mode                               
          223, 224,                                                                                           
          225                                                                                                 

  181     137                  VTEC          WWA               Public                         SW              C - Critical

  189     111                  VTEC          WWA               Public                         SW              C - Critical

  144     89                   VTEC          WWA               Public      Clearing zones     SW              C - Critical
                                                                           early                              

  185                          VTEC          WWA               Public                         SW              C - Critical

  192                          VTEC          WWA               Public                         SW              C - Critical

  229               89, 90     VTEC          WWA               Public      Two VTEC Lines for                 C - Critical
                                                                           One Action                         

  235               95         VTEC          WWA               Public      Wind Advisory                      C - Critical

  280               138        VTEC          WWA               Public      Followup a                         C - Critical
                                                                           previously                         
                                                                           cancelled hazard                   

  301               151        VTEC          WWA               Public      Bad VTEC line                      C - Critical

  306                          VTEC          WWA               Public      UPG Headline not                   PE - Potential
                                                                           accurate                           Enhancement

  336                          VTEC          WWA               Public      Add Administrative                 R - Routine
                                                                           Message (ADR) to                   
                                                                           WWA Hazard Menu                    

  305               153        VTEC          WWA               Public      Winter Storm                       R - Routine
                                                                           Warning Test                       

  321                          VTEC          WWA               Public      Wrong starting ETN                 R - Routine
                                                                           Frost/Freeze                       

  190               42         VTEC          WWA               Public      VTEC Dense Smoke   Template        U - Urgent
                                                                           Advisory Headline                  
                                                                           Wording                            

  160                          VTEC          WWA               Public      Slow WWA           SW              U - Urgent
                                                                           Performance (esp                   
                                                                           Live Comms Test                    
                                                                           Mode)                              

  334               182, 183   VTEC          WWA               Public      Bad MND for Winter                 U - Urgent
                                                                           Outlook                            

  335               184        VTEC          WWA               Public      Extreme Cold                       U - Urgent
                                                                           Products Listed in                 
                                                                           Wrong Place in WWA                 
                                                                           Hazard Menu                        

  337               185        VTEC          WWA               Public      Add Watch                          U - Urgent
                                                                           Definition to all                  
                                                                           WSW & NPW CTA                      
                                                                           Templates                          

  151     193, 195,            VTEC          WWA               Public      Dual time zone     Template        U - Urgent
          237                                                              issues                             

  193               44         VTEC          WWA               Public                         Template        U - Urgent

  194               57         VTEC          WWA               Public      Parenthesis in     Template        U - Urgent
                                                                           headline                           

  195     194       57         VTEC          WWA               Public                         Template        U - Urgent

  255               109, 55    VTEC          WWA               Public      Event Code of New                  U - Urgent
                                                                           vs. EXB/EXT in WWA                 

  270               110, 129   VTEC          WWA               Public      ETN error when                     U - Urgent
                                                                           switching domains                  

  299               149        VTEC          WWA               Public      Missing Zone name                  U - Urgent

  325                          VTEC          WWA               Public      No EXT for change                  U - Urgent
                                                                           in beginning time                  
                                                                           of Freeze Warning                  

  112     154, 215             VTEC          WWA               Public      ETN\'s on WWA                      
                                                                           products not                       
                                                                           starting at 0001                   

  331                          VTEC          WWA               Public      Event Beginning                    
                                                                           Date/Time VTEC                     
                                                                           String all Zero\'s                 

  134     136, 185,            VTEC          WWA               Public      EXB headline error Template        
          191, 192,                                                                                           
          236                                                                                                 

  136     134                  VTEC          WWA               Public                         Template        

  139     145, 298,            VTEC          WWA               Public      VTEC ETN           SW              
          290                                                              spontanous                         
                                                                           increment with                     
                                                                           EXT/B                              

  209               56, 69     VTEC          WWA                           VTEC Product       Template        U - Urgent
                                                                           Template Test                      
                                                                           Phrase Wording                     

  106     199, 228,            VTEC          WWA                           EXP usability      Operator error  
          146                                                              issue                              
  --------------------------------------------------------------------------------------------------------------------------