***Hydrologic Deterministic Verification***

***Test Plan***

By

Henry Herr

Office of Hydrologic Development

National Weather Service

Table of Contents

[1.0 Overview 3](#__RefHeading___Toc162235425)

[2.0 Prerequisites 3](#__RefHeading___Toc162235426)

[3.0 General Test Steps 3](#__RefHeading___Toc162235427)

[3.1 Vfyruninfo Test Details 3](#__RefHeading___Toc162235428)

[3.2 IVP Test Details 4](#__RefHeading___Toc162235429)

[3.3 IVP Batch Program Test Details 6](#__RefHeading___Toc162235430)

[3.4 IVP Batch Builder Test Details 7](#__RefHeading___Toc162235431)

[3.5 Other Details 7](#__RefHeading___Toc162235432)

[4.0 Conclusion 7](#__RefHeading___Toc162235433)

1.  []{#__RefHeading___Toc162235425 .anchor}***Overview***

The Interactive Verification Program (IVP) and IVP Batch Program are
being changed for OB8.2 in order to satisfy requirements as stated in
the *Concept of Operations and Requirements Specification Document* for
the Complete Deterministic Verification HOSIP project. Described herein
is a plan for independently testing the new IVP against the requirements
stated in Appendix B of the document above. The requirements will be
referred to by number within this document, so it is recommended any
reader have a copy of the requirements on hand.

2.  []{#__RefHeading___Toc162235426 .anchor}***Prerequisites***

The primary prerequisite before testing the software is the presence of
an RFC archive database populated with test data. The test performed
will require a consistent canned set of data to work with, so that the
correct results can be identified and compared against captured
screenshots and baseline output files. In order to run the test, the
tester will be responsible for the following:

-   Installation of the test archive database, and what settings for
    environment variables and apps-defaults tokens are needed to access
    the database.

-   Installation of the test data package and knowledge of the directory
    where the package is located.

-   Having a copy of the test procedure.

3.  []{#__RefHeading___Toc162235427 .anchor}***General Test Steps***

The following programs must be tested:

-   Vfyruninfo Editor: The editor must be tested to determine if a
    location can be added and removed from the table. A location is
    defined by the lid, pe, dur, extremum, and forecast type source, and
    has associated with it an active flag (yes for active, no for not
    active), a national flag, a list of sensor preferences (observed
    type sources), and a response time. Each field must be editable for
    all rows in the GUI.

-   IVP: The GUI must be tested to be sure that it allows for the user
    to select locations to verify, define parameters of verification,
    create an initial scatter plot and time series plot of the pairs
    used in verification, select parameters of a graphic to create, and
    create the graphic.

-   IVP Batch Program: The batch program must be tested to be sure it
    can load any pairs from the vfypairs table, calculate statistics
    specified in the requirements, and generate any graphics that can be
    generated within the GUI.

-   IVP Batch Builder: The batch builder must be tested to be sure it
    allows for editing all commands or actions within the batch
    language, run the batch file using the IVP Batch Program, and view
    any output files.

The test procedures will provide the step-by-step instructions
associated with each of the following sections.

1.  []{#__RefHeading___Toc162235428 .anchor}***Vfyruninfo Test
    Details***

The vfyruninfo table defines what verification locations are available
for verification within the IVP software. As such, the test of the
Vfyruninfo Editor tests the following requirements:

*1.1.2, 1.1.6.1, 1.2.1.1-1.2.1.4, 1.2.1.6, 1.2.1.9, 1.2.1.11*

The following tests should be performed using the Vfyruninfo Editor:

1.  Make sure Vfyruninfo Editor starts correctly.

2.  In the first GUI, the IVP Vfyruninfo Table Editor main window, test
    that the contents of the two displayed tables correspond correctly
    to the contents of the ingestfilter and vfyruninfo archive database
    tables.

3.  Verify that new rows can be added to the table with each having a
    unique combination of location id, physical element, duration,
    extremum, and forecast type source. Each will also specify a sensor
    preference list, a response time, an active flag, and a national
    flag. Rows can be added to the vfyruninfo table either based on the
    ingest filter table or manually.

4.  Verify that existing rows can be changed in the table.

5.  Verify that the contents can be saved to and loaded from the
    vfyruninfo table.

6.  Make sure the Vfyruninfo Editor exits correctly.

    1.  []{#__RefHeading___Toc162235429 .anchor}***IVP Test Details***

The following tests should be performed using the IVP software. Any step
that tests a listed requirement will have the number of that requirement
provided in italics at the end of that step's description.

1.  Make sure IVP starts.

2.  Verify that each field can be edited via the Verification Group
    Manager window using the tools provided.

    a.  Verify that parameters can be loaded from a batch file.

    b.  Verify that each parameter specified via the window can be
        edited.

> *Requirements:1.2.1.5, 1.2.1.6, 1.2.1.7, 1.2.1.8, 1.2.1.9, 1.2.1.10,
> 1.2.1.11, 1.2.2.4, 1.2.2.5, 1.2.2.6*

3.  Verify that the Verification Location Manager window can be opened
    from the Verification Group Manager window.

4.  Verify that all features of the Verification Location Manager
    function properly:

    a.  Verify that all locations listed in the vfyruninfo table of the
        archive database are visible in the table within this window,
        and that the rivercrit information given in the as, fs, modfs,
        majfs, and rs columns are correct. This requires performing
        postgres queries and comparing the results to the GUI.

    b.  Verify that the categories listed and locations chosen match
        those specified in the batch file (loaded in step 2.a above).

    c.  Verify that one or more rows can be selected from the table.

    d.  Verify that rows can be selected based on physical element,
        duration, extremum, forecast type source, and active/not active.

    e.  Verify that the forecast and observed categories can be
        specified from this window.

    f.  Verify that rows can be chosen to be displayed in the IVP Data
        Display window.

> *Requirements: 1.2.1.1, 1.2.1.2, 1.2.1.3, 1.2.1.4, 1.2.1.12, 1.2.2.1*

5.  Verify that the IVP Data Display can be opened from the Verification
    Group Manager window.

    a.  Verify that if a parameter is improperly specified (such as if
        location data types do not have the same unit), then the IVP
        Data Display will not be created.

> *Requirements: 1.2.1.13, 1.2.1.14, 1.2.1.15*

6.  Verify that the IVP Data Display interacts correctly with the
    Verification Group Manager and Verification Location Manager.

    a.  Change some setting in both the Verification Group Manager and
        Verification Location Manager, and verify that the IVP Data
        Display must be recreated due to parameter changes.

    b.  Verify that all locations chosen for display in the Verification
        Location Manager are present in the location list panel.

7.  Verify that the menu options of the IVP Data Display work.

    a.  Verify that the IVP Data Display can open the Verification Group
        Manager, Verification Location Manager, and Plot Definition
        Manager windows.

    b.  Verify that the chart displayed in the window can be saved to a
        PNG or JPEG file.

    c.  Verify that the chart allows for zooming.

    d.  Verify that the line y=x can be plotted on the chart.

    e.  Verify that a Chart Property Manager window can be opened
        allowing for users to edit features of the chart.

> *Requirements: 5.1.5*

8.  Verify that the Verification Pairs Data Manager window can be opened
    up from the IVP Data Display.

    a.  Verify that the data matches the data in the chart.

    b.  Verify that colors are used to distinguish rows visible in the
        chart from those not visible in the chart (due to zooming).

    c.  Verify that the data can be saved to a file.

    d.  Verify that pairs can be marked as good or bad, and that the
        plot reflects those marks.

> *Requirements: 5.1*

9.  Verify that the Plot Definition Manager window can be opened up from
    the IVP Data Display window.

10. Verify that the options in the Plot Definition Manager can be set
    correctly.

    a.  Verify that statistics can be chosen for display against right
        or left-hand y-axes from IVP Statistic Chooser Manager windows.

    b.  Verify that special plots can be selected and that the window
        options become disabled as appropriate depending on the special
        plot.

    c.  Verify that the parameter displayed along the x-axis can be
        specified.

    d.  Verify that the parameter used as the comparison variable can be
        specified.

    e.  Verify that a category can be chosen, restricting data used in
        generating the graphic.

    f.  Verify that a graphics template file can be specified.

> *Requirements: 1.2.2.2, 1.2.2.3, 1.2.3, 1.3.1, 1.3.3, 1.3.4, 1.3.5,
> 1.3.6*

11. Verify that the IVP Statistic Display window can be opened from the
    Verification Plot Definition manager, and that it displays an
    appropriate chart based on user settings.

12. Verify that the menu options in the IVP Statistic Display work.

    a.  Verify that the chart displayed in the window can be saved to a
        PNG or JPEG file.

    b.  Verify that a batch file can be created that recreates chart
        displayed in the window.

    c.  Verify that a Chart Property Manager window can be opened
        allowing for users to edit features of the chart.

> *Requirements: 1.4.1, 1.4.2, 5.1.2, 5.1.4, 5.1.1, 5.1.4, 5.1.5, 5.1.6*

13. Verify that the IVP Statistics Data Viewer can be opened up from the
    IVP Statistic Display.

    a.  Verify that the data matches the data in the chart.

    b.  Verify that the data can be saved to a file.

> *Requirements: 1.6, 5.1.3*

14. Verify that the Chart Properties Manager window can be used to edit
    properties of the chart displayed in either the IVP Data Display or
    IVP Statistic Display.

    a.  Verify that each of the parameters required to be editable via
        the Chart Properties Manager can be edited.

    b.  Verify that chart properties can be saved to and loaded from a
        template file via the Actions menu of the IVP Statistics Display
        and the Options menu of the IVP Data Display.

> *Requirements: 1.3.2*

15. Create many different graphics by changing the settings in the Plot
    Definition Manager and verify that they are created correctly.

16. Verify that the windows can close down.

    a.  Click on the "Close" menu item in the IVP Statistic Display.

    b.  Click on the "x" in the upper corner of every other open window,
        except for the IVP Data Display. This should close each window.

    c.  Click on the "Exit" menu item of the IVP Data Display. The
        program should shut down.

NOTE: The program should also shut down if you click on the "Close"
button of the IVP Data Display while the Verification Group Manager is
not open, or by closing the Verification Group Manager while the IVP
Data Display is not open.

1.  []{#__RefHeading___Toc162235430 .anchor}***IVP Batch Program Test
    Details***

The following test should be performed using the IVP Batch Program. Any
step that tests a listed requirement will have the number of that
requirement provided in italics at the end of that step's description.

1.  Verify that every required parameter can be set via the IVP Batch
    Program batch language.

    a.  Verify that all pairing parameters can be edited.

    b.  Verify that all location parameters can be edited.

    c.  Verify that all group parameters can be edited.

    d.  Verify that all graphics parameters can be edited.

> *Requirements: 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.2, 1.3.1,
> 1.3.3, 1.3.4, 1.3.5, 1.3.6*

2.  Verify that pairs can be built by the software, and that the pairs
    are correct.

    a.  Verify that the data paired is specified by the vfyruninfo table
        and parameters of the batch file.

    b.  Verify that output pairs are stored in the vfypairs table.

> *Requirements: 1.1.7*

3.  Verify that every required statistic can be calculated and output to
    an ascii file, and that the output statistics are correctly
    calculated.

    a.  Verify that output file can be generated and that statistics
        shown therein are correctly computed.

    b.  Verify that a pairs file can be generated and that it contains
        the correct pairs data.

> *Requirements: 5.1.1, 5.1.2, 5.1.3, 5.2, 5.3*

4.  Verify that any graphic that can be produced via the IVP GUI can
    also be produced via the IVP Batch Program.

    a.  Verify that a batch file saved from the IVP GUI can be executed
        via the IVP Batch Program to generate an identical graphic.

    b.  Verify that the template file can be specified using the batch
        tokens.

    c.  Verify that the graphics can be produced and output to a PNG or
        JPG file.

    d.  Verify that the data comprising a graphic can be output to an
        ASCII text file.

> *Requirements:1.3.2, 1.4.3, 5.1.4, 5.1.5, 5.1.6*

1.  []{#__RefHeading___Toc162235431 .anchor}***IVP Batch Builder Test
    Details***

IVP Batch Builder is software designed to guide the user through
building and running an IVP Batch Program batch file. The following test
should be performed using the IVP Batch Builder:

1.  Verify that any batch command or action of the IVP Batch Program can
    be edited via the IVP Batch Builder interface.

2.  Verify that commands displayed in the command list can be saved to
    or loaded from a batch file.

3.  Verify that the IVP Batch Builder can run the IVP Batch Program
    using the batch file displayed in the command list, and that any
    output file (both graphics files and ascii text files) generated
    from the execution can be viewed.

    1.  []{#__RefHeading___Toc162235432 .anchor}***Other Details***

The test above will use canned data, so that results will be predictable
and easy to check. The canned data will be comprehensive and
representative of data found in the field.

4.  []{#__RefHeading___Toc162235433 .anchor}***Conclusion***

Upon completion of these tests, the software should be fully tested and
all requirements are satisfied.