***Interactive Verification Program***

***User's Manual***

By

Henry Herr

Office of Hydrologic Development

National Weather Service

***Table Of Contents***

[1.0 Overview 5](#__RefHeading___Toc135190916)

[1.1 Using This Manual 5](#__RefHeading___Toc135190917)

[2.0 Notation 5](#__RefHeading___Toc135190918)

[3.0 Execution 6](#__RefHeading___Toc135190919)

[4.0 Directory Structure 6](#__RefHeading___Toc135190920)

[5.0 Apps-defaults Tokens 6](#__RefHeading___Toc135190921)

[6.0 File Types 7](#__RefHeading___Toc135190922)

[7.0 Instructions 8](#__RefHeading___Toc135190923)

[8.0 Verification Group Manager 9](#__RefHeading___Toc135190924)

[8.1 Start Time and End Time 9](#__RefHeading___Toc135190925)

[8.2 Time Step 9](#__RefHeading___Toc135190926)

[8.3 Lead Time Start, Lead Time End, and Lead Time Step
10](#__RefHeading___Toc135190927)

[8.4 Fcst Type Sources 10](#__RefHeading___Toc135190928)

[8.5 Physical Elements 10](#__RefHeading___Toc135190929)

[8.6 River Responses 10](#__RefHeading___Toc135190930)

[8.7 Persistence 10](#__RefHeading___Toc135190931)

[8.8 Buttons 10](#__RefHeading___Toc135190932)

[9.0 Choose Date/Time Window 12](#__RefHeading___Toc135190933)

[9.1 Specifying a Date 12](#__RefHeading___Toc135190934)

[9.2 Buttons 12](#__RefHeading___Toc135190935)

[10.0 Choose Relative Date/Time Window 13](#__RefHeading___Toc135190936)

[10.1 Specifying a Relative Date 13](#__RefHeading___Toc135190937)

[10.2 Buttons 13](#__RefHeading___Toc135190938)

[11.0 Select FCST_TS and Select PE Windows
14](#__RefHeading___Toc135190939)

[11.1 Buttons 14](#__RefHeading___Toc135190940)

[12.0 Verification Location Manager 15](#__RefHeading___Toc135190941)

[12.1 Terminology 15](#__RefHeading___Toc135190942)

[12.2 The Table 15](#__RefHeading___Toc135190943)

[12.3 Buttons 16](#__RefHeading___Toc135190944)

[13.0 Observed and Forecast Category Editor
17](#__RefHeading___Toc135190945)

[13.1 Boundary List 17](#__RefHeading___Toc135190946)

[13.2 Specifying a Boundary String 17](#__RefHeading___Toc135190947)

[13.3 Buttons 18](#__RefHeading___Toc135190948)

[14.0 IVP Data Display 19](#__RefHeading___Toc135190949)

[14.1 Highlighted Location 19](#__RefHeading___Toc135190950)

[14.2 Data Tracker 19](#__RefHeading___Toc135190951)

[14.3 Zooming In and Out 19](#__RefHeading___Toc135190952)

[14.4 Menu Items 20](#__RefHeading___Toc135190953)

[15.0 Verification Pairs Data Manager 22](#__RefHeading___Toc135190955)

[15.1 Sorting the Rows 22](#__RefHeading___Toc135190956)

[15.2 Visible Rows 22](#__RefHeading___Toc135190957)

[15.3 Buttons 23](#__RefHeading___Toc135190958)

[16.0 Verification Plot Definition Manager
24](#__RefHeading___Toc135190959)

[16.1 Primary and Secondary Statistics 24](#__RefHeading___Toc135190960)

[16.2 Primary and Secondary Statistic Plot Types
24](#__RefHeading___Toc135190961)

[16.3 X-Axis Variable 25](#__RefHeading___Toc135190962)

[16.4 Analysis Interval and Lead Time Interval Used
25](#__RefHeading___Toc135190963)

[16.5 Observed Category and Forecast Category
25](#__RefHeading___Toc135190964)

[16.6 Graph Template File 26](#__RefHeading___Toc135190965)

[16.7 Buttons 26](#__RefHeading___Toc135190966)

[17.0 IVP Statistic Chooser Manager 27](#__RefHeading___Toc135190967)

[17.1 Buttons 27](#__RefHeading___Toc135190968)

[18.0 IVP Statistic Display 28](#__RefHeading___Toc135190969)

[18.1 Menu Items 28](#__RefHeading___Toc135190970)

[19.0 IVP Statistics Data Viewer 30](#__RefHeading___Toc135190971)

[19.1 Buttons 30](#__RefHeading___Toc135190972)

[20.0 IVP Batch File Save Manager 31](#__RefHeading___Toc135190973)

[20.1 File Name Text Fields 31](#__RefHeading___Toc135190974)

[20.2 Buttons 31](#__RefHeading___Toc135190975)

[21.0 IVP Chart Properties Manager 33](#__RefHeading___Toc135190976)

[21.1 Template Files 34](#__RefHeading___Toc135190977)

[21.2 Default Values 34](#__RefHeading___Toc135190978)

[21.3 All Titles (Chart Titles) 34](#__RefHeading___Toc135190979)

[21.4 All Numerical Axis Limits (Chart Others)
35](#__RefHeading___Toc135190980)

[21.4.1 Buttons 36](#__RefHeading___Toc135190981)

[21.5 Legend (Chart Titles) 37](#__RefHeading___Toc135190982)

[21.5.1 Buttons 37](#__RefHeading___Toc135190983)

[21.6 X-Axis Labels (Chart Others) 37](#__RefHeading___Toc135190984)

[21.6.1 Buttons 38](#__RefHeading___Toc135190985)

[22.0 Tips and Information 39](#__RefHeading___Toc135190986)

[22.1 System Settings File 39](#__RefHeading___Toc135190987)

[22.2 NOAA and NWS Logos 39](#__RefHeading___Toc135190988)

[22.3 Performance 39](#__RefHeading___Toc135190989)

[22.4 Running IVP on AWIPS LX Machines 40](#__RefHeading___Toc135190990)

[22.5 Debugging 40](#__RefHeading___Toc135190991)

[Appendix A: System Settings File 41](#__RefHeading___Toc135190992)

[A.1 Overview 41](#__RefHeading___Toc135190993)

[A.2 Location 41](#__RefHeading___Toc135190994)

[A.3 System Settings File 41](#__RefHeading___Toc135190995)

[A.3.1 Commands 41](#__RefHeading___Toc135190996)

[A.4 Generic Font Format 48](#__RefHeading___Toc135190997)

[A.4 Generic Color Format 49](#__RefHeading___Toc135190998)

[Appendix B: The Statistics 50](#__RefHeading___Toc135190999)

[B.1 Error Statistics 50](#__RefHeading___Toc135191000)

[B.2 Categorical Statistics 50](#__RefHeading___Toc135191001)

[B.3 Quantiles and Extremes 51](#__RefHeading___Toc135191002)

[B.4 Leadtime Statistics 51](#__RefHeading___Toc135191003)

[B.5 Sample Size 52](#__RefHeading___Toc135191004)

[B.6 References 52](#__RefHeading___Toc135191005)

1.  []{#__RefHeading___Toc135190916 .anchor}***Overview***

Number range CHAPTERThe Interactive Verification Program (IVP) is a tool
for verifying hydrologic forecasts. It provides the ability to do the
following:

-   Define and view forecast-observed data pairs for use in verification
    according to the location, physical element, forecast validtime,
    forecast leadtime, forecast value, observed value, and forecast type
    source. Data can be viewed graphically and in a table format.

-   Save the forecast-observed data pairs to a properly formatted pairs
    file.

-   Define and view verification statistics, including error statistics,
    categorical statistics, quantiles, extremes, and sample sizes.
    Statistics can be viewed in a table and in a plot. More than one
    statistic can be displayed against a single y-axis if those
    statistics have the same scale, and two y-axes can be included in
    the plot. Statistics can be plotted against location, analysis (or
    validtime) interval, lead time interval, forecast category, and
    observed category.

-   Save plot to an image file (jpg or png).

-   Save table containing the statistics to a file.

-   Save a batch file that can be processed by the IVP Batch Program in
    order to regenerate graphics visible in the IVP.

This manual provides instructions on how to use the IVP.

1.  []{#__RefHeading___Toc135190917 .anchor}***Using This Manual***

Section 7 provides instructions on how to use this software to generate
verification graphics. Sections 8-21 provide information on each window
of the IVP. It is recommended that the user follow the instructions in
Section 7 when first using this software, and, as the user interacts
with the IVP, refer to the Section pertaining to a window if he or she
has a question about it (see the Table of Contents for the page number).

Screenshots for each window are included at the END of the section
pertaining to the window.

2.  []{#__RefHeading___Toc135190918 .anchor}***Notation***

Number range CHAPTER

The following notational conventions are used in this document:

-   A GUI component, including a window, panel, menu, menu item, or
    button, is displayed in **bold**.

-   A menu item within a menu is displayed as **\[menu name\] \>\>
    \[menu item name\]**. For example, **Actions \>\> Close** is the
    **Close** menu item within the **Actions** menu.

-   Terms to remember and NOTEs will be denoted in *italics*.

-   Number range CHAPTER\<key\> indicates a keyboard key. For example
    \<shift\> means the shift key.

-   Text to be entered at a command line is displayed in this font.

-   Directories and file names are in this font.

-   A directory corresponding to an apps-defaults token is denoted
    \$(token).

3.  []{#__RefHeading___Toc135190919 .anchor}***Execution***

To execute the IVP, enter:

cd \$(get_apps_defaults vsys_dir)/scripts

ivp

Upon execution, the **Verification Group Manager** window will open up.

4.  []{#__RefHeading___Toc135190920 .anchor}***Directory Structure***

The following directory structure must be in place for the IVP to
execute properly:

> \$(vsys_dir)/bin/RELEASE/rfc.ob7.2.jar
>
> \$(vsys_dir)/bin/RELEASE/dbgen.jar
>
> \$(vsys_dir)/input/
>
> \$(vsys_dir)/files/\$(LOGNAME)/templates/
>
> \$(vsys_dir)/output/\$(LOGNAME)/
>
> \$(vsys_dir)/scripts/ivp

\$(vsys_dir) refers to the value of the apps-defaults token vsys_dir,
which points to the base directory, typically /rfc_arc/verify.
\$(LOGNAME) refers to the user name; any directory above with this in it
should be created for each user who is to use the software.

5.  []{#__RefHeading___Toc135190921 .anchor}***Apps-defaults Tokens***

The following apps-defaults tokens are used by the IVP:

-   adb_name : \<depends upon RFC\>

-   verify_dir : /rfc_arc/verify

-   vsys_dir : \$(verify_dir)

-   vsys_input : \$(vsys_dir)/input

-   vsys_output : \$(vsys_dir)/output

-   vsys_files : \$(vsys_dir)/files

-   rax_pghost : ax \<optional; if not present, it uses the PGHOST
    environment variable, instead\>

-   pguser : pguser \<optional; if not present, it uses the PGUSER
    environment variable, instead\>

-   pgport : 5432 \<optional; if not present, it uses th PGPORT
    environment variable, instead\>

Each of the above directories must exist for the IVP to run properly. It
is also recommended that the following directory be created for each
user:

\$(vsys_input)/\$LOGNAME

If this recommendation is followed, then the apps-defaults site file
should override the setting of vsys_input as follows:

-   vsys_input : \$(vsys_dir)/input/\$(LOGNAME)

All of these directories should be constructed prior to running IVP.

6.  []{#__RefHeading___Toc135190922 .anchor}***File Types***

The following types of files are used within the IVP (recommended file
extensions are given in parentheses; for the output image files, the
extensions are required)

-   **Batch Input Files (.bat)**: These files are input files to the IVP
    Batch Program, and can be loaded by the GUI in order to specify
    parameters of the data used and plots generated. These files are, by
    default, assumed to be in the \$(vsys_input) directory.

-   **Graph Template Files (.txt)**: These files specify properties of
    the charts to create, including labels, fonts, colors, sizes, axis
    limits, and others. These files are, by default, assumed to be in
    the \$(vsys_files)/\$LOGNAME/templates directory.

-   **Output Image Files (.png, .jpg, .jpeg)**: These files are images
    of the charts generated by the IVP or IVP Batch Program. These files
    are, by default, assumed to be in the \$(vsys_output)/\$LOGNAME
    directory. Any such file MUST have one of the extensions listed.

-   **Output Data Files (.dat)**: These files are ASCII format files
    that provide the data plotted to a chart in a tabular format. These
    files are, by default, assumed to be in the
    \$(vsys_output)/\$LOGNAME directory.

-   **System Settings File**: This file is an ASCII format file used to
    specify defaults for the appearance of the IVP and its charts. The
    file must be in the directory \$(vsys_dir)/app-defaults and have the
    name IVP_SYSTEM_FILE.txt. See Appendix A for more details.

7.  []{#__RefHeading___Toc135190923 .anchor}***Instructions***

Prior to using the IVP to generate a verification graphic, the user
should know the answers to the following:

-   Which locations are to be included in the verification analysis?
    What are the location ids for those locations?

-   Over what analysis time period is the verification analysis to be
    performed? Is it to be broken down into intervals? If so, what is
    the width of the intervals?

-   For which forecast lead times is verification to be performed? Is
    the overall lead time period to be broken down into smaller
    intervals? If so, what is the width of the intervals?

-   Are any restrictions to be placed on the physical elements or
    forecast types of the forecasts to be analyzed?

-   Are persistence forecasts to be included in the analysis?

-   Is performance relative to categories (e.g. above flood stage, below
    flood stage) to be analyzed? If so, what categories are to be used?

To generate a verification graphic, do the following:

1.  Start the IVP (see Section 3.0).

2.  Enter the analysis interval information, lead time information, and
    any other restrictions into the **Verification Group Manager**
    window.

3.  Click on **Edit Locations** to open up the **Verification Location
    Manager** window.

4.  Select the desired locations from the **Locations List**.

5.  Set the categories using the **Observed Category Editor** and
    **Forecast Category Editor**.

6.  Click on **Create Display** in the **Verification Group Manager** to
    create the **IVP Data Display**.

7.  Examine the data for a general feeling about the skill of the
    forecasts.

8.  Click on **Actions \>\> Show Plot Mgr** to open up the
    **Verification Plot Definition Manager**.

9.  Specify a desired graphic by changing the settings in the
    **Verification Plot Definition Manager**.

10. Click on **Create Plot** to create an **IVP Statistics Display**.

11. Many graphics can be produced. At this point, the data used in
    verification is already loaded into memory. So producing a graphic
    is relatively fast and the user should feel free to produce as many
    graphics as needed to do a proper verification analysis.

12. If the user would like to produce a particular graphic on a regular
    basis, the user should save a batch file by clicking on **Actions
    \>\> Save Batch File** and fill out the fields in the **IVP Batch
    File Save Manager** as desired. It is important to use relative
    dating for the start and end times in the **Verification Group
    Manager** window if the user would like this plot to be generated in
    batch mode for different analysis intervals.

13. Save any desired graphics by clicking on **Actions \>\> Save Image
    File** in either the **IVP Data Display** or **IVP Statistic
    Display**.

```{=html}
<!-- -->
```
8.  []{#__RefHeading___Toc135190924 .anchor}***Verification Group
    Manager***

The **Verification Group Manager** window provides tools to manage and
edit parameters of a *verification group*. A verification group is a
collection of locations and parameters that define how to do statistic
calculations, for which one set of verification statistics is to be
calculated. Parameters of a verification group include:

-   locations: collection of locations within which a forecast-observed
    data pair's location must exist in order for it to be included in
    the calculations

-   analysis intervals: start time and end time defining the period of
    analysis, and a time step breaking down the period into subintervals

-   lead time intervals: start lead time and end lead time defining a
    range that restricts the lead time of the forecast, and a time step
    breaking down the range into subintervals

-   forecast type sources: list of type sources for forecasts to include
    in the calculations

-   physical elements: list of physical elements for the observed and
    forecast data to include in the calculations

-   response time: list of response times, restricting data to include
    in verification by the corresponding location's response time

-   persistence: parameter defining if persistence forecasts (those with
    a type source of "FR") are to be included in the analysis

Each parameter is managed in a graphical way, using components of the
**Verification Group Manager**.

*NOTE: It is recommended that, if you have an appropriate batch file
already constructed that defines the group for which you wish to compute
statistics, then you should use the* **Load From Batch** *button to load
the parameters from the file. This is easier than specifying each field
manually.*

1.  []{#__RefHeading___Toc135190925 .anchor}***Start Time and End
    Time***

The *start time* and *end time* of the analysis interval specify a range
within which the forecast valid time of a forecast-observed data pair
must exist in order to be included in the verification computations. In
the **Verification Group Manager**, the start time is given in the
**Start Time** text field and the end time is given in the **End Time**
text field. Both can be specified manually or via a **Choose Date/Time**
**Window** or **Choose Relative Date/Time** **Window**.

*NOTE: See Section 8.0 of the* IVP Batch Program User's Manual for
Verification *for acceptable formats for the text field.*

2.  []{#__RefHeading___Toc135190926 .anchor}***Time Step***

The *time step* specifies a time interval that breaks down the overall
analysis interval (start to end) into equal sized subintervals. In the
**Verification Group Manager**, the time step is given by the **Time
Step Quantity** spinner and **Time Step Unit** choice box. The former
specifies an integer value greater than 0, and the latter specifies a
unit of time ("weeks", "days", "hours") or "NONE". A unit of "NONE"
implies that no subintervals are to be used and the **Time Step
Quantity** is to be ignored. Combined, the quantity and unit define an
interval of time.

3.  []{#__RefHeading___Toc135190927 .anchor}***Lead Time Start, Lead
    Time End, and Lead Time Step***

The *lead time start*, *lead time end*, and *lead time step* specify a
range within which the forecast lead time (valid time -- basis time) of
a forecast-observed data pair must exist in order to be included in the
verification computations, and further breaks down that interval into
equal sized subintervals. In the **Verification Group Manager**, each is
specified using a spinner and choice box: **Lead Time Start Quantity**
spinner, **Lead Time Start Unit** choice box, etc.

4.  []{#__RefHeading___Toc135190928 .anchor}***Fcst Type Sources***

The *fcst type sources* define a list of type sources within which the
forecast's type source of a forecast-observed data pair must exist in
order to be included in the verification computations. In the
**Verification Group Manager**, it is specified as a comma delimited
list in the **Forecast Type Sources** text field. It is edited manually
or via a **Select FCST_TS Window**.

5.  []{#__RefHeading___Toc135190929 .anchor}***Physical Elements***

The *physical elements* define a list of physical elements within which
the physical element of a forecast-observed data pair must exist in
order to be included in the verification computations. In the
**Verification Group Manager**, it is specified as a comma delimited
list in the **Physical Elements** text field. It is edited manually or
via a **Select PE Window**.

6.  []{#__RefHeading___Toc135190930 .anchor}***River Responses***

The *river responses* define a list of response times within which a
locations response time must be in order for points for that location to
be included in the verification computations. In the **Verification
Group Manager**, it is specified using **Response Time** radio buttons
that allow for a setting of "ALL" or a setting that can include any
combination of "SLOW", "MEDIUM", and "FAST".

*NOTE: If the current river response setting is "ALL", then the other
three radio buttons will be deactivated and cannot be selected. To make
them active, first deselect "ALL".*

7.  []{#__RefHeading___Toc135190931 .anchor}***Persistence***

The *persistence flag* defines if forecast-observed data pairs that are
for persistence forecasts are to be included in the verification
computation. In the **Verification Group Manager**, it is specified by
using **Persistence** radio buttons that allow for a setting of "ON" or
"OFF".

*NOTE: Setting the* **Persistence** *to "ON" is equivalent to including
the type source "FR" in the* **Forecast Type Sources** *text field.*

*NOTE: If calculations are only to be made for persistence forecasts,
then set the **Forecast Type Sources** text field to "FR" and set this
radio button to either "ON" or "OFF" (it doesn't matter, since the text
field includes "FR").*

8.  []{#__RefHeading___Toc135190932 .anchor}***Buttons***

-   **Start Time Fixed** button: Opens up a **Choose Date/Time Window**
    for editing the **Start Time** text field.

-   **Start Time Rel** button: Opens up a **Choose Relative Date/Time
    Window** for editing the **Start Time** text field.

-   **End Time Fixed** button: Opens up a **Choose Date/Time Window**
    for editing the **End Time** text field.

-   **End Time Rel** button: Opens up a **Choose Relative Date/Time
    Window** for editing the **End Time** text field.

-   **Fcst Type Sources Edit List** button: Opens up a **Select FCST_TS
    Window** for editing the **Forecast Type Sources** text field.

-   **Physical Elements Edit List** button: Opens up a **Select PE
    Window** for editing the **Physical Elements** text field.

-   **Edit Locations** button: Opens up the **Verification Location
    Manager** to allow for changing categories for locations and
    choosing locations to be displayed in the **IVP Data Display**.

-   **Load From Batch** button: Opens up a file browser that can be used
    to find a previously created batch file for the IVP Batch Program
    and set the parameters within this **Verification Group Manager**
    according to that file. It will read parameters until the first
    verification group is defined (via a DEF_GRP batch file action; see
    the *IVP Batch Program User's Manual*) and sets the parameters
    herein according to those read in.

-   **Create Display** button: Creates an **IVP Data Display**, showing
    forecast-observed data pairs specified by this **Verification Group
    Manager** and the **Verification Location Manager**. Pairs shown
    correspond to chosen locations in the **Verification Location
    Manager** and all chosen locations.

![](media/image1.jpeg){width="4.129166666666666in"
height="4.863194444444445in"}

9.  []{#__RefHeading___Toc135190933 .anchor}***Choose Date/Time
    Window***

The **Choose Date/Time Window** is used to specify a date in the format

yyyy-mm-dd hh:mm:ss TZC

1.  []{#__RefHeading___Toc135190934 .anchor}***Specifying a Date***

The date is chosen as follows:

1.  Select a month from the **Month** choice box.

2.  Select a year from the **Year** choice box.

3.  Click on the desired day of the month.

4.  Specify the hour in the **Hour** spinner either manually or by using
    the up and down arrows.

5.  Specify the minute in the **Minute** spinner.

6.  Specify the second in the **Second** spinner.

7.  Select a time zone in the **Time Zone** choice box.

8.  Click **OK**.

    1.  []{#__RefHeading___Toc135190935 .anchor}***Buttons***

-   **OK** button: Closes the window and updates the corresponding date
    text field in the **Verification Group Manager**.

-   **Cancel** button: Closes the window with no change in the
    **Verification Group Manager**.

![](media/image2.jpeg){width="2.8194444444444446in"
height="2.484722222222222in"}

10. []{#__RefHeading___Toc135190936 .anchor}***Choose Relative Date/Time
    Window***

The **Choose Relative Data/Time Window** is used to specify a date
relative to the current time. It is a string of this format:

\* \<+/-\> \<quantity\> \<unit\> \<quantity\> \<unit\>\...

where \<quantity\> is a positive integer and \<unit\> is "weeks",
"days", or "hours". For example, "\* - 2weeks" is valid, and is equal to
"\* - 14 days" or "\* - 1 weeks 7 days" or "\* - 1weeks 6 days 24
hours".

1.  []{#__RefHeading___Toc135190937 .anchor}***Specifying a Relative
    Date***

A relative date is chosen as follows:

1.  Select either the **Plus (+) Radio** button or the **Minus (--)
    Radio** button.

2.  Into the **Quantity** text field, enter the quantity to use.

3.  From the **Unit** choice box, select the unit to use.

4.  Click on the **\>\>** button. The quantity and unit will enter into
    the **Value** text field.

5.  Repeat steps 2 and 3 until the relative date/time is completely
    specified.

6.  Click on **OK**.

    1.  []{#__RefHeading___Toc135190938 .anchor}***Buttons***

-   **OK** button: Closes the window and updates the corresponding date
    text field in the **Verification Group Manager**.

-   **Cancel** button: Closes the window with no change in the
    **Verification Group Manager**.

![](media/image3.jpeg){width="2.8305555555555557in"
height="1.4513888888888888in"}

11. []{#__RefHeading___Toc135190939 .anchor}***Select FCST_TS and Select
    PE Windows***

The **Select FCST_TS Window** and **Select PE Window** perform similar
functions: they allow the user to select items to use from a list of
items, or select "ALL" if all items are to be used. To select an item
for use, click on the desired items in the list of items. To deselect
the item, click on the selected item again. When finished, the user must
click **OK** to accept the list or **Cancel** to return without
accepting the list.

1.  []{#__RefHeading___Toc135190940 .anchor}***Buttons***

-   **OK** button: Closes the window and updates the corresponding date
    text field in the **Verification Group Manager**.

-   **Cancel** button: Closes the window with no change in the
    **Verification Group Manager**.

![](media/image4.jpeg){width="2.4in" height="2.0083333333333333in"}

12. []{#__RefHeading___Toc135190941 .anchor}***Verification Location
    Manager***

The **Verification Location Manager** provides tools to manage the list
of verification locations to be included in the computation of
verification statistics. A *verification location* is uniquely
identified by a location id (lid from the IHFS and archive databases)
and physical element, and also contains a response time, various
critical stages found in the rivercrit table of the archive data, and
categories for both the observed and forecast value.

1.  []{#__RefHeading___Toc135190942 .anchor}***Terminology***

These terms must be defined in order to describe the **Verification
Location Manager**:

-   *selected* *location*: a selected location is a verification
    location that is currently selected from the table by clicking on
    the location's row. Selected locations are highlighted in blue.

-   *chosen location*: A chosen location is a verification location that
    is chosen to be displayed in the **IVP Data Display**. A location is
    chosen by first selecting the row and then clicking on **Choose**.
    Chosen locations are highlighted in pink and red. All chosen
    locations must have the same number of forecast categories and the
    same number of observed categories.

*NOTE: Verification locations must be chosen in order for the* **Create
Display** *button of the* **Verification Group Manager** *to build the*
**IVP Data Display***. All chosen locations MUST have the same number of
observed categories and forecast categories. If not, the display will
not be created.*

1.  []{#__RefHeading___Toc135190943 .anchor}***The Table***

The **Location Table** displays the verification locations defined in
the vfyruninfo table. Each row of the table lists the following
attributes of a location:

-   location: The 8-character identifier for a location, which along
    with the pe uniquely defines the verification location to which this
    row corresponds. Often called the lid in the IHFS and archive
    databases.

-   pe: The 2-character physical element, which along with the location
    uniquely defines the verification location to which this row
    corresponds.

-   as: The action stage, defined for this location in the rivercrit
    table. If this stage is not defined, it will be -999.0.

-   fs: The flood stage, defined for this location in the rivercrit
    table. If this stage is not defined, it will be -999.0.

-   modfs: The moderate flood stage, defined for this location in the
    rivercrit table. If this stage is not defined, it will be -999.0.

-   majfs: The major flood stage, defined for this location in the
    rivercrit table. If this stage is not defined, it will be -999.0.

-   rs: The record stage, defined for this location in the rivercrit
    table. If this stage is not defined, it will be -999.0.

-   Forecast Category: The string defining the categories for the
    forecast value. This field can be edited via the **Forecast Category
    Editor** window.

-   Observed Category: The string defining the categories for the
    observed value. This field can be edited via the **Observed Category
    Editor** window.

Within the table, locations are selected by clicking on the
corresponding row. Multiple locations can be selected by clicking and
dragging, or by using the \<ctrl\> and \<shift\> buttons while clicking.
A selected location can be deselected by pressing \<ctrl\> while
clicking on its corresponding row.

Clicking on a column header within the table will sort the table entries
by the column, either descending or ascending depending on which sort
was last performed (ascending is the initial sort order).

1.  []{#__RefHeading___Toc135190944 .anchor}***Buttons***

-   **Edit Fcst Cat** button: Opens the **Forecast Category Editor** to
    allow for editing forecast categories for the selected locations.

-   **Edit Obs Cat** button: Opens the **Observed Category Editor** to
    allow for editing forecast categories for the selected locations.

-   **All** button: Selects all locations. This may require two clicks
    for it to work.

-   **None** button: Deselects all locations.

-   **Choose** button: Chooses all currently selected locations for
    display. After clicking, the selected rows will have a pink and red
    background and all rows will be deselected.

-   **Clear** button: Unchooses all currently selected locations. After
    clicking, the selected rows will have a white and gray background
    and all rows will be deselected.

-   **Clear All** button: Unchooses all locations. Any currently
    selected rows will be deselected.

-   **Show** button: Highlights the currently selected location in the
    **IVP Data Display**. This button will do nothing if the selected
    location is not chosen or if more than one row is currently
    selected.

![](media/image5.jpeg){width="6.5in" height="2.3493055555555555in"}

13. []{#__RefHeading___Toc135190945 .anchor}***Observed and Forecast
    Category Editor***

The **Observed Category Editor** and **Forecast Category Editor** are
used to specify categories for verification locations. The categories
are defined by a comma delimited list, where each delimited item is a
*boundary string* specifying a boundary of a category. A boundary string
must one of the following:

-   MIN: -INFINITY (no lower bound)

-   MAX: INFINITY (no upper bound)

-   \<absolute number\>: Any fixed number (i.e. for stage data, 6 means
    6 feet)

-   \<scalar\>\*\<stage\>: A multiple of one of the stages associated
    with a verification location. The scalar can be any fixed number.
    The stage must be one of AS (action stage), FS (flood stage), ModFS
    (moderate flood stage), MajFS (major flood stage), and RS (record
    stage).

For example, the following string defines two categories: above flood
stage and below flood stage:

"MIN, 1\*FS, MAX"

The following defines three categories: below flood stage, between flood
stage and 12 ft (assuming the data's units is in feet), and above flood
stage:

"MIN, 1\*FS, 12, MAX"

*NOTE: The category boundaries are sorted into ascending order when
determining the categories, implying that the order of boundary strings
within the category string does not matter. So, in the second example
above, if 12 were less than the flood stage for a location, the middle
category would include stages larger than 12 ft and less than the flood
stage.*

1.  []{#__RefHeading___Toc135190946 .anchor}***Boundary List***

The **Boundary List** lists the currently specified boundary strings
defining the category string. It is a list of strings on the right-hand
side of the window above the **Delete** button with one boundary string
per line. List items can be selected by clicking on them and deselected
by clicking on them a second time.

2.  []{#__RefHeading___Toc135190947 .anchor}***Specifying a Boundary
    String***

A boundary string is constructed by doing the following:

1.  Click on the **MIN** button if one of the categories is to have no
    lower bound.

2.  Click on the **MAX** button if one of the categories is to have no
    upper bound.

3.  For each absolute number that defines a boundary, type the number
    into the **Absolute** text field and click on the **Absolute \>\>**
    button (the **\>\>** button next to the **Absolute** text field).

4.  For each multiple of a stage that defines a boundary, type the
    scalar into the **Scalar** text field, select the stage from the
    **Stage** choice box, and click on the **Scalar \>\>** button (the
    **\>\>** button next to the **Stage** choice box).

5.  Select the locations to which you wish to apply these categories.

6.  Click on **OK** to make the changes to the **Verification Location
    Manager**.

After clicking on the **OK** button, the **Boundary List** will be
turned into a comma delimited string and displayed in the appropriate
column of the **Location Table**. The **Category Editor** window will
also close.

1.  []{#__RefHeading___Toc135190948 .anchor}***Buttons***

-   **Add MIN** button: Adds the MIN string to **Boundary List**.

-   **Add MAX** button: Adds the MAX string to the **Boundary List**.

-   **Absolute \>\>** button: Adds an \<absolute number\> to the
    **Boundary List**. This is the **\>\>** button next to the
    **Absolute** text field.

-   **Scalar \>\>** button: Adds a \<scalar\>\*\<stage\> to the
    **Boundary List**. This is the **\>\>** button next to the
    **Scalar** choice box.

-   **Delete** button: Removes the currently selected boundary strings
    from the **Boundary List**.

-   **OK** button: Converts the **Boundary List** into a comma delimited
    category string and inserts the category string into the appropriate
    column within the **Location Table** for the currently selected
    locations. Also closes the window.

-   **Close** button: Closes the window. Upon opening the window again,
    the **Boundary List** displayed upon last closing the window will
    again be visible.

![](media/image6.jpeg){width="2.8305555555555557in"
height="2.267361111111111in"}

14. []{#__RefHeading___Toc135190949 .anchor}***IVP Data Display***

The **IVP Data Display** graphically displays all of the
forecast-observed data pairs specified by the parameters in the
**Verification Group Manager** and locations chosen in the
**Verification Location Manager**. The y-axis of the plot is the
forecast value, the x-axis is the observation. Any data pair that would
be included in the verification is shown in the plot.

By default, the title of the plot provides the RFC name of the data, the
overall analysis interval, the overall lead time interval, and a list of
the ids of the locations chosen for display. Also, by default, the NOAA
logo is displayed in the upper left corner and the NWS logo in the upper
right corner.

Associated with the **IVP Data Display** chart are chart properties,
including the title, axis labels, legend, fonts, etc. These properties
can be customized via the **Chart Properties Manager**.

*NOTE: The* **IVP Data Display** *is directly linked with, and must be
consistent with, the* **Verification Group Manager** *and*
**Verification Location Manager***. If parameters in either window are
changed, then the* **IVP Data Display** *must be re-created (i.e. click
on the* **Create Display** *button in the* **Verification Group
Manager***). The next time the mouse pointer passes over the window, the
user will be told this and the* **IVP Data Display***, and all related
windows including the* **Verification Pairs Data Manager** *and*
**Verification Plot Definition Manager***, will close.*

1.  []{#__RefHeading___Toc135190950 .anchor}***Highlighted Location***

By selecting one and only one location and then clicking on the **Show**
button of the **Verification Location Manager**, the user can highlight
a location in the **IVP Data Display**. A highlighted location is
displayed in yellow, and the boundaries defining the categories for the
location are drawn with blue lines. The forecast category boundaries
will be displayed as horizontal lines, and the observed will be
displayed as vertical lines.

2.  []{#__RefHeading___Toc135190951 .anchor}***Data Tracker***

The **Data Tracker** consists of coordinates displayed in the upper
right corner of the window in the menu bar. When the mouse pointer
passes over the plot, the **Data Tracker** displays the values currently
pointed to by the mouse pointer.

3.  []{#__RefHeading___Toc135190952 .anchor}***Zooming In and Out***

To zoom in on portion of the **IVP Data Display** plot, put the mouse
pointer at one corner of the region you want to zoom in on, move the
mouse to the opposite corner, and release the button. This will draw a
box on the plot. Then click on **Options \>\> Zoom in**. The plot will
be redrawn to show only the selected region, and the **Options \>\> Zoom
in** menu item will change to **Options \>\> Zoom out**.

To zoom back out, click on **Options \>\> Zoom out**. The plot will be
redrawn to look as it originally did, and the **Options \>\> Zoom out**
menu item will revert to **Options \>\> Zoom in**.

*NOTE: Zooming in and out can also be done by changing the axis limits
in the* **Chart Properties Manager**.

4.  []{#__RefHeading___Toc135190953 .anchor}***Menu Items***

-   **Actions \>\> Show Plot Mgr** menu item: Opens up the
    **Verification Plot Definition Manager** and makes it the active
    window.

-   **Actions \>\> Show Group Mgr** menu item: Makes the **Verification
    Group Manager** visible and active.

-   **Actions \>\> Show Location Mgr** menu item: Makes the
    **Verification Location Manager** visible and active.

-   **Actions \>\> Show Pairs Data** menu item: Open up the
    **Verification Pairs Data Manager** and makes it active.

-   **Actions \>\> Save Image File** menu item: Open up a file browser
    to allow the user to select a jpeg or png file name to which to save
    the image. If the file name does not end in either a ".jpg" or
    ".jpeg" extension for JPEG images, or a ".png" extension for PNG
    images, then the file type will be unrecognized and no image will be
    saved.

-   ![](media/image7.jpeg){width="2.540277777777778in"
    height="1.5618055555555554in"}**Actions \>\> Close** menu item:
    Close the **IVP Data Display** window.

-   **Actions \>\> Exit** menu item: Exit the IVP program.

-   **Options \>\> Show line y=x (Hide line y=x)** menu item: Draws the
    line y = x (i.e. forecast = observation) on the plot. The line will
    be green.

-   **Options \>\> Edit Chart Properties** menu item: Opens up the
    **Chart Properties Manager** for this chart and makes it active,
    enabling a user to change the properties of this chart.

-   **Options \>\> Save Properties Template** menu item: Opens up a file
    browser to save the current chart properties for this chart to a
    template file.

-   **Options \>\> Load Properties Template** menu item: Opens up a file
    browser to select a template file and apply the settings to the
    chart properties for this chart.

-   ![](media/image8.jpeg){width="2.540277777777778in"
    height="1.5618055555555554in"}**Options \>\> Zoom in (Zoom out)**
    menu item: Zooms in on a region of the plot, or zooms out to the
    original plot. See Section 11.3, above.

    ![](media/image9.jpeg){width="4.722222222222222in"
    height="4.222222222222222in"}

15. []{#__RefHeading___Toc135190955 .anchor}***Verification Pairs Data
    Manager***

The **Verification Pairs Data Manager** shows in a table the
forecast-observed pairs currently displayed and visible in the **IVP
Data Display**. Each row corresponds to a record of the vfypairs table
in the archive database. Each field of the record is displayed as a
column of the table in the **Verification Pairs Data Manager**. The
columns are:

-   Index: The original row number. After sorting the table, the
    original ordering of the rows can be reacquired by sorting by the
    index column.

-   Vis: If the point displayed on the row is currently visible in the
    **IVP Data Display** plot, then this column value will be 'Y'.
    Otherwise, the value will be 'N'. This is useful if the user has
    zoomed in on a region of the plot.

-   location: 8-character location id

-   pe: physical element (SHEF *PE*dtsep)

-   dur: duration (SHEF pe*D*tsep)

-   idur: integer representation of the duration

-   fts: forecast type source (SHEF ped*TS*ep)

-   e: extremum (SHEF pedts*E*p)

-   p: probability (SHEF pedtse*P*)

-   validtime: forecast valid time

-   basistime: forecast basis time

-   ots: observation type source (SHEF ped*TS*ep)

-   obstime: observed time

-   fcst value: forecast value (for stage data, the unit is feet)

-   obs value: observed value

-   quality code: the integer quality code stored with the forecast
    value

    1.  []{#__RefHeading___Toc135190956 .anchor}***Sorting the Rows***

To sort the table by a given column, just click on the header for that
column. For example, to sort alphabetically by location, click on the
"location" column header. The first time the table is sorted by a
particular column, the sort will be in ascending order. The second time,
it will be in descending order. After that, the sort order will continue
to alternate between ascending and descending.

To retrieve the original order of the rows, sort by the "Index" column.

2.  []{#__RefHeading___Toc135190957 .anchor}***Visible Rows***

If a row is visible in the current **IVP Data Display** plot, then the
background of the row will include colors, such as yellow, green, and
gray, and the "Vis" column will be 'Y'. If the row is not visible, the
background will be white, and the "Vis" column will be 'N'. To more
easily focus on visible or not-visible rows, sort by the "Vis" column.

The rows that are visible and displayed in color within the table
changes whenever the user zooms-in on a region of the **IVP Data
Display** plot.

3.  []{#__RefHeading___Toc135190958 .anchor}***Buttons***

-   **Save to File** button: Opens up a file browser to allow the user
    to select the file to which to save the data within this table. The
    format of the file will be that of a pairs file (see the *IVP Batch
    Program User's Manual for Verification*).

-   **Close** button: Close the **Verification Pairs Data Manager**.

![](media/image10.jpeg){width="6.168055555555555in"
height="3.604861111111111in"}

16. []{#__RefHeading___Toc135190959 .anchor}***Verification Plot
    Definition Manager***

The **Verification Plot Definition Manager** provides tools to manage
parameters of an IVP statistic plot. Parameters of the plot include the
following:

-   primary statistics: statistics displayed against the left hand
    y-axis of the generated plot.

-   primary plot type: the plot type of the primary statistics (line,
    scatter, or bar plot).

-   secondary statistics: statistics displayed against the right hand
    y-axis of the generated plot.

-   secondary plot type: the plot type of the secondary statistics.

-   x-axis variable: the variable to display along the x-axis (location,
    analysis interval, lead time interval, observed category, forecast
    category).

-   analysis interval used: the analysis interval used in the plot, as
    dictated by the x-axis variable.

-   Leadtime interval used: the lead time interval used in the plot, as
    dictated by the x-axis variable.

-   observed category: the observed category used to restrict what pairs
    are included in the analysis.

-   forecast category: the forecast category used to restrict what pairs
    are included in the analysis.

Each parameter is managed in a graphical way, using components of the
**Verification Plot Definition Manager**.

*NOTE: It is recommended that, if you have an appropriate batch file
already constructed that defines the parameters of the graphic you wish
to produce, then you should use the **Load From Batch** button to load
the parameters from that file. This is easier than specifying each field
manually.*

1.  []{#__RefHeading___Toc135190960 .anchor}***Primary and Secondary
    Statistics***

The *primary statistics* and *secondary statistics* define the
statistics displayed on the left-hand and right-hand y-axes,
respectively. All of the statistics displayed against a single axis must
have the same scale. For example, the mean absolute error and mean error
can be displayed against the same y-axis, because both have the same
units (feet, in the case of stage data). However, mean absolute error
and probability of detection cannot be displayed against the same
y-axis, because the probability of detection is a probability, between 0
and 1.

Selecting primary and secondary statistics is done through an **IVP
Statistic Chooser Manager**, which ensures only statistics with the same
scale can be displayed against a single axis. The **IVP Statistic
Chooser Manager** is opened by clicking on the corresponding **Choose
Stat** button. The **Primary Statistics** text field displays the
currently selected primary statistics, and the **Secondary Statistics**
text field displays the secondary statistics.

2.  []{#__RefHeading___Toc135190961 .anchor}***Primary and Secondary
    Statistic Plot Types***

The *plot type* for a statistic can be one of the following:

-   Scatter: Statistic values are plotted as points. A scatter plot is
    good when the x-axis does not show trends (i.e. location).

-   Line: statistic values are plotted as points and are connected by
    lines. A line plot is good when the x-axis does show trends (i.e.
    analysis interval, lead time interval).

-   Bar: statistic values are plots as vertical bars rising to the
    statistic value from the bottom of the plot. A bar plot is good when
    the y-axis value is a count, such as the statistic sample size or
    average lead time of detection.

The primary statistic plot type is chosen from the **Primary Plot Type**
choice box. The secondary statistic plot type is chosen from the
**Secondary Plot Type** choice box.

1.  []{#__RefHeading___Toc135190962 .anchor}***X-Axis Variable***

The *x-axis variable* for a plot can by one of the following:

-   Location: Statistic values are plotted for each location separately
    in the order in which the locations appear within the **Verification
    Location Manager** (top to bottom).

-   Analysis Interval: Statistic values are plotted against each
    interval defined by the start time, end time, and time step of the
    **Verification Group Manager**.

-   Lead Time Interval: Statistic values are plotted against each
    interval defined by the lead time start, lead time end, and lead
    time step of the **Verification Group Manager**.

-   Observed Category: Statistic values are plotted against each
    category defined for the observed value in the **Verification
    Location Manager**.

-   Forecast Category: Statistic values are plotted against each
    category defined for the forecast value in the **Verification
    Location Manager**.

    1.  []{#__RefHeading___Toc135190963 .anchor}***Analysis Interval and
        Lead Time Interval Used***

The *analysis interval used* and the *lead time interval used* inform
the user what intervals are being used to compute the statistics. For
example, if the x-axis variable is location, observed category, or
forecast category, then the analysis interval used will be the entire
interval, from start time to end time. Also, the lead time interval used
will be the entire interval, from lead time start to lead time end. If
the x-axis variable is analysis interval, then the analysis interval
used will be "All user-defined intervals will be used." If the x-axis
variable is lead time interval, then the lead time interval used will be
"All user-defined intervals will be used."

2.  []{#__RefHeading___Toc135190964 .anchor}***Observed Category and
    Forecast Category***

The *observed category* and *forecast category* define any restrictions
on the data to use in verification, based on the observed or forecast
value. The choice is made by selecting an item from the **Observed
Category** choice box or **Forecast Category** choice box. The choices
are the following:

-   Do Not Use: Do not use this category to add a restriction to the
    data. If chosen, then the corresponding choice box will be disabled
    and the other category choice box will become enabled, allowing a
    selection to be made.

-   All Categories Combined/Use Only Category: If only one category is
    defined, then data pairs must be in that category in order to be
    used in the computations. If multiple categories are defined, then
    the data must be in at least one of the categories in order to be
    used in the computations.

-   Category #: The data pair must be in the category chosen for the
    corresponding data value (i.e. observed or forecast) in order to be
    used in the computations.

If the x-axis variable is observed category or forecast category, then
the two choice boxes are disabled, since the x-axis variable dictates
how the categories are used. Only one of the two choice boxes can be
active at a time. If the observed category is Do Not Use, then the
forecast category must be chosen as something else, and vice versa.

*NOTE: To make it so that all data is used in computing statistic
regardless off category, one of the two choice boxes must be "Do Not
Use" and the other must be "All Categories Combined"/"Use Only
Category". Also, the categories must include boundary strings* MIN *and*
MAX*.*

1.  []{#__RefHeading___Toc135190965 .anchor}***Graph Template File***

The *graph template file* specifies properties (that are not default) of
the chart to create, including labels, fonts, colors, sizes, axis mins
and maxes, and others. For more information on the graph template file,
see the section on the **Chart Properties Manager** (Section 21). The
text field specifies the template file to use when the **IVP Statistic
Display** is first created. If the text field is empty or "NONE", then
default chart properties will be used to construct the display.

2.  []{#__RefHeading___Toc135190966 .anchor}***Buttons***

-   **Primary Statistic** **Choose** **Stat** button: Opens up the IVP
    Statistic Chooser Manager for selecting the primary statistics.

-   **Secondary Statistic** **Choose** **Stat** button: Opens up the IVP
    Statistic Chooser Manager for selecting the secondary statistics.

-   **Graph Template Select** button: Opens up a file browser for
    selecting the template file.

-   **Load From Batch** button: Opens up a file browser that can be used
    to find a previously created batch file for the IVP Batch Program
    and set the parameters within this **Verification Plot Definition
    Manager** according to that file. It will read parameters until the
    first graphic is generated (via a GEN_GRAPH batch file action; see
    the *IVP Batch Program User's Manual*) and sets the parameters
    herein according to those read in.

-   **Create Plot** button: Creates the **IVP Statistic Display**
    showing the plot specified by this **Verification Plot Definition
    Manager**. If no primary statistics are selected to be displayed,
    then the plot will not be constructed.

![](media/image11.jpeg){width="3.9791666666666665in"
height="3.6326388888888888in"}

17. []{#__RefHeading___Toc135190967 .anchor}***IVP Statistic Chooser
    Manager***

The **IVP Statistic Chooser Manager** is used to select primary or
secondary statistics for the IVP statistic plot. The statistics are
placed in groups, so that all of the statistics in a group have the same
scale or unit. Each group is given a master check box, with an
appropriate name, so that if that checkbox is checked, then all
statistics within the group will be checked.

The statistic groups are as follows (the short hand notation for the
statistic in the IVP Batch Program output file is given in parentheses):

-   Error Statistics: root mean squared error (RMSE), maximum error
    (MAXERR), mean absolute error (MAE), and mean error (ME)

-   Categorical Statistics: probability of detection (POD), hydrologic
    false alarm rate (HFAR), traditional false alarm rate (TFAR), over
    forecast rate (OFR), under forecast rate (UFR), and critical success
    index or Gilbert score (CSI)

-   Quantiles and Extremes: minimum (MIN), 25% quantile (QUAN 0.25),
    median (MEDIAN), 75% quantile (QUAN 0.75), and maximum (MAX)

-   Lead Time Statistics: average lead time of detection (AVGLDTM)

-   Sample Size: statistic sample size (NUM SAMPLES)

Statistics are chosen by clicking on checkboxes. If a statistic of one
group is checked, then all statistics of all *other* groups will be
unchecked. Only statistics of one group can be checked at a time.

The statistics are defined in Appendix B.

1.  []{#__RefHeading___Toc135190968 .anchor}***Buttons***

-   **OK** button: Close the window and update the corresponding text
    field of the **Verification Plot Definition Manager** to list the
    chosen statistics.

-   **Cancel** button: Close the window and do nothing else.

![](media/image12.jpeg){width="5.816666666666666in"
height="2.828472222222222in"}

18. []{#__RefHeading___Toc135190969 .anchor}***IVP Statistic Display***

The **IVP Statistic Display** graphically displays statistics calculated
by the IVP software according to parameters of the **Verification Plot
Definition Manager**, **Verification Group Manager**, and **Verification
Location Manager**. The legend defines what statistics are displayed,
with the left (or only) column of the legend displaying the primary
statistics, and the right column displaying the secondary statistics.

Associated with the chart in the IVP Statistic Display window are chart
properties, including the chart title, axis labels, legend title, fonts,
colors, etc. Default values for the chart properties are constructed
based on the data and user settings. These properties can be customized
via the **Chart Property Manager**. They can also be loaded from an
existing template file via **Actions \>\> Load Chart Properties**.

The following features are the same as those in the **IVP Data Display**
(see Section 11):

-   **Data Tracker** (Section 14.2)

-   **Actions \>\> Edit Chart Properties** (Section 14.4)

-   **Actions \>\> Save Properties Template** (Section 14.4)

-   **Actions \>\> Load Properties Template** (Section 14.4)

-   **Actions \>\> Save Image File** (Section 14.4)

-   **Actions \>\> Close** (Section 14.4)

Additional features are given below.

1.  []{#__RefHeading___Toc135190970 .anchor}***Menu Items***

In addition to the menu items identical to those in the **IVP Data
Display**, the following menu items are available:

-   **Actions \>\> Save Batch File** menu item: Opens up an **IVP Batch
    File Save Manager** for saving a batch file to reconstruct the image
    within the **IVP Statistic Display**.

-   **Actions \>\> Show Data**: Opens up **IVP Statistics Data Viewer**
    for this plot.

![](media/image13.jpeg){width="2.3402777777777777in"
height="1.757638888888889in"}

![](media/image14.jpeg){width="5.9534722222222225in"
height="4.117361111111111in"}

19. []{#__RefHeading___Toc135190971 .anchor}***IVP Statistics Data
    Viewer***

The **IVP Statistics Data Viewer** shows in a table the statistics
currently displayed in the **IVP Statistic Display**. The columns of the
table are as follows:

-   Index: An indexing number shows the order in which the rows were
    initially given in the table.

-   \<X-axis Variable\>: The value of the variable displayed along the
    x-axis. This column has a pink background.

-   \<statistic values\>: One column per statistic displayed on the
    plot. These columns have a gray background.

    1.  []{#__RefHeading___Toc135190972 .anchor}***Buttons***

```{=html}
<!-- -->
```
-   **Save to File** button: Opens up a file browser to allow the user
    to select the file to which to save the data within this table. The
    file produced will include group information at the top and the
    table at the bottom.

-   **Close** button: Closes the **IVP Statistics Data Viewer**.

![](media/image15.jpeg){width="4.417361111111111in"
height="1.8090277777777777in"}

20. []{#__RefHeading___Toc135190973 .anchor}***IVP Batch File Save
    Manager***

The **IVP Batch File Save Manager** is used to complete specification of
the batch file to construct that will allow for the IVP Batch Program to
reconstruct the image in the corresponding **IVP Statistic Display**. To
build the batch file, the names of the image file and data file to
create must be specified, the template file must be specified (if used),
and the batch file name must be given. The *image file* is the file to
contain the image in the **IVP Data Display**, being either a PNG or
JPEG file. The *data file* is the text file to contain the contents of
the **IVP Statistics Data Viewer**. The *template file* is the file
containing information about the chart properties to use when
constructing the chart. The *batch file* is the batch file for the IVP
Batch Program that will reconstruct the image file, and optionally the
data file.

1.  []{#__RefHeading___Toc135190974 .anchor}***File Name Text Fields***

There are three file names that must be specified:

-   Image file: The name of the image file the IVP Batch Program is to
    construct using this batch file. Unless specified, this file is
    assumed to be in the directory \$(vsys_input). This file is
    required.

-   Data file: The name of the data file the IVP Batch Program is to
    construct using this batch file. Unless specified, this file is
    assumed to be in the directory \$(vsys_output)/\$LOGNAME. This file
    is optional. If it is not used, the data file should be left blank
    or set to "NONE".

-   Template file: The name of the graph template file the IVP Batch
    Program is to use when constructing the chart. Unless otherwise
    specified, this file is assumed to be in the directory
    \$(vsys_files)/\$LOGNAME/templates. This file is optional. If it is
    not to be used, then the radio button **Use Defaults** should be
    selected, or the template file should be left blank or set to
    "NONE". If the template file needs to be created, the button
    **Create New Template** should be selected. If the template file
    already exists, the button **Use Existing Template** should be
    selected.

-   Batch File Name: The name of the batch file, itself. Unless
    specified, this file is assumed to be below the directory
    corresponding to the vsys_input apps-defaults token.

    1.  []{#__RefHeading___Toc135190975 .anchor}***Buttons***

```{=html}
<!-- -->
```
-   Select buttons: Each select button opens up a file browser to choose
    the name of the file. Clicking on the **Open** or **Save** button of
    the browser will close the window and update the corresponding text
    field. Clicking on **Cancel** will close the window, with no
    updates.

-   **OK** button: Closes this window and creates the file, using
    parameters specified in the **Verification Group Manager**,
    **Verification Location Manager**, **Verification Plot Definition
    Manager**, and in this window. The template file will also be
    created if the **Create New Template** radio button is selected.

-   **Cancel** button: Closes this window and do nothing else.

![](media/image16.jpeg){width="5.2868055555555555in"
height="2.977777777777778in"}

21. []{#__RefHeading___Toc135190976 .anchor}***IVP Chart Properties
    Manager***

The **IVP Chart Properties Manager** window is used to customize the
appearance of a displayed chart. It allows for customizing the following
*chart components*:

-   Chart Title: The text, font, and color of the chart title can be
    changed.

-   X-Axis Title: The text, font, and color of the x-axis title can be
    changed.

-   Primary Y-Axis Title: The text, font, and color of the primary
    (left-hand) y-axis title can be changed.

-   Secondary Y-Axis Title: The text, font, and color of the secondary
    (right-hand) y-axis title can be changed.

-   Legend Title: The text and font of the legend title can be changed.

-   Legend Components: The text and color of the legend components can
    be changed. The font is always equal to the legend title font.

-   Tick Mark Font: The fonts of the tick marks used for all of the axes
    can be changed. All three axis tick mark labels are in the same
    font.

-   X-Axis Limits or Labels: If the x-axis is numerical (as with the
    **IVP Data Display**), then the minimum and maximum values of the
    axes can be changed. If the x-axis is by label (as with the **IVP
    Statistics Display**), then the labels along the x-axis can be
    changed.

-   X-Axis Annotation: The direction of printing for the x-axis labels
    can be changed.

-   Primary Y-Axis Limits: The minimum and maximum for the primary
    y-axis can be changed.

-   Secondary Y-Axis Limits: The minimum and maximum for the secondary
    y-axis can be changed.

-   Chart Dimensions: The dimensions (height and width) of the window
    containing the chart can be changed.

The **IVP Chart Properties Manager** window is organized into two
panels, each of which has a tab on it: **Chart Titles** and **Chart
Others**. These two panels further contain other tabbed panels. The
organization of these other tabbed panels, and the chart component each
panel is associated with, is as follows:

> **Chart Title (Chart Titles)**: Chart Title
>
> **Y-Axis 1 Title (Chart Titles)**: Primary Y-Axis Title
>
> **Y-Axis 2 Title (Chart Titles)**: Secondary Y-Axis Title
>
> **X-Axis Title (Chart Titles)**: X-Axis Title
>
> **Legend Title (Chart Titles)**: Legend Title
>
> **Legend (Chart Titles)**: Legend Components
>
> **Tick Mark Font (Chart Titles)**: Tick Mark Font
>
> **Chart Dimension (Chart Others)**: Chart Dimensions
>
> **Primary Y-Axis (Chart Others)**: Primary Y-Axis Limits
>
> **Secondary Y-Axis (Chart Others)**: Secondary Y-Axis Limits
>
> **X-Axis (Chart Others)**: X-Axis Limits or Labels
>
> **X-Axis Annotation (Chart Others)**: X-Axis Annotation

Each tabbed panel provides tools to edit the corresponding chart
component of the chart. The tabbed panels are described below, with a
small screenshot of the appearance of the panel within the **Chart
Property Manager** window.

1.  []{#__RefHeading___Toc135190977 .anchor}***Template Files***

A *template file* is a file that records the properties corresponding to
the **Chart Property Manager**. Specifically, it records the properties
of a graphic that differ from the default properties. This file can be
created by clicking on **Options \>\> Save Properties Template** in the
**IVP Data Display**, clicking on **Actions \>\> Save Properties
Template** in the **IVP Statistics Display**, or by choosing to create a
new template within the **IVP Batch File Save Manager**. The settings
within a template file can be applied to a chart by clicking on
**Options \>\> Load Properties Template** in the **IVP Data Display**,
clicking on **Actions \>\> Load Properties Template** in the **IVP
Statistics Display**, or by specifying the template file in the
**Verification Plot Definition Manager**.

Template files allow for a user to customize the appearance of a
graphic, save the chart properties, and then apply those properties when
generating other images, both in the IVP and IVP Batch Program.

2.  []{#__RefHeading___Toc135190978 .anchor}***Default Values***

Every property of a chart has a default value associated with it.
Initially, all chart properties are assumed to be their default. For all
tabbed panels except for those associated with labels, the radio button
**Default** specifies if a default value is being used. For those
associated with labels, the radio button **Default (Text & Font)**
specifies if both the text and font of the label are default, and
**Default (Text Only)** specifies if the text is default but the font is
not.

In any case, if a change is made in a panel, the corresponding
**Default** radio button will be deselected. To reacquire the default
value, the user may click on the corresponding **Default** radio button.
Note that, for labels, clicking on **Default (Text Only)** will only
recover the default label text. Clicking on **Default (Text & Font)**
will recover both the default label text and font.

3.  []{#__RefHeading___Toc135190979 .anchor}***All Titles (Chart
    Titles)***

The **Chart Title**, **X-Axis Title**, **Y-Axis 1 Title**, **Y-Axis 2
Title**, and **Legend Title** panels, as well as the **Tick Mark Font**
panel, all have the same basic appearance, so only the **Chart Title**
panel is shown here. The exceptions are that neither the **Legend
Title** panel nor the **Tick Mark Font** panel have a **Text Color**
button. Also, the text within the **Tick Mark Font** panel is ignored;
it is only there to show what the font looks like for numbers.
Furthermore, only the chart title on a chart can display multiple lines
of text. For the others, if multiple lines of text are present, they
will be merged into one line (new-lines will be removed) prior to use.

The Font is specified by choosing a font name from the **Font** choice
box, a size from the **Font Size** choice box, and selecting bold or
italics using the **Bold** and **Italics** buttons. The text color is
chosen by clicking on the **Text Color** button and selecting a color
from the **Choose Color** window that pops up.

The text area will display the text as it will be seen on the chart.

1.  *Buttons*

-   **Bold** button: Selects the font to be bold, if toggled (i.e.
    shaded).

-   **Italics** button: Selects the font to be italics, if toggled.

-   **Text Color** button: Opens up a **Choose Color** window for
    selecting the color of the text.

![](media/image17.jpeg){width="5.629166666666666in"
height="3.317361111111111in"}

![](media/image18.jpeg){width="3.198611111111111in"
height="3.1145833333333335in"}

1.  []{#__RefHeading___Toc135190980 .anchor}***All Numerical Axis Limits
    (Chart Others)***

The **X-Axis** limits (for the **IVP Data Display**), **Primary Y-Axis**
limits, **Secondary Y-Axis** limits, and **Chart Dimensions** panels all
have the same basic appearance, so only the **Primary Y-Axis** limits
panel is shown here. Each contains two sliders: one for the upper limit
(or chart height) and one for the lower limit (or chart width). The
value of a limit (or dimension) can be changed by using the slider
(click on it and drag, or click to either side of it) or by entering the
desired number into the text field and pressing \<Enter\>.

*NOTE: To specify a limit outside the slider's minimum and maximum
range, type the limit value into the text field and press* \<Enter\>*.
Do not use the slider in these cases.*

The slider itself can also be customized by clicking on **Reset**, which
will pop up a **Reset Slider** window. This window allows for setting
the slider's minimum value, maximum value, initial value, major tick
spacing, minor tick spacing, and scale.

*NOTE: Any changes to the slider will be lost when the window closes.*

1.  []{#__RefHeading___Toc135190981 .anchor}*Buttons*

-   **Reset** button: Opens up the Reset Slider window, allowing for
    customizing the slider.

![](media/image19.jpeg){width="5.629166666666666in"
height="3.317361111111111in"}

![](media/image20.jpeg){width="2.079861111111111in"
height="1.8138888888888889in"}

1.  []{#__RefHeading___Toc135190982 .anchor}***Legend (Chart Titles)***

The Legend panel is used to change the color and text of legend
components. Each component is provided in a **Legend Component**
**List**, with the background color being the color used within the
legend. To change a legend component, do the following:

1.  Click on the corresponding item in the **List**.

2.  Type the name of the legend component into the text field.

3.  Click on **Update**. A **Choose Color** window will pop up.

4.  Select the desired color and click on **OK**.

After completing the last step, the **Legend Component List** should
change to reflect the new component.

*NOTE: The shapes used in the charts cannot be edited via the* **Chart
Property Manager***. However, they can be edited via the*
IVPStatChartPrimarySymbols *and* IVPStatChartSecondarySymbols *settings
of the System Settings file (see Section A.3.1).*

1.  []{#__RefHeading___Toc135190983 .anchor}*Buttons*

-   **Update** button: Opens up a **Choose Color** window for choosing
    the legend component color.

![](media/image21.jpeg){width="5.629166666666666in"
height="3.317361111111111in"}

1.  []{#__RefHeading___Toc135190984 .anchor}***X-Axis Labels (Chart
    Others)***

The **X-Axis (Chart Others)** panel is used to change the x-axis tick
mark labels for the **IVP Statistics Display**. The current x-axis
labels are provided in an **X-Axis Label List**. To change an x-axis
label, do the following:

1.  Click on the corresponding item in the **List**.

2.  Type in the desired label in the text field.

3.  Click on **Update**.

After completing the last step, the **X-Axis Label List** should be
updated to display the new label.

1.  []{#__RefHeading___Toc135190985 .anchor}*Buttons*

-   **Update** button: Updates the selected x-axis label in the **X-Axis
    Label List** to be the text currently in the text field.

![](media/image22.jpeg){width="6.49375in" height="3.8270833333333334in"}

22. []{#__RefHeading___Toc135190986 .anchor}***Tips and Information***

This section provides tips about how to operate the IVP and how the
windows interact.

1.  []{#__RefHeading___Toc135190987 .anchor}***System Settings File***

An IVP system settings file exists allowing for customization of the
various windows and plots. It is described in Appendix A. If you need to
change the size of windows, fonts used, column names in tables, or other
items, check the appendix to see if it can be done and edit the system
settings file appropriately. The system settings file is located at the
following directory:

\$(vsys_dir)/app-defaults/IVP_SYSTEM_FILE.txt

where \$(vsys_dir) is the directory corresponding to apps-defaults token
vsys_dir.

2.  []{#__RefHeading___Toc135190988 .anchor}***NOAA and NWS Logos***

The NOAA logo is displayed in the upper left corner of any graphic
within the IVP, and the NWS logo is displayed in the upper right corner.
These images are drawn from these two jpeg files, respectively:

\$(vsys_dir)/app-defaults/ivp_noaa.jpg

\$(vsys_dir)/app-defaults/ivp_nws.jpg

If you need to change the logos, then just edit these files. If you need
to remove the logos, then just remove these files or change their names.

3.  []{#__RefHeading___Toc135190989 .anchor}***Performance***

For large scale (tens of thousands of points or more) verification
projects, the IVP can be a slow, memory intensive program. Two factors
control the performance of the IVP: (1) system memory and (2) database
usage. If the IVP performs very poorly (either being very slow or
running out of memory), then you should take the following actions, in
order, until the problem is resolved:

-   Turn off the SHEF decoders and any other programs that are accessing
    the archive database.

-   Run IVP on a computer with better hardware (i.e. more memory and
    faster processor) that can see the archive database (with the
    rax_pghost and pguser apps-defaults tokens being changed
    appropriately). See the next Section for how to setup IVP to run on
    an AWIPS LX machine.

-   Decrease the size of the analysis interval in order to lessen the
    number of forecast-observed pairs included in the analysis. If the
    product must be produced for the larger, original analysis interval,
    then save a batch file using this smaller run (via **Actions \>\>
    Save Batch File** in the **IVP Statistics Display**), edit the batch
    file to be for the desired analysis interval (see the *IVP Batch
    Program User's Manual For Verification*), and run the IVP Batch
    Program with the batch file. The IVP Batch Program is usually
    capable of handling more pairs than the IVP.

-   Employ the --m option of the ivp script to increase the amount of
    memory in the Java heap space. Run ivp --hh to see how to use this
    option.

    1.  []{#__RefHeading___Toc135190990 .anchor}***Running IVP on AWIPS
        LX Machines***

Generally, the AWIPS lx machines have greater resources than the archive
machines and can see the archive database. To run the IVP software on an
AWIPS lx machine, do the following:

1.  Tar up the entire directory structure under the directory
    corresponding to apps-defaults token vsys_dir:

tar --cvf ivp.tar \$(get_apps_defaults vsys_dir)/\*

2.  Put the tar file on the system where the IVP software is to run.

3.  Select the directory where the software is to be located. The
    directory /awips/hydroapps/lx/rfc/verify may be a good choice. This
    directory will be denoted \<basedir\> below.

4.  Untar the tar file into that directory:

cd \<basedir\>

tar --xvf ivp.tar

5.  In your apps-defaults site file, change the token vsys_dir to point
    to that directory, or do the following:

export vsys_dir=\<basedir\>

6.  Set the rax_pghost apps-defaults token to be your archive (ax)
    machine. This usually works:

export rax_pghost=ax

7.  Set the pguser apps-defaults token to be a valid user.

8.  Make sure all apps-defaults tokens needed by the software are
    present on the AWIPS LX system.

At this point the software should execute using the command given in
Section 3.

1.  []{#__RefHeading___Toc135190991 .anchor}***Debugging***

The apps-defaults token vsys_debug can be used to acquire debugging
output from all of the verification software (IVP, IVP Batch Builder,
IVP Batch Program, Vfyruinfo Editor, etc.). To set it, export it as
follows:

export vsys_debug=40

As the value of the vsys_debug goes up, the amount of output increases.
Here are some common settings and what they will include in the screen
output:

> 0 No screen output, except what is output from the wrapper scripts
>
> 1 Standard (default)
>
> 20 Additional progress info, including locations as they are being
> processed
>
> 40 Database queries and number of records found will be output to the
> screen
>
> 128 All output will be given, including every individual data pair
> used or created (not recommended)

[]{#__RefHeading___Toc135190992 .anchor}***Appendix A: System Settings
File***

[]{#__RefHeading___Toc135190993 .anchor}***A.1 Overview***

The IVP windows appear different on different operating systems.
Therefore, it is necessary to allow for user to adjust the default
display to accommodate multiple operating systems. The system settings
file allows the user to change the appearance of the windows. This
appendix describes that file.

[]{#__RefHeading___Toc135190994 .anchor}***A.2 Location***

The following apps-defaults token must be set in order for the system
settings file to be found:

vsys_dir

The file is assumed to exist in the directory \$(vsys_dir)/app-defaults
with the name "IVP_SYSTEM_FILE.txt".

[]{#__RefHeading___Toc135190995 .anchor}***A.3 System Settings File***

The system settings file uses the same file format as an IVP Batch
Program batch file. However, rather than being read in sequentially, it
is processed and stored at run-time so that each GUI component can
extract the settings that pertain to it. If a command is given more than
once within the system settings file, the last instance will over-ride
the earlier instances.

[]{#__RefHeading___Toc135190996 .anchor}*A.3.1 Commands*

Commands, indicated in **bold**, set parameter values that control the
appearance of various IVP windows. The \<value\> on the line of the
system settings file that contains the command is usually restricted,
depending on that command. Acceptable Values are listed for each
command. The default value shown is the value assumed if the system
settings command is not found.

The following are IVP system settings commands:

**BatchSaveWindowHeight = \<number\>**

*Description*: Controls the pixel height of the **IVP Batch File Save
Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 225

**BatchSaveWindowWidth = \<number\>**

*Description*: Controls the pixel width of the **IVP Batch File Save
Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 600

**ButtonFont = \<font\>**

*Description*: Controls the font used on button labels in the windows.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Courier;plain;12\"

**CategoryEditorWindowHeight = \<number\>**

*Description*: Controls the pixel height of the **Forecast Category
Editor** and **Observed Category Editor**.

*Acceptable Values*: positive integer value.

*Default Value*: 250

**CategoryEditorWindowWidth = \<number\>**

*Description*: Controls the pixel width of the **Forecast Category
Editor** and **Observed Category Editor**.

*Acceptable Values*: positive integer value.

*Default Value*: 320

**ChartBGColor = \<color\>**

*Description*: Controls the background color of the charts.

*Acceptable Values*: Any color string (see Section A.5)

*Default Value*: white

**ChartCatLineColor = \<color\>**

*Description*: Controls the category line color of the **IVP Data
Display**.

*Acceptable Values*: Any color string (see Section A.5)

*Default Value*: blue

**ChartFGColor = \<color\>**

*Description*: Controls the foreground color of the charts.

*Acceptable Values*: Any color string (see Section A.5)

*Default Value*: black

**ChartGridColor = \<color\>**

*Description*: Controls the grid color of the charts.

*Acceptable Values*: Any color string (see Section A.5)

*Default Value*: "160,160,160"

**ChartPairsColor = \<color\>**

*Description*: Controls the scatter point color in the **IVP Data
Display**.

*Acceptable Values*: Any color string (see Section A.5)

*Default Value*: red

**ChartShownLocationColor = \<color\>**

*Description*: Controls the scatter point color in the **IVP Data
Display** for a highlighted location.

*Acceptable Values*: Any color string (see Section A.5)

*Default Value*: yellow

**ChartXYLineColor = \<color\>**

*Description*: Controls the line y = x color in the **IVP Data
Display**.

*Acceptable Values*: Any color string (see Section A.5)

*Default Value*: green

**ChartZLineColor = \<color\>**

*Description*: Controls the color in the chart.

*Acceptable Values*: Any color string (see Section A.5)

*Default Value*: dark_gray

**ChartWindowHeight = \<number\>**

*Description*: Controls the default height of a **Chart** window.

*Acceptable Values*: positive integer value.

*Default Value*: 500

**ChartWindowWidth = \<number\>**

*Description*: Controls the default width of a **Chart** window.

*Acceptable Values*: positive integer value.

*Default Value*: 650

**DateChooserWindowHeight = \<number\>**

*Description*: Controls the default height of the **Choose Date/Time**
window.

*Acceptable Values*: positive integer value.

*Default Value*: 275

**DateChooserWindowWidth = \<number\>**

*Description*: Controls the default width of a **Choose Date/Time**
window.

*Acceptable Values*: positive integer value.

*Default Value*: 320

**DestroyOldDisplayUponCreate = \<ON/OFF\>**

*Description*: If set to **ON**, the IVP will destroy the old **IVP Data
Display** before creating a new one.

*Acceptable Values*: **ON** or **OFF**.

*Default Value*: **OFF**, meaning that if the display fails to create,
the old display will still be available.

**IVPDataChart\*Buffer = \<number\>**

*Description*: Controls a buffer around the **IVP Data Display**, moving
the position of the plot within the window.

*Acceptable Values*: positive integer value.

*Default Value*: 0

**IVPDataChartWindowHeight = \<number\>**

*Description*: Controls the default height of the **IVP Data Display**.

*Acceptable Values*: positive integer value.

*Default Value*: 700

**IVPDataChartWindowWidth = \<number\>**

*Description*: Controls the default width of the **IVP Data Display**.

*Acceptable Values*: positive integer value.

*Default Value*: 800

**IVPDataTableMgrWindowHeight = \<number\>**

*Description*: Controls the default height of the **Verification Pairs
Data Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 400

**IVPDataTableMgrWindowWidth = \<number\>**

*Description*: Controls the default width of the **Verification Pairs
Data Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 700

**IVPStatChart\*Buffer = \<number\>**

*Description*: Controls a buffer around the **IVP Statistics Display**,
moving the position of the plot within the window.

*Acceptable Values*: positive integer value.

*Default Value*: 0

**IVPStatChartPrimaryColors = "\<color\>;\<color\>;\..."**

*Description*: Controls the colors used for primary statistics within
the **IVP Statistics Display**. If more statistics are displayed than
colors listed, they will be plotted using system defaults.

*Acceptable Values*: list of colors, each of which satisfies Appendix
A.5 requirements

*Default Value*: "red;blue;green;yellow;orange;magenta"

**IVPStatChartPrimaryLineSize = \<number\>**

*Description*: Controls the width of lines drawn for primary statistics
in the **IVP Statistics Display**.

*Acceptable Values*: positive integer value.

*Default Value*: 1

**IVPStatChartPrimarySymbols = "\<symbol\>;\<symbol\>;\..."**

*Description*: Controls the symbols used for primary statistics within
the **IVP Statistics Display**. If more statistics are displayed than
symbols listed, they will be plotted using system defaults.

*Acceptable Values*: list of symbols, each of which is one of the
following: "box", "circle", "cross", "diamond", "none", "square",
"start", "triangle", "vert_line" (vertical line).

*Default Value*: "dot;box;triangle;diamond;square;star;cross"

**IVPStatChartPrimarySymbolSize = \<number\>**

*Description*: Controls the size of symbols drawn for primary statistics
in the **IVP Statistics Display**.

*Acceptable Values*: positive integer value.

*Default Value*: 8

**IVPStatChartSecondaryColors = "\<color\>;\<color\>;\..."**

*Description*: Controls the colors used for secondary statistics within
the **IVP Statistics Display**. If more statistics are displayed than
colors listed, they will be plotted using system defaults.

*Acceptable Values*: list of colors, each of which satisfies Appendix
A.5 requirements

*Default Value*: "96,96,96;128,128,128;160,160,160"

**IVPStatChartSecondaryLineSize = \<number\>**

*Description*: Controls the width of lines drawn for secondary
statistics in the **IVP Statistics Display**.

*Acceptable Values*: positive integer value.

*Default Value*: 1

**IVPStatChartSecondarySymbols = "\<symbol\>;\<symbol\>;\..."**

*Description*: Controls the symbols used for secondary statistics within
the **IVP Statistics Display**. If more statistics are displayed than
symbols listed, they will be plotted using system defaults.

*Acceptable Values*: list of symbols, each of which is one of the
following: "box", "circle", "cross", "diamond", "none", "square",
"start", "triangle", "vert_line" (vertical line).

*Default Value*: "dot;box;triangle;diamond;square;star;cross"

**IVPStatChartSecondarySymbolSize = \<number\>**

*Description*: Controls the size of symbol drawn for secondary
statistics in the **IVP Statistics Display**.

*Acceptable Values*: positive integer value.

*Default Value*: 8

**IVPStatChooserWindowWidth = \<number\>**

*Description*: Controls the default width of the **IVP Statistics
Chooser Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 820

**IVPStatChartWindowHeight = \<number\>**

*Description*: Controls the default height of the **IVP Statistics
Display**.

*Acceptable Values*: positive integer value.

*Default Value*: 525

**IVPStatChartWindowWidth = \<number\>**

*Description*: Controls the default width of the **IVP Statistics
Display**.

*Acceptable Values*: positive integer value.

*Default Value*: 775

**IVPStatChooserWindowHeight = \<number\>**

*Description*: Controls the default height of the **IVP Statistics
Chooser Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 390

**IVPStatChooserWindowWidth = \<number\>**

*Description*: Controls the default width of the **IVP Statistics
Chooser Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 820

**IVPStatTableMgrWindowWidth = \<number\>**

*Description*: Controls the default width of the **IVP Statistics Data
Viewer**.

*Acceptable Values*: positive integer value.

*Default Value*: 200

**IVPStatTableMgrWindowHeight = \<number\>**

*Description*: Controls the default height of the **IVP Statistics Data
Viewer**.

*Acceptable Values*: positive integer value.

*Default Value*: 500

**LabelFont = \<font\>**

*Description*: Controls the default font used on labels within the
various windows.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Courier;plain;12\"

**LegendFont = \<font\>**

*Description*: Controls the default font used within the legend of the
various **Chart** windows and **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;9\"

**ListFont = \<font\>**

*Description*: Controls the font used within lists in the program,
including the **Select FCST_TS Window**, for example.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"TimesRoman;plain;12\"

**MainChartWindowHeight = \<number\>**

*Description*: Controls the default height of a **Data Source Display**
window.

*Acceptable Values*: positive integer value.

*Default Value*: 600

**MainChartWindowWidth = \<number\>**

*Description*: Controls the default width of a **Data Source Display**
window.

*Acceptable Values*: positive integer value.

*Default Value*: 750

**MenuFont = \<font\>**

*Description*: Controls the font used in the menus of the various
windows.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Courier;plain;12\"

**RelDateChooserWindowHeight = \<number\>**

*Description*: Controls the default height of the **Relative Choose
Date/Time** window.

*Acceptable Values*: positive integer value.

*Default Value*: 160

**RelDateChooserWindowWidth = \<number\>**

*Description*: Controls the default width of a **Relative Choose
Date/Time** window.

*Acceptable Values*: positive integer value.

*Default Value*: 320

**SegmentWindowHeight = \<number\>**

*Description*: Controls the height of a **Select Data Source** window.

*Acceptable Values*: positive integer value.

*Default Value*: 625

**SegmentWindowWidth = \<number\>**

*Description*: Controls the width of a **Select Data Source** window.

*Acceptable Values*: positive integer value.

*Default Value*: 400

**TableFont = \<font\>**

*Description*: Controls the default font used for table data, including
the tables in the **Verification Pairs Data Manager** and **IVP
Statistics Data Viewer.**

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Courier;plain;12\"

**TextFieldFont = \<font\>**

*Description*: Controls the default font used for text fields within the
software.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"TimesRoman;plain;12\"

**TickLabelFont = \<font\>**

*Description*: Controls the default font used for the tick mark labels
along all axes.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;9\"

**TitleFont = \<font\>**

*Description*: Controls the default font used for the title in the
**Chart** windows and the **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;12\"

**VfyGrpMgrSectionLabelFont = \<color\>**

*Description*: Controls font for the section labels in the
**Verification Group Manager**.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: "TimesRoman;bold;14"

**VfyGrpMgrWindowHeight = \<number\>**

*Description*: Controls the default height of the **Verification Group
Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 575

**VfyGrpMgrWindowWidth = \<number\>**

*Description*: Controls the default width of the **Verification Group
Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 500

**VfyLocMgrWindowHeight = \<number\>**

*Description*: Controls the default height of the **Verification
Location Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 300

**VfyLocMgrWindowWidth = \<number\>**

*Description*: Controls the default width of the **Verification Location
Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 850

**VfyLocMgrColumnName\<#\> = \<string\>**

*Description*: Sets the name of the \<#\>-th column of the **Location
Table** within the **Verification Location Manager** to be the specified
string.

*Acceptable Values*: a string, which must be in quotes if spaces are
included

*Default Value*: depends upon \<#\>; see the **Verification Location
Manager**

**VfyLocMgrColumnWidth\<#\> = \<number\>**

*Description*: Sets the width of the \<#\>-th column of the **Location
Table** within the **Verification Location Manager** to be the specified
string.

*Acceptable Values*: positive integer value

*Default Value*: depends upon \<#\>: {81, 30, 75, 60, 60, 60, 60, 60,
163, 163}.

**VfyLocMgrChosenColor = \<color\>**

*Description*: Controls the background color of the non-stage columns
when a location is chosen for display in the **Location Table** of the
**Verification Location Manager**.

*Acceptable Values*: a color (see Section A.5)

*Default Value*: pink

**VfyLocMgrChosenStageColor = \<color\>**

*Description*: Controls the background color of the stage columns when a
location is chosen for display in the **Location Table** of the
**Verification Location Manager**.

*Acceptable Values*: a color (see Section A.5)

*Default Value*: red

**VfyLocMgrUnchosenColor = \<color\>**

*Description*: Controls the background color of the non-stage columns
when a location is not chosen for display in the **Location Table** of
the **Verification Location Manager**.

*Acceptable Values*: a color (see Section A.5)

*Default Value*: white

**VfyLocMgrUnchosenStageColor = \<color\>**

*Description*: Controls the background color of the stage columns when a
location is not chosen for display in the **Location Table** of the
**Verification Location Manager**.

*Acceptable Values*: a color (see Section A.5)

*Default Value*: light_gray

**VfyPlotDefWindowHeight = \<number\>**

*Description*: Controls the default height of the **Verification Plot
Definition Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 440

**VfyPlotDefWindowWidth = \<number\>**

*Description*: Controls the default width of the **Verification Plot
Definition Manager**.

*Acceptable Values*: positive integer value.

*Default Value*: 540

**XAxisLabelFont = \<font\>**

*Description*: Controls the default font used for the x-axis in the
**Chart** windows and the **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;12\"

**YAxisLabelFont = \<font\>**

*Description*: Controls the default font used for the left y-axis in the
**Chart** windows and the **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;12\"

**YAxis2LabelFont = \<font\>**

*Description*: Controls the default font used for the right y-axis in
the **Chart** windows and the **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;12\"

[]{#__RefHeading___Toc135190997 .anchor}***A.4 Generic Font Format***

All font settings for the IVP must be entered in the system settings
file with the following value format:

\<font name\>;\<font style\>;\<font size\>

where each component can have the following values:

\<font name\>: \"Serif\", \"SanSerif\", \"TimesRoman\" or \"Courier\"

\<font style\>: \"plain\", \"bold\", \"italics\" or \"bold-italics\"

\<font size\>: any integer representing a valid size for that font

If an invalid size is specified, the system will use the default value,
which varies based on the font.

[]{#__RefHeading___Toc135190998 .anchor}***A.4 Generic Color Format***

All color settings for the IVP must be entered in the system settings
file with the following value format:

\<color\> -or- \<red\>,\<green\>,\<blue\>

where each component can have the following values:

\<color\>: black, blue, cyan, dark_gray, gray, green, light_gray,
magenta, orange, pink, red, white, yellow.

\<red\>,\<green\>,\<blue\>: The red, blue, and green numerical
specifiers for a color. Each component must be an integer.

If an invalid size is specified, the system will use the default value,
which varies based on the font.

[]{#__RefHeading___Toc135190999 .anchor}***Appendix B: The Statistics***

The statistics are computed numerically from a sample of
forecast-observed data pairs. For the equations below, *N* is the number
of samples satisfying the restrictions for a particular x-axis value,
*f~i~* is the forecast value in the *i-*th data pair, *o~i~* is the
observed value in the *i-*th data pair,

[]{#__RefHeading___Toc135191000 .anchor}***B.1 Error Statistics***

Error statistics are calculated by using the difference between the
forecast and observed value for each data pair in a formula of some
kind. The statistics are calculated as follows:

RMSE =
$\sqrt{\frac{\sum_{i = 1}^{N}\left( f_{i} - o_{i} \right)^{2}}{N}}$

MaxErr
=$\text{MAX}\left\{ \left| f_{i} - o_{i} \right|:i = 1,\text{.}\text{.}\text{.},N \right\}$

MAE = $\frac{\sum_{i = 1}^{N}\left| f_{i} - o_{i} \right|}{N}$

ME =$\frac{\sum_{i = 1}^{N}\left( f_{i} - o_{i} \right)}{N}$

[]{#__RefHeading___Toc135191001 .anchor}***B.2 Categorical Statistics***

Categorical statistics are calculated relative to a particular category.
Regardless of whether the user chooses to use forecast categories or
observed categories, the equations used to compute categorical
statistics are constant. This means that the user only controls the
boundaries defining the categories. It is usually best to plot the
categorical statistics with an x-axis of either forecast categories or
observed categories. After creating the display, the x-axis label should
be changed to "Categories", and each of the x-axis tick marks should be
given an appropriate label, such as "Below FS" (below flood stage).

The following table will be used to describe the computation of each
categorical statistic:

Number range CHAPTER

  -------------------------- --------------------- ---------------------- ---------------------
                             Observed was\...                             

  Forecast was\...           [below]{.underline}   [within]{.underline}   [above]{.underline}

  [above]{.underline}        **A**                 **B**                  **C**

  [within]{.underline}       **D**                 **E**                  **F**

  [below]{.underline}        **G**                 **H**                  **I**
  -------------------------- --------------------- ---------------------- ---------------------

As an example, the above table can be read as: "B is the number of pairs
for which the observed is within the interval and the forecast is above
the interval". The statistics plotted against the left y-axis are as
follows:

-   *Probability of Detection (POD)*: The probability that, given an
    observed value is within the category, the forecast value is also
    within the category, or E/(B+E+H).

-   *Traditional False Alarm Rate (TFAR)*: The probability that, given a
    forecast value is within the category, the observed value is not
    within the category (either above or below), or (D+F)/(D+E+F).

-   *Hydrologic False Alarm Rate (HFAR)*: The probability that, given a
    forecast value is within the category, the observed value is below
    the category (useful when categories represent flood levels), or
    D/(D+E+F).

-   *Under Forecast Rate (UFR)*: The probability that, given an observed
    value is within the category, the forecast value is below the
    category, or H/(B+E+H).

-   *Over Forecast Rate (OFR)*: The probability that, given an observed
    value is within the category, the forecast value is above the
    category, or B/(B+E+H).

-   *Critical Success Index (CSI)*: Commonly used in verification, the
    CSI integrates information from the POD and FAR, and is calculated
    as (D+E+G+H)/(A+B+D+E+F+G+H+I). Currently, this is only available in
    the IVP Batch Program.

    []{#__RefHeading___Toc135191002 .anchor}***B.3 Quantiles and
    Extremes***

The quantiles and extremes are computed by, first, building a set of
points, and then calculating empirical probabilities based on the set.
Given a probability *p*, a *quantile* is that value, *x~p~*, for which
*P*(*X* ≤ *p*) = *x~p~*. The steps in computing the quantiles are as
follows:

1.  Define which x-axis variable you wish to use.

2.  If you wish to calculate the forecast (observed) quantile, in the
    **Verification Plot Definition Manager**, select "Do Not Use" from
    the **Forecast Category** (**Observed Category**) choice box.

3.  For each x-axis variable, a set is built which includes all pairs
    that satisfy the condition placed on the category NOT chosen in Step
    2, above. For example, if you wish to compute the forecast
    (observed) quantile, then the pairs used in calculations must be in
    the selected observed (forecast) category in order to be used in the
    computation.

4.  Estimate the quantiles by building an empirical cumulative
    distribution function using probability estimates based on Bayesian
    plotting positions (Krzysztofowicz, 1992) and using linear
    interpolation to find the forecast (or observed) value that
    satisfies the quantile definition.

5.  Calculate the minimum and maximum by finding the largest and
    smallest forecast (observed) value used in estimating the empirical
    cumulative distribution function.

    []{#__RefHeading___Toc135191003 .anchor}***B.4 Leadtime
    Statistics***

The average lead time of detection is computed by finding all points
within cell E of the categorical statistics table and averaging their
lead time. The lead time is calculated as (valid time -- basis time) of
the forecast within a forecast-observed data pair. This is usually best
plotted as a secondary statistic with a bar plot type. Its best paired
with primary statistics that are categorical statistics.

[]{#__RefHeading___Toc135191004 .anchor}***B.5 Sample Size***

The statistic sample size is the number of samples used to calculate a
statistic; i.e. *N*. With an x-axis of location, it is the number of
forecast-observed data pairs found for each location. With an x-axis of
analysis interval or lead time interval, it is the number of pairs found
for the interval. With an x-axis of observed category or forecast
category, it is the number of pairs found for the category.

[]{#__RefHeading___Toc135191005 .anchor}***B.6 References***

Krzysztofowicz, R., "A Bayesian Estimator of an Empirical Distribution,"
working paper, University of Virginia, 1992.