***Interactive Verification Program User's Manual***

1.  ***Overview***

Number range CHAPTERThe Interactive Verification Program (IVP) is a tool
for verifying hydrologic forecasts. It provides the ability to view
pairs of observations and forecasts, compute statistics by category, and
transform the data into normalized space. This manual describes how to
operate the IVP.

2.  ***Notation***

Number range CHAPTER

The following notational conventions are used in this document:

-   A GUI component, including a window, panel, menu, menu item, or
    button, is displayed in **bold**.

-   Number range CHAPTER\<key\> indicates a keyboard key. For example
    \<shift\> means the shift key.

-   Text to be entered at a command line is displayed in this font.

3.  ***Execution***

To execute the IVP, enter:

cd \$(get_apps_defaults vsys_scripts)

ivp

Upon execution, the **Select Data Source** window, an example of which
is shown in Figure 1.

4.  ***Apps-defaults Tokens***

The following apps-defaults tokens are used by the IVP:

-   verify_dir : \$(apps_dir)/rfc/verify

-   vsys_dir : \$(verify_dir)

-   vsys_output : \$(vsys_dir)/output

The first token should be defined in the national apps-defaults file and
is used to located the system settings file (see Appendix A). The second
token is used in constructing the third token, which is used to
determine the initial directory to search for pairs files.

5.  ***Select Data Source Window***

The **Select Data Source** window, shown in Figure 1, is used to select
the forecast-observed pairs to use in verification. It is opened when
the IVP first executes or when either of the **Open Data Source** or
**Change Data Source** menu items within the **Actions** menu of the
**Data Source Display** window (see Section 6.0) is selected. Number
range CHAPTERThis section describes the components of the **Select Data
Source** window.

1.  ***Files To Load List***

The **Files To Load** list is displayed in the upper text area.
Forecast-observed data pairs used in verification must be loaded from
verification pairs files. These files are constructed via the Verify
program of release OB3 and earlier, or via the PAIRS_FILE command used
by the verification function of the IVP Batch Program (see the IVP Batch
Program User's Manual for Verification). More than one file can be
included in the list of files. Adding and removing files from the list
are accomplished via the **Add File** and **Remove File** buttons.
![](media/image1.wmf){width="3.109027777777778in"
height="5.202083333333333in"}

2.  ***Available LIDS List***

The **Available LIDS** list is displayed in the lower text area. It is a
list of the location ids (lids) and physical elements available for
verification, as well as the number of pairs found within the files of
the **Files to Load** list for each combination of lid and physical
element. The list is a complete listing of lids and physical elements
for which at least one forecast-observed data pair exists. The list is
constructed whenever the **Fill LIDs** button is clicked.

3.  ***The Buttons***

Managing the **Files to Load** list and the **Available LIDS** is
accomplished through the use of seven buttons in the window. The buttons
are as follows:

-   **Add File**

> Add a file to the **Files to Load** list. Clicking on this button
> spawns a file browser dialog. Find the file you wish to add and click
> on the **Open** button within the dialog. The only requirement is that
> the file's name contains the phrase ".pairs" in it somewhere. If the
> Verify program (OB3 and earlier) is used to build the pairs file, this
> will be handled automatically.

-   **Remove File**

> Remove a file from the **Files to Load** list. Selecting a file and
> clicking on this button will remove the selected file from the list.

-   **Fill LIDs**

> Construct the **Available LIDS** list using the pairs within the files
> of the **Files to Load** list. This list never changes unless the
> **Fill LIDs** button is clicked.

-   **Select All**

> Select all entries in the **Available LIDS** list.

-   **Clear**

> Unselect all entries in the **Available LIDS** list.

-   **Okay**

> Open up a **Data Source Display** window using the selected pairs. By
> selecting entries in the **Available LIDS** list and clicking on this
> button, a Main Display window is opened displaying, as a scatter plot,
> the forecast-observed data pairs.

-   **Cancel**

> Close the **Select Data Source** window and do nothing. If this
> **Select Data Source** window was opened upon execution of the IVP,
> then the program will exit. Otherwise, the window will close and the
> **Data Source Display** window that spawned this window will become
> active.

6.  ***Data Source Display Window***

The **Data Source Display** window, shown in Figure 2, is opened when
the user clicks the **Okay** button in the **Select Data Source**
window. It displays the forecast-observed data pairs to be used in
verification as a scatter plot. The different lids and physical element
combinations that make up the scatter plot are used to build the legend
to the right of the scatter plot.

![](media/image2.png){width="5.891666666666667in"
height="4.711805555555555in"}

The title of the window is constructed as follows:

IVP Number range CHAPTERData Source #\<num\>

where \<num\> is the identification number for the **Data Source
Display**. The menu bar for this window is displayed above the plot. To
the right of the menu bar is the **Data Tracker**, which displays the
coordinates currently pointed to by the mouse cursor.

The following sections describe the menus that make up the menu bar.

1.  ***The Actions Menu***

The **Actions** menu, shown in Figure 3, contains menu items associated
with opening and changing the data source, closing the data source,
saving the current display as a gif, and exiting the program. The menu
items are as follows:

-   **Open New Data**

> Open a new data source. Selecting this menu item will spawn a **Select
> Data Source** window and allow the user to open up a new data source.

-   **Change Data
    Source**![](media/image3.png){width="4.375694444444444in"
    height="2.3958333333333335in"}

> Change this data source. Selecting this menu item will spawn a
> **Select Data Source** window and allow the user to change the data
> displayed in this **Data Source Display** window and reset the
> categories defined in the **Bounds Editor** window (see Section 7.0).
> Note that any **Chart** windows spawned by this **Data Source
> Display** will not change automatically. To change the chart display,
> the user must select the **Refresh** menu item from the **Actions**
> menu of the **Chart** window (see Section 9.1).

-   **Close**

> Close this **Data Source Display** window. If this is the only **Data
> Source Display** window that is currently open, the IVP will exit.

-   **Save as Jpeg/Save as PNG**

> Save the current plot displayed in this **Data Source Display** window
> to a jpeg or png file. This will spawn a file dialog. Specify the name
> of the file to create (without extension) and click **Open**. The
> image file will be created with an appropriate extension (.jpg or
> .png).

-   **Exit**

> Exit the IVP.

1.  ***The Options Menu***

The **Options** menu, shown in Figure 4, is used change the appearance
of the scatter plot display in some way. The menu items are as follows:

-   **Apply NQT...**

> Apply the Normal Quantile Transform (NQT) to the scatter plot data.
> Number range CHAPTERSelecting the **Apply NQT\...** menu item applies
> the NQT, described in Appendix B, to both the observed and simulated
> (forecast) stage values, transforming each into standard normal
> variables. It has two sub-menu items, one of which must be selected
> for anything to happen:

**By Segment**: Apply the NQT for each lid independently. This is the
recommended approach.

**For All**: Apply the NQT for all the lids lumped together.

![](media/image4.png){width="4.401388888888889in"
height="2.4027777777777777in"}

> Upon selecting one of the two sub-menu items, the following will
> happen: (1) the scatter plot will change to display the transformed
> data, (2) the **Bounds Editor** window will refresh, and (3) the
> **Apply NQT\...** menu item will be disabled while the **Recover
> Original Data** menu item will be enabled.

-   **Recover Original Data**

> Recover the original data before application of the NQT. Selecting the
> **Recover Original Data** will result in the following: (1) the
> scatter plot will change to display the original data, (2) the
> **Bounds Editor** window will refresh, and (3) the **Recover Original
> Data** menu item will be disabled while the **Apply NQT\...** menu
> item will be enabled.

-   **Edit Bounds**

> Open the **Bounds Editor** window associated with this **Data Source
> Display** window. See Section 7.0.

-   **Clear Bounds**

> Reset the categories displayed within the **Bounds Editor** window.

-   **Show Line y = x / Hide Line y = x**

> Show or hide the line y = x (i.e. the line where the observed and
> simulated values are equal) in the scatter plot. Selecting this menu
> item will result in a green line being added to the plot (for
> **Show**) or removed (for **Hide**) from the plot. Upon selecting
> **Show Line y = x**, the menu item will change to **Hide Line y = x**,
> and vice versa.

-   **Edit Labels**

> Open up a **Chart Label Mgr** window to allow for editing the labels
> on the current chart. See Section 8.0.

-   **Edit Title Bar**

> Edit the title of the window. Selecting this menu item will result in
> an **Edit Title Bar** dialog being opened. To change the title, type
> in the desired new title and click **Okay**.

-   **Zoom In / Zoom Out**

> Zoom in or out in the current scatter plot. Number range
> CHAPTERSelecting the **Zoom In/Zoom Out** menu item either turns on
> zoom mode, which allows the user to zoom in on a region of the scatter
> plot, or recovers the original plot dimensions. Which of these actions
> it performs is decided according to the current label of the menu
> item. Instructions on how to zoom in on a region are given in Section
> 10.2.

![](media/image5.png){width="4.370833333333334in"
height="2.3930555555555557in"}

1.  ***The Charts Menu***

The **Charts** menu, shown in Figure 5, is used to select different
types of charts available in the IVP. Selecting any menu item (and
sub-menu item) within this menu will result in a **Chart** window being
spawned. The categories used by the **Chart** window, where appropriate,
are determined by this **Data Source Display** window's associated
**Bounds Editor** window. See Section 11.0 for the different charts.

7.  ***Bounds Editor Window***

The **Bounds Editor** window, shown in Figure 6, is opened when the user
selects the **Edit Bounds** menu item in the **Options Menu** of the
**Data Source Display** window. The categories are displayed in
mathematical notation in two lists, the left list for the x-axis (or
observed data) and the right list for the y-axis (or simulated/forecast
data). Initially, these two lists each contain only one entry: (-∞,∞).
Whenever either of these two lists change, the scatter plot within the
associated **Data Source Display** window will change, with blue lines
being drawn for each of the category
boundaries.![](media/image6.png){width="4.700694444444444in"
height="2.8006944444444444in"}

The title of the window is constructed as the title of the **Data Source
Display** window associated with this **Bounds Editor** window, with a
": Bounded Regions Manager" attached at the end. The **X-axis** and
**Y-axis** checkboxes determine which categories are currently selected
for editing. For a set of categories to be edited, the corresponding
check box must be checked. The **Enter Value** text field is an input
mechanism used by the **Actions** menu items **Create Default...** and
**Add Value To Regions**, both described in Section 7.1.

The **Actions** menu is described in the following section.

1.  ***The Actions Menu***

The **Actions** menu, shown in Figure 7, provides menu items associated
with managing the categories displayed in the **Bounds Editor**. The
menu items are as follows:

-   **Create Default...**

> Calculate default categories. Selecting the **Create Default...** menu
> item will setup default categories for the currently selected axis or
> axes. It has three sub-menu items, one of which must be selected for
> the categories to be calculated:
>
> **Minimum Size**: Selecting the **Minimum Size** sub-menu item will
> read in an integer from the **Enter Value** text field and create a
> set of categories such that no category has fewer pairs than the
> integer.
>
> **Number of Regions**: Selecting the **Number of Regions** sub-menu
> item will read in an integer from the **Enter Value** text field and
> create a set of categories such that the number of categories is that
> integer and each interval is of equal size, or as close as possible.
>
> **Flood Stages**: Selecting the **Flood Stages** sub-menu item will
> attempt to set the categories based on the floodcat table of the IHFS
> database specified by the apps-defaults token "db_name". This will
> only work if the IHFS database is accessible from the machine on which
> IVP is being run (typically OB3 and earlier versions of IVP).
>
> Once one of the three sub-menu items it selected, the categories for
> the selected axis or axes will be removed and the default categories
> will be added. The scatter plot in the associated **Data Source
> Display** window will be redrawn to show the new category
> bounds.![](media/image7.png){width="3.2090277777777776in"
> height="1.9583333333333333in"}

-   **Add Value to Regions**

> Define a new category boundary. Selecting the **Add Value to Regions**
> menu item will create a new category defining boundary based on the
> value in the **Enter Value** text field. See Section 10.3 for
> instructions on how to create a new category defining boundary.

-   **Remove Selected Region(s)**

> Remove selected categories from the list of categories for the
> selected axis or axes.

-   **Merge Selected Categories**

> Merge the selected categories into one large category for the selected
> axis or axes. The categories must be adjacent in order to be merged.

-   **Clear Regions**

> Clear all of the categories for the selected axis or axes.

-   **Close**

> Close the **Bounds Editor** window. The categories cannot be edited
> while this window is closed.

8.  ***Chart Label Mgr Window***

The **Chart Label Mgr** window, shown in Figure 8 with its **Actions**
menu, is used to edit the labels displayed in the plot within the
associated **Data Source Display** window or **Chart** window. The title
of the window is constructed as the title of the **Data Source Display**
window or **Chart** window associated with this **Chart Label Mgr**
window, with a ": Chart Label Mgr" attached at the
end.![](media/image8.png){width="5.228472222222222in"
height="3.145138888888889in"}

There are three components to this window. The first is the **Label
Selection** choice box at the top. From this choice box, the label to
change can be selected. The second is the **Font Selection** panel. In
this panel are choice boxes that can be used to select the font to use.
The third is the **Text Value** panel. In this panel is displayed the
current text for the label. See Section 10.4 for instructions on how to
edit the labels.

The **Actions** menu is described in the following section.

1.  ***The Actions Menu***

There are only two menu items within the **Actions** menu of the **Chart
Label Mgr** window.

-   **Update**

> Update the associated **Data Source Display** window or **Chart**
> window to display the new labels.

-   **Change Data Source**

> Close the **Chart Label Mgr** window.

9.  ***Chart Window***

The **Chart** window, examples of which are shown in Figures 10 -- 17,
displays verification graphics calculated for the associated **Data
Source Display** window using categories from the associated **Bounds
Editor** window, where appropriate. It is spawned when one of the menu
items (and possibly sub-menu items) within the **Chart** menu of the
**Data Source Display** window is selected. If the chart includes a
legend, it will be drawn at the right. See Section 11.0 for a
description of each individual chart.

Each type of **Chart** window displays a different chart, but they do
have similarities. First, the title of all **Chart** windows is
constructed as the title of the **Data Source Display** window
associated with this **Chart** window, with a chart descriptor appended
to the end. Second, each type of **Chart** window provides the same
**Actions** menu. That menu is described in the following section.

1.  ***Actions Menu***

The **Actions** menu of a **Chart** window, shown in Figure 9, provides
five menu items, three of which are identical to the similarly named
menu items in the **Data Source Display** window. The menu items are as
follows:

-   **Refresh**

> Redraw the chart. Selecting the **Refresh** menu item will result in
> the chart being redrawn using the data provided in the associated
> **Data Source Display** window scatter plot and the categories in its
> associated **Bound Editor** window. This will allow the plot to be
> redrawn if the data source changes or if the categories change.

-   **Edit Labels**![](media/image9.png){width="2.9993055555555554in"
    height="1.9375in"}

> Edit the labels on the chart. See Section 6.2 for a description.

-   **Edit Title Bar**

> Edit the title bar of this **Chart** window. See Section 6.2 for a
> description.

-   **Save as Gif**

> Save the current chart display as a gif file. See Section 6.1 for a
> description.

-   **Close**

> Close this **Chart** window.

10. ***Instructions***

Number range CHAPTERThis section provides instructions for various
actions that a user can take while running IVP.

1.  ***Build Verification Data Set***

To build a verification data set, do the following within the **Select
Data Source** window (see Section 5.0):

1.  Decide on which pairs files you wish to use as a source of the
    forecast-observed pairs.

2.  For each file, do the following:

    a.  Click on the button **Add File**. This opens up a file browser
        dialog.

    b.  Navigate to and select the file you wish to use as a source of
        pairs.

    c.  Click **Open** within the file browser dialog. The dialog will
        close and the file will be added to the **Files To Load** list.

3.  Click on the **Fill LIDs** button. The files will be searched for
    all unique lid and physical elements pairs. The **Available LIDS**
    list will be filled accordingly.

4.  While holding down the \<ctrl\> key, click on every lid and physical
    element pair you wish to verify together. Optionally, you may use
    the **Select All** button to choose them all.

5.  Click on **Okay**.

Upon completing Step 5, either a new **Data Source Display** window will
be spawned providing a scatter plot of the pairs, or an existing window
will be changed to display a scatter plot of the pairs. The later will
occur only if the **Change Data Source** menu item (see Section 6.1)
within the **Data Source Display** window resulted in spawning the
**Select Data Source** window. Otherwise the former will occur.

1.  ***Zooming In and Out***

Within the **Data Source Display** window, to zoom-in or zoom-out on a
section of scatter plot, do the following:

1.  Click and drag on the scatter plot to draw a rectangle encompassing
    the area you want to zoom in on.

2.  Select the **Zoom In** menu item within the **Options Menu**. This
    will redraw the plot, zooming in on the area within the drawn
    rectangle. Also, the **Zoom In** menu item will become a **Zoom
    Out** menu item.

3.  To zoom back out, select the **Zoom Out** menu item within the
    **Options Menu**.

Only one level of zooming in can be performed within this program.

1.  ***Defining Categories***

To define categories within the **Data Source Display** window for
either the x-axis (observed) values or the y-axis (simulated or
forecasted) values, do the following:

1.  Select the **Edit Bounds** menu item from the **Options** menu. The
    **Bounds Editor** window associated with this **Data Source
    Display** window will open.

2.  Click on the **X-axis** and/or **Y-axis** check box for the axis or
    axes for which you wish to create categories.

3.  Boundaries specifying the categories can be created by doing one of
    the following:

-   Selecting the **Create Default\...** menu item of the **Actions**
    menu (see Section 7.1),

-   selecting the **Add Value to Regions** menu item of the **Actions**
    menu (see Section 7.1) to construct the categories one boundary at a
    time,

-   selecting the **Flood Stages** menu item of the **Actions** menu
    (see Section 7.1) to construct categories corresponding to flood
    stages for a location (if only one lid is involved), or

-   clicking where you wish the boundary to be within the scatter plot
    of the **Data Source Display** window to construct the categories
    one boundary at a time.

> It is recommended you use the **Data Tracker** to guide you selecting
> category boundaries using the scatter plot (see Section 6.0).

Step 3 should be performed until every desired category is defined.

1.  ***Editing Plot Labels***

To edit the labels for the plot associated with the **Data Source
Display** window or a **Chart** window, do the following:

1.  Select the **Edit Labels** menu item within the **Options** menu of
    the **Data Source Display** window or the **Actions** menu of a
    **Chart** window. This will spawn a **Chart Label Mgr** window
    described in Section 8.0.

2.  For each label you wish to edit, do the following:

    a.  Select the label to edit from the **Label Selection** choice
        box.

    b.  Select the font for the label from the choice boxes within the
        **Font Selection** panel.

    c.  Set the text value within the text field in the **Text Value**
        panel.

3.  Once label editing is complete, select the **Update** menu item from
    the **Actions** menu (see Section 8.1).

Upon completing Step 3, the labels should change on the plot within the
corresponding window accordingly.

11. ***The Charts***

Number range CHAPTERThis section describes each of the charts and how
they are constructed. Creating a chart requires, first, defining
categories as described in Section 10.3, and then selecting the
corresponding menu item (or sub-menu item) of the **Charts** menu in the
**Data Source Display** window. A description of how the data sets are
prepared will now be provided, followed by a description of each type of
available chart.

1.  ***Data Set Preparation***

The statistics generated by the IVP are calculated using categories that
are defined using the **Bounds Editor** window. The algorithm used to
prepare the data for statistics calculation begins with identifying the
category variable. The category variable is that variable which is used
to determine to which category a pair belongs. For example, the user may
wish to break down the pairs based on the observed value. Hence, the
observed value becomes the category variable. Other potential category
variables include the forecast value or the location id (see **Error By
Lid Chart**).

Statistics are calculated independently for each category by calculating
the subset of the pairs that belong to the category and calculating the
statistics for that subset. Continuing with the previous example,
suppose the user defines the categories (-∞, 5.5), \[5.5, 6.5), and
\[6.5, +∞) for the observed values. Then three sets of statistics will
be produced, one for each category, where the pairs included in the
statistics for the first category are those pairs for which the observed
value is less than 5.5. The pairs included in the statistics for the
second category will be those pairs for which the observed value is at
least 5.5 and less than 6.5. The third category will include pairs for
which the observed value is at least 6.5.

This is the general mode of computation for all charts, with the
exception of the **Categorical Statistics Chart**. For that chart, both
the forecast and observed values are used in categorizing pairs and
calculating statistics. See Section 11.4 for more detail.

Lastly, the **Error By Lid Chart** uses the combination of location id
and physical element to break down the pairs into categories. See
Section 11.7 for more details.

2.  ***Probability Bar Plot
    Chart***![](media/image10.png){width="5.0569444444444445in"
    height="3.8895833333333334in"}

Number range CHAPTERThe **Probability Bar Plot** **Chart**, shown in
Figure 10, is calculated using whichever variable is not the category
variable. For example, if the user has selected that the observed
regions are used, the statistics are computed based on the forecasted
values. The statistics computed are the 0.25, 0.50, and 0.75 probability
quantiles, where for probability p the probability quantile is x~p~ such
that P(X ≤ x~p~) = p. The quantiles are displayed in the form of a bar
chart, which is interpreted as,

"When the observed value is within a specific interval, the forecast has
a 25% chance of being in blue region, a 25% chance of being in the
yellow region (or 50% chance of being in the yellow or blue regions), a
25% chance of being in the red region (or 75% chance of being in the
red, yellow, or blue regions), and a 25% chance of being above the red
region."

3.  ***Error By Region Chart***

The **Error By Region Chart**, shown in Figure 11, displays the
following verification statistics:

-   root mean squared error (RMSE),

-   maximum error (MaxErr),

-   mean absolute error (MAE), and

-   mean error (ME).

The legend will show which symbol corresponds to which error statistic.

![](media/image11.png){width="5.111111111111111in"
height="3.926388888888889in"}

1.  ***Categorical Statistics Chart***

The **Categorical Statistics Chart**, shown in Figure 12, is constructed
using both forecast and observed categories, requiring that both be
equal. Hence, the sub-menu items **using Observed Regions** and **using
Simulated Regions** only specify what categories to use. The following
table will be used to describe the computation of each categorical
statistic:

Number range CHAPTER

  -------------------------- --------------------- ---------------------- ---------------------
                             Observed was\...                             

  Forecast was\...           [below]{.underline}   [within]{.underline}   [above]{.underline}

  [above]{.underline}        **A**                 **B**                  **C**

  [within]{.underline}       **D**                 **E**                  **F**

  [below]{.underline}        **G**                 **H**                  **I**
  -------------------------- --------------------- ---------------------- ---------------------

The table reads as, "B is the number of pairs for which the observed is
within the interval and the forecast is above the interval". The
statistics plotted against the left y-axis are as follows:

-   Probability of Detection (POD): The probability that, given an
    observed value is within the category, the forecast value is also
    within the category, or E/(B+E+H).

-   Traditional False Alarm Rate (TFAR): The probability that, given a
    forecast value is within the category, the observed value is not
    within the category (either above or below), or (D+F)/(D+E+F).

-   Hydrologic False Alarm Rate (HFAR): The probability that, given a
    forecast value is within the category, the observed value is below
    the category (useful when categories represent flood levels), or
    D/(D+E+F).![](media/image12.png){width="5.097222222222222in"
    height="3.9208333333333334in"}

-   Under Forecast Rate (UFR): The probability that, given an observed
    value is within the category, the forecast value is below the
    category, or H/(B+E+H).

-   Over Forecast Rate (OFR): The probability that, given an observed
    value is within the category, the forecast value is above the
    category, or B/(B+E+H).

-   Critical Success Index (CSI): Commonly used in verification, the CSI
    integrates information from the POD and FAR, and is calculated as
    (D+E+G+H)/(A+B+D+E+F+G+H+I). Currently, this is only available in
    the IVP Batch Program.

Number range CHAPTERThe average lead time of detection is provided as a
gray bar plotted against the right y-axis. This statistic is computed by
averaging the difference between the basis time and valid time for each
of the detections, or pairs for which the forecast and observed are both
within the category.

The usefulness of the HFAR is best described through an example. Suppose
for a specific lid, a minor flood stage and a major flood stage are
available. Then, the HFAR over the interval from minor to major flood
stage will tell the user the rate at which a minor flood is forecasted
but no flooding occurs. Similarly, the HFAR over the interval above
major flood stage will tell the user the rate at which a major flood is
forecasted but either no flooding or minor flooding occurs.

1.  ***Probability Range Chart***

The **Probability Range Chart**, shown in Figure 13, is similar to the
**Probability Bar Plot Chart** except that the quantiles are displayed
as points. The statistics displayed are the probability quantiles (see
Section 11.2) as well as the minimum and maximum. The legend will show
which symbol corresponds to which quantile, minimum, or maximum.

![](media/image13.png){width="5.097222222222222in"
height="3.9208333333333334in"}

2.  ***Murphy Interval Histogram Chart***

Number range CHAPTERThe **Murphy Interval Histogram Chart**, shown in
Figure 14, is a means by which the ability of the forecaster to
discriminate between the categories can be ascertained. There are no
statistics produced. Rather, the density function of the variable for
which the statistics are normally produced is displayed. By displaying
the density function, the spread of the statistics variable can be seen.
The density is approximated by performing the following algorithm for
each user selected category, where the statistics variable is that
variable which is not the category variable:

1.  Break down the total range of the statistics variable into 10 equal
    intervals.

2.  Count the number of values of the statistics variable within each
    interval.

3.  Divide the count by the total number of pairs within the user
    selected category.

4.  Plot the proportion from Step 3 against the midpoint of the
    interval.

For example, suppose the user selected the observed categories, and one
of the intervals is \[5.5, 6.5). The first step is to create a subset of
the pairs for which the observed value is within \[5.5, 6.5). Next,
suppose that the forecasted values for the pairs in this subset range
from 4 to 8. For this chart, the interval \[4, 8\] is broken down into
10 sub-intervals of equal size, which means each sub-interval has a
width of 0.4, or (8 - 4)/10. Then, for each sub-interval, the number of
forecasted values within that interval is counted and divided by the
total number of points within the interval \[4, 8\]. Then, that
proportion is plotted against the midpoint of the sub-interval. So, a
proportion is generated for \[4, 4.4\] and plotted against 4.2, for
(4.4, 4.8\] and plotted against 4.6, for (4.8, 5.2\] and plotted against
5.0, and so on.

![](media/image14.png){width="5.077083333333333in" height="3.9in"}

1.  ***Error By LID Chart***

Number range CHAPTERThe **Error By LID Chart**, shown in Figure 15, is
constructed using the lid to determine the categories. The statistics
computed are the exact same as the **Error By Region Chart**.

2.  ***Kolmogorov-Smirnov Chart***

Number range CHAPTERThe **Kolmogorov-Smirnov Chart**, shown in Figure
16, is a means by which the ability of the forecaster to discriminate
between the categories can be ascertained. The statistics displayed are
the Kolmogorov-Smirnov (K-S) Test Statistics is the maximum vertical
difference between the distributions shown in the **Cumulative
Distribution Chart**, and indicates the similarity of the conditional
distribution in one Subset with the conditional distributions in the
other subsets.

One important characteristic of forecasts is discrimination. When
forecasts are skillful, the distribution of the forecast (or observed)
values for different observed (or forecasted) categories will be
distinct from each other. When the forecasts are unskillful, the
distributions will not differ much and the densities approximated in the
**Murphy Interval Histogram** chart will overlap.

3.  ***Cumulative Distribution Chart***

Number range CHAPTERThe **Cumulative Distribution Chart**, provided in
Figure 17, shows the conditional distributions derived using the
Interval Subset Approach as cumulative distribution functions. They are
shown in the **Murphy Interval Histogram Chart** as density functions.

![](media/image15.png){width="5.118055555555555in"
height="3.93125in"}![](media/image16.png){width="5.090972222222222in"
height="3.9055555555555554in"}

***Appendix A: System Settings File***

***A.1 Overview***

The format of the IVP windows look different on different operating
systems. A system settings file allows the user to change the appearance
of the windows. This appendix describes that file.

***A.2 Location***

The following apps-defaults token must be set in order for the system
settings file to be found:

vsys_dir

The file is assumed to exist in the directory \$(vsys_dir)/app-defaults
with the name "IVP_SYSTEM_FILE.txt".

***A.3 System Settings File***

The system settings file uses the same file format as an IVP Batch
Program batch file. However, rather than being read in sequentially, it
is processed and stored at run-time so that each GUI component can
extract the settings that pertain to it. If a command is given more than
once within the system settings file, the last instance will over-ride
the earlier instances.

*A.3.1 Commands*

Commands, indicated in **bold**, set parameter values that control the
appearance of Vfyruninfo Editor. The \<value\> on the line of the batch
input file that contains the command is usually restricted, depending on
that command. Acceptable Values are listed for each command. The default
value shown is the value assumed if the system settings command is not
found.

The following are IVP system settings commands:

**ButtonFont = \<font\>**

*Description*: Controls the font used on button labels in the windows.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Courier;plain;12\"

**ChartLabelMgrWindowHeight = \<number\>**

*Description*: Controls the pixel height of the **Chart Label Mgr**
window.

*Acceptable Values*: positive integer value.

*Default Value*: 300

**ChartLabelMgrWindowWidth = \<number\>**

*Description*: Controls the pixel width of the **Chart Label Mgr**
window.

*Acceptable Values*: positive integer value.

*Default Value*: 500

**ChartWindowHeight = \<number\>**

*Description*: Controls the default height of a **Chart** window.

*Acceptable Values*: positive integer value.

*Default Value*: 500

**ChartWindowWidth = \<number\>**

*Description*: Controls the default width of a **Chart** window.

*Acceptable Values*: positive integer value.

*Default Value*: 650

**LabelFont = \<font\>**

*Description*: Controls the default font used on labels within the
various windows.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Courier;plain;12\"

**LegendFont = \<font\>**

*Description*: Controls the default font used within the legend of the
various **Chart** windows and **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;9\"

**ListFont = \<font\>**

*Description*: Controls the font used in the lists within the **Select
Data Source** window and the **Bounds Editor** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Courier;plain;12\"

**MainChartWindowHeight = \<number\>**

*Description*: Controls the default height of a **Data Source Display**
window.

*Acceptable Values*: positive integer value.

*Default Value*: 600

**MainChartWindowWidth = \<number\>**

*Description*: Controls the default width of a **Data Source Display**
window.

*Acceptable Values*: positive integer value.

*Default Value*: 750

**MenuFont = \<font\>**

*Description*: Controls the font used in the menus of the various
windows.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Courier;plain;12\"

**SegmentWindowHeight = \<number\>**

*Description*: Controls the height of a **Select Data Source** window.

*Acceptable Values*: positive integer value.

*Default Value*: 625

**SegmentWindowWidth = \<number\>**

*Description*: Controls the width of a **Select Data Source** window.

*Acceptable Values*: positive integer value.

*Default Value*: 400

**TitleFont = \<font\>**

*Description*: Controls the default font used for the title in the
**Chart** windows and the **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;12\"

**XAxisLabelFont = \<font\>**

*Description*: Controls the default font used for the x-axis in the
**Chart** windows and the **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;12\"

**YAxisLabelFont = \<font\>**

*Description*: Controls the default font used for the left y-axis in the
**Chart** windows and the **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;12\"

**YAxis2LabelFont = \<font\>**

*Description*: Controls the default font used for the right y-axis in
the **Chart** windows and the **Data Source Display** window.

*Acceptable Values*: Any font string (see Section A.4)

*Default Value*: \"Serif;bold;12\"

***A.4 Generic Font Format***

All font settings for the Vfyruninfo Editor must be entered in the
system settings file with the following value format:

\<font name\>;\<font style\>;\<font size\>

where each component can have the following values:

\<font name\>: \"Serif\", \"SanSerif\", \"TimesRoman\" or \"Courier\"

\<font style\>: \"plain\", \"bold\", \"italics\" or \"bold-italics\"

\<font size\>: any integer representing a valid size for that font

If an invalid size is specified, the system will use the default value,
which varies based on the font.

***Appendix B: The Normal Quantile Transform***

Number range CHAPTERThe NQT is applied by first computing an empirical
estimate, F, of the cumulative distribution function for the data set X.
Then, for every sample value, x, in the data set X, a new sample value
is generated as

z = Q^-1^(F(x)),

where Q is the standard normal cumulative distribution function. The
data set Z containing all the transformed values from X is then the NQT
transformed data set.