# Presentation: 242714

## Interactive Verification ProgramTraining for AWIPS OB8.2

- Hank Herr
- Office of Hydrologic Development

*RFC Verification Workshop, 08/15/2007*

## Overview

- Introduction
- IVP Terminology
- Software Components
- Database Usage
- Output Files
- Design
- Demonstration
- Tips
- Future Possibilities
- Exercises
- Discussion

## Original IVP written around 2000-2001 for old HP based verify software
Verification moved to archive database for OB4
IVP Batch Program software written
Old IVP GUI still ran, but used pairs files created during batch processing; it did not access the archive database directly
New IVP GUI written for OB7.2
Provided ability to build verification products via GUI, modify components of a chart and save modifications, create batch file to reconstruct graphics via IVP Batch Program
Accessed archive database directly
Limitations:
Designed for stage data
Accessed only three data tables of the archive database
Limited number of statistics
No quality control of pairs
Inflexibility in building batch files from GUI
JClass Desktop Views used to render charts (limited charting capabilities; upgrade cost too much money)

- Verification moved to archive database for OB4
- IVP Batch Program software written
- Old IVP GUI still ran, but used pairs files created during batch processing; it did not access the archive database directly
- New IVP GUI written for OB7.2
- Provided ability to build verification products via GUI, modify components of a chart and save modifications, create batch file to reconstruct graphics via IVP Batch Program
- Accessed archive database directly
- Limitations:
- Designed for stage data
- Accessed only three data tables of the archive database
- Limited number of statistics
- No quality control of pairs
- Inflexibility in building batch files from GUI
- JClass Desktop Views used to render charts (limited charting capabilities; upgrade cost too much money)

- IntroductionHistory of IVP

## IntroductionNew for OB8.2

- Designed for verification of stage, flow, temperature, and precipitation forecasts
- Can do other data types as long as data is stored within accessible tables
- Accesses 10 data tables of the archive database
- Uses all data tables except pempsep, pemrsep, and pairedvalues
- Additional statistics
- Moments, CDF Plot, ROC, etc.
- Quality control of pairs via GUI
- View pairs in a time series plot, in addition to existing scatter plot
- Mark pairs as ‘bad’ so that verification ignores them
- Change forecast or observed values within a pair and save changes to database
- More flexible batch file construction
- Batch file can be built that will construct current chart for other combinations of locations and analysis periods
- Chart Director used for rendering
- Relatively quick, memory efficient chart rendering
- Used previously for HydroGen, so a run-time license already exists

## IVP Terminology

- Verification Location
- A forecast point or location and a data type for which verification is to be performed
- Uniquely specified by location id (lid within database) and SHEF pedtsep code.
- Includes following information:
- Forecast and observed categories
- Observation type (raw or processed)
- Active flag (yes or no)
- National flag (yes or no)
- Sensor preferences (ordered list of observed type source, representing preferences for which observed values to pair with a forecast)

## IVP Terminology

- Verification Group
- A set of locations and parameters that define the data to load and other parameters of verification
- Locations must have the same number of categories and must be of the same type: stage (height), flow (discharge), temperature, precipitation
- Includes following parameters:
- Analysis period and intervals
- Lead time period and intervals
- Issuance time-of-day (0-24 hours) period and intervals
- Restrictions on location response time, active flag, and SHEF components (forecast type source, physical element, duration, and extremum)
- Whether or not to compute statistics independently for each lid or forecast type source

## IVP Terminology

- Forecast-Observed Data Pair
- A forecast value paired with an observed value
- Observed value can be “raw” or “processed” depending on if the value was posted to the database by the raw or processed SHEF decoder
- Each forecast can have one raw pair and one processed pair associated with it
- “Best” candidate observation is chosen for each forecast based on following criteria:
- SHEF qualifier code (G, M, P, V, S, Z, T, F, and Q, in order)
- sensor preference (observation type source) rank
- difference between forecast validtime and observation obstime
- A pair can be “good” or “bad”
- Bad pairs are not used in verification computations

## Software Components

- Vfyruninfo Editor GUI
- Manage vfyruninfo table
- IVP
- Data quality control and define verification products graphically
- IVP Batch Program
- Build pairs and verification products via a batch language
- IVP Batch Builder
- Edit IVP Batch Program batch files graphically
- Verify Pairs Ingestor
- Ingest pairs directly from text files of specific format into the vfypairs or vfyprocpairs tables
- del_bad_pairs
- Removes bad pairs from the vfypairs and vfyprocpairs table

## Database Usage

- Verification Location Information
- Vfyruninfo table
- Defines locations for which verification can be performed.
- Individual entries for each verification location and sensor preference component (observed type source)
- Edited via Vfyruninfo Editor GUI
- Location table
- Information used in building labels, includes RFC name, full location name, etc.
- Rivercrit table
- Flood stages and flows (if location is of appropriate data type).

## Database Usage

- Building Pairs
- Tables of data for pairing:
- Forecasts: pedfsep, pehfsep, peqfsep, pedcsep
- Raw Observations: pecrsep, pedrsep, peoosep
- Processed Observations: pedpsep, pehpsep, peqpsep, peoosep
- Vaddadjust table
- Adjust observations prior to pairing with a forecast
- Only affects forecast-observed pairs
- Is not dated; there can be only one entry per location
- Sensok table
- Allows for specifying time ranges during which sensors should not be used (i.e. observations should not be used for pairing)

## Database Usage

- Storing Pairs
- Vfypairs table
- Stores pairs constructed using raw observations
- Field quality_code defines if the pair is good or bad
- Can be very large (millions of records)
- Vfyprocpairs table
- Stores pairs constructed using processed observations
- Field quality_code defines if the pair is good or bad
- Can be very large, but will likely be a fraction of the size of vfypairs
- A forecast can have up to two pairs built for it: one stored in vfypairs and one stored in vfyprocpairs
- Only one of the two pairs can be analyzed at one time

## Output Files

- Output Statistics File (batch only)
- Verification statistics output in an ASCII tabular format
- Pairs File
- Pairs used to compute statistics
- ASCII tabular format
- Output Image File (.png, .jpg)
- PNG or JPG files displaying a plot of verification statistics
- Image Data File (.dat)
- Statistic values comprising an output image plot
- ASCII tabular format

## DesignFoundation

- Archive database provides the following:
- Forecasts and observations
- Verification location information
- Forecast-observed data pairs

- User

## DesignVfyruninfo Editor

- Vfyruninfo
- Editor

- User

- Vfyruninfo rows

- Vfyruninfo Editor is used to specify verification locations and saves them to the vfyruninfo table of the archive database

## Design IVP Batch Program

- Processor

- User

- pairs

- Design IVP Batch Program

- Vfyruninfo
- Editor

- Vfyruninfo rows

- forecasts,
- observations,
- location info

- IVP Batch Program provides the user the ability to:
- Pair forecasts with observed values according to user specified parameters

- parameters

- Batch

- Batch file

## DesignIVP Batch Program

- Verification
- Processor

- Pairing
- Processor

- User

- pairs

- ASCII

- pairs,
- location info

- forecasts,
- observations,
- location info

- Vfyruninfo
- Editor

- Vfyruninfo rows

- pairs,
- statistics

- IVP Batch Program provides the user the ability to:
- Calculate statistics and output results to tabular ASCII files

- parameters

- Batch

- Batch file

## DesignIVP Batch Program

- Verification
- Processor

- Pairing
- Processor

- User

- ASCII

- pairs

- Graphics
- Processor

- Graphics

- statistics

- params

- Vfyruninfo
- Editor

- Vfyruninfo rows

- forecasts,
- observations,
- location info

- pairs,
- location info

- pairs,
- statistics

- IVP Batch Program provides the user the ability to:
- Generate user customizable graphical verification products displaying statistics

- parameters

- Batch

- Batch file

## DesignIVP GUI

- Verification
- Processor

- Pairing
- Processor

- User

- GUI

- Graphics

- ASCII

- pairs

- pairs,
- statistics

- DesignIVP GUI

- Vfyruninfo
- Editor

- Vfyruninfo rows

- forecasts,
- observations,
- location info

- pairs,
- location info

- IVP provides the user the ability to:
- Specify, build, and view graphics

- Graphics
- Processor

- statistics

- params

- parameters

- Batch

- Batch file

## DesignIVP GUI

- Verification
- Processor

- Pairing
- Processor

- User

- GUI

- Graphics

- ASCII

- pairs

- pairs,
- statistics

- DesignIVP GUI

- Vfyruninfo
- Editor

- Vfyruninfo rows

- forecasts,
- observations,
- location info

- pairs,
- location info

- IVP provides the user the ability to:
- Modify forecast-observed data pairs

- Graphics
- Processor

- statistics

- params

- pairs

- parameters

- Batch

- Batch file

## DesignIVP GUI

- params

- Verification
- Processor

- Pairing
- Processor

- User

- GUI

- Graphics

- ASCII

- pairs

- parameters

- pairs,
- statistics

- DesignIVP GUI

- Batch

- Vfyruninfo
- Editor

- Vfyruninfo rows

- forecasts,
- observations,
- location info

- pairs,
- location info

- IVP provides the user the ability to:
- Create batch files to produce graphics via IVP Batch Program

- Graphics
- Processor

- statistics

- params

- pairs

- Batch file

- batch
- tokens

## DesignIVP Window Layout

- DesignIVP Window Layout

- 1. Load Pairs

- 2. View and QC Pairs

- 3. Define Products

- 4. View Products

## Demonstration

- Feature availability:
- Both OB7.2 and OB8.2
- OB8.2 only*

## Start Vfyruninfo Editor
Filter ingestfilter table contents*
Select locations from ingestfilter table*
Create locations from scratch
Specify location parameters
Active*
National*
Response time
Sensor preferences
Save to archive database

- Filter ingestfilter table contents*
- Select locations from ingestfilter table*
- Create locations from scratch
- Specify location parameters
- Active*
- National*
- Response time
- Sensor preferences
- Save to archive database

- DemonstrationVfyruninfo Editor

- ref:* Vfyruninfo Editor User’s Manual*

## Commands:
Time window: START_TIME, END_TIME
Specify locations: LOCATION, DUR*, EXTREMUM*, FCST_TS*, PE*
Pairing window: PAIRING_WINDOW
Observation type: OBS_TYPE (replaces OBS_TABLE)*
Persistence flag: PERSISTENCE
Action:
Build the pairs: BUILD_PAIRS

- Time window: START_TIME, END_TIME
- Specify locations: LOCATION, DUR*, EXTREMUM*, FCST_TS*, PE*
- Pairing window: PAIRING_WINDOW
- Observation type: OBS_TYPE (replaces OBS_TABLE)*
- Persistence flag: PERSISTENCE
- Action:
- Build the pairs: BUILD_PAIRS

- DemonstrationPairing Batch File

- ref:* IVP Batch Program User’s Manual for Pairing*

## Example (buildpairs_template.bat):

- DemonstrationPairing Batch File

**# Pair all locations. Use default settings of **
**# ALL for all other location specifying commands**
**location       = ALL  **
**# Build pairs for a two week window prior to **
**# current system time**
**start_time     = "* - 14 days"**
**end_time       = "*"**
**# Parameters         **
**pairing_window = 1**
**obs_type       = RAW**
**build_pairs    = true**

- ref:* IVP Batch Program User’s Manual for Pairing*

## Commands:
Time Window: START_TIME, END_TIME
Specify Locations: DUR*, EXTREMUM*, FCST_TS, PE
Action:
Compute statistics: NATLSTATS
Recommendation*:
Set the national flag for all locations that are part of the national program to ‘Y’ (use Vfyruninfo Editor)
Give the NATLSTATS action a value of NATIONAL

- Time Window: START_TIME, END_TIME
- Specify Locations: DUR*, EXTREMUM*, FCST_TS, PE
- Action:
- Compute statistics: NATLSTATS
- Recommendation*:
- Set the national flag for all locations that are part of the national program to ‘Y’ (use Vfyruninfo Editor)
- Give the NATLSTATS action a value of NATIONAL

- DemonstrationNational Statistics Batch File

- ref:* IVP Batch Program User’s Manual for Verification*

## Example (natlstats_template.bat):

- DemonstrationNational Statistics Batch File

**# Compute stats for September, 2003.**
**START_TIME  = “2003-09-01 00:00:00”**
**END_TIME    = “2003-09-30 23:59:59”**
**# Default Values are ‘ALL’**
**# FCST_TS   = ALL**
**# PE        = ALL**
**# DUR       = ALL**
**# EXTREMUM  = ALL**
**NATLSTATS   = NATIONAL**

- ref:* IVP Batch Program User’s Manual for Verification*

## Start IVP
			go vsys_scripts
			ivp
Specify parameters of a verification group
Analysis period and intervals
Lead time period and intervals
Issuance time period and intervals*
Miscellaneous parameters

**	****	****	****go vsys_scripts**
**	****	****	****ivp**
- Specify parameters of a verification group
- Analysis period and intervals
- Lead time period and intervals
- Issuance time period and intervals*
- Miscellaneous parameters

- DemonstrationIVP: Verification Group Manager

- ref:* IVP User’s Manual*, Section 9

## Specify verification locations that comprise a group
Filter available locations*
Choose locations
Specify categories
Specify observation type (from which pairs table to acquire data)*

- Filter available locations*
- Choose locations
- Specify categories
- Specify observation type (from which pairs table to acquire data)*

- DemonstrationIVP: Verification Location Manager

- ref:* IVP User’s Manual*, Section 13

## Create IVP Data Display
Loading data progress window
Open the Verification Group Manager and Verification Location Manager
Save the image

- Loading data progress window
- Open the Verification Group Manager and Verification Location Manager
- Save the image

- DemonstrationIVP: IVP Data Display

- ref:* IVP User’s Manual*, Section 15

## Examine data through two views:
Scatter Plot View
Select a location* 
In OB7.2, use Show button in Verification Location Manager
Display info about a location*
Line x = y
Time Series View*
Specify time series to view
Emphasize a time series

- Scatter Plot View
- Select a location*
- In OB7.2, use Show button in Verification Location Manager
- Display info about a location*
- Line x = y
- Time Series View*
- Specify time series to view
- Emphasize a time series

- DemonstrationIVP: IVP Data Display

- ref:* IVP User’s Manual*, Sections 15.1, 15.2

## Draw a selected region rectangle
Zoom and the navigator panel
Zoom in on scatter view
Zoom in on time series view*
Panning through the time series by zooming
Shift the zoomed region via the navigator panel and keyboard*
Zoom out

- Zoom and the navigator panel
- Zoom in on scatter view
- Zoom in on time series view*
- Panning through the time series by zooming
- Shift the zoomed region via the navigator panel and keyboard*
- Zoom out

- DemonstrationIVP: IVP Data Display

- ref:* IVP User’s Manual*, Sections 15.6, 15.7

## View Pairs via the Verify Pairs Viewer
Select pairs
Sort table
Create a pairs file

WARNING: Do not view very large (30,000+) sets of pairs... the table rendering may be too slow

- Select pairs
- Sort table
- Create a pairs file
- WARNING: Do not view very large (30,000+) sets of pairs... the table rendering may be too slow

- DemonstrationIVP: IVP Data Display

- ref:* IVP User’s Manual*, Section 16

## Mark pairs as good or bad via IVP Data Display*
Edit pairs via the Verify Pairs Editor*
Select pairs
Mark pairs as good or bad
Edit forecast or observed value manually
Focus on a pair
Edit forecast or observed value graphically
Scatter view
Time series view
Change the mouse mode*

WARNING: Do not select large sets (1000+) of pairs for editing... the table rendering speed may be too slow

- Edit pairs via the Verify Pairs Editor*
- Select pairs
- Mark pairs as good or bad
- Edit forecast or observed value manually
- Focus on a pair
- Edit forecast or observed value graphically
- Scatter view
- Time series view
- Change the mouse mode*
- WARNING: Do not select large sets (1000+) of pairs for editing... the table rendering speed may be too slow

- DemonstrationIVP: IVP Data Display

- ref:* IVP User’s Manual*, Sections 15.11, 17

## Open the Plot Definition Manager
Select statistics 
IVP Statistic Chooser Manager
Select plot type
Select x-axis variable
Select comparison variable*
Edit statistics parameters*
Edit group parameters*
Spawns a window nearly identical to Verification Group Manager

Note how variable selection affects analysis, lead time, and issuance time-of-day intervals
Note how variable selection affects category choice boxes

- Select statistics
- IVP Statistic Chooser Manager
- Select plot type
- Select x-axis variable
- Select comparison variable*
- Edit statistics parameters*
- Edit group parameters*
- Spawns a window nearly identical to Verification Group Manager
- Note how variable selection affects analysis, lead time, and issuance time-of-day intervals
- Note how variable selection affects category choice boxes

- DemonstrationIVP: Verification Plot Definition Manager

- ref:* IVP User’s Manual*, Sections 18, 19

## Errors
Categorical Statistics
Quantiles and Extremes
Computed relative to whichever variable whose category used is set to “Do Not Use”
Moments*
Sample Size
The number of pairs used in the computations 
–or–
Number of pairs within the category (if x-axis or comparison variable is forecast or observed categories)
Misc.*
Special Plots*

- Categorical Statistics
- Quantiles and Extremes
- Computed relative to whichever variable whose category used is set to “Do Not Use”
- Moments*
- Sample Size
- The number of pairs used in the computations
- –or–
- Number of pairs within the category (if x-axis or comparison variable is forecast or observed categories)
- Misc.*
- Special Plots*

- DemonstrationIVP: The Statistics

- ref:* IVP Batch Builder User’s Manual*, Appendix B

## Categorical Statistics use 3x3 Contingency Table

Examples:
HFAR = D/(D + E + F)
TFAR = (D + F)/(D + E + F)
CSI = (A+B+D+E)/(A+B+D+E+G+H+C+F) 
In a 2x2 contingency table
HFAR is always 0 for the lowermost category
TFAR and HFAR are equivalent for uppermost category
POD + UFR + OFR = 1 in all cases

- Examples:
- HFAR = D/(D + E + F)
- TFAR = (D + F)/(D + E + F)
- CSI = (A+B+D+E)/(A+B+D+E+G+H+C+F)
- In a 2x2 contingency table
- HFAR is always 0 for the lowermost category
- TFAR and HFAR are equivalent for uppermost category
- POD + UFR + OFR = 1 in all cases

- DemonstrationIVP: The Statistics

- ref:* IVP Batch Builder User’s Manual*, Appendix B

## DemonstrationIVP: Verification Plot Definition Manager

- x-axis

- primary
- y-axis

- comparison
- variable

- secondary
- y-axis

- Statistic
- short name
- (matches
- output
- files)

## Choose categories used
Disabled if x-axis or comparison variable is forecast or observed categories
Select “Do Not Use”
Only one of the two choice boxes can be “Do Not Use”
Impact of selections on how statistics are computed
Category used restricts pairs used in computations to only those within the category
Some statistics are calculated relative to the variable not used to categorize data (e.g. quantiles, cdf plot*, pdf plot*)
Special case: ROC plot*
Category used defines observation threshold

- Disabled if x-axis or comparison variable is forecast or observed categories
- Select “Do Not Use”
- Only one of the two choice boxes can be “Do Not Use”
- Impact of selections on how statistics are computed
- Category used restricts pairs used in computations to only those within the category
- Some statistics are calculated relative to the variable not used to categorize data (e.g. quantiles, cdf plot*, pdf plot*)
- Special case: ROC plot*
- Category used defines observation threshold

- DemonstrationIVP: Verification Plot Definition Manager

- ref:* IVP User’s Manual*, Section 18.7

## Create IVP Statistic Display
Data is always loaded from memory even if the overridden group parameters imply additional data is needed (such as a wider analysis interval, for example)
Allows for relatively quick plot generation
Save the image
View the data via the IVP Statistics Data Viewer
Create an ASCII format image data file

- Data is always loaded from memory even if the overridden group parameters imply additional data is needed (such as a wider analysis interval, for example)
- Allows for relatively quick plot generation
- Save the image
- View the data via the IVP Statistics Data Viewer
- Create an ASCII format image data file

- DemonstrationIVP: IVP Statistic Display

- ref:* IVP User’s Manual*, Section 21

## Create a batch file
Change the analysis period*
Change the included locations*
Create other groups for which to generate products*
Specify output and template file names
Replacement strings are needed if images are to be produced for multiple groups in order to avoid overwriting output files*
Specify batch file name

- Change the analysis period*
- Change the included locations*
- Create other groups for which to generate products*
- Specify output and template file names
- Replacement strings are needed if images are to be produced for multiple groups in order to avoid overwriting output files*
- Specify batch file name

- DemonstrationIVP: IVP Statistics Display

- ref:* IVP User’s Manual*, Section 23

## Open up the Chart Property Manager
Click on menu item or tool bar item
Click on plot component*
Change labels
Change axis limits or chart dimensions
Change legend components
Change x-axis labels (for IVP Statistic Display)
Create a template file
Don’t modify manually unless absolutely necessary
Apply changes from a template file

- Click on menu item or tool bar item
- Click on plot component*
- Change labels
- Change axis limits or chart dimensions
- Change legend components
- Change x-axis labels (for IVP Statistic Display)
- Create a template file
- Don’t modify manually unless absolutely necessary
- Apply changes from a template file

- DemonstrationIVP: Chart Property Manager

- ref:* IVP User’s Manual*, Sections 24, Appendix D

## DemonstrationIVP: Chart Property Manager

- X-Axis Title

- Y-Axis 1
- Title

- Legend

- Y-Axis 2
- Title

- Legend
- Title

- Tick Mark Font

- Plot Title

## Start IVP Batch Builder
Load batch file
Insert, append, delete command lines
Edit command lines
Comment and uncomment command lines
Check for errors
Run batch file

- Load batch file
- Insert, append, delete command lines
- Edit command lines
- Comment and uncomment command lines
- Check for errors
- Run batch file

- DemonstrationIVP Batch Builder and the IVP Batch Program

- ref:* IVP Batch Builder User’s Manual*

## Create one batch file to generate products for many locations
@FILE Command to refer to other files
Replacement strings to avoid overwriting output files

- @FILE Command to refer to other files
- Replacement strings to avoid overwriting output files

- DemonstrationIVP Batch Builder and the IVP Batch Program

- ref:* IVP Batch Builder User’s Manual*

## Organize output by using subdirectories of $(vsys_output)*
If producing graphics for each location, use  @ONELID or @ONELIDSHEF to define directories: 
OUTPUT_FILE = “@ONELID/output.txt,c”
If producing graphics for time periods, use @STARTTIME or @ENDTIME:
OUTPUT_FILE = “@STARTTIME{[CCYY]}/output.txt,c”

- If producing graphics for each location, use  @ONELID or @ONELIDSHEF to define directories:
- OUTPUT_FILE = “@ONELID/output.txt,c”
- If producing graphics for time periods, use @STARTTIME or @ENDTIME:
- OUTPUT_FILE = “@STARTTIME{[CCYY]}/output.txt,c”

- DemonstrationIVP Batch Builder and the IVP Batch Program

- ref:* IVP Batch Builder User’s Manual*

## To minimize time spent querying database, load all needed pairs at once
Only specify the overall analysis period and choose the locations within the Verification Group Manager and Verification Location Manager
Specify other parameters via the Plot Definition version of the Verification Group Manager (click on Edit Group Parameters in the Plot Definition Manager)
Important when generating many products
To minimize the chart rendering time for the IVP Data Display, keep the number of pairs small
Do one location at a time for a couple years
Do several locations at a time for one month
Important when quality controlling pairs data
See the Memory Limitations Tips later for recommended numbers of pairs to load

- Only specify the overall analysis period and choose the locations within the Verification Group Manager and Verification Location Manager
- Specify other parameters via the Plot Definition version of the Verification Group Manager (click on Edit Group Parameters in the Plot Definition Manager)
- Important when generating many products
- To minimize the chart rendering time for the IVP Data Display, keep the number of pairs small
- Do one location at a time for a couple years
- Do several locations at a time for one month
- Important when quality controlling pairs data
- See the Memory Limitations Tips later for recommended numbers of pairs to load

- Tips:Verification Group Manager Usage

## Use OBS_TYPE “RAW” for paired observations if...
Observed data was posted to archive database via raw SHEF decoder, AND
First letter of type source is ‘R’.
Use OBS_TYPE “PROCESSED” if...
Observed data was posted to archive database via processed SHEF decoder, OR
First letter of type source is not ‘R’
Example:
If processed observations were posted via raw SHEF decoder (to peoosep table), then specify PROCESSED

- Observed data was posted to archive database via raw SHEF decoder, AND
- First letter of type source is ‘R’.
- Use OBS_TYPE “PROCESSED” if...
- Observed data was posted to archive database via processed SHEF decoder, OR
- First letter of type source is not ‘R’
- Example:
- If processed observations were posted via raw SHEF decoder (to peoosep table), then specify PROCESSED

- Tips:Deciding Between PROCESSED or RAW

## Specifies parameters of graphical user interfaces
Fonts
Window Sizes
Colors
Others
Location
Default:

$(get_apps_defaults vsys_dir)/app-defaults/*_SYSTEM_FILE.txt

Directory can be overridden by exporting environment variable SYSTEM_SETTINGS_FILE_DIR
Can allow for user specific system settings
Format
ESPADP/IVP batch language standard “<token> = <value>” format

- Fonts
- Window Sizes
- Colors
- Others
- Location
- Default:
- $(get_apps_defaults vsys_dir)/app-defaults/*_SYSTEM_FILE.txt
- Directory can be overridden by exporting environment variable SYSTEM_SETTINGS_FILE_DIR
- Can allow for user specific system settings
- Format
- ESPADP/IVP batch language standard “<token> = <value>” format

- Tips:System Settings Files

- ref:* IVP User’s Manual*, *IVP Batch Builder User’s Manual*, *Vfyruninfo Editor User’s Manual*, Appendix A

## Tips: Memory Limitations

- Recommended number of pairs:
- Limits can be changed by using the ivp and ivpbatch script –m and –s options (run “ivp –hh” or “ivpbatch –hh” to read about these options)
- Limits above are for IVP GUI; IVP Batch Program can deal with more pairs because not all pairs need to be stored in memory at one time
- If IVP GUI runs out of memory, an error message will pop-up and the pairs will be discarded (in most cases)
- The software may crash in other cases
- Verify Pairs Viewer window is a memory hog
- Add about 25% to the recommended value if not using the Viewer

- ref:* IVP User’s Manual*, *IVP Batch Builder User’s Manual*, *Vfyruninfo Editor User’s Manual*, Appendix A

## Debug problems
Export vsys_debug token prior to execution to see additional output
Useful levels are 0 (no output), 1, 20, 40, 60, 80, 100, 120, 128.
“export vsys_debug=20” to see additional progress output
“export vsys_debug=40” to see database query where clauses and timing results for chart rendering
“export vsys_debug=128” to see everything (lots and lots of output)
Performance
If IVP Data Display chart rendering is slow...
Reduce number of pairs viewed
Zoom in on regions (points outside current zoom area are not rendered)
Run on faster machines (port code to LX and execute software there)
If database query speed is slow...
Turn off SHEF decoders for a short time (if you can)
Reduce number of locations or width of analysis period

- Export vsys_debug token prior to execution to see additional output
- Useful levels are 0 (no output), 1, 20, 40, 60, 80, 100, 120, 128.
- “export vsys_debug=20” to see additional progress output
- “export vsys_debug=40” to see database query where clauses and timing results for chart rendering
- “export vsys_debug=128” to see everything (lots and lots of output)
- Performance
- If IVP Data Display chart rendering is slow...
- Reduce number of pairs viewed
- Zoom in on regions (points outside current zoom area are not rendered)
- Run on faster machines (port code to LX and execute software there)
- If database query speed is slow...
- Turn off SHEF decoders for a short time (if you can)
- Reduce number of locations or width of analysis period

- Tips:Debugging and Performance

- ref:* IVP User’s Manual*, Section 25

## Future Possibilities

- Confidence Intervals
- Algorithms need to be developed
- Ability to produce CI values and plot them needs to be added
- GUI redesign
- Reduce the number of windows
- Reorganize window so that the Verification Group Manager does not need to be in two places
- Add ability to handle multiple comparison variables
- Add ability to modify other properties of a chart
- Tick spacing, line style (solid, dashed, etc), line width, etc.
- Requires additions to Chart Property Manager
- Create baseline batch files requiring no changes in order to produce standard graphics
- Standard graphics need to be determined
- Add ability to put changed forecasts and observed values back into original data tables and rebuild all affected pairs on-the-fly
- IVP would become a more powerful, general QC tool for archive data

## Documentation

- Available on the web

- http://www.nws.noaa.gov/oh/hrl/verification/verification_doc_ob82.php

## Exercises

- Described in hand-outs
- The data sets are very large, so while the pairs are loading, discuss the plots that are needed in order to answer the exercise questions
- What statistics are to be displayed?
- What are the x-axis and comparison variables?
- Are there any observed or forecast category restrictions?
- Group assignments

## Discussion

- Review exercises
- Pre-workshop questions
- Have they been answered?
- General comments on software