**Pacific Northwest Hydrography Framework**

**Event Migration Tools -- Software Requirements**

**April 12, 2007**

Prepared by:

Dana Baker, Arron Heriford, Tim Smith, Jay Stevens

# Introduction

The purpose of this document is to record the software requirements for
the proposed PNW Hydrography Framework Event Migration Tools. These
tools are to be a shared set of components to migrate event data that is
referenced to the PNWHF LLID route system to the new NHD event table
format. Based on initial scoping, it is expected that event migration
will be iterative -- requiring multiple passes of the tools using
different tolerances. For instance, where the source route system is
coincident with the target route system, it is likely that only a single
pass with a very small tolerance will be needed. In areas where the
input route system is not coincident with the target route system, one
or more additional passes with higher tolerances may be needed.

# General Application Requirements

This section defines requirements for the overall migration tool
application.

1.  The system will migrate event data from the PNWHF LLID format to the
    NHD -- Hydro Event management Tools format.

2.  Tools will be designed to optimize processing speed.

3.  Tools will be designed to minimize user interaction.

4.  Tools will be designed so that they are easy to troubleshoot.

5.  The event migration tools will automate the processes defined in
    this document.

6.  The system will utilize the ArcMap Transform Route Events
    functionality. This toolset has tested wel in conflating events from
    one route system to another and works quickly.

7.  The system will avoid using Network Traces -- this will slow down
    the tools.

8.  The system will provide a user interface that walks the editor
    through the data migration process.

9.  Tools will write an error log showing successful progress through
    the tool modules and any errors.

10. The input format for event table structures is:

> WC_LLID_NR
>
> Watercourse longitude/latitude identifier number.
>
> WC_BEGIN_AD
>
> Watercourse begin address. The beginning, or downstream, index of the
> watercourse segment. The address or index at which an event or
> attribute can be associated to a route.
>
> WC_END_AD
>
> Watercourse end address. The ending, or upstream, index of the
> watercourse segment. The address or index at which an event or
> attribute can be associated to a route.

11. The output format for the event table structure is defined in the
    *PNW Hydro Framework Event Management Tools User Guide -- Appendix
    1: HEM Event Table Structure.*

# Migration Process Map

![](media/image1.wmf)

# Process Automation Requirements

This section defines the processes to be automated by the event
migration tools.

**Process #1 -- Unique Identifier Check**

> Input: Event Table
>
> Summary: Validates the unique identifier for the input event table.
> This unique identifier is needed in later processing.

1.  Prompts the user to select the unique identifier of the input event
    table.

2.  Process validates that the unique identifier is truly unique.

3.  If one does not exist, one is added to the table and unique values
    are generated for each record.

**Process #2 -- First Pass Event Transform**

> Inputs: Event Table, LLID Route System, NHD Flowline
>
> Outputs: Event Table referenced to the target route system
>
> Summary: Transforms the event table from the source route system to
> the target route system.

1.  Run the input event table through the Event Transform process with a
    small (user defined) tolerance. This process will conflate the LLID
    referenced event data onto the NHD linework.

2.  Process produces an event table referenced to NHD that are potential
    good events.

**Process #3 -- Detect Records that did not Transform**

> Inputs: Original Input Event Table, Output from Process #2
>
> Outputs: Table of records that did not transform
>
> Summary: Examines the transformed event table to determine which
> records were dropped during the event transformation performed in
> Process #2.

1.  Compare the records from the original input event table to the
    output from Process #2. Using the unique identifier, create an
    output table of records that were dropped during the event
    transformation.

**Process #4 -- Export Events**

> Inputs: Output from Process #2
>
> Outputs: Featureclass generated from the transformed event table.
>
> Summary: Creates a featureclass (with geometry and length fields) from
> the transformed event table.

1.  Display the product from Process #2 as an event theme.

2.  Export the event theme to a featureclass (with geometry).

3.  Create and calculate a length in meters field.

4.  Create a field to record the tolerance used during the event
    transformation. Calculate the values accordingly.

**Process #5 -- Export LLID Events**

> Inputs: Original Input Event Table, LLID Route System
>
> Outputs: Featureclass of Input Events with Geometry
>
> Summary: Creates a featureclass (with geometry and length fields) from
> the source event table; used to compare to the transformed event table
> for QA/QC purposes.

1.  Display the original input event theme on the LLID route system as
    an event theme.

2.  Export the event theme to a featureclass (with geometry).

3.  Create and calculate a length in meters field.

**Process #6 -- Event Transform QC**

> Inputs: Transformed Events Featureclass, Original Events Featureclass,
> Length Comparison Tolerance
>
> Outputs: Validated Events, Rejected Events
>
> Summary: Compares the length of the transformed events to the length
> of the event in the original event table. A length comparison will
> detect events that only partially transformed or transformed
> incorrectly.

1.  Summarize the output from Process #4 on the event unique identifier
    field. Sum the total length for each unique event identifier.

2.  Compare the total geometry length in the summary table to the output
    from Process #5.

3.  If the lengths are equal (within the defined length comparison
    tolerance), then put the event record into the "Good Events" bucket.

4.  If the lengths are NOT equal (within the defined length comparison
    tolerance), then put the records into the "bad events" bucket.

**Process #7 -- Merge Event Featureclasses**

> Inputs: Two Transformed Events Featureclasses (outputs from process
> #6, multiple runs)
>
> Outputs: Single Featureclass
>
> Summary: If multiple iterations of the event transformations are
> needed, then the tools must merge the output datasets back together.

1.  Merge the two input datasets into a single output dataset with the
    same input data structure.

**Process #8 -- HEM Event Conversion**

> Inputs: Outputs from Process 4
>
> Outputs: HEM Formatted Event Featureclass
>
> Option: Is Multi-Route Event?
>
> Summary: Once events are transformed and validated, they must be
> converted to the HEM event featureclass structure.

1.  Create a new HEM event featureclass.

2.  If the destination event type is multi-route linear event, then:

    a.  Summarize the input featureclass on the event unique identifier.

    b.  For each input unique identifier:

        -   Retrieve all records in the input featureclass for the
            unique identifier.

        -   Merge the geometries for all of the records selected.

        -   Simplify the output geometry.

        -   Query the output from Process #5 for the original event
            record.

        -   Retrieve the start point from the geometry of the original
            record.

        -   Retrieve the start and end points from the new output
            geometry.

        -   Use the near command to find which of the new points are
            closest to the original start point. This is the new start
            point of the event.

        -   Insert a new record into the output HEM event featureclass
            that contains the simplified geometry and the correct From
            Reach Code, From Reach Measure, To Reach Code, and To Reach
            Measure data. Insert entries into the output HEM event M
            table that contains the Reach Code and Measure information
            for the individual segments.

        -   Insert the event unique identifier into the output HEM
            featureclass.

3.  If the destination event type is single-route linear event or point
    event, then:

```{=html}
<!-- -->
```
a.  For each input record:

-   Insert a new record into the output HEM featureclass that contains
    the geometry from the input featureclass and populate the
    appropriate Reach Code and Reach Measure.

-   Insert the event unique identifier into the output HEM featureclass.

**Process #9 -- Merge Event Tables**

> Inputs: Two event tables (outputs from process #6, multiple runs)
>
> Outputs: Single Featureclass
>
> Summary: Merges the records that did not transform with the records
> that failed the QA/QC process.

1.  Merge the two input datasets into a single output dataset with the
    same input structure.

**Process #10 -- Merge Event Tables that Failed Transformation**

> Inputs: Two datasets in the format of the source event table.
>
> Outputs: A single dataset in the format of the source event table.

1.  Merge the two input datasets into a single output dataset with the
    same input structure.

**Process #11 -- Load Rejected Events (Optional)**

> Inputs: Dataset of events that did not transform in the format of the
> source event table and/or events that failed QA/QC.
>
> Summary: Loads the rejected events into the output HEM event
> featureclass so that they can be manually corrected using the HEM
> event editing tools.

1.  For each event record that did not transform, retrieve the geometry
    for that record from the QC featureclass.

2.  Load the record into the HEM event featureclass.

3.  Calculate the value of the event unique identifier.

4.  Set the ReachCode and Measure fields equal to -1 in the output event
    featureclass.

5.  Set the value of the event transform tolerance attribute equal to
    -1.