# Presentation: 840311

## ISEMP Objectives

**Subbasin-scale**
**Programmatic coordination, design, planning and implementation**
**Indicators and metric development and testing**
**Protocol development, refinement and testing**
**Sampling design development and testing**
**Effectiveness and status and trend monitoring experimental design and implementation**
**Program-scale**
**Evaluation tools development and testing**
**Data management tools development and testing**

## Requirements to meet objectives

- Large volume of data from many agencies and projects
- Ability to summarize based on how, when, and where data was collected
- Ability to support range of analytical methods (hypothesis testing, time series, structural equations, mapping, etc)
- Ability to adapt to changing requirements

**Notes:**
Transition: To get a sense of the scope of data that we would be working with, we began compiling historic data from the John Day Basin

## Compilation of Historic Data in the John Day Basin

- Acquiring data:
*~80 hours in 3 month period*
- Process data:
*~1300 hours over 12 months*
- Analyze data:
- -in progress

**Notes:**
Let me show you some examples of the formats we were working with.

Data entry and processing tools were developed in response to a data management need created by the compilation of historical data for the pilot basins.  The intention with the historical compilation was to collect ‘historic’ monitoring data collected by agencies.  We found a large quantity of data (e.g. over 3000 collection events of water temperature within the basin over the last 20 years, averaging about ~300 sites/year) and were overwhelmed with data storage and processing needs.  There were multiple data types  and data were in varied formats (.txt, .xls, etc).  

## What we found...

- Large quantity of historic data
- Multiple data types
- 	water quality
- 	habitat
- 	fish abundance
- No consistent data format
- No central data storage within agencies
- Agencies struggling with multiple reporting responsibilities

## Limiting Factor for any M&E Program

-  ‘Messy’
- field data

- Regional data analysis

- -centralized source
- -standardized format
- -metadata
- -processing tool
- -summary analysis
- -distribution

- -large volume
- -multiple types
- -multiple sources
- -multiple formats

**Notes:**
How do you integrate local projects with broad scale monitoring objectives. This issue is not unique to ISEMP. It has to be dealt with and we are in the best spot to deal with it.

## ISEMP Data Management Strategy

- Data Storage and Analysis
- RME Status and Trend Monitoring Database
- Metadata tracking
- Protocol manager tool
- Data dictionary
- Data Processing and Reporting
- Database templates
- Data Distribution
- Salmon Data Management website
- ISEMP website
- Geo-databases

**Notes:**
Transition: We propose the following strategy.

## RME Status and Trend Monitoring Database

-  Central Data Repository
- provides data validation, long-term storage, and backup
- supports multi-users
- supports web based user interface
-  Supports Data Analysis
- protocols attached to individual data value
- summary statistics for sites and years
- derived metrics
- selections based on network location (upstream/downstream)

## RME Status and Trend Monitoring Database Schema

- Protocol
- Manager

- Field Data and Derived Data
- (Individual Values)

- Geo-Spatial
- (NHD Plus)

- Watershed

- Stream

- Reach

- Monitoring Event

- Collection Characteristics

- Site Characteristics

- AttrName

- AttrValue

- AttrUnits

**Notes:**
This is a quick slide that highlights two simple points. Highlights that protocols and metadata are attached to individual data values. Also, highlights geo-spatial attributes which allow reach, watershed, or basin summary and upsteam/downstream navigation. Next slide.

## ISEMP Data Management Strategy

- Data Storage and Analysis
- RME Status and Trend Monitoring Database
- Metadata tracking
- Protocol manager tool
- Data dictionary
- Data Processing and Reporting
- Database templates
- Data Distribution
- Salmon Data Management website
- ISEMP website
- Geo-databases

## Metadata Tracking

- Protocol Manager
- Allows users to select from existing, modify existing, or create new protocol documentation
- Increases ease, accuracy, and efficiency of documenting protocols
- Data Dictionary
- Defines all metrics and values within protocols

## ISEMP Data Management Strategy

- Data Storage and Analysis
- RME Status and Trend Monitoring Database
- Metadata Tracking
- Protocol manager tool
- Data dictionary
- Data Processing and Reporting
- Database templates
- Data Distribution
- Salmon Data Management website
- ISEMP website
- Geo-databases

## Database Templates

- Data capture and reporting
- Standard format for data entry
- Supports multiple reporting responsibilities
- Technology transfer
- Leverages insights from historic data compilation
- Simple database design
- Supports new attributes or summary metrics
- Data Analysis
- Automates standard metrics and analysis
- Frees staff to ask more advanced questions

## Slide 13

**Notes:**
It’s a MS Access database that is relatively easy to understand. A project can have many sites and a site can participate in one to many projects. Sites have many data collection events. A data collection event consists of crew, protocol, equipment, and date/time. A data collection event can have many observations. In the WQ database template, we store observations of different water quality parameters in separate tables and currently we support 7 water quality parameters.

## Slide 14

**Notes:**
Forms allow easy data entry for field staff and provide data validation

## ISEMP Data Management 

- Data Storage and Analysis
- RME Status and Trend Monitoring Database
- Protocols
- Protocol manager tool
- Data dictionary
- Data Entry, Processing, and Reporting
- Database templates
- Data Distribution
- Salmon Data Management website
- ISEMP website
- Geo-databases

## Slide 16

**Notes:**
Screen shot of RME Status and Trend Monitoring Website

## Slide 17

## Slide 18

**Notes:**
Website to distribute information about the project, reports, maps, database templates, and other products.

## Geo-database

- MS Access database managed by ArcGIS
- Stores geo-spatial data (GIS layers)
- Contains metadata
- Single projection
- Easily distributed via web or CD
- Supports advanced geo-spatial analysis

## Slide 20

## Regional Data Analysis

- Central Repository

- Data Collection

- Database Template

- Protocol Manager

- Standard Analyses

- Approved Protocol

- Validation

## Deliverables

- 1.  Agency and regional tools for data processing:
- 	Template databases
- 2.  Centralized data repository
- Data storage
- Metadata tracking
- Data analysis
- 3.  Tools for data distribution
- 	Web interface database
- 	General project website
- 	Geo-databases

## The End

## Data storage and analysis

- EPA’s STORET
- No protocols
- No motivation from agencies
- No tools for analysis
- Streamnet
- Houses project descriptions, not necessarily  raw data
- USFS NRIS
- Fixed protocols

## Protocol Manager Schema

- _Protocol_

- _Program_

- _Project_

- _Method_
- _Attribute_

- _Method_
- _Shape_

- _Organization_

- _Reference_

**Notes:**
I could easily skip this slide. The point here is that Protocols are defined by their program, project, reference, organization, and methods.

## Adaptive Data Management

- Data Collection

- Protocol Manager

- Central Repository

- Regional Data Analysis

- Database Template

- Approved Protocol

- Standard Analyses

- Validation

- Protocols

**Notes:**
Protocol Manager provide protocols for data collection, while the database templates provide validation during data entry. The Central Repository provides persistent data storage, but can be updated on periodic basis. Regional Analysis performs scientific inquiry and defines approved protocols. As standard analyses are codified, then can be pasted down to local database templates to support analysis at local level. Additional results from protocol comparisons can be stored in Protocol Manager, thereby defining “Approved Protocol”. Both the Database Templates and the Protocol Manager are the “flexible” storage contains that allow local adaptation and incorporation of new methods or attributes.