# Presentation: 630952

## Designing and Managing Fisheries Data Systems that Support the NOAA Data Quality Act: A Case Study Using the Hawaii Longline Observer Program

- Karen Sender and Janet Pappas
- NOAA Fisheries - Pacific Islands Fisheries Science Center
- Honolulu, Hawaii
- FIS Conference 2003
- 18 November 2003

## Asking the Hard Questions...

- Can I prove that my data is as good as I think it is?
- Do I really know why we’ve been collecting that piece of data for the past ten years?
- Do I worry that my data will be misinterpreted when it is released?
- Can I really be sure that my data hasn’t been corrupted?
- Do I dread running into my data users in the coffee room?
- Can I handle even a minor system modification?
- Is linking related data sets a nightmare?
- Are my users all reporting off the same instance of the data?

## Poor data quality
Data contamination during processing steps
Inadequate documentation
No change control procedures
Poor communication between data role groups
Unreasonable time for data dissemination
Summary reporting from multiple – and different – data sources

- Data contamination during processing steps
- Inadequate documentation
- No change control procedures
- Poor communication between data role groups
- Unreasonable time for data dissemination
- Summary reporting from multiple – and different – data sources

- A Mission Born out of Passion and Frustration...

## What Is Data Management?

*It’s Not Just Archiving!*

**Notes:**
Understanding the scope of data management...

## Slide 5

**Notes:**
       Identification of data to be included in the data resource
       What data need to be shared?
       Will sharing a data set increase its value and the value of an existing data set?

Defining business rules for the data elements
       What are the valid ranges of a given data element?
       What does a null value mean versus a zero in a data field?
Designing the data models
       Is the data model compatible with other data sets with which it needs to be integrated?
Designing data element naming/format standards
       Can users easily understand what the data fields represent?
       (6 names for temperature and multiple units of measure)
Complying with hardware/software standards
       Are we using agency-approved hardware and software tools?
       Are our current tools maximizing data quality and facilitating data access?
Performing routine and disaster recovery system administration
Are we performing both preventive maintenance and planning for catastrophic failures? 

## Slide 6

**Notes:**
Identifying not only the data elements desired but also any additional information that will be required to make use of those data.
       Do data records require unique identifiers?
       Do any reference tables need to be designed or can existing ones be used?
Developing clear, concise manuals and instructions for data collection.
Are the instructions for data fields unambiguous?
Designing user-friendly paper and electronic forms for data capture and entry 
       Is data quality being compromised at data collection because the forms are too complicated or difficult to fill out?
Building data validation schemes in data entry applications 
       Are unreasonable data excluded from the data set at data entry?
       Do you ever want to see north longitudes in your data? 
Maintaining data set documentation and history
Can you reproduce the path of a piece of data from data collection through data extraction should you be required by some legal review process? 

## Slide 7

**Notes:**
Storing data in the centralized data resource
       Are data-transfer routines automated and fully documented?
       Have they been proved not to introduce errors or modify the data?
Detecting and reporting errors via data monitoring applications
       Do you want to have your users catch the errors before you do?
Maintaining a history of when and by whom data are added, modified or deleted
       Not only is this required for data security considerations, it provides information that can be used for process improvement.
       How quickly is the data set growing?
       What are the most common errors that are edited post data entry and can we eliminate them?
Periodically auditing data by tracking data from collection to dissemination via the data path.
       Are we ensuring that data are not lost or contaminated?
       The first time your data are audited, do you really want it to be by an outside auditor?
Developing and following formal change control procedures
       Are all changes documented?
       Have the changes been tested in a development environment?
       Has proper notice been provided to all role groups?
Developing and documenting data processing procedures
       Can you fully document the path of your data?  
Developing and testing applications before release 
       Can data downtime and user frustration be minimized?

## Slide 8

**Notes:**
Providing user support via training and problem resolution
Are the users provided with proper tools and information to allow access to the data and metadata?  
Complying with data accessibility issues
Have data access tools been reviewed and tested by an outside, independent reviewer?
Ensuring data availability and ease of integration with other data sets, as needed
(Whether this is within the lab or across the agency.)
Establishing data security
Are all appropriate users granted access in a timely manner?
Is access revoked promptly when necessary?
Complying with data publication formats and deadlines
Do final data products such as maps, reports and web pages include standardized logs, scales and necessary metadata and disclaimers?
Do dates clearly indicate the point in time when data was extracted? 

## Data Management Areas

## Who Is Responsible forData Management?

*All data role groups!*

**Notes:**
Understanding the data management role groups...

## Slide 11

**Notes:**
All data management role groups must live within the policies and guidelines of DOC, NOAA and NMFS.

## Slide 12

**Notes:**
Data Stewards set policy for the collection, management and accessibility of the data set and its metadata to ensure compliance with agency data management policies, mandates and relevant state/federal laws.

Data Stewards
Have planning and policy-level responsibility in their functional area.
Document policies and procedures for access and use of the data set.
Ensure that the data set is stored, managed and accessed in the enterprise database per the agreed upon data management policy.
Periodically review costs and benefits of continuing to maintain the data set.

## Slide 13

**Notes:**
Data Managers
Perform and supervise operational management of their data sets per the data management policy, data administration guidelines and the data set policies set by the data stewards.

Data Managers
Have operational-level responsibility for data management activities related to data
Capture
Maintenance
Dissemination

## Slide 14

**Notes:**
Database Administrator
Has responsibility for the physical data resource:
Generating physical database schema
Performing database tuning
Creating database backups
Planning for database capacity
Implementing data security requirements

## Slide 15

**Notes:**
Data Users are individuals requiring access to agency data in the course of meeting the requirements of their position and anyone in the public who wishes access to public information held in the data resource.

## Slide 16

## Our Mission...

- Ensure access and dissemination of quality Fisheries data and metadata to appropriate users in a timely manner.

**Notes:**
Committing to solving the data problems...

## Jan Pappas:
PIFSC systems DBA
Oracle Project Team member
Karen Sender:
JIMAR contractor
DBA/Computer Specialist for ITS Oracle Team 
Scientific Information Systems background

- PIFSC systems DBA
- Oracle Project Team member
- Karen Sender:
- JIMAR contractor
- DBA/Computer Specialist for ITS Oracle Team
- Scientific Information Systems background

- The Technical Team

## Our Ground Rules...

- Look for data management resources that already exist
- Commit to data quality and data transparency
- Develop reusable data management tools

## Our Ground Rules ...(cont.)

- Involve all the data role groups
- Adopt the data stewardship paradigm
- Design in process improvement schemes
- Minimize points-of-failure

## Our Ground Rules ...(cont.)

- Completely document every aspect of the data system
- Provide data management education at every opportunity
- Keep it simple
- Try the plan on an existing data system

## Hawaii Longline Observer Program

- Collecting data since 1994
- 5-25% coverage of Hawaii Longline fishing trips
- Related data sets – Logbook, landings
- Post trip data entry by the observer in office

## Stuart “Joe” Arceneaux:
PIRO Observer Training Coordinator
Thomas Swenarton:
PIRO Debriefer
Tina Chang:
FIS IT Coordinator/DBA
NASDAQ and NASA Hubble systems development

- PIRO Observer Training Coordinator
- Thomas Swenarton:
- PIRO Debriefer
- Tina Chang:
- FIS IT Coordinator/DBA
- NASDAQ and NASA Hubble systems development

- The Support Team

## Data Role Group Concernsof the Existing System

- End User complaints on data quality
- No clearly defined process for reporting/resolving data issues
- Database model deficiencies
- Excessive _[points-of-failure](slide25.xml)_
- Data transfer arduous
- Labor intensive effort for any system modification
- Redundant and ambiguous data elements
- Burden on data collectors

## Observer E Data Flow

## Data Role Group Concernsof the Existing System

- End User complaints on data quality
- No clearly defined process for reporting/resolving data issues
- Database model deficiencies
- Excessive points-of-failure
- Data transfer arduous
- Labor intensive effort for any system modification
- Redundant and ambiguous data elements
- Burden on data collectors

## Team Focus Areas

- Data Quality
- Ease of Use (collection and data entry)
- Data Management at the source (PIRO)
- Data Security
- System reliability and stability
- Data Harmony with national FIS objectives
- Compliance with NOAA Data Quality Act

## Data Quality

- Document definitions for all data elements
- Develop Data Issues Management tool
- Develop and enforce change control procedures
- Track Data set history/activities
- Promote the Data Stewardship model

## Ease of Use

- Use web based interface
- Design in data validation at data entry
- Develop user controlled data set validation scheme
- Secure web interface using database roles
- Design user friendly feedback and error messaging

## Data Management at the Source

- Administration tools at PIRO to manage
- reference codes
- validation algorithms
- users
- database roles
- Project circulars, activities log, news, FAQ

## Data Security

- Each data entry person and debriefer to log on with own user name and password
- Database user roles to provide appropriate read, insert, update, and delete privileges
- History – who, when, what – for each database transaction to be maintained in the database
- Use NOAA FITS approved technologies

## System Reliability

- Connection between PIRO and HL is through direct T1 line, independent of internet
- Reduction in number of points-of-failure
- Automated nightly backup of database

## Harmony with National FIS

- Design to consider ACCSP(FIS) data structure and objectives
- Standardize field names and data types
- Manage metadata for all data elements in a data registry

## NOAA Data Quality Act

- Requires a process for reporting/resolving errors
- Requirement for data transparency, quality, integrity, and utility
- Need for clearly defined data management practices

## Accomplishments to Date

- Improved communication between data role groups
- Improved data quality and data utility
- Better data, happier users
- Additional data in the enterprise database
- Elimination of redundant data items
- Computer calculation replaces human calculations

## Accomplishments (cont.):Data Model Enhancements

- Model design that ensures data stability and integrity
- Use of appropriate data types to facilitate summary calculations
- Data history tables track inserts, updates and deletes as well as who, when and what data was changed
- Reference code management allows code changes without model changes

## Accomplishments (cont.):LODS Applications

- LODS Home
- Data Entry Application
- Data Element Registry
- Data Validation Manager
- Data Issues Manager
- Reference Code Management
- LODS Admin

## Accomplishments (cont.):LODS Applications

- LODS Home
- Data Entry Application
- Data Element Registry
- Data Validation Manager
- Data Issues Manager
- Reference Code Management
- LODS Admin

## LODS Home Page

## Accomplishments (cont.):LODS Applications

- LODS Home
- Data Entry Application
- Data Element Registry
- Data Validation Manager
- Data Issues Manager
- Reference Code Management
- LODS Admin

## Trip Tab

## Set Tab

## Issues Tab

## Trip Data History Tab

## Trip Summary Tab

## Catch Tab

## Birds Tab

## Turtles Tab

## Accomplishments (cont.):LODS Applications

- LODS Home
- Data Entry Application
- Data Element Registry
- Data Validation Manager
- Data Issues Manager
- Reference Code Management
- LODS Admin

## Data Element RegistryDatabase Information Tab

## Data Element RegistryData Collection Tab

## Data Element RegistryReferences Tab

## Data Element RegistryHistory Tab

## Data Element RegistryData Access Tab

## Accomplishments (cont.):LODS Applications

- LODS Home
- Data Entry Application
- Data Element Registry
- Data Validation Manager
- Data Issues Manager
- Reference Code Management
- LODS Admin

## LODS Data Validation

## Accomplishments (cont.):LODS Applications

- LODS Home
- Data Entry Application
- Data Element Registry
- Data Validation Manager
- Data Issues Manager
- Reference Code Management
- LODS Admin

## LODS Data Issues Manager

## Accomplishments (cont.):LODS Applications

- LODS Home
- Data Entry Application
- Data Element Registry
- Data Validation Manager
- Data Issues Manager
- Reference Code Management
- LODS Admin

## Reference Code Management

## Measurement Type Codes

## Reference Code Versioning

## LODS Reference Codes

## Accomplishments (cont.):LODS Applications

- LODS Home
- Data Entry Application
- Data Element Registry
- Data Validation Manager
- Data Issues Manager
- Reference Code Management
- LODS Admin

## Project Activities Log

## FAQ Manager

## LODS News

## Project Circulars Management

## Persons Management

## Data Role Management

## Species Code Management

## Table Version Management

## Future Opportunities

- Apply tools and techniques developed for LODS to other data systems
- System process improvement from Issues Management and Data History statistics
- Integration of documentation tools and techniques into _[In](slide82.xml)__[P](slide82.xml)__ort metadata repository_
- Work with FIS Professional Specialty Groups (PSG) to develop and promote NMFS data management best practices

## Reviewing the Process

- Overcoming the politics of change

## Reviewing the Process

- Overcoming the politics of change
- Doing it the *Hard Way*

## Reviewing the Process

- Overcoming the politics of change
- Doing it the *Hard Way*
- FIS to the rescue

## Everything I know, I learned at sea...

- Change happens, enjoy it!
- Passion and energy can often overcome a lack of resources
- If you want to keep the boat afloat, you need the whole crew
- If you want the boat to go in the right direction, you need a captain

## Final Thoughts...

- The most accurate data are not defensible without adequate documentation
- Data management policies and guidelines need to be adopted
- Data management needs to be a forethought, not an afterthought when planning and budgeting projects
- Quality data management is not rocket science
- GIS, FGDC, NDQA, Data Integration require good data management

## Data are the foundation of our success

## EndDesigning and Managing Fisheries Data Systems that Support the NOAA Data Quality Act: A Case Study Using the Hawaii Longline Observer Program

- Karen Sender and Janet Pappas
- NOAA Fisheries - Pacific Islands Fisheries Science Center
- Honolulu, Hawaii
- FIS Conference 2003
- 18 November 2003

## AddendumThe following ten slides are presented to describe InPort, our working model of an integrated data registry and metadata repository

- Karen Sender and Janet Pappas
- NOAA Fisheries - Pacific Islands Fisheries Science Center
- Honolulu, Hawaii
- FIS Conference 2003
- 18 November 2003

## InPort Metadata Repository

- Maintain a complete inventory of data
- Metadata are readily available on-line
- Automatically generate near final FGDC metadata records
- Reduce the waste of time and money
- Facilitate the sharing of data and information

## InPort Information Objects

## InPort Common Components

## InPort Relationships

## InPort Main Page

## InPort Search

## InPort Search Results

## InPort Object Profile

## InPort Object Profile

## InPort Object Profile