# Presentation: 573951

## Multi-tiered Peer-to-Peer Data Distribution for Spatially Enabled Data

**Nazila Merati OAR/PMEL**
**Ann Matarese NMFS/AFSC**
**Jason Marshall NOS/CSC**

## Overview

- Data transfers are integral to NOAA’s mission
- In some cases, data are still transferred using CDs/DVDs and via “Sneakerweb” due to firewalls and security concerns
- Inter-disciplinary projects, which rely on data from other organizations for developing models and scientific research, can be delayed by slow data distribution

## How do we get data to collaborators faster?

- Peer to peer data distribution has been used by the physical science community to distribute model results and large scale datasets to collaborators.
- Works well in a academic setting with large servers and different security requirements
- Overnet and Gnutella as examples of file sharing systems

## Challenges we face

- Security concerns – getting data and keeping data secure
- Bandwidth – file sharing of large files can slow down other traffic
- Trusting peers and data sources – whose data are most correct and up to date?
- Data integrity – how can transfers keep data uncorrupted?

## Solutions

- Using COTS software that will allow us to transfer data in its native format – ArcPublisher/ArcReader
- Using a online collaborative tool to view maps/review and whiteboard solutions – Toucan and GrooveSite
- Testing a secure peer to peer network – BestPeer for GIS data transfers/JXTA
- Building a central repository of data sources and creating a trusted peer network within NOAA for data distribution using Webservices

## Test Case – Alaska Fisheries Science Center 

- AFSC Recruitment Processes group has developed an Atlas of Fisheries Abundance for major icthyoplankton groups in the North Pacific.
- Data are stored in a GIS, but currently only a hard copy Atlas is available.

## Physical Data 

- AFSC needs access to PMEL/FOCI data
- Data currently stored in netCDF, but conversion tools exist to convert data to shapefiles
- Cruises of data can be delivered via ArcPublisher

## Data delivery – who are peers?

- Data (occurrence and size distribution) need to be distributed to collaborators within AFSC working on ecosystem assessments as well as to collaborators at PMEL in the FOCI program and vice versa.
- Data are also needed by outside collaborators at the UW Fish Collection.

## Constraints

- Data are to be distributed with a time lag and to very specific groups
- Physical data (CTD, MBT and XBT) taken at the same time are also needed for recruitment studies
- All groups need base layers – bathymetry, shoreline, graticules, isobaths

## First Technique tried

- ArcPublisher for ArcGIS 9.1 was used to create maps of icthyoplankton abundance and distribution for peers within and outside the Recruitment Processes group network and sent via email
- Maps can be customized with a corporate look and feel (NOAA logo, disclaimers, information and links)
- Maps can have specific access-control attributes

## ArcPublisher/ArcReader for data transfer

- Maps are viewed using ArcReader, a stand-alone free client available from ESRI or can be added to existing ArcGIS application
- Data can be extracted and edited or just viewed depending on the level of trust within the network

## ArcPublisher

- Map authored in ArcGIS including layout
- Publisher will allow map creator to determine levels of access control in Publisher options
- Identical look and feel to ArcGIS, so integrates seamlessly with existing projects
- Easy to author and package

## ArcPublisher Security and Control

- Password Protection to open map in ArcGIS or an IMS application
- Data Packaging and map/data timeouts are options – good for data that need to be refreshed
- Raster, vector data can be distributed in map document

## More enhancements in ArcPublisher 9.2

- Publish Geodatabases of data
- Connections to ArcWeb Services
- More service oriented architecture, integrating with ArcGIS 9.2

## GeoCollaboration on the Web 

- Many on line project management sites are used by cross-line office projects
- (Grovesite, Groovesite, Basecamp) all work to help organize large teams with geographically separated collaborators to work more efficiently
- Many tools can be extended by third party extensions to work with mapping applications

## Geocollaboration can be used in a secure networking environment

- Data and documents are placed in secure areas for teams to review and comment on using messaging areas or white boards. In some cases, versioning of data is handled. Can be crucial with regularly updated maps and figures.
- Can link to ArcIMS or Webmapping services to provide static updates to site specific projects
- Downside – some PM tools may still be platform dependent
- May be built, may not be used by PIs and managers

## Groovesite and Toucan Navigate

- Groovesite is used by groups large and small for project collaboration. Developers have added GIS functionality to Groovesite to allow real time on line collaboration in a map room.
-     Uses Toucan systems’ own GIS, but can edit and view shapefiles and MapInfo Format files

## True Peer to peer solutions – build your own and they will use it

- Programmer code exists to build your own peer to peer network that has reliable security and will work with large scale data sets
- JXTA (Sun Microsystems) has large code repository and deals with many of the security issues faced by NOAA
- Interoperable and platform independent
- Successful in distributing large volume data sets –used in many universities for parallel and grid computing
- Requires programmer time and strong support from NOAA IT security officers

## BestPeer Architecture

- BestPeer application is written specifically for GIS file types, uses UDDI and LIGLO for data discovery
- Secure security model and global identities to keep peers uniquely identified
- Maybe tested at AFSC, used in University communities
- AFSC is more one to many case

## Long term options for data distribution

- Building a distributed data management system within NOAA that would allow data sharing of base data layers so that we could
- Reduce duplication of effort and data storage of most used data sets
- Create a practice and protocol for peer recognition (using webservices and service discovery using UDDI)

## Potential Applications 

- Project Management on a NOAA wide level using geocollaboration framework
- Disseminating data and maps to first responders – via the web and hard copies with the best available data – coastal storms, oil spills, tsunami response in field conditions
- Distributed data and sharing of geospatial data within NOAA – avoiding duplication of effort by using a trusted peer network
- Integrated Decisions Support Systems – collaboration tools in emergency planning and mitigation

## Next Steps – Peer to peer collaboration

- Investigate the use of Groovesite and Toucan for collaborative map making
- Test larger data sets in ArcPublisher to see how file transfer works, install new release and test extended functionality
- Test a open-source peer to peer data network using JXTA framework to test security models and data delivery with PMEL, CSC and AFSC