# Presentation: 498608

##  SNAAP Eric Kihn and Mikhail ZhizhinNOAA/NGDC

**GO-ESSP Meeting**
**Sept 18th, 2008**

- Simple NOAA Archive Access Portal

## What is CLASS?

## SNAAP Goals

**Integrate diverse systems via a standards based, user focused WS interface**
**Demonstrate the concept of “fundamental separation” of archive and storage from access**
**Create a tool that shows the benefit of API’s for users**
**Discover new technology and evaluate cutting edge tools for handling integration and presentation**
**Document integration of multiple data types and sources through an single API (time-series, grid, swath, etc..)**

**Notes:**
The bit about “fundamental separation”  comes from the CLASS project history.  Basically at one point it was believed CLASS would do all access, storage and user interaction for NOAA data. The SNAAP demo shows that you can have multiple systems hidden under a common layer and make it look like one to the user. 

The SNAAP development is being documented in a series of white papers (GIS, metadata, OE, etc) that are released as we cover a new area.

## The Data Environment Now

- CLASS ADS

- Web Interface

- Subscriptions

- Human

- Machine

- Data

- Providers

- Visualization

- Inventory

- Order

- NMMR/Metadata

- NGDC/SPIDR

- Web Interface

- Web Interface

- Metadata

- SPIDR WS

- SNAAP

- Web Interface

- Metadata

- Data Access

**Notes:**
First this is an animated slide so do it in sequence.   Basically it’s the was things are now to the way we’d like them to be.  From diverse and separate metadata, data, enterprise data to an integrated set of resources.

## Three Simple Items

**Search for appropriate data sets**
**Discover data by time, type, location, abstract, etc. **
**Support for multiple metadata schema (FGDC, IEEE, SPASE) **
**Standards based (xQuery and REST)**
**This is in CLASS 4.3.3 – awaiting policy**
**Order Data**
**Needs a standard way to describe ordering options**
**This is done via Ordering Extensions (OE)  which are linked to standard metadata**
**Needs some framework for service integration (i.e. transforms)**
**Provider Integration Layer**
**We are using OGSA-DAI for the demo **
**We offer both a SOAP and REST interface (different functionality)**
**Its important that the backend retrieval not effect the consumer.**

**Notes:**
This is the key slide for the whole talk.  For all the complication in system integration this says its really three simple things,  “what do you have?”, “how do I ask you for it?”,  how do I create a back end so diverse systems can be accessed via a common layer.  People usually get lost when we start dealing with semantics, units, translation and mapping.  The point is this API just lets you get to the data with a description and doesn’t try and solve more.  Higher order systems can.

## So a new  provider wouldhave to ...

**Create a standard metadata record for the data set.  (They should be doing this anyway).**
**Create an Ordering extension document.  If they are offering web distribution this is trivial.**
**Write an adapter to the middle layer, unless they have a supported service (like JDBC, FTP, etc..) in which case adapt an existing one.**

**Notes:**
Emphasis here is that a provider would face minimal hurdles to integrate with the API.  We’ve found that if you make it tough you eliminate large percentages of people no matter how clever the functionality provided.

## Advantages

**This approach mirrors that of the highly successful OGC specification for map services (what, make, retrieve)**
**It minimizes entry burden for new providers**
**Its simplicity makes it scalable**
**It doesn’t preclude or interfere with on-going activity, just opens up new integration possibilities.**

**Notes:**
The parallel with the OGC is important here.  Basically that API has the what, how, do  functionality and we think the same applies for data.  The fact that the translations here are not data specific but simply ordering syntax means that the load is minimal and it scales well.

## Current snapshot of the SNAAP API architecture

**Notes:**
Metadata retrieved via rest call.
Ordering extensions parsed out and submitted to API.
Appropriate plug-in is invoked and the order extracted.
We are using OpenDap as a delivery option not an accessor, though an OGSA-DAI adapter could be written.
Delivery options are data type specific (grid, time-series, swath, etc...)

## Data Order Flow

**Notes:**
The big item here is that it is XML flowing around and being translated to the appropriate target (like a user display for options).  Same flow as described in the previous slide just broken down.

## CLASS Ordering Extensions (OE) schema upper level

**Notes:**
Now we are looking at the OE schema.  The point here is just to show the types of things it covers at a high level like, Maps, stations, etc...
We’ll drill down into details in the next couple of slides.  A point here is that the client whoever it is would just need to parse this in ordeer to place an order with SNAAP.

## CLASS OE schema: parameters and time

**Notes:**
Here’s an example of parsing the items.  The OE XML is on the left and the OE parsed into a HTML form is on the right.

## CLASS OE schema: map and ROI

**Notes:**
Further example using maps.  Maps have all sorts of optional layers like grids, default regions, stations, points, etc.  Its all about being able to describe spatially selectable items.

## CLASS OE schema: station map

**Notes:**
Down further into station XML.  The OE is NOT a complete specification yet.  We think we can make it so with some review and new data to test against.  Right now it handles time-series, stations, grids, swaths, and profiles.  I expect it will need to be extended as it gets deployed.

## Why OGSA-DAI?

**Standard tool in the Grid community**
**Includes a “process” for extraction**
**Built in support for asynchronous transactions**
**Includes support for domain login**
**Looked at alternatives like OpenDap, WCS, Custom applications some documentation forth coming**
**Problem: Its very complex and not well accepted in NOAA**

**Notes:**
Two key things in the choice of DAI are  1) It allows for a process as part of extraction.  Data retrievals are often multi-step in our demo.  So a simple example might be retrieve via FTP and TAR the process document describes this.  2) Support for asynchronus transactions is key like in a tape extraction.  This is where we couldn't use OpenDap.

## Visualization screens – 1time series and map server

**Notes:**
Some demo screen shots of data accessed via the API.  First just time-series data returned via XML from a point and a grid returned and contoured for a region.

## Visualization screens – 2NASA World Wind plugins

**Notes:**
Swath, grid and image data displayed in NASA’s WorldWind client and retrieved via an adapter for SNAAP.  The DMSP image was first sent to Thredds and then accessed from the client. 

## MS Excel and NASA World Windinteroperability for trajectory data

**Notes:**
Here we define a track in Excel and then use the API to retrieve data along that track and populate a spreadsheet.  This was a real case in which migratory birds were being correlated with weather grids.

## Future plans

**Extend the OE documentation to the CLASS development team and provide support**
**Extensive work on the connector layer**
**Expand proto-type to cover more NOAA and non-NOAA data types and services and document lessons learned**
**Release the work to the public **_**[http://sourceforge.net/projects/vxoware/](http://sourceforge.net/projects/vxoware/)**_
**Seek outside review**
**Enhance the connection with GEO-IDE**
**Will identify a production build and date for SNAAP operations**

## BACK UP SLIDES

## Asynchronous Ordering

**CLASS API Demonstration**

## Orbit Inventory List

**Notes:**
When you search for swath (orbit data) the return is a file listing which is parsed here.

## User Basket (Order & Check-Status)

**Notes:**
Those files are added to a user “basket”  which is really a collection of granule status markers on the server.  The client can then poll the server to
Find out the status of a given granule (done, processing, not-ready, error).

## Order System Entities

***Order System Entities***

- Granule List now using ECHO XML in the return
- New entity “Data Basket” – handle to collection of order
- Demo currently orders all data centrally will need to be adapted

**Notes:**
The entities involved in asynchronous ordering.

## Order Sequence Diagram

- Data basket ID is the API handle
- OGSA-DAI activities handle the extraction and packaging
- Status check is currently by basket only (XML return)
- Finished granules return a URL handle to the entity