# Presentation: 289716

## Server-side OPeNDAP Analysis – Concrete steps toward a generalized framework via a reference implementation using F-TDS

- Roland Schweitzer
- Weathertop Consulting, LLC
- Steve Hankin and Ansley Manke NOAA/PMEL

## Stuff I thought I’d get done this summer, but now can only talk about and ask for help.

- Roland Schweitzer
- Weathertop Consulting, LLC

## Highlights

- Server-side analysis
- Motivation
- Implementation
- LAS as OPeNDAP Client and Server
- Implementation of the server (and why you might care)‏
- Community participation and “call to action”
- Summary

## Server-side Analysis

- In general server-side analysis is a computation made by an OPeNDAP server at the request of a client.
- The specification of the computation is transmitted to the server via the OPeNDAP URL.

## Motivation and Implementation

- We are interested in server-side analysis for use with the Live Access Server (LAS).
- Primarily to implement comparisons between data defined on different grids
- Our implementation borrows heavily from GDS and our experience of running a legacy application (Ferret) from within a Java runtime environment.

## The Live Access Server (LAS)‏

- The Live Access Server (LAS)‏

## LAS Architecture

**LAS Architecture**

- Product
- Server

- SQLBackend Service

- DRDSBackend Service

- Metadata(XML)‏

- LAS Product Server

- client

- product

- metadata

- product request XML (REST)‏

- back endrequest (SOAP)‏

- OPeNDAP server

- DRDS server

- Local netCDF data

- FerretBackend Service

- Ferret

## Comparing OPeNDAP datasets

**Comparing OPeNDAP datasets**

- metadata

- product request XML (REST)‏

- back endrequest (SOAP)‏

- Product
- Server

- SQLBackend Service

- DRDSBackend Service

- Metadata(XML)‏

- LAS

- user

- product

- OPeNDAP server

- OPeNDAP server

- Suppose the variables are on different grids?

- FerretBackend Service

- Ferret

## LAS as an OPeNDAP Server

- Data on grids which are available via LAS are guaranteed to be geo-referenced and at least COARDS compliant.
- We can often “repair” (including re-gridding) the data and/or metadata by writing a script of Ferret commands.
- Wouldn’t it be nice to make these “repaired” data available via OPeNDAP?

## The Ferret-THREDDS Data Server

- F-TDS makes this possible.
- F-TDS provides an OPeNDAP view of the data being served by LAS and can make any transformations specified by the associated script before serving the data.
- F-TDS also implements server-side analysis (including the ability to pass in external data sources).

## A GDS Digression

- The GrADS Data (DODS) Server was an early implementation of this concept.
- Both GDS and F-TDS use the Java Runtime environment to invoke the associated legacy app (GrADS or Ferret) to do the heavy lifting.

## F-TDS Capabilities

- F-TDS takes advantage of several characteristics of Ferret.
- New "virtual" data variables can be defined
- Can build the metadata (netCDF header described by dimensions, coordinate variables and the structure of data variables) without performing any heavy calculations for both data read from files and “virtual” data variables
- Only performs calculations when the data are requested
- Only calculates the minimal set needed to fulfil the current request

## F-TDS Evolution

- The Java netCDF library allows new data container formats to be plugged-in by implementing the I/O Service Provider interface.
- Once “plugged-in” clients using nj22 have access to the data from this container.
- We implemented a Ferret I/O Service provider which can read Ferret command scripts and direct Ferret to perform the calculations as needed to satisfy data requests.

## The THREDDS Data Server

- TDS is an OPeNDAP server based on netCDF Java.
- nj22 I/O Service Providers can be plugged in to TDS.
- The combination of the Ferret I/O Service Provider and TDS (aka F-TDS) serves via OPeNDAP data which are represented by Ferret command scripts (both data read from disk by Ferret and virtual data computed on-the-fly by Ferret).

## The Why You Should Care

- If you already serve data via TDS you can add server-side analysis to your TDS for the cost of installing Ferret, and a few Java classes and configuration files...
- and what every cycles get used by the analysis...
- which may be cheaper than the data transfer

## F-TDS and Server-Side Analysis

- A DataSource handler can also be plugged in to TDS which allows custom handling of OPeNDAP requests based on the contents of the URL.
- We built such a DataSource handler which recognizes URLs of the form:
**http://machine:port/thredds/dodsC/**
**_expr_****{dataset1,dataset2,...}**
**{expression1;expression2;...}**
**.URLsuffix?projection**

## LAS Example

## F-TDS Specific Example

- _[http://server.gov/thredds/dodsC/data/levitus](http://server.gov/thredds/dodsC/data/levitus)_
- _expr_
- {http://server.gov/thredds/dodsC/data/coads}{let sst_2_regrid=sst[d=2,gxy=temp[d=1]]}
- .dods?SST_2_REGRID[0:1:0][67:1:131][0:1:14]
- /thredds/dodsC/data/levitus
- .dods?TEMP[0:1:0][67:1:131][0:1:14]
- /coads
- .dods?SST[0:1:0][33:1:66][124:1:179]

## Community Action

- Can we define a general syntax for server-side analysis requests?
- Some common operations (ave, diff, linear interp.) with standard syntax
- Still allow server-specific (native) operations
- Can we define a general syntax for geo-referenced projections?
- ERDDAP’s (180),10,(360) syntax
- TDS’s north=17.3&south=12.088&west=180&east=360.0 netCDF subset syntax

## OPeNDAP “Working” Group

- Working is in quotes because we have not been doing any work recently
- See:
- http://docs.opendap.org/index.php/
- Server-side_Functions

## Summary

- Server-side analysis is critical for LAS.
- NetCDF Java and the THREDDS Data Server is a great platform for implementing this type of analysis with a legacy analysis application (like Ferret).
- A community-developed server-side analysis framework would make it easier to get the advantages of server-side analysis from other servers.

## IOSP*

- *Stolen directly from John Caron with only this measly acknowledgement.

- Application

- The low-level part of the NetCDF-Java version 2.2 architecture

- NetCDF-3

- HDF5

- I/O service provider

- GRIB

- GINI

- NIDS

- NetcdfFile

- NetCDF-4

- Nexrad

- DMSP

- Java Runtime

- Ferret

- GrADS

- readData

- open

- isValid