# Presentation: 761381

## DAPPER: An OPENDAP Server for In-Situ Data

- Joe Sirott
- Donald W. Denbo, Willa H Zhu
- University of Washington
- PMEL/NOAA

## What is Dapper?

- Web server that provides distributed access to in-situ data via OPeNDAP protocol
- Clients include GrADS, Matlab
- Available data:
- PMEL EPIC database
- NODC World Ocean Database 2001

## Problems with in-situ data

- Irregular (x,y,z,t)
- Not all variables available in all profiles
- Data from a given profile may be revised as QC is applied
- Each profile usually stored as individual file
- No standard APIs

## Problems with in-situ data

- Irregular (x,y,z,t)
- Not all variables available in all profiles
- Data from a given profile may be revised as QC is applied
- Each profile usually stored as individual file
- Use Climate Data Portal for aggregation

## Problems with in-situ data

- No standard APIs
- Use OPeNDAP Sequence datatype

## Dapper architecture

**CORBA (IIOP)**

** **

**Climate Data Portal **

** **

**MySQL**

** **

**Java netCDF**
**library**

**Dapper**

**OPeNDAP client**

**OPeNDAP Protocol**
**(HTTP)**

## Climate Data Portal (CDP)

- CORBA based
- Metadata in database (MySQL)
- Variables (measurements) in individual netCDF files
- Developed by Don Denbo and Willa Zhu

## OPeNDAP

- Sequence example (header):
- Dataset {
-     Sequence {
-         Float32 longitude;
-         UInt32 time[tsize = 2];
-         Float32 latitude;
-         Int32 _id;
-         Sequence {
-            Float32 depth;
-            Float32 temperature;
- 		  } profile;
-     } location = { { -42.01, {77, 3850926208}, 53.3402, 3645, { { 3, 16.698 }, { 5, 16.71 }, ...

## OPeNDAP (cont.)

- Sequences accessible to clients via Java or C++ libraries
- _id field in “outer” sequence allows unique identification of an individual profile

## Dapper design

- Consists of configurable *services*
- Climate Data Portal service
- netCDF service
- Services route OPeNDAP request
- Implemented as Java servlet

## Dapper architecture revisited

**CORBA (IIOP)**

** **

**Climate Data Portal **

** **

**MySQL**

** **

**Java netCDF**
**library**

**Dapper**

**OPeNDAP client**

**OPeNDAP Protocol**
**(HTTP)**

## CDP service example

- HTTP request:
- 	_[http://foo.org/.../](http://foo.org/.../dset.cdp?lon,lat&lat%3E0&lat%3C20)__[dset.cdp.dods?lon,lat](http://foo.org/.../dset.cdp?lon,lat&lat%3E0&lat%3C20)__[, ](http://foo.org/.../dset.cdp?lon,lat&lat%3E0&lat%3C20)__[depth,_id,temp&lat](http://foo.org/.../dset.cdp?lon,lat&lat%3E0&lat%3C20)__[>0&lat<20](http://foo.org/.../dset.cdp?lon,lat&lat%3E0&lat%3C20)_
- Dapper contacts CDP via CORBA IIOP
- CDP accesses MySQL for metadata, appropriate profile files for data
- CDP returns data to Dapper via CDP API
- Dapper returns to client as OPeNDAP sequence

## CDP service performance

- < 500 us/profile for coordinate info
- 5 ms/profile to read one variable with z axis containing 1000 pts
- netCDF library enhancements:
- Parsing of netCDF header/attributes is optional
- Use String(char[]) constructor instead of String(byte[]) constructor
- 2X speed up

## netCDF service

- Individual profile files available as OPeNDAP Grid datatype
- netCDF library enhancements:
- True streaming server

## Future work

- Stream data from CDP
- Distribute netCDF file processing using scatter/gather patterns

## Availability

- _[http://www.epic.noaa.gov/dapper](http://www.epic.noaa.gov/dapper)_
- Joe.Sirott@noaa.gov