Server­side Data Reduction and Analysis with Script Workflow Analysis for MultiProcessing (SWAMP)
Creators: name: Wang, Daniel L.
name: Zender, Charles S.
name: Jenks, Stephen F.
Publisher: name: Zenodo
Year: 2007

description: This slide deck was presented as part of the OPeNDAP Developer's Workshop, Winter 2007 that was held from February 21-23, 2007 at the NCAR Foothills Laboratory in Boulder, CO. 

Abstract: 

Despite the inexorable advance of faster, better, and cheaper computing hardware, terascale data reduction and analysis remain elusive for most. Massive amounts of netCDF data remain underutilized due to scientists' limited bandwidth and computational capacity. Our strategy relocates computation closer to data sources to minimize bandwidth usage and increase efficiency. We apply our server-side analysis framework, the Script Workflow Analysis for MultiProcessing (SWAMP) system to specify and perform computational workflows. Instead of running locally, scientists' netCDF Operator (NCO) data analysis scripts are processed and sent via Data Access Protocol (DAP) to a modified OPeNDAP server, where they are parsed, optimized, scheduled, and executed. Instead of receiving raw input data to be computed locally, scientists distribute computation and merely receive the reduced output data, while leveraging their legacy script-based analysis methods. Our benchmarks quantify the reduced bandwidth and improved execution time. Local computation and bandwidth requirements are drastically reduced, freeing the scientist to perform exploratory analysis and discovery in wider scopes and finer resolutions, and reducing time to discovery.

 

AGENDA OF THE WORKSHOP 

[Almost all of the presentation listed here are available in the OPeNDAP Community Zenodo library.] 

Wednesday February 21, 2007

Session 1

0900-1000 OPeNDAP Current Status



Introduction by Peter Fox, President of OPeNDAP


1000-1015 Break

Session 2 (moderator: P. Fox)

1015-1200 The OPeNDAP 4 Data Server (aka Hyrax)



James Gallagher and Nathan Potter - Customizing and Extending Hyrax

Patrick West - Harnessing the power of the Server 4 Back-End Server modular framework

Jose H. Garcia - Numerical Grid Computations with the OPeNDAP Back End Server (BES)

Patrick West - OPeNDAP Server 4 Back-End Server Authentication and Authorization


Lunch 1215-1330

Session 3 (moderator: P. Cornillon)

1330-1530 Security



John Relph and Kenneth S. Casey - Software Development and Security at NOAA

John Caron - The THREDDS Data Server and OPeNDAP security

Tim Pugh - BMRCs OPeNDAP data service, how far can it reach?

Discussion


1530-1545 Break

Session 4 (moderator: J. Gallagher)

1545-1730 Semantics



Benno Blumenthal- Using an RDF framework to carry metadata for datasets

Rob Raskin - SWEET - an upper-level ontology for Earth and Space Sciences

(withdrawn) Luis Bermudez - MMI and CF

Peter Fox - OPeNDAP and semantics

Discussion


Dinner

No-host dinner, Cafe Gondolier

 

Thursday February 22, 2007

Session 5 (moderator: D. Holloway)

0900-1030 APIs and Clients



Roberto De Almeida - Data Access Protocol meets Python

Peter Cornillon - The Matlab OPeNDAP GUI Suite


1030-1100 Break



Poster: Denis Nadeau - Enhancing Access to NASA Satellite Data OGC Web Services using OPeNDAP.


Session 5 continued

1100-1215 APIs and Clients (continued)



John Chamberlain - OPeNDAP: User Versus Programmatic Access

Greg Janee - Data Discovery in a Distributed Environment and Darren Hardy's AGU presentation.


Lunch 1215-1315

Session 6 (moderator: P. Cornillon)

1315-1445 Data Providers



Roy Mendelssohn - OPeNDAP at ERD with suggestions for future development

Jim Potemra - IPRC data transport, LAS and EPIC server

Christopher Lynnes - OPeNDAP Developments at the Goddard Earth Sciences DISC

Thomas LOUBRIEU - OPeNDAP in European oceanography data management


Session 7 (moderator: N. Potter)

1445-1715 Server-Side Operations with 1530-1600 Break



Wenli Yang - The ROSES ACCESS OPeNDAP/OGC Gateway Project

Roland Schweitzer - Server-side OPeNDAP Analysis - A General Approach Utilizing Legacy Applications through TDS

Daniel L. Wang - Server-side Data Reduction and Analysis with Script Workflow Analysis for MultiProcessing (SWAMP)

Thomas LOUBRIEU - Dap4cor: A Dapper-like server for CORIOLIS in-situ data centre


Dinner

Dinner on your own

 

Friday February 23, 2007

Session 9 (moderator: P. Fox)

0900-0930 Server-side functions (continued)



James Gallagher - Server-side Functions for Geo-spatial Selection


0930-1200 Strategies and Directions



Discussion (with nominal time allocations) focusing on...

community forums for deciding on standards processes within (and beyond) the OPeNDAP community, who is involved, timeframe, prototypes, etc. (1hr)

security (20min)

semantics (profiles, vocabularies) (20min)

server-side (aggregation) (20min)

response types (e.g. get_capabilities) (20min)

metrics (10min)

others: gateways, Server4, TDS and other general direction issues


Lunch

Hyrax Tutorial (FL2/1001): 1300-1630

Dinner

No-host dinner, location TBA