# Presentation: 099884

## Collaboration Tools and Techniques for Large Model Datasets

**Collaboration Tools and Techniques for Large Model Datasets**

- Rich Signell,USGS
- Woods Hole, MA

## Motivation

- Today model outputs is typically 4D, and often many gigabytes.
- Collaborators don’t want it all.
- Each model uses different conventions, requiring special tools for each model.
- There is a better way...

## Approach

- Model providers create files that meet 4D geospatial data standards, where coordinate systems, data types are self-defined
- Each data provider distributes data via web services
- Common clients can be built that work with all models, accessing only necessary data over the web

## NetCDF

- _Machine independent_, _self-describing_, _binary format_ for multidimensional scientific data
- Efficient extraction of data subsets
- Interfaces: Fortran, C, C++, Java, Perl, Matlab, IDL, Python
- Free, supported by NSF at Unidata

## DODS/OpenDAPhttp://www.opendap.org

- Open Data Access Protocol for delivery of multidimensional scientific data via http
- DODS allows efficient slicing from data via the web, just as NetCDF works for local files.
- DODS serves not just NetCDF, but also Matlab, HDF (also GRIB, BUFR, etc...)

## Accessing DODS data

- Call DODS APIs (C++, Java, Fortran)
- Use any NetCDF client, relinked with DODS netCDF library
-  ncdump => dncdump
-  ncview => dncview
- Your Fortran, C, C++, Python, Perl, Java code...

## DODS & Matlab

- DODS GUI and command line tools
- Relinked mexnc.dll, which can enable  _all_ Matlab tools that read NetCDF!
- (e.g.) NetCDF/Matlab toolbox
- >> url=‘http://long_path/myfile.nc’
- >> nc=netcdf(url);
- >> lon=nc{‘lon’}(:);
- Google on:   “mexnc”

## DODS/OpenDAP 

- Serving DODS data requires almost no effort on the part of the data provider:
- Download DODS server binaries to the cgi-bin directory on the web server
- Put your NetCDF files on the web server
- Go have a coffee to celebrate !
-  	(Note: most people don’t know that getting a DODS server going is this easy!)

**Notes:**
DODS server binaries are available for: Linux RH7.3, RH8.0, RH FC1, Spark/Solaris, SGI/IRIX, Compaq Tru64, MAC OS/X

## DODS Success Story

- DODS at sea:  in limited-bandwidth situation, grabbed only 200 k OBC region instead of 18 Mb NetCDF file.
- 30 second download instead of 45 minutes!

## Need for Conventions

- One of the greatest things about NetCDF is that it places few demands on the data provider - they are free to specify whatever attributes they want, or none at all
- This is also one of the worst things, making it hard to develop flexible software
- Software for ROMS won’t work for POM, NCOM, HOPS, ECOM, etc (and vice versa)

## CF Conventions I

- Google: “CF” “ucar”

## CF Conventions II

## Generic Visualization Tools via Standards: IDV

**Generic Visualization Tools via Standards: IDV**

## IDV

- Freeware supported by the Unidata Program Center (new app, version 1.2)
- Java, utilizing Java3D and VisAD (VIS5D)
- Runs on Windows, Mac, Solaris (VIS5D is limitation)
- Reads NetCDF, DODS, ADDE, GeoTiff, Arc Shapefiles
- Slices, dices, animates

## IDV: Hurricane Katrina

**IDV: Hurricane Katrina**

**Notes:**
The same IDV tool used to visualize sediment in ROMS can be used to visualize any earth-science data that conforms to CF or several other standards.  For example, here we look at realtime radar data and nowcase meterological model results.   IDV doesn’t know anything specific about any of these models or data – only that they conform to standards! 

## Recommendations

- Make model output CF-compliant!
- Sign up for CF mailing list and demand support for unstructured grids
- Distribute your model output via DODS
- Allow “packing” of data for efficient internet  delivery (and disk utilization)
- Help support software development for CF-compliant data