Unidata

20 September 2004
Mohan Ramamurthy
Unidata Program Center
UCAR Office of Programs
Boulder, COMission
 Mission Statement:
Provide data, tools, and community leadership forenhanced Earth-system education and research.
At the Unidata Program Center, we• Facilitate [Real-time] Data Access• Provide Tools• Support Faculty and Staff• Build and Advocate for a Community

Unidata Program Center Conception: Circa 1983
 Current Funding Sources:
Primarily NSF/ATM, withadditional funding from
NSF/EHR and NASA.
 Governance:• Community governed• Policy Committee (Appointed byUCAR President)• Users Committee• Technical Committee(s) – asneeded

UNIDATAThe University Data System
John Dutton et alInternet Data Distribution
 About 150+ sites are

participating in Unidata
Internet Data Distribution(IDD) system

ModelSourceApproximately 2 GB ofRadar
data injected/hour fromSource
distributed sources;
Unidata IDD/LDM uses
more of the Internet2
than any other advancedapplication;
 Approx. 15 Terabytes
of data transmitted eachweek (~4% of I2 traffic).

LDM

LDM

Satellite
LDM
LDM

LDM
LDM

SourceInternet

LDM

LDM

LDM

Unidata and Internet 2
Volume of Data Moved by LDM each week via Internet 2 for thepast year

The LDM is now
ranked #3(behind HTTPand NNTP) inInternet 2usage.

16Volume (Terabytes)

14121086It recently
surpassed FTP.

4201357

9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51

Week (most recent period to the right)

Technology Portfolio
1) McIDAS: A client/server analysis and display packagethat emphasizes image processing of data fromsatellite-borne sensors;
2) GEMPAK: An analysis, display, and productgeneration package for meteorological data;
3) Integrated Data Viewer: Java-based, platformindependent data analysis and 3D visualization tools;
4) NetCDF: A software interface for platformindependent access to self describing datasets;
5) Local Data Manager: Software for capturing,
disseminating, and organizing data in near-real time;
It is the heart of the Internet Data Distribution (IDD)
system;6) THREDDS: A project to facilitate remote access tothematic, distributed, interdisciplinary data servers;

Real-Time Monitoring
Unidata’s reach is
now global.
There are sites inSouth America,
Europe, and Asia
that receive realtime data via theIDD.

Real-time Statistics

Hong Kong Univ. - HDS

Univ. Washington - ALL

Ohio State Univ. - FNEXRAD

Univ. Wisconsin - CONDUITTechnology Transfer: Operational
LDM Use in the NWS

The U. S. National Weather Service is now using the Unidata LDM
technology operationally to distribute NEXRAD Level II data.
The Korean Meteorological Administration has started using the LDM for some of
their internal data distribution to/from nearly 40 weather service offices.

SuomiNet GPS Met and Geodetic
Sites

Unidata’s LDM is used for real-time data transport

Unidata 2008: Shaping the Futureof Data Use in the Geosciences
Unidata user community is interdisciplinary - 2/3rdof sites have users outside atmospheric sciences.
We are moving from an era of data provision towards one inwhich data- and related web-services are important;Multidisciplinary integration and synthesis are emphasized.

Ongoing Endeavors
 Community and Support Services• Endeavor 1. Responding to a broader andmore diverse community.
• Endeavor 2. Comprehensive support services

 Data Services, Systems and Tools• Endeavor 3. Real-time, self-managing dataflows
• Endeavor 4. Software to analyze andvisualize geoscience data
• Endeavor 5. Distributed, organizedcollections of digital material• Endeavor 6. Improved data accessinfrastructure

Integrated Data Viewer•IDV testing in the IHOP
field project•S-POL 3D radarreflectivity from NCAR
•Albedo (color-shaded)•Aircraft Track•Different sources,
protocols, resolutions
and time-scales•IDV will be used in RICOExamples of Remote Visualizations

Thunderstorm simulation

Sea-level Pressure andUpper-level Jet
S-POL Radar Cross-sectionof a thunderstorm
Upper-mantle convection

NO2 concentration

Hurricane Ivan
SuomiNet Precipitable Water

Thematic Real-time EnvironmentalDistributed Data Servers (THREDDS)
Discovery

OpenDAP, ADDE,&FTP protocols

Hydrology
Data, e.g.
IDD

IDD

DLESE

• To make it possible toDigital Library for
Earth-System Education

publish, locate, analyze,
visualize, and integrate avariety of environmental data

DLinterchange
protocolGeophysicalData, e.g.

IDD

IDD

PeopleSatellite
Satellite
Satellite
Satellite
Images,e.g.
e.g.
Images,Images,e.g.
Imagery...

Discovery andPublication Tools

IDD
Discovery andPublication Services• Combines IDD “push” with• About 25 data providers areDocuments

THREDDS
Middlewareog olstal ToCa tion
raneGeseveral forms of “pull” and DL
discovery
partners in THREDDS

Analysis andVisualization Tools

Data Services

DataSe Catarvilce ogsUser applications:e.g., McIDAS, IDV,LAS, IDL, MatLab...

Data• Connecting People withDocuments and Data

NetCDF/HDF5 Merger

netCDF-3Interface

netCDF-4 Library
HDF5 Library
Access to netCDF-3, netCDF-4, and HDF5 data
created through netCDF-4 interface

17

LEAD: A Large Grid Computing
Project Linked Environments for
Atmospheric Discovery• Identify, Access,
Assimilate, Predict,Manage, Mine, andVisualize a broad array of
meteorological data andmodel output,
independent of format
and physical location• A range of Grid and Web
Services will be
developed for dynamic,
on-demand, end-to-end
weather prediction• Institutions: U.
Oklahoma, Unidata, U.
Alabama, U. Illinois, U.
Indiana, Millersville U.,
Howard U. and Colorado
State U.

LEAD Prototype 4IDD
DataStreamGWSTBs

Decoders

ADAS

IDV,NCL

ADAS
Output

WRF
Model

WRF
OutputADAM

Employ components of WRF prediction as a seriesof linked web services in a Grid Environment.