# Presentation: 657132

## Integrating the Scientific Community with a Measurement Based, Multi-Sensor Data Processing and Distribution System

**Bryan Franz**, Donna Thomas, Sean Bailey, Norman Kuring, Gene Feldman,  Chuck McClain, John Wilding
- NASA Ocean Biology Processing Group

- AGU Meeting, December 2005, San Francisco, CA

## The Ocean Biology Processing GroupA Component of NASA’s Missions-to-Measurements Initiative

- Heritage: SeaWiFS, SIMBIOS, SeaBASS, SeaDAS
- Designated NASA team responsible for the processing and distribution of ocean color measurements and SST from various spaceborne instruments.
- 		Ocean Color:   CZCS, OCTS, SeaWiFS, MODIS/Aqua
- 		SST:                MODIS/Aqua, MODIS/Terra
- Product Evaluation & Test Element (PEATE) for OC and SST, and Science Team  Lead for OC on NPP/VIIRS.
- Designated as the software development, processing, and distribution element for Sea Surface Salinity measurements from Aquarius.

## NASA’s Goal

- To make available the highest quality ocean data products to the broadest user community in the most timely and  efficient manner possible, and to provide sufficient documentation and support to facilitate the use of that data for global and regional studies.

## Approach

- highly integrated project structure with all elements co-located
- flexible, mission-independent data processing system that constantly upgrades procedures, technologies, and hardware
- comprehensive, centralized calibration and validation program closely coupled to the data processing and quality control system
- multi-mission software development
- common code for MODIS, SeaWiFS, OCTS, CZCS, future sensors
- common Level-2 & Level-3 formats
- a project philosophy designed to support and involve as large a community as possible
- providing opportunities for user input, evaluation, and feedback

## Data Processing and Storage

- database controlled, distributed processing and storage system
- many Linux-based, processing nodes (34 dual 3GHz Xeon)
- 200X processing for MODIS/Aqua ocean color
- 4000X processing through-put for SeaWiFS ocean color
- 9-month OCTS mission can be reprocessed over lunch
- all data online for direct access (100TB RAID-5 array)
- capacity for global mission testing prior to reprocessing, allowing opportunity for community participation in reprocessing decisions
- direct and near real-time user access to data archives

- processing node

- storage node

- database server

- gigabit ethernet

## Current Level-1 & Level-2 Data LatencyMinimizing Time Between Observation and Data Availability

- MODIS/Aqua Level-1 & Level-2 Ocean Color & SST
- Quicklook, average latency 4 hours, 52 minutes
- Refined, available within 2-8 days
- MODIS/Terra Level-1
- Quicklook, average latency 4 hours, 38 minutes
- SeaWiFS Level-1 & Level-2 Ocean Color
- Quicklook, 1-13 hours (2 downlinks per day)
- Refined, available within 4 days
- Latency minimization facilitates near real-time applications
- coastal monitoring (HAB detection), GHRSST, cruise support

## All Level-3 data available
bin & map files
daily, 8-day, monthly
seasonals & climatologies

Rolling archives of Level-1 & 2
quicklook
refined
30-days

All ancillary files
meterological data
ozone data
attitude & ephemerides

- Simplifying Data Access

- All Level-3 data available
- bin & map files
- daily, 8-day, monthly
- seasonals & climatologies
- Rolling archives of Level-1 & 2
- quicklook
- refined
- 30-days
- All ancillary files
- meterological data
- ozone data
- attitude & ephemerides

## Level-3 Multi-Sensor Browse & Distribution

## Level-1 & 2 Multi-Sensor Browse & Distribution

## Single File Browse and Download

## Multiple File Browse and Order

- users can order one file or the entire multi-mission data set

## Regional and Product Subsetting of Order

- users only download what they need

## specify region of interest
specify type of data (Level-1, Level-2, Quicklook or Refined, Day/Night)
files automatically staged for ftp transfer

- users can fully automated future data acquisition

- specify region of interest
- specify type of data (Level-1, Level-2, Quicklook or Refined, Day/Night)
- files automatically staged for ftp transfer

## Files Distributed to Users4.6 Million Since February 2004

- Files Distributed to Users4.6 Million Since February 2004

## Geographical Distribution of File Downloads

- Web Browser Downloads

- FTP Downloads

- European and Asian researchers prefer web access, while American and Southern Ocean folks prefer ftp.

## SeaDASproviding the user with the required toolsexpanding the standard product suite

- SeaDASproviding the user with the required toolsexpanding the standard product suite

- software to display and analyze all products produced by the OBPG
- software AND source code to process from Level-0 through Level-3
- allows users to do local reprocessing (reproduce standard products)
- provides for user-specific product suites

- Some Product Algorithms in SeaDAS
**water-leaving radiances**
**remote sensing reflectance**
**SST (thermal and short-wave IR)**
**chlorophyll (8 algorithms)**
**diffuse attenuation of sea water **
**IOP (GSM01, Carder, QAA):**
**particulate organic carbon**
**total suspended matter**
**calcite concentration**
**fluorescence line height**
**PAR, IPAR**
**aerosol products (AOT, Angstrom)**
**intermediate products (Lr, La, etc.)**

## Direct Broadcast Supportfacilitating real-time product generation

- 12 minute MODIS pass from University of Wisconsin DB Station, processed from Level-0 through Level-2 with SeaDAS

- Standard 5-minute MODIS granule processed from Level-0 to Level-2 by OBPG

- True Color

- Chlorophyll

- SST

## Near Real-Time Mapped Image Support supporting field campaigns for collection of validation data

- Near Real-Time Mapped Image Support supporting field campaigns for collection of validation data

-  select region
-  select products
-  select map characteristics
-  receive daily images
-  email or ftp

- current subscriptions for image support

## Near Real-Time Mapped Image Support

- True Color

- Chlorophyll-a

- Option to receive corresponding Level-1 and Level-2 HDF files, extracted to the region

## Slide 20

## Slide 21

## http://oceancolor.gsfc.nasa.gov/

## Notes

## The Ocean Color Web http://oceancolor.gsfc.nasa.gov was developed and is maintained by the Ocean Biology Processing Group (OBPG) at NASA's Goddard Space Flight Center as a means for disseminating information and data in support of NASA's Ocean Color program. The OBPG is responsible for the processing, validation, and distribution of Ocean Color and SST data from MODIS, and Ocean Color data from SeaWiFS, OCTS, and CZCS. The Ocean Color web is the public interface to a fully automated data system for acquisition, processing, analysis, and distribution of data from these spaceborne sensors. Key features include web and ftp-based data access, timely availability of data products, data subscription services, parameter and regional subsetting, image support for field campaigns, extensive documentation, and user support forums. This paper presents an overview of the valuable Ocean Color Web resources that allow the scientific community to browse, download, and analyze Ocean Color and SST data from multiple satellites. 

## Slide 25

## Slide 26

## Slide 27

## Slide 28

## Operational MODIS-Aqua Data Flow

- table

- L0
- Ingest
- process

- L1A-L1B
- (MOD_PR02)

- L1B-L2
- (MSl12)

- MODISA L2
- table

- Archive - Distrib
- Server

- L3BIN
- (l2bin)

- MODISA L3-bin
- table

- Archive - Distrib
- Server

- MODISA L3-map
- table

- Archive - Distrib
- Server

- Software process

- Hardware system

- Database table

- Ancillary input data

- Sensor
- CAL

- Sensor
- attribs

- Atm
- corr

- Browser
- CGI /
- httpd

- Ocean Color
- Web Server

- User
- Community

- MySQL
- DB

- Ozone

- L3MAP
- (smigen)

- MET

- MET, Ozone, and OISST data are dynamically selected for each L1A granule

- Product meta data are populated from production DB

- OISST

- Level-0
- Ingest

- MODISA L0
- table

- Archive - Distrib
- Server

- ATT/EPH
- Ingest
- process

- MODISA atteph
- table

- Archive - Distrib
- Server

- L0-L1A
- (MOD_PR01)

- Geo-Location
- (MOD_PR03)

- MODISA L1
- table

- Archive - Distrib
- Server

- Full-resolution day- and nighttime via SEN (~60 GB per day)

- NOAA Realtime
- System

- NASA EDOS
- System

## Slide 30

## Time-Series Utility

## MODIS Direct Broadcast Receiving Stations