# Presentation: 763285

## ORA requirements for CLASS

- M. Goldberg
- 12/11/03

## Requirements

- Standards for data formatting, metadata, documentation, and how software is archived.
- Reprocessing capability
- Data Access – for experimental reprocessing at ORA (connected to CLASS via Dark Fiber).
- Or better yet – parallel reprocessing systems within CLASS.
- Parallel processing systems can be used by Data Centers and ORA.

## Requirements

- Archive of thinned datasets for  large datasets like AIRS, CrIS, VIIRS, MODIS, GOES-R
- Thinned datasets will be used for reprocessing pathfinders as well as providing easily accessible data to the external community.

- Thinned AIRS gridded dataset,
- 1/30 size of original

## Requirements

- Access to past data for EDR validation.
- Access to past data for resolving anomalies.
- Validation datasets (in situ, aircraft campaigns, etc) need to be stored and easily accessed.
- All ancillary data for reprocessing needs to be archived.

## Importance of Reprocessing

- To generate Climate Data Records from historical data with algorithm based on today’s knowledge
- To correct for problems in a time series
- To validate new EDR algorithms to ensure that the algorithm is robust and accurate for all seasons.

## Current Reprocessing Activities

- Ozone from SBUV/2
- Microwave Hydrological products
- Precipitation - GPCP
- AMSU temperatures
- AVHRR polar products
- New pilot projects

## Scientific Data Stewardship Pilot Studies – AVHRR and AIRS

- M. Goldberg
- 12/11/03

- Observing System Performance Monitoring
- Detect problems early
- Production of near real-time CDRs
- Monitor current state of climate system and short -term variations
- Reprocessing of CDRs for long-term records
- Consistent, seamless, high quality time series with minimized bias
- Climate research and applications
- Joint activities with external community
- Archive and distribution
- Includes output of above activities, metadata, and timely distribution
- Above are guided by climate science teams – experts in instrument characterization, algorithms, validation, data management, applications, and observing system performance monitoring

- An end-to-end process for scientific stewardship of  CDRs
-       will likely consist of 5 functional areas

## Importance of Pilot Studies

- Provides a pathfinder for implementation of  Scientific Data Stewardship.
- Provides experiences, i.e. lessons learned.
- Provides team members a new way of thinking.

## Two Pilot Studies at ORA

- AVHRR Reprocessing
- AIRS Reprocessing

## Why Reprocess AVHRR?

- - Most versatile satellite data set, many products can be derived
- 	from AVHRR
-         Cloud cover and properties
-         Aerosol optical depth
-         Radiation budget
-         Snow cover
-         Vegetation index
-         SST
-         Fires
-  AVHRR data record is global and mostly complete
-  The AVHRR data record will be carried forward in time by VIIRS,    	so potential for  very long imager data record
- Revamp the processing system – so that it can be used for real-time 	EDR/CDR processing

## Hardware

- 16 Intel Xeon 2.8 GHz Processors
- 14 TB of Disk Space
- 32 GB of Memory
- Rack Mounted, connected by Gigabit Ethernet

##           AVHRR Operational              Products Flowchart

- Aerosol Optical
- Depth & SST

- SNOW

- NDVI

- Hazards

- Radiation
- Budget

- Level 1B Calibrated
- and Navigated data

- L1B
- Imagery

## Reprocessing Ideas

- Reprocess 20 years of AVHRR data
- Fix the navigation where required
- Redo the calibration
- Ingest the calibrated and navigated AVHRR data into one software package that produces ALL the products

##    Proposed AVHRR             Products Flowchart

**Cloud  Mask**

- Level 1B  with improved
- Calibrated and Navigated
- Orbital or Granule data

- Level0 or 1b Data

- Calibration
-       and
- Navigation

- Mapped L1B
- Cal/Nav
- Daily Data

-   Snow         Radiation Budget             Hazards
-   NDVI        Aerosol Optical Depth     SST
-   Cloud Products

- Ancillary
- Data

## Why One Package

- Results of scientific code changes are easy to track
- Consistent science used by all products
- Easier to maintain
- Reduce time lag between research code to operational code

## From the Past to the Future

- Package built to reprocess AVHRR data
- Package will be modified to read in MODIS data
- AVHRR algorithms can be modified to process MODIS data – similar algorithms for both instruments
- Directly compare AVHRR and MODIS products

## NPOESS Preparation

- AVHRR package modification to ingest and process MODIS data is to prepare the AVHRR algorithms for VIIRS data
- AVHRR package will be ready to process simulated VIIRS data before launch of NPP

## AVHRR Summary

- Reprocess 20 years of AVHRR
- Create one software package to produce the AVHRR products
- Ingest and process MODIS data to prepare for NPP and NPOESS
- Embrace the principles of Scientific Stewardship

## Reprocessing AIRS

- Thinned gridded radiances
- Provides pathfinder for reprocessing of full data volume
- Start after algorithms have matured.
- AIRS science team oversight
- Also studying optimal data compression for archiving

- Thinned data is 30x smaller
-  than full data.

## AIRS/MODIS Hardware

- 7 TB of Disk Space for AIRS/MODIS development machine
- Link the raids from server, MODIS operational, AIRS operational, and development machines into one file system accessible to all machines
- Development machine is now the backup operational machine for AIRS
- Development machine can now be used as a reprocessing machine

## AIRS/MODISHardware Configuration

- Server
- 2 CPUs
- 5 TB

- MODIS OPS
- 64 CPUs
- 2.3 TB

- AIRS OPS
- 32 CPUs
- 2.3 TB

- Development
- 32 CPUs
- 650 GB

## AIRS/MODIS NewHardware Configuration

- Server
- 2 CPUs

- MODIS OPS
- 64 CPUs

- AIRS OPS
- 32 CPUs

- Development
- 32 CPUs

- 16.5 TB of Disk Space

## AIRS Summary

- Reprocessing capability based on thinned AIRS data.
- Thinned radiance files are stored locally.
- Should be stored on CLASS
- Start reprocessing of AIRS – Oct 04.
- We also produce full resolution compressed radiances – should be CLASS archived and accessed.  (10x reduction – 3 GB per day)
- Our data compression studies  -- pathfinder for IASI and CrIS

## Backup

## Current AVHRR Operational Products

- Snow Map – Not fully automated
- NDVI
- OLR
- SST
- Aerosol Optical Depth
- Hazards --  Not fully automated
- Cloud Products (Not in Operations yet)

## Current Operations

- The products are generated by separate software systems
- Different cloud masks are used by each product