# Presentation: 764279

## AIRS Data Decimation  

- for Climate Trending

- 30 March  2004
- Hartmut H. Aumann

## The excellent data quality demonstrated by AIRS during the first 18 months of data promises some potentially very interesting contributions to climate trending.

- Key to successful climate trending  using AIRS data is to make the data readily available to the climate research community.
- The AIRS spectrometer produces 1.3Mbits/sec  = 14 GBytes/day
- The AIRS PGE produces about 12 GBytes/day L2
-                                                  37 GBytes/day L1b
- If one day of L1b data is  requested from the DAAC over a 1MByte/sec internet connection, it would take 14 hours to transmit.
- The expected 7 years of AIRS data would require 80 TBytes storage.
- At an internal 10MByte/sec transfer rate it would take 92 days to read.
- Access difficulties  will hamper the effective and wide spread use of the AIRS data, no matter how good the data are.

## The approved AIRS project plan identifies three methods to  deal with

- this issue:
- Write out the data more efficiently than is done currently without
- 	loss of information (presentation by Evan Manning)
- Generate level 3 products which provide globally mapped means
- 	suitable for climate research (presentation by Steph Granger)
- Generate spatial, spectral and/or temporal level 1b data subsets
- 	suitable for climate research

## Generate spatial, spectral and/or temporal level 1b 

- at 10 MBytes/sec in one day =   1 TByte
- Currently there are at least four types of subsets being generated,
- which could be evaluated, augment and promoted to routine
- production status at the DAAC.
- We discuss the four and propose  augmentations for routine
- production at the DAAC

**Generate spatial, spectral and/or temporal level 1b **
**data subsets suitable for climate research.**

## NESDIS BUFR file

- NESDIS Radiance 0.5x2 degree gridded product (all channels,center fov)
- NESDIS Radiance 3x3 degree gridded product (entire “golfball”)
- NESDIS AIRS Level 2 Support file @ 0.5 x 2 degree gridded product
- UMBC clear ocean subset
- JPL calibration clear ocean subset

## The JPL calibration clear ocean subset was optimized to support the long term trending of the AIRS radiometric and spectral calibration.

- This file is generated every day. It saves for every
- “reasonably clear” AIRS footprint
- 	70 “key” AIRS channels (2 per module ++)
- 	the matching mean and stdev vis1, vis2 and vis3
- 	the matching AMSU 1,2,3,4,5,6 and 15
- 	the matchup with the RTGSST.
- Typical 100,000 points per day
- Average size (not bzip compressed) 18 Mbytes/day
- Clear biased data subset and climate application may conflict
- Factor 1800 less than the current 32 GBytes/day
- Seven year volume 0.05 TByte

- JPL calibration clear ocean subset

## The AIRS sst2616 tracks the RTG.SST

- at night with 0.64K bias and 5mK/year stability.

- 0.35K of the cold bias is due to the day night bias in the RTG.SST and the skin effect. The remaining 0.28K bias is due to residual cloud contamination

- rms=0.4K

- uncorrected
- night bias
- -0.62K

## UMBC clear ocean subset

- Every fairly “clear” ocean spectrum is saved for all 2378 channels
- in rtp format plus the ECMWF matchup T(p), q(p) and  TSurf
- Typically 	20,000 spectra per day
- 		500 Mbytes/day in 240 files
- Clear biased data subset and climate application may conflict
- Factor 60 reduction in data volume compared to current level 1b
- Seven year volume 1.3 TBytes

## NESDIS 0.5x2 degree gridded product

- All 2378 channels are saved for the first spectrum which
- falls into the 0.5 x 2 degree grid for ascending and descending
- orbits.
- Unbiased
- 720 x 90 gridpoints * 2 * 2378 points *  2 bytes/point =
- 616 MBytes in two files per day
- Factor 54 volume reduction compared to current L1b
- Seven year volume 1.6 TBytes
- All 2378 channels for first 3x3 golfball in 3x3degree box same.

## NESDIS AIRS BUFR file

- For every second AMSU footprint all 15 AMSU channels
- and 340 AIRS channels are saved (i.e. every 0.53 seconds)
- Each day  163,000 points * 370 samples * 2 bytes/sample
- = 120 MBytes/day
- Factor 270 decrease in data volume
- Seven year volume 0.3 TBytes
- Currently this file is not saved at NESDIS

## Enhanced  0.5x2 degree gridded product

- Save 400 channels  (with AMSU and vis) from the first
- 3x3 golfball which falls into the grid for ascending and
- descending orbits.
- unbiased
- 720 x 90 gridpoints * 2 * 9 * 400 points *  2 bytes/point =
- 933 MBytes in two files per day
- Factor 34 volume reduction compared to current L1b
- Seven year volume 2.4 TBytes

## For every day save for every “reasonably clear” land and ocean 

- For every day save for every “reasonably clear” land and ocean
- AIRS footprint
- 	all  AIRS channels
- 	the matching mean and stdev vis1, vis2 and vis3
- 	the matching AMSU 1-15
- 	the matchup with the surface analysis (RTGSST over ocean).
- Typical 150,000 points per day
- Average size (not bzip compressed) 800 Mbytes/day
- Clear biased data subset and climate application may conflict
- Factor 40 less than the current 32 GBytes/day
- Seven year volume 2.0 TByte

- Augmented “clear” subset

## For climate every eighth day may be adequte

## 8th day sampled 0.5x2 degree gridded product

- Save all 2378 channels  (plus  AMSU and vis) from the first
- 3x3 golfball which falls into the grid for ascending and
- descending orbits.
- Unbiased
- 720 x 90 gridpoints * 2 * 9 * 2400 points *  2 bytes/point =
- 5.6 GBytes in two files every eighth day
- Factor 34 volume reduction compared to current L1b
- Seven year volume 1.8 TBytes

## With the version 4.0 PGE we include the production of 

- With the version 4.0 PGE we include the production of
- at least five  subsets in support of climate (not counting L3)
- All channels all footprints compacted to 2 bytes.
- 	compressed to 1 byte. 2400 bytes per spectrum
- All channels all footprints 80 static principle components
- 	saved as 400 bytes per spectrum.
- The NESDIS  0.5 x 2 degree gridded product for every day
- The NESDIS 3x3 degree gridded product for first golf ball
- 	for all 2389 channels
- The all  channels “clear” land and ocean set every day
- The climate data subsets are reprocessed to generate a
- complete record starting in September 2002.

- Proposal/Plan