# Presentation: 421489

## AIRS Level-1B Data Format

**Evan Manning**
**Jet Propulsion Laboratory**
**California Institute of Technology**

## AIRS Level-1B Format

**Problem:  The large size of AIRS Level-1B IR products makes it difficult for users to order and store as much data as they would like.**
- 121 MB per granule
- 29 GB per day.
- 11 TB per year.
**Level-1B V/NIR and Level-2 Cloud-Cleared Radiances are each ~10% of this.**
**Main reason (95% of Level-1B IR) is radiance field -- one 32-bit floating-point number per channel per FOV.**

## AIRS Level-1B Format

**Investigated solutions (cumulative):**
- 1. Scale radiances from 32-bit floating-point to 16-bit fixed-point
- 2. Rearrange radiances from 135 x 90 spectra to 2378 images to improve compressibility
- 3. Compress: HDF internal or external

## AIRS Level-1B Format

**1. Scale radiances from 32-bit floating-point to 16-bit fixed-point**
**	****Pro:**
- Faster to ftp data.
- Less disk/tape space to store.
**	****Con:**
- Harder to use – users are responsible for rescaling.
- Some "data" lost:
- Extreme values that do not fit in range
- Low "noise" bits
- Change from current practice will inconvenience current users

## AIRS Level-1B Format

**2. Rearrange Radiances from 135 x 90 spectra to 2378 images.**
**	****Pro:**
- Improves compressibility:
- Faster to ftp data.
- Less disk/tape space to store.
**	****Con:**
- Harder to use – violates HDF-EOS swath format conventions, may interfere with standard subsetting tools.
- Change from current practice will inconvenience current users
** ****Tradeoff:**
- Adds no value unless compression is also being used.

## AIRS Level-1B Format

**3. Compress**
**	****Pro****:**
- Faster to ftp data.
- Less disk/tape space to store.
**	****Con:**
- Slower access, especially when all data must be decompressed to access a subset.
-  Harder to use – see next slide.

## AIRS Level-1B Format

**3. Compress (Cont)**

| 
Method | Size Reduction (Percent) | Size Reduction (Factor) |
| --- | --- | --- |
| Scaling | 47 | 1.89 |
| Scaling + Internal Compression | 63 | 2.70 |
| Scaling + External Compression | 71 | 3.45 |
| Scaling + Reorder + Internal Compression | 65 | 2.86 |
| Scaling + Reorder + External Compression | 72 | 3.57 |

## AIRS Level-1B Format

**Recommendations:**
**Implement scaling of AIRS Level-1B IR radiances with factor of 1⁄2 to 1⁄4 * NeN****Ideal**** determined by further experiments.**
- Investigate using same factors for AIRS Level-2 cloud-cleared radiances.
- Investigate using similar scaling for AIRS Level-1B Vis/NIR radiances.
- Investigate dynamic scaling factors
**Do not reorder radiances or implement compression in any AIRS product because the additional size savings does not justify the additional complexity.**
**Encourage users (and GSFC DAAC) to implement external compression or internal compression (using HDF tool hrepack) where this fits their usage profile.**

## Backup Material

**The long form...**

## AIRS Level-1B Format

**Problem:  The large size of AIRS Level-1B IR products makes it difficult for users to order and store as much data as they would like.**
- 121 MB per granule
- 29 GB per day.
- 11 TB per year.
**Level-1B V/NIR and Level-2 Cloud-Cleared Radiances are each ~10% of this.**
**Main reason (95% of Level-1B IR) is radiance field -- one 32-bit floating-point number per channel per FOV.**

## AIRS Level-1B Format

**Investigated solutions (cumulative):**
- 1. Scale radiances from 32-bit floating-point to 16-bit fixed-point
- 2. Rearrange radiances from 135 x 90 spectra to 2378 images to improve compressibility
- 3. Compress: HDF internal or external

## AIRS Level-1B Format

**1. Scale radiances from 32-bit floating-point to 16-bit fixed-point**
**	****Pro:**
- Faster to ftp data.
- Less disk/tape space to store.
**	****Con:**
- Harder to use – users are responsible for rescaling.
- Some "data" lost:
- Extreme values that do not fit in range
- Low "noise" bits
- Change from current practice will inconvenience current users

## AIRS Level-1B Format

**Scale radiances (Details)**
** ****Tradeoffs:**
- Keeping more low bits reduces “noise” loss but decreases compressibility.
- Dynamic scaling factors increase range and compressibility but are more difficult for users.

## AIRS Level-1B Format

**Proposed scaling: Constant per-channel scale factor of 1/2 * NeN****Ideal**** for each detector, where  NeN****Ideal**** is determined by a fit to the best detectors in each module.**

## AIRS Level-1B Format

**1. Scale radiances from 32-bit floating-point to 16-bit fixed-point (Details)**
**Experiment: Sept 6, 2002 data was processed through retrieval 3 ways:**
- Current Baseline
- After scaling by NeNIdeal/2
- After scaling by NeNIdeal/4
**Results:**
- When scaling with NeNIdeal/4 yield, Tsurf bias, and Tsurf RMS were all indistinguishable from baseline.
- When scaling with NeNIdeal/2 bias
- Tsurf bias and RMS were still indistinguishable from baseline
- Yield fell by 0.13%, but this is within experimental uncertainty
**Recommendation:**
- Scale Level-1B IR radiance product using a scaling factor of 1⁄2 to 1⁄4  * NeNIdeal determined by results of further studies

## AIRS Level-1B Format

**2. Rearrange Radiances from 135 x 90 spectra to 2378 images.**
**	****Pro:**
- Improves compressibility:
- Faster to ftp data.
- Less disk/tape space to store.
**	****Con:**
- Harder to use – violates HDF-EOS swath format conventions, may interfere with standard subsetting tools.
- Change from current practice will inconvenience current users
** ****Tradeoff:**
- Adds no value unless compression is also being used.

## AIRS Level-1B Format

**3. Compress**
**	****Pro****:**
- Faster to ftp data.
- Less disk/tape space to store.
**	****Con:**
- Slower access, especially when all data must be decompressed to access a subset.
-  Harder to use – see next slide.

## AIRS Level-1B Format

**3. Compress (Details)**
** ****Tradeoffs:**
- Added time:
- ~30 sec to compress a 6-minute granule.
- ~15 sec to decompress a 6-minute granule.
- External compression puts responsibility on users and requires decompression for even trivial data access.
- The most aggressive compression utilities (bzip2) are not universally supported.
- HDF-EOS supports internal compression which is transparent to users, but not very aggressive.
- New HDF-EOS 5 supports more aggressive internal compression, but would be an even greater effort to implement and even harder for existing users to adjust to.
- HDF “hrepack” tool may allow users to apply or reverse HDF internal compression to product files like external compression.

## AIRS Level-1B Format

**3. Compress (Details - Cont)**
** ****Experiment:**
- Scale one granule by NeNIdeal/2.
- With and without reordering.
- Best HDF internal compression is skipping Huffman.
- Best tested external compression is “bzip2 –9”.

## AIRS Level-1B Format

**3. Compress (Cont)**

| 
Method | Size Reduction (Percent) | Size Reduction (Factor) |
| --- | --- | --- |
| Scaling | 47 | 1.89 |
| Scaling + Internal Compression | 63 | 2.70 |
| Scaling + External Compression | 71 | 3.45 |
| Scaling + Reorder + Internal Compression | 65 | 2.86 |
| Scaling + Reorder + External Compression | 72 | 3.57 |

## AIRS Level-1B Format

**3. Compress (Details - Cont)**
**Notes on Experiment:**
- Greater compression would be realized with dynamic scaling.
- Lower compression would be realized with smaller scaling bin size.
- Sample size for this experiment is small (1 granule), and exact ratios vary with scene characteristics, so all numbers are approximate.
- Overall compression ratios are limited by uncompressed 5% “overhead” of “metadata” and other fields (geolocation, QA, etc.).

## AIRS Level-1B Format

**Recommendations:**
**Implement scaling of AIRS Level-1B IR radiances with factor of 1⁄2 to 1⁄4 * NeN****Ideal**** determined by further experiments.**
- Investigate using same factors for AIRS Level-2 cloud-cleared radiances.
- Investigate using similar scaling for AIRS Level-1B Vis/NIR radiances.
**Do not reorder radiances or implement compression in any AIRS product because the additional size savings does not justify the additional complexity.**
**Encourage users to implement external compression or internal compression (using HDF tool hrepack) where this fits their usage profile.**

## Deep Backup Material

**Ideal NeDTs for each module**

## AIRS Level-1B Format

**Ideal (fit) NeDT for Modules M1 & M2**

## AIRS Level-1B Format

**Ideal (fit) NeDT for Modules M3 & M4**

## AIRS Level-1B Format

**Ideal (fit) NeDT for Modules M5 – M8**

## AIRS Level-1B Format

**Ideal (fit) NeDT for Modules M9 – M12**