# Presentation: 702135

## Onboard Science Data Compression: 

**Technology Development and Status at GSFC**
*	*
*	**      **Pen-Shu Yeh*
- NASA/Goddard Space Flight Center, 301.286.4477, pen-shu.yeh@gsfc.nasa.gov
- May 23, 2003

## CONTENT

***Lossless Compression*****:**** **
**Development History**
**Performance**
** ****Status**
** *****High Performance Compression*****:**
**Requirement for Space Applications**
**GSFC/NASA Technique**
**CCSDS WG Selection**
**Performance**
**Technology Status**
** *****Conclusion***

## LOSSLESS DATA COMPRESSION DEVELOPMENT HISTORY

***History***
**1988-1989: requirements established**
**Applicable to various instruments and missions**
**Adaptive to statistics and easily implementable with low memory, high speed**
**Easily interfaced with packet data system w/o performance penalty**
**1989-1990: **
**algorithm comparison (Rice, LZW, arithmetic, Huffman)**
**Rice algorithm selected, added enhancement at low entropy**
**mathematical proof established**
**1990-1993: ASIC development, S/W by-product**
**1994-1995: algorithm submitted to CCSDS as candidate**
**1997: algorithm became CCSDS recommendation, Blue Book and Green Book published. First flight-validation on COBRA/DOE**
**~present: project support**

## LOSSLESS DATA COMPRESSION Algorithm Architecture

**LOSSLESS DATA COMPRESSION ****Algorithm Architecture**

**Rice Algorithm Architecture**

## E_Rice Algorithm Theoretical Performance on Data with Laplacian Model After De-correlation

**Performance Curve for Each Option**

**Overall Performance Curve**

**E_Rice Algorithm Theoretical Performance on Data with Laplacian Model After De-correlation**

## E_Rice Algorithm Performance on Science DataArchive Applications

**Objective**:  Compare performance of CCSDS lossless (szip),  LZW (*compress, gzip*) and arithmetic coding(*az*) techniques on EOSDIS data sets, with two measurements: **Compression Ratio** (CR) and **speed**.
**Test Data Source**: MAS, TRMM, AVHRR, TOVS, ASTER, SeaWifs,  TOMS in a total **data volume** of 930 Mbytes.
**Data Format**: Level 1B, 2,3,4,  Grid, Swath, byte, int*2, real*4
**Results Summary: (on Sun Sparc20)**
- 				***szip******	******gzip   compress******	******az***
**   ****CR****	****	****              ****	****	**3.24	2.44	2.06	2.38
**  ****Time**(compress,Seconds)**	****               **353        8112      1973    10516
-   **Time**(decompress,Seconds)**	**394	1264        790      7341

## E_Rice Algorithm Performance on Sensor Data

**E_Rice Algorithm Performance on Sensor Data**

## E_Rice Algorithm Performance on Sensor Data

- Decompression Time Comparison for Real Sensor Data

**E_Rice Algorithm Performance on Sensor Data**

## NASA LOSSLESS DATA COMPRESSION TECNOLOGY

**USERS**

**Mission ****	****Launch****	****Lead Agency****	****Implementation**
- Mars Observer	09/92	NASA/JPL	SW
- SERTS-96	11/96	NASA/GSFC	HW
- (Sounding Rocket)
- Mars-96	11/96	RSA	SW
- COBRA	   /97	DOE	HW
- LEWIS/SSTI	08/97	NASA/HQ	HW
- CASSINI Cosmic Dust	10/97	NASA/JPL	SW upload after launch
- Analyzer (CDA)
- SERTS-97	11/97	NASA/GSFC	HW
- SWAS/SMEX-3	01/99	NASA/GSFC	SW
- EO-1	12/99	NASA/GSFC	HW
- KOMPSAT-1	   /99	KARI	HW
- IMAGE/MIDEX-01	02/00	NASA/JPL	SW
- THEMIS/Mars Odyssey	04/01	NASA/JPL	HW
- VCL/ESSP-01	    /01	NASA/GSFC	HW
- MAP/MIDEX-02	07/01	NASA/GSFC	SW
- SIRTF	12/01	NASA/JPL	?
- EOS CHEM-1/AURA	12/02	NASA/GSFC	HW
- ROSETTA	01/03	ESA	HW
- Space-Based Infrared Sys	Multiple	DOD	HW
- INTEGRAL SPI	 2002	ESA	SW in ADA
- HDF4/5	 2003/5	NASA/NCSA	SW release/ground archive
- MESSENGER(MLA)	 2004/5	NASA	SW
- GIFTS/EO-3	 200?	NASA/LaRC	HW
- PICARD	 2005	CNES	SW on DSP
- NPP	 200?	NOAA/NASA	HW
- GOES/ABI	 200?	NOAA/NASA	HW
- JWST	 200?	NASA	HW
- GPM	 200?	NASA/GSFC	SW

## LOSSLESS DATA COMPRESSION FOR SPACE APPLICATIONS

***Benefits for NASA Missions***
- Reduces bandwidth requirement, onboard storage or station contact time
- Reduces ground archive volume with savings in $$M

***Technology Features***
- Algorithm adopted as CCSDS recommendation
- Works well with large data quantization range and packet data system without penalty on performance
- ASIC offers real-time operation > 40 Msamples/sec in space environment
- Compresses faster and better than commercial techniques

***Users***
- COBRA/DOE, SWAS/SMEX, MAP/MIDEX, EOS-CHEM, KOMPSAT, IMAGE, CASSINI, INTEGRAL, SERTS, SBIRS/DOD, MARS ODYSSEY, NPP, EO-3, MLA/MESSENGER, ABI/GOES, EOSDIS(HDF), GPM, NPOES, JWST.
***Information***
- http://www.ccsds.org/ccsds/ccsds_document_access.html
- http://www.cambr.uidaho.edu

- Lossless Compression Board

## CONTENT

***Lossless Compression*****:**** **
**Development History & Requirement**
**Performance**
** ****Status Summary**
** *****High Performance Compression*****:**
**Requirement for Space Applications**
**GSFC/NASA Technique**
**CCSDS WG Selection**
**Performance**
**Technology Status**
** *****Conclusion***

## HIGH PERFORMANCE DATA COMPRESSIONrequirement for space applications

- Requirements established by Consultative Committee for Space Data Systems (CCSDS) Compression Working Group in 1998:
-  Offer “Royalty free” algorithm
- Process both non-frame based (push broom) and frame based input source data.
-   Offer adjustable data rate.
-   Work with large source quantization ranges up to 16 bit-per-pixel
-  Offer real-time processing >= 20 Msamples/sec,
- 			      at <= 1 watt/Msamples/sec.
-  The power consumption includes all buffering and support electronics.
-   Require minimum ground interaction during operation.
-   Allow packetization for error containment.
-  Allow progressive transmission/Decoding (optional)

## 	GSFC/NASA TECHNIQUE

- De-correlator:  Discrete Cosine Transform, Lapped Transform, Wavelet Transform

## BIT PLANE ENCODER

**BIT PLANE ENCODER**

**Transformed Coefficients in Bit Planes:**

- BP: 5

- BP: 4

- BP: 3

- BP: 2

- BP: 1

- Block 1

- Block 4

- Coded  BP 5

- Coded  BP 4

- Coded  BP 3

- Coded  BP 2

- Coded  BP 1

**Output Coded Bit String Direction**

**Multiple Bit Planes**

## BIT PLANE ENCODER

**BIT PLANE ENCODER**

- Scanning on each bit plane
- Direction: F0 => F1 => F2
- Coding: 3 main levels/block
- Output: embedded bit string
-  => progressive decoding

- * No look up table
- * > 20 Msamples/sec
- * Radiation Tolerant implementation
- * Progressive decoding for quick-look

**Blocks on Bit Plane**

## CCSDS WG SELECTION

- CCSDS Image Data Compression Working Group (Sub-Panel 1C) has been trying to select an algorithm as recommendation for space implementation.
- After nearly 5 years of work on evaluating performance, implementation issues, impact on science it finally made a decision (April, 03) to adopt:
-  Discrete Wavelet Transform: 9/7 floating and 9/7 (or 5/3) integer
-  Bit Plane Encoder
- With integer wavelet, the scheme will provide from high compression ratio, to visually lossless and to mathematically lossless performance.
- Agency review on Red Book will commence after October 03, with Blue Book/Green Book and S/W projected July 2004.

## TEST IMAGES

**Mars**

**SPOT**

**ICE(AVHR)**

**INDIA(AVHR)**

**OCEAN(AVHR)**

**FOREST(AVHR)**

## TEST IMAGES

**SOLAR**

**SUNSPOT**

**SAR**

**WFPC**

**FOC**

## PERFORMANCE

**Visual Evaluation Performed at 1.0 bpp**

**Original**

**2DMLT**

**JPEG**

**JPEG2000**

## PERFORMANCE

**Visual Evaluation Performed at 1.0 bpp**

**Original**

**CCSDS**

**JPEG**

**JPEG2000**

## PERFORMANCE

**Visual Evaluation Performed at 0.5 bpp**

**Original**

**CCSDS**

**JPEG**

**JPEG2000**

## PERFORMANCE

**PERFORMANCE**

**Original**

**CCSDS**

**JPEG**

**JPEG2000**

## PERFORMANCE

**PERFORMANCE**

**Original**

**CCSDS**

**JPEG**

**JPEG2000**

## TECHNOLOGY STATUS

**Earlier version on Lewis for Hyper-Spectral-Imager (data cube compression, ‘97) **
**DCT/EDCT RT chip fabricated (**upto 16-bit input**), tested at 35 Msamples/sec**
** ****2D DWT chip planned 2004-5 design/fabrication**
** ****Bit Plane Encoder chip under design ==> 2004 fabrication**
** ****System power estimated at 0.36 watt/Mpixel/sec**
** ****Software simulation performed on various types of images**
** ****Performance impact on science product under study**
**-- **sea surface temperature --on NOAA-14 data,
- 		mean error < 0.01k from 0.25 - 2 bpp, 5/00
**-- **cloud detection using MODIS algorithm
- -- Retrieval for sounder (HES) data (GOES-R)

## CONCLUSION

** ****A lossless compression technique has been developed for space applications, and inserted into over 15 missions with $$M savings to NASA. The technique is applicable to 1D, 2D, 3D (and *****m*****D) data.**
**An image data compression technique for space pushbroom  and frame applications is being developed. **
**The technique is implementable in current rad-tolerant, SEU/SEL immune electronics technology to achieve >= 20 Mpixels/sec and <= 1 watt/Mpixels/sec.**
**The technique produces embedded bit string, allows accurate rate control, requires no table upload.**
**Performance on quantitative measurement and visual evaluation is comparable to JPEG2000.**
**Further study will be performed on possible impact on science product.**
**The DWT+BPE algorithm is selected by CCSDS as recommendation for image data compression. With integer DWT, it achieves lossless as well.**