# Presentation: 170360

## The use of principal components in processing AIRS data

- Mitchell D. Goldberg
- NOAA/NESDIS/Office of Research and Applications
- 01/25/02

## Applications of Eigenvectors to AIRS Data

- NESDIS will provide AIRS data in near-real time to NWP centers.
- Data size is very large compared with current sounders  (1 orbit ~ 2GB vs. 8 MB)
- Information is not independent.  Eigenvectors provide an effective way to extract independent pieces of information.
- Eigenvector expansion coefficients (principal component scores) can be provided instead of the individual channels.
- Individual channels can be reconstructed with minimal signal loss.

**Notes:**
I am going to talk about how we plan to use eigenvectors for processing AIRS data. The main problem is that the data size is large compared with current sounders.   Transmitting the data to the ground is not a problem, because the data rate is only 1.5 mbits/s.  The problem is distributing the data to users and also NWP users generally archive the data they assimilate, so archiving 2 gb per orbit , 14 orbits per day would overwhelm there archive system.

The goods news is the information from AIRS 2000 plus channels is not independent.  In fact most of the channels are highly correlated with at least 100 other channels.

## 2378

- 8461

- 1305

- AIRS

- IASI

- CrIS

## AIRS/AMSU/HSB Simulations

- Includes surface topography and variable surface pressure
- Daytime and nighttime conditions
- T(p),q(p),o3(p) from surface to .005 mb.
- cloud liquid water profiles
- multiple level cloudy  conditions with spectrally varying cloud emissivity and reflectivity, consistent with atmospheric conditions (clouds from global model,but cloud amounts are randomized)
- variable surface skin temperature, surface emissivity and surface reflectivity
- variable land coverage with coastlines, lakes, etc.
- variable view  and solar zenith angles

## Example of simulated AIRS window channels:  LW,  SW

## Simulated AMSU

- Real AMSU

## Principal Component Analysis

- Principal component analysis (PCA) is often used to reduce data vectors with many components to a different set of data vectors with much fewer components that still retains most of the variability and information of the original data
**R**   =    *r*1.**i**1 + *r*2.**i**2 + *r*3.**i**3 + .... +  *r*n.**i**n
-         where **i**1  = (1,0,0,0,0....0n) ; **i**2  = (0,1,0,0,0....0n)
- Data are rotated onto a new set of axes, such that the first few axes have the most explained variance.
**R**   =    *p*1.**E**1 + *p*2.**E**2 + *p*3.**E**3 + .... +  *p*n.**E**n
-          where  **E  **are eigenvectors and  *p*1  =  **R** **E**1
- So instead of **R** vectors of length n,  we can have a truncated **P** vectors of length m,  where m << n

## Eigenvectors are used for

- Data compression
- Quality control
- Retrievals of  geophysical parameters
- Atmospheric temperature, moisture and ozone
- Surface temperature and emissivity

## Topics

- Generation of eigenvectors
- Applications of eigenvectors in AIRS data processing.
- Monitoring of representativeness of eigenvectors

## Generating AIRS eigenvectors

- Each AIRS data vector has 2103 radiance values.
- The radiances are normalized by expected instrumental noise (signal to noise)
- Compute the covariance matrix S
- Compute the eigenvectors E and eigenvalues  
-                             S = E ET
-  E = matrix of orthonormal eigenvectors (2103x2103)
-       = vector of eigenvalues (explained variance)

##   Applying AIRS eigenvectors 

- On independent data – compute principal component scores.
-     P    =    ET  R    ;   elements of R  = (ri- ri ) /ni
- Invert equation and compute reconstructed  	radiances  R*.
-     R*    =    E P
- Reconstructed radiances are used for quality control.
-  Reconstruction score  =  [ 1/N (R*i - Ri)2 ]1/2
-  				                                        i = 1 ....N channels

## AIRS applications

- AIRS is currently being simulated in near-real time.
- Each day a spatial subset of AIRS data is produced (200 mbytes vs 30 GB full data)
- Eigenvectors are generated daily.
- A static set of eigenvectors is used, but the ensemble is occasionally updated with new structures.
- When the ensemble is updated a new set of eigenvectors is also updated.

## Locations used in generating eigenvectors

**Notes:**
We have very good global representation

## Slide 14

**Notes:**
However the cumulative explained variance is an inadequate metric in determining the number eigenvectors to use because it does not adequately 

## Square root of the eigenvalues

**  ****2  1670.40**
**  ****3   945.52**
**  ****4   496.01**
**  ****5   284.01**
**  ****6   266.30**
**  ****7   156.95**
**  ****8   139.67**
**  ****9    88.27**
** ****10    72.83**
** ****11    60.03**
** ****12    53.42**
** ****13    45.01**
** ****14    39.72**
** ****15    34.54**
** ****16    26.57**
** ****17    22.62**
** ****18    17.60**

-  **19    14.68**
** ****20    13.49**
** ****21    12.28**
** ****22    11.32**
** ****23    10.70**
** ****24     9.08**
** ****25     8.24**
** ****26     7.85**
** ****27     6.77**
** ****28     5.98**
** ****29     5.83**
** ****30     5.39**
** ****31     5.34**
** ****32     4.98**
** ****33     4.34**
** ****34     4.09**
** ****35     3.62**
** ****36     3.48**

-  **37     3.38**
** ****38     3.11**
** ****39     2.82**
** ****40     2.53**
** ****41     2.41**
** ****42     2.39**
** ****43     2.34**
** ****44     2.24**
** ****45     2.03**
** ****46     1.86**
** ****47     1.78**
** ****48     1.71**
** ****49     1.65**
** ****50     1.61**
** ****51     1.54**
** ****52     1.52**
** ****53     1.35**
** ****54     1.34**

-  **55     1.25**
** ****56     1.19**
** ****57     1.16**
** ****58     1.15**
** ****59     1.09**
** ****60     1.05**
** ****61     1.02**
** ****62     0.98**
** ****63     0.90**
** ****64     0.86**
** ****65     0.81**
** ****66     0.80**
** ****67     0.78**
** ****68     0.77**
** ****69     0.73**
** ****70     0.72**
** ****71     0.70**
** ****72     0.66**

**Notes:**
This slide shows the square root of the eigenvalues which is the standard deviation of the principal component scores.  Each principal component score is some linear combination of all channels.  Since the eigenvectors are generated from radiance divided by noise, the square root of the eigenvalue can be thought of as the signal to noise of that principal component score.  So the first pcs has a signal to noise of 7500. 

## Slide 16

##  				                                        i = 1 ....N channels

-  				                                        i = 1 ....N channels

##  				                                        i = 1 ....N channels

-  				                                        i = 1 ....N channels

## Channel reconstruction scores for 25 and 50 pcs

- 97.9 % > 1

- 52.5 % > 1

## Channel reconstruction scores for 100 pcs

- 10.4 % > 1

## Channel reconstruction scores for 150 and 200 pcs

- 1.3 % > 1

- 0.3 % > 1

## Noise filter

- Truncating eigenvectors also results in noise filtering.
- AIRS has very low noise  -- so reconstructing the noise is not an issue.
- But for other sounders, noise filtering may be very desirable.

## Slide 23

## Noise filtering   IASI

## BAND 3  RECONSTRUCTION- SINGLE CASE

## BAND 2  RECONSTRUCTION- SINGLE CASE

## Monitoring Eigenvectors

- Monitoring eigenvectors is critical
- Eigenvectors may need to be updated due to new structures that were not in the original ensemble

## Slide 28

**Notes:**
To monitor the eigenvectors we construct daily maps of reconstruction scores.  These are reconstruction scores based on 200 eigenvectors which has a mean score of about .95  For this experiment the eigenvectors we generated from clear sky locations so large reconstruction scores generally indicate regions of cloud cover.

## Slide 29

## 12/4/00   reconstruction scores

## Monitoring reconstruction score is important

## Clear eigenvectors                            all-sky eigenvectors

## Slide 33

**Notes:**
Displaying maps of principal component scores can be very interesting. Each pcs tells a story, the problem is that you have to figure out what the story is.

## Slide 34

## Slide 35

## Slide 36

## PC Regression retrievals

## Compression Factors

- 200 pc scores ~ 9.5 %  (lossy compression).
- For near lossless compression.   Store 100 pc scores plus the residual.  The residual can be stored as 1 byte as long as residual is +- 1.28 K.
- Not all channels will be within +- 1.28 K, but no more than 20 channels per fov.
- Full data set  2100 x 4 bytes = 9400
- Compression  100 pc  x 4 bytes + 2100 ch x 1 byte + 20 outliers x 4  = 2580  ~ 27 %
- Bit trimming can provide even greater lossless data compression (perhaps 10%).

## Summary

- All sky eigenvectors
- Monitor reconstruction scores
- Update eigenvectors periodically during first 6 months.
- Provide to NWP centers 200 pc scores (normalized by sqrt(eigenvalues)).
- Provide subset of individual channels.
- Eigenvector regression is quite accurate from the  middle troposphere to upper stratosphere.
- Need to study noise filtering feature of eofs.