# Presentation: 874118

## Level-3 Product & Web Site Review 

- MODIS Atmosphere
- Level-3 Product & Web Site Review

- Paul A. Hubanks          Science Systems and Applications, Inc.

## Presentation Outline:

1.  Level-3 (L3) Primer  --  High-level algorithm & product description

2.  Web Site Review  --  Web-based tools and key features

3.  Case Studies  --  Using web & interactive tools to diagnose algorithm issues

4.  L3 Updates  --  Collection 005 change summary

- 1.  Level-3 (L3) Primer  --  High-level algorithm & product description
- 2.  Web Site Review  --  Web-based tools and key features
- 3.  Case Studies  --  Using web & interactive tools to diagnose algorithm issues
- 4.  L3 Updates  --  Collection 005 change summary

## 
Topic 1.

The L3 Algorithm

**Topic 1.**
** **
**The L3 Algorithm**

## L3 Daily Algorithm: Characteristics

- Important Limitations:
- Limited input - only L2 files
- Fixed grid - equal-angle 1x1 degree
- Subsampling - L2 pixels sampled at geolocation resolution
- No valid range check - inconsistently defined and implemented at L2 - potential to mask L2 algorithm problems
- Statistical Computation:
- Aggregation - using L2 QA flags (e.g: ice vs. water cloud phase)
- QA-weighted Statistics - using L2 QA “confidence flags”
- Comparison Statistics - joint histogram and/or regression statistics may be computed of one L2 parameter against another

**Notes:**
Reduce Text Here ... bullet points!

## 

Sub-sampling

**Sub-sampling**

## L3 Sub-sampling Impact

| Product 
Family |  L2 Data 
Resolution | L2 Geolocation 
Resolution |  L2 Input Pixels (Max) 
 per 1° L3 Grid (Equator) | Impact |
| --- | --- | --- | --- | --- |
| Aerosol
04 | 10km | 10 km |  121 out of 121 | No |
| Water Vapor 
05 | 1 km | 5 km | 484 out of 12,231 | Yes |
| Cirrus Detection
06_CD | 1 km | 5 km | 484 out of 12,231 | Yes |
| Cloud Top Properties
06_CT | 5 km | 5 km | 484 out of 484 | No |
| Cloud Optical Properties
06_OD | 1 km | 5 km | 484 out of 12,321 | Yes |
| Atmosphere Profile
07 | 5 km | 5 km |  484 out of 484 | No |

## 

The meaning of “QA”

**The meaning of “QA”**

**Notes:**
Last Meeting Robert Pincus gave a great talk on the innovative design of the L3
Now I want to focus a bit more on the actual implementation and computational characteristics ... 

- QA or Quality Assurance is a loosely defined term that covers a myriad of ancillary information produced along with with many L2 parameters
- QA is described in the QA Plan document	 (available on the modis-atmos web site -- being updated for Collection 005)
- The “runtime” QA flags produced by Level 2 algorithms are typically stored as packed bit-strings in “Quality Assurance” SDS’s. These flags cover many aspects of L2 retrieval parameters including:
- Input (ancillary) data sources
- Approach (retrieval method)
- Various scene attributes
- Success or failure of various criteria
- Data quality or confidence estimate
- 	(*QA confidence flags*)

**Notes:**
Working on updating the QA Plan!  

- 	QA confidence flags, when computed, are set for every L2 pixel.  The four categories (for non-fill L2 data) are:
- 0 = No Confidence →  0x weight
- 1 = Questionable or Marginal →  1x weight
- 2 = Good Confidence →  2x weight
- 3 = Very Good Confidence →  3x weight

**Notes:**
Most L2 QA confidence flags have reserved 3 bits ... The old version of the plan mentions that we might set as many as
8 levels of confidence however the L3 code was written to handle only 4 levels (2 bits max) ... which is the number of confidence levels currently set by L2 teams

## Who is setting “meaningful” L2 QA?

| Product 
Family | L2 QA Set? | Detail |
| --- | --- | --- |
| Aerosol
04 | Yes | Differentiate experimental from non-experimental results |
| Water Vapor 
05 | Yes | Negligible impact at L3 |
| Cirrus Detection
06_CD | No | No QA bits reserved in QA Plan |
| Cloud Top Properties
06_CT | No | Dropped in Summer 2002 |
| Cloud Optical Properties
06_OD | Yes | Based on joint distribution of 
Tau and Re |
| Atmosphere Profile
07 | No |  No impact visible at L3 |

## 

Unique Characteristics of the 
L3 Multiday 
(8-Day and Monthly)
Algorithm

**Unique Characteristics of the **
**L3 Multiday **
**(8-Day and Monthly)**
**Algorithm**

## L3 Multiday Algorithm: Characteristics

- 1. Only the L3 daily files used as input (highly efficient)
- 2. Identical Grid, SDS dimensions, and Histogram Bins
- 3. Three weighting schemes used to compute multiday statistics from daily statistics
- Unweighted
*	**(time-averaged mean, meaningful for computing temperature averages)*
- Pixel count weighted
*	**(**count-averaged mean, ensures computed means match means from histograms)*
- Pixels count weighted with pixel count screen

## L3 Multiday Weighting Scheme Specification

| Product 
Family | Multiday Weighting 
Scheme * | Detail |
| --- | --- | --- |
| Aerosol
04 | Pixel Count Weighted 
with Pixel Count Screen | Each daily grid weighted by the pixel count & 
daily grids with counts < 6 are excluded |
| Water Vapor 
05 | Pixel Count Weighted | Each daily grid weighted by the pixel count |
| Cirrus Detection
06_CD | Unweighted | Each daily grid given the same weight |
| Cloud Top Properties
06_CT | Unweighted | Each daily grid given the same weight |
| Cloud Optical Properties
06_OD | Pixel Count Weighted | Each daily grid weighted by the pixel count (categorical, by phase) |
| Atmosphere Profile
07 | Pixel Count Weighted | Each daily grid weighted by the pixel count |

**** Weighting is applied to all non-count based statistics (mean, std); but not applied to histograms*

## Example of “Pixel Count Weighted with Pixel Count Screen”  (Multiday Weighting Scheme Update - before and after images) 

- Implemented in L3 on 1 January 2004

## 
Topic 2. 

The MODIS-Atmosphere Web Site 

modis-atmos.gsfc.nasa.gov

**Topic 2. **
**The MODIS-Atmosphere Web Site **
**modis-atmos.gsfc.nasa.gov**

**Notes:**
Group effort: Bill Ridgway, Jay Dinsick, Mark Gray, Eric Moody, & Charles Gatebe 

## Slide 16

## 

New Product Sections

**New Product Sections**

## Slide 18

## Slide 19

## Slide 20

## Slide 21

## 

Data Product Processing and 
Availability Calendar

**Data Product Processing and **
**Availability Calendar**

## Slide 23

## Slide 24

**Notes:**
Describes evolution of science algorithms

## Slide 25

## 

New Tools Section

**New Tools Section**

## Slide 27

## Slide 28

## Slide 29

## 

The Images Section

**The Images Section**

## Slide 31

## Slide 32

## Slide 33

## Slide 34

## Slide 35

## LL vs. HA Images:  Data Detail Comparison

**Equal Angle Grid**
**	****1x1° Lat-Lon**
**	****Native L3 Grid**
**Equal Area Grid**
**	****Hammer-Aitoff**
**	****Computed in IDL from Lat-Lon Image**

## Slide 37

## Slide 38

## Slide 39

## Slide 40

## Slide 41

## Slide 42

## Slide 43

## 

Web Site Usage Statistics

**Web Site Usage Statistics**

## 
March 2002 – March 2003
Unique visitors = 63,000
Total downloads = 137 Terrabytes

July 2003 – July 2004
Unique visitors = 148,000
Total downloads = 742 Terrabytes

**March 2002 – March 2003**
**Unique visitors = 63,000**
**Total downloads = 137 Terrabytes**
**July 2003 – July 2004**
**Unique visitors = 148,000**
**Total downloads = 742 Terrabytes**

- Usage Statistics – Comparisons from 1 year ago

## 
Topic 3.

Using web & interactive tools 
to diagnose L2 algorithm issues

3 Case Studies

**Topic 3.**
** **
**Using web & interactive tools **
**to diagnose L2 algorithm issues**
**3 Case Studies**

**Notes:**
Combination of the web, ability to write IDL tools on the fly, a good experienced (sometimes intuitive) analyst.

## 

Case 1. 
The “Sahara Streak”
(Aerosol)

**Case 1. **
**The “Sahara Streak”**
- (Aerosol)

**Notes:**
Now Fixed – All Aqua Collection 004 & Terra Collection 004 after 1Jan2004

## Slide 48

**Notes:**
L3 images (web based) are very useful for quickly diagnosing gross errors.

## Computation of Geolocation in L2 Atmosphere Products

**5-km resolution**
- 5-km geolocation is **copied** from center (3,3) 1-km L1B input pixel in each 5x5-km area

**10-km resolution**
- 10-km geolocation is **computed** by averaging the 4 central (5,5), (5,6), (6,5), (6,6) 1-km L1B input pixels in each 10x10-km area

**Notes:**
Lucky that I was in meetings discussing the new L2 joint product and the two sets (5km and 10km) of  geolocation ... 
and I had just asked Allen Chu how they computed the 10km geolocation

## Slide 50

**Notes:**
L3 images (web based) are very useful for quickly diagnosing gross errors.

## Slide 51

## 

Case 2. 
Anomalous Water Vapor scans
(Water Vapor Near Infrared Clear)
Clear = bright land and ocean sunglint

 Similar anomalous property found in Cirrus Detection Parameters

**Case 2. **
**Anomalous Water Vapor scans******
- (Water Vapor Near Infrared Clear)
- Clear = bright land and ocean sunglint
****** ****Similar anomalous property found in Cirrus Detection Parameters**

**Notes:**
Fixed for Collection 005

## Dec03 Monthly L3 - Water Vapor (NIR) Image

- (”Clear” = bright land and ocean sunglint only)

## 03Dec03 Daily L3 - Water Vapor (NIR) Image

- (”Clear” = bright land and ocean sunglint only)

## L1B (Alaska) vs. Daily Water Vapor (NIR)

- 03 Dec 03

## L1B image  vs. L2 Water Vapor (NIR)

- 7.5 cm

- 0.0 cm

**Notes:**
Wrote a L2 and L2 QA bit flag visualizer in IDL ... Important characteristic of diagnostic tools is not to remap data.  If data is 2030x1354 make the image 2030x1354 (remapping can mask problem) 
Check Notes to BoCai and Rich for details !!!!!

## L1B image  vs. L2 Surface Type Flag (NIR)

- Sunglint

- Cloudy

- Ocean

- Bright Land

**Notes:**
Wrote a L2 and L2 QA bit flag visualizer in IDL ... Important characteristic of diagnostic tools is not to remap data.  If data is 2030x1354 make the image 2030x1354 (remapping can mask problem) 
Check Notes to BoCai and Rich for details !!!!!

## 

Case 3. 
 Unphysical results from Cirrus Fraction (IR) 
(Cloud Top Properties)

**Case 3. **
** ****Unphysical results from Cirrus Fraction (IR) **
- (Cloud Top Properties)

**Notes:**
This is one of the fixes implemented for Collection 005

## Slide 59

## Two “CO2 Slicing Algorithm” Cloud Flags set in 06_L2  Quality_Assurance_5km  array

- Parameter Definitions:
**Cirrus Cloud: **
**  ****CTP ≤ 700mb and CEE ≤ 0.95**
**High Cloud: **
**  ****CTP < 400mb**

- →

- →

## Cirrus Cloud:   CTP ≤ 700mb and CEE ≤ 0.95 →

- Cirrus Cloud:   CTP ≤ 700mb and CEE ≤ 0.95 →

- High Cloud:   CTP < 400mb →

## CO2 Slicing Algorithm

**Old Cloud Top Properties CO****2**** slicing subroutine:**
**High Cloud Flags initialized using “window retrieval” CTP’s. **
**	****High Cloud Flags overwritten with valid “CO****2**** retrieval” CTP’s.**
**Cirrus Flags were *****not***** initialized. **
**	****Cirrus Flags overwritten with valid “CO****2**** retrieval” CTP’s and CEE’s.**
**Bug Fix Implemented for Collection 005**
_**Cirrus Flags initialized using “window retrieval” values.**_** **
**	**

## Cirrus Cloud:   CTP ≤ 700mb and CEE ≤ 0.95 →

- High Cloud:   CTP < 400mb →

## 
Topic 4.

Level-3 Collection 005 
Change Summary

**Topic 4.**
** **
**Level-3 Collection 005 **
**Change Summary**

## Collection 005 Updates

- derived from
-                   **Aerosol (04_L2)**
** ****Combined Land & Ocean**
** ****Land only**** **

***L2 Aerosol  algorithm improvements:***
***- reduced snow contamination in land retrievals at the snow/ice edge***

- Collection 005 Updates

**Notes:**
Primer on reading table 1 dot in the table = Parameter + Statistic = L3 SDS ... L3 SDS names can be inferred ... Parameter name is the SDS prefix, the statistic is the SDS suffix. ... No known bugs in the L3 software ... bugs or limitations in the L2 products get propagated to L3... Almost all the L3 parameters will show improvement in C5 due to improved L1B and Cloud Mask (could have highlighted almost all the parameters yellow) .. I highlighted the aerosol retrieval parameters yellow here to denote the improvement made at L2 (reducing snow contamination over land at any the snow/ice edge)Green and Red highlight show additions and deletions to L3.  Each dot represents a L3 SDS (in other words a particular statistic for a given parameter) ... Some SDS’s are more space or size intensive  (note dimensions)
Multiday products (E3 and M3) have nearly the same set of parameters (exceptions are PRL and CRL shown here)
There is a slightly expanded list of statistics in the multiday products.  For simplicity will focus on the D3 here.
Angstrom Exp Land is regressed with CODL (1st soln at .47 microns)

All Fluxs are deleted ... not working correctly and not fixable at L2.
Two new parameters added ... Cloud Fraction derived from the Aerosol Cloud Mask
And an SDS representing the number of 1km pixels used to create each 10km L2 retrieval
The L3 statistics then show statistics of those counts for a given 1 degree L3 grid cell

## Collection 005 Updates

- derived from
**                  ****Aerosol (04_L2)**
** ****Ocean only**** **

- Collection 005 Updates

**Notes:**
Almost all the L3 parameters will show improvement in C5 due to improved L1B and Cloud Mask (could have highlighted almost all the parameters yellow)

## Collection 005 Updates

- derived from
-                  **Atm. Profile (07_L2)****  **

- Collection 005 Updates

**Notes:**
Almost all the L3 parameters will show improvement in C5 due to improved L1B and Cloud Mask (could have highlighted almost all the parameters yellow)

## Collection 005 Updates

- derived from
-                **Water Vapor (05_L2)**
-                              &
**               ****Cloud (06_L2)**
** ****Cirrus detection**
** ****Cloud Top properties**

- Collection 005 Updates

**Notes:**
Almost all the L3 parameters will show improvement in C5 due to improved L1B and Cloud Mask (could have highlighted almost all the parameters yellow)

## Collection 005 Updates

- derived from
-                  **Cloud (06_L2)  **
**  ****Cloud Optical properties**

- Collection 005 Updates

- File Size Change Est.
- C004 D3 = 440 mb
- C005 D3 = 550 mb

**Notes:**
Almost all the L3 parameters will show improvement in C5 due to improved L1B and Cloud Mask (could have highlighted almost all the parameters yellow)