# Presentation: 470011

## Fred Patt
NASA NPP SDS
NASA Goddard Space Flight Center

NOAA CLASS Users’ Workshop
August 7-8, 2006
Boulder, CO

fred.patt@gsfc.nasa.gov

**Fred Patt**
**NASA NPP SDS**
**NASA Goddard Space Flight Center**
**NOAA CLASS Users’ Workshop**
**August 7-8, 2006**
**Boulder, CO**
**fred.patt@gsfc.nasa.gov**

**Product Array-level Metadata**

## Background

**Self-describing data formats (including HDF 4 and HDF5) support descriptive attributes on data objects (e.g., science data arrays) within files.**
**HDF5 extends attribute support to groups and complex data types.**
**NASA EOS missions and programs (e.g, MODIS, SeaWiFS) have taken advantage of this capability to add standard descriptive metadata to the arrays in order to make them truly self-describing.**
**Sample array-level metadata:**
**units**
**long_name**
**valid_range**
**slope**
**intercept**
**fillvalue**

## Issue

**Current NPOESS data product formats (as specified in the CDFCB) do not include array-level metadata.**
**Product Profiles, as proposed by NGST/Raytheon, are separate files that contain most of the descriptive information about arrays.**
**Scaled integer implementation (where applicable) uses separate data arrays for scaling information; this precludes use of standard tools for reconstituting data.**
**While both of these satisfy the “letter of the requirement” to describe the data products, they both require additional steps and/or product-specific code in order to read and interpret the data in the arrays.**

## Recommendation

**Incorporate scaling and descriptive information about data as array-level metadata using HDF5 attributes.**
- Metadata values can be set to static values which do not require input from data processing.
- Use of attributes on data arrays establishes the implicit and unbreakable connection between metadata and data.
- Attribute names and values are automatically listed together using generic HDF tools.
- No extra files to maintain.
- Use of standard scaling attributes allows standard software to reconstitute data from scaled integers without customization.
- Descriptive attributes can be automatically read and displayed by analysis and display software (e.g., SeaWiFS Data Analysis System (SeaDAS)).
- Establishment of valid ranges and flag values using attributes enables automated quality checking and validation of array data.
** **

## Example – Data Product Listing

-  short nLw_412(Number of Scan Lines, Pixels per Scan Line) ;
-                 nLw_412:long_name = "Normalized water-leaving radiance at 412 nm" ;
-                 nLw_412:slope = 0.001 ;
-                 nLw_412:intercept = 0. ;
-                 nLw_412:units = "mW cm^-2 um^-1 sr^-1" ;
-  float chlor_a(Number of Scan Lines, Pixels per Scan Line) ;
-                 chlor_a:long_name = "Chlorophyll Concentration, OC3 Algorithm" ;
-                 chlor_a:slope = 1. ;
-                 chlor_a:intercept = 0. ;
-                 chlor_a:units = "mg m^-3" ;
- short tau_869(Number of Scan Lines, Pixels per Scan Line) ;
-                 tau_869:long_name = "Aerosol optical thickness at 869 nm" ;
-                 tau_869:slope = 1.e-04 ;
-                 tau_869:intercept = 0. ;
-                 tau_869:units = "dimensionless" ;

**Array-level Metadata in Current NASA MODIS Ocean Color Product**

## Example – Data Analysis Tool

**SeaDAS, a standard NASA tool with nearly 1000 user sites worldwide, makes use of array-level metadata in data displays**

## Robert Wolfe
NASA NPP SDS
NASA Goddard Space Flight Center

NOAA CLASS Users’ Workshop
August 7-8, 2006
Boulder, CO

robert.e.wolfe@nasa.gov

**Robert Wolfe**
**NASA NPP SDS**
**NASA Goddard Space Flight Center**
**NOAA CLASS Users’ Workshop**
**August 7-8, 2006**
**Boulder, CO**
**robert.e.wolfe@nasa.gov**

**VIIRS Product Aggregation and Packaging**

## Background

**There are three broad areas where the end-users can help define the VIIRS products distributed by NOAA:**
**Temporal aggregation**
**Band packaging**
**Geolocation packaging**
**Good decisions in these areas will make the products more useful to the community**
**NASA NPP VIIRS scientist and engineers would like to develop a community consensus in these areas to help make our job easier – it will make evaluation of the xDRs easier for us**
**Note that other product specific format concerns (scaling, QA bits) are important but not part of this presentation**
**Our proposals in these areas are based on our experience from NASA EOS missions and programs (e.g, MODIS, SeaWIFS) which are used by a broad operational and scientific community**
**MODIS science team and associates alone are close to 200 users**
**Other users, including direct broadcast users, number in the 1000s**

## VIIRS Temporal Aggregation – Background

**IDPS’s basic unit of production for a VIIRS swath xDR is a 28 sec. granule – this size is driven by processing efficiencies, not by end-user needs or for ease of archiving**
**NESDIS, in their instance of the IDPS, can choose how the 28 sec. granules are temporally aggregated**
**Temporal aggregation can reduce the number of VIIRS swath xDRs granules that need to be handled by a factor of 10 or more**
**Aggregation options considered:**
**5 minute granules (MODIS heritage)**
**1 orbit granules (AVHRR heritage)**
**12.5 to 16.7 minute granules (1/8 to 1/6 orbit) (unknown heritage)**

## VIIRS Temporal Aggregation – Our Proposal

**We are proposing that the VIIRS xDRs be aggregated in 5 minute granule aggregates**
**Rationale: many users are happy with MODIS 5 minute granules**
**This aggregate size was chosen to keep the multiple band L1 (SDR) EDR files to a reasonable size (< 500 MB) while minimizing the total number of files**
**Proposal advantages:**
**Users needing to inter-compare VIIRS data with MODIS 5 minute aggregates will benefit from a VIIRS 5 minute granule aggregate**
**For operational users, five minutes is a good compromise for size vs. latency**
**Granule Aggregates are more easily visualized because they are roughly square**
**Works well with band and geolocation packaging (below)**
**Disadvantage of even larger aggregates:**
**Even larger aggregates would further reduce the number of granules – but this comes at the cost of increased latency and granule size**

## VIIRS SDR Band Packaging – Background

**IDPS’s basic unit of production for a VIIRS SDRs (calibrated radiances) are granules containing individual bands – this was also driven by processing efficiencies, not by end-user needs or for ease of archiving**
**NESDIS, in their instance of the IDPS, can choose how the SDR bands are packaged in a product**
**Band packaging can reduce the number of VIIRS SDR granules that need to be handled by a factor of 10 or more**
**Band packaging options considered:**
**Individual bands – no band packaging (unknown heritage)**
**Resolution, e.g. all M bands together (MODIS heritage)**
**Spectral, e.g., all thermal bands together (ASTER heritage)**
**By end use, e.g., all “Ocean color bands together” (Ocean Color subset heritage)**

## VIIRS SDR Band Packaging – Proposal

**We are proposing that the VIIRS SDRs be aggregated by resolution in three files: I bands, M bands and D/N band**
**Rationale: many users are happy with MODIS granules that are packaged by band**
**Advantages:**
**Users needing to use or inter-compare VIIRS granules with MODIS granules will benefit from having all bands with the same resolution in one product**
**Operational users will benefit from a reduction in the number of granules needed to be handled and tracked**
**Disadvantages:**
**Some users will get some bands that they don’t necessarily need – this could be mitigated by NESDIS/CLASS providing a band subsetting capability**

## VIIRS Geolocation Packaging – Background

**IDPS’s produces 6 different VIIRS geolocation products, two geolocation data sets for each resolution (M, I and D/N)**
**Both the intersection with the Ellipsoid and with the Terrain are produced**
**In addition to the spatial coordinates (lat, long, and terrain height) the viewing geometry is also produced: solar and sensor azimuth and zenith angles, range, and for the D/N bands, lunar azimuth and zenith angles**
**NESDIS, in their instance of the IDPS, can choose how the geolocation data is packaged in the SDR and EDR products**
**Geolocation can also be requested as standalone products**
**Proper geolocation packaging can make the products more useable**
**However, packaging geolocation in every product can make the overall storage volume very large**
**IDPS does not have a sub-sampled geolocation packaging option – i.e., provide the geolocation (or just the viewing geometry) for every n****th**** line/sample**
**This is one option that is currently being used in MODIS L1B and L2 atmosphere products**

## VIIRS Geolocation Packaging – Proposal

**We are proposing that:**
**Terrain corrected VIIRS geolocation be packaged with SDR data packaged by band resolution (see above)**
**In addition, store terrain corrected geolocation data separately for each resolution – so that users can order it with any EDR product**
**Do not store the ellipsoid geolocation for any resolution**
**Advantages:**
**We are not aware of any users of the ellipsoid geolocation**
**Altogether, the volume stored in the archive is equivalent to the volume produced**
**Disadvantages:**
**EDR users will either have to also order the SDR products or separate geolocation products**

## VIIRS SDR Band Packaging – Wish List

**An additional option (on our wish list) would be to spatially aggregating the I bands and D/N bands to the M band resolution (similarly for the D/N band to the I bands)**
**virtual M (and I) bands are created**
**this would involve a resampling step for the D/N bands since they are not nested within the M or I bands**
**MODIS also creates virtual bands by spatially aggregating the finer resolution 250m and 500m bands**
**Some I bands may not need to be aggregated because they are spectrally the same (I1 and I2) or similar (I3 and I4) to the M bands**
**Advantages:**
**this allows users without a need for the higher spatial resolution to easily use the VIIRS I bands and the D/N band**
**it would also allow the D/N band to be more easily used with the I bands because it would have the same geometry**
**Note that this option is not something the current IDPS system has the capability to produce (so augmentation would be needed)**

## VIIRS SDR Geolocation Packaging – Wish List

**An additional option (on our wish list) would be a sub-sampled geolocation packaged with some EDRs**
**Package geolocation (or just the viewing geometry) for every n****th**** line/sample**
**Advantages:**
**Allows users (e.g. of atmosphere products) who do not need the full resolution geolocation to get EDRs packaged with just the geolocation they need**
**Smaller files sizes**

## Liam Gumley
NASA NPP SDS
Space Science and Engineering Center
University of Wisconsin-Madison

NOAA CLASS Users’ Workshop
August 7-8, 2006
Boulder, CO

liam.gumley@ssec.wisc.edu

**Liam Gumley**
**NASA NPP SDS**
**Space Science and Engineering Center**
**University of Wisconsin-Madison**
**NOAA CLASS Users’ Workshop**
**August 7-8, 2006**
**Boulder, CO**
**liam.gumley@ssec.wisc.edu**

**Aggregation and Packaging for****VIIRS and CrIS/ATMS**

## Background

**Our recommendations are based on our experience in working with data from MODIS and AIRS**
**The aggregation strategies chosen for MODIS granules have made the data easier to use, namely**
**Temporal: 5-minute granules (any scan beginning in a given 5-minute window are included in the granule)**
**Spatial/Band: Having all bands available at native spatial resolution and resampled resolution makes it easier for algorithms which use multiple bands (e.g., cloud mask)**
**1KM file contains bands 1-36 at 1Km resolution**
**HKM file contains bands 1-7 at .5Km resolution**
**QKM file contains bands 1-2 at .25Km resolution**
**We routinely co-locate MODIS and AIRS granules (i.e., locate MODIS pixels within each AIRS FOV), and have devised methods for dealing with 5-min. MODIS vs. 6-min AIRS granules. **

## Recommendations

**VIIRS**
**Temporal aggregation into 5-minute granules (concur with Land PEATE)**
**Any scan where start of earth-view is within 5-minute window is included in the granule (no partial scans)**
**Band aggregation**
**One file containing all M-bands at native resolution with geolocation at M-band resolution for every pixel**
**One file containing all I-bands at native resolution with geolocation at I-band resolution for every pixel**
**One file containing D/N band with geolocation for every pixel**
**CrIS/ATMS**
**Temporal aggregation into 6-minute or 8-minute granules (concur with Sounder PEATE)**
**CrIS and ATMS data in separate files**
**All channels contained in one file**
**Each file contains geolocation for every FOV**

## Wish List

**VIIRS**
**SDR: Would like to have all bands available at M-band spatial resolution (very helpful for cloud mask), i.e.,**
**one file containing all bands at M-band 750 m resolution (I-bands and D/N band resampled)**
**one file containing I-bands and D/N band at I-band 375 m resolution (D/N band resampled)**
**EDR: Would like to have atmosphere EDRs aggregated into one file (e.g., like MODIS cloud product MOD06_L2 which contains cloud top pressure, emissivity, phase, etc.)**

## Robert Ando
NASA NPP SDS 

NOAA CLASS Users’ Workshop
August 7-8, 2006
Boulder, CO

robert.r.ando@jpl.nasa.gov

**Robert Ando**
**NASA NPP SDS **
**NOAA CLASS Users’ Workshop**
**August 7-8, 2006**
**Boulder, CO**
**robert.r.ando@jpl.nasa.gov**

**CrIMSS Aggregation and Packaging**

## Background

**IDPS has the capability of aggregating NPP data supplied to the NOAA CLASS / LTA for archiving.  Aggregation results in a reduced number of files to be transferred and the associated bookkeeping.  **
**Aggregation, packaging, and data format for CrIMSS (CrIS and ATMS) data are being discussed.  **
**NASA SDS was requested to bring preferences to this venue for public review and/or concurrence.  **

## CrIS and ATMS Issues

**Aggregation**
**What is the preferred temporal aggregation for CrIS and ATMS granules to assist current evaluation of xDRs and for long term archive? **
**How will missing data be handled? **
**Do aggregates contain fixed numbers of granules? **
**Will CrIS and remapped ATMS aggregations be temporally aligned? **
**Packaging**
**Should all channels/bands be packaged together in one file? **
**Should Geolocation information be packaged with the sensor data and/or be supplied as a separate file? **
**Which xDRs should be packaged together? **
**Data Format**
**Units for Latitude, Longitude, other Geolocation parameters.**
**Floating point data output as scaled integers. **

## CrIS and ATMS Aggregation (1 of 2)

**Assumptions:**
**Aggregates are fixed periods of time, not fixed amounts of data.**
**Aggregations are not synchronized with granule times, so aggregations could contain varying numbers of granules.  **
**Every granule will be put into one and only one aggregate based on either the start or end time of the granule (No duplication of granules).  **
**If no granules are available in an aggregation period, then no aggregation will be produced.  **
**Missing granules will not be replaced with fill.  **
**Granules containing missing data will have the missing data filled with the appropriate HDF fill value during (or before) aggregation.  **
**Remapped ATMS xDR granules will exactly correspond to CrIS granules, and thus ATMS aggregates can also correspond to CrIS aggregates.  **

## CrIS and ATMS Aggregation (2 of 2)

**CrIMSS scan period and granules are 8-seconds.  **
**For production efficiency IDPS is considering outputting four 8-second granules at a time. **
**Temporal Aggregation period:  **
**Should be evenly divisible by 8-second granule. **
**Should be an integer number of minutes.**
**Should be evenly divisible into 24 hours.  **
**Multiples of 2-minutes (15 x 8 seconds) satisfies the above conditions.  **
**Users are experienced/comfortable with 6-minute AIRS L1B and L2 granules.  **
**If IDPS chooses to output 4 granules at a time, then 4, 8, 12 minute (etc.) aggregations are more suitable.  **
**  **

## Typical AIRS Granule Map (ascending node, 6-minute granules)

## CrIS and ATMS Packaging

**Band Packaging - We expect all CrIS bands (LW, MW, SW) and all wavelength channels to be output together in the xDR granules.  **
**CrIMSS geolocation is much smaller than the sensor data.  **
**CrIS SDR, remapped ATMS, and CrIMSS EDR which correspond in time/space could conceptually be packaged together for user convenience.  Similarly ATMS SDR & TDR can be packaged together.  Would result in a small savings in redundant information.  **
**CrIS SDRs contain 2 “QC” words (the imaginary part of the signal and a noise estimate) for every 1 measurement word (the real part of signal).  Sounder PEATE will ingest all the data, but end users might prefer the option of receiving a smaller product containing a summary of phase and noise.  **

## CrIS and ATMS Data Format

**Observation:  A recent CDFCB (June ‘06) specifies radians as the units of some parameters (Roll, Pitch, Yaw, Latitude, Longitude and Solar angles) in ATMS and CrIS SDR Geolocation Summaries.  **
**Latitude, Longitude traditionally use degrees.  **
**Many different units may be used for the other parameters, but it is important to know the frame of reference and convention.  **
**White Paper ‘IDPS Scaled Integers’ Richard Elsbernd, James Austin, 10/31/2005, specifies scaled integers for CrIS and ATMS SDRs and EDRs.  **
**CDFCB generally specifies floats for real parameters but some TBDs remain.  **
**Data accessed as floating-point numbers in physical units is much easier for users. **
**Supported by HDF, and HDF internal compression can produce files as small as a file of scaled integers.  **

## CrIMSS Community Recommendations

**For CrIMSS data, the Sounder and Atmosphere PEATEs prefer ****6-minute aggregations, but if IDPS outputs CrIS and ATMS granules four at a time (32-seconds), then 8-minute aggregations are preferred.  **
**CrIS and ATMS aggregations should be temporally aligned.  **
**All channels and all bands should be combined in one file. **
**Each file produced should contain the Geolocation for every FOV.  All geolocation information should be copied into ATMS and CrIS product aggregates.  **
**Prefer Latitude and Longitude Units to be degrees, not radians.  **
**Floating-point numbers in physical units rather than scaled integers.  **
**Prefer CrIS and ATMS in separate files.  **
**Sounder PEATE would like *****the option***** of packaging multiple data types together:  **
**CrIS SDR, + remapped ATMS + CrIMSS EDR.  **
**ATMS SDR + TDR**

## Wish List

**CLASS data users may find useful:**
**A channel or band subsetting capability.  **
**A separate geolocation product.  **
**Re-packaging data types (different combinations).  **
**Summarized QC, involving pre-processing of the original data.  **

## OMPS Aggregation and Packaging

**OMPS Aggregation and Packaging**

**Mike Linda**
**NASA NPP SDS**
**NOAA CLASS Users’ Workshop**
**August 7-8, 2006**
**Boulder, CO**
**Mike.Linda@gsfc.nasa.gov**

## Background (OMPS)

**What We Know*:**
**RDR EV****	****    (TC, NP, LP) ****	****7.6, 38, or 18.72 sec/granule**
**RDR Cal****	****    (TC, NP, LP) ****	****20 min/granule [undocumented]**
**RDR Other****	****    (TE, DI)****	****(size unknown)**
**SDR****	****    (TC, NP, LP) ****	****(size unknown)**
**EDR****	****    (TC, NP, LP) ****	****38 sec/granule [suspect]**
**What We Don’t Know:**
**Format of OMPS SDRs and EDRs**
**Format of IPs**
**Production format of upload files and databases**
**Getting copies of upload files**
**Getting copies of databases **
**Therefore ... much TBD!**
**	****	****	****	****	*********Data Format Control Books (DFCBs) have too many TBDs**
**	****	****	****	****	****EV = Earth View****	****TC = Total Column**
**	****	****	****	****	****TE = Telemetry ****	****NP = Nadir Profile**
**	****	****	****	****	****DI = Diagnostic ****	****LP = Limb Profile (de-manifested)**

## Ozone Community Preferences – 1 of 2

**One HDF5 file per orbit of data**
**File covers day-lit side**
**	****Given an ****ascending****-node orbit, aggregate file boundary is at start of first sub-granule following ****descending**** equator crossing.**
**Whole sub-granules in file**
**No data duplicated between files**
**All wavelengths included in each file**
**Geolocation included in each data file**

## Example of Orbital Data Filefor Ascending Orbit

- Not Desirable 

- Desirable

- aggregate file boundary is at start of first sub-granule following ascending equator crossing

- aggregate file boundary is at start of first sub-granule following descending equator crossing

**Flight Dynamics increments orbit numbers at Equator crossing when the spacecraft is ascending (South-to-North).**

## Ozone Community Preferences – 2 of 2

**All* metadata included in each data file**
**(no separate met files, profiles, or file specs)**
**Floating-point numbers in physical units**
**(not scaled integers)**
**Missing data filled with well-defined fill values**
**Compatible format from CLASS as from IDPS**
**Compatibility with heritage ozone mission formats**
**	****	****	****	*****granule- and lower-level (not collection-level) metadata**

## Ozone Community Assumptions

**CLASS* stores all that is necessary for reprocessing****
**xDRs**
**Delivered IPs**
**Engineering data ****(incl. C****3****S data if not in RDRs)**
**Upload files **
**Databases**
**Software (source code)**
**Calibration Artifacts**
**Data**
**Analysis tools**
**Tables**
**Logs**
**Notebooks**
**Instrument design**
**All Project Documentation**
**All Scientific Papers**
**All Reports**
**	****	****	****	****	****  *****If not CLASS, then who?  Need **_***permanent***_** storage.****	****	****	****	******Long-term climate analysis requires reprocessing with algorithm tweaks**

## Ozone Community Preference Summary

**~14 (xDR) files per product per day (one file per orbit)**
**OMPS files < 500 MB**
**Simplicity via all-inclusive files (incl. metadata, geo, all bands)**
**Need Low Impact on the Atmospheric Chemistry Community**
**Hundreds of research scientists**
**Thousands of users**
**Used to analyzing data in a particular form (evolving formats)**
**Many tools and compatible products exist**
**30 years of data – one of NASA’s longest ranging**
**TOMS and OMI data consistent with AURA formats**
**AURA formats use-tested after years of negotiating**
**	****	****http://www.eos.ucar.edu/hirdls/HDFEOS_Aura_File_Format_Guidelines.pdf**
**OMPS* format should be compatible with OMI/Aura**
**	****	****	****	****	****	****	****	*********LP de-manifested**