# Presentation: 481019

## CLASS WS/NPP ProductsNPP Data Format, Packaging and Metadata Issues

**Richard.Ullman@nasa.gov; NASA GSFC NPP SDS/ IPO**
**Robert.Rank@noaa.gov; NOAA NESDIS/CLASS**

## NPOESS – What is it?

**NPOESS is a system of polar-orbiting satellites in low earth orbit to be operated cooperatively between the Department of Defense (DoD) and the National Oceanographic and Atmospheric Administration (NOAA) with participation of the National Aeronautics and Space Administration (NASA).**
**NPOESS will be the Nation’s PRIMARY collector of satellite weather and climate data for weather prediction**
**NPOESS will collect over 50 environmental measurements which are crucial to timely, accurate, weather forecasts by military and civilian organizations. It will enable**
**– ****Precision military combat operations**
**– ****Enhanced environmental knowledge for military advantage**
**– ****Increased accuracy in severe storm warnings and forecasting**
**– ****Improved drought analysis and flood warnings**
**BETTER WEATHER FORECASTING**
**CONTINUED CLIMATE MONITORING**

## NPP Origins & Mission Objectives

**The NPOESS Preparatory Program (NPP), overseen by the Integrated Program Office (IPO), is a joint effort of the NASA, the Department of Defense and Department of Commerce.**
**Feasibility study began in November 1998.**
**Formulation began in November 1999.**
**To satisfy the nation’s critical civil and national security requirements for spacebased, remotely sensed environmental data:**
**Bridge Mission: To provide NASA with continuation of a group of global change observations initiated by the Earth Observing System (EOS) TERRA, AQUA, and Aura missions.**
**NPOESS Risk Reduction: To provide the NPOESS operational community with pre-operational risk reduction demonstration and validation for selected NPOESS instruments, and algorithms, as well as ground processing.**

## Mission Data Products

**NPP (and later the NPOESS Space Segment) will produce sensor data for processing by the NPOESS Interface Data Processing Segment (IDPS)**
**The IDPS will produce Data Records (xDR’s) of geophysical products in an operational environment while minimizing data latency**
**NASA, NOAA, and other agencies will leverage the operational EDR’s in performing climate research**
**NOAA/CLASS will archive the NPOESS Data Products**
**NPP SDS is intended to be a research tool for the primary purpose of assessing EDRs for climate quality.**
**NOAA (including NESDIS, CLASS and NDE) and NASA share an IDPS production system operated at NESDIS.**
**xDR packaging is the same for all NESDIS IDPS customers.**

## Context [NPP Example]

***Data Request***
***Subscriptions***

***Areas of Influence***

- Future?

- Easy

## Data Format Description[Separate Presentation]

**Kim D Tomashosky <KDTomashosky@Raytheon.com>**
**NPOESS Data Format Control Book **

## Breakout Purpose

**NPP and NPOESS products will be archived in CLASS and distributed for the benefit of the user community.  **
**NPP/NPOESS products are produced by the NPOESS IDPS in HDF5.**
**The NESDIS instance of IDPS has flexibility in packaging the data to serve the NESDIS community overall including CLASS.**
**All customers of the NESDIS IDPS will receive the same product packaging.**
**Community input is needed to help CLASS determine the packaging most beneficial to end users.**
**The charge to this forum:**
**understand the current design**
**make recommendations for product packaging and future improvements. **

## Discussion Topics

** ****Data packaging including HDF5 and metadata**
**HDF5 **
**How the NPOESS files use HDF5**
**Packaging and Aggregation**
**Arrangement of information within the HDF5 file**
**Method for inclusion of information such as metadata**
**Aggregation size (see later charts)**
**Packaging with related information (such as other data products or geolocation or quality flags)**
**Metadata**
**Classes of metadata (not discussion of values of individual attributes)**
**Provenance records (history)**
**For another forum**
**Granule size**
**Algorithm changes**
**Calibration/Validation programs**
**Transformation to alternate standards (BUFR, netCDF, HDF-EOS) **
**Services on data (OPeNDAP, WCS, etc) **

## Context [NPP Example]

***Data Request***
***Subscriptions***

***Areas of Influence***

- Future?

- Easy

## HDF-NPOESS

**HDF-NPOESS is a convention for use of HDF5 and defines the way that NPOESS IDPS packages and distributes data to the civil and military weather “centrals”**
**There is no “HDF-NPOESS” library, HDF-NPOESS is a design convention using the native HDF5 library. **
**Documented in the NPOESS CDFCB-X**
**Common Data Format Control Book-eXternal**
**Copies available by request.**
**This presentation describes the current convention.**

## Granule: Definition in NPOESS

| NPP Sensor | Granules per Orbit | Granule Length (sec) |
| --- | --- | --- |
| CrIS/ATMS | 761 | 8 |
| VIIRS | 213 | 28.58 |
| OMPS NP | 160 | 38 |
| OMPS TC | 800 | 7.6 |

**Granule: Definition in NPOESS**

**NPOESS IDPS is designed for high performance operation and one aspect of that design is that data granules are relatively small.  Granule size may change, current definitions are:**

## Aggregation

**For convenience of distribution to and use by primary customers, the NPOESS Granules are packaged in HDF5 as “aggregations” of the same kind of granule.  **
**An aggregation may contain as few as one granule, and for swath data, as many as an orbit of granules.  **

## Aggregation

**Aggregation**

**HDF-NPOESS provides packaging by aggregations of whole granules**
**Fields are stored as HDF datasets.**
**An aggregation is represented as a set of object reference pointers to the whole of respective field datasets.**
**A granules is represented as a set of region reference pointers to sections of the respective field datasets.**
**Metadata is attached to the sets of references.**
**Aggregate metadata to the _Agg object reference arrays.**
**Granule metadata to the _GranN region reference arrays.**

## Data_Products/
-xDR*/
_Aggr*
Object pointer array

_Gran_0*
Region pointer array

_Gran_1*
Region pointer array

* Indicates that metadata is attached here

**-xDR*/**
**_Aggr***
**Object pointer array**
**_Gran_0***
**Region pointer array**
**_Gran_1***
**Region pointer array**
*** Indicates that metadata is attached here**

**All_Data/**
**-xDR_All/**
**Field 1 Dataset Array**
**Field 2 Dataset Array**
**Field 3 Dataset Array**

**Aggregation**

## Aggregation

**For swath data:**
**Granules are time-ordered with gran_0 the earliest granule in the aggregation.**
**Aggregations are a covering of all granules of a specific data product that span the time from the beginning time of gran_0 through the ending time of gran_n.**
**Gaps in granules are filled using the defined missing_data fill value.**
**Missing whole granules are missing.**

## Aggregation: CLASS User Recommendation

| NPP Sensor | Granules per Orbit | Granule Length (sec) | Granules per Aggregation | Aggregation 
Length (sec [min]) |
| --- | --- | --- | --- | --- |
| CrIS/ATMS | 761 | 8 | 45 | 360 [6] |
| VIIRS | 213 | 28.58 | 10 | 285.8 [~5] |
| OMPS NP | 160 | 38 | 160 | 6080 [~101] |
| OMPS TC | 800 | 7.6 | 800 | 6080 [~101] |

**Aggregation: CLASS User Recommendation**

**The recommended time aggregation for NPP/NPOESS granules is:**
**NASA SDS recommendation:**

## Data DeliveryHDF5 Packaging

**HDF5 Packaging Functionality implemented in Build 1.4**
**Scenario: EDR1 and EDR2 have the same associated Geolocation**
**Option 1: Order EDRs and Geolocation together with “packaging on”.**
**Alternate 1a: Order each EDR with Geolocation “packaging on” in separate requests. **
**Option 2: Order EDRs and Geolocation together with “packaging off”.**
**Suggested by NASA SDS (but not in current IDPS design)**
**Proposed: Order EDRs packaged together with Geolocation separate.**

- v 1.4

- Suggested by NASA SDS

- 2 HDF5 files
- “EDRs together
- Geo separate”

## Packaging: CLASS User Recommendation

**If possible, this workshop should recommend which xDRs should be packaged together.**
**Geolocation included.**
**Other packaging recommendations? **
**quality flag inclusion.**
**See separate table of xDR products**

## Metadata

**Classes of metadata**
**Collection:  Describes attributes that are the same from instance to instance of a product.  NPOESS program has sometimes called this “Quasi-static” metadata**
**Aggregation: Describes attributes of the packaged file**
**Granule: Describes attributes related to a particular granule**
**Dataset Array: Describes attributes related to a particular field array (an HDF dataspace)**
**Element: Quality flags, most often a parallel array to the data array.**
**Not all of these metadata are available in machine readable form, and only some are available as native HDF attributes**

## Collection Metadata

**NPOESS Collection metadata is not provided in a machine readable form, however the information needed to populate a collection record is available in various configuration controlled documents.**

## Aggregation Metadata

**Describes the aggregation file.  Two sources:**
**The HDF User Block, the leading part of the HDF file, readable without HDF tools.**
**In XML, according to a schema documented in the CDFCB-X Volume V.**
**Attached as HDF attributes, to the _Agg group**

## Metadata: HDF5 Attribute Values

**Metadata: HDF5 Attribute Values**

**Metadata attributes that are packaged within the HDF5 product.  **
**A spreadsheet correlating NPOESS metadata attribute names to the FGDC standard is available. **
**Metadata is documented in CDFCB-X Volume V.**
**Metadata attributes are stored as HDF attributes attached to “Data_Products” hierarchy:**
**The particular product group**
**The array of aggregation field references **
**The array of granule field references. **
**No metadata attributes are attached to the “All_Data” hierarchy.**
**No metadata attributes are attached to array fields (datasets)**

## Metadata: Array metadata

**Each NPOESS product is described by a XML “profile” that describes the structure and data type of each parameter field. Metadata that appears in the XML profile includes:**
**Fill Values, Dimension Names, Measurement Units, Datum Description and**** “L****egend ****E****ntries” for bit-mapped fields are located in the XML profile of the product and not in the HDF5 packaging of the product. **
**Some parameter field arrays are packaged as scaled values. The Scale and Offset values for scaled field arrays are found in a separate data array from the values array.  The association between the scale factor array and the field is found in the XML profile of the product and not in the HDF5 packaging of the product. **

## Metadata: Element Quality Flags

**Field element metadata (Quality Flags)**
**Quality flags associated with field elements are represented as bits and packed into byte arrays.**
**The association between data field and quality flags is defined in the XML product profile.  There is no explicit relationship in the HDF5 file, except that they reside in the same xDR group and are congruent.**
**Information required to extract the quality flags and a legend of their definition are provided in the XML product profile and not in the HDF5 file.**
**All quality flags defined for a given xDR are always included.**

## Metadata: Possible Class User Workshop Recommendation

**All metadata should be provided using HDF5 native facilities (i.e. HDF5 attributes attached to groups or datasets) **
**Parsing of XML (product profiles, user block, delivery records) should be unnecessary for full description of the product.**
**Collection metadata should be attached to the All_Data/Product group.**
**Aggregate metadata should include all metadata that properly describes the aggregation and should be attached to the All_Data group**
**Metadata descriptive of arrays should be attached to the array that they describe using HDF5 attributes.**
**Metadata descriptive of granules should be attached to the gran_n reference region array (as is presently the design).**
**Element metadata (quality flags) legend should be attached to the quality flag array.**
**CLASS should provide the ability to order data without quality flag metadata.**

## Breakout Group Discussion Additional User Questions

**What test data and needs are to be satisfied for the user community?**
**What metadata is required? **
**Quality/stability & uncertainty of measurements**
**Provenance/History**
**Quasi-static metadata: (Performance monitoring, Event logs)**
**What are the needs for tools (formats, subsetting)?**
**What documentation should be archived (instrument, algorithms)?**
**Should each data format from CLASS allow options for calibration into a number of geophysical units?  If so, which ones? (Radiance, brightness temperature, etc)**
**After products are re-processed (i.e. for climate applications), should all previous versions of the products remain in the archives?**