# Presentation: 577414

## Profile of National Polar-Orbiting Operational Satellite System (NPOESS) HDF5 Files

**Kim Tomashosky, Ken Stone, Pat Purcell, Ron Andrews**
**NPOESS Program**
**Raytheon Company**
**Aurora, Colorado**

## NPOESS HDF5 General Overview

**Ron Andrews**

## HDF5 Conceptual Diagram

## HDF5 XML User Block

**The XML User Block for NPOESS Data Products provides a ‘quick-look’ into the metadata of the associated HDF5 file**
- The size of the HDF5 XML User Block will be a multiple of 512 bytes
**The XML User Blocks are defined in the following volumes:**
- Common Data Format Control Book – External (CDFCB-X) Volume V – Metadata – contains the XML User Block formats for the Raw Data Records (RDR), Sensor Data Records (SDR), Temperature Data Records (TDR), Environmental Data Records (EDR), Appliaction Related Products (ARP), and Intermediate Products (IP)
- CDFCB-X Volume VI – Ancillary, Auxiliary, Reports, and Messages – contains the XML User Block formats for the Ancillary and Auxiliary data file that are delivered in HDF5
**Example elements:**
- N_Processing_Domain
- Mission, Platform, and Instrument Names
- N_Dataset_Type_Tag
- Number_of_Data_Products
- CollectionShortName(s)
- Aggregation Information
- Timestamps (Creation Timestamp, Observation Timestamps)
- Percent Missing Data

## General HDF5 File Structure

## NPOESS HDF5 Metadata Locations

**The NPOESS HDF5 Metadata is organized hierarchically, from the top down in order to reduce duplication in information**
- Root Group
- Product Group
- Product Aggregation Dataset
- Product Granule Dataset

## HDF5 Conceptual Diagram - Data

## NPOESS Quality Flags Overview

**The concept is to provide for consistently stored, high density, quality information about the delivered data – simplifying usability while maintaining storage efficiency**
**Quality flags are qualifications of one or more consecutive bits in each byte.**
**Quality flag arrays follow the structure of the data product**
- The size of the arrays are equal to or less than the size of the data to which the quality information applies (dimensions correspond to the data product arrays)
**Quality flags will appear in the HDF5 files as *****n***** number(s) of two or three dimensional, 1-byte arrays. **
- The number of arrays is dependant on the quality flag definitions, specific to each data product
- Quality flags will be ordered such that each flag is entirely contained within a single byte, occasionally resulting in a byte with reserved or meaningless bits
- Byte alignment is the same for every quality flag array
- First bit (left-most) is the LSB
**Each byte may contain multiple bit-level flags**

## 2-Dimensional Array Example

## Detailed NPOESS UML Models

**Ken Stone**

## RDR UML Model

## Common RDR Layout

## SDR/TDR UML Model

## EDR UML Model

## Ancillary/Auxiliary UML Models

- Ancillary

- Auxiliary

## NPOESS Sample Data 

**Patrick Purcell**

## VIIRS I1 Band Environmental Data Record  Example UML Model

**VIIRS I1 Band Environmental Data Record  Example UML Model**

- These arrays are different for each data product

## HDF View - ImageView

## Product Profile Example

## HDF5 API – C++ Reader Sample Code

- Note:  A granule holds a reference to a _**dataset region**_ ...

- ... while an aggregation contains a reference to an _**object.**_

## NPOESS HDF5 Summary

**The NPOESS Program delivers the official deliverable data products (RDR, SDR/TDR, EDR/ARP/IP) and dynamic ancillary data and auxiliary data in HDF5 Files**
**The HDF5 Files have an XML User Block that can be accessed without HDF5 tools - provides a “quick-look” into the metadata before opening the HDF5 file**
**Metadata within the HDF5 files are stored as attributes**
**There are general UML Models for the NPOESS official delivered data that provide a common framework **
**Official deliverable data products are organized by reference objects (aggregations) which contain one or more reference regions (granules)**
**Although data may be accessed directly through the All Data group, the Data Products group provides integrated access:**
- Allows the user to access both metadata and data through a common HDF5 group
- Metadata is accessed directly by reading the Attribute values
- Datasets may be accessed by dereferencing the object ID stored in the Data Products Group for the aggregation or granule
**NPOESS HDF5 files provide flexibility for a variety of end users. **

## Backup Slides

## HDF5 File Root Group Metadata ElementsExample CDFCB-X Table

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Instrument_Short_Name  | X | X | X | X | X |  |  |
| Mission_Name | X | X | X | X | X | X | X |
| N_Dataset_Source | X | X | X | X | X | X | X |
| N_GEO_Ref |  | X | X |  |  |  |  |
| N_HDF_Creation_Date | X | X | X | X | X | X | X |
| N_HDF_Creation_Time | X | X | X | X | X | X | X |
| N_Software_Version | X | X | X | X |  |  | X |
| Platform_Short_Name | X | X | X | X | X |  | X |

## HDF5 File Product Group Metadata ElementsExample CDFCB-X Table

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| N_Anc_Type_Tasked |  |  | X | X |  |  |  |
| N_Collection_Short_Name | X | X | X | X | X | X | X |
| N_Dataset_Type_Tag | X | X | X | X | X | X | X |
| N_Instrument_Flight_SW_Version |  | X | X |  |  |  |  |
| N_Processing_Domain | X | X | X | X | X | X | X |
| Operational_Mode |  | X | X | X |  |  |  |

## HDF5 File Product Aggregation Dataset Metadata ElementsExample CDFCB-X Table

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AggregateBeginningDate  | X | X | X | X | X |  |  |
| AggregateBeginningGranuleID  | X | X | X | X |  |  |  |
| AggregateBeginningOrbitNumber  | X | X | X | X |  |  |  |
| AggregateBeginningTime  | X | X | X | X | X |  |  |
| AggregateEndingDate  | X | X | X | X | X |  |  |
| AggregateEndingGranuleID  | X | X | X | X |  |  |  |
| AggregateEndingOrbitNumber  | X | X | X | X |  |  |  |
| AggregateEndingTime  | X | X | X | X |  |  |  |
| AggregateNumberGranules  | X | X | X | X |  |  |  |

## HDF5 File Product Granule Dataset Metadata ElementsExample CDFCB-X Table

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Calendar_Date |  |  |  |  |  | X | X |
| Ascending/Descending_Indicator |  | X | X | X |  |  |  |
| Band_ID |  | X | X |  |  |  |  |
| Beginning_Date | X | X | X | X |  |  |  |
| Beginning_Time | X | X | X | X |  |  |  |
| Cloud_Cover |  |  | X |  |  |  |  |
| East_Bounding_Coordinate |  | X | X |  | X |  |  |
| Ending_Date | X | X | X | X |  |  |  |
| Ending_Time | X | X | X | X |  |  |  |
| G-Ring_Latitude |  | X | X | X | X |  |  |
| G-Ring_Longitude |  | X | X | X | X |  |  |
| N_Anc_Filename |  | X | X |  |  |  |  |
| N_Anc_Type |  |  |  |  |  | X |  |
| N_Aux_Filename |  | X | X |  |  |  |  |
| N_Beginning_Orbit_Number | X | X | X | X |  |  |  |
| N_Beginning_Time_IET | X | X | X | X |  |  | X |

## HDF5 File Product Granule Dataset Metadata ElementsExample CDFCB-X Table (cont.]

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| N_Columns |  | X | X | X |  |  |  |
| N_Day_Night_Flag |  | X | X | X |  |  |  |
| N_Creation_Date | X | X | X | X |  |  |  |
| N_Creation_Time | X | X | X | X |  |  |  |
| N_Graceful_Degradation |  |  | X |  |  |  |  |
| N_Ending_Time_IET | X | X | X | X |  |  |  |
| N_Granule_ID | X | X | X | X |  |  |  |
| N_Granule_Status |  | X | X |  |  |  |  |
| N_Granule_Version | X | X | X | X |  |  |  |
| N_Input_Prod |  | X | X | X |  |  |  |
| N_LEOA_Flag | X | X | X | X | X |  |  |
| N_Levels |  | X | X |  |  |  |  |
| N_LUT_Version |  | X | X |  |  |  | X |
| N_Nadir_Latitude_Max |  | X | X |  |  |  |  |
| N_Nadir_Latitude_Min |  | X | X |  |  |  |  |
| N_Nadir_Longitude_Max |  | X | X |  |  |  |  |

## HDF5 File Product Granule Dataset Metadata ElementsExample CDFCB-X Table [cont.]

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| N_Nadir_Longitude_Min |  | X | X |  |  |  |  |
| N_NPOESS_Document_Ref | X | X | X | X |  |  |  |
| N_Packet_Type | X |  |  |  |  |  |  |
| N_Packet_Type_Count | X |  |  |  |  |  |  |
| N_Percent_Erroneous_Data |  | X | X |  |  |  |  |
| N_Percent_Missing_Data | X | X | X |  |  |  |  |
| N_Percent_Not-Applicable_Data |  |  | X |  |  |  |  |
| N_Reference_ID | X | X | X | X | X | X | X |
| N_Rows |  | X | X | X |  |  |  |
| N_Satellite/Local_Azimuth_Angle_Max |  | X | X |  |  |  |  |
| N_Satellite/Local_Azimuth_Angle_Min |  | X | X |  |  |  |  |
| N_Satellite/Local_Zenith_Angle_Max |  | X | X |  |  |  |  |
| N_Satellite/Local_Zenith_Angle_Min |  | X | X |  |  |  |  |
| N_Solar_Azimuth_Angle_Max |  | X | X |  |  |  |  |
| N_Solar_Azimuth_Angle_Min |  | X | X |  |  |  |  |
| N_Solar_Zenith_Angle_Max |  | X | X |  |  |  |  |

## HDF5 File Product Granule Dataset Metadata ElementsExample CDFCB-X Table

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| N_Solar_Zenith_Angle_Min |  | X | X |  |  |  |  |
| N_Spacecraft_Manuever |  | X | X | X | X |  |  |
| N_Spacecraft_SW_Version |  | X | X |  |  |  |  |
| North_Bounding_Coordinate |  | X | X |  | X |  |  |
| South_Bounding_Coordinate |  | X | X |  | X |  |  |
| West_Bounding_Coordinate |  | X | X |  | X |  |  |
| Time_of_Day |  |  |  |  |  | X | X |

## HDF5 File Spacecraft Diary Group Metadata ElementsExample CDFCB-X Table

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| N_Collection_Short_Name | X |  |  |  |  |  |  |
| N_Dataset_Type_Tag | X |  |  |  |  |  |  |
| N_Packet_Type | X |  |  |  |  |  |  |
| N_Packet_Type_Count  | X |  |  |  |  |  |  |

## HDF5 File Spacecraft Diary Aggregation Metadata ElementsExample CDFCB-X Table

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AggregateBeginningDate  | X |  |  |  |  |  |  |
| AggregateBeginningTime  | X |  |  |  |  |  |  |
| AggregateEndingDate  | X |  |  |  |  |  |  |
| AggregateEndingTime  | X |  |  |  |  |  |  |

## HDF5 File Spacecraft Diary Dataset Metadata ElementsExample CDFCB-X Table

| Field Name | RDR | SDR/TDR | IP/ARP/EDR | GEO | Gridded IP | ANC | AUX |
| --- | --- | --- | --- | --- | --- | --- | --- |
| N_Beginning_Time_IET | X |  |  |  |  |  |  |
| N_Ending_Time_IET | X |  |  |  |  |  |  |

## 3-Dimensional Array Example