# Presentation: 704637

## GRIB2 conversion and usage at NCEP

- V. KRISHNA KUMAR &
- STEPHEN A GILBERT
- NOAA/NWS/NCEP/Central Operations

- 10th Workshop on Meteorological Operational Systems ECMWF User Orientation
- 14-18 November 2005

- Room 302, WWB
- Camp Springs, MD

## Overview

- GRIB1 and GRIB2
- Format differences between GRIB1 and GRIB2
- Challenges/Issues on GRIB2
- NCEP GRIB2 utilities
- Operational GRIB2 products
- Data Packing – Compression Experiments

## GRIB1 Sections

- 0 – Indicator Section
-     **- ‘GRIB’ (octet 1-4 = 32bits),  length of message (octet 5-7=24bits), edition number -1 (octet 8=8bits)**
**      ****- 1 octet = 1byte = 8bits**
**       ****- octets are numbered 1, 2, 3, etc., Bit positions within octets are from bit 1 to bit 8.**
**       ****- bit 1 = most significant bit &  bit 8 = least significant)**
- 1 – Product Definition Section (PDS)
-     **- Ref time, orig center, grid, parameter, level, valid time ... **
- 2 – Grid Definition Section (GDS) – optional
-     **- Length of sec., sec. no, defn of grid surf and geometry of data values within the surface**
- 3 – Bit-Map Section (BMS) – optional
-     **- Length of sec., sec. no, indication of presence or absence of data at each of the grid points, as applicable example sst **
- 4 – Binary Data Section (BDS)
-     **- Length of sec., sec. no, data values**
- 5 – End Section = ‘7777’

## GRIB2 Sections

- 0 – Indicator Section
**                   ****- ‘GRIB’, length, version, discipline – (16 octets long)**
- 1 – Identification Section
**                   ****- Ref time, orig center, table versions, ...(length of the section is expressed as the first four octets (32 bits)                -the section number in the fifth octet and octets beyond 21 are reserved for future use)**
- 2 – Local Use Section – optional      **==================**
- 3 – Grid Definition Section                 **=======**            |
- 4 – Product Definition Section            **====**    **|                   |**
**                   ****- parameter, level, valid time                                                   |      |                   |                  **
- 5 – Data Representation Section               **|**   **|**           |
**                   ****- packing algorithm used, info needed to unpack                  |      |                  |**
- 6 – Bit Map Section                                  **|**   |           |
- 7 – Data Section                                  **==================**
**                   ****-packed data**
- 8 – End Section = ‘7777’

## GRIB2 – Documentation

- GRIB2 specification from WMO contains template definitions and table entries: _[http://www.wmo.ch/web/www/DPS/grib-2.html](http://www.wmo.ch/web/www/DPS/grib-2.html)_
- Local Tables: _[http://www.nco.ncep.noaa.gov/pmb/docs/grib2](http://www.nco.ncep.noaa.gov/pmb/docs/grib2)__ _

## GRIB1 & GRIB2 Major Differences

- One GRIB2 message can contain multiple fields.                   - **Sequences of GRIB sections 2 to 7, sections 3 to 7 or sections 4 to 7 may be repeated within a single GRIB message.**
**         ****- Once a section is repeated all the following sections must appear in order.**
**         ****- A section remains in effect until redefined.**
**         ****- Could be used for vector fields such as u & v components, or ensemble members**
**         ****- IEEE single precision floating point representation used instead of standard IBM representation. The representation occupies four octets.**
- GDS, PDS, DRS, and DS are all template driven.
-       **Octet assignments within a section are based on the Template specified.**
-  Parameters are defined by three values:
**         ****Discipline (specified in Section 0) **
**         ****Category (PDS) **
**         ****Parameter number (PDS).**

## GRIB1 and GRIB2 Major differences contd.

- Table versions, originating center, initial reference time and grid ID removed from GRIB1 PDS. Most are now in Identification Section.
- All longitudes MUST be in degrees East and be between 0 and 360 degrees. No negatives. No degrees West. No change in latitude definitions –900  to 900.
- GDS and BMS no longer optional. They must be included. Only Section 2 (Local Use) is optional.
- Latitudes, Longitudes are in units of micro-degrees (10 –6 degrees). Includes distance between grid points for Lat/Lon grids (GRIB1 was
-      milli-degrees)
- All other grid point increments are in millimeters (GRIB1 was in meters)
- Floating point values are now stored in IEEE format (GRIB1 was IBM 360/195 format HDS)

## GRIB2 in NCEP operations

- GRIB2 Encoding Routines
**GRIB2 message can contain gridded fields for many parameters on a number of different grids.**
**Start a new GRIB2 message with a call to:**
** ****(1)   call GRIBCREATE – encodes sections 0 and 1 at the beginning of message.**
** ****(2)   call ADDLOCAL – adds a Local Use Section (Section 2) optional**
** ****(3)   call ADDGRID – encodes a grid definition into Section 3. Defines the geometry of the **
**                                      ****data values in the fields that f   follow it.  ADDGRID can be called **
**                                      ****again to change the grid definition describing subsequent data fields.**
** ****(4)   call ADDFIELD – adds each data field which adds Sections 4, 5, 6, and 7 to the message.**
** ****(5)   call GRIBEND – adds the final section 8 to the message which updates the length of **
**                                      ****the message. GRIBEND is required for each GRIB2 message.  **

## GRIB2 in NCEP operations contd.

- GRIB2 Decoding Routines
**call GB_INFO – finds out how many Local Use sections and data fields are contained in a given GRIB2 message. Also returns the number of octets of the largest Local Use section in the message. Ensures the dimensionality of the input array of GETLOCAL **
** ****call GETLOCAL – returns the requested occurrence of Section 2 from a given GRIB2 message.**
** ****call GF_GETFLD – gets all information pertaining to the nth data field in the message – returns all the unpacked values for each Section and Template in a f90 “type gribfield” with options to unpack the Bit-map (if applicable) and the data values or just return the field description information.**
** ****call GF_FREE(gfld)  – frees up memory held by gfld as many elements in derived type gribfield are dynamically allocated when decoded.**
**Example:**
**      ****use grib_mod**
**       ****type(gribfield) ::gfld   **
**          ...**
**          ...**
**         ****call gf_getfld(cgrib,lengrib,1,.true.,.false.,gfld,ierr)**
**         ****print*,’Num of grid points =‘,gfld%ngrdpts**
**         ****print*,’First data value=‘,gfld%fld(1)**
**         ****call gf_free(gfld)**
** ****How to extract GRIB2 fields from a GRIB2 file?**
**GETGB2 – extracts a specified field from a file containing many GRIB2 messages.**

## GRIB2 in NCEP operations contd.

- Create GRIB2 Message
**call PUTGB2 – if a variable of derived type of “gribfield” already exists, this subroutine can be used to pack it up and write it to a file. **
- NCEP GRIB2 Utilities
**   ****/grb2index grib2file grib2indexfile**
**   ****- creates a GRIB2 index file **
** ****/cnvgrib –h         (packing options:  1 => 2,   2=>1,   2=>2)**
**   ****-converts all fields in a file between GRIB1 and GRIB2**
** ****/degrib2 grib2file**
**  ****-dumps the meta-data for each GRIB2 field in grib2file**
** ****- basically prints contents of each element in derived type gribfield**
** ****- wgrib2 for Grads in development**

## GRIB2 in NCEP Operations

- So far we have only offered new products in GRIB2 format. We are starting planning for transition from GRIB1 to GRIB2.
- To ease users’ transition to GRIB2, we have conversion utility that can be used for existing applications that read only GRIB1.

## GRIB2 Section 7 – Data Section

- Data coded with the minimum number of bits necessary to provide the accuracy.
- Simple scaling – data multiplied by  an appropriate  power of 10 before forming the non-negative differences, and then using the binary scaling to select the precision of the transmitted value.
- Data coded in the form of non-negative scaled differences from a reference value of the whole field plus, if applicable, a local reference value. (reference value is normally the minimum value of the data set which is represented).

## GRIB2 Section 7 – Data Section contd.

- Complex packing for grid-point values – split the whole set of scaled data values into groups on which local references (local minima) are removed – pre-processing may be needed on the scaled data.
- Complex packing for spectral data.
- The original data value Y can be recovered with the formula
-       Y*10D=R+(X1+X2)*2E
-       where E=Binary scale factor, D=Decimal scale factor, R=Reference value of the whole field, X1=0, X2=Scaled (encoded) value.

## GRIB2 – Packing Methods

- GRIB2_com – Group packing – DR Template 5.2
- GRIB2_1od – Group packing –with 1st order differencing Template 5.3
- GRIB2_2od – Group packing – with 2nd order differencing Template 5.3
-                    Image compression algorithms within the GRIB2 standard.
- Both treat grid point data as a grey scale image (single component)
- JPEG_2000 (Part-1 standard) (ISO/IEC 15444-1)
-        _[http://www.jpeg.org/JPEG2000.html](http://www.jpeg.org/JPEG2000.html)_
- PNG  (Portable Network Graphics image compression format)
-        http://www.libpng.org/pub/png/.

## Some GRIB2 packing results 

- Packing results are sensitive to the grouping algorithms one chooses. More sophisticated grouping algorithms may result in better overall compression. The following results are based on a small constant group size.
- Since then a more sophisticated grouping algorithm (from Dr. Glahn, Meteorological Developmental Lab) has been incorporated resulting in better compression.

## Slide 16

## Slide 17

## Slide 18

## Slide 19

## Slide 20

## Slide 21

## Slide 22

## Slide 23

## Slide 24

## Slide 25

## Slide 26

## Slide 27

## Slide 28

## Slide 29

## Slide 30

## Slide 31

## Slide 32

## Slide 33

## NCEP conversion to GRIB2

- NCEP is currently making available high resolution GFS and NAM output in GRIB2 format on its FTP servers and on NOAAPORT
- These include:
- GFS 0.5 degree global grid (F00-F180)
- NAM 12 km CONUS grid (F00-F84)
- NAM 12 km AK grid (F00-F84)

## NCEP conversion to GRIB2

- Transition to GRIB2 is ongoing
- All new products added to NOAAPORT & FTP servers since July 2004 have been in GRIB2 format.
- Tentatively planning to make GRIB2 data sets available for all existing GRIB1 data sets on FTP servers by April 2006.
- The GRIB1 data sets will then be phased out over the following 6-12 month period.