ecCodes: Advanced Topics Part II
Shahram NajmDevelopment Section
Forecast Department
Slide 1COM ecCodes, ECMWF 2016 © ECMWF

Slide 1

Overview Parameter database
 GRIB1 to GRIB2 conversion

 Local configuration

Slide 2COM ecCodes, ECMWF 2016 © ECMWF

Slide 2

GRIB 1
“10 metre U component of wind”
indicatorOfParameter = 33 [u-component of wind (m/s)]
table2Version = 3
indicatorOfTypeOfLevel = 105 [Specified height level above ground (m)]level = 10Slide 3

COM ecCodes, ECMWF 2016 © ECMWF

Slide 3

GRIB 2
“10 metre U component of wind”
discipline = 0 [Meteorological products]
parameterCategory = 2 [Momentum]
parameterNumber = 0 [u component of wind (m s-1) ]typeOfFirstFixedSurface = 103 [Specified height level above ground (m)]scaleFactorOfFirstFixedSurface = 0
scaledValueOfFirstFixedSurface = 10
typeOfSecondFixedSurface = 255 [Missing ]scaleFactorOfSecondFixedSurface = MISSING

scaledValueOfSecondFixedSurface = MISSING
Slide 4COM ecCodes, ECMWF 2016 © ECMWF

Slide 4

GRIB 1 ECMWF local coding“10 metre U component of wind”

indicatorOfParameter = 165 [10 metre u-component of wind (m/s)]
table2Version = 128

Slide 5COM ecCodes, ECMWF 2016 © ECMWF

Slide 5

ecCodes solution
 Use a VIRTUAL (computed) key to decouple
user level from coding level User code gets/sets a virtual key and ecCodes
gets/sets the appropriate coded keys
 Local configuration is available to deal withlocal codes

Slide 6COM ecCodes, ECMWF 2016 © ECMWF

Slide 6

ecCodes parameter ecCodes provides some edition independent keys toidentify a parameter : paramId
 shortName name units centreSlide 7

COM ecCodes, ECMWF 2016 © ECMWF

Slide 7

ecCodes parameters(GRIB 1)

 centre table2Version
 indicatorOfParameter levelType level…

Slide 8COM ecCodes, ECMWF 2016 © ECMWF

Slide 8

ecCodes parameters(GRIB 2)

 discipline parameterCategory
 parameterNumber typeOfFirstFixedSurface
 scaleFactorOfFirstFixedSurface
 scaledValueOfFirstFixedSurface
 typeOfSecondFixedSurface
 scaleFactorOfSecondFixedSurface
 scaledValueOfSecondFixedSurface
 productDefinitionTemplateNumber
Slide 9
…
COM ecCodes, ECMWF 2016 © ECMWF

Slide 9ecCodes - Design
Client Software(C, FORTRAN, Python …)

Command line tools

set and get
key / values

GRIB 1GRIB 2

ecCodesDefinition
Files

BUFR 3 BUFR 4

Definition files are external text files describing
the decoding rules
New decoding rules can beSlide
added
without
10
rebuilding the ecCodes

COM ecCodes, ECMWF 2016 © ECMWF

Slide 10

Concept: shortName
GRIB 2‘2t’ = { discipline = 0 ; productDefinitionTemplateNumber = 8 ;parameterCategory = 0 ; parameterNumber = 0 ; typeOfFirstFixedSurface =103 ; scaleFactorOfFirstFixedSurface = 0; scaledValueOfFirstFixedSurface =2; typeOfSecondFixedSurface = 255 ; scaleFactorOfSecondFixedSurface =missing(); scaledValueOfSecondFixedSurface = missing(); }GRIB1‘2t’ = { indicatorOfParameter=11; table2Version=3;
levelType=103; level=2;}

GRIB1 ECMWF local coding
Slide 11‘2t’ = { indicatorOfParameter=167; table2Version=128;}COM ecCodes, ECMWF 2016 © ECMWF

Slide 11

Concept: paramId
GRIB 2167 = { discipline = 0 ; productDefinitionTemplateNumber = 8 ;parameterCategory = 0 ; parameterNumber = 0 ; typeOfFirstFixedSurface =103 ; scaleFactorOfFirstFixedSurface = 0; scaledValueOfFirstFixedSurface =2; typeOfSecondFixedSurface = 255 ; scaleFactorOfSecondFixedSurface =missing(); scaledValueOfSecondFixedSurface = missing(); }GRIB1
167 = { indicatorOfParameter=11; table2Version=3;
levelType=103; level=2;}

GRIB1 ECMWF local coding
Slide 12167 = { indicatorOfParameter=167; table2Version=128;}COM ecCodes, ECMWF 2016 © ECMWF

Slide 12

GRIB Parameters in ecCodes
 The Parameters database is accessible here:

http://apps.ecmwf.int/codes/grib/param-db/

Slide 13

COM ecCodes, ECMWF 2016 © ECMWF

Slide 13

Parameters database
WEBParameters
database

ecCodes

Slide 14COM ecCodes, ECMWF 2016 © ECMWF

Slide 14

Parameters: Practicals
To get the practicals:
tar xvf ~trx/ecCodes/grib_parameters.tar
1. You have two grib messages start.grib1 and start.grib2
2. Create the file 10u.grib1 setting shortName=10u instart.grib1
3. Create the file 10u.grib2 setting shortName=10u instart.grib2
4. Do grib_ls –n parameter 10u.grib1 10u.grib2 . Do
you see any difference
5. Compare the grib_dump –O of the two files and of thetwo messages in each file
Slide 15

COM ecCodes, ECMWF 2016 © ECMWF

Slide 15

GRIB1 to GRIB2 conversion
grib_set –s edition=2 in.grib1 out.grib2

conversion of time geography
 vertical parameter local dataCOM ecCodes, ECMWF 2016 © ECMWF

Slide 16

Slide 16

GRIB1 to GRIB2 conversion
 Parameter conversion is particularly complex due to thedifference between the two coding standards and thelocal tables used by some meteorological centres
 The conversion is based on the parameter’s unique
identifier “paramId”

Slide 17

COM ecCodes, ECMWF 2016 © ECMWF

Slide 17

paramId based conversion
 How to produce a GRIB for a “2 metre temperature”
grib_set –s paramId=165 in.grib1 out.grib1

grib_set –s paramId=165 in.grib2 out.grib2

 How to convert a GRIB1 to GRIB2
grib_set –s edition=2 in.grib1 out.grib2

 During the conversion to edition 2 ecCodes copies theparamId value from the GRIB1 to the GRIB2:
1. get paramId(=165) from GRIB1
2. change edition to 2 producing a GRIB2Slide 18

3. set paramId(=165) in GRIB2
COM ecCodes, ECMWF 2016 © ECMWF

Slide 18

paramId based conversion
 The conversion is possible only if a paramId isdefined for both editions
 Check on the parameters database website if aconversion is possible

Slide 19

COM ecCodes, ECMWF 2016 © ECMWF

Slide 19

Parameters: Practicals
We refer to the same files produced in theprevious practical
1. Convert 10u.grib1 to its GRIB2 version
10u_converted.grib2.
2. Do grib_ls -n parameter 10u.grib2
3. Do grib_ls –n parameter 10u_converted.grib2
4. Take the first message from start.grib1 and save it toecmf.grib1
5. Set the paramId of ecmf.grib1 to 162089. Save it asecmf.162089.grib1

6. Convert ecmf.162089.grib1 to GRIB edition 2. Why
Slide 20
does it fail?
COM ecCodes, ECMWF 2016 © ECMWF

Slide 20

Local configuration
 The external text files defining the decoding rules usedby the decoding engine are called definition files

 For each installation there is a default set of definition
files
 The ECCODES_DEFINITION_PATH environment variable
can be set to use local definition files instead of thedefinition files provided within the distribution

Slide 21COM ecCodes, ECMWF 2016 © ECMWF

Slide 21

Local configuration
 For GRIB, the parameter descriptions for a given“centre” are contained in the files shortName.def,
paramId.def, units.def, name.def in the directories
BASE_DIR/definitions/grib1/localConcepts/[centre:s]
BASE_DIR/definitions/grib2/localConcepts/[centre:s]
Note: ‘centre:s’ means the centre as a string e.g. ecmf, kwbc, cnmc etc

 The general parameter descriptions are contained in
the files shortName.def, paramId.def, units.def,
name.def in the directories
BASE_DIR/definitions/grib1
BASE_DIR/definitions/grib2
Slide 22COM ecCodes, ECMWF 2016 © ECMWF

Slide 22

Local configuration

ECCODES_DEFINITION_PATH=/my/definitions:/eccodes/definitions

 The library searches for each required definition file first in/my/definitions and then in /eccodes/definitions

 If the file is found in /my/definitions then it used by thedecoding engine
 The user can override all the definition files with his/herown definition files
 We suggest you only override the definition files
containing the parameter information
Slide 23COM ecCodes, ECMWF 2016 © ECMWF

Slide 23

Local configuration: defining a GRIB
parameter locally Get the directory of the definition files with the utility
codes_info set the environment variable
ECCODES_DEFINITION_PATH=local_dir:default_definition_dir

 Create the directories:
local_dir/grib1/localConcepts/[centre:s]
local_dir/grib2/localConcepts/[centre:s]
And add files shortName.def, paramId.def, name.def &units.def.
Slide 24COM ecCodes, ECMWF 2016 © ECMWF

Slide 24

Local configuration: defining aparameter locally Example from paramId.def (for GRIB1)#Direction of wind waves'500072' = {table2Version = 112;
indicatorOfParameter = 101;}

Example from shortName.def (for GRIB1)#Total precipitation of at least 10 mm'tpg10' = {table2Version = 131;
indicatorOfParameter = 62;}
Slide 25

COM ecCodes, ECMWF 2016 © ECMWF

Slide 25

Local configuration: GRIB PracticalGet the practicals: tar xvf ~trx/ecCodes/localConfig.tar
1. What parameter is contained within the two files x.grib1 andx.grib2?
2. Run codes_info to find the location of the default definitions
3. Now set ECCODES_DEFINITION_PATH to include the“mydefs” directory e.g.
exportECCODES_DEFINITION_PATH=`pwd`/mydefs:/path/to/defaults

4. Now see if ecCodes recognizes the name, units etc
5. Test the GRIB1 to GRIB2 conversion. Compare the output
with the provided x.grib2 file
6. Study the directory structure of “mydefs”
Slide 26 and its local
concept files
COM ecCodes, ECMWF 2016 © ECMWF

Slide 26

Local configuration: BUFR local tables
 Warning: The use of local tables in messages intended for
non-local or international exchange is strongly discouraged

 The local BUFR tables are stored in the following
definitions directories:
definitions/bufr/tables/[masterTableNumber]/
local/[localTablesVersionNumber]/[bufrHeaderCentre]/[bufrHeaderSubCentre]
e.g. definitions/bufr/tables/0/local/101/98/0/ (98 is for ecmwf)

 The environment variable ECCODES_DEFINITION_PATH
can be set to locate local BUFR tables. We follow the sameconvention as we saw previously:
Slide 27
ECCODES_DEFINITION_PATH=/path/to/local/defs:/path/to/default/defs

COM ecCodes, ECMWF 2016 © ECMWF

Slide 27

Local configuration: BUFR PracticalGet the practicals: tar xvf ~trx/ecCodes/localBufrTable.tar
1. Try decoding the BUFR file ikco_217.local.bufr using
bufr_dump. This file uses a local table version(localTablesVersionNumber=66)
2. Run codes_info to find the location of the default definitions
3. Now set ECCODES_DEFINITION_PATH to include theprovided “mydefs” directory e.g.
exportECCODES_DEFINITION_PATH=`pwd`/mydefs:/path/to/defaults

4. Now see if ecCodes can decode the BUFR file
5. Search the output of bufr_dump for the overridden unit
“MyOwnUnits”. This was added as an example of a centre
Slide 28
defining its own parameter attributes

COM ecCodes, ECMWF 2016 © ECMWF

Slide 28

Questions ?

Slide 29

COM ecCodes, ECMWF 2016 © ECMWF

Slide 29