MARS - Webinar

Webinar Starts at(10:00am UK time)

PRESENTER:
Cristian Simarro

Please remember tomute your microphoneand camera 

HOST:Carsten Maass

Slide 1

1

MARS – Webinar
Efficient retrievals and new interpolation software
Cristian Simarro and Carsten Maass
User Support

Slide 2

© ECMWF June 21, 2019MARS Webinar May 2019

Content of this Webinar

MARS Introduction



How to retrieve large datasets efficiently

MIR: the Meteorological Interpolation and Regridding library

Slide 3MARS Webinar May 20193

MARS eLearning modules
https://www.ecmwf.int/en/learning/education-material/elearning-online-resources

Slide 4MARS Webinar May 20194

MARS Introduction

Meteorological Archival and Retrieval System

Meteorological data (GRIB: fields, BUFR, ODB: observations)
Large amount of data (size of archive & number of fields)
Operational & Research environment
Complex batch system
Large number of users with different requirements:
large datasets rarely ↔ few fields very often
Slide 5MARS Webinar May 20195

Introduction – Some figures
Users- +100,000 registered users- ~1,000 daily users

~1.8 million requests per day:
- 300 million retrieved fields (200 Terabytes)- 250 million archived fields (230 Terabytes)



Total volume in tapes 200 PiB in 27,000 tapes- In the last 2 years, ~90 PiB have been deleted

Total volume in disk 5 PiB

More than 380 * 109 meteorological fields

Metadata size: 2.5 Tbytes

Slide 6COM INTRO 2017: MARS Introduction and basic concepts

6

Introduction – MARS components

Client/Server architecture

Clients: workstations, supercomputers, Web API requests …



6 instances of MARS servers

~50 linux CentOS 7.4

Tape management software: HPSS 7.5.3

Slide 7MARS Webinar May 20197

Why do MARS requests need to be efficient?
The resources are shared among many usersThe number of tapes drives is limited

Mounting the same tape many times can damage it

Waiting times in the queues might be longTools to help creating efficient MARS requests:- MARS Catalogue (MARS tree)- List verb

Tool to monitor the activity- MARS Activity

Slide 8MARS Webinar May 2019

LIST,
CLASS
TYPE
STREAM
EXPVERLEVTYPE
PARAM
DATE
TIME
STEP
NUMBEROUTPUTTARGET= OD,= PF,= ENFO,= 0001,= SFC,= 134.128/151.128/165.128,= 20190301,= 0000,= 0/1/2/3/4/5/6/7/8/9,= 1/to/50,= cost,= list2.txt

8

MARS Catalogue – http://apps.ecmwf.int/mars-catalogue/
Content browsing of every field in the archive- more up to date than static content documentation

URL based on MARS requests (can be edited & bookmarked)

Create MARS requests (without checking availability)Check availability of dataCheck estimated size of the retrieval

Retrieval in GRIB and NetCDF for few fields (to so some tests)
Archive license

https://apps.ecmwf.int/archive-catalogue/
Slide 9Public users

http://apps.ecmwf.int/datasets/
MARS Webinar May 20199

Retrieving data - MARS tree

date x time x
level x parameter

date x time x
parameter

date x time x
level x parameter

Slide 10

step x parameter

MARS Webinar May 201910

MARS activity – http://apps.ecmwf.int/mars-activity/
Server activity / MARS queue Show system activity
 Monitor your requests
 Learn how the queuing system works- Reason for queued requests

Slide 11MARS Webinar May 201911

MARS activity – http://apps.ecmwf.int/mars-activity/Slide 12

MARS Webinar May 201912

Retrieving data – Efficiency
Explore data in archive catalogue - collocation

Estimate amount of data (list command)- Number of fields (up to tens of thousands / request)- Data size (up to several Gigabytes / request)Retrieve as much data from the same tape as possible- Avoid constantly accessing the same tapeReduce number of tapes involved (better scheduling)Use local target disk (e.g. $SCRATCH for MS users)
Slide 13MARS Webinar May 201913

MARS Efficient requests use caseSome users want to retrieve large chunks of data
- ENS: Atmospheric (enfo) Perturbed Forecast surface- HRES: Atmospheric (oper) Model Levels forecast- long periods of time

Depending on how a user retrieves the data
- Faster extractions- Performance of the system

Use case example:
Slide 14https://software.ecmwf.int/wiki/display/UDOC/Guidelines+to+write+efficient+MARS+requests

MARS Webinar May 201914

Questions about MARS efficiency
Contact:ServiceDesk@ecmwf.int

Slide 15

… and answers

MARS Webinar May 201915

MIR: the Meteorological Interpolation and Regridding libraryImplementation plan
- Used in MARS, Metview & product generation (dissemination)

New features

Differences between MARS / MIR and MARS / EMOSLIB- Interpolation method- Parameter classification- Spectral to grid transformation- Sub-areas- Land-sea mask
- Wave data
Slide 16MARS Webinar May 201916

Implementation plan: MARS
Beta testing ofMARS / MIR available
as “mars –m”

MARS / MIR available
as “mars –m”

MARS / EMOSLIB default

MARS/ MIR as default
MARS / EMOSLIB
available as nondefault

We are here
18 October 2017
Availability of beta
version of MARS / MIR
announced to CompReps
6 February 2018
Announced to all users

July 2018
First full release of MIR• MARS / MIR “ready forproduction”• MARS / EMOSLIB still
the default MARSMARS Webinar May 201926 January 2019
• MARS / MIR
becomes thedefault versionof MARS
Slide 17
• Metview uses
MIR by default

December 2019MARS / EMOSLIB
decommissionedDates reflect current
expectations and maychange in light of actual
progress madeOctober 29, 201417

New features

No Land-Sea Mask processingGrid to grid interpolation- Interpolations from

global regular or reduced Gaussian or lat/lon grid to regular or reduced Gaussian or lat/lon
grid is supported

MIR uses an intermediate grid when transforming from Spherical Harmonics to latlon grids- It can be changed using MARS keywords

Faster spectral transform ( 2-3 times )Slide 18

MARS Webinar May 201918

Interpolation methods

MIR uses by default a linear method (based on a triangular mesh)- replaces the EMOSLIB bilinear method (based on 4 points)

MIR computes distance in 3D- nearest neighbour method has improve as a result

Slide 19

MARS Webinar May 201919

Parameter classification
The parameters have been classified into groups with an associated defaultinterpolation methodThese parameters will be interpolated using nearest neighbour:- index parameters: integer values associated to a predefined table- wave spectra and associated parameters: intensity as a function of discrete
wavelengths- vegetation cover ratio parameters

Slide 20

MARS Webinar May 201920

Spectral to grid point transformations (1/3)
The default transformation with MIR is via an intermediate regular Gaussian grid.

The post-processing keyword RESOL is replaced by two new MARS keywords- TRUNCATIONDescribes how to treat the incoming SH, before the transformation to grid points

The default TRUNCATION=AUTO is derived from the intermediate grid

- INTGRIDDescribes the intermediate grid to which the transform is performedThe default INTGRID=AUTO uses an intermediate regular Gaussian grid, derived from theoutput GRID

Spherical Harmonics

MARS Webinar May 2019

Slide 21

Reduced Gaussian grid O3221Spectral to grid point transformations (2/3)
Transformation from T1279 to regular lat/lon 1/1

Automatic truncation: via intermediate F90 grid (with an automatic truncation to T179)
This is the default behaviour- TRUNCATION=AUTO, INTGRID=AUTO, GRID=1/1

User specified truncation: via intermediate F90 grid (with a user-specified truncation toT179)- TRUNCATION=179, INTGRID=F90, GRID=1/1

No truncation of the SH input field: no truncation of the source, and via intermediate gridbased on the source (T1279 corresponds to O1280)- TRUNCATION=NONE, INTGRID=SOURCE, GRID=1/1
Slide 22MARS Webinar May 201922

Spectral to grid point transformations (3/3)
MIR
TRUNCATION=AUTOEMOSLIBINTGRID=AUTO

TRUNCATION=NONE,INTGRID=SOURCETRUNCATION=179

INTGRID=F90

Slide 23

MARS Webinar May 201923

SubareasCropping subareas and frames directly from global reduced Gaussian grids,
including the octahedral reduced Gaussian grid, is now supported
MIR can work with staggered grids (no point at latitude=0, longitude=0)

AREA=89.5/0.5/-89.5/359.5,
GRID=1/1
MARS/MIR: honours the user defined area and returns theinner grid
MARS/EMOSLIB: expands the area to fit in the original grid
Slide 24

MARS Webinar May 201924

Land-Sea masks

By default no Land-Sea Mask processing- LSM processing for both input and output grids can be requested with the new LSM keyword

LSM = ON

- As usual, LSM can be set to value OFF in subsequent requests- Land-Sea Mask in MIR is based on a much finer 1 km resolution fieldWhen interpolating the Land-Sea Mask field (paramId= 172) to a regular lat/longrid, MARS/MIR returns fractional values [0,1]lsm ≥ 0.5 : land point
lsm < 0.5 : sea point

Slide 25

MARS Webinar May 201925

Wave dataInterpolation from limited-area wave data on a regular lat-lon grid is now supportedNearest neighbour points might be different compared to EMOSLIB

Different treatment of neighbouring missing values produces a smootherinterpolation close to coasts
all "Spectra-related" parameters have been classified and are interpolated in thesame way as the 2D spectra, using a nearest neighbour in MIR instead ofEMOSLIB's bilinear method

Slide 26MARS Webinar May 201926

Differences in parameter values between EMOSLIB and MIR
MIR has undergone a thorough validation process and tests to ensure its quality andcorrectness. You may see some differences when using MARS/MIR compared toMARS/EMOSLIB
Possible reasons for differences in interpolated fields between MIR and EMOSLIB:- land sea-mask processing- default interpolation method- selection of nearest points- handling of missing values
For full details see:Slide 27https://software.ecmwf.int/wiki/display/UDOC/MARS+interpolation+with+MIR

MARS Webinar May 201927

Examples

Slide 28MARS Webinar May 201928

MIR article

Slide 29MARS Webinar May 2019https://doi.org/10.1016/j.cpc.2017.07.006

29

Additional resources

MARS documentation- https://confluence.ecmwf.int/display/UDOC/MARS+user+documentation



MARS Web Applications- http://apps.ecmwf.int/mars-catalogue/- https://apps.ecmwf.int/mars-activity/

Guidelines to write efficient MARS requests- https://software.ecmwf.int/wiki/display/UDOC/Guidelines+to+write+efficient+MARS+requests

eLearning- https://www.ecmwf.int/en/learning/education-material/elearning-online-resources

MARS interpolation with MIR- https://software.ecmwf.int/wiki/display/UDOC/MARS+interpolation+with+MIR

Newsletter article about MIR

Slide 30

- https://www.ecmwf.int/sites/default/files/elibrary/2017/17439-newsletter-no-152-summer-2017.pdf
MARS Webinar May 201930

Questions …
Contact:
ServiceDesk@ecmwf.int

Slide 31

… and answers

MARS Webinar May 201931