Atmosphere Virtual Lab (AVL)
2nd ESA EarthCARE Validation Workshop
Sander Niemeijer, S[&]T25-28 May 2021 (online)
ESA UNCLASSIFIED – For ESA Official Use Only1

Atmosphere Virtual Lab• Continuation and evolution of the ESA Atmospheric Toolbox2

Atmosphere Virtual Lab• Atmospheric Toolbox consists of:• CODA – file reading library/tools supporting large variety of data formats• HARP – library/tools for easy ingestion, data comparison, regridding, etc.
• QDOAS – DOAS retrieval application• VISAN – desktop application for visualisation• See also presentation on ESA Atmospheric Toolbox (ATS) at the first EarthCARE
validation workshop (13 June 2018)https://atmospherictoolbox.org

3

Atmosphere Virtual Lab• Focus in AVL project is on bringing the tools to cloud platforms(e.g. DIAS, Exploitation Platforms)• Most cloud platforms focus on Python and Notebooks for interactive analysis• Also allow running AVL standalone on local computers• Solution will be based on JupyterLab• Use what is already there (numpy, matplotlib, cartopy, plotly, etc.)• Improve/extend python interfaces for CODA/HARP/QDOAS• Improve interactive geographical plots for atmospheric data
(replace VISAN as main visualisation environment)• Project result will be software that can be deployed in exploitation platforms;
AVL is not a cloud service itself
4

Atmosphere Virtual Lab• Besides the Lab environment, the AVL project also covers:• support for new missions such as EarthCARE• Create publicly available Use Cases and Examples• Project only started recently• No support for EarthCARE yet, but coming soon

5

Atmosphere Virtual Lab• Most important component for validation community is HARPRinterfacePythoninterface

Matlabinterface

IDL
interfaceharpconvert

harpdump

harpcollocate

other...

harpmerge

harpcheckother...

other...

HARP C libraryIngestion modules:
Sentinel 5P, OMI/TES/MLS/HIRDLS, GOME-2/IASI,GOMOS/MIPAS/SCIAMACHY, GOSAT, Odin/OSIRIS,
Atmospheric CCI, CAMS, NDACC, ...

CODA C library

Unit
conversion

Derivedvariables

Collocation

Filtering

Regridding

Import/ExportHDF4 library

HDF5 library6

Atmosphere Virtual Lab• HARP is not just software, but also a set of conventions:• simple data structure like netcdf (product with variables; no grouping)• global attributes, variable attributes• similar/complementary to e.g. netCDF-CF, but not fully compatible (CF couplesdimensions to coordinate variables; HARP keeps these fully separate)• variable naming convention•

similar things should be named similarly•

different things should be named differently•

measurements from different instruments or different retrieval algorithms
belong in different files (allows use of same variable name)7Atmosphere Virtual Lab8

Atmosphere Virtual Lab• HARP variable naming convention• Already proper distinction for e.g.:•densities (total column, partial column profile, concentration profile)•number density vs. mass density (also for mixing ratios)•

total air mixing ratios vs. dry air mixing ratios• Variables names are generic, and sometimes differ from domain specific terms.
•e.g. a vertical ‘water path’ would become a ‘column density’.

• Cloud/Aerosol domain still work in progress. Complicated due to waterphases (vapor, liquid, ice), precipitation types, cloud phases, aerosol
types/sizes, etc.
http://stcorp.github.io/harp/doc/html/conventions/index.html
9

Atmosphere Virtual Lab• HARP operations• filtering• derivations• vertical regridding• binning• spatial binning• applying averaging kernels• Operations are provided as a ‘;’ separated string parameter during ingestion of a
file:

10

Atmosphere Virtual Lab
Collocation workflow with HARP:$ harpcollocate -d 'datetime 12 [h]' -d 'point_distance 500 [km]' -ny point_distance-nx datetime .../sat_data_dir .../ref_data_dir collocations.csv$ harpmerge -a 'collocate_left("collocations.csv")' .../sat_data_dir sat_data.nc$ harpmerge -a 'collocate_right("collocations.csv")' .../ref_data_dir ref_data.nc
You can then further manipulate the data (e.g. smoothing, profile -> column, ...)$ harpconvert –a '.....' data.nc data_modified.nc
or:>>> harp.import_product('data.nc', '…..')

11

Atmosphere Virtual Lab• Upcoming support for EarthCARE:• Ingestion support for all L2A and L2B products (main variables only)
http://stcorp.github.io/harp/doc/html/ingestions/index.html• Ingestion support for reference data (e.g. EVDC, GEOMS, cloudnet, …)• Review/adapt HARP conventions for cloud/aerosol• Improve/revise HARP operations for cloud/aerosol domain• Support EarthCARE with AVL visualisation functions• Publish Use Cases / Examples for handling EarthCARE data with AVL/HARP

12

Atmosphere Virtual Lab
HARP is used in many operational services already:13

Atmosphere Virtual Lab
HARP is used in many operational services already:14

Atmosphere Virtual Lab
HARP is used in many operational services already:15

Atmosphere Virtual Lab

https://forum.atmospherictoolbox.org
16