Package: lisflood-utilities
Lisflood UtilitiesThis repository hosts source code of LISFLOOD utilities.
Go toLisflood OS pagefor more information.Other useful resourcesProjectDocumentationSource codeLisfloodModel docshttps://github.com/ec-jrc/lisflood-codeUser guideLisvapDocshttps://github.com/ec-jrc/lisflood-lisvapCalibration toolDocshttps://github.com/ec-jrc/lisflood-calibrationLisflood Utilitieshttps://github.com/ec-jrc/lisflood-utilities(this repository)Lisflood Usecaseshttps://github.com/ec-jrc/lisflood-usecasesIntroLisflood Utilities is a set of tools to help LISFLOOD users (or any users of PCRaster/netCDF files)
to execute some mundane tasks that are necessary to operate lisflood.
Here's a list of utilities you can find in lisflood-utilities package.pcr2ncis a tool to convert PCRaster maps to netCDF4 files.convert a single map into a NetCDF4 fileconvert a time series of maps into a NetCDF4 mapstacksupport for WGS84 and ETRS89 (LAEA) reference systemsfine tuning of output files (compression, significant digits, etc.)nc2pcris a tool to convert a netCDF file into PCRaster maps.convert 2D variables in single PCRaster mapsnetCDF4 mapstacks are not supported yetcutmapsis a tool to cut netcdf files in order to reduce size, using eithera bounding box of coordinatesa bounding box of matrix indicesan existing boolean area maska list of stations and a LDD (in netCDF or PCRaster format)Note: PCRaster must be installed in the conda envthresholdsis a tool to compute the discharge return period thresholds from netCDF4 file containing a discharge time series.compareis a package containing a set of simple Python classes that helps to compare
netCDF, PCRaster and TSS files.water-demand-historicis a package allowing to generate sectoral (domestic, livestock, industry, and thermoelectric) water demand maps with monthly to yearly temporal steps for a range of past years, at the users’ defined spatial resolution and geographical extent. These maps are required by the LISFLOOD OSwater use modulewaterregionsis a package containing two scripts that allow to create and verify a water regions map, respectively.The package contains convenient classes for reading/writing:PCRasterMapPCRasterReaderNetCDFMapNetCDFWriterInstallationRequisitesThe easy way is to use conda environment as they incapsulate C dependencies as well, so you wouldn't need to install libraries.Otherwise, ensure you have properly installed the following software:Python 3.5+GDAL C library and softwarenetCDF4 C libraryInstallIf you use conda, create a new env (or use an existing one) and install gdal and lisflood-utilities:condacreate--namemyenvpython=3.7-cconda-forge
condaactivatemyenv
condainstall-cconda-forgepcrastergdal
pipinstalllisflood-utilitiesIf you don't use conda but a straight python3 virtualenv:source/path/myenv/bin/activate
pipinstalllisflood-utilitiesIf GDAL library fails to install, ensure to install the same package version of the
C library you have on your system. You may also need to setup paths to gdal headers.To check which version of GDAL libraries you have installed on your computer, use gdal-configsudoapt-getinstalllibgdal-devlibgdalexportCPLUS_INCLUDE_PATH=/usr/include/gdalexportC_INCLUDE_PATH=/usr/include/gdal
gdal-config--version# 3.0.1pipinstallGDAL==3.0.1Note: if you previously installed an older version of the lisflood-utilitiies, it is highly recommended to remove it before installing the newest version:pipuninstalllisflood-utilities
pipinstall-e./pcr2ncUsageNote:This guide assumes you have installed the program with pip tool.
If you cloned the source code instead, just substitute the executablepcr2ncwithpython pcr2nc_script.pythat is in the root folder of the cloned project.The tool takes three command line input arguments:-i, --input: It can be a path to a single file, a folder or a unix-like widlcard expression like/path/to/files/dis00*-o, --output_file: Path to the output nc file-m, --metadata: Path to a yaml or json file containing configuration for the NetCDF4 output file.Unless the input is a single file, the resulting NetCDF4 file will be a mapstack according to a time dimension.Input as a folder containing PCRaster maps. In this case, the folder must contain ONLY PCraster files and the output will be a mapstack.pcr2nc-i/path/to/input/-o/path/to/output/out.nc-m./nc_metadata.yamlInput as a path to a single map. In this case, the output won't be a mapstack.pcr2nc-i/path/to/input/pcr.map-o/path/to/output/out.nc-m./nc_metadata.yamlInput as aUnix style pathname pattern expansion. The output will be a mapstack.Note that in this case the input argument must be contained in double quotes!pcr2nc-i"/path/to/input/pcr00*"-o/path/to/output/out.nc-m./nc_metadata.jsonWriting metadata configuration fileFormat of resulting NetCDF4 file is configured into a metadata configuration file. This file can be written in YAML or JSON format.An example of a metadata configuration file is the followingvariable:shortname:disdescription:Dischargelongname:dischargeunits:m3/scompression:9least_significant_digit:2source:JRC Space, Security, Migrationreference:JRC Space, Security, Migrationgeographical:datum:WGS84variable_x_name:lonvariable_y_name:lattime:calendar:proleptic_gregorianunits:days since 1996-01-01Variable sectionInvariablesection you can configure metadata for the main variable:shortname: A short name for the variablelongname: The long name versiondescription: A description for humansunits: The units of the variablecompression: Optional, integer number between 1 and 9, default 0 (no compression). If present the output nc file will be compressed at this level.least_significant_digit: Optional, integer number, default 2. From NetCDF4 documentation:If your data only has a certain number of digits of precision
(say for example, it is temperature data that was measured with a precision
of 0.1 degrees), you can dramatically improve zlib compression by quantizing
(or truncating) the data using the least_significant_digit keyword argument
to createVariable. The least significant digit is the power of ten of the
smallest decimal place in the data that is a reliable value.
For example if the data has a precision of 0.1,
then setting least_significant_digit=1 will cause data the data to be
quantized usingnumpy.around(scale*data)/scale, wherescale = 2**bits,
and bits is determined so that a precision of 0.1 is retained
(in this case bits=4). Effectively, this makes the compression 'lossy'
instead of 'lossless', that is some precision in the data is sacrificed for the sake of disk space.Source and referencesourceandreferenceadd information for the institution that is providing the NetCDF4 file.Geographical sectionIn thegeographicalsection you can configuredatumand name of the x and y variables. Asvariable_x_nameandvariable_y_nameyou should use 'lon' and 'lat' for geographical coordinates (e.g. WGS84) and 'x' and 'y' for projected coordinates (e.g. ETRS89).Currently, pcr2nc supports the following list fordatum:WGS84ETRS89GISCOTime sectionThis section is optional and is only required if the output file is a mapstack (a timeseries of georeferenced 2D arrays)
In this section you have to configureunitsandcalendar.units: Can be one of the following strings (replacing placeholders with the actual date):hours since YYYY-MM-DD HH:MM:SSdays since YYYY-MM-DDcalendar: A recognized calendar identifier, likeproleptic_gregorian,gregorianetc.nc2pcrThis tool converts single maps netCDF (time dimension is not supported yet) to PCRaster format.Usagenc2pcr-i/path/to/input/file.nc-o/path/to/output/out.map[-c/path/to/clone.mapoptional]If input file is a LDD map, you must add the-lflag:nc2pcr-i/path/to/input/ldd.nc-o/path/to/output/ldd.map-l[-c/path/to/clone.mapoptional]Cutmaps: a NetCDF files cookie-cutterThis tool cut netcdf files, using a mask, a bounding box or a list of stations along with a LDD map.Usage:The tool accepts as input:a mask map (either PCRaster or netCDF format) using the -m argument oralternatively, using the -i argument, matrix indices in the formimin imax jmin jmax(imin, imax, jmin, jmax  must be integer numbers)alternatively, using the -c argument, coordinates bounding box in the formxmin xmax ymin ymax(xmin, xmax, ymin, ymax can be integer or floating point numbers; x = longitude, y = latitude)alternatively, using the -N and -l arguments, list of stations with coordinates and a LDD map.a path to a folder containing netCDF files to cut or a static dataset path like LISFLOOD static files.a path to a folder where to write cut files.The following command will cut all netcdf files inside/workarea/Madeira/lai/folder
and produced files will be writte in current folder.
The cookie-cutter that will be used is/workarea/Madeira/maps/MaskMap/Bacia_madeira.nc.
This file is a mask (boolean map with 1 only in the area of interest) where cutmaps takes the bounding box from.
The mask can also be in PCRaster format.cutmaps-m/workarea/Madeira/maps/MaskMap/Bacia_madeira.nc-f/workarea/Madeira/lai/-o./Indices can also be passed as an argument (using -i argument instead of -m). Knowing your area of interest from your netCDF files,
you can determine indices of the array and you can pass in the formimin imax jmin jmax(imin, imax, jmin, jmax  must be integer numbers).cutmaps-i"150 350 80 180"-f/workarea/Madeira/lai/-o./Example with coordinates (using -c argument)xmin xmax ymin ymax(xmin, xmax, ymin, ymax can be integer or floating point numbers; x = longitude, y = latitude) and path to EFAS/GloFAS static data (-S option), with -W to allow overwriting existing files in output directory:cutmaps-S/home/projects/lisflood-eu-c"4078546.12 4463723.85 811206.57 1587655.50"-o/Work/Tunisia/cutmaps-WExample with stations.txt and LDDGiven a LDD map and a list of stations in a text file, each row having coordinates X/Y or lon/lat and an index, separated by tabs:4297500	1572500 1
4292500	1557500 2
4237500	1537500 3
4312500	1482500 4
4187500	1492500 5cutmaps-S/home/projects/lisflood-eu-lldd.map-Nstations.txt-o/Work/Tunisia/cutmapsIf ldd is in netCDF format, LDD will be converted to PCRaster format, first.cutmaps-S/home/projects/lisflood-eu-lldd.nc-Nstations.txt-o/Work/Tunisia/cutmapsIf you experience problems, you can try to pass a path to a PCRaster clone map.cutmaps-S/home/projects/lisflood-eu-lldd.nc-Carea.map-Nstations.txt-o/Work/Tunisia/cutmapsYou will find the produced mask.map and mask.nc for your area in the same folder of ldd map; you will need it for lisflood/lisvap executions.
You will also have outlets.map/outlets.nc based on stations.txt, which let you produce gauges TSS if configured in LISFLOOD.compare utilityThis tool let you compare two netcdf datasets. You can configure it with tolerances (atol, rtol, thresholds for percentage of tolerated different values).
You can also set the option to write diff files, so that you can inspect maps and differences with a tool like Panoplyusage: compare [-h] -a DATASET_A -b DATASET_B -m MASKAREA [-M SUBMASKAREA]
               [-e] [-s] [-D] [-r RTOL] [-t ATOL] [-p MAX_DIFF_PERCENTAGE]
               [-l MAX_LARGEDIFF_PERCENTAGE]

Compare netCDF outputs: 0.12.12

optional arguments:
  -h, --help            show this help message and exit
  -a DATASET_A, --dataset_a DATASET_A
                        path to dataset version A
  -b DATASET_B, --dataset_b DATASET_B
                        path to dataset version B
  -m MASKAREA, --maskarea MASKAREA
                        path to mask
  -e, --array-equal     flag to compare files to be identical
  -s, --skip-missing    flag to skip missing files in comparison
  -D, --save-diffs      flag to save diffs in netcdf files for visual
                        comparisons. Files are saved in ./diffs folder of
                        current directory.For each file presenting
                        differences, you will find files diffs, original A and
                        B (only for timesteps where differences are found).
  -r RTOL, --rtol RTOL  rtol
  -t ATOL, --atol ATOL  atol
  -p MAX_DIFF_PERCENTAGE, --max-diff-percentage MAX_DIFF_PERCENTAGE
                        threshold for diffs percentage
  -l MAX_LARGEDIFF_PERCENTAGE, --max-largediff-percentage MAX_LARGEDIFF_PERCENTAGE
                        threshold for large diffs percentagethresholdsThe thresholds tool computes the discharge return period thresholds using the method of L-moments.
It is used to post-process the discharge from the LISFLOOD long term run.
The resulting thresholds can be used in a flood forecasting system to define the flood warning levels.Usage:The tool takes as input a Netcdf file containing the annual maxima of the discharge signal. LISFLOOD computes time series of discharge values (average value over the selected computational time step). The users are therefore required to compute the annual maxima. As an example, this step can be achieved by using CDO (cdo yearmax), for all the details please refer tohttps://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-190001.2.5The output NetCDF file contains the following return period thresholds [1.5, 2, 5, 10, 20, 50, 100, 200, 500], together with the Gumbel parameters (sigma and mu).usage: thresholds [-h] [-d DISCHARGE] [-o OUTPUT]

Utility to compute the discharge return period thresholds using the method of L-moments.
Thresholds computed: [1.5, 2, 5, 10, 20, 50, 100, 200, 500]

options:
  -h, --help            show this help message and exit
  -d DISCHARGE, --discharge DISCHARGE
                        Input discharge files (annual maxima)
  -o OUTPUT, --output OUTPUT
                        Output thresholds filewater-demand-historicThis utility allows to create water demand maps at the desired resolution and for the desired geographical areas. The maps indicate, for each pixel, the time-varying water demand map to supply for domestic, livestock, industrial, and thermoelectric water consumption. The temporal discretization is monthly for domestic and energy demand, yearly for industrial and livestock demand. The maps of sectoral water demand are required by the LISFLOOD OSwater use module. Clearly, the sectoral water demand maps and the scripts of this utility can be used also for other applications, as well as for stand-alone analysis of historical water demand for anthropogenic use.InputThe creation of the sectoral water demand maps requires a template map that defines the desired geographical area and spatial resolution. The generation of the maps relies on a number of external datasets (examples are theGlobal Human Settlement - Datasets - European Commission (europa.eu)andFAO AQUASTAT Dissemination System). The locations of the template map, of the input datasets and files, of the output folder, and other users’ choices (e.g. start year and end year) are specified in a configuration file. The syntax of the configuration file is pre-defined and an example is provided to the users. The complete list of external datasets, the instructions on how to prepare (i) the external dataset, (ii) the template map, (iii) the input folder, (iv) the output folder, and (v) the configuration file are explained into detailshereOutputFour sectoral water demand maps in netCDF-4 format. The geographical extent and the spatial resolution are defined by the template map (users-defined input file). Each netCDF-4 file has 12 months, for each year included in the temporal time span identified by the user. Sectoral water demand data with lower (yearly) temporal resolution are repeated 12 times.UsageThe methodology includes five main steps. The instructions on how to retrieve the scrips, create the environment including all the required packages, and use the utility are providedhereImportant notes on documentation and data availabilityThe complete list of external datasets, the instructions on how to retrieve the external datasets, the methodology, and the usage of the scripts are explained into detailshere. The README file provides detailed technical information about the input datasets and the usage of this utility. The methodology is explained in the manuscript: Choulga, M., Moschini, F., Mazzetti, C., Grimaldi, S., Disperati, J., Beck, H., Salamon, P., and Prudhomme, C.: Technical note: Surface fields for global environmental modelling, EGUsphere, 2023 (preprint).The global sectoral water demand maps at 3 arcmin (or 0.05 degrees) resolution, 1979-2019, produced using the scripts of this utility can be downloaded fromJoint Research Centre Data Catalogue - LISFLOOD static and parameter maps for GloFAS - European Commission (europa.eu)waterregionsThe modelling of water abstraction for domestic, industrial, energetic, agricoltural and livestock use   can require a map of the water regions. The concept of water regions and information for their definition are explainedhere.
Since groundwater and surface water resources demand and abstraction are spatially distributed inside each water region, each model set-up must include all the pixels of the water region. This requirement is crucial for the succes of the calibration of the model. This utility allows the user to meet this requirement.
More specifically, this utility can be used to:create a water region map which is consistent with a set of calibration points: this purpose is achieved by using the script define_waterregions.verify the consistency between an existing water region map and an exixting map of calibration catchments: this purpose is achieved by using the script verify_waterregions
It is here reminded that when calibrating a catchment which is a subset of a larger computational domain, and the option wateruse is switched on, then the option groudwatersmooth must be switched off. The explanation of this requirement is provided in the chapterWater useof the LISFLOOD documentation.Requirementspython3, pcraster 4.3. The protocol was tested on Linux.define_waterregionsThis utility allows to create a  water region map which is consistent with a set of calibration points. The protocol was created by Ad De Roo (Unit D2, Joint Research Centre).InputList of the coordinates of the calibration points. This list must be provided in a .txt file with three columns: LONGITUDE(or x), LATITUDE(or y), point ID.LDD map can be in netcdf format or pcraster format. When using pcraster format, the following condition must be satisfied:PCRASTER_VALUESCALE=VS_LDD.Countries map in netcdf format or pcraster format. When using pcraster format, the following condition must be satisfied:PCRASTER_VALUESCALE=VS_NOMINAL. This map shows the political boundaries of the Countries, each Coutry is identified by using a unique ID. This map is used to ensure that the water regions are not split accross different Countries.Map of the initial definition of the water regions in netcdf format or pcraster format. When using pcraster format, the following condition must be satisfied:PCRASTER_VALUESCALE=VS_NOMINAL. This map is used to attribute a water region to areas not included in the calibration catchments. In order to create this map, the user can follow the guidelines providedhere.file.yamlor.jsonto define the metadata of the output water regions map in netcdf format. An example of the structure of these files is providedhereInput data provided by this utility:This utility provides three maps ofCountries IDs: 1arcmin map of Europe (EFAS computational domain), 0.1 degree and 3arcmin maps of the Globe . ACKNOWLEDGEMENTS: both the rasters were retrieved by upsampling the original of the World Borders Datase provided byhttp://thematicmapping.org/(the dataset is available under a Creative Commons Attribution-Share Alike License).OutputMap of the water regions which is consistent with the calibration catchments. In other words, each water region is entirely included in one calibration catchment.  The test to check the consistency between the newly created water regions map and the calibration catchments is implemented internally by the code and the outcome of the test is printed on the screen.
In the output map, each water region is identified by a unique ID. The format of the output map can be netcdf or pcraster.UsageThe following command lines allow to produce a water region map which is consistent with the calibration points (only one commad line is required: each one of the command lines below shows a different combination of input files format):python define_waterregions.py -p calib_points_test.txt -l ldd_test.map -C countries_id_test.map -w waterregions_initial_test.map -o my_new_waterregions.mappython define_waterregions.py -p calib_points_test.txt -l ldd_test.nc -C countries_id_test.nc -w waterregions_initial_test.nc -o my_new_waterregions.nc -m metadata.test.jsonpython define_waterregions.py -p calib_points_test.txt -l ldd_test.map -C countries_id_test.nc -w waterregions_initial_test.map -o my_new_waterregions.nc -m metadata.test.yamlThe input maps can be in nectdf format or pcraster format (the same command line can accept a mix of pcraster and netcdf formats).It is imperative to write the file name in full, that is including the extension (which can be either ".nc" or ".map").The utility can return either a pcraster file or a netcdf file. The users select their preferred format by specifying the extension of the file in the output option (i.e. either ".nc" or ".map").The metadata file in .yaml format must be provided only if the output file is in netcdf format.The code internally verifies that the each one of the newly created water regions is entirely included  within one calibration catchments. If this condition is satisfied, the follwing message in printed out:“OK! Each water region is completely included inside one calibration catchment”. If the condition is not satisfied, the error message is“ERROR: The  water regions WR are included in more than one calibration catchment”. Moreover, the code provides the list of the water regions WR and the calibration catchments that do not meet the requirment. This error highlight a problem in the input data: the user is recommended to check (and correct) the list of calibration points and the input maps.The input and output arguments are listed below.usage: define_waterregions.py [-h] -p CALIB_POINTS -l LDD -C COUNTRIES_ID -w
                              WATERREGIONS_INITIAL -o OUTPUT_WR

Define Water Regions consistent with calibration points: {}

optional arguments:
  -h, --help            show this help message and exit
  -p CALIB_POINTS, --calib_points CALIB_POINTS
                        list of calibration points: lon or x, lat or y, point id. File extension: .txt,
  -l LDD, --ldd LDD     LDD map, file extension: .nc or .map
  -C COUNTRIES_ID, --countries_id COUNTRIES_ID
                        map of Countries ID, fike extension .nc or .map 
  -w WATERREGIONS_INITIAL, --waterregions_initial WATERREGIONS_INITIAL
                        initial map of water regions, file extension: .nc or .map
  -o OUTPUT_WR, --output_wr OUTPUT_WR
                        output map of water regions, file extension: .nc or .map 
  -m METADATA, --metadata_file METADATA
                        Path to metadata file for NetCDF, .yaml or .json formatverify_waterregionsThis function allows to verify the consistency between a  water region map and a map of calibration catchments. This function must be used when the water region map and the map of calibration catchments have been defined in an independent manner (i.e. not using the utilitydefine_waterregions). The function verify_waterregions verifies that each water region map is entirely included in one calibration catchment. If this condition is not satisfied, an error message is printed on the screen.InputMap of calibration catchments in netcdf format.Water regions map in netcdf format.OutputThe output is a message on the screen. There are two options:'OK! Each water region is completely included inside one calibration catchment.''ERROR: The  water regions WR are included in more than one calibration catchment’: this message is followed by the list of the water regions and of the catchment that raised the isuue.
In case of error message, the user can implement the functiondefine_waterregions.UsageThe following command line allows to produce a water region map which is consistent with the calibration points:python verify_waterregions.py -cc calib_catchments_test.nc -wr waterregions_test.ncThe input and output arguments are listed below. All the inputs are required.usage: verify_waterregions.py [-h] -cc CALIB_CATCHMENTS -wr WATERREGIONS

Verify that the Water Regions map is consistent with the map of the
calibration catchments

optional arguments:
  -h, --help            show this help message and exit
  -cc CALIB_CATCHMENTS, --calib_catchments CALIB_CATCHMENTS
                        map of calibration catchments, netcdf format
  -wr WATERREGIONS, --waterregions WATERREGIONS
                        map of water regions, netcdf formatNOTE:
The utilitypcr2nccan be used to convert a map in pcraster format into netcdf format.Using lisfloodutilities programmaticallyYou can use lisflood utilities in your python programs. As an example, the script below creates the mask map for a set of stations (stations.txt). The mask map is a boolean map with 1 and 0. 1 is used for all (and only) the pixels hydrologically connected to one of the stations. The resulting mask map is in pcraster format.fromlisfloodutilities.cutmaps.cutlibimportmask_from_lddfromlisfloodutilities.nc2pcrimportconvertfromlisfloodutilities.readersimportPCRasterMapldd='tests/data/cutmaps/ldd_eu.nc'clonemap='tests/data/cutmaps/area_eu.map'stations='tests/data/cutmaps/stations.txt'ldd_pcr=convert(ldd,clonemap,'tests/data/cutmaps/ldd_eu_test.map',is_ldd=True)[0]mask,outlets_nc,maskmap_nc=mask_from_ldd(ldd_pcr,stations)mask_map=PCRasterMap(mask)print(mask_map.data)