Package: pyESD
Python Package for Empirical Statistical Downscaling (v1.01) :sun_behind_rain_cloud: :cloud_with_snow: :cloud_with_rain: :fire: :thermometer:PyESDis an open-source framework of the Perfect Prognosis approach of statistical downscaling of any climate-related variable such as precipitation, temperature, and wind speed using reanalysis products eg. ERA5 as predictors. The package features all the downscaling cycles including data preprocessing, predictor selection, constructions (eg. using transformers), model selection, training, validation and evaluation, and future prediction. The package serves as the means of downscaling General Circulation Models of future climate to high resolution relevant for climate impact assessment such as droughts, flooding, wildfire risk, and others.
The main specialties of the pyESD include:Well designed in an OOP style that considers weather stations as individual objects and all the downscaling routines as attributes. This ensures fewer lines of code that cover the end-to-end downscaling of climate change variable products.PyESD features many machine learning algorithms and predictor selection techniques that can be experimented with toward the selection and design of robust transfer functions that can be coupled with GCM to generate future estimates of climate changeAnd many other functionalities that are highlighted in the paper description of the package (to be submitted).The main component and the work flow of the package are summarised in the modeling outline:Installation :hammer_and_wrench:Install the standard version:pip install pyESDfrom PyPI or git clonegit@github.com:Dan-Boat/PyESD.gitcd to the folder|pip install .Install in editable mode:pip install -e pyESDorpip install -e .in the package base folder clone from github
The installation might require some dependencies that must be installed if not successful from the distribution from PyPI:cartopy,xarray,sciki-learn,scipyand the other scientific frameworks such as NumPy, pandas,Matplotlib, andseabornAlternatively, to ensure the installation in an isolated environment, virtual python environment usingcondaorvirtualenvcan be used to create a separate env for the package installationDocumentation :blue_book:The package documentation is still in progress. The initial structure is accessible atgithub-pagesExamplesThe package has been used for downscaling precipitation and temperature for a catchment located in southwestern Germnany. We have also used it for generating future rainfall products for all the synoptic weather stations in Ghana. Their respective control scripts are located in theexamples folder. Generally, the control scripts follow the modeling workflow as shown in:.
For instance, the downscaling framework show below can be experimented with to select the robust predictor selection method and emprical transfer function for a specific location and predictand variable.Workflow demonstration: To use the PP-ESD model to downscale climate model, weather station and reanalysis datasets are required. The predictors are loaded in as netCDF files and the predictand as csv file. Let assume that the various predictor variables are stored locally in theera5_datadirdirectory/home/daniel/ERA5/and the predictand variable eg. precipitation is stored instation_dirThe files should have the same timestamp as the interested predictand variableimport all the required modulesfrom pyESD.Weatherstation import read_station_csv
from pyESD.standardizer import MonthlyStandardizer, StandardScaling
from pyESD.ESD_utils import store_pickle, store_csv
from pyESD.splitter import KFold
from pyESD.ESD_utils import Dataset
from pyESD.Weatherstation import read_weatherstationnames

import pandas as pdRead the datasetsERA5Data = Dataset('ERA5', {
    't2m':os.path.join(era5_datadir, 't2m_monthly.nc'),
    'msl':os.path.join(era5_datadir, 'msl_monthly.nc'),
    'u10':os.path.join(era5_datadir, 'u10_monthly.nc'),
    'v10':os.path.join(era5_datadir, 'v10_monthly.nc'),
    'z250':os.path.join(era5_datadir, 'z250_monthly.nc'),define potential predictors and radius of predictor construction, time range for model training and evaluationradius = 100 #km
predictors = ["t2m", "tp","msl", "v10", "u10"]

from1958to2010 = pd.date_range(start="1958-01-01", end="2010-12-31", freq="MS") #training and validation

from2011to2020 = pd.date_range(start="2011-01-01", end="2020-12-31", freq="MS")  # testing trained dataRead weather stations as objects and apply the downscaling cycle attributes. Note that running the model the first time for a specific location extract the regional means using the define radius and location of the station. The extracted means are stored in a pickel files in the directory calledpredictor_dirvariable = "Precipitation"
SO = read_station_csv(filename=station_dir, varname=variable)
        
        
# USING ERA5 DATA
# ================


#setting predictors 
SO.set_predictors(variable, predictors, predictordir, radius,)

#setting standardardizer
SO.set_standardizer(variable, standardizer=MonthlyStandardizer(detrending=False,
                                                               scaling=False))

scoring = ["neg_root_mean_squared_error",
            "r2", "neg_mean_absolute_error"]

#setting model
regressor = "RandomForest"
SO.set_model(variable, method=regressor, 
               daterange=from1958to2010,
               predictor_dataset=ERA5Data, 
               cv=KFold(n_splits=10),
               scoring = scoring)


# MODEL TRAINING (1958-2000)
# ==========================


SO.fit(variable, from1958to2010, ERA5Data, fit_predictors=True, predictor_selector=True, 
            selector_method="Recursive" , selector_regressor="ARD",
            cal_relative_importance=False)
   
score_1958to2010, ypred_1958to2010 = SO.cross_validate_and_predict(variable, from1958to2010, ERA5Data)
   
score_2011to2020 = SO.evaluate(variable, from2011to2020, ERA5Data)

ypred_1958to2010 = SO.predict(variable, from1958to2010, ERA5Data)
   
ypred_2011to2020 = SO.predict(variable, from2011to2020, ERA5Data)

ypred_2011to2020.plot()
plt.show()Package testingThe package is tested using theunittestframework with synthetic generated data. The testing scripts are located in thetestfolder. Running  the various scripts with -v flag (higher level of verbose), would validate the modified version of the package.PublicationsThe package description and application paper is currently under preparation (to be submitted to GMD)
Its application for weather station in Ghana was presented at the AGU22Linkand the paper is under preparationCitation:
Upload on zenodo:https://doi.org/10.5281/zenodo.7748769Future versionsThe pacakage is still in planing stage (v.0.0.1) (The stable version would be uploaded on pypi with the version number v.1.0.1)Collaborators are welcomed: interms of model application, model improvement, documentation and expansion of the package!@ Daniel Boateng (linkedin) :University of Tuebingen:incoming_envelope:dannboateng@gmail.com=======
History1.0.7 (2023-06-30)Bug fixes* Fix the issues with installation on pypi

New indicators
--------------
* added additional teleconnection indicies for Southern Hemisphere
* Test the framework with daily data but reqiure additional models like wet day classifiers
* missing data in the predictand values can be fill with the implemented impute methods

To do
------
* Extend the package to use spatial datasets
* Implement wet and dry day classifiers before model training
* implement the various calibration metric for daily data 
* Add simple MOS method (quantile mapping)