Package: evalPM
evalPMThis repository contains the frameworkevalPMfor creating, evaluating, and comparing immission models of particulate matter (PM), as well as corresponding example datasets.The datasets have been created using data fromSächsisches Landesamt für Umwelt, Landwirtschaft und Geologie, fromLuftgütemessnetz Brandenburg, fromDeutscher Wetterdienst, fromARPA Lombardia, from theEuropean Environment Agency, fromThe Norwegian Meteorological Institute, and from theCopernicus Atmosphere Monitoring Serviceand theCopernicus Climate Change Service.
The individual datasets and their data sources are documented by aREADME.txtin the corresponding data folders.
Note that all times in the supplied datasets are in UTC. If you use data of locations which are in a different time zone, i.e. especially outside of Europe, it is advised to use timestamps of the corresponding local time, since the feature calculation mainly aggregates by date (as specified in the datasets), which obviously should align with the actual local days.Using the FrameworkA simple example of using the framework may look as follows:importevalPMaspmtraining_data,test_data=pm.features.load_data([2020],[2021])pm.helpers.train_and_evaluate_model_at_all_stations(pm.models.GradientBoostingRegressor,training_data,test_data,test_metrics=["RMSE","SMAPE"])Here, a separate Gradient Boosting model is trained per station on data from 2020 and tested on data of the same station from 2021, with the evaluation metrics RMSE and SMAPE being reported per station.Notable options ofload_data()are especially the lists of training and test years, but it also supports the argumentfeature_filewhich should point to a JSON file specifying the features to calculate on the raw data (the format of such files is defined below in the sectionFeature Specification).
Additionally, it supports the argumentdirpathwhich should point to a directory containing the raw data files (one per station), on which the features are to be calculated; the default isdata/data-saxony/.The modulepm.helperscontains many functions for easy model training and evaluation.
As the basis, the functiontrain_and_evaluate_model()expects a model instance and DataFrames with training and test data. The model needs to be an instance of any class in the modulepm.models; if you would like to add a different kind of model, just extend the classAbstractModelto ensure compatibility with the framework.
Additional parameters oftrain_and_evaluate_model()aretrain_metricsandtest_metrics, which both can be either a list of error metrics (see the example above) or the string"all"to return all available metrics.test_metricsmay also be set to"predictions"to return the model's predictions rather than error metrics.More advanced functions of thepm.helpersmodule includetrain_and_evaluate_model_at_all_stations()which expects a model type rather than a model instance and dictionaries mapping station names to their training and test data DataFrames (as returned byload_data()), while also supportingmodel_parameterswith which to instantiate the model of each station as well as the sametrain_metricsandtest_metricsastrain_and_evaluate_model().
In order to not only train and evaluate a separate model per station, but also vary the model parameters, you can use the functionhyperparameter_grid_search(); it takes mainly the same arguments astrain_and_evaluate_model_at_all_stations(), but differentiates betweenstatic_model_parametersandvarying_model_parameters, across which all parameter combinations are used to train and evaluate separate models.
Additionally, it can train and evaluate multiple models in parallel, if supplied with the accordingnum_jobsargument; note however that using this function, because of the potentially parallel execution, might require you to call it from within aif __name__ == "__main__":block.Trained models can also be persisted in order to load them later on.
This can be done for an individual model or using the helper functionstrain_and_evaluate_model()andtrain_and_evaluate_model_at_all_stations().
It is advisable to save relevant metadata alongside the model (feature set, dataset, model type ...), which can be done using the parameterpersistence_metadata.
To load a saved model, use the methodload_model()of the appropriate model class.For further details regarding available functions and their arguments, take a look at the code and the docstrings within it.DependenciesThis framework uses the following libraries:pandas(version 1.4)numpy(version 1.22)scikit-learn(version 1.0)tensorflow(version 2.8)deepsig(version 1.2)matplotlib(version 3.5)experiment.pyTo run an extensive grid search across model types, the amount of training data, boundary layer height features, and hyperparameters, you can simply run the scriptexperiment.pywhich utilizes this framework.
Note that the runtime of the unmodified script will be multiple days.
To get the same results as us, make sure to install the same dependency versions usingpipand the suppliedrequirements.txt.
The script has been tested with python version3.9.12.
Result files will be written to the directoryresults.
To switch to a different dataset than Saxony, simply adjust the variablesdata_folder,training_years, andtest_yearsinside the script as needed.Feature SpecificationFeatures are specified by a JSON file, which can be passed as an argument to theload_datafunction. The default file isfeatures_default.json, which is based on the features ofKlingner et al.(the filesfeatures_blh-mean.jsonandfeatures_blh-min-max.jsonare based on them as well).The structure of the JSON file has to be as follows:{"date_attribute":"date","time_attribute":"time","features":[{}]}date_attributeandtime_attributespecify the names of the date and time column in the data files.featuresis a comma-separated list of feature objects, each of which has the following structure:{"name":"rain2","source_attribute":"rain","aggregation":{"type":"mean","start":"12:30:00","end":"16:00:00"},"shift":0,"delta":0}All values fromstarttoendof each day (both times are inclusive; corresponding to thetime_attribute) of the column specified assource_attributeget grouped by thedate_attributeand aggregated according to the aggregationtype. The name of the resulting feature is specified byname. In this case, the mean of all rain values from 12:30 to 16:00 of each day would become the featurerain2.Possible values of the aggregationtypeincludesum, mean, median, min, max, prod, std, var.If thestarttime is after theendtime or both are equal, the time range is interpreted to be from one day to the next day, e.g. from 16:30 to 06:30. In that case, the date of the calculated feature is the date of the latter day (i.e. the time range is from the previous day to the current day).To aggregate over the whole day, simply omit bothstartandend.Theshiftparameter can be used to shift the calculated feature by n days, e.g. to get an average value of the previous day (shift=1). Thedeltaparameter can be used to calculate differences of the feature between n days, e.g. the change of the feature from the previous to the current day (delta=1). Bothshiftanddeltacan also be omitted instead of setting them to0.Special casesIn addition to these general options, there are also a few special types of aggregation.Firstly, there is the aggregation typedelta, which calculates a difference between two values of thesource_attributeper day. The timestamps of these values are specified viastartandend(i.e. they are required for this aggregation type); specifying astartgreater than or equal toendresults instartbeing on the previous day here as well. The feature is calculated as the end value minus the start value. This is especially useful for calculating the change of a variable within one day:{"name":"temp_gradient","source_attribute":"temperature","aggregation":{"type":"delta","start":"04:00:00","end":"12:00:00"}}Secondly, there are the aggregation typesconsecutive_days_zeroandconsecutive_days_nonzero, which count the number of consecutive preceding days (including the current day) where thesource_attributeis zero throughout the whole day (consecutive_days_zero) or nonzero for at least one timestamp of the day (consecutive_days_nonzero), respectively.
It is optionally possible to specify a time range viastartandendin order to only consider the specified part of each day.
These aggregation types could e.g. be used to calculate the number of preceding days with or without precipitation:{"name":"days_without_rain","source_attribute":"rain","aggregation":{"type":"consecutive_days_zero"}}Thirdly, there is the option to group by a different column than thedate_attribute. To achieve this, set theaggregationblock's optional parametergroup_byto the column name by which to group thesource_attribute. The necessary mapping from the resulting group keys to thedate_attributegroups is found by grouping thegroup_bycolumn by thedate_attribute(each day should only have a single unique value (i.e. repeating for every time of that day) of thegroup_bycolumn, so the mean is always used for this aggregation). Do NOT use this parameter to group by thedate_attribute; for that case, simply omit thegroup_byparameter, since grouping by thedate_attributeis the default behavior. Grouping by a different column than thedate_attributecurrently does not support selecting a sub-range of thetime_attributeviastartandend. An example use-case for this aggregation option is calculating the mean PM concentration per weekday, and supplying this value as a feature to every date with that weekday:{"name":"pm_weekday","source_attribute":"pm","aggregation":{"type":"mean","group_by":"weekday"}}NOTE: The aggregation typesdeltaandconsecutive_days_(non)zeroare mutually exclusive to specifying a differentgroup_bycolumn, i.e. they cannot be performed within a single feature. However, all of these can optionally be combined with either or both ofshiftanddelta(outside of theaggregationblock, see the general example above).ImplementationThe framework consists of five modules, which interact with each other, as well as datasets.The datasets consist of one CSV file per station, which contain columns for the date, time, and measurements.
These datasets are used by thefeaturesmodule to calculate various features as well as the target variable using a feature definition as detailed above.Themodelsmodule acts as an interface between the framework and specific model implementations from other libraries.
Therefore, existing implementations of various model types can be used and are available through a uniform interface within the framework.
All hyperparameters are passed to the constructor of the relevant interface class, such that the training and prediction methods only require the actual training or test data.
The currently included model interfaces includelinear regressionandgradient boostingbased onscikit-learnas well asfeedforward neural networksbased ontensorflow.
However, more interfaces can be simply added as needed, as long as they adhere to the definition of theAbstractModel.Thecomparemodule facilitates evaluation and comparison of model predictions, either through various metrics like RMSE and SMAPE or through a statistical test based ondeep-significance(model comparison only).Thevisualmodule offers functions for visual interpretations by usingmatplotlib, either through a line plot of observations and predictions or through a heatmap of evaluation results.Thehelpersmodule utilizes the modulesmodelsandcompareto wrap model training and evaluation into a single function call and thereby simplify the use of the framework.
This module also includes a function for hyperparameter optimization using grid search.AttributionMaterial preparation, data collection, analysis, and implementation were performed byJonas DeepeandLucas Woltmann.