Package: dynabench
Dynabench: A benchmark dataset for learning dynamical systems from low-resolution dataThis is the repository containing the data generation algorithms as well as all baseline models for theDynabench: A benchmark dataset for learning dynamical systems from low-resolution datapaper (accepted at ECML-PKDD 2023)DynaBench is a benchmark dataset for learning dynamical systems from data. Dynamical systems are physical systems that are typically modelled by partial differential equations (e.g. numerical weather prediction, climate models, fluid simulation, electromagnetic field simulation etc.). The main challenge of learning to predict the evolution of these systems from data is the chaotic behaviour that these systems show (small deviation from the initial conditions leads to highly different predictions) as well as data availability. In real world settings only low-resolution data is available, with measurements sparsly scattered in the simulation domain (see following figure illustrating the distribution of weather monitoring stations in europe).In this benchmark we try to simulate this setting using synthetic data for easier evaluation and training of different machine learning models. To this end we generated simulation data by solving five different PDE systems which were then postprocessed to create low-resolution snapshots of the simulation.There main tasks for which the dataset has been generated is forecasting - predicting the next state(s) of the systemThe six included different equations were selected to be both sufficiently complex, as well as sufficiently variable to simulate different physical systems (first and second order, coupled equations, stationary and non-statinary).An example (wave equation) of a simulated system is shown below:EquationsThere are four different equations in the dataset, each with different characteristics summarized in the following table:EquationComponentsTime OrderSpatial OrderAdvection111Burgers'212Gas Dynamics412Kuramoto-Sivashinsky114Reaction-Diffusion212Wave122SetupAutomated setupIf needed create a virtual environment and activate it
You can then install all dependencies by runningsh scripts/install_requirements.shfrom the main project directoryManual installationAlternatively you can manually install the dependencies from therequirements.txtfile:pip install -r requirements.txtIt is recommended to first create a virtual environment, for example:python -m venv venv
source venv/bin/activateAdditionally you need to install pytorch geometric, following the instructions ontheir website.GenerationTo generate the data for a specific equation runpython generate.py num_simulations=NUM_SIMULATIONS equation=EQUATION split=DATASET_SPLITWhereNUM_SIMULATIONSindicates how many times each equation is simulated,EQUATIONis one of (advection, burgers, gas_dynamics, kuramoto_sivashinsky, reaction_diffusion, wave), andDATASET_SPLITis one of train, test, valThe full benchmark dataset contains 7000 simulations for the training set, 1000 for the validation set and 1000 for the test set, all divided into chunks of 500 simulations.Warning: this can take a long time.Data formatThe data is stored in.tararchives in chunks of 500 simulations. Each simulation consists of one file called XXXXXXX.data containing the simulation values for the given setting (cloud/grid + number of points) as well as a file called XXXXXXX.data containing the coordinates of the points at which the measurements were recorded.UsageTo reproduce the experiments from our paper run:
python main.py equation=EQUATION model=MODEL support=cloud num_points=NUM_POINTSThis will start the training for a specific setting. The parameters specify which model, task, support structure, number of points etc. should be run. The available choices of parameters are:EQUATION = [brusselator, gas_dynamics, kuramoto_sivashinsky, wave, advection]

MODEL = [persistence, point_gnn, point_net, point_transformer, gat, gcn, feast, kernelNN, graphpde]To run the experiments for the grid models run:
python main.py equation=EQUATION model=MODEL support=grid num_points=NUM_POINTS datamodule=torch" lightningmodule=gridmodule support=gridwithMODELselected from [neuralpde, resnet, cnn]Additionally, to use the benchmark for your own research use the included datasets.The repository contains two dataset classes to handle the generated data.A pytorch dataset class, where each sample has the form $X\in\mathbb{R}^{L\times N\times D}$, where L is the lookback, N is the number of points and D is the number of target variables. See documentation of the dataset for details. To initialize the dataset class:DynaBenchBase(mode:str='train',equation:str='gas_dynamics',task:str='forecast',support:str='grid',num_points:str='high',base_path:str='data',lookback:int=1,rollout:int=1,test_ratio:float=0.1,val_ratio:float=0.1,merge_lookback:bool=True,*args,**kwargs)Initializes a pytorch dataset with selected parameters. The data is loaded lazily.Args:mode(str, optional):  the selection of data to use (train/val/test). Defaults to "train".equation(str, optional):  the equation to use. Defaults to "gas_dynamics".task(str, optional):  Which task to use as targets. Defaults to "forecast".support(str, optional):  Structure of the points at which the measurements are recorded. Defaults to "grid".num_points(str, optional):  Number of points at which measurements are available. Defaults to "high".base_path(str, optional):  location where the data is stored. Defaults to "data".lookback(int, optional):  How many past states are used to make the prediction. The additional states can be concatenated along the channel dimension if merge_lookback is set to True. Defaults to 1.rollout(int, optional):  How many steps should be predicted in a closed loop setting. Only used for forecast task. Defaults to 1.test_ratio(float, optional):  What fraction of simulations to set aside for testing. Defaults to 0.1.val_ratio(float, optional):  What fraction of simulations to set aside for validation. Defaults to 0.1.merge_lookback(bool, optional):  Whether to merge the additional lookback information into the channel dimension. Defaults to True.A graph dataset, specifically used for Message Passing Neural Networks implemented using thePytorch Geometricmodule. It has a similar structure as the base DynaBench dataset.Benchmark ResultsThe following tables show the results of our experimentsforecast task, 900 points (1-step MSE):modelAdvectionBurgersGas DynamicsKuramoto-SivashinskyReaction-DiffusionWaveCNN5.30848e-050.01109880.004203680.0006698370.000369180.00143387FeaSt0.0001303510.01161550.01620.01178670.0004888480.00523298GAT0.009601130.04399860.0374830.06670570.009152080.0151498GCN0.0263970.138990.08426110.4365630.1646780.0382004GraphPDE0.0001370980.01073910.01947550.007198220.0001421140.00207144KernelNN6.31157e-050.01061460.0133540.006686980.0001870190.00542925NeuralPDE8.24453e-070.01123730.003734160.0005369580.0003031760.00169871Persistence0.08120810.03676880.1869850.1422430.1471240.113805Point Transformer4.41633e-050.01030980.007248990.004897110.0001412480.00238447PointGNN2.82496e-050.008825280.009016490.006730360.0001360590.00138772ResNet2.15721e-060.01480520.003212350.0004901040.0001567520.00145884forecast task, 900 points (16-step rollout MSE):modelAdvectionBurgersGas DynamicsKuramoto-SivashinskyReaction-DiffusionWaveCNN0.001613310.5545540.9953821.260110.01834830.561433FeaSt1.482880.5611970.8195943.744480.1301491.61066GAT41364.10.8333531.214365.689253.855062.38418GCN3.51453e+1313.08767.206331.70612e+241.75955e+077.89253GraphPDE1.079530.7298790.9692082.10440.08002351.02586KernelNN0.8974310.727160.8540152.003340.06352781.57885NeuralPDE0.0002703080.6597890.4434981.055640.02241550.247704Persistence2.393930.6792611.4571.897520.2756782.61281Point Transformer0.6170250.5038650.6428792.097460.05643991.27343PointGNN0.6606651.043420.7592572.820630.05822931.30743ResNet8.64621e-051.863520.4802841.06970.007046120.299457LicenseThe content of this project itself, including the data and pretrained models, is licensed under theCreative Commons Attribution-ShareAlike 4.0 International Public License (CC BY-SA 4.0). The underlying source code used to generate the data and train the models is licensed under theMIT license.