DynaDojo: An Extensible Benchmarking Platform for Scalable Dynamical System Identification
Authors: Logan Mondal Bhamidipaty
Tommy Bruzzese
Caryn Tran
Rami Ratl Mrad
Max Kanwal
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: system identification
benchmarking
scaling
TL;DR: We unveil DynaDojo, an extensible benchmarking platform for advancing system identification by evaluating any learning algorithm's sample efficiency, scalability, and generalizability across diverse dynamical systems.

Abstract: Modeling complex dynamical systems poses significant challenges, with traditional methods struggling to work across a variety of systems and scale to high-dimensional dynamics. In response, we present DynaDojo, a novel benchmarking platform designed for data-driven dynamical system identification. DynaDojo enables comprehensive evaluation of how an algorithm's performance scales across three key dimensions: (1) the number of training samples provided, (2) the complexity of the dynamical system being modeled, and (3) the training samples required to achieve a target error threshold. Furthermore, DynaDojo enables studying out-of-distribution generalization (by providing multiple test conditions for each system) and active learning (by supporting closed-loop control). Through its user-friendly and easily extensible API, DynaDojo accommodates a wide range of user-defined $\texttt{Algorithms}$, $\texttt{Systems}$, and $\texttt{Challenges}$ (scaling metrics). The platform also prioritizes resource-efficient training for running on a cluster. To showcase its utility, in DynaDojo $\texttt{0.9}$, we include implementations of 7 baseline algorithms and 20 dynamical systems, along with many demo notebooks. This work aspires to make DynaDojo a unifying benchmarking platform for system identification, paralleling the role of OpenAI’s Gym in reinforcement learning.

DynaDojo: An Extensible Benchmarking Platform for Scalable Dynamical System Identification
Logan Mondal Bhamidipaty" Stanford University loganmb0cs stanford.edt
Tommy Bruzzese" Stanford University tbruêes. stanford. edu
Caryn Tran"
Northwestern University
carynuu. nowtwnssttttw..wccomomoo edu
Rami Mrad UC Berkeley
Max Kanwal
Stanford University
ranirratimraa@@errkkeee...c
kanwal@stanford. edu
Abstract
Modeling complex dynamical systems poses significant challenges, with tradi- tional methods struggling to work across variety of systems and scale to high- dimensional dynamics. In response, we present DynaDojo, a novel benchmarking platform designed for data-driven dynamical system identification. DynaDojo enables comprehensive evaluation of how an algorithm's 's performance scales across three key dimensions: (1) the number of training samples provided, (2) the complex- ity of the dynamical system being modeled, and (3) the training samples required to achieve target етrоr threshold. Furthermore, DynaDojo enables studying out-of- distribution generalization (by providing multiple test conditions for each system) and active learning (by supporting closed-loop control). Through its user-friendly and easily extensible API, DynaDojo accommodates a wide range of user-defined Al Algoritie thms, Systems, and Challenges (scaling metrics). The platform also prioritizes resource-efficieen training for running on a cluster. To showcase its utility, DynaDojo 0.9, include implementations of baseline algorithms and 20 dynamical systems, along with many demo notebooks. This work aspires make DynaDojo unifying benchmarking platform for system identification, paralleling the role of OpenAl's I's Gym in reinforcement learning
Introduction
Dynamical systems, fundamental to disciplines like physics, engineering, economics, and neuro- science, are difficult to predict and control when nonlinear and high-dimensional. Traditional methods that rely known underlying model structure fall short when faced with modern problems like stock market forecasting or modeling human social interactions, where the structure is either unknown non-existent. This has prompted shift toward data-driven modeling (system identification). and especially model-free methods, bypassing the need for predefined equations []] To benefit from these data-driven approaches, however, researchers and practitioners need tailored benchmarks to easily evaluate and compare methods for system identification in their area of study.
In this work, present DynaDojo, open, extensible benchmarking platform standardize the process of benchmarking any learning algorithm on any dynamical system. Modeled after OpenAl's Gym 2 and Procgen DynaDojo introduces abstractions over algorithms, systems, and challenges
"qual contribution corresponding author
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

1
STEP 2 Select System evaluate on
STEP Instantiate Challenge
STEP 4 Get Results
Select your Algorithm (with optional control)
Fixed Complexity
Linear
DMD
SINDY
ODE
Fixed # Samples
DNN
CNN
LPR
Fixed Error
complexit
Your Custom System
Your Custom Challenge
Algorithm
STEP 5 Repeat to compare different settings
tittt--Diitrbbution (OOD)
Active Learning
Algorithm Comparison
Cross-System Generalization
In Dist
ODD
LPR
LPR w/ Control
SINDY
DMD
Heat
Epidemic
Figure I: Pipeline for how to use DynaDojo. Select an algorithm (Step 1) and a system (Step 2), then instantiate challenge (Step 3), and evaluate get results (Step 4). It is easy to repeated DynaDojo challenges to compare performances on in- vs. out-of-distribttion data, with vs. without active learning, between algorithms, and across systems (Step 5).
to promote ease of use, modularity, and broad compatibility. It includes growing library of reference algorithms and tunable dynamical systems. Uniquely, DynaDojo facilitates benchmarking scalability via procedural generation and customizable challenges that systematically vary training set size, system complexity, and target error rates. Challenges also allow for assessing out-of-distribution generalization, active learning, algorithm comparison, and cross-system performance. Through its simple yet flexible API, parallelized execution, and interactive demos, DynaDojo aims serve as an accessible yet rigorous benchmarking platform for the system identification community.
2 Related work
Because of the shared contexts the fields system identification and reinforcement learning (RL), draw upon and connect literature from both areas motivate the development of our benchmarking platform. While system identification focuses on accurately modeling system dynamics. RL aims optimize inputted actions system to maximize reward. Despite their distinct objectives, they both seek to model and interact with complex environments and can often used to solve similar or overlapping problems. For summary comparison of DynaDojo against existing benchmarks, Table
2.1 Benchmarks and benchmarking platforms
Single-System Benchmarks In the field of system identification, benchmarks have been created that use data on one specific dynamical system from a physical phenomenon of interest 54. These single-system benchmarks have been used evaluate specific learning algorithms (6) or compare different approaches (7). These narrowly focused benchmarks do not facilitate the evaluation of an algorithm's generalization across diverse set of systems, which is central aim of our work.
Benchmark suites offer broader scope for evaluating system identification algorithms by covering larger subset of system classes. These suites, however, are restricted to specific types of systems or representations, such as chaotic systems (8. physical systems [TO], or partial differential equations [I]] 12). DynaDojo is agnostic to the type of dynamical system allow
Benchmark Suites

---

Active Learning
Sample Efficiency
Complexity
Systems
Algorithms
Extensible
OOD
Measure
Single-System
Chaos [8]
131
Fixed
PDEArena
5
13
PDEBench
12
11
3
mn-benchmark 10
4
Procgen'
16
Binary Tunable
DynaDojo
20
7
Table I: Comparison with related system identification benchmarks. DynaDojo, to the best of our knowledge, is the first extensible dynamical systems benchmark that evaluates out-of-distribution (OOD) trajectories, supports active learning, benchmarks sample efficiency, and implements dynami- cal systems with tunable complexity measures. "Note: Procgen is RL benchmark.
for diverse range of systems to be included. We implement 20 such systems (Figure) and provide a simple wrapper interface to use to add more.
OpenAl Gym To address the fragmentation and lack of standardization in system identification benchmarks, we adopt similar approach OpenAl's Gym environment (2]. OpenAI Gym offers common interface for RL benchmarking tasks and, with over 6,500 citations, has become a standard benchmark framework in RL. Our work creates an extensible, standardized gym-like platform to unify system identification benchmarks. Contrary to OpenAl's focus on environments and nor agents [2], we provide abstractions over both entities (systems and algorithms, respectively) and additionally implement challenges that orchestrate benchmark evaluation while scaling parameters such as system complexity and training set size. In Subsection 2.2 we motivate our focus on scaling system complexity and training samples. And in Section 3.4 we explain why we implement challenges for benchmark orchestration.
2.2 Generalization and scaling
In designing DynaDojo, we drew upon variety of literature to determine desirable features and evaluation metries to implement in our platform.
Out-of-Distribution Generalization In system identification, there is interest in understanding how well models can generalize beyond the training distribution. [13) probes how deep learning
models generalize to trajectories from out-of-distribution initial conditions for dynamical systems. [14 investigates whether deep neural networks learning cellular automata show disti--iist generalization for unseen initial configurations with different rule sets. Recent RL benchmarks have sought to split train and test data draw from different gaming environments in response to problems of overfitting on training environments [15] 16]. Motivated by this work, DynaDojo enables evaluating algorithms on both in-distribution and -istti-iiittibution data.
Cross-System Generalization While we are unaware of any work in system identification that explores generalization across different systems, the concept has been explored in RL. Generalization in RL has expanded from considering unseen states to learning across different domains, as exempli- fied by AlphaZero's ability learn Go, Chess, and Shogi [17]. compared AlphaGo's specialization in Go [88 It is also similar the field of multi-task learning in which one general policy might train on several different environments [9 This trend is reflected in RL benchmarks and toolkits that measure performance across a diverse variety of environments 22 3 Our work takes an analogous approach by facilitating easy evaluation of an algorithm's 's performance across different classes of dynamical systems, thereby capturing the algorithm's 's cross- system generalization capabilities.
Scaling Complexity There is desire understand how algorithms perform systems of varied complexity. For example, within system identification, [14] and [20] test deep neural networks on their generalization capabilities for learning cellular automata with varied neighborhood sizes.
In RI., benchmarks have been designed train and test algorithms on game environments with varying difficulty levels 33. This work games, however, uses imprecise notions of difficulty which
3

---

20 PROCEDURALLY-GENERATED DYNAMICAL SYSTEMS
System type
Measure Complexity
Generalized Lorenz
Cellular Automata (CA)
Chaotic behavior and complex attractors in fluid dynamics
Grid-based models with simple interaction rules
Y
Continuous
Y Discrete
^ of Equations
M Neighborhood radius
Linear Dynamical System (LDS) Solvable system: x = Ax + Bu
S
Opinions
Opinions change due to pair- wise and group interactions
Continuous
Rank(A)
Discete
of Agents
Threshold-Linear Network (TLN)
Epidemic
Disease spread between Individuals and populations
LDS with ReLU nonlinearity:
À - [Ax b].
Hybrid
① Continuous
ا # of Agents
/ Rank(A)
Spiking Neural Network (SNN)
Lotka Volterra (LV)
Nonlinear, higher-dimensional embedding LDS
Competitive inter- and intra- species
Hybrid
Continous
Rank(A)
Rank(A)
Kuramoto
m
N-Body
Collective synchronization of coupled oscillators
Gravitational Interactions among celestial bodies
Continuous
Y
Continuous
^ of Oscillators
x of Bodies
X
Forward-Backward Backward Stochastic Neural Networks (FBSNN)
2D Heat Equation
Heat diffusion over planar domain
Approximates coupled high- dimensional PDEs without grids
Continuous
Continuous
Plate Area
Dimension
WITH FLEXIBLE OVERRIDES
Initial distribution
Complexity
Trajectory time
Set number of initial conditions, ranges for in and out-of-distribution generation
Adjust the latent and embedded dimenssions for any System
Adjust the absolute time interval and number timesteps to simulate
AND SMART DEFAULTS
Control
Noise
Evaluation
Actively control each step of trajectory, while staying under an settable cost
W
Add an adjustable amount of noise each trajectory
Define System's error metric with standard options (MSE] or a custom function
Figure 2: The 20 dynamical systems currently packaged in DynaDojo. Systems are annotated with the system type (discrete, continuous, hybrid) and their measure of complexity. Users adjust nearly all DynaDojo parameters on these systems, including how trajectories are initialized, simulated, and controlled, well system complexity and evaluation.

---

are calibrated roughly on the training speeds of baseline algorithms, but this definition is inconvenient because it is hard to generalize and test.
In system identification, [8] provides a suite of over 100 benchmark datasets of known chaotic dynamical systems. Fach system is annotated with mathematical properties reflecting the complexity of the system. These amnotations facilitate comparison of learning methods across dynamical systems of varying complexity; our work similarly motivated. In DynaDojo, instead of providing fixed datasets corresponding to different complexity levels, dynamical system classes are defined to allow their complexity levels to be programmatically scaled. This mirrors the approach (2T) which evaluates physics-infomme neural networks on a pair of dynamical systems with tunable parameters that control complexity.
Active Learning In RI., active beaning-whert the algorithm selects control inputs to intelligently explore gaate-space-enhaneee sample efficiency. While dynamical systems can accept control inputs and therefore can support active learning, this property is often underutilized in system identification benchmarks that predominantly use static datasets 4 5 8 10 12 our work, we enable algorithms to provide control inputs to dynamical systems, thereby facilitating active learning approaches and interactive data generation.
The ability to learn effectively from a limited number training samples is particularly relevant it directly influences algorithm's 's performance in real-world scenarios where data may be scarce, expensive to obtain, or hard to simulate (22). In DynaDojo, we'r 'e designed metrics that measure how algorithm's 's performance scales with changes in system complexity and training dataset size. This feature enables users to assess an algorithm's 's sample efficiency, particularly as it handles increasing complexity.
Sample Efficiency
3 Overview of DynaDojo
DynaDooj operates on three core objects: Algorithms, Systems, and Challenges. Run a Challenge with any given Algorithm and System to evaluate how a learning algorithm's 's performance scales (Figure Dp. DynaDojo currently provides a suite of 7 Algorithms, 20 Systems, and 3 Challenges to be used, Additionally, DynaDojo provides abstract interfaces for Algorithms, Systems, and Challenges that can be extended to support custom implementations (Figure
Develop an Algorithm
Develop a System
Develop a Challenge
Implement a wrapper around any existing algorithm to test it using our platform.
Determine how data is generated and loss is calculated for a dynamical system.
Decide how to test an algorithm on system parameter of interest scales.
Implement fit and prediet
Implament make init for in- and out-of- distribution data
conds init_conds
Implement execute job to train and test an algorithm on system
Optionally, implement Net to control data generation
Implement make_daaaa data
Decide how to plot the data
Implement cale error
Optionally, implement cale_contte costrol, cost
Figure 3: Users can extend DynaDojo by implementing their own Algorithm, System, or Challenge.
3.1 Algorithms
Algorithms are subclasses of AbstractAlgorithhn an interface designed to wrap any learning algorithm that can be used for system identification. initialization, algorithms are provided the dimensionality of the data which can be used to define the appropriate parameters for the algorithm,
5

---

for example, the number of layers in neural network or degree of polynomial in polynomial regression. The AbstractAlgori interface further abstracts the details of the algorithm under fit, predict, and act methods for ease of use and simplicity. See the online documentation for additional details on the algorithms included in DynaDooo.
3.2 Systems
Systems are subclasses of AbstractSystem, an interface to wrap any procedurally generated dy- namical systems. Systems initialized with latent dimension and embedding dimension which determine the complexity of the system and its generated trajectories. The AbstractSystem in- terface abstracts the details of simulating the system under the make _nnit_conds and make make_data methods. Initial conditions can be produced in-distribution or out-of-distribution. Data can be generated with noise optional control inputs. The AbstractSystem interface also abstracts the evaluation metric for any particular dynamical system with the calc_errow method. For example, continuous systems, such linear dynamical systems, might implement mean-squared error to evaluate the accuracy of predicted trajectories, whereas a binary system, such as cellular automata, might use Hamming distance instead. Other system-specifie metrics, such achieving stability with control, can likewise be defined in calc_ calc_error. error. See Figure 2 for the systems that we package with DynaDojo and the adjustable parameters available for systems.
3.3 Challenges
Challenges are subclasses of AbstractChallenge. an interface to orchestrate the evaluation of algorithms on systems. Challenges simplify and parallelize repeated trials of training and testing algorithms on systems while parameter (of the system, algorithm, or training process) scales, challenge is run via the evaluate method: This method handles parameter scaling, seed generation (for reproducible execution), and job parallelization. evaluate calls on execute_job, which is where one defines the protocol for algorithm and system instantiation, data generation, and testing for a single trial. The plot method visualizes the results from the challenge evaluation.
In DynaDojo 0.9, we implement three challenges to evaluate scaling: FixedComplexityy FixedTrainSize, and FixedError. In FixedComplexity, we repeatedly train and test an al- gorithm on a system of fixed complexity while scaling the number of training samples. In FixedTrainSize, we repeatedly train and test an algorithm on systems of increasing complex- ity while fixing the number of training samples. In FixedError, samples necessary for an algorithm to achieve a target error rate on systems of increasing complexity. In Figure we visualize the relationship between these three scaling challenges.
search for the number of training
Error
Samples
Ereer
Fixed Error
Saergies ee
Froedd
Borro
Gampal
Fhed
Fixed
food
# Samplien
Comp
Rand
Flaed Complexity
Fixed # of Samples Given fixed amount of data, how well can an algorithm perform on increasingly complex systems?
Fixed Complexity
Fixed Error
Complexity
Given fixed system complexity, how much does more data reduce the error?
How much data do need to hit target error? As system complexity increases, need more data?
Figure 4: DynaDojo Challenges provide a snapshot of an Algorithm's 's scaling behavior along one
slice of a "performance landscape" relating three dimensions: system complexity (C), prediction error (E), and number of training samples (N). Each Challenge varies parameters along one dimension while holding a second one constant, to measure the algorithm's 's performance on the third.

---

3.4 Design
Unified but decoupled. Rather than providing a single or suite of benchmarks, DynaDojo is benchmarking platform. We designed abstractions over algorithms and systems to unify system identification benchmarking under one platform. This allowed us to package Systems and Algorithms in DynaDojo and standardize comparing them via challenges for better consistency. While each system and algorithm work seamlessly together, they are not locked into the platform. To ameliorate the inflexibility of requiring interfaces over both algorithms and systems, designed them to be decoupled from another and thus independently usable outside of DynaDojo (see Section
Focused on scaling, not error. Often, algorithms are evaluated by their performance on a single task. On the contrary, we are focused on how that performance scales as the task gets harder or more data is provided (see Figure D. Specifically in system identification, we saw an opportunity to numerically define task difficulty via measures of complexity. To evaluate scaling, many rounds of instantiating algorithms and systems, training, and testing must be repeated as parameters scale. This assource-intenive intensive workloads. Thus, we conceived of Challenges to manage job parallelization across cores and compute nodes. We also packaged a DynaDojo Docker container for easy use on cluster environments.
can lead to embarrassingly parallel resource-
Procedurally generated, not static. Rather than a static dataset, dynamical systems in DynaDojo procedurally generate data train and test time in order support active learning, where system trajectories are altered by control inputs provided from an algorithm. Additionally, we require a tunable, continuous measure of complexity for each system support scaling metrics. This requires that dimensionality of system trajectories must be dependent on the system complexity annothe reason for procedural data generation. Compared pre-computed datasets, limitation of our work is that procedural generation of data can be costly, especially for systems of high complexity.
Simple by default. A key challenge in system identification is being able to compare one algorithm against others across many systems. This is difficult because various existing algorithms and systems either lack a simple API or all have different APIs. Wrestling together APIs and setting the right parameters is major barrier to benchmarking in system identification. We designed DynaDojo with a focus on simplicity of the API to ensure that different algorithms can painlessly run on any system. To enable this, all systems and algorithms must come with default presets for all parameters, with optional overrides. This requires developers to do work upfront to determine reasonable or adaptive settings for the algorithms or systems they contribute.
Flexible and versatile. We support a variety of settings to enable features such as OOD data generation and control inputs in order to ensure DynaDojo covers broad use-cases (Figure 2). Our extensible interfaces for algorithms, systems, and challenges to ensure that DynaDojo can be adapted to use-cases we have not yet covered (Figure 3}. To show the versatility of DynaDojo benchmarking platform, provide numerous example Jupyter notebooks for implemented algorithms, systems, and challenges, available in our GitHub repository.
4 Example Usage
4.1 Running a single algorithm on a single system
DynaDojo algorithms and systems can be used independently of challenges. To train and test single algorithm instance on single system instance, first instantiate the system and create the training and test data (which, in this example, is OOD).
latent_dimg rmbed_dimg train_size, test_size, timesteps (3, 3, 50, 10, 50)
lorenz= LorenzSystem (latent, dim, embed_ding seed=100)
x0 lorenz. make, iiteeeiiit___eeeeeeeeeeeeeeeeeee
coniittoooddeeeeee conds (train_size)
y0 lorenz. make www./aaaaeeieeaaaaaaaaaerra caait__codsttett_e_ae
in_dist=Palse)
X lorenz, make lite_ddttaee (x0, timesteps-timeateel
y lorenz make_ fatee_datappe (yO, timesteps=timestep noisy=True)

---

Then instantiate the algorithm, fit on the training data, predict, and calculate error. Plotting utilities, demos, and examples are provided in our GitHub repository.
prod truth
sindy SINDy (embed_dim, timesteps,
seed-100)
sindy (fit(x)
20
10
predict trajectories
y-pred = sindy predict 0], timesteps)
-10
-20
lorenz calc_error(s y-pred)
10
fig, ax dynadojo. wtwlu.tillppl.... (Iy pred,
yl, target_ dinget_dimmmi(3,, (stent_dim)) labels- (""red", "truth"])
0
20
30
(b) SINDY algorithm prediction for a Lorenz system showing high overlap with ground truth.
(a) Code for fitting and testing SINDy LorenzSystem using DynaDojo. Plotting utilities are provided to visualize high dimensional systems.
4.2 Running challenge
To evaluate, for example, how linear regression generalizes to linear dynamical systems of increasing complexity, run FixedTrainSize. First, decide the complexities (latent dimensions) scale across, the number of training samples, the number of trials, and the training and testing conditions. Supply these arguments instantiate FizedTrainSize challenge. Then, evaluate the challenge with the algorithm class and plot the results. By default, challenges are run without parallelization; and computers are provided GitHub.
however, code examples showing parallelization
Fixed Train Size, 100
challenge FixedTrainSizec
10*
Distribution
wer.npp.llloo logspace 3, 20, train_size=100,
nod, eioo
10
timestepa=50,
control_horizonss
mas_contro__cost_prr_ddm
system_cca=1DSSSytten,
trials=100,
10
test_examples=50,
10
test_timesteps-50
data challenge evaluatee
13
LinearRegression,
14
noisy=True,
10'
15
ood-True)
(b) FixedTrainSize plot for linear regression on a lin- car dynamical system with 100 training samples. Latent dimensionality is the z-axis and error on the y-axis. Curves for testing in- and out-of-distribution (OOD) are overlapping. showing OOD generalization. ter L = 100, error rapidly increases, showing weaker sample efficiency with high system complexity.
17 FixedTrainSize plot (data)
(a) Code for running FixedTrain challenge with on a training set size of 100 for LinearRegression on LDSSystem with latent dimensions from 10 to 1000.
4.2.1 Analyses
With a suite of baseline algorithms and dynamical systems, DynaDojo is designed to support running repeated challenges to analyze algorithms across systems, schematically depicted in Figure [

---

Out-of-Distributin Generalization To test how an algorithm generalizes to out cuttof-diitributtin (OOD) data, run a challenge with the ood parameter enabled. Challenges will test the algorithm on data simulated both from initial conditions drawn from the same distribution- as the training set initial conditions and from those drawn OOD. See Figure 7 for an example analysis of OOD generalization in a FixedComplexity challenge.
Active Learning DynaDojo algorithms can optionally implement an act method which generates control inputs that DynaDojo systems accept when generating data. To compare the effect of active learning, run a challenge for given algorithm without control and the same algorithm with control on given system. See Figure g for an example analysis of active learning in FixedError challenge.
Comparing Algorithms To compare algorithms, run challenge for two different algorithms on the same system. See Figure 9 for example comparison of two algorithms in FixedError challenge.
Cross-System Generalization To investigate whether an algorithm's 's performance is generalizable across systems, run challenge with an algorithm on two different systems. See Figure TO for an example analysis of cross- cross-system generalization in a F FixedTrainSize challenge.
16
Floed Comp sory isterie5
Entrr, target emro--10
10'
4
Mode
16'
16
(Coneroled)
16'
Figure/ 7: Comparing OOD generalization for deep neural networks (DNN) on linear dy- namical systems (latent dimension 5) in FixedComplexity challenge. In-distribution test error (blue line) decreasing steeply but OOD test error (orange line) is constant number of training samples (π-axis) increases, showing lack of OOD generalization as training size scales.
Figure 8: Comparing active learning for Low- est Possible Radius (LPR) on cellular automata (CA) in a FixedError challenge. As CA com- plexity (x-axis) scales, LPR w/ control (blue line) requires less training samples (y-axis) than LPR w/o control (orange line) to achieve zero error.
12'
10'
10'
(a) FixedError for LR on LDS, target error=
(b) FixedError for DNN on LDS, target error= le0.
Figure 9: Comparing two algorithms: Linear regression (LR) and deep neural network (DNN) are evaluated on linear dynamical systems in a FixedError challenge. As complexity (z-axis) of the system scales, LR (left) achieves a much lower target error of 10 with fewer or comparable number of training samples (y-axis) than needed by a DNN (right) for higher target error rate of 1.

---

Floo Train Bice, 1030
Daltibution
19
Flod Tren Stew ne3000
Diort
sod uno
15
12"
12"
12
4x100
3x100
4x10
(a) FixedTrainSize with SINDy and Lorenz.
(b) FixedTrainSize with SINDy and LDS.
Figure 10: Cross-system generalization: SINDy is evaluated on Lorenz Systems (left) and linear dynamical systems (LDS) (right) in a FixedTrainSize challenge. Error (y-axis) decreases with complexity (z-axis) for Lorenz, but increases and plateaus with complexity for LDS. Also, OOD (orange) and in-distribution (blue) error are matched, showing OOD generalization.
5
Future work
DynaDojo is still a work in progress. As of our most recent release, our platform has certain limitations worth noting: While we implement parallel computing across all Challenges, the FixedError challenge is still especially resource-intensive and susceptible to noise. We aim to implement more sophisticated root-finding search algorithms to replace our exponential search.
To improve the rigor of our scaling results, we would like to move to a more objective measure of system complexity (e.g., the intrinsic dimension of the objective landscape (23). We also seek to develop scaling metrics that summarize the results plotted by DynaDojo. Furthermore, we would like to define unified generalization metric that captures algorithm's 's capacity work OOD test. data, across scales of complexity, and on different dynamical systems altogether.
We will, of course, always look for ways to include more state-of-the-ar algorithms and dynamical systems of interest. In particular, we would like to develop simple wrappers for OpenAl Gym environments and algorithms to immediately accommodate their vast library, and vice-versa, wrapping DynaDojo Systems and Algorithin thms for OpenAI Gym.
To further broaden the scope and applicability of DynaDojo, we plan on introducing new Challenges of interest to the community. For example, we aim to incorporate an optimal control challenge involving stabilizing system around a target trajectory, transfer learning challenge focusing on fine-tuning to new system data, a multi-task learning challenge around maintaining consistent performance across multiple dynamical systems, and curriculum learning challenge focusing on leveling up system complexity without retraining an algorithm from scratch. All of the Challenges can also be extended to measure prediction/control error multiple timescales.
Lastly, in light of the emerging importance of scaling laws in deep learning, we hope to incorporate new scaling dimensions. These will include the number of model parameters. computational cost of training, and activation sparsity, on top of the three existing scaling dimensions. By including these dimensions, we aim to offer a more comprehensive view of how algorithms scale.
10

---

Author contributions
Using the Contributor Roles Taxonomy (CRediT
LMB contributed to the software (lead on design, documentation; support on algo- rithms/systems), data curation (equal on demo notebooks).
TB contributed to the visualization (lead on creative direction; co-lead on figures), software (lead on algorithms/systemn support on documentation), data curation (equal on demo notebooks), writing (editing).
CT contributed to the writing (co-lead), software (lead on reproducibility, parallelization, testing; support on design), visualization (co-lead on figures), and led the investigation, validation.
RM contributed to the software (support on algorithns/yystemms).
MK led the conceptualization, supervision, project administration. methodology, and con- tributed to the writing (co-lead), software (all-around support).
Acknowledgments
LMB was supported by Stanford's CURIS program. was supported by Stanford's Bioengineering REU program. MK was supported by an NSF Graduate Research Fellowship. The authors thank Stanford's Brains in Silicon Lab, Northwestern's Delta Lab, and Sean Yoon for helpful feedback on the writing. This work used Stanford's Sherlock and Northwestem''s Quest clusters to collect results.
References
[1] Oliver Nelles. Nonlinear dynamic system identification, Springer, 2020.
[2] Greg Brockman et al. "OpenAl Gym". In: CoRR abs/1606.01440 (2016). arXiv: 1606. 01540
URL: httt:://arrav.orr//bbb////eeeeee argg/bbs/6606 01540
[3] Karl Cobbe et al. "Leveraging Procedural Generation to Benchmark Reinforcement Learning"
In: CoRR abs/1912.0155 (2019). arXiv: 1912. 01588
URL: /ttt:///rxiv.... org/ abs
1912.01588
[4] Florent Bonnet et al. "An extensible Benchmarking Graph-Mesh dataset for studying Steady- State Incompressible Navier-Stokes Equations". In: ICLR 2022 Workshop Geometrical and Topological Representation Learning. 2022. URL: https://openreeveww net/ fttf/oormmmi/
rqudi4-kpeq
[5] Jean-Philippe Noel and Maarten Schoukens. "Hysteretic benchmark with a dynamic nonlinear-
ity". In: Workshop on nonlinear system identification benchmarks. 2016, pp. 7-14.
[6] Max Schüssler, Tobias Münker, and Oliver Nelles. "Deep Recurrent Neural Networks for Non- linear System Identification". In: 2019 JEEE Symposium Series on Computational Intelligence (SSCI). 2019, pp. 448 454. DOI: 10. 00009//SS44811 2019. 9003133
[7] Max Schüssler, Tobias Münker, and Oliver Nelles. "Local Model Networks for the Identifi- cation of Nonlinear State Space Models". In: 2019 JEEE 58th Conference on Decision and Control (CDC). 2019, pp. 6437-6442. DO1: 10. 1109/ CCO0//CCC40024-00 2019. 9028945
[8] William Gilpin. "Chaos as an interpretable benchmark for forecasting and data-driven mod- elling". In: Thirty-fitth Conference Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2). 2021. URL: httpt::////ttttc
https:///peerrvviee /pwprevereevee.e.mom neteeewwweeeeeeeeeee feet/forum ? id=
enYjtbjYJrf
[9] Luā Streit, Vikram Voleti, and Tegan Maharaj. "Understanding Generalization and Robustess Learned Representations of Chaotic Dynamical Systems". In: ICML 2022: Workshop on Spurious Correlations, Invariance and Stability. 2022. URL: hnttps://ppereviier net/ forum?id=sJg. BRg
[10] Karl Otness et al. "An Extensible Benchmark Suite for Learning to Simulate Physical Systems". In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round 1). 2021. URL:
https://credit. niso. org/

---

[11] Jayesh K Gupta and Johannes Brandstetter. "Towards Multi-spatiotemprrall scale Generalized PDE Modeling". In: arXiv preprint arXiv:2209. 15666 (2022).
[12] Makoto Takamoto et al. "PDEBench: An Extensive Benchmark for Scientific Machine Learn- ing". In: Thirry-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022. http::tttts::::orrreeeee
net/forum?id dh, _MXoofrX
[13] Rui Wang et al. "Learning dynamical systems requires rethinking generalization.. In: NeurIPS 2020 Workshop on Interpretable Inductive Biases and Physically Structured Learning. 2020. URL: https: / www amazon science wwtps//pbbliiiiiioooioioa learning dynami cal
[14] Marcel Aach, Jens Henrik Göbbert, and Jenia Jitsev. "Generalization over different cellular automata rules learned by deep feed-forwart neural network", In: CoRR abs/2113.14488 URL: htttps://arxai /ariivv.//ivvv/// www.aiiww/....a...a.....eaooooie
(2021). Xrv: 2103. 14886
org/ htt///bb/22111 14886
[15] Karl Cobbe et al. "Quantifying generalization in reinforcement learning". In: International Conference on Machine Learning. PMLR. 2019, pp. 1282-1289.
[16]
Alex Nichol al. "Gotta learn fast: A new benchmark for generalization in rl". In: arXiv preprint arXiv: /wrXivv/04..000 087400370 (2018).
[17] David Silver et al. "A general reinforcement learning algorithm that masters chess, shogi, and
Go through self-play". In: Science 362.6419 (2018), pp. 140-1 144. DoL: 10. 1126/science aar6404 eprint: https: / www science org /dor /df /10 1 science aar6404
URL:: htt.:h/ppp//ww...cceeee...rrrr/p
10. 1126 science.....c. a4r6404
[18] David Silver et al. "Mastering the game of Go with deep neural networks and tree search". In: nature 529.7587 (2016), pp. 484 489.
[19] Tianhe Yu et al. "Meta-
Benchmark and Evaluation for Multi-Tass and Meta Re-
inforcement Learning".. In: CoRR abs/1910. 10897 (2019). arXiv: 1910. 10897
URL: http
metxxv/.org/ab//110.or 10897
[20]
William Gilpin. "Cellular automata as convolutional neural networks". In: Phys. Rev. E
100 Sept. 2019), p. 032402. DOL: 10 1103 Phyo//hhyRev 100 032402
URL: https
//link. aps prg//oi/10/ 1103/PhysRevis 100. 032402
[21] Alexander New et al. "Tunable complexity benchmarks for evaluating physics-informed neural networks on coupled ordinary differential equations". In: 2023 57th Annual Conference on
Information Sciences and Systems (CISS). IEEE. 2023, pp. 1-8.
[22] Andrei Kramer, Ben Calderhead, and Nicole Radde. "Hamiltonian Monte Carlo methods for efficient parameter estimation in steady state dynamical systems". In: BMC bioinformatics 15 (2014), pp. 1-11.
[23] Chunyuan Li al. "Measuring the Intrinsic Dimension of Objective Landscapes". In: Interna- tional Conference on Learning Representations. 2018.
12


Reviews:
[{"review_id": "NZfYEdBfWk", "reviewer": "Reviewer_33kq", "summary": "", "questions": "", "limitations": "Yes, the authors discussed the limitations of the approach.", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The benchmark is well-motivated, and feels like it fills a gap in the existing work.  From a usability perspective, this benchmark seems much more user friendly than many of the other submissions that I have reviewed for this track at NeurIPS.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Mq8gIWrTtv", "reviewer": "Reviewer_j4gM", "summary": "", "questions": "", "limitations": "It would seem to me that the only limitations to their method are inherent to the ones present in the `sklearn` library (if that is the case, it would be good if the authors could briefly mention this in the paper as well).", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main strength of the paper is unifying a number of existing test environments in a single framework. Identifying common feature of existing environment, in order to unify them, is the main contribution. \nFour prototypical classes of dynamical systems are used to illustrate the framework.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dE2PBmp98D", "reviewer": "Reviewer_Nmnd", "summary": "", "questions": "", "limitations": "It seems that the benchmark currently does not take a look at the computational complexity of the methods being considered, something which is (arguably) as important as the other analyses in Figure 3. I understand that this is hard to do from an engineering point of view, either by asking derived classes of AbstractModel to self report parameter sizes or somehow track them like PyTorch does, but I believe it deserves a discussion.\nAdditionally, the generality of the API means that it is difficult to use for more specialized applications. For example, implementing a mesh based system is possible but you’d have to throw away mesh specific information. While I don’t expect this platform to handle every specialized case, I do think that it deserves discussion especially as other related work can in some cases.\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Analyzing the use of data driven methods is considerably more opaque than traditional methods. Often, a model builder will publish the results of a large model, trained with unknown amounts of data for unknown amounts of time, making it look very impressive. I believe this platform is very useful to the scientific community that is thinking about using such methods, because it aligns with things that researchers care about. For example, how do methods compare if one cares about achieving a fixed accuracy on a problem. Or, given that one has a fixed amount of data available, which method performs better or has a lower uncertainty for a certain regime. \nAdditionally, the the API seems well architectured and general enough to allow for broader comparisons along the chosen cross sections of analysis.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Y8HJrHGHTH", "reviewer": "Reviewer_rZxW", "summary": "", "questions": "", "limitations": "Although the paper mentions some limitations, system identification is much more complex than presented in the paper.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper presents an extensible framework to study a very important and relevant problem of system identification, which has several applications across domains.\n\n2. The paper is well-organized and the explanations highlight well the features of DynaDojo that allow exploration of active learning algorithms to improve sample efficiency. This expands the research landscape in dynamical system identification by providing access to the data-generating process.\n\n3. The benchmarks, systems and models implemented provide a reasonable initial set of problems that demonstrate the benefits of the approach.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper presents a solid benchmark for dynamical systems, and 3 out of 4 reviewers were in favor of the contribution being accepted. The issues raised by reviewer rZxW have (to a substantial degree) been addressed in the updated version of the paper, and I hence recommend acceptance.
Decision: Accept (Poster)