SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Out-of-Distribution Generalization To test how an algorithm generalizes to out-of-distribution (OOD) data, run a challenge with the ood parameter enabled. Challenges will test the algorithm on data simulated both from initial conditions drawn from the same distribution—as the training set initial conditions—and from those drawn OOD. See Figure 7 for an example analysis of OOD generalization in a FixedComplexity challenge.
Active Learning DynaDojo algorithms can optionally implement an act method which generates control inputs that DynaDojo systems accept when generating data. To compare the effect of active learning, run a challenge for a given algorithm without control and the same algorithm with control on a given system. See Figure 8 for an example analysis of active learning in a FixedError challenge.
Comparing Algorithms To compare algorithms, run a challenge for two different algorithms on the same system. See Figure 9 for an example comparison of two algorithms in a FixedError challenge.
Cross-System Generalization To investigate whether an algorithm's performance is generalizable across systems, run a challenge with an algorithm on two different systems. See Figure 10 for an example analysis of cross-system generalization in a FixedTrainSize challenge.
Figure 7: Comparing OOD generalization for deep neural networks (DNN) on linear dynamical systems (latent dimension 5) in a FixedComplexity challenge. In-distribution test error (blue line) is decreasing steeply but OOD test error (orange line) is constant as number of training samples ( $x$ -axis) increases, showing lack of OOD generalization as training size scales.
Figure 8: Comparing active learning for Lowest Possible Radius (LPR) on cellular automata (CA) in a FixedError challenge. As CA complexity  $(x$ -axis) scales, LPR w/ control (blue line) requires less training samples  $(y$ -axis) than LPR w/o control (orange line) to achieve zero error.
(a) FixedError for LR on LDS, target error  $= 1\mathrm{e} - 5$
(b) FixedError for DNN on LDS, target error  $= 1\mathrm{e}0$
Figure 9: Comparing two algorithms: Linear regression (LR) and deep neural network (DNN) are evaluated on linear dynamical systems in a FixedError challenge. As complexity ( $x$ -axis) of the system scales, LR (left) achieves a much lower target error of  $10^{-5}$  with fewer or comparable number of training samples ( $y$ -axis) than needed by a DNN (right) for a higher target error rate of 1.
(a) FixedTrainSize with SINDy and Lorenz.
(b) FixedTrainSize with SINDy and LDS.
Figure 10: Cross-system generalization: SINDy is evaluated on Lorenz Systems (left) and linear dynamical systems (LDS) (right) in a FixedTrainSize challenge. Error  $(y$ -axis) decreases with complexity  $(x$ -axis) for Lorenz, but increases and plateaus with complexity for LDS. Also, OOD (orange) and in-distribution (blue) error are matched, showing OOD generalization.
5 Future work
DynaDojo is still a work in progress. As of our most recent release, our platform has certain limitations worth noting: While we implement parallel computing across all Challenges, the FixedError challenge is still especially resource-intensive and susceptible to noise. We aim to implement more sophisticated root-finding search algorithms to replace our exponential search.
To improve the rigor of our scaling results, we would like to move to a more objective measure of system complexity (e.g., the intrinsic dimension of the objective landscape [23]). We also seek to develop scaling metrics that summarize the results plotted by DynaDojo. Furthermore, we would like to define a unified generalization metric that captures an algorithm's capacity to work on OOD test data, across scales of complexity, and on different dynamical systems altogether.
We will, of course, always look for ways to include more state-of-the-art algorithms and dynamical systems of interest. In particular, we would like to develop simple wrappers for OpenAI Gym environments and algorithms to immediately accommodate their vast library, and vice-versa, wrapping DynaDojo Systems and Algorithms for OpenAI Gym.
To further broaden the scope and applicability of DynaDojo, we plan on introducing new Challenges of interest to the community. For example, we aim to incorporate an optimal control challenge involving stabilizing a system around a target trajectory, a transfer learning challenge focusing on fine-tuning to new system data, a multi-task learning challenge around maintaining consistent performance across multiple dynamical systems, and a curriculum learning challenge focusing on leveling up system complexity without retraining an algorithm from scratch. All of the Challenges can also be extended to measure prediction/control error on multiple timescales.
Lastly, in light of the emerging importance of scaling laws in deep learning, we hope to incorporate new scaling dimensions. These will include the number of model parameters, computational cost of training, and activation sparsity, on top of the three existing scaling dimensions. By including these dimensions, we aim to offer a more comprehensive view of how algorithms scale.
Using the Contributor Roles Taxonomy (CRediT)2:
- LMB contributed to the software (lead on design, documentation; support on algorithms/systems), data curation (equal on demo notebooks).
- TB contributed to the visualization (lead on creative direction; co-lead on figures), software (lead on algorithms/systems; support on documentation), data curation (equal on demo notebooks), writing (editing).
- CT contributed to the writing (co-lead), software (lead on reproducibility, parallelization, testing; support on design), visualization (co-lead on figures), and led the investigation, validation.
- RM contributed to the software (support on algorithms/systems).
- MK led the conceptualization, supervision, project administration, methodology, and contributed to the writing (co-lead), software (all-around support).
LMB was supported by Stanford's CURIS program. TB was supported by Stanford's Bioengineering REU program. MK was supported by an NSF Graduate Research Fellowship. The authors thank Stanford's Brains in Silicon Lab, Northwestern's Delta Lab, and Sean Yoon for helpful feedback on the writing. This work used Stanford's Sherlock and Northwestern's Quest clusters to collect results.
[1] Oliver Nelles. Nonlinear dynamic system identification. Springer, 2020.
[2] Greg Brockman et al. "OpenAI Gym". In: CoRR abs/1606.01540 (2016). arXiv: 1606.01540. URL: http://arxiv.org/abs/1606.01540.
[3] Karl Cobbe et al. "Leveraging Procedural Generation to Benchmark Reinforcement Learning". In: CoRR abs/1912.01588 (2019). arXiv: 1912.01588. URL: http://arxiv.org/abs/1912.01588.
[4] Florent Bonnet et al. "An extensible Benchmarking Graph-Mesh dataset for studying Steady-State Incompressible Navier-Stokes Equations". In: ICLR 2022 Workshop on Geometrical and Topological Representation Learning. 2022. URL: https://openreview.net/forum?id=rqUUi4-kpeq.
[5] Jean-Philippe Noel and Maarten Schoukens. "Hysteretic benchmark with a dynamic nonlinearity". In: Workshop on nonlinear system identification benchmarks. 2016, pp. 7–14.
[6] Max Schüssler, Tobias Munker, and Oliver Nelles. "Deep Recurrent Neural Networks for Nonlinear System Identification". In: 2019 IEEE Symposium Series on Computational Intelligence (SSCI). 2019, pp. 448-454. DOI: 10.1109/SSCI44817.2019.9003133.
[7] Max Schüssler, Tobias Münker, and Oliver Nelles. "Local Model Networks for the Identification of Nonlinear State Space Models". In: 2019 IEEE 58th Conference on Decision and Control (CDC). 2019, pp. 6437-6442. DOI: 10.1109/CDC40024.2019.9028945.
[8] William Gilpin. "Chaos as an interpretable benchmark for forecasting and data-driven modelling". In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021. URL: https://openreview.net/forum?id=enYjtbjYJrf.
[9] Luā Streit, Vikram Voleti, and Tegan Maharaj. “Understanding Generalization and Robustness of Learned Representations of Chaotic Dynamical Systems”. In: ICML 2022: Workshop on Spurious Correlations, Invariance and Stability. 2022. URL: https://openreview.net/forum?id=sJg-BRQcLxr.
[10] Karl Otness et al. "An Extensible Benchmark Suite for Learning to Simulate Physical Systems". In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). 2021. URL: https://openreview.net/forum?id=pY9MHwrmrymR.
$^{2}$ https://credit.niso.org/
[11] Jayesh K Gupta and Johannes Brandstetter. "Towards Multi-spatiotemporal-scale Generalized PDE Modeling". In: arXiv preprint arXiv:2209.15616 (2022).
[12] Makoto Takamoto et al. "PDEBench: An Extensive Benchmark for Scientific Machine Learning". In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022. URL: https://openreview.net/forum?id=dh_MkXOQfrK.
[13] Rui Wang et al. "Learning dynamical systems requires rethinking generalization". In: NeurIPS 2020 Workshop on Interpretable Inductive Biases and Physically Structured Learning. 2020. URL: https://www.amazon.com/science/publications/learning-dynamical-systems-requires-rethinking-generalization.
[14] Marcel Aach, Jens Henrik Göbbert, and Jenia Jitsev. "Generalization over different cellular automata rules learned by a deep feed-forward neural network". In: CoRR abs/2103.14886 (2021). arXiv: 2103.14886. URL: https://arxiv.org/abs/2103.14886.
[15] Karl Cobbe et al. “Quantifying generalization in reinforcement learning”. In: International Conference on Machine Learning. PMLR. 2019, pp. 1282–1289.
[16] Alex Nichol et al. "Gotta learn fast: A new benchmark for generalization in rl". In: arXiv preprint arXiv:1804.03720 (2018).
[17] David Silver et al. "A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play". In: Science 362.6419 (2018), pp. 1140-1144. DOI: 10.1126/science.aar6404. eprint: https://www.science.org/doi/pdf/10.1126/science.aar6404. URL: https://www.science.org/doi/abs/10.1126/science.aar6404.
[18] David Silver et al. "Mastering the game of Go with deep neural networks and tree search". In: nature 529.7587 (2016), pp. 484-489.
[19] Tianhe Yu et al. "Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning". In: CoRR abs/1910.10897 (2019). arXiv: 1910.10897. URL: http://arxiv.org/abs/1910.10897.
[20] William Gilpin. "Cellular automata as convolutional neural networks". In: Phys. Rev. E 100 (3 Sept. 2019), p. 032402. DOI: 10.1103/PhysRevE.100.032402. URL: https://link.aps.org/doi/10.1103/PhysRevE.100.032402.
[21] Alexander New et al. "Tunable complexity benchmarks for evaluating physics-informed neural networks on coupled ordinary differential equations". In: 2023 57th Annual Conference on Information Sciences and Systems (CISS). IEEE. 2023, pp. 1-8.
[22] Andrei Kramer, Ben Calderhead, and Nicole Radde. "Hamiltonian Monte Carlo methods for efficient parameter estimation in steady state dynamical systems". In: BMC bioinformatics 15 (2014), pp. 1-11.
[23] Chunyuan Li et al. "Measuring the Intrinsic Dimension of Objective Landscapes". In: International Conference on Learning Representations. 2018.