Factors Influencing Generalization in Chaotic Dynamical Systems
Authors: Luã Streit
Vikram Voleti
Tegan Maharaj
Venue: Submitted to ICLR 2023 (2023)
Keywords: generalization
multi-task learning
chaos
dynamical systems
representation learning

Abstract: Many real-world systems exhibit chaotic behaviour, for example: weather, fluid dynamics, stock markets, natural ecosystems, and disease transmission. While chaotic systems are often thought to be completely unpredictable, in fact there are patterns within and across that experts frequently describe and contrast qualitatively. We hypothesize that given the right supervision / task definition, representation learning systems will be able to pick up on these patterns, and successfully generalize both in- and out-of-distribution (OOD).
Thus, this work explores and identifies key factors which lead to good generalization. We observe a variety of interesting phenomena, including: learned representations transfer much better when fine-tuned vs. frozen; forecasting appears to be the best pre-training task; OOD robustness falls off very quickly outside the training distribution; recurrent architectures generally outperform others on OOD generalization.
Our findings are of interest to any domain of prediction where chaotic dynamics play a role.

Under review as a conference paper at ICLR 2023
FACTORS INFLUENCING CHAOTIC DYNAMICAL SYSTEMS
GENERALIZATION
IN
Anonymous authors
Paper under double-blind review
ABSTRACT
Many real-world systems exhibit chaotic behaviour, for example: weather, fluid dynamics, stock markets, natural ecosystems, and disease transmission. While chaotic systems are often thought to be completely unpredictable, in fact there are patterns within and across that experts frequently describe and contrast qualita- tively. We hypothesise that given the right supervision task definition, represen- tation learning systems will be able to pick up on these patterns, and successfully generalize both in- and -uito--distr (OOD). Thus, this work explores and identifies key factors which lead to good generalization. We observe a variety interesting phenomena, including: learned representations transfer much better when fine-tuned vs. frozen; forecasting appears to be the best pre-training task; OOD robustness falls off very quickly outside the training distribution; recurrent architectures generally outperform others OOD generalization. Our findings are of interest to any domain of prediction where chaotic dynamics play a role.
INTRODUCTION
There are many reasons to be interested in understanding and predicting behaviour of chaotic sys- tems. For example, the current climate crisis arguably the most important issue of our time. From atmospheric circulation and weather prediction to economic and social patterns, there are chaotic dynamics in many data relevant to mitigate impact and adapt to climate changes. Most natural ecosystems exhibit chaos; better understanding of the mechanisms of our impact ment is essential to ensuring a sustainable future on our planet. The spread of information in social networks, many aspects of market economies, and the spread of diseases, all have chaotic dynam- ies too, and of course these are not isolated systems they all interact in complex ways, and the interaction dynamics can also exhibit chaos.
environ-
This makes chaotic systems a compelling challenge for machine learning, particularly representation learning: Can models learn representations that capture high-level patterns and are useful across other tasks? Which losses, architectures, and other design choices lead to better representations? These are some of the questions which we aim answer. Our main contributions are:
The development of a lightweight evaluation framework, ValiDyna, to evaluate represen- tations learned by deep-learning models in new tasks, new scenarios, and new data.
The design of experiments using this framework, showcasing its usefulness and flexibility.
A comparative analysis of popular deep-learning architectures using these experiments.
Table 1: Summary of the generalisation results. S. C and F stand for the tasks of Supervised featurisation, Classification, and Forecasting. -4 and + indicate strict (see section (.2) and loose (see section 5.3) feature-transfer from task A to task B. All runs generalise in-distribution. V and indicate whether or not the model-run pair achieves OOD generalisation in the final task.
model
S
C
F
F
S
C
F C S C.
F F C C-F
GRU LSTM Transformer N-BEATS
V
V
V
V
Y
V
/
V
V
/

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Many works have studied factors influencing generalization for deep networks; see Maharaj (2022) for review, and Arjovsky (2021) for OOD specifically. To our knowledge, ours is the first such analysis for data exhibiting chaotic dynamics.
Our work relies on that of Gilpin (202I), which presents dataset of dynamical systems that show chaotic behaviour under certain conditions. They benchmark statistical and deep-learning models typically used with time series for variety of tasks including forecasting and tataset- and highlight some connections between model performance and chaotic properties.
Although not directly addressing chaos, the intersection of physics physics-informed informed and dynamical sys- tems literature with representation learning holds relevance for chaotic dynamies, e-g. |Raissi et al. 2019) show how to train models whose predictions respect the laws of physics, by employing partial differential equations as regularisation. Yin et al. (2022) propose framework to learn con- textual dynamics by decomposing the learned dynamical function into two components that capture context-invaree and copeext-specific patterns.
As Al systems are increasingly deployed in the real world, researchers have increasingly noted short- comings of standard practice (i.e. performance on validation/test set) for comprehensively evaluating learned representations. An increasing number of evaluation frameworks have been proposed to help address this, e.g. Gulrajani & Lopez-Paz (2021) propose model selection algorithms and develop framework (DomainBed) for testing domain/OOD generalisation. Of particular relevance, WWang et al.) (2020) discuss the difference between generalisation to new data domains and to new ODE parameters in the context of dynamical systems. They show that ML techniques generalise badly when the parameters of a test system/data are not included in the train set (extrapolation).
3 DATA
Our data is generated using dysts, Python library of 130+ chaotic dynamical systems published by Gilpin (2021). In dysts, each dynamical system can be integrated into a trajectory with any desired initial condition, length and granularity, thus allowing generate an unlimited number of trajectories. It can also generate trajectories of similar time scales across different chaotic systems. See Figure for examples and Figure A9 for further examples.
Arnedoo
Figure Sample trajectories from two re- lated chaotic attractors. Both systems have two 'lobes'; Arneodo (Ieft) has a characteris- tic shell shape with one lobe inside the other, while Lorenz (right) shows characteristic butterfly shape with lobes at an angle one another. This the kind of high-level pattern experts describe for many real- world chaotic systems, which we hypothesize representa- tion learning systems could pick up on.
95
85
0.5
10
10
-10_05
-05
-35
-1.2
-1.0
0z
(2g
2
P
by
02
de
0=
0C Ty
8z
3.1 THE DATA GENERATION PROCESS
We sample data from each dynamical system by picking different initial conditions. This leads to trajectories that are sufficiently different from each other, but representative of the underlying chaotic system. However, dysts relies on numerical ODE solvers to generate trajectories, which could fail due to numerical instabilities when the initial condition is too extreme. To avoid that, we generate the default trajectory for each system, compute the component minima and maxima, and use percentage p of the resulting intervals sample random initial conditions for that system. In addition to the properties of the trajectory. the parameters this process are the random seed and the percentage p of the observed initial condition range be used for sampling.
2

---

Under review as a conference paper at ICLR 2023
3.2 OUR DATASETS AND THEIR PARAMETERS
We generate three sets of data for all our experiments: training, validation, and test. The training set used to optimise the model weights. The validation set is used for early stopping and learning rate adaptation (see Appendix A.3), and to measure in-distribution generalisation. The test set is used to measure OOD generalisation.
The train and validation sets come from the same data distribution, while the test set comes from larger distribution containing the former. All the sets contain trajectories with the same length (5 periods) and granularity (50 points per period). The parameters used to generate the data can be found in Table A2 We choose to only include dysts systems of 3 dimensions in the datasets (i.e. 100 out of 131 systems, ef. Table A3 to avoid adapting models for variable input dimensions, and for faster training. The default trajectory from each included system can seen Figure A9
THE VALIDYNA EVALUATION FRAMEWORK
The exploratory and comparative nature of this work results in the need to have common experi- mental framework for consistency and configurability across experiments, including different tasks and combinations of losses. We present ValiDyna, open-source, lightweight framework built on top of Pytorch and Lightning. It is built with extensibility in mind, so that new model architectures, metries and training objectives can be easily added. The framework saves a large amount of code repetition and complex indexing/references, e.g. in multi-task experiments.
4.1 TASKS
ValiDyna currently includes three tasks on learned representations from with time series data:
1. (Task S) Self-supervised featurisation (aka feature extraction) involves extracting features from time series such that similar time series have similar features, similarity defined as coming from the same dynamical system. We use triplet margin loss, which takes 3 features as input: that of an anchor time series a, a positive series p similar to it, and negative (dissimilar) series nı:
Lniiter( p, n)) = (nax(d(a,))) - d(a, n)) + m,,)
where distance metric (euclidean our case) and mn the margin of tolerance, i.e. the minimum difference between the positive and negative distances for the loss to be non-zero. The number of features to be extracted and the margin value are the main parameters of this task.
2. (Task C) Classification involves predicting single discrete class for each time series, in case the chaotic system from which it came. We use cross-entropy loss to measure how close the model's output is to the true class. The main parameter is the data-dependent number of classes.
3. (Task F) Forecasting, perhaps the most popular task for time-series, involves predicting the future values of time series based on its past values. We use the mean squared error (MSE) loss. Although the number of time steps in the past and in the future need not necessarily be fixed, we do so due to N-BEATS' architecture (ef. Section 4.2). Thus, this task is parameterised by the number Tip of time steps that are input to the model, and the number Tour of time steps output by the model.
Each of these tasks is implemented in ValiDyna as a separate Lightning module ([Slice] Featuriser, Classi fi er and Forecaster) that wraps around a model architecture to allow for easy training and metric logging. All such modules log the corresponding loss during training for all data sets, while the Classi fier module additionally logs the classification accuracy.
4.2 MODEL ARCHITECTURES
ValiDyna currently includes 4 machine learning architectures often used for temporal data:
GRU (Cho et al. 2014) and LSTM (Hochreiter & Schmidhuber 1997): these Recurrent Neural Networks (RNNs) are likely the most popular MI. architectures be used for time series as they allow crunching series of variable size into fixed-size representations.
Transformer VVsswam et al.) 2017): an attention- architecture that achieves state-of- the-art performance for seq2seq, and has replaced L.TMMs in many time series tasks.
3

---

Under review as a conference paper at ICLR 2023
N-BEATS (Oreshkin al. 2020): a purely deep neural ofate-of-the-ar forecasting archi- tecture based on residual blocks. Originally written in TensorFlow, we provide a Py ToTchh implementation based on that of Herzen al. (202I).
The main challenge of our multi-task setup involves adapting the model architectures above for tasks they were not originally built for. The most straightforward way is to use the architecture (or part of it) as a feature extractor, and then attach a classification or forecasting head. For RNNs, we consider the outputs of the last layer as the "features". For the Transformer, we use its encoder as feature extractor, and completely discard the decoder. For N-BEATS, we choose the concatenation of the forecast neural basis expansions of all blocks the "features".
To ensure fair comparisons, the framework makes it easy to ensure a fixed number of features (Nreatures) across model architectures. To accomplish this, we insert simple linear layer with N Vieatures output units between the vanilla feature extractor and the task-specific heads. These models are implemented in ValiDyna as sub-classes of a Mul TaskeMdeel with all of the functionality above, as shown in Figure 2
For further details on the framework, see Appendix A.I
EXPERIMENTS
Input data
We now present some experiments that help us better under- stand how different deep-lcarning models represent chaotic data, and showcase the ValiDyna framework. For all experi- ments, we show table with the metrics achieved by the var- ious model runs during the last epoch training, and show the corresponding training curves in the Appendix. The con- figuration values used across experiments are listed Ap- pendix A.3 for the sake of transparency and reproducibility.
8x Tm
multi-task model feature extractor
"natural" feature extractor
8 Natural
linear layer
5.1 RANDOM SAMPLING
Meatures
In this baseline experiment, we measure the dependence of model performance on the specific set of trajectories used for training. We construct 5 subsets (using the random seeds O to 4) sampling 75 % of the available trajectories in each set without replacement. For each sub-sampled set, we train each model architecture for each of the 3 tasks. The results in Ta- ble also serve as performance baseline for the more com- plex experiments that follow.
classification head
foracasting head
x Nelecana
x T X
Schematics of a multi-task model. B=batch size, D=space dimen- sion, T=number of time steps. Numurat depends on model hyper-parametert but Nicaturos can be chosen. N-BEATS needs no separate forecasting head.
Figure 2:
Table 2 shows the following: N-BEATS consistently per- forms poorly for fcaturisation and classification (an alternate featuriserfoorecatter decomposition could potentially perform better); OOD) forecasting generalisation is bad overall; GRU, LSTM and Transformer perform better on the validation set than on the train set (possibly an effect of dropout regularisation).
The training curves in Figure A2 show that: different seeds result in different training times although final performance is stable; test curves are noise-like (no improvement) for N-BEATS on all tasks, and for forecasting with all models.
5.2 STRICT FEATURE- TRAUREFETRANFEER ACROSS TASKS (FROZEN WEIGHTS)
With this experiment, we seek to evaluate the usefulness of transferring learned representations from of the three tasks to another. We expect forecasting to rely implicitly learning the system of time series to generate better predictions, and hope that pre-training for tasks that do this explicitly will be beneficial. Moreover, since classification and featurisation try to achieve very similar goals, we expect that training a model for one is beneficial for the other.
In each
1. pre-train model for one task (the pre-training task); 2. freeze the parameters associated to the feature extractor (cf. Figure 3, train for another task (the main task). Note that featurisation cannot be main task freezing the featuriser results in no learning.
4

---

Under review as a conference paper at ICLR 2023
Table 2: Random sampling experiment: final metric means and standard deviations aggregated over the 5 different random sampling seeds. See Table A4 for the full results per sampling seed. We highlight the best mean value obtained on each set. N-BEATS performs poorly for classification and featurisation. GRU is consistently among the best performers, LSTM is close behind. Validation metrics are equal or better than the train ones for all models but N-BEATS. Generalisation to the OOD test set is generally good, except for forecasting.
set
train
validation
test
set
train
validation
test
GRU
.12::.099
.094.008
.104.008
GRU
5.1+.32
4.1+.23
1e
LSTM
.12:t0099
.10:t.009
.11+:009
LSTM
5.2:
44:34
2.4.+8:42.344
Transformer
.25+0005
.21+0001
.21+0002
Transformer
9.14.94
8.641.8
1.9e+8+9..55
N-BEATS
.48..035
.48+..04
1.60+1.800
N-BEATS
7=0.34
7.2+.34
1.5e+8
(a) (Task S) Supervised featurisation loss (1)
(b) (Task F) Forecasting loss x 1000 (4)
set
train
validation
test
set
train
validation
test
GRU
.281.036
.21=.032
.33::033
GRU
92:11.1
95=0,9
934.89
LSTM
.31+.052
.20=:042
.36::.06
LSTM
91:1.5
9.21.2
92+1.2
Transformer
.52+.050
.50=.055
.65=t.022
Transformer
85::1.6
86=2.11
85:22.1
N-BEATS
2.30:10099
2.30+.041
.70=2.522
N-BEATS
40::0.6
402.66
391.75
(c) (Task C) Classification loss (1)
(d) (Task C) Classification accuracy % (t)
Table Feature-freeze experiment:
final task metrics as a function of the pre-training task. We highlight the best metric value obtained for each model-dataset pair.
In general, pre-training on other tasks results worse performance. Interestingly, the representa-
tions learned by Transformer during forecasting seem transfer well to classification. Freezing the feature extractor entirely seems to prevent learning.
set
train
validation
test
pre-task
C
C
S
S
GRU
37
35.5
4.8
28
27.7
3.8
1.35e+8
1.42e+8
1.6e+8
LSTM
41.4
40.8
4.75
32.1
32
3.62
1.32++8
1.42c+8
1.56c48
Transformer
13
21.4
8.51
12.6
19.8
7.96
1.16e+8
9,48c+7
1.18e+8
N-BEATS
170
236
6.22
169
237
6.36
1.12248
8.3e47
3.88e+7
(a) (Task F) Forecasting loss 1000 (1)
set
train
validation
train
validation
test
pre-task
S
F
S
F
S
F
S
F
S
F
GRU
1.64 1.15 .178
1.31
913
.117
1.41
1.08
.267
52.6
65.7
94.7
63.2
73.5
97
61.9
72 95.5
LSTM
1.96
1.45
.354
1.61
1.13
2245
1.76
1.26
.378
44.4
57.4
89.5
54.6
67.8
93.5
53.1
66.3
91.5
Transformer
1.44
.448
529
1.11
.41
448
1.19
.655
.648
55,8
87.9
84,4
66.1
87.5
87.8
64.7
85.6
85.7
N-BEATS
2.28 3.37 2.03 2.28
3.38
2.04
21.6
3.45
43.7
38.7
19.1
46.1 38.6
19
46
37.3
18.5 44.7
(b) (Task C) Classification loss (L)
(c) (Task C) Classification accuracy % (f)
Table that in almost all cases, pre-training other tasks results in worse performance. The only exception when pre-training Transformer for forecasting and then classifying, but even then GRU and LSTM still achieve better classification accuracy with pre-training. Figure A3 shows that pre-training for other tasks puts models a better initial position during the main training, but then performance stops improving and ends up worse than without pre-training, Given this initial performance boost, we speculate that learned features are useful across tasks, but freezing the featuriser is too extreme and prevents learning during the main training phase.
5.3 PROBING FOR OTHER TASKS (FINE-TUNING)
The goal of this experiment is to better understand how training a model for one task impacts its performance other tasks. The flexibility of our framework allows write this experiment in simple manner, by implementing a new "prober" metric that actively treats a task module A as if were the module for task B, and logs metrics for B.
5

---

Under review as a conference paper at ICLR 2023
It is to be noted that to probe for some tasks, model must have already been pre-trained for that task. For instance, one cannot probe for classification or forecasting during featurisation training. However,
probe for fcaturisation while training for other tasks, as the featuriser is used for those tasks (except N-BEAT'S which does not train its featuriser layer). A side-effect of this experiment is that solves the main limitation of the previous one, by allowing the transfer of features across tasks without freezing any component of the model.
task(s)
set
clas.
train
cles,
+> feat
val
fore.
feat.
Triplet loss
N-BEATS
Transformer
107
Iriplct loss
10a
10
8
10
12
14 spoch
16
18
20
22
24
26
28
0
2
8
10
12
14 epoch
16
18
20
22
24
26
28
Figure 3: Prober experiment: training curves per model and pre/main task combination. A running average length 500 (roughly a quarter an epoch) is used for readability. The best task performance is obtained when the model is being trained for that specific task. Metrics of tasks different to the training task seem to stay stable during training.
First, we look at task performance using pre-training, extending the observations of Section 5.2 Table show that: the best classification performance with forecasting pre-training all models except N-BEATS perform forecasting at least as well on the train/val sets with classification pre- training; featurisation loss is generally better with pre-training than without (ef. Table 2a baseline).
Then, for task probing, Sable|showee that the classification metrics are consistently bad during the training of other tasks: accuracy Rs 1% (the random baseline), and the loss is an order of magnitude higher than its counterparts on all sets. This is not surprising. Consider C - F, as the featuriser is updated during forecasting training, it is no longer compatible with the classification head.
Considering the evolution of metrics during training, Figure 3 shows that the performance of task when training for task B is initially good, but either collapses after one or two epochs, or stays stable. This agrees with our theory that the featuriser and task-specific heads become incompatible.
In summary, this experiment shows that pre- mreitiannig on other tasks can be greatly beneficial for another task, likely because it places the model parameters in region of space that is casier to optimise. However, the update of the featuriser weights during training renders the previously- trained task-specific heads useless they cannot adapt the new features that they receive.

---

Under review as a conference paper at ICLR 2023
Table 4: Prober experiment: final metrics per pre-training task, training task, model and dataset. The best metric value is highlighted for each model-set pair. Forecasting features transfer well to classification and the reverse is often true. Classification and forecasting features transfer decently to featurisation (cf. baseline in Table 2ap.
se1
train
validation
test
train
validation
test (=100)
pre-task
C
F
C
F
C
F
C
F
C
F
task
S
S
F
S
S
C
F
S
S
C
F
F
C
S
C
F
F
GRU
.08 .09
84
06
.7
84
.08
.1
.85
.46
46
28
28
36
34
28
28
16
16
13
16
LSTM
.09 .08
.79
00
.63
8
64
.42
42
27
27
33
32
27
28
16
16
13
17
Transformer
.24 .28
1.2
81
11
1.3
.22
.24
1.3
.81
80
28
28
76
28
13
12
13
9.9
N-BEATS
.45
.44
1.1
.99
45
44
1.1
.99
4.1
2.6
8.5
.60
2
28
42
.6
2
28
42
2.7
8.8
13
13
(a) (Task S) Supervised featurisation loss (4)
(b) (Task
F) Forecasting loss
X 100
(4)
set
train
validation
test
train
validution
test
pre-task
C
C
F
C
F
C
F
C
F
task
C
S
F
C
S
F
C
C
S
F
C
C
S
F
C
C
S
F
C
C
F
C
GRU
21
4.7
4.8
12
15
4.7
4.8
08
29
4.7
4.8
23
93.8
96.3
96.3
96.3
1
1.01
97.9
94.6
1.01
1
96.6
LSTM
29
4.7
4.6
15
21
4.7
4.6
09
35
4.7
4.6
21
91.4
95.6
94,5
101
1.01
97.8
92.8
1
96.5
Transformer
45
6.3
4.7
.7
63
6.3
85.6
0.97
89.2
0.97
85.7
0.99
N-BEATS
2
4.8
4.7
4.8
4.7
1.
38
4.8
4.7
45.9
1.84
50.2
45.7
1.84
50.1
44.3
1.79
0.99
48.5
(c) (Task C) Classification loss (L)
(d) (Task
C) Classification accuracy % (f)
5.4 FEW-SHOT LEARNING
With this experiment, we hope to better understand how our models adapt to a distribution shift consisting of new environment with a new chaotic system in it. We focus on the dynamical system SprottE. We consider two sets of systems: set of toy chaotic systems with simple equations similar to SprottE's s (Sproti 1994): SprottA, SprottB, SprottC and SprottD; and a set of 4 systems with more complex diffcrential equations: Arneodo, Lorenz, Sakarya and QiChen. We show the default trajectory and differential equations of each system in Figure AS
The experiment is set as follows. In some cases, pre-train a model on one set of systems (similar or different), then add SprottE to the data and train fully. In other cases, train models directly on set of 5 systems (SprottE+ + similar/different). Runs are identified by the similarity of the other systems (similar vs. different), and by whether SprottE is included ("no" during pre-training, "no "no->yes" after pre-training, and "yes" when SprottE is there from the beginning).
Since care about model performance SprottE in partieular, we introduce new metrics: for forecasting, MSE loss only on SprotE i.e. S-MSE; for classification, sensitivity i.e. true positive rate (TPR) and specificity i.e. true negative rate (TNR) of SprouE vs the other classes; for featurisation, the standard deviation of the features extracted from SprottE series. Note that these metrics can only be tracked when SprottE is included in the data (i.e. not during pre-training!
Table 5) confirms our choice of similar and different systems: classification accuracy and featurisa- tion loss are better for different systems (i.e. easier differentiate), while forecasting loss is better for similar (i.e. reusable representations). Pre-training does not have significant impact classi- fication or forecasting metrics, but is better for featurisation, in particular for different systems. This could be due to the triplet margin loss needing more samples to be optimised than the two others.
Figure A6 mainly shows that convergence is faster for pre-trained models, as expected. There does not seem to be any relationship between SprottE feature standard deviation and performance.
In this experiment, we also visualise the features learned by the models using 2D PCA projections. Figure 4 shows features learned under the feature extraction task for the 4 different architectures. Full plots for all settings in Figure All
7

---

Under review as a conference paper at ICLR 2023
Table 5: Few-shot learning experiment: final metrics per training task, system similarity (= or *), and SprottE status, averaged over the train/validation/ees sets. See Table A5 for the full table. Forecasting MSE is significantly better for the similar systems. Featurisation loss and classification accuracy are better for the different systems.
Sprott??
yyes
yes
no-tyes
yes
systems
7
7 MSE
MSE
MSE
S-MSE
MSE
S-MSE
MSE
S-MSE
MSE
S-MSE
GRU
26
.76
44
3.3
1.1
2.3
1.6
.78
1.3
1.8
1.1
.87
1.1
LSTM
.38
2
.29
57
1.6
4.4
1.2
2.2
1.4
1.1
1.7
2.7
1.6
1.1
1.3
Transformer
.63
3
42
2.3
.54
2.4
2
.54
1.7
.94
.7
1.2
2.4
1.4
.98
1.3
N-BEATS
2.7
3.8
2
4.5
5.2
4.1
1.8
.68
2.1
1.4
.63
.97
2.2
1.4
.91
1.2
(a) (Task Feat. loss
110 (4)
(b) (Task Forecasting metrics x 1000 (L) (S-MSE SprottE MSE)
SprotE?
no >yes
yes
no
no -yes
yes
systems
#
acc
TNR TPR
acc
TNR TPR
acc
TNR
TPR
acc
TNR
TPR
GRU
1.10 .800
1.5 0.47
100 100
100
100
100
100
100
99
100
100
100
100
100
100
LSTM
0.61 0.68
1,4 0.91
100 97
100
100
100
100
100
98
100
100
100
100
100
100
Transformer
0.88 0.45
0.8 0.81
100 98
100
100
100
100
100
99
100
100
100
100
100
100
N-BEATS
1.50 1.60 1.8 0.44
99 97
99
93
97
100
99
98
100
97
95
100
99
94
(c) (Task S) Featurisation std
(d) (Task C) Classification metrics % (1)
sisder 5t you 57.8%)
viriler 57
simiirr (57 you 99.9%)
simsdar 50 yos 75.0%)
LSTM
GRU
NBEATS
TRANSFORMER
1.0
0.5
0.0
10
1.5
Figure 4: Comparison of the features extracted by the 4 different architectures. Note the effective low dimensionality of the N-BEATS features compared to the others. In these examples and in general, recurrent architectures LSTM and GRU appear to have the most separable learned features.
very noticeable result is that the two principal axes of the PCA projections of the features gener- ated by N-BEATS always explain at least 95% of the feature variance (we call this value 'r'). We speculate that, although the feature extractor of N-BEATS has an output dimension of 32, its effec- tive number of degrees of freedom (i.e. its effective capacity) is much lower, around 2, which would explain why features for all systems are mixed a linear or "V" shape of most PCA projections.
5.5 COMBINING TASK LOSSES
In this experiment, we explore optimising the three task losses simultaneously. Concretely, we implement a new Sl ceModule whose loss is weighted mean of all task losses. In particular, we consider forecasting as a main task, and use the other losses to explicitly enforce the learning of series' system, by setting the weights Lital aLMSE + a) (Lurpll L.wos).
Figure A10 shows no evident benefit from enforcing shared representations across tasks.
LIMITATIONS AND FUTURE DIRECTIONS
Despite ValiDyna's configurability and extensibility, it has some limitations inherent in choosing experimental scope:
Limitations:
It is built with time series data at its core, and would not work with other types of data such as static images (e.g. forecasting is time series-specific).
While adding new architectures is simple, adapting new models to multi-task learning requires some expertise.
8

---

Under review as a conference paper at ICLR 2023
There many ways to extract features for some of the model architectures (e.g. for N-BEATS), and the current version only support a single scheme per model architecture.
It only supports training models on single dataset, and has no way of distinguishing between different training environments corresponding for example to different ecosystems.
There are also limitations in our experiments:
They done for particular timescale of data, and might differ substantially when consider- ing coarser or finer granularities. This a challenge in general for climate data as some events occur in large timescales and others in smaller ones, and climate models must account for both.
Out-of-distributie generalisation was only explored in the context of extrapolation initial con- dition of trajectories.
Although the dysts library allows generating noisy trajectories, the data used in the experiments is free of noise, while real-world measurements often contain noise.
Apart from evaluating further architectures and adding new losses to the existing few ways in which would like to expand our framework.. While initial experiments with meta-learning were not promising (this is why we did not focus the framework around them), adding the ability to do multi-dataset losses such as meta-learning losses is something we would like to pursue. We hypothesize that the benefits of this approach might require massively multi-environmeen settings show. We would also like exploring generative modelling and the role it can play in encouraging good representations.
Future directions:
framework, there
We would also like explore further experimental settings, such mixing different data timescales, dynamical system parameters (e.g. changing 3 in Lorenz), and trajectory noise levels.
Finally, of ongoing work is perform comparison with real ecological measurements and see how the results differ from the synthetic case.
7 CONCLUSION
In summary, we present an experimental analysis of factors influencing generalization for data ex- hibiting chaotic dynamics. To do so, built configurable and extensible model evaluation frame- work called Validyna. Using Validyna, we constructed and ran five experiments - random sam- pling, transfer learning with frozen weights, fine-tuning with unfrozen weights (probing), few-shot learning, multi-task loss - to better understand the quality of representations learned by four pop- ular machine learning architectures GRU, LSTM, Transformer, N-BEATS on three tasks feature extraction, classification, forecasting - for in- and out-of--istribun generalization.
Takeaways. Summarizing our extensive experiments, the main takeaways from our work are:
All four model architectures generalise poorly to unseen data distribution for the forecasting task. This likely due to the chaotic nature our data.
Our feature extractor for N-BEATS performs very poorly, while the others perform better.
All four model architectures robust data sub-sampling in the sense that their final perfor- mance is stable, but training times can vary considerably.
Dropout seems be an effective regularizer for in- and -i-tt--diitiitttion generalisation.
Learned representations can transfer well across tasks, especially from forecasting to classifica- tion, but not when the feature extraction module is frozen.
There no straightforward relationship between optimising for the triplet or cross-entropy loss, although they try to achieve very similar goal.
The cross-entropy loss and the classification accuracy of a model do not necessarily follow each other when models optimised for other losses.
There is no evident benefit from enforcing shared representations across tasks.
These results provide insights and starting points for future research representation learning of chaotic dynamical systems.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky. Out of Distribution Generalization in Machine Learning. PhD thesis, New York University, 2021.
KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. CoRR, abs/1409. 1259, 2014. URL.
http://arxis> wttp/////////..///////// 1259
William Gilpin. Chaos as an interpretable benchmark for forecasting and data-driven modelling.
arXiv:2110.05200 [nlin], October 2021. URL httpppp//aaxi..orr///////1113333 arXiv: 2110.05266.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. URU/ttpps://peezevierr www.nnetfoorrne
1d=1@dxeXDoott.
Julien Herzen, Francesco Lässig, Samuele Giuliano Piazzetta. Thomas Neuer, Léo Tafti, Guillaume Raille, Tomas Van Pottelbergh, Marek Pasieka, Andrzej Skrodzki, Nicolas Huguenin, Maxime Dumonal, Jan Košcisz, Dennis Bader, Frédérick Gusset, Mounir Benheddi, Camila Williamson, Michal Kosinski, Matej Petrik, and Gaäl Grosch. Darts: ''ser-Friendly Modern Machine Learning for Time Series. arXiv: I 10.03224 fes, stat], October 2021, URL htt.htttp///rxxi.crrrte
2110.03224 arXiv: 2110.03224.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735- 80, 12 1997. doi: http///cco.199..9...
Tegan Maharaj. Generalizing in the Real World with Representation Learning. PhD thesis, Montreal Polytechnique, 2022.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv: 10437 [cs, stat), February 2020. URL URL httt:://arxiii wwwprxivv....gvaae.
arXiv: 10437.
M. Raissi, Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions.
Journal of Computational Physics, 378:686 707, February 2019. ISSN 0021 19919 doi:
10. c.p0166j..pp.0118 2018 10.045. URL htttps://iiiiinguu.eeeeee... pi1/50021999118332
J. C. Sprott. Some simple chaotic flows. Phys. Rev. E, 50:R647-R650, Aug 1994. doi: 10. 1103/PhysReE..50..667. URL 50.R647
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. N0ti://706.3766 [cs], De- cember 2017. URL
wttp://r.gbbbb//...ooooo 03762 arXiv: 1706.03762.
Rui Wang, Danielle Robinson, Christos Faloutsos, Yuyang (Bernic) Wang, and Rose Yu. Learning dynamical systems requires rethinking generalization. In
Workshop Interpretable Inductive Biases and Physically Struc- tured Learning, 2020. URL http://www..................earr
NeurlPS 2020
amazon. http://wwww..acccoocc.ceeeeaaaaa
synll-systenne
Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. LEADS:
Learning Dynamical Systems that Generalize Across Environments. arXiv:2106.0455 [cs, stat], February 2022. URL http: arxiv.iiv..////v/a/aaaaaaaaaaavv wttp:///g///b//aaaa 04546
arXiv: 2106.04546.
10

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 ADDITIONAL DETAILS ON VALIDYNA EVALUATION FRAMEWORK
A.1.1 DATA PROCESSING
Entire trajectories cannot be fed to the models due to N-BEATS' limitation of requiring fixed number of input and output time steps in and Tut. Also, task-specific data i.c. positive and negative examples for the (S) featurisation task need to be constructed. And, although the generation process transparent to the different scales of system values (cf. Section 3.1, the generated trajectories still need to be scaled.
To address these issues, we: 1. compute the component-wist minima and maxima of the train trajectories of each system; 2. them to scale the trajectories per system in all 3 sets; 3. map each (scaled) trajectory of length N into the list of all the possible contiguous slices of length Tin + 4. split each such slice two parts of length in and Tour 5. attach the system name which generated it, encoded as a number. Thus, single data sample in our setting is triplet (Xin Xout, c) i.c. model input for all tasks, the true future for forecasting, and the class label for classification.
In addition, given a batch of anchor time series, the Featuriser module can retrieve a batch of one positive and one negative example per anchor.
A.1.2 FRAMEWORK ARCHITECTURE, EXTENSIBILITY
Introducing a new multi-task model to the framework is straightforward, as it suffices implement a new Mu t TaskMode I class and adapt the model to all available tasks. Adding new metrics to be tracked for specific tasks experiments is also very straightforward, and the process is briefly explained in the experiments of Sections 5.3 and [.44 Creating variations of current tasks can also be done quickly, c.g. adding Forecaster that optimises the mean average error (MAE) instead of the MSE is trivial. However, introducing new task can be complicated, as it requires: adapting all existing multi-task models it; implementing the core training objective in new $ $1 SliceModules adapting Mult t See Figure Al for a high-level class diagram.
and the data processing pipeline if new kinds of data are necessary.
A.2 EXAMPLE OF EXPERIMENT CODE
With our framework, one run for the striet feature-transfer experiment could be simply written as:
# Python 3.9 pseudocode
model: MultiTaskModel (RU dataset: MultiTaskDataset Pre-train for classification classifier Classifier (model) classifier. fit (dattate # Freeze feature extractor weights model. freeze_featuriaa Train model for forecasting forecaster Forecaster (model)
forecaster. fit (dataset)
A.3 EXPERIMENTAL SETTINGS
We use the values Tin = Tour for data processing and for the models. For each dataset, we shuffle the data and use batches of size 1024. After every training batch, we compute the metrics of interest for a random validation and test batch. All models are optimised using Py TyTrchhss 's Adam W optimiser, with starting learning rate of 0.01 that is divided by 5 when the validation loss does not improve, with a patience of epoch. All training procedures include early stopping, so that training stops when the validation loss stops decreasing, with patience of epochs. A maximum of 100
11

---

Under review as a conference paper at ICLR 2023
MutiTasilMode!
int
MUBRINI
McbILSTM
MueWBEATS
space_ closses: maybelint)
int
[_feetures: int
n. feeturee: maybefint)
n. out maybelini]
forecaster: maybefhead]
daesiler: maybe[]vead]
MAlIGUU
clsssifee:: maybe(nead]
forecaster: maybe(head)
dlsesllle: maybe[head]
Custom N-BEATS code
Uses PoToch RNNs
Torch''s Transformer
Handles muli- logic
Whaps
MuliToaDDataoe
around
Contains
train and validation
sample fin, out, dlass)
Legend:
abstrect clase
MllceMooute
Data proceesing
instanbable class
abstract loss)) metrica
Positive and negalive sempling
Model training logic
SliceClassifer
Data scaling
SliceFeelturzer
BliceForecaster
Metric logging
loss cross, entropy motrics accuraty
Sapple__marghe
mse
Figure A1: Class diagram of the ValiDyna Framework.
training epochs is allowed, but no training run attains it. Each model training run deterministically. by setting Lightning's s random seed to 2022.
We also want to make sure that our models have a comparable representative power, and we use the number of parameters a proxy. Table shows the number of parameters obtained for each model using the hyper-parametent that follow. All models use the value Nieauures 32. All forecasting and classification heads in our models are simple feed-forward neural networks with 3 hidden layers of width equal to Nieatures = 32, and ReLU activations after each hidden layer. For N-BEATS, we use 4 stacks of 4 blocks each, neural-basie expansion dimension of 4, and the fully-connected' network of each block has 4 hidden layers of 8 units each and ReLU activation. For the two RNNs, we use dropout probability of 0.1 and layers, with GRU having 30 hidden units per layer and LSTM 26. The Transformer uses 4 encoder layers, each having a feed-forward dimension of 6, as well attention heads, an embedding dimension of 16, and dropout probability of 0. in the feed-forward and self self-attention networks. The margin of the triplet loss is equal to the default of 1.
Table A1:
The number of parameters of each model architecture as used in the experiments. We consider the number of parameters used the featuriser and in total (including any task specific heads). The total parameter count is very stable (around 11-12k), while there is slightly more vari- ability for the featuriser one (around 7-9k).
model
GRU
LSTM
Transformer
N-BEATS
params (featuriser)
8730
8840
6816
7856
# params (total)
11933
11915
12079
11136
B EXTRA TABLES
Table A2: Parameters used to generate trajectories for each data set. Theoretically, the trajectories from the train and validation sets come from the same distribution, while those from the test set come from a larger distribution containing the other. *see Section 3.1
parameter
random seed
# trajectories
p"
periods
#
points persod
points
train
0
80
%
5
50
250
validation
20
50
250
test
2
30
10 %
50
250
12

---

Under review as a conference paper at ICLR 2023
Table A3: The distribution of attractor dimensions in dysts. Most attractors have dimension 3.
dimension
6
count
100
3
Table A4: Random sampling experiment: full final metrics per random sampling seed.
set
train
validation
test
seed
0
1
2
3
4
1
3
0
2
4
GRU
100
113
.16
.16
.18
.0812
.0917
.092
.0916
.103
.0897
.0991
.102
.113
LSTM
.18
.07
114
119
.131
.105
.0838
.101
.0949
.107
.114
.0936
.10
.16
.115
Transformer
.23
.257
.255
255
.255
.207
.203
2006
.206
206
.216
.209
.23
.213
.233
N-BEATS
.455
.529
.444
.495
.46
.458
.531
.447
.497
.464
2.92
2.19
5.37
.526
1.8
(a) (Task S) Supervised featurisation loss (4)
set
train
validation
test
seed
2
0
2
GRU
5.12
4.94
5.71
4.96
4.98
4.06
3.9
4.49
4.01
4.02
1.96++8
1.72c+8 2.36e+8 2.36e+8 2,36e+8
LSTM
5.34
4.64
5.07
5.94
4.89
4.2
3.65
3.98
4.54
3.87
2.36e+8
2.36e+8 2.36c48
2.36e+8
2,36e+8
Transformer
9.14
10.7
8.86
8.36
8.36
8.65
10.3
8.39
7.86
7.86
1.89e+8
2.03e+8 1.82c+8
1,82e+8
1.82e+8
N-BEATS
6.79
6.97
6.85
7.65
6.96
7.07
7.13
7.02
7.83
7.14
4. 12e+8 5.49c+7 3.46c+7
1.6e+8 7.84++7
(b) (Task F) Forecasting loss (1)
set
train
validation
test
seed
2
3
4
2
3
4
0
4
GRU
274
.274
.332
.232
225
.206
.198
.259
.17
.203
.32
.277
.379
.3
.336
LSTM
307
.377
.35
.366
.232
.23
2211
.267
.28
.172
.347
331
.394
.387
.283
Transformer
558
.488
.49
.598
.49
.486
.549
.43
555
.443
.642
.7
.603
.711
.603
N-BEATS
2.34
2.23
2.29
2.29
2.29
2.36
2.25
2.3
2.3
2.3
147
101
33.8
33.8
33.8
(c) (Task C) Classification loss (L)
set
train
validation
test
seed
2
3
4
3
4
3
4
GRU
.999
.988
.001
.93
.911
944
.948 .932
.956
.98
.932
.933
.96
.94
.932
LSTM
9088 .906
.96
.891
.93
.94
.938
.93
.926
.966
.924
.922
9914
.91
.942
Transformer
836 .855
.859
.823
.859
.66
.84
.883
.842
.883
.847
.823
.86
.824
.666
N-BEATS
.396 .413
.401 .401
.401
.394
.413
.402
.402
.402
.384 .403
.387
.387
.387
(d) (Task C) Classification accuracy (random baseline of 0.01) (f)
13

---

Under review as a conference paper at ICLR 2023
Table A5: Few-shot learning experiment: full final metrics.
SprotE?
no
no-
yes
attractors
*
metric
acc
acc
TNR TPR
acc
TNR
TPR TNR
acc
TPR
acc
TNR
TPR
acc
GRU
.999
.997
.999
.999 .996
.998
LSTM
.998
.973
.999
.999
.997 .98
1
1
.998
.997
train
Transformer
.993 .981
.998
.999
.999 .996
1
1
.998
.997
N-BEATS
.996 .972 .995
.932
.978
.997
.994 .982
.975
.953
.998
.993
.951
GRU
.999 .997
.998
.999 .995
I
.999
.998
LSTM
.998 .97
.997
.998 .979
1
1
.999
.998
validation
Transformer
.998 .983
.999
.999
.997
1
1
.999
.999
.997
N-BEATS
.991 .966
.993
.93
.971
.995
.992 .975
.999
.975
.948
.997
.99
.945
GRU
.994 .995 .999
.997
.998
.997
.993
.999
.999
.998
.966
.994
LSTM
.966 .969 .999
.997
.998
.995
.974
.999
.997 .996
.997
.993
Transformer
.996 .977
.997
.998
1
.995
.992
.977
.998 .966
.995
.993
N-BEATS
.933 .61 .92
.925
.962
.995
.99
.974
.988
.971
934
.999
.99 .936
(a) (Task C) Classification metrics (1)
SprottE?
no->yes
aftractors
*
metric
MSE
MSE
MSE
S-MSE
MSE
S-MSE
MSE
S-MSE
MSE
S-MSE
GRU
00196
.00455
.0014
.00139
0000933
.000974
.0011
.000991
.0011
.00103
LSTM
.00599
.00188
.00122
.00132
.00129
.00146
.00181
.00162
.00124
.00112
train
Transformer
0000006
.000567
.00065
0000533
000592
.005077
.00106
.000915
.000553
.000791
N-BEATS
.000624
.00062
.0009
.000221
.00082
.000438
.000888
.000694
.000718
.000648
GRU
.0024
.00119
.00148
.00058
.000545
.000363
.0012
.000372
.000886
.000364
LSTM
00356
.00119
.00164
.000588
.000845
.000648
.00194
.000737
.000881
.000518
validation
Transformer
.00137
.000577
.0015
.000396
.000566
.000382
.00198
.000686
.00087
.000885
N-BEATS
.00194
.000808
.00207
.000813
.000618
.00052
.00207
,000868
.000966
.000768
GRU
.00568
.000697
.00404
.00282
.000859
.00245
.00317
.00197
.000822
.00191
LSTM
.07705
.000853
.00343
.00241
.00166
.00308
.00438
.0025
.00103
.00229
test
Transformer
.00392
.000463
.00306
.01886
.00996
.00777
.00419
.00258
.00111
.00249
N-BEATS
.00289
.00061
.00325
.02258
0000781
.00194
.00375
.00268
.00104
.00228
(b) (Task F) Forecasting metrics (1)
SprottE?
no
no- yes
yes
attractors
#
#
#
metric
L
L
L
0
L
I
I
GRU
.00254
.00547
.0022
1.07
.00568
.795
.0019
1.49
.0062
.466
LSTM
.00271
.0163
.00152
.605
.00351
.673
.00545
1.37
.0123
.903
train
Transformer
.00954
.0265
.00682
.853
.0215
.442
.00952
.769
.0221
.788
N-BEATS
.019
.0249
.00823
1.55
.0339
1.57
.0389
1.82
.0356
.444
GRU
.00161
.00398
.00166
1.08
.00626
.803
.0014
1.5 .0056 .466
LSTM
.00244
.0178
.00115
.607
.0036
.685
.00999
1.37
.0153 .908
validation
Transformer
.00292
.026
.00133
.891
.0199
.454
.00211
.821
.0231
.888
N-BEATS
.0293
.053
.0154
1.55
.0552
1.56
.0597
1.82
.0468 .443
GRU .00395
.0128
.00405
1.08
.0108
.803
.0098
1.5 .012 .467
LSTM
.00632
.027
.00596
.608
.01
.685
.0148
1.37
.0202 .907
test
Transformer
.00648
.0363
.00454
.898
.0271
.456
.00462
.821
.0255 .819
N-BEATS
.0325
.037
.0362
1.55
.0463
1.56
.0576
1.82 .0402
.443
(c) (Task S) Supervised featurisation metrics (L L)
14

---

Under review as a conference paper at ICLR 2023
C
EXTRA FIGURES
GRU
LSTM
104
train
Triplt 1OGE
100
N-BEATS
Transformer
104
Triplet 1062
100
10
15
20
100 epoch
15
20
epoch
(a) (Task S) Supervised featurisation loss (4)
GRU
LSTM
10
102
seed
set
MSE loos
train
N-BEATS
Transformer
10
10
MSE 1o00
10A
0
10
15
20
0
10
15
20
(b) (Task F) Forccasting loss (4)
Figure A2: Random sampling experiment: training curves. A running average length 700
(roughly half epoch) used for readability. The random seeds don't 1 seem to impact the final performance of the models, but they do impact the training times and speed of convergence.
15

---

Under review as a conference paper at ICLR 2023
GRU
LSTM
train
Coose-nzrony loss
204
N-BEATS
Transformer
Mo 12
Cross-nnoo any loss
204
15
20
30
10
15
20
25
30
(c) (Task C) Classification loss (L)
GRU
LSTM
0.8
Clocification accuraoy
0.6
0.2
0.0
N-BEATS
Transformer
0.8
Cossiiication accurayy
0.6
0.2
0.0
10
15
25
30
1.0
20
25
30
(d) (Task C) Classification accuracy (t)
Figure A2: Random sampling experiment: training curves (cont.)
16

---

Under review as a conference paper at ICLR 2023
1.0
"octeriction
8EATS
To Transfor mer
ww
221
20 op4rss
(a) (Task F) Forecasting loss (1)
GRU
LSTM
1.9
anotraiing
28-1
BEATS
To nondforraa
222
1241
30
30
(b) (Task C) Classification loss (L)
3.6
(c) (Task C) Classification accuracy (1)
Figure A3: Feature-freeze experiment: training curves for the classification and forecasting tasks. running average length 500 (roughly a quarter an epoch) is used for readability. Pre-training seems to help at the start of training, but performance quickly plateaus. This is likely due to the frozen featuriser weights preventing learning from happening.
17

---

Under review as a conference paper at ICLR 2023
GRU
baskis)
10a
clas, feat.
vol
fore.
test
Tripat loss
100
10-1
N-BEATS
Transformer
107
Trigett loos
100
10-4
10
14 epach
10
10
1.8
epoch
(a) (Task S) Supervised featurisation loss (4)
10a
10a
taskiss
set
MSE to0s
clas, fore.
train
107
fore. clas,
test
fore.
10:1
Transformer
10
10a
MSE lo0s
10a
10-4
2
20
12 opoch
24
26
18
20
22
24
10
12 opoth
14
16
18
20
22
24
(b) (Task F) Forecasting loss (1)
Figure A4: Prober experiment: training curves per model and pre/main task combination. running average of length 500 (roughly quarter of an epoch) is used for readability. The best task performance is obtained when the model is being trained for that specific task. Metrics of tasks different to the training task seem stay stable during training.
18

---

Under review as a conference paper at ICLR 2023
GRU
LSTM
toskis)
10
dlss.
train
clas. feat.
Coossenniooy loss
Tore clos.
10F
100
10-1
N-BEATS
Transforerr
10
Cooseentroy bos
10a
100
10-1
10
14
16
18
20
22
24
26
28
10
14
18
20
22
24
26
2B
epoch
(c) (Task C) Classification loss (L)
GRU
LSTM
Clocafication accursoy
tasels)
das. feat. dlas fore.
0.4
Test
0.0
N-BEATS
Transformer
Cassiiication accuracy
0.4
0.2
0.0
2
10
14
18
22
26
10
34
18
20
24
26
28
(d) (Task C) Classification accuracy % (f)
Figure A4: Prober experiment: training curves (cont.)
19

---

Under review as a conference paper at ICLR 2023
Sprott
10
40
-10
0x y2
0t
By x
y 0t
d= 1
4r
01
(a) The SprottE systems.
SprotTC
10
1.0
1.0
1.3
0.
00
-05
00
-05
0.5
100
10
10
12.05
10_03
10
0.5
10
1.0
88
05
1.0
03
1.0
01
0x
Ox
0r
01.
8t
12
öt
y
0t
Oy
By T
By
0y. T + 2
Ø + y
8t
y
T y
Bt
2
0= ry
0= I2 + 3y²
Dr
81
8t
8t
(b) The set of systems similar to SprottE.
Ameodo
Lorenz
QiChen
Sakarya
0.5
05
05
00
10
1.0
10
1.
0.5
-1505
-18
-1.0
0.5
0.0
-10
-55
00
-18
-05
0.0
-10
80
-1.0
005
0.0
-10
25
05
-10
035
0.0
y
21
y
2
y
Us
03 hy + sy2
dy
x
2)
dy
7t
CT
0g
by
pz + quz
de
Ty
3z
05
Ty
bz
8: C3
01
ray
The of systems different SprottE.
Figure A5: Few-shot learning experiment: The set of systems used in the experiment. The default
trajectory of 500 points per period and 10 periods is shown for each. Each trajectory component re-scaled to be in the range [-1, 1]. The differential equations of SprottE and the similar systems simpler, with at most a sum two elementary products, while those the different systems involve more complicated terms.
20

---

Under review as a conference paper at ICLR 2023
GRU
LSTM
10a
attractors, pre-training ar. yes
set
similia, no
Val
different, yes
test
Triplet loss
10:1
different,
10
10
N-BEATS
Transformer
tripitlos
10
10
10
10
14
16
10
14
16
epoch
epocch
(a) (Task S) Featurisation loss (L)
GRU
LSTM
attractors, pre-training
set
2.0
ar, yes
Feature standrdd deviation
simil ar, no
val
different, yes
test
1.5
differant,
1.0
0.5
N-BEATS
Transformer
2.0
Peature standard deviation
1.5
1.0
0.5
2
10
14
16
14
epoch
epoch
(b) (Task S) Featurisation SprottE std
Figure A6: Few-shot learning experiment: Supervised Featurization: training curves (only runs with SprottE are included). A running average of length 100 (roughly an epoch) is used for read- ability.
All runs with pre-training converge faster except those of N-BEATS and classification sensitivity.
21

---

Under review as a conference paper at ICLR 2023
attractors, pre-treining
set
similar
train
simil ar, no
val
different, yes
10-1
aifferent,
MSE loxs
10a2
10
N-BEATS
Transformer
10-1
MSE loss
10a²
10
0
10
12
14
16
18
20
22
24
26
28
0
4
6
10
12
14 epoch
16
18
20
22
24
26
28
(c) (Task F) Forecasting MSE loss (L)
GRU
LSTM
attractors, pre-training
set
simil
train
simil ar. no
val
different, yes
Sprotte MSE
different,
10
10-1
N-BEATS
Transformer
10a
Sproct MSE
10-1
10
0
6
10
12
14
16
18
20
22
24
26
28
0
4
10
12
14
16
18
20
22
24
26
28
epoch
(d) (Task F) Forecasting SprottE MSE (J)
Figure A6: Few-shot learning experiment: Forecasting: training curves (only runs with SprottE included). running average of length 100 (roughly an epoch) is used for readability. All runs with pre-training converge faster except those of N-BEATS and classification sensitivity.
22

---

Under review as a conference paper at ICLR 2023
GRU
LSTM
10a
Cross caoss-entoopy loss
1
10
10
N-BEATS
Transformer
Cress Cress eetropy less
10-4
10-2
attractors, pre-training
train similar, yes
different, yes
test
10-3
different,
16
18
20
22
24
26
18
20
22
24
26
epoch
epoch
(a) (Task C) Classification loss (L)
GRU
LSTM
1.0
Clessification accuracy
attractors, pre-training
Set
simil ar, yes
train
similar
val
different, yes
test
0.2
N-BEATS
Transformer
Clasification accuracy
0.6
0.4
02
32
14 epoch
16
18
22
24
26
10
12
14 epoch
16
18
20
22
24
26
(b) (Task C) Classificatiinn accuracy (t)
Figure A7: Few-shot learning experiment: Classification loss, accuracy: training curves (only runs with SprottE are included). A running average of length 100 (roughly an epoch) is used for read- ability.
All runs with pre-training converge faster except those of N-BEATS and classification sensitivity.
23

---

Under review as a conference paper at ICLR 2023
1.0
0.8
Sprott sersibiiity
0.4
attractors, pre-training
set
ar, yes
0.2
simillar no
val
different, yes
test
different,
N-BEATS
Transformer
1.0
0.8
Sprotte sensiivity
0.4
0.2
10
12
14 epoch
16
18
20
22
24
26
0
2
4
6
8
10
12
14
16
18
20
22
24
26
epoch
(a) (Task C) Classification true positive rate (t)
1.0
0.9
Sprott speafiiitt
0.7
attractors, pre-training ar. yes
set
simillar no different, yes
val
different,
N-BEATS
Transformer
1.0
0.9
Sprottk specificity
0.7
10
32
14 epoch
16
18
20
22
24
26
0
2
4
6
8
10
12
14 epoch
16
18
20
22
24
26
(b) (Task C) Classification true negative rate (+)
Figure A8: Few-shot learning experiment: Classification TPR, TNR: training curves (only runs with SprottE are included). A running average of length 100 (roughly an epoch) is used for readability. All runs with pre-training converge faster except those of N-BEATS and classification sensitivity.
24

---

Under review as a conference paper at ICLR 2023
Aizswa
Anidkchhenkkkttkktov
ArnelBeitramihhiidess
10
1.0
1.8
05
6.0
0.0
0.
00
0.5
0.5
0.5
0.5
-1.0
25
05
1.0
-10
05
60
10
-1.0
03
85
1.0
-1.0
-1.0
05
1.0
-10
BeerRNN
BelbusasZ2abotticky
Bickleyjet
Blastus
14
10
10
13
000
00
20
0.
10
1.0
10
1.
0.5
-10
20
88
05
0.0
-1.
-85
50
05
2.0
-.0
25
05
1.0
0.0
-10 .05
5D
10
BinkingRotlet
BlikkigWVorte
Bouell
Bouall2
1.0
1.5
1.0
1.2
03
0.0
0.
800
0.0
2.5
00
03
-05
14
1.0
1.0
1.
-10
0.0
-10 .05
3.0
0.0
-10 -10.05
0.
00
00
50
-10
-1.0
-10
-10
BurkeShaw
Ca CatwoPius
Co CoTwootusSQuasiarfood
CallsarNeurllletteert
1.
1.0
05
05
0.5
05
00
35
-05
1.0
-1.0
0.5
-1
05
60
-2.5
4.5 08
-0.5
-1. -10.005
-03
-1.0
-1.0
-1.0
-1.
Chiin
ChenLeee
Chura
Coevaliinghrddatofeyy
05
95
05
85
25
05
-35
05
1.0
1.
1
-10
2,5
85
0.5
0.5
00
-10
-1.2
-1.0
-10
(a) Systems 1-20.
Figure A9: The 100 dynamical systems of dimension in synthetic dataset (only Torus is non- chaotic). A single trajectory is shown for each, with the default initial condition, 500 points per period, and 10 periods. Each trajectory component re-scaled to be in the range [-1,
25

---

Under review as a conference paper at ICLR 2023
Colpitts
Coullet
Dadras
DequanLi
05
05
0.5
855
055
-55
05
1.0
1.0
3.
1.2.05
4.0.
-10.0
88
0.
-10
-10
-10
-1.0
DaubleCyre
Duffing
ExcitableCall
Anance
1.0
1.0
1.0
0.5
05
0.5
80
0.0
-05
-05
0.
-100.35
P-55p
0.5
-10
-5
0a
1.0
-1.0
1.
-1.0
FluldTrampoline
ForcadBrusselator
ForsodRttHughhiiiuumo
ForcadVonDerPol
1.0
1.0
1.0
1.9
5.0
08
20
00
1.0
1.0
1.0
1.
0.5
3.5
0.5
0.5
-10
25
09
05
0.0
05
05
2.0
-1.0
08
05
1.0
0.0
-1.0
.0
3D
05
1.0
Genesio Tesi
Guskenheimerrlllmes
Ho dley
Halvorsen
1.0
1.0
1.0
1.9
05
0.0
0.
8.0
0.0
110
1.0
1.0
113
10.05
10
0.8
18.05
3.0
10.55 00
0.0
10.05 9.0
10
0.9
05
60
05
HastingsPowwe owolll
HindmarshRose
loothomalChemical
IskkankssTumrr
10
1.0
05
65
00
05
-05
-.0
-10 3.5
0.
05
-1.0
-0.5
0.0
..5 0.
-1.5
-0.5
3.0
-1.0
o.
-1.0
-0.5
0.0
-1.0
-05
(b) Systems 21-40.
Figure A9: The 100 chaotic dynamical systems of dimension 3. (cont.)
26

---

Under review as a conference paper at ICLR 2023
JerkCrrcutt
KawcoomskiSiricchak
1.0
1.0
1.0
1.9
8.
00
1.0
1.0
0
0.5
0.5
10.55
0.0
18.
10.
10.0000
10.05
08
05
60
0.
08
05
58
1.0
1.0
1.0
1.0
LoranzBounded
1.
1.0
1.0
1.
0.0
0.0
9.0
0.0
-10
-10
-10
-10
-10
0.0
-14
.05 00
2.0
-1.0
0.5
0.0
-10
.05
08
3D
65
85
65
05
LiChenCheng
MooreSplegal
MuttiChu
NewtonLiepnik
1.0
1.0
0.5
50
0.
-05
-1.
-10
1.0
-10
-10
02
05
0.5
0.0
0.5
0.
05
0.5
-1.0
0.
00
-1.0
..0.0.
Noselloover
Oscillatingflow
PanXuZhou
PehlivanWei
1.0
05
85
05
0.5
85
0.5
0.5
0.5
-.0
-10
-12
1.0
14
10.
0.0
18.05
10.05 00
10.05
0.9
-0.
-0.5
-0.5
-0.5
1.0
1.0
110
QiChen
RablovidhFarriiant
RaylelghBenard
RikttkkDDnaamo
0.5
35
05
0.5
25
05
35
-05
1.0
1.0
-10
0.0
2.55
-05
-0.5
-05
-05
-1.
-1.
-10
-1.0
(c) Systems 41-60.
Figure A9: The 100 chaotic dynamical systems of dimension 3. (cont.)
27

---

Under review as a conference paper at ICLR 2023
Rossler
Rucklidge
Sakarys
SaltonSea
1.0
1.0
1.0
1.
5.0
00
-10
-10
-.0
-10
10 10
0.0
10.05
3.0
1..55
0.0
10 10.05 9.
08
05
00
05
00
05
SanUmSrtsuchnmwnn
ShimizuMortoka
SprottA
Sprottt
1.
1.0
05
05
90
00
00
-05
-10
1.0
-10
0.0
La5
0.0
0.0
-10.65
88
-0.
-0.5
-0.5
-05
-10
-1.5
-1.0
-1.0
SprottC
SprottD
SprottE
SprottF
0.5
05
05
05
-5
-05
45
-05
-100
1.
0.0
-10
0.5
0.5
0.5
-10
-1.0
-1.
-1.
SprottG
SprottH
Sprotti
SprotU
0.5
05
05
0.5
80
85
05
0.5
3.5
0.5
110
-10
100
-10
-1.2
-10
1.0
-10
SprottJerk
SprottK
SprottL
1.0
1.0
05
05
80
05
1.0
2.5
0.5
-10
05
1.0
-1.0
0.0
-10
60 05
1.
-1.2
3.0
1.0
08
1.0
-1.0
110
.05
10
-1.0
(d) Systems 61-80.
Figure A9: The 100 chaotic dynamical systems of dimension 3. (cont.)
28

---

Under review as a conference paper at ICLR 2023
SprotMore
SprottN
SprottO
SprotUP
0.5
85
05
0.5
85
0.5
0.5
0.
-10
1.0
1.
18
0.0
-0.5
-0.5
00
-0.5
-05
-100
-1.0
-1.0
-1.0
Sprotti
SprottR
SprottS
Sprottlorus
0.5
25
05
85
-55
-5
45
-0
1.0
-10.5
L15
-1.03
-1.
-1.
-10
-1.0
StickSilpDsccllsttor
Thomas
ThomasLabythth
05
0.5
80
35
-1.0.25
-1.00
-1.0.05
-10
-1.2
-10
1.0
-1.0
Tsucs2
TurchinHanski
ValliEINino
WangSuun
1.
1.0
0.5
8.0
08
5.
0.0
-1.0
0.5
0.5
0.5
0.5
-1.0
35
05
10
-10
05
60
05
-1.9
-1.0
03
85
1.0
-100
-10
05
14
-10
WindmiReduced
YiWling
KiWWing?
ZhouChen
1.8
1.0
1.0
1.
80
88
30
00
10
1.0
10
1.
0.5
-10 -25
88
05
0.
-1 -85
60
05
2.0
-1.0
00
05
0.0
-10
.05
3D
05
0.
(e) Systems 81-100.
Figure A9: The 100 chaotic dynamical systems of dimension 3. (cont.)
29

---

Under review as a conference paper at ICLR 2023
3x
2 10
train
18x x 10
1.6
1.4
12x 10
10a
0.33
0.50
0.67
0.80
alpha
(a) Total loss (4)
-0
train
val
2x10
0.33
0.50
0.67
0.80
alpha
(b) (Task Featurisation loss (L)
2x 10
14x
16x10'
*
train
10"
val
12x 10
10a
10
10-
x 10
0.33
0.50
0.67
0.80
1.00
alpha
(c) (Task F) Forecasting loss (4)
6 10-1
train
val
40-0--
3x
-0-1
0.33
0.50
0.67
0.80
alpha
(d) (Task C) Classification loss (4)
Figure A10: Multi-task loss experiment: final GRU performance for different a (MSE weight in Lona).
The forecasting train/val loss decreases with a, showing that GRU does not benefit from the enforced system representations from the other tasks. The featurisation and classification losses generally increase with o, so they do not benefit from the forecasting representations either.
30

---

Under review as a conference paper at ICLR 2023
(a) GRU
Aftereet] 5t no BLI
Atteert yeur 263
[Hfewent -yee 72
Sprattc
sindar] 57 mr 13.89)
you
sinvior 57 yes
Soroth
Scrotid
valcetion
poal
peal
pea1
(a.1) Classification task
dfftrrntt 57 52.5%)
Mirene 57 yes 50250
Eftrenn 57 yeu y 63.5%
dit attrr
Scent
Ameeds
cinliar (5 no yru tha
embar (T yeu (9.2%)
cibarr ST yes 72.7%)
Scrotii
Stenn
Stetti
Scrotp
Daan
(a.2) Forecasting task
(ffloren) y 10.4%)
diferont 33.00%
ciffeeni| you 5555%
Strot
Amoods
Sakanyu
dintar] S1 nour 75.78
slmibr St yes Ba1
rimiirr St yes $7.451
Spratt
ScrotIC
Scrito
act
Sta
(a.3) Feature extraction
Figure A11: Few-shot learning experiment: PCA projection of the features learned by the models as a function of the other attractors used in the experiment (similar or different), and whether the SprottE attractor is included (shown 'S?"). A different PCA is performed for each model run, using the 32-dimensional features obtained for the train/validationntest datasets, and we show the percentage of variance captured by the two main PCA axes 'r'.
31

---

Under review as a conference paper at ICLR 2023
(b) LSTM
dffereet: 57 no y 17.0%
dttereet 27 no yed C4.0%)
ctteeen 57 -P-yee 77-5700 57.0%)
Ktacar
Scrote
Amoods
Sakanyo
sinar] 51 ro 93.09
simber 57 ro yos 65.04)
simior 57 yes $9,491
Scroti
Scritt
Scrott
(b.1) Classification task
dffereet] St no y 1800)
dftowet St no yre 979993
[ffewent] yes s0.55)
Strett
Amoads
simdor] 51 mee 13.99
similr 57 you 05.751
simrler 57 you - 2045L
Sprots
Scritid
Sznettc
Soropp
14
1.5
100
15
(b.2) Forecasting task
dftereet] 57 no y 10130
dfferree 57 no yee $4.2%)
ctMeren]] 57 yeu 59250
Scont
Amoods
Sakanyo
simtar] 51 -o 74.55
almia 5t yeu 57.841
simile 57 yeu 16.211
Scroti
Scrit
ScrotD
-3.0
6.0
1.0
15
-1.8
-0.
0.0
3.0
15
(b.3) Feature extraction task
Figure A11: Few-shot learning experiment: PCA projection of learned features (cont.)
32

---

Under review as a conference paper at ICLR 2023
(c) Transformer
dftereee: 57 no ST.00
dtteeet 22 yee 00.7%)
cPteeen) 57 -ayye (725%
Scrotic
Amoods
Sakanyo
dnnar] 51 ro 61.7%)
sim 57 ro yes $9.941
simbor S7 yesrr 66.15)
Scrati
Scritt
Scratt
(c.1) Classification task
ditewett St more ROS
dttowett St no yes 74,093
atternt (S) yes in 68.0%)
Scrett
Ameads
Senterr 137 rot (20%)
simia 57 wo you 15.06)
simiar 57 you 20. (0.0%)
Sprots
Scrotid
Sznetrc
Scrotp
(c.2) Forecasting task
cfffewet (57 ro Ir 03 391
dfffrert 27 no yes $2 2%
omternt (St yeer 13.7%
Scrett
Amoods
Sakanyo
simiiar (5) no (1.55
aimia 57 wo yeu $2.51
aimiarr (5 you 7235%
Scroti
Scriti
Scrott
-1.8
-0.5
65
3,8
-1.0
-0.5
-6.5
3.0
3,5
(c.3) Feature extraction task
Figure A11: Few-shot learning experiment: PCA projection of learned features (cont.)
33

---

Under review as a conference paper at ICLR 2023
(d) N-BEATS
atternt 57 94.5%0
dffeent 57 no ooyee 0 90.05)
dfffrent (51 yes 250
Scrote
Ameras
simir| 57 no ir- 150, (4))
smnar] S no yestr- 100.9)
smbry 57 yeu ml)
Szrotbk
Screttc
Scroeo
2.5
2.0
-25
0.0
255
10.0
75
10.0
(d.1) Classification task
dfferent (57- no 200 050
diffrert 15t no os yeu 919%1
iireett - you iy 90 (9.550
Sprottc
Ameods
0.2
cintar (5 ra I- 50.850
siniaer SP i per (r 100.5%1
smibrr 57 yearr 66.1%)
ScrotiA
Scridis
0.2
-0.4
08
-2.4
(d.2) Forecasting task
dPPerett (5 no i do 250)
dtferet 57 no yet tr $8.00 98.0%)
ePterent] 51 yeu y 100.0%
aaarar
0.3
Ameads
Scroue
Sakanyo
[nbr] 50 mir- 65.90)
siniar 57 - o yes 93.551
simbar (S yes in
Scrotin
Scent
ScrotD
0.35
0.50
(d.3) Feature extraction task
Figure A11: Few-shot learning experiment: PCA projection of learned features (cont.)
34


Reviews:
[{"review_id": "uijAzgLMAX0", "reviewer": "Reviewer_KoFr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors construct a framework for assessing generalization performance of models for learning dynamical systems. For this they collect a variety of simulated benchmark data sets and introduce different tasks (forecasting, generalization, classification), and different evaluation measures and procedures. They exemplify their framework with LSTM, GRU, Transformer and N-Beats.  ", "summary_of_the_review": "The question of generalization is certainly very important, but this paper in my mind does not address it. It lacks the theoretical and empirical background necessary to do so.", "strength_and_weaknesses": "Strength:\n- addresses an interesting \\& highly relevant question\n\nWeaknesses:\n- does actually not really test OOD generalization, as claimed\n- doesn't consider relevant benchmarks from the lit.\n- misses important tests for generalization", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fRsHvdzKZls", "reviewer": "Reviewer_k5uo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents an empirical investigation of generalization and transfer in learning to predict dynamical systems. A set of experiments involving classification, featurization and forecasting for 4 main model types is implemented in a framework the authors call ValiDyna. ", "summary_of_the_review": "The authors carry out an investigation of performance of deep time series models for forecasting, classification and self-supervised training on small-dimensional chaotic systems. Some preliminary empirical findings are provided (e.g., freezing after pertaining does not work, some degree of transfer from forecasting to classification, or from forecasting to featurisation). The experiments are bundled with a high-level framework built on PyTorch Lightning. The paper would benefit from a narrower scope and more depth in the analysis.", "strength_and_weaknesses": "Strengths:\n* Understanding transfer in deep time series models  (from a theoretical or even purely empirical perspective) remains an open problem in the literature, and this paper promises a step forward in that direction.\n* I appreciate the design of certain experiments, in particular finding the probing metric idea valuable. Some of the findings are potentially interesting, but as will be made clear below this paper suffers from lack of focus and depth in the analysis which in my opinion makes me lose confidence in the findings.\n\nConcerns (high-level, clarity):\n* Please argue with facts when presenting your contribution (e.g. \"The framework saves a large amount of code\nrepetition and complex indexing/references, e.g. in multi-task experiments.\" is a weird way to start your method section, especially when these statements are not contextualized e.g., w.r. to what? how much is saved?)\n* I don't understand why the \"framework\" is presented as the core technical contribution. What makes this different than any other time series paper doing forecasting or classification or self-supervision?. As this is primarily an empirical investigation, I would have personally preferred more details on the experiments themselves rather than attempts to justify the novelty of performing classification, forecasting and self-supervised training on time series in a single repository. I understand some implementation choices have been made to carry out the experiments, leading to some abstractions (the diagram in A1), but I am not convinced these are any better or easier to use than just working in PyTorch Lightning directly, and the authors do not spend time justifying their choices. The end result is a paper that is uncomfortably in-between a systems paper and an empirical analysis paper, with no justification for the abstraction and no convincing takeaways from the empirical analysis.\n\nConcerns (experimental):\n* Elaborating on the above point. There is a surprising lack of motivation for anything attempted in this paper e.g., (1) why have these four model classes been chosen? If the focus was covering as many mainstream deep time series models, the list is missing temporal convolutions, deep state space models, neural differential equations and more. If the focus was on identifying a particular property of a model class, the paper misses the mark as there is no analysis, nor any ablation to attempt to validate their observations. For example: you observe transfer from forecasting to classification in Transformers? Show that this holds across datasets, varying the architecture, attempt to motivate why you think that is. In my opinion, few readers will be convinced by the evidence you presented in support of your findings, especially due to the large number of moving pieces and the large scope of the paper, which leaves little space for depth. The authors might want to see [1] for a completely empirical work that puts forth a hypothesis, verifies it and then provides further compelling evidence for it. (2) What makes any of this analysis tailored to chaotic systems, beyond the title and choice of datasets? There is no attempt to ablate the empirical findings against a dataset of non-chaotic or turbulent systems. \n* The low performance of N-BEATS seems to go against a lot of time series literature, and I cannot check the implementation since the code was not provided. From the loss divergence seen in Figure A2 and others it appears the hyperparameters might have been poorly chosen. \n* If classification accuracy reaches ~100% for all models, it is a sign that perhaps a more challenging task or variation of the task is required. There is no signal here to distinguish models.\n\n\n[1] ResNet strikes back: An improved training procedure in timm", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DgxgxZd4xja", "reviewer": "Reviewer_hEVM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The purpose of this paper is to present an experimental analysis of factors that influence generalization for data exhibiting chaotic dynamics. For this purpose, the authors built a configurable and extensible model evaluation framework (Validyna). A number of experiments were constructed and run using Validyna.", "summary_of_the_review": "overall, I think the paper's main idea is substantial, and interesting; but needs technical improvements.", "strength_and_weaknesses": "\n**Weaknesses**:\n\n1) Some parts of the paper are unclear. For example, regarding the sentence \"*We sample data from each dynamical system by picking different initial conditions. This leads to trajectories that are sufficiently different from each other, but [...]*\" (on page 2, Section 3.1), there are the following questions:\n- What does \"sufficiently different\" mean in this sentence? Both *sufficiently* and *different* need to be clarified.\n\n- How does picking different initial conditions lead to *qualitatively* different trajectories for a dynamical system? For instance, suppose that one chooses different initial conditions from the basin of attraction of an attractor. Then, all the generated trajectories can be qualitatively the same (topologically equivalent).  \n\n2) If possible, it would be better to increase the size of some plots (e.g. the plots on pages 31-34 which are not clear enough). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject