Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction
Authors: Manuel Brenner
Christoph Jürgen Hemmer
Zahra Monfared
Daniel Durstewitz
Venue: NeurIPS 2024 poster (2024)
Keywords: recurrent neural networks
dynamical systems
chaos
attractors
interpretability
Primary area: machine_learning_for_physical_sciences
TL;DR: We introduce Almost-Linear Recurrent Neural Networks (AL-RNNs) to derive highly interpretable piecewise-linear models of dynamical systems from time-series data.

Abstract: Dynamical systems theory (DST) is fundamental for many areas of science and engineering. It can provide deep insights into the behavior of systems evolving in time, as typically described by differential or recursive equations. A common approach to facilitate mathematical tractability and interpretability of DS models involves decomposing nonlinear DS into multiple linear DS combined by switching manifolds, i.e. piecewise linear (PWL) systems. PWL models are popular in engineering and a frequent choice in mathematics for analyzing the topological properties of DS. However, hand-crafting such models is tedious and only possible for very low-dimensional scenarios, while inferring them from data usually gives rise to unnecessarily complex representations with very many linear subregions. Here we introduce Almost-Linear Recurrent Neural Networks (AL-RNNs) which automatically and robustly produce most parsimonious PWL representations of DS from time series data, using as few PWL nonlinearities as possible. AL-RNNs can be efficiently trained with any SOTA algorithm for dynamical systems reconstruction (DSR), and naturally give rise to a symbolic encoding of the underlying DS that provably preserves important topological properties. We show that for the Lorenz and Rössler systems, AL-RNNs derive, in a purely data-driven way, the known topologically minimal PWL representations of the corresponding chaotic attractors. We further illustrate on two challenging empirical datasets that interpretable symbolic encodings of the dynamics can be achieved, tremendously facilitating mathematical and computational analysis of the underlying systems.

Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction
Manuel Brenner Christoph Jürgen Hemmer!-3, Zahra Monfared', Daniel Durstewitz!22
Dept. of Theoretical Neuroscience, Central Institute of Mental Health, Medical Faculty, Heidelberg University. Germany
Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University, Germany Faculty of Physics and Astronomy, Heidelberg University, Heidelberg, Germany
Abstract
Dynamical systems (DS) theory is fundamental for many areas of science and engineering. It can provide deep insights into the behavior of systems evolving in time, as typically described by differential or recursive equations. A common approach to facilitate mathematical tractability and interpretability of DS models involves decomposing nonlinear DS into multiple linear DS separated by switching manifolds, i.e. piecewise linear (PWL) systems. PWL models are popular in engineering and a frequent choice in mathematics for analyzing the topological properties of DS. However, hand-crafting such models is tedious and only possible for very low-dimensional scenarios, while inferring them from data usually gives rise to unnecessarily complex representations with very many linear subregions. Here we introduce Almost-Liness Recurrent Neural Networks (AL-RNNs) which automatically and robustly produce most parsimonious PWL representations of DS from time series data, using as few PWL nonlincarities as possible. AL-RNNs can be efficiently trained with any SOTA algorithm for dynamical systems reconstruc- tion (DSR), and naturally give rise to symbolic encoding of the underlying DS that provably preserves important topological properties. We show that for the Lorenz and Rössler systems, AL-RNNs discover, in purely data-driven way, the known topologically minimal PWL representations of the corresponding chaotic attrac- tors. We further illustrate on two challenging empirical datasets that interpretable symbolic encodings of the dynamics be achieved, tremendously facilitating mathematical and computational analysis of the underlying systems.
1 Introduction
Dynamical systems (DS) underlie many real-world phenomena of scientific and practical relevance. Complex chaotic DS are believed to govern market dynamics [65], the rhythms of the brain [16], climate systems [100], or ecosystems [71]. A by now rapidly growing field in scientific ML is dynamical systems reconstruction (DSR), where the goal is learn DS model directly from data that constitutes a generative surrogate model of the data- getiaaaeenratnggggg DS. DSR increasingly relies on deep learning, especially in contexts where dynamics are too complex be captured by simple equations or where the underlying processes are not fully understood.
One way of making DSR models mathematically accessible is piecewise linear (PWL) designs, popu- lar among engineers for decades [8, 17, 43, 81, 91]. In the mathematical theory of DS, PWL models also play special role and simplify many types of analysis [3, 54], such as the characterization of
'These authors contributed equally to this work. Corresponding authors: (manuel.brenme christoph.hemmee hemmer, durstewtt @ mi--manneeiddee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

bifurcations [27, 42, 40, 77.777 70]. This is because linear DS are well-understood and straightfor- ward to analyze, while nonlinear DS lack an equally simple deseription [78, 94, 13]. RNNs based on PWL. activation functions, like rectified- linear units (ReI.Us), have been proposed recently for learning mathematically tractable DSR models. Such piecewise- lineer RNNs (PIRNNs),, combined with effective training techniques for controlling gradient flows [67, 391, achieve state-of-the-an (SOTA) performance across a wide range of DSR tasks, including challenging (high-dimensional. noisy, chaotic, partially observed ...) empirical time series [25, 11, 39, 12]. However, while featuring a PWL design, the resulting constructions are often complex, with a large number of linear subregions required to capture the data properly, hence still impeding effective analysis. On the other hand, a class of switching linear DS has been proposed to decompose nonlinear DS into linear regions combined with switching states that determine the transitions between these regions [1, 31, 28, 58, 56, 57, 2]. However, the underlying assumptions of these models and the complexity of the inference mecha- nisms these entail, often make their training challenging and impede their efficient application to DSR problems, especially when moving to real-world scenarios and higher-dimensional systems.
Here we propose almost-linear RNNs (AL-RNNs) which combine linear units and ReLUs, but can use as few of the latter as necessary to achieve a most parsimonious representation in terms of linear subregions. AL-RNNs are easy and effective to train by any SOTA algorithm for DSR. Through this, they are able to robustly identify topologically or geometrically minimal representations of well-known chaotic systems. Their structure translates naturally into symbolic coding that preserves important topological properties. These features make AL-RNNs highly interpretable and mathematically tractable, enabling to harvest tools from symbolic dynamics [74, 55], including the representation of empirically observed DS via minimal topological and computational graphs.
2
Related Work
Dynamical systems reconstruction The field of data-driven DSR has been rapidly expanding in recent years. On the one hand there are approaches based on function libraries for approximating unknown vector fields, which have become particularly popular in some areas like physics [53, 64). Among these, Sparse Identification of Nonlinear Dynamics (SINDy) and its variants [14, 60, 45, 21, 66, 44] is probably the most popular. Since in these models sets of differential equations are directly formulated in terms of known, predefined function libraries, instead of using NN black-box approximators, they have some level of interpretability in the sense that they are readabee and can easily be related to established mathematical building blocks in physical or biological theories [37, 83]. This does not necessarily make them mathematically tractable, however, since systems of nonlinea differential equations are in themselves ves usually hard to analyze (in fact, their behavior is much of the core topic of DS theory [78]). They also have other limitations, including a difficulty in capturing complex and noisy empirical data [11, 391, as they usually require considerable prior knowledge about the system's 's underlying structure (i.e., which terms to include in the function library). This somewhat limits their applicability for discovering novel phenomena. On the other hand, many recent powerful DSR methods rely on universal approximators, in particular the fact that sufficiently large RNNs can approximate any underlying DS [29, 47, 34]. Such methods may be grouped into several broad classes, including reservoir computers [76. 79, 80], neural ODEs/PDEs [20, 46, 4, 48], neural/ Koopman operators [15, 63, 75, 6, 73, 30, 104], and RNNs [99, 25, 101, 19, 11, 84, 391. The latter are commonly trained by variants of backpropagation through time (BPTT, [101, 102, 11]), combined with specific control techniques [67] to remedy the exploding/ vanishing gradients problem [9, 67, 11, 39]. While DSR algorithms based on universal approximators achieve SOTA performance on DSR tasks, and often work particularly well on empirical time series [11, 39J, they commonly deliver a complex model structure that is difficult to interpret and parse mathematically.
Nonlinear dynamics via linear DS The idea of approaching nonlinear DS through our good grasp of linear DS has been around for quite long time, reflected in important theoretical results like the Hartman-Grobman
theorem [36]] or Koopman operator theory [49, 15]. While linear DS are easy to analyze and well understood [78, 94], however, they cannot properly capture most real-world systems, as they cannot produce many important DS phenomena such as limit cycles, chaos, or multistability. This has motivated the modeling of complex dynamics in terms of compositions
'The Hartman Grobman theorem states that nonlinear hyperbolic DS are topologically conjugate to linear DS in some neighborhood of the system's 's equilibrium points.

---

of locally linear dynamics, as the next best alternative, i.e. piecewise-linear (PWL) maps ODE systems [18, 41, 22]. PWL models have been popular in engineering and mathematics for several decades for these reasons, including earlier attempts for learning such models directly from data [93, 24]. Switching linear DS are one particular brand PWI. models with a long tradition in DS and control theory [23, 95, 1, 31, 28). These systems model nonlinear dynamics through set of linear (or affine) DS combined with a switching rule which decides which linear DS is currently active. Likewise, in mathematics PWL models served well for investigating generic properties of nonlinear systems, e.g. the tent map which is topologically conjugate to the logistic map [3]. PWL models often lend themselves to particularly convenient symbolic representations [54, 3], based on which important topological properties of the underlying system, e.g. the nature and number of unstable
periodic orbits embedded within chaotic attractor, can be analyzed [69, 74, 98].
More recently, various modern approaches for inferring PWL models from data have been formulated. For instance, switching state space models combine hidden Markov models with linear DS, jointly inferring the state of a switching (random) variable with the linear DS parameters conditioned on these states [3 []. Various extensions of this basic setup like recurrent and hierarchical switching linear DS and fully Bayesian inference methods have been advanced in recent years [92, 58, 56]. However, inference in these models is often complex and not necessarily optimized for DSR, limiting their applicability to mainly now-dimensional scenarios with comparatively simple dynamics. Most of these models are also discontinuous in their dynamics across switches, while most commonly require the state variables to evolve continuously across the whole of state space. Piecewise- linear RNNs (PLRNNs), and, relatedly, threshold- lineerr networks [33, 105, 1091, on the other hand, based on familiar ReL.U and hence change continuously across their switching manifolds [25,51, 51, 88). They also have some biological justification [33, 25]. Commonly they are trained by variants of BPTT backed up by specific tohtrol- approaches like sparse [67] or generalized [39] teacher forcing which make them SOTA on many DSR tasks. Different PLRNN architectures have been proposed to enhance expressivity or reduce the dimensionality of trained models [11, 391. Yet, while these advances may yield comparatively low-dimensional state spaces, the number of different linear subregions that need to be allocated usually remains very high, hampering efficient mathematical analysis.
would
3 Methodological and Theoretical Prerequisites
3.1
AL-RNN Model
Consider piecewise linear recurrent neural network (PLRNN, [25]):
= Fo(z Az. + Wø(z + h,
where diagonal A E MMM contains linear self-connections, W E MRM are nonlinear con- nections between units, E RM is a bias term, and d(z) = max[0, z| is an element- wiiee ReLU nonlinearity. To expose the piecewise linear structure of this model more clearly, by noting that the slope of the ReLU is either 0 or depending on the sign of 2m,t, one can reformulate this
2. = (A + W D + h =: W 2 + h.
(2)
where Dese) == diag(d a diagonal matrix and doxe)) = (d1, d2.... dd)) ) an indicator vector with dhm (2m,1) = 1 whenever Zm,t > 0 and zero otherwise [26]. For the 2M different configurations of Doay Dipt E {1,2, .......... 2M,, the phase space of system eq. is divided into 2M subregions with linear dynamics
2111
Wa* zt 2 h,
Wa := A WD
Empirically, M often needs to be quite large (at least on the order of the number of observations) for achieving good reconstructions of observed DS. Since the number of subregions grows Md analyzing inferred models in terms of the subregions can thus become very challenging. We therefore introduce a novel variant of the PLRNN in which only subset of P << M units equipped with a ReLU noplinearity, yielding
= Az, + W (z,-1) h
(4)
3

---

where
('(x,) (21,5"" 2M P,t. max(0, 2M-P41,), max(0, zar,:)l
(5)
In this formulation, we thus only have 2n different linear subregions, while still accommodating a sufficiently large number of latent states for capturing unobserved dimensions in the data and disentangling trajectories sufficiently [96, 863.2
The model is trained on the N N-dimensional observations { T, R. by a variant of sparse teacher forcing called identity teacher forcing [67, 11]. In identity teacher forcing, the first N latent states ('readout neurons') are replaced by the N-dimensional observations every time steps, where is chosen such to optimally control trajectory and gradient flows, avoiding exploding gradients while providing the model sufficient opportunity to unroll into the future to capture the underlying DS' long-term behavior (see [67, 39] for details); see Appx. A.2 for details training. We emphasize that sparse teacher forcing is only used for training the model, and is turned off at test time where the model generates new trajectories completely independent from the data.
L linear non-readout neurons
N linear readout neurons
piecewise linear neurons
2P2
WYY
zt time
z
J
14 WT
01
11
2P1
11
00
10
Z c+1
Z++1
Figure
Illustration of the AL-RNN architecture.
3.2 Theoretical Background: Symbolic Dynamics and Symbolic Coding of AL-RNN
The mathematical field of symbolic dynamics formulates conditions under which a DS has a unique symbolic representation and discusses how harvest this symbolic representation properties of the underlying system, which otherwise may be more difficult to address [74, 55). In fact, symbolic dynamics has led many powerful insights and formal results in DS theory, e.g. about the properties of chaos or type and number of periodic orbits [32, 106]. An appealing feature of symbolic dynamics for the field of ML/AI is that it links concepts DS theory computational concepts like finite state automata or formal languages, as well as graph theory [55. 35]. It can thus facilitate the computational interpretation of natural or trained dynamical systems, like RNNs.
prove certain
Assume we have an alphabet of n symbols AA = {O...., ,, n 1). from which we form infinite sequences (bidirectionally or only in forward-time) the dot separating past from future (i.e., indices k: < 0 indicate backward time, and k 2 0 present and forward time). Then the space all possible sequences, together with the so-called (left) shift operator given by
1a----0-ppppppp 02... with 0g A, and
a (a) = a(... a0 1.0001...a1112
(6)
defines the full shift space A² We denote by ṭ² = o o ....o o a the k-times iteration the shift. Now consider a DS (S, Φ) consisting of metric (state) space S and recursive (flow) map d. The flow map Φ advances the system's 's current state x by At and may be thought of as the solution operator of the underlying DS & f(x) [78]. When training RNNs 2, (=.- on time series of observations (x( = 1... T, from the underlying DS, where g is the observation function, we are trying to approximate this flow map. Assume the whole state space S can partitioned into finite set = (U.... ... Un-1} } of disjoint open sets U.. such that = U U., i.e. covered by the union of the closures of these sets. We call this a topological partition of S [55].
The central idea now to assign unique symbol d, A to each U. U, with n U |A|.
As a trajectory (+) (t) of the underlying DS travels through the system's 's state space S, observed at times
SStrictly. in this formulation. for the linear units the diagonal entries in W are redundant to those in A and could be omitted, but found in practice this hardly makes difference.

---

kAt as it passes through different subregions Ue. it gives rise to specific symbolic sequence dæ.e (with unique symbol assigned at each time step via h S - A. FRAI + ag). We may thus think of the shift operator as moving along a trajectory in correspondence with the flow map Φ If the symbolic coding of each trajectory unique, U may constitute a Markov partition (see Appx. for formal definition). We denote by (As,s. 0) the shift of finite type induced by the flow á which picks out from the full shift space A only those admissible symbolic sequences that correspond to valid trajectories of (S, Φ) (we will use the term 'induced by' to refer to this property). The set of admissible blocks constitutes the language of (As.o, o). Every shift of finite type has graph representation G E) (Fig. 2), with either the edges Etg E & or vertices v, E of the graph encoding the permitted transitions among symbols ag A of admissible sequences
A [55].
The finite collection U {Uo...Un-Ih, n 2", of linear subregions of an AL-RNN defined in eq. 4, separated by the switching manifolds Ei.j U, nU, between every pair of neighboring subregions U, and U.. forms topological partition. Here we use this partition as the basis of our symbolic coding and the respective symbolic dynamics (A, σ) induced by the AL-RNN, where we assign to each state 2; E S (i.e., at each time point) the unique symbol E A such that a as iff 2, E U, (Fig. 2).² In the corresponding symbolic graphs, we identify vertices is with symbols a, E A and draw a directed edge Gij from U; v whenever Fo(U; n B) # 88, where B is the attracting set of interest (see Appx. A. for details). As we will show further below, this particular partition has useful theoretical properties that makes the symbolic coding topologically interpretable w.r.t. the AL-RNN map Fe. In fact, a large literature in symbolic dynamics has dealt with the relation between the dynamics in finite shift space and that of PWL map, like, e.g., the tent map, with partition of S into the map's 's different linear subregions as we use here for the AL-RNN [55, 3, 68].
...
Tepological
graph representaton
plecewise neureas
symbollc groph
weighted transttion grapht
000
100
100
110
011
011
011 001 R4
01.
110
pasothle boundary crnssinge
001
2P
110
011
222
Figure 2: Illustration of symbolic approach (3 panels on the left) and geometrical graphs (right).
4
Theoretical Results
Recall that within each subregion Ue the map Fe is monotonic and the dynamics are linear (ruling out certain possibilities, like chaos isolated cycles occurring within just one subregion). We furthermore assume that the dynamics are globally non-diverging (this could be strictly enforced through 'state-clipping" and constraints matrix A in eq. 4, see Hess et al. [39], but will also be the case for a well-trained AL-RNN). Here we claim that for hyperbolic AL-RNNs F we have 1:1 relations between important topological objects in the AL-RNN's state space and those of the symbolic coding formed from the linear subregions U, of the AL-RNN, as expressed in the following theoretical results.
Consider a hyperbolic, non- www.ggoball-yyyygiiiie AL-RNN Fo, 4, and topological partition 2 of the state space into its linear subregions U. S S, 0... .g. - 1. Denote by (Au.Fa, o) the
finite shift induced by (S, with each a. E A its alphabet A associated with exactly one linear subregion Ue E U1 and let consider the system's evolution only in forward time. Then the following holds:
Theorem 1. An orbit Us -2......22..... of the AL-RNN Fe asymptotically fixed (i.e., converges to fixed point) if and only if the corresponding symbolic sequence a
For simplicity we will ignore here and in the following the borders between subregions, on which the coding is ambiguous.
"By hyperbolic RNN we mean the AL-RNN is hyperbolic in each of its linear subregions, ie. none of its Jacobians A W D. has eigenvalues of absolute magnitude 1. The chances that this condition is not met in practice in trained AN-S, Le. the non-hyperbolic case, are close numerically.
5

---

(a1 1io0s...a-1((aa1 E Fu is an eventually fixed point of the shift map a (where by "eventu- ally' we mean it exactly lands on the point in the limit, see Appx. for a precise definition).
Proof. See Appx. B.
Theorem An orbit Gs {Z)...., 2.....) of the AL-RNN Fg asymptotically p-periodic if and only if the corresponding symbolic sequence
a = (a102- aN aN- (a.a....... E Au.
an eventually p-periodic orbit of the shift map o.
Proof. See Appx. B.
Theorem 3. An orbit Ug = (zi...., ......} an asympiotically aperiodic (irregular) orbit of the AL-RNN Fe if and only if the corresponding symbolic sequence (a1...., an....) is aperiodic.
Proof. See Appx. B.
Loosely speaking, these results confirm that fixed points of our symbolic coding correspond to fixed points of the AL-RNN, cycles cycles, and chaos chaos, thus preserving important topological properties in the symbolic representation.
5 Experimental Results
To assess the quality of DSR, we employed established performance criteria based on long-term, invariant topological, geometrical, and temporal features of DS [51, 11, 39]. Due to exponential trajectory divergence in chaotic systems, eqean-sqaared prediction errors rapidly grow even for well-trained systems, and hence only of limited suitability for evaluating DSR quality [108, 67]. Thus, we prioritize the geometric agreement between true and reconstructed attractors, quantified by Kullback-Leebbe divergence (Dstop Appx. A.2) [51]. Additionally,
examine the long-term temporal agreement between true and reconstructed time series by evaluating the average dimension- wise Hellinger distance (D+) between their power spectra (Appx. A.2). We first confirmed that the AL-RNN is at least on par with other SOTA methods for DSR. We then tested AL-RNNs on two commonly employed benchmark DS for which minimal PWL representations are known, the famous Lorenz-63 model of atmospheric convection [61] and the chaotic Rössler system [85]. We finally explored the suitability of our approach on two real-world examples, human electrocardiogram (ECG) and human functional magnetic resonance imaging (fMRI) data.
5.1 SOTA performance
While our goal here a technique that constructs topologically minimal, interpretable DS representa- tions, at the same time we do not want to compromise on DSR performance which should still be within the same ballpark as existing SOTA methods. We checked this on the Lorenz-63, Rössler and ECG data noted above, and in addition on the higher-dimensional chaotic Lorenz-96 system [62] and on human electroencephalogran (EEG) data. Table the Appx. confirms that the AL-RNN not only par with, but indeed outperforms most other techniques when trained with sparse teacher forcing (which may be rooted in its simple and parsimonious design).
5.2 Reconstructed Systems Occupy a Small Number of Subregions
Fig. 3 illustrates reconstruction performance for varying numbers of ReLU nonlinearities at constant network size M. We found that a small number of PWL units already significantly improves perfor- mance, especially for the Lorenz-63 and Rössler systems, and that beyond that number performance starts to plateau (or even briefly decrease again). Additionally, some linear units are necessary to sufficiently expand the space, but they cannot compensate for insufficient number of PWL units (Fig. 10).3 Moreover, as shown in Fig. (left), the number of linear subregions explored by the
Note that the number of linear units does affect the interpretability or symbolic coding of the model.

---

Rössler
Lorenz 63
ECG
fMRI
10
2.5
1df
D00
1.5
10"3
0.5
10
15
10
5
10
20
10
15
20
25
1.0
0.5
0.5
0.5
0.6
0.4
0.1
0.4
0.3
10
20
PWL units (P)
# PWL units (P)
PWL units (P)
# PWL units (P)
Figure 3: Quantification of DSR quality in terms of attractor geometry disagreement (Datp- top row) and disagreement in temporal structure (Dμ, bottom row) as a function of the number of ReLUs (P) in the AL-RNN (Rössler: M 20, Lorenz-63; M 20, ECG: M 100, fMRI: M 50). The little humps at P = 3 for the Lorenz-63 indicate that performance may sometimes first degrade again when passing the number of minimally necessary PWL units (see also Fig. 9). Error bars SEM.
Data covered (96)
# subregions
10
Rössler Lorenz
80
ECG
60
Total
10a
40
Rössler
20
3
15
10
30
50
60
PWL units (P)
Number subregions
Figure 4: Left: Number of linear subregions traversed by trained AL-RNNs as function of the number P of ReLUs. Theoretical limit (2") in red. Right: Cumulative number of data (trajectory) points covered by linear subregions in trained AL-RNNs (Rössler: M 20, P 10. Lorenz-63:
100, = 10), illustrating that trajectories on an attractor live in relatively small subset of subregions.
M = 20, = 10, ECG:
trained dynamics saturates well below the theoretical limit of 2 once this performance threshold reached. Within this already small subset of explored subregions, generated network activity is furthermore concentrated within an even smaller number of dominant subregions: For instance, for the Rössler system 4 out of 45 subregions used cover 80% the data (Fig. 4, right). This substantial reduction of necessary linear subregions strongly f'ciilitates the analysis of trained models with respect to fixed points and k-cycles, which naively would require examining 2 and 2 combinations of subregions, respectively. To select the optimal number of PWL units, the point where performance starts plateauing (as in Fig. 3) may be chosen. Alternatively, one may restrict the number of linear subregions employed through regularization, adding penalty for ReLU nonlincarities. This approach, as Fig. 9 illustrates, results in the same number of selected PWL units.
5.3 Minimal PWL Reconstructions of Chaotic attractors
Topologically minimal reconstructions Investigating reconstructions with the minimal number of PWL units needed for close-to-optimal performance (Fig. 3), we found that the AL-RNN would deliver reconstructions capturing the overall structure of the attractor using only three (Lorenz-63 system) or two (Rössler system) linear subregions (Fig. 5a), explaining the strong performance gains in Fig. for and PWL units, respectively. These representations, and their symbolic coding (Fig. Se), expose the mechanisms of chaotic dynamics (Fig. 5b). Notably, these closely agree with the minimal topologically equivalent PWL representations of the two chaotic DS described in Amaral et al. [5]: The Lorenz-63 system has at its core two unstable spiral points in the two lobes, separated by the saddle node in the center (Fig. 5b). For the Rössler system, the topologically minimal PWL representation indeed consists of just two subregions [5], one containing an unstable spiral in the x-y plane and the other 'half-spiral" almost orthogonal to that plane (Fig. 5b). The AL-RNN automatically and robustly discovers these representations from data: across multiple training runs, performance values are very similar (Figs. 23, 25), the assignment subregions to different parts of the attractor remains almost the same (Figs. 24 & 25), and the regions with linear dynamics
7

---

a b
0.10
O
0.05
Abssler
Lorenz-63
0.10
$0.05
0.00
Rössler
Lonenz-63
Figure 5: a: Color-coded linear subregions of minimal AL-RNNs representing the Rössler (top) and Lorenz-63 (bottom) chaotic attractor. b: Illustration of how the AL-RNN creates the chaotic dynamics. For the Rössler, trajectories diverge from an unstable spiral point (true position in gray, learned position black) into the second subregion, where after about half a cycle they are propelled back into the first. For the Lorenz-63, two unstable spiral points (true: gray; learned: black) create the diverging spiraling dynamies in the two lobes, separated by the saddle node in the center. c: Topological graphs of the symbolic coding. While for the Rössler it is fully connected. for the Lorenz-63 the crucial role the center saddle region distributing trajectories onto the two lobes is apparent. d: Geometrical divergence (Dssp) among repeated trainings of AL-RNNs (n = 20), separately evaluated within each subregion, shows close agreement among different training runs. Likewise, low normalized distances between fixed point locations and f: relative differences in maximum absolute eigenvalues omax across 20 trained models indicate that these topologically minimal representations robustly identified.
closely agree both in terms of their topology and geometry (in fact, the topological graphs remained identical). This is in contrast to the standard PLRNN, where assignments strongly varied among multiple training runs (Figs. 23, 25). We quantified this further by computing across training runs separately for each subregion Dasp (Fig. 5d), the normalized distances between fixed points (Fig. Se). and the normalized differences between the maximum absolute eigenvalues max of the AL-RNN's Jacobians (Fig. 5f), obtaining values close zero in all three cases (see Fig. for absolute values). While Amaral et al. [5] explicitly handcrafted such minimal PWL representations, the AL.-NNN extracts them automatically without the provision any prior knowledge about the system.
Geometrically minimal reconstructions While these reconstructions capture the topology of the underlying DS, they do not yet capture the full geometry and temporal structure the attractor (Fig. 3). Fig. 6 illustrates for the Rössler system that as the number of PWL units is further increased to = 10, the geometrical agreement becomes almost perfect. Although the mapping from latent to observation space is not 1:1 (since M N), points close in observation space still tended fall into the same latent subregion, such that the observed system's 's attractor still decomposed into distinct subregions, as confirmed by proximity matching (see Appx. A. For the Rössler system there is just one nonlinearity, the x 2 term in the temporal derivative of (eq. 14). Accordingly, the AL-RNN devotes most of its subregions to the lobe along the coordinate, while dynamics in the (z, y) plane is geometrically faithfully represented by only 4 subregions. Hence, the AL-RNN utilizes additional subregions to express finer geometric details where dynamics are more nonlinear. This is apparent from more geometrical graph representation (see Fig. 2, right), where in addition to topological information transition probabilities among subregions are being used construct node distances via the graph Laplacian (see Appx. A.1), see Fig. for the Rössler and Fig. 11 for the Lorenz-63.
5.4 PWL Reconstructions of Real-World Systems
Topologically minimal reconstructions ECG data (with 1 membrane potential recordings delay-embedded into N = 5, see Appx. A.3) and fMRT recordings (with N 20 time series extracted, ef. Appx. A.3) from human subjects performing three different types of cognitive task [50, 52].
next considered two experimental datasets, human

---

a
b
301
Ground Truth Freely Generated
363
3000
Figure 6: Geometrically minimal reconstruction and graph representation of the Rössler attractor (M = 30, P = 10, D stsp = 0.08, D 0.06). a: Provided sufficient number of linear subregions, the geometry of the attractor is almost perfectly captured. b: Reconstruction with linear subregions color-coded by frequency of visits (dark: most frequently visited regions, yellow: least frequent re- gions). c: Corresponding geometrical graph, which contains information about transition frequencies via node distances, visualized using the spectral layout in networkx. Note that self-connections were omitted in this representation. d: Connectome of relative transition frequencies between subregions.
Q Wave
Amplttuee (mv]
1.0
pww Units
R Riive
0.5
0.0
20
-0.5
Depsiartation Repearictton
0.000
0.236
0,471
0.707
0.000
0.235
0.471
0,707
Time [s]
Time [s]
Figure 7: a: Freely generated ECG activity using an AL-RNN with 3 linear subregions (color-coded according to subregion) and ground truth time series in black. b: After activation of the Q wave in the third subregion, the second PWL unit is driven far below O, whose activity, consistent with the known physiology [89], mimics the latent de- and re-polarization process of the interventricular septum. c: Symbolic graph representation of the trained AL-RNN.
As for the Lorenz-63, for the ECG data observed strong performance gain for just P 2 PWL units, see Fig. 3. Indeed, Fig. 7a confirms that the complex activity pattern and positive max. Lyapunov exponent (Amax 1.96 s-1 ground truth: Aiue = 2.19 s [39]) of the ECG time series could be achieved with P = 2 in only 3 linear subregions. These subregions corresponded to distinct parts of the ECG activity: ramping-up uup phases (light blue, node # 1), declining activity (dark blue, node #2), represented by two unstable spirals with shifted phases (Fig. 12), and the Q wave (medium blue, node #3). The activity in the third region (a (ama 1.34, other two: maar 1.02, Fig. 12) caused a strong inhibition in the second PWL unit (Fig. 7b) that captures the critical transition initiated by the Q wave. The Q wave triggers the depolarization of the interventricular septum during the QRS complex [89], indicating that this latent depolarization process is captured by the model. These results suggest that the AL-RNN cannot only learn dynamically but also biologically interpretable latent representations. The core aspects of this representation were furthermore consistent across successful reconstructions (Fig. 13). The symbolic sequences corresponding the graph in Fig. 7 reveal the nearly periodic yet chaotic dynamics of the ECG (Fig. 14).
For the short (T 360) fMRI time series, P 3 often resulted in reconstructions matching the complex activity patterns reasonably well (cf. Fig. 3, right). Fig. 15 illustrates results for an example subject using& 8 linear subregions. The second most visited subregion implemented unstable spiral, while the most visited region had a stable virtual fixed point also located in the second subregion. These two regions covered over 50% of the data and were strongly connected (Fig. 15b). This balance between stable and unstable activity suggests mechanism through which the network implements chaotic dynamics, with the stable virtual fixed point pulling activity into the second subregion from which it then diverges again (Fig. 15d).
"In general, we also found that quantities computed from symbolic sequences like the topological entropy correlated highly with the maximum Lyapunov exponent, Fig. 16.

---

11
M
80LD
230
400
000
800
1000
Time [+]
00
10
10
and Inatractio
-5.
-2.5
01 11
CRT
CMT
Figure 8: Reconstructions from human fMRI data using an AL-RNN with M 100 total units and P= 2 PWL units. a: Mean generated BOLD activity color-coded according to the linear subregion. Background color shadings indicate the task stage. b: Generated activity (trajectory points) in the latent space of PWL units with color-coding indicating task stage as in a.
Task stages align with linear subregions To integrate the fMRI signal with cognitive (task-stage) information, in addition to the linear decoder for the BOLD signal, we coupled the PWL units z to categorical decoder model (eq. 12) which predicts the three cognitive task stages and the 'Rest and Instruction' period, as in [52, 12]. Using only P = 2 PWL units on the fMRI data made challenging to capture the complex, chaotic long-term activity patterns in the freely generated activity of the AL-RNN. However, the dynamics were still -ell-approximated locally (Fig. 17). To maintain temporal alignment with the task stages when sampling from the AL-RNN the AL-RNN's readout units were reset to the observations every 7 time steps (Fig. Sa). Fig. 8a-b show that in this setup, the four linear subregions of the AL-RNN often closely aligned with the different task stages of the experimental time series. Similar results were obtained across different subjects, with an average classification accuracy of p 0.78 + 0.05 (mean + SEM) (see Appx. A.4 for details). While the categorical decoder aids in separating latent states according to task stage, there is nothing that would bias this separation align with the linear subregions. The observed alignment therefore suggests that the AL-RNN learns to leverage distinct linear dynamics in each subregion to represent differences across cognitive tasks. Furthermore, as shown in Fig. 18, the network weights of the PWL units were significantly larger than those of other units, indicating their critical role in modulating the dynamics and representing task-related variations in brain activity. This approach also demonstrates how local context-aligned linear approximations can be achieved using the AL-RNN, which is useful in areas such as model -redei-treeietvve control [72, 23,58) 58].
6 Conclusion
Here we introduced a novel variant of a PLRNN, the AL-RNN, which learns to represent nonlinear DS with as few PWL nonlinearities possible. Despite its simple design and the minimal hyperparameter tuning required, the AI-RNN robustly and automatically identifies highly interpretable, topologically minimal representations of complex nonlinear DS, reproducing known minimal PWL forms of chaotic attractors [5]. Such minimal PWL forms that allow for an interpretable symbolic and graph-theoretie representation were discovered even from challenging physiological and neuroscientific data. They also profoundly ease subsequent model analysis. For instance, with only a few linear subregions to consider, the search for fixed points or cycles becomes very fast and efficient [26].
Limitations While this seems promising, how to determine whether a topologically minimal and valid reconstruction from empirical data has truly been achieved remains open topic. Performance curves as in Fig. 3 or Fig. 9 give an indication of how many PWL units may be required to yield an optimal minimal representation, but whether there is more principled way of automatically inferring the optimal number P of PWL nonlinearities from data may be an interesting future direction. Finally, the current finding that for empirical ECG and fMRI data few linear subregions (< 8) suffice for faithful reconstructions is encouraging. Whether this more generally will be the case in empirical scenarios is another interesting and open question. Not all types of (empirically observed) dynamical systems may easily allow for such topologically minimal representations.
All code created is available at https: / //tpt/gggttt//tt// wwt.://wwwwwwwwti/eeeeeeoeo Bww.://LLLAAAAANN--A DSR.
10

---

Acknowledgements
This work was funded by the Federal Ministry of Science, Education, and Culture (MWK) of the state of Baaeen-Wirttebbeee Württemberg within the AI Health Innovation Cluster, by the German Research Foundation (DFG) within Germany's 's Excellence Strategy EXC 2181/1 390900948 (STRUCTURES) and through DFG individual grant Du 354/15-1 to DD. ZM was funded by the Federal Ministry of Education and Research (BMBF) through project OIDLITDSM, 01IS24061.
References
[1] G. Ackerson and K. Fu. On state estimation in switching environments. IEEE Transactions on Automatic Control, 15(1):10-17, February 1970. ISSN 1558-2523, doi: 10. 1 10.119A9AC11900 1099359. URL https: //ieeexpllrreeee/e/oo/ee.//aeeee ieee. www.oo///ge///oeeeee55e 1099559. Conference Name: IEEE Transactions on Automatic Control.
[2] Xavier Mamedda-Pineda Vincent Drouard, and Radu Patrice Horaud. Variational Inference and Learning Piecewise Linear Dynamical Systems. IEEE Transactions on Neural Networks and Learning Systems, (3(8):3753-376 August 2022. ISSN 2162-2388 doi: 10.1 109//NNN.
2021.3054480 URL https: //ieeexplore ieee. .ee...rg/eeeeeeeeee..eeee.....ee httg/ddcument99333350
Conference
Name: IEEE Transactions on Neural Networks and Learning Systems.
[3] Kathleen T. Alligood, Tim D. Sauer, and James A. Yorke. Chaos: An Introduction to Dynamical Systems. Textbooks in Mathematical Sciences. Springer, 1996. ISBN 978-0-387-94677-11 978-0-387-22492-3. doi: 10.1007/b97589.
[4] Victor M. Martinez Alvarez, Rareş Rosea, and Cristian G. Fälcuţeseu. DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control, September 2020.
[5] Gleison F. V. Amaral, Christophe Letellier, and Luis Antonio Aguirre. Piecewise affine models of chaotic attractors: the Rossler and Lorenz systems. Chaos (Woodbury, N.Y.), 16(1):013115, March 2006. ISSN 1054-1500. doi: 10. 10.106//1.214957.
[6] Omri Azencot, N. Benjamin Erichson, Vanessa Lin, and Michael W. Mahoney, Forecasting Sequential Data using Consistent Koopman Autoencoders. In Proceedings of the 37th Interna- tional Conference on Machine Learning, 2020. URL http://:///aat
herg/abs/220 02236.
[7] Mikhail Belkin and Partha Niyogi. Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering. In Advances in Neural Information Processing Systems, vol- neurips cw/paper.pp.p......pc.. laappprrrllss files/ ume 14. MIT Press, 2001. URL httttp:://proceeeeee Abw.rbbbbbttttt......o html. paper http://ss//.111be99222eeeeeeeeee
[8] Alberto Bemporad, Francesco Borrelli, and Manfred Morari. Piecewise linear optimal con- trollers for hybrid systems. In Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No. 00CH36334), volume 2, pages 1190-1194. IEEE, 2000.
[9] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994, ISSN 1045-9227. doi: 10. 109/72.279881.
[10] Mario Di Bernardo, Mark I. Feigin, Stephen J. Hogan, and Martin Homer. Lo- cal Analysis of C-Bifurcations in n-Dimensional Piecewise-Smooth Dynamical Sys- tems. Chaos, Solitons and Fractals: the interdisciplinary journal of Nonlin- ear Science, and Nonequilibrium and Complex Phenomena, 11(10):1881-108 1999. ISSN 0960-0779. URL https://wwat elsevier- -ww.-b611111----01 000999550-311 310f bww.-dddfffeeeeee 4bd77717494.
http////eeree////eeeeeeeeeeeeeee wetetttaaeeeemmmmm.em. element.
[11] Manuel Brenner, Florian Hess, Jonas M. Mikhaeil, Leonard F. Bereska, Zahra Monfared, Po-Chen Kuo, and Daniel Durstewitz. Tractable Dendritie RNNs for Reconstructing Nonlinear Dynamical Systems. In Proceedings of the 39th International Conference on Machine Learning, pages 2292-2320. PMLR, June 2022. URL www.hhttt//p/otoeee htt.httt///prrccednggss mlr. www.s/eeesssveee
brenner224 a......... html. ISSN: 2640-3498.

---

[12] Manuel Brenner, Florian Hess, Georgia Koppe, and Daniel Durstewitz. Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics. In Proceedings of the 41st International Conference on Machine Learning, pages 4482-43 1812-456. PMLR, July 2024. URL
https://proceediing .mlr http:sswww......e.eeeemmmmmeeeee wataas//223/bbbeeeeee
html. ISSN: 2640-3498.
[13] Steven L Brunton and Nathan Kutz. Data-driven science and engineering: Machine learning. dynamical systems, and control. Cambridge University Press, 2019.
[14] Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings tfe National Academy of Sciences USA, 113(15):3932-33)) 3937, 2016. ISSN 0027-8424. doi: 10. 10.10a3/pnas. 1517384113.
[15] Steven L. Brunton, Marko Budišić, Eurika Kaiser, and J. Nathan Kutz. Modern Koopman Theory for Dynamical Systems, October 2021. arXiv:2102. 12086 [cs, eess, math].
[16] Gyorgy Buzsaki. Rhythms of the Brain. Oxford University Press, August 2006, ISBN 978-0-19-804125-2. Google-Books-ID: Idz58irprjYC.
[17] Victoriano Carmona, Emilio Freire, Enrique Ponce, and Francisco Torres. On simplifying and classifying piecewise-linear systems. IEEE Transactions Circuits and Systems I: Fundamental Theory and Applications, 49(5):60--200, 2002.
[18] Martin Casdagli. Nonlinear prediction of chaotic time series. Physica D: Nonlinear Phenom- ena, (5(3):335-356 May 1989. ISSN 0167-2789. doi: 00.10166016--27889999990177771
[19] Rok Cestnik and Markus Abel. Inferring the dynamics of oscillatory systems using recurrent neural networks. Chaos: An Interdisciplinary Journal of Nonlinear Science, 193318 June 2019. ISSN 1054-1500. doi: 10.1063/1.50969) URL https://doi. org 1063/
5096991..
[20] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. In Advances Neural Information Processing Systems 31, 2018. URL http: //arxi/aaiiii/// org/ 1ate/180...o 07366.
[21] Alexandre Cortiella, Kwang-Chun Park, and Alireza Doostan. Sparse identification of non- linear dynamical systems via reweighted 11 -l-reuularized least squares. Computer Methods in Applied Mechanics and Engineering, 376:113620, April 2021. ISSN 0045-7825. doi:
10. 10..1066jj.cm...oo cma.2220.00.... 13620.
[22] Antonio C. Costa, Tosif Ahamed, and Greg J. Stephens. Adaptive, locally linear models of complex dynamics. Proceedings of the National Academy of Sciences, 116(5):150--1510, January 2019. doi: (0.1073/pas..111476111 Academy of Sciences.
Publisher: Proceedings of the National
[23] J. Daafouz, P. Riedinger, and C. lung. Stability analysis and control synthesis for switched systems: a switched Lyapunov function approach. IEEE Transactions Automatic Control, 47(11):1833-1877 November 2002. ISSN 1558-2523.
doi: 11100/TA..2022.844444
URL https: / feeexpllree www.ee.eeeeeeeeeeeeeeeeeeeeeeeee wwwp///wweee////o/////////////// 1047016. 1044000.... Conference Name: IEEE Transactions on Automatic Control.
[24] Oscar De Feo and Marco Storace. Piecewise- Linear Identification of Nonlinear Dynamical Systems in View of Their Circuit Implementations. IEEE Transactions on Circuits and Systems I: Regular Papers, 54(7):1542-1554, July 2007. ISSN 1558-0806. doi: 10.1109/TCSI.00 899613. URL htt.ttt:pp/eeeee/ll///e http///peeeeeeeeeeeeeeee/e///eee ieee. org/ ceccumnt/4266440 Conference Name: IEEE Transactions Circuits and Systems! I: Regular Papers.
[25] Daniel Durstewitz. state space approach for piecewise- recurrent neural networks for identifying computational dynamics from neural measurements. PLoS Comput. Biol., 13(6): e1005542, 2017. ISSN 1553-7358. doi: 10.1371/joummal.po 1c0..100552.. 1005542.
[26] Lukas Fisenmann, Zahra Monfared, Niclas Göring, and Daniel Durstewitz. Bifurcations and loss jumps in RNN training. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[27] Mark I Feigin. The increasingly complex structure of the bifurcation tree of a piecewise- smooth system. Journal of Applied Mathematics and Mechanics, 59(6):553-863, 1995. ISSN 0021-8928. doi: 10. 10:10166002218882895)000188 URL htt.s: http://www.ieneereeeeeeeee
[28] Emily Fox. Erik Sudderth, Michael Jordan, and Alan Willsky. Nonparamet- ric Bayesian Learning of Switching Linear Dynamical Systems. In Advances
in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008.
URL https://papers nips.cc/ http/appr_ffffeee wttpff//lpeleee/0///./ pptpp///00e///asst 95o4455ccbbaaaa188a663366666 -eee-bbbtsrtttee html.
[29] Ken-ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural Networks, 1989. doi: 10.1016/0992-6080099999000000000
[30] Nicholas Geneva and Nicholas Zabaras. Transformers for modeling physical systems. Neural 201111.2021..1..22.
Networks, 146:272-2899 February 2022. ISSN 0893-6080. doi:
[31] Z. Ghahramani and G. E. Hinton. Variational learning for switching state-space mod- els. Neural Computation, 12(4):831-864, April 2000. ISSN 0899-7667. doi: 10.1162/ 089976600300015619.
[32] John Guckenheimer and Philip Holmes. Nonlinear Oscillations, Dynamical Systems, and Bi- furcations of Vector Fields, volume 42 of Applied Mathematical Sciences. Springer, New York, NY, 1983. ISBN 978-1-4612-7020-- 978-1-461 10-44612211400- 100-- doi: 10. 10.1007/988-1-461--11444 140-2. URL httptttp//iikkkppp...sst.
wwwwco.mo////oc 1007 1001119900--1111-111111-
[33] Richard Hahnloser and H. Sebastian Seung. Permitted and Forbidden Sets in Symmetric Threshold- Networks. Advances in Neural Information Processing Systems, vol- ume 13. MIT Press, 2000.
[34] Joshua Hanson and Maxim Raginsky. Universal simulation of stable dynamical systems by recurrent neural nets. In Proceedings of the 2nd Conference Learning for Dynamics and Control, volume 120 of Proceedings of Machine Learning Research, pages 384-392. PMLR, 10-11 Jun 2020. URL https: /pppe//d//n/ceggggss//.e
aaaiingsss.rr....mmmmm....m.mmmm mlr press htt////////////hae/ html.
[35] Bailin Hao and Weimou Zheng. Applied symbolic dynamics and chaos. World Scientific, 1998.
[36] Philip Hartman. A lemma in the theory of structural stability of differential equations. Pro- ceedings of the American Mathematical Sociery, 1 11(4):610666 620, 1960. ISSN 0002-9939, 1088-6826. doi: 10. 10.1090/S0002 9990900000009999-000009 1900---999----2220 1960-012154-77.. URL https: /www.wwwo ams org/
proc/ 1roc/1960-ae 011 10011104/00022 9939 10000999999999----0111---------1
[37] Niklas Heim, Václav Šmidl, and Tomáš Pevny. Rodent: Relevance determination differential /992..
equations. arXiv preprint arXiv: (rXXiv/9/2..06560 2019. URL 00656.
[38] Christoph Jürgen Hemmer, Manuel Brenner, Florian Hess, and Daniel Durstewitz. Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction. Proceedings of the st International Conference on Machine Learning, pages 18174-18200 PMLR, July 2024.
URL httpps:/pproceedinngg mlr wwwps/e2sss...e. 017/3///hemee2444 html. ISSN: 2640-3498.
[39] Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized Teacher Forcing for Learning Chaotic Dynamics. In Proceedings of the 40th International Conference Machine Learning, pages 13017-13049. PMLR, July 2023. URL //ppoe/e//o..c.c........
mlr, press
preess/v02// htt///22//hheeeeeee 2r202/hess23.... html. ISSN: 2640-3495
[40] Stephen J. Hogan, L. Higham, and C. L. Griffin. Dynamics of piecewise linear map with gap. Proceedings of the Royal Society A: Mathematicul, Physical and Engi- neering Sciences, 463(2077):49-63 2007. doi: 10. 10.1098//spa.2000 1735. URL https:
//royalsocietypuuleeiiiii org/oo//gg /w///oi/aaaa abs/ 10. ww0.0///....... 2006 1735.
13

---

[41] Anthony R. Ives and Vasilis Dakos. Detecting dynamical changes in nonlinear time series using locally linear state-space models. Ecosphere, 3(6):art58 2012. ISSN 2150-8925. doi: 10.1 1990/8811-000447.... _eprint http:///nniielllraayyllllyooo//p 1-900/// 00347.1.
[42] Parag Jain and Soumitro Banerjee. Border-collision bifurcations in one-dimensional discon- tinuous maps. Int. J. Bifurcation Chaos, 13(11):3341-3351 2003. ISSN 0218- 1274. doi:
10.1 10.1142/5021812244000333333 URL https: http://wwwwwwwwccnnnnniii..e.... wwt.coooamc/c///eeeetoooaaooaatt 1142/$00188274403033333
[43] A.L. Juloski, S. Weiland, and W.P.M.H. Heemels. Bayesian approach identification of hybrid systems. IEEE Transactions on Automatic Control, 50(10)15500-1333 October 2005. ISSN 1558-2523. doi: 10.1 20.1009/AC..2055.00 86668856649. URL wttttttss://eeepplllee
.ieee org/
document/151622 Conference Name: IEEE Transactions on Automatic Control.
[44] Kadierdan Kaheman, Steven L. Brunton, and J. Nathan Kutz. Automatic differentiation to simultaneously identify nonlinear dynamics and extract noise probability distributions from data. Machine Learning: Science and Technology, 3(1):015031, March 2022. ISSN 2632-2153. Publisher: IOP Publishing.
doi: 10. 10110/2633-2133//55565
[45] E. Kaiser, J. N. Kutz, and S. L. Brunton. Sparse identification nonlinear dynamics for model predictive control in the low-data limit. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 474(2219):0010333, November 2018. doi: 10.10p8/ 2018.0333 Publisher: Royal Society.
[46] Daniel Karlsson and Olle Svanström. Modelling Dynamical Systems Using Neural Ordinary Differential Equations, 2019, URL httph: //hp1 /hdl. hww.lal met//e.500. (2380/256888)
[47] Masahiro Kimura and Ryohei Nakano. Learning dynamical systems by recurrent neural networks from orbits. Neural Networks, 11(9):1589-155) 1559, 1998.
[48] Joon-Hyuk Ko, Hankyul Koh, Nojun Park, and Wonho Jhe. Homotopy-based training of NeuralODEs for accurate dynamics discovery, May 2023. URL http://arxiv. orgoorgg//ooo harg/b////a 2210. 01407. (.Xiv:2210.014:0 [physics].
[49] B. O. Koopman and v. Neumann. Dynamical Systems of Continuous Spectra. Proceedings of the National Academy of Sciences, 18(3):255-22) 263,, March 1932. doi: 10.1073/pmas.18.33 Publisher: Proceedings of the National Academy of Sciences.
[50] Georgia Koppe, Harald Gruppe, Gebhard Sammer, Bernd Gallhofer, Peter Kirsch, and Stefanie Lis. Temporal unpredictability of stimulus sequence affects brain activation differently depending on cognitive task demands. Neurolmage, 101:236-244, 2014. ISSN 1095-9572.
doi: 10. /00.///.cuuoommaee20000 20..4.0..08..
[51] Georgia Koppe, Hazem Toutounji, Peter Kirsch, Stefanie Lis, and Daniel Durstewitz. Identify- ing nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI. PL.Ó Computational Biology, 15())ee1007263, 2019. ISSN 1553-7358. doi:
10. 10.1371/journal wwb...piib...... 1007263.
[52] Daniel Kramer, Philine L Bommer, Carlo Tombolini, Georgia Koppe, and Daniel Durstewitz. Reconstructing nonlinear dynamical systems from of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1 16613-11633, PMLR, 17-23 Jul 2022. URL https: / //proceedings... mlr. press wttps//ii1//k/maeeeeeee .html.
time series. In Proceedings
[53] William La Cava, Patryk Orzechowski. Bogdan Burlacu, Fabrício Olivetti de França, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason H. Moore. Contemporary Symbolic Regression Methods and their Relative Performance, July 2021. URL http://arxiv, org/ abs/2107 14351. arXiv:2107. 14351 [cs].
[54] Douglas Lind and Brian Marcus. namics and Coding. Cambridge doi: 10. 10110177C3009780511666622
An Introduction to Symbolic Dy- University Press, Cambridge, 1995.
URL https: //wwww cambridge.eooooo org/
14

---

core/ http://oootcc......oocoooooooooo (BDDEESDD6613300118331000000
coding/
[55] Douglas Lind and Brian Marcus. An Introduction to Symbolic Dynamics and Coding. Cam- bridge Mathematical Library. Cambridge University Press, 2 edition, 2021.
[56] Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Sys- tems. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pages 914-922 PMLR, April 2017. URL https: atttps:/pproceednng mlr press// 354/1inderman176 html. ISSN: 2640-3498.
[57] Scott W. Linderman and Matthew J. Johnson. Structure-Exploittng variational inference for recurrent switching linear dynamical systems. In 2017 IEEE 7th International Workshop on Computational Advances in Multi- Adaptive Processing (CAMSAP), pages 1-5, December 2017. doi: 10.1 10.1109/CAMSAP201....11 org/ ddccument/8333132
URL htttt:://eeeepppaeeeee ieee.
[58] Scott W. Linderman, Andrew C. Miller, Ryan P. Adams, David M. Blei, Liam Paninski, and Matthew J. Johnson. Recurrent switching linear dynamical systems. arXiv:1610.085 [stat], October 2016. URL bectp://arxie
httg/bb/1660 11611.08666. 08466. arXiv: 1610.08466.
[59] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020. URL UtLhttps://ppemte //tp///vee///eeee//////e//e//eee www.:nee//eeee net/forum? id=rkgZ2aEKDr.
[60] Jean-Christophe Loiseau and Steven L. Brunton. Constrained sparse Galerkin regression. Journal Fluid Mechanics, 838:42-67, March 2018. ISSN 0022-1120, 1469-7645. doi: 10. 10.1017/jfm.200 2017.823. Publisher: Cambridge University Press.
[61] Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2):
130-141, 1963.
[62] Edward N Lorenz. Predictability: problem partly solved. In Proc. Seminar on predictability, volume 1996.
[63] Bethany Luseh, Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nat Commun, 91):4950, December 2018. ISSN 2041- 1723. doi: 10. arXiv: 1712.09707.
URL attt:://arivv org/aos///////oooooooeo...ooeo 1712 09707.
[64] Nour Makke and Sanjay Chawla. Interpretable scientific discovery with symbolic regression: a review. Artificial Intelligence Review, 57(1):2, January 2024. ISSN 1573-7462, doi: 10. 10.10077/10442-003--- 10622-0.
URL https: /atts::/coe http://oooooo//////// orgg 10. s000000//00222000 023-00022-000000 o.
[65] Benoit Mandelbrot and Richard L. Hudson. The Misbehavior of Markets: Fractal View of Financial Turbulence. Basic Books, March 2007. ISBN 978-0-465-00468-3, Google-Books-
ID: GMKeUquiPOOC.
[66] Daniel A. Messenger and David M. Bortz Weak SINDy: Galerkin-Based Data-Driven Model Selection. Multiscale Modeling & Simulation, 19(3):1474-1497, January 2021. ISSN 1540-3459. doi: 10.1137/20M1343116. URL https: / /epubs.. siam. www...ogg//aa......... org//oi/10..ooo 1137/ 20M1343168 Publisher: Society for Industrial and Applied Mathematics.
[67] Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with RNNs. Advances in Neural Information Processing Systems, 35: 35:11297-11311 2022. December 2022. URL StRL:t /tppoceep///rrooeeedngg neurips wwtpppcppppppeeeeeeeeeeeec http:p/pprreeeeeeeeeeeeeeeeeeeee /202//
haal//96655..17000cccadaa1100002 htwp://bbbbbbaa10044oaede--oaate Abstract Conference oorrradddddbbtttarrr-oooeeeeeeee html.
[68] John Milnor. On the concept of attractor. Communications in Mathematical Physics, 99(2): 177-195, June 1985. ISSN 1432-0916. doi: 10.1007/BF012112880 URL https://o/// org/ 1007 BF01222280.
15

---

[69] K. Mischaikow, M. Mrozek, J. Reiss, and A. Szymezak. Construction of Symbolic Dynam- ics from Experimental Time Series. Physical Review Letters, 8266):114-1117, February 1999. doi: 10. wet.eee...........1.11........ URL https: //inkk....////////////////// aps. bttg/doo/1..10000000 PhysRevLett. 82. 1144. Publisher: American Physical Society.
[70] Zahra Monfared and Daniel Durstewitz. Existence of n-cycles and border-collision bifurcations in piecewise- linear continuous maps with applications to recurrent neural networks. Nonlinear URL
Dyn, 1012):1037-1052 2020. ISSN 1573-269X. doi: 10700000
Attps://áon org/ 1111111.1000001111111110
05841-x88
[71] Peter J. Mumby, Alan Hastings, and Helen J. Edwards. Thresholds and the resilience of caribbean coral reefs. Nature, 450(7166):98-101) 2007. doi: 10.1038/maturr006255. URL
https: //doi. org/10. 2038//nature62222
[72] Kenneth R. Muske and James B. Rawlings. Model predictive control with linear
models. AIChE Journal, 39(2):262-287, 1993. ISSN 1547-5905. doi: 10.1002/ aic.690390208. URL https: / htt::///nnnnnnnnnneeeeeeeee wiley com http:/doo/a/////ee// wttpaa......///////...........tt 1002/ aic.
690390208. _eprin::
[73] Ilan Naiman and Omri Azencot. A Koopman Approach to Understanding Sequence Neural Models. crXiv:2102.000 00284 [cs, math), October 2021. URL 07824. arXiv: 2102.07824.
[74] George Osipenko and Stephen Campbell. Applied symbolic dynamies: attractors and filtrations.
Discrete and Continuous Dynamical Systems, 5(1):43-60, September 1998. ISSN 1078-0947. doi: 10.3934/dcds.. 1999.5.43, URL https: //t://wwww /www. aimsci ences. org/ en/
10. 3934/ //344/dcca /cds. 1999 5. 43. Publisher: Discrete and Continuous Dynamical Systems.
[75] Samuel E. Otto and Clarence W. Rowley. Linearly-Recurren Autoencoder Networks for Learning Dynamics, January 2019. URL http://arxiv http:///axxiv.rrgabbb... art./bb/11/22..1333 arXiv:1712.0133 [cs, math, stat].
[76] Jaideep Pathak, Zhixin Lu, Brian R. Hunt, Michelle Girvan, and Edward Ott. Using Machine
Learning Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27112) December 2017. ISSN 1054-1500, 1089-7682. doi: (0.1063/1.5010300 URL http::/ttxiiiiiii attp///www...gaaaaaa/aa/aaa abw..//11..bb
07313. arXiv: 17 1710.07313.
[77] Mahashweta Patra. Multiple Attractor Bifurcation in Three-Dimensional Piecewise Linear Maps. Int. J. Bifurcation Chaos, 28(10):183033,, 2018. ISSN 0218-1274 doi: 10.1142/ $021811741830032XX URL $0218127418300322
[78] Lawrence Perko. Differential equations and dynamical systems. Number 7 in Texts in applied mathematics. Springer, New York, 3rd ed edition, 2001. ISBN 978-0-387-951 16-4.
[79] Jason A. Platt, Stephen G. Penny, Timothy A. Smith, Tse-Chun Chen, and Henry D. I. Abarbanel. A Systematic Exploration of Reservoir Computing for Forecasting Complex Spatiotemporal Dynamics, January 2022. URL http:://arrt org/ abs/2201..89110 MxXiv:2211008111 [cs].
[80] Jason A Platt, Stephen G Penny, Timothy A Smith, Tse-Chun Chen, and Henry DI Abarbanel. Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(10), 2023.
[81] Anders Rantzer and Mikael Johansson. Piecewise linear quadratic optimal control. IEEE transactions on automatic control, 45(4):629-63) 2000.
[82] Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. Deep ppg: Large-scale heart rate estimation with convolutional neural networks. Sensors. 3974), 2019, 2019.
16

---

[83] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 165):06-211, May 2019.
ISSN 2522-5839. doi: 10. 1038//42256-019 -01-00009-- URL https: //ww//aauure com/ articles/ss2256-000 019- x. Publisher: Nature Publishing Group.
[84] T Konstantin Rusch, Siddhartha Mishra, N Benjamin Erichson, and Michael W Mahoney. Long expressive memory for sequence modeling. In International Conference on Learning Representations, 2022.
[85] O.E. Rössler. An equation for continuous chaos. Physics Letters 5755)397-38, 1976. ISSN 0375-9601, doi: 0001001600375--900177999000 com/ http:/sinnee/articeo httpa////r....eeee////////00///0
URL http:://ww.ssdinccciieeet sciencedirect..
[86] Tim Sauer, James A Yorke, and Martin Casdagli. Embedology. Journal of statistical Physics, 65(3):579-616, 1991.
[87] Gerwin Schalk, Dennis J. McFarland, Thilo Hinterberger, Niels Birbaumer, and Jonathan R. Wolpaw. BC12000: general-purpose brain-computer interface (BCT) system. IEEE transactions on bio-medical engineering, 51(6):1034-1043, 2000. ISSN 0018-9294. doi: 10. 10.1000TBBEE.2244.2777
[88] Dominik Schmidt, Georgia Koppe, Zahra Monfared, Max Beutelspacher, and Daniel Durste- witz. Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies. In Proceedings of the 9th International Conference on Learning Representa- tions, 2021. URL http: http///axvvvv/..///////e.ee..ava org/ abs /9100 034471.
[89] Demetrio Poli-Paalaress, María Isabel Rodriguez, Leonardo O. Chait, and Rudolf Zuckermann. The activation of the interventricular septum. American Heart Journal, 4164):569-608 April 1951. ISSN 0002-8703. sciencedirect.. ctt http:///cacecceaataaaa//////////
doi: 10. 00.006/0002-777335119000444 URL https //www.
[90] Justin Solomon. PDE Approaches to Graph Analysis, April 2015. UtL.ttp://arxdv..rr
0ts/1505.001880 1r0iv:1505.00111 [cs, math].
[91] E. Sontag. Nonlinear regulation: The piecewise linear approach. IEEE Transactions on Automatic Control, 26(2):346-355 April 1981. ISSN 1558-2523, doi: 10.1 10.1 1102596. URL https: //teeeepeeeeeeeeeeeeeeeeeeeeeeee feee. .ee...rrg//eeeeeeee............e www.uoeeetttneeeeeee IEEE Transactions Automatic Control.
Conference Name:
[92] Ioan Stanculescu, Christopher K. I. Williams, and Yvonne Freer. A Hierarchical Switch- ing Linear Dynamical System Applied to the Detection Sepsis in Neonatal Condition Monitoring. In Proceedings of the 30th Conference
Uncertainty in Artificial Intelli- gence (UAI 2014), 2014. URL hattps::/wwit research. ed. wu/ http:/ubllccciooo a-hierarchic cal swi tching- wwt.ssswwwhhiieneee-nee-eeeeeaee www.--nnnnannnnnaallalaaaaa sys-sm--sssssee--- applied to-eeddttttttte thethe
[93] M. Storace and O. De Feo. Piecewise-linear approximation of nonlinear dynamical systems. IEEE Transactions on Circuits and Systems I: Regular Papers, 51(4):830 842, April 2004. ISSN 1558-0806. doi: 10. WWT.CCSS..0048226666 Conference Name: IEEE Transactions on Circuits and Systems I: Regular Papers.
[94] Steven H. Strogatz. Nonlinear Dynamics and Chaos. CRC Press, 1 edition, 2015. ISBN 978-0-429-80 000422--9611-33 doi: 10.1201/9780429449553. URL https: taylorfracoiis com/books/ http:/book////0000/ /970029961111.
[95] Zhendong Sun. Switched Linear Systems: Control and Design. Springer Science & Business Media, March 2006. ISBN 778-1-84628-133 131--0.-11-....1111111111110-011 Google- Bookkl-Books-DD:: u4GArZNIlmse
[96] Floris Takens. Detecting strange attractors in turbulence. In Dynamical Systems and Turbulence, Warwick 1980, volume 898, pages 366-381. Springer, 1981. ISBN 978-3-540-11177 978-3- 540-38945-3, URL http: //link////nk... springer com/ 10. 1001110777/BOO BBB.0999200999.....
[97] Sachin S. Talathi and Aniket Vartak. Improving performance of recurrent neural network with relu nonlinearity. Proceedings of the 4th International Conference on Learning Representations, 2016. URL http:tp:/// /ttpvv.//vvvv//////a//////./.a wwgg/bs//t//eeoeeoooo abs//511.111111///////1////11/11 0t////511.0777/1 03771.
17

---

[98] M. Tomas-Roos Rodriguez and S. P. Banks. Linear approximations to nonlinear dynamical systems with upplicationn to stability and spectral theory. IMA Journal of Mathematical Control and Information, (0(1))89-103, March 2003. ISSN 0265-0754. doi: 10. (0.109//mamci/20, 1.89.
httpsstss://di.. org. /togg///...0 1093/ 1mamci/20000 1.89.
[99] Adam P. Trischler and Gabriele M.T. D' D'Eleuterio. Synthesis of recurrent neural networks for dynamical system simulation. Neural Networks, 80:67-78, 2016. ISSN 08936080. doi: 10. mew.:::..cummet2110..44444 pis/80893608016300314
URL cettp://linkiing htw.:nnniinnnu....eeeeeeeee. elsevvier.cmrreerieeee.. com/retrieve/
[100] Eli Tziperman, Harvey Scher, Stephen E. Zebiak, and Mark A. Cane. Controlling spatiotem- poral chaos in a realistic el niño prediction model. Phys. Rev. Lett. 79:1034-1037, Aug 1997. doi: 10.1 10.1103/hysRevll.tt 79. 1034. URL httth::///inniii aps wwwp///////oooooooo PhysRevLett. 79. 1034.
[101] Pantelis R. Vlachas, Wonmin Byeon, Zhong Y. Wan, Themistoklis P. Sapsis, and Petros Koumoutsakos. Data-driven forecasting of high-dimensional chaotic systems with long short- term memory networks. Proc. R. Soc. A., 474(2113)20110844, 2018. ISSN 1364-5021, 1471-2946. doi: 10. wa..1098/ppa.2217..... 2017.0844. URL htt.aatpps///oolllooceyyllllaaap org/
doi/10. 10w./r999...... 201rs 2011.00000 0844..
[102] Pantelis R. Vlachas, Jaideep Pathak, Brian R. Hunt, Themistoklis P. Sapsis, Michelle Girvan, Edward Ott, and Petros Koumoutsakos. Backpropagation Algorithmss and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics. arXiv: /0tiv.99/00055 1910.05266 [physics], February 2020. uttp:tt::/ ptattp://axiii... org/ abs/ /0tbs//990 1910.. 05266. arXiv: 1910.05266.
[103] Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie. On Yyaрunov exponents for RNNs: Understanding information propagation using dynamical systems tools. Frontiers in Applied Mathematics and Statistics, 8, 2022. ISSN 2297-4687. URL https:
//www.frontieree.e. www.fferrittnnnnnmm org/ articles/ 10. 3389 / fams. 818779..
[104] Rui Wang, Yihe Dong, Sercan Ö Arik, and Rose Yu. Koopman Neural Forecaster for Time Series with Temporal Distribution Shifts, October 2022. URL http:://as/t orgoorggggoo abs/ 2210. 03675. arXiv:2210.0588
[cs, stat].
[105] H. Wersing, W. J. Beyn, and H. Ritter. Dynamical stability conditions for recurrent neural networks with unsaturating piecewise linear transfer functions. Neural Computation, 13(8): 1811-1825, August 2001. ISSN 08899-7667. doi: 10. 00.116//0899766615525500
[106] Stephen Wiggins. Global Bifurcations and Chaos, volume 73 of Applied Mathemati- cal Sciences. Springer, New York, NY, 1988. ISBN 978-1-4612-1041- 978-1-4612- 1042-9. doi: 10.1007/978- 10..007977--14412-100000 116-1-4612-142---1
URL http://link, springer 10..m/10..10
[107] Alan Wolf, Jack B. Swift, Harry L. Swinney, and John A. Vastano. Determining lyapunov exponents from a time series. Physica D: Nonlinear Phenomena, 16(3):285-317, 1985. ISSN 0167-2789. doi: 10. 00:101660167--7888559900000
URL https: //tp::///wwwww... www.wieeinneeeir.......
com/science con/scciece/arree htap://sinneeeetttcc///a//////20
[108] Simon N. Wood. Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466(7310):1002-111 August 2010. ISSN 1476-4687. doi: 10.1038/nature093199 URL
https://wwas nature. com/ http///ticcls//ature053333 Publishing Group.
Number: 7310 Publisher: Nature
[109] Zhang Yi, K. Tan, and T. H. Lee. Multistability Analysis for Recurrent Neural Networks with Unsaturating Piecewise Linear Transfer Functions. Newral Computation, 15(3):639- 662, March 2003, ISSN 0899- doi: 10.162/0889766002219911
URL https:
//doi. org /0/./11.111 100.122/0009660022119977
18

---

A
Appendix
A.1 Graph Representation
The symbolic graph construction followed the rules given in Sect. 3.2: linear subregion Ug is assigned node (symbol), and a directed edge is drawn between nodes i, + whenever (U) nU; * Z. Here, however, we are mostly interested in the topologiaa graphs representing particular chaotic attractors (like the Lorenz-63 Rössler attractor), and hence restrict the graph representation to the nodes corresponding to subregions visited by trajectories on the attractor B, and edges drawn when FU, n B) n U3 # Ø. More specifically, we first sampled a long trajectory Z = [z1.... .r) with T = 100, 000 time steps, removed the first 1000 time steps transients, and counted all transitions between any two subregions U,, Uy. To obtain a more nuanced geometrical/ statistical picture, we also evaluated the relative number of time steps the trajectory spent in each subregion U, (i.e., estimate of the occupation measure), 22 E U 1000}1/(100, 000- - 1000), as well as the relative frequency of transitions between any two subregions U. Jz [2z, 2++1 weight for each edge, or a distance between nodes through the graph's 's Laplacian (see below). To enhance the readability of the larger graphs in Figs. 6, 11, 15 and 21, removed self-connections (time steps where the trajectory remained within a single subregion).
Most generally, each
1000, 2, E U. Fo (z)) (1)0/ 000 1001), yielding
Laplacian matrix The Laplacian matrix of graph is defined L -= D A where A is the adjacency matrix of the graph containing the weights (transition probabilities), and D the out- degree matrix, which is a diagonal matrix where each element is the sum of the outgoing edge weights of node i, Di, = E, Asg. The spectral layout in networkx uses the eigenvectors of the Laplacian matrix corresponding to the smallest non-zero eigenvalues as positions for the nodes. This tends to group more tightly connected nodes closer together. The Laplacian is more widely used as dimensionality reduction technique in MI., for example in Laplacian eigenmaps [7], and has also been used to represent discretizations of PDEs as graphs [90].
Proximity matching The mapping from latent space to observation space is not unique because M > N, and hence the nullspace of the linear observation model is non-empty (does not contain just the 0 vector). For the AL-RNN this is evident from the fact that the non-readout neurons, particularly the PWL neurons, do not contribute to the observations. However, in practice, for the freely generated activity of trained AL-RNNS, points that fall into the same linear subregion in latent space also were close in observation space. This implies that attractors are segmented into different subregions in latent space in accordance with the observable dynamics. To numerically confirm this, we found that proximal points in observation space were typically related to the same linear subregion when generating activity from trained AL-RNNs: We conducted proximity matching by defining threshold distance (e.g. d = 0.05, corresponding to 5% of the data variance) and assessing whether generated latent trajectory points proximal in observation space fall into the same or different subregions. We found that for the geometrically minimal reconstruction of the Rössler system (Fig. 6), only 6% of proximal data points (within d) belonged to different subregions, while for the Lorenz-63 -63 attractor (Fig. 11) 4% of proximal data points (within d) belonged to different subregions, confirming that the attractors were segmented into relatively distinct patches.
A.2 Methodological Details
Training method Our training method rests on variant of sparse teacher forcing. This approach has recently been proven to effectively tackle gradient divergence when training on observations from chaotic DS [67] and has shown SOTA performance on benchmark and real-world systems in DSR [1]. In sparse teacher forcing, the idea is to directly replace latent states (or a subset of them) with states inferred from observations at intervals while leaving the network to evolve freely otherwise. To obtain forced states, the observation model needs to be 'pseudo-inverted'. Here we employ specific variant of sparse teacher forcing called identity teacher forcing [67, 11], where this pseudo-inversion becomes trivial by adopting an identity mapping as the observation model:
St Izee
19

---

with I E RNXM, and I, for the N read-out neurons, 1' S N, and zeroes elsewhere. During training. the read-out states are replaced with observations every T time steps:
Fo( if ET 2++1 = Fo(zz)
(8)
else
with T = (IT + 1hEN,, and ž; = (Ii, *N+1:M,))" Employing identity teacher forcing splits the AL-RNN into essentially three types of units, the N linear readout-neurons 2 which are equivalent to the predicted observations and teacher-forced during training, the remaining M - P- N linear neurons zi. and the P nonlinear neurons z/- The overall model and architecture illustrated in Fig. The AL-RNN can be trained using Mean Squared Error (MSE) loss over model predictions and observations:
T | 2.122
MSEE X xxX N Σ
where X are the model predictions and X denotes the training sequence of length T. We found that performance was better when the read-out neurons were linear rather than ReLUUbasee.. Note that the M - N non-readout neurons, including the PWL neurons, do not explicitly con- tribute the loss function. We used rectified adaptive moment estimation (RADAM) [59] as the optimizer, with L = 50 batches with S = 16 sequences per epoch. Further, we chose M = {00, 20, 100, 100, 100, 130),
{16, 8, 20, 10), T (200, 300, 50, 72, 50,50, 100), initial learning rates 7utt = {10-3, 10-3 2. 10 10-3,5 10-3 10-3 10 10-3), nead = 10 and epochs {2000, 3000, 4000, 2000, 3000, 2000) for the (Lorenz-63, Rössler. ECG, LorenL 96.EEG dataset, respectively. Parameters in W were initialized using a Gaussian initialization with a = 0.01, as a vector of zeros, and A as the diagonal of normalized positive-definite random matrix [11, 97]. The initial latent state 21
Læ,]" J" is estimated from 21 using matrix L E R (M-N)xXX which is jointly learned with the other model parameters. Additionally, for the Rössler and Lorenz systems, we added 5% observation noise during training. Across all training epochs of a given run, we consistently selected the model with the lowest Daup- Each individual training run of the AL-RNN was performed on a single CPU. Depending on the training sequence length, a single epoch took between 0.5 to 3 seconds.
Geometric agreement For evaluating attractor geometries, we use a state space measure Daup based on the Kullback-Leiler (KL) divergence, which assesses the (mis)match between the ground truth spatial distribution of data points, (2)) and the distribution Pgen(zz [z]=) ) of trajectory points freely generated by the inferred DSR model. These probability distributions can be approximated in different ways from the observed/ generated trajectories. Here, we usually sampled long trajectories corresponding to the test set length (usually T = 100, 000 time steps, but sometimes shorter for the empirical time series) from trained systems, removing transients to ensure that the system has reached limit set. For low-dimensional systems, the KI. divergence can be straightforwardly calculated through a discrete binning approximation [11]:
K
Datap (Prrue(x), Pgen 2)) 2 Σ
Pirue (z) log
Pirne x
(10)
Pgen x
where K = m is the total number of bins, with m bins per dimension and N being the dimension of the ground truth system. A bin number of m = 30 per dimension was chosen as a good compromise for distinguishing between successful and bad reconstructions for 3d systems. Since the number of data required to fill the bins scales exponentially with N, for the ECG time series (N = 5) we reduced the number of bins to m = 8, as suggested [38].
Temporal agreement To assess temporal agreement, we computed Hellinger distances DH between power spectra [67, 39J. We first simulated long time series T' = 100, 000 (as with Datep above). After standardization, we computed dimension-wist Fast Fourier Transforms (FFT). The power spectra were smoothened using a Gaussian kernel and normalized, and the extended, high-frequency tails, which predominantly contained noise, were truncated. The Hellinger distance between smoothed ground-trute spectra F(w) and generated spectra G(w) is given by:
H(F(w), G(w)) = vi-f - V((w)G(W)do E [0.1]
(1))
20

---

Maximum Lyapunov exponent The maximum Lyapunov exponent of a system quantifies how fast nearby trajectories diverge, and for flow map can be computed in the limit T oo from the system's 8 product of Jacobians. To approximate the maximum exponent numerically, we first iterated a trained model forward from some randomly drawn initial condition and discarded transients. Given that for chaotic systems the product of Jacobians itself explodes [67], we employed numerical algorithm from [107, 103] that re-orthogonalizes the series of Jacobian products at regular intervals using a QR decomposition.
Categorical decoder given by
We coupled categorical observations the / PWL neurons via a link function
(3) 2
exp
vE {1 K K-1 -1} 1}
(12)
K- 1+ j=1
exp (5f 2.,
K
K-1 Ej=1 j=1 exp (gT
The regression weights B, E Px1 are learned for each category 1... K - 1, while the normalization ensures that the total probability over all categories sums to one, >
iml nT = 1.
A.3 Benchmark datasets
Lorenz-63 The Lorenz-63 system, formulated by Edward Lorenz in 1963 [61] model atmospheric convection, is defined by
dæ
d/ o ay x))
(13)
dy
dt x1p - 2)
dz xy _2,
dt
where a, p. 3, are parameters that control the dynamics of the system (here set to = 10, B = 8 and p = 28, in the chaotic regime). The system was solved numerically with integration time step At = 0.01 using scipy. integrate with the default RK45 solver.
Rössler The Rössler system, intended by Otto Rössler in 1976 [85] further simplification of the Lorenz model, produces chaotic dynamics using a single nonlinearity in one state variable:
dx
dt y
(14)
dy x. + az
df
dz = b + 2(x -c),
dt
where a, b, c, are parameters controlling the dynamics of the system (here set to a = 0.2, = 0.2, and 5.7, in the chaotic regime). The system was solved with integration time step At 0.08 using integrate with the default RK45 solver.
Human electrocardiogram The electrocardiogram (ECG) time series was taken from the PPG- DaLiA dataset [82]. With a sampling frequency of 700H,, the recording duration spanned 600 seconds, yielding a time series of T = 419, 973 time points. Initially, a Gaussian smoothing filter with a = was applied, resulting in a filter length of = So + = 49. We standardized time series and applied temporal delay embedding using the Dynami calSystems..... jl Julia library, resulting an embedding dimension of m For model training, the first T 100, 000 time steps (approximately 143 seconds) were used, downsampled to every 10th datapoint.
21

---

Human IMRI data The functional magnetic resonance imaging (fMRI) data from human subjects performing three cognitive tasks is publicly available on GitHub [52]. We followed Kramer al. [52] and selected the first principal component of BOLD activity in each of the 20 brain regions. Subjects underwent five rounds three cognitive tasks ('Choice Reaction Task [CRT]', 'Continuous Delayed Response Task [CDRT]' and 'Continuous Matching Task [CMT]'), together with 'Rest' and 'Instruction' period. The time series per subject were short (7' = 360) and reconstructions in [52] exhibited a positive maximum Lyapunov exponent, indicating the chaotic nature of the underlying system.
Lorenz-96
The Lorenz-96 system, formulated by Edward Lorenz in 1996 [62], is defined by dæ; dt (xi+1 21-2) I; F,
(15)
with system variables T,, = N, and forcing term /' (here, F' = 8, in the chaotic regime). Furthermore, eyclic boundary conditions are assumed with z-1 NN-1- Ty NN, EN+1 = 21, and the system was solved with integration step At = 0.04 using scipy integrate with the default RK45 solver.
Human EEG data Electroenccephalogra (EEG) data were taken from a study by Schalk et al. [87]. These are 64-channel EEG recordings obtained from human subjects during different motor and imagery tasks. The signal was standardized and smoothed using a Hann function and a window length of 15, as in [11].
A.4 Further Results
Table 1: Comparison of AL-RNN to different SOTA models for dynamical systems reconstruction. Comparison values, datasets and evaluation measures as [39], based on code provided on GitHub by the authors. id-TF: identity teacher forcing, GTF: generalized teacher forcing. D Dstsp: state space divergence, DH: Hellinger distance across power spectra. Reported values are median median absolute deviation.
Dataset
Method
Dau
Dh
AL-RNN id-IF
0.34
0.081
360
shPLRNN GTF
0.26:0.00
0.090 0.007
365
dend-RNN + id-IF
0.9
0.03
361
Lorenz-
Reservoir Computer
0.12
0.19
603
(3d)
LSTM-TBPTH
0.222
0.11
1188
SINDy
0.24
0.091
30
Neural-On ODEE
0.63
0.15
353
Long Expressive Memory
0.39
0.12
360
RNN id-TF
3.0
0.29 0.04
2808
shPLRNN GTF
4.30.6
0.02
2785
dendPLRNN id-TF
5.6
0.37 0.06
3245
Reservoir Computer
53
0.39 0.05
5000
BCG
LSTM-TBPTT
15.2
0.73
5920
SINDy
diverging
diverging
3960
Neural-ODE
1.72
0.7 0.03
4955
Expressive Memory
16.3
0.56 0.04
4872
AL-RNN 4 id-TF
1.04
0.089 + 0.001
4623
shPLRNN GTF
1.68 0.06
0.072
0.001
4540
dendPLRNN id-TF
1.65 0.05
0.083 1.000
5740
Lorenz-
Reservoir Computer
0.15
0.14
12000
(20d)
LSTM-TBPTT
5
0.31 0..M
10580
SINDY
1.59
0.00
4620
Neural-ODD ODE
1.77+-0.07
0.076 + 0.01
4530
Long Expressive Memory
7.2
0.54
4620
AI-RNN
0.3
17955
shPLRNN GTF
2.2
0.11
17952
dendPLRV LRNN id-TF
3+1
0.13+00.0
18099
BEG
Reservoir Computer
1417
0.54
28672
(64d)
LSTM-TBPIT
30
0.2
51584
SINDY
diverging
diverging
133120
Neural-ODE
0.5
0.47
17995
Long Expressive Memory
10.2-41..5
0.38 + 0.06
18304
22

---

15
D Doop
10
10
Apn 10.0
1.0
A 0.5
A
0.1
A
0.01
Figure 9: Top: DSR quality (assessed by Dstsp) as a function of strength of regularization on the number of nonlinearities for the AL-RNN trained on Lorenz-63. Bottom: Number of selected piecewise-linear units P function of regularization strength. As in Fig. 3, first optimum consistently occurs for = 2. To select the number of nonlinear units through regularization, we replaced the standard ReLU by leaky ReLU max max(a; z)), az (0, 1). for each of the i ===1.. -1...M M units. The slope a; = (74)) is determined through a steep sigmoid, (7%) = 1/(1 + cxp( -500(% 0.5))), via trainable parameter "ss ensuring that it is either close to 0 close to 1. To encourage linearity, we include a loss term Lin = Aim EM la; 1/. pushing slopes towards 1. After training. units with a; R 1 are classified as linear, while all remaining units were considered nonlinear to provide an estimate for
Task stages align with subregions To test the alignment of the reconstructed AL-RNN activity in the subspace of the P = 2 PWL units with the 4 task stages, we trained 10 models on each the 10 subjects without visible movement artifacts (yielding 100 trained models). We then checked which assignment of the four subregions (00, 01, 10, and 11) to the four task stages (Rest and Instruction, CRT, CDRT, and CMT) gave the highest alignment (Fig. 8), and used this to determine the average classification accuracy as the percentage of time points correctly assigned to their respective task stage based the four subregions.
Geometrically minimal reconstructions of empirical time series Increasing the number of PWL
units also improved geometric agreement for the empirical time series (Fig. 3, P 10 for the ECG data), while dynamics remained confined to relatively small subset of linear subregions (Fig. 4). This confinement within only a few subregions allows for the efficient computation of dynamical objects like fixed points. For the ECG data, real and virtual fixed points in the linear subregions were primarily located within a 3d hyperplane of the 5d data. Principal component analysis showed that this hyperplane harboring the fixed points aligned with the first principal component of the data, explaining approximately 40% of the data's variance (Fig. 20a/b). similar pattern was observed in the fMRI data, where fixed point locations often aligned with PC1 of the data (Fig. 21).
23

---

Lorenz 63 (P = 1)
10.5
1.00
10.0
0.98
9.5
0.96
9.0
0.94
20
40
60
80
20
40
60
80
Linear units
# Linear units
Lorenz 63 (P 2)
0.7
10.0
0.6
7.5
0.5
5.0
0.4
0.3
0
5
10
15
10
15
# Linear units
Linear units
Figure 10: Reconstruction performance on Lorenz-63 system for an AL-RNN (M = 20) as function of the number of linear units, once for the case where the number of PWL units was insufficient for topologically accurate reconstruction (P = 1, top), and once for the case where it was sufficient (P bottom). Results indicate performance cannot be improved by adding more linear units P is too small, but can be to some saturation level when P is sufficiently large. Error bars SEM.
00010
Relative Transition Frequenoy
0.008
0.008
-0.004
-0.002
-0.000
Figure 11: Optimal geometric reconstruction of a Lorenz-63 using the AL-RNN with P 8 PWL units. Left: reconstruction with subregions color-coded by frequency of trajectory visits (dark: most frequently visited regions, yellow: least frequent regions). Center: Resulting geometrical graph structure (using transition probabilities for placing the nodes) visualized using the spectral layout in networkx. Note that self-connections and directedness of edges were omitted in this representation. The resulting graph shadows the layout of the reconstructed system. Right: Connectome of transitions between subregions.
neurons
0.6
region
Amplttude [mV]
Amplitude [mv]
200
%6
0.2
region 2nd region
1st region region
0.0
3rd region
-200
0.000
0.307
0.214
0.323
0.000
0.107
0.214
0.321
Time [s]
Time
Figure 12: "Linearized" dynamics (i.e., considering the linear map from each subregion) within the three linear subregions of the AL-RNN trained on the ECG data from Fig. The first two subregions host weakly unstable spirals with shifted phase, corresponding to the excitatory/ inhibitory phases of the ECG. The strongly divergent activity in the third subregion induces the wave.
24

---

Amplitude [mV]
Amplitude [mV]
644
0464
Time [s]
Time [s]
Amplitude [mV]
Amplitude [mV]
-1
0.0
214
0.429
0 644
0.0
0
214
0.429
0.644
Time [s]
Time [s]
Figure 13: Freely generated ECG activity using an AL-RNN with linear subregions (color-coded) shows consistent assignment of the wave to distinct subregion across multiple successful recon- structions.
Amplitude [mV]
Aman than
2.1
Time [s]
b
35
30
Symbols
25
15
#
0.0
Time [s]
iteration
Figure 14: a: Freely generated ECG activity by the AL-RNN with M 100 total units and PWL units. b: Symbolic coding of the dynamics (with each shade of blue different symbol/ linear subregion), reflecting the QRS phase with alternating excitation/inhibtionn (lighter shades of blue) following the short Q wave burst (dark blue). Time histogram of distinct symbols along the symbolic trajectory, exposing the mildly chaotic nature the reconstructed ECG signal [39].
25

---

b
1900 (700
/
M
M
V
Tral, most frequentyy visted region
stable floed point
Suddle spiaal
Time
4
Sympoll
1350
2550
Figure 15: Freely generated fMRI activity using AL-RNN with M = 100 total units and P PWL units. a: Mean generated activity color-coded according to linear subregions, with background shading highlighting the most frequently visited subregion. b: Geometrical graph representation of connections between linear subregions, with edge weights representing relative transition frequencies (self-connections omitted). c: Time series of the symbolic coding of dynamics according to linear subregions. d: Dynamics in the two most frequently visited linear subregions in the subspace of the three PWL units, with the boundary between subregions in gray. The dark blue trajectory bit in the first subregion moves towards virtual stable fixed point located near the center of the saddle spiral the second subregion. The yellow trajectory illustrates an orbit cycling away from this spiral point and eventually crossing into the first subregion. From there, trajectories are pulled back into the second subregion through the virtual stable fixed point located close the saddle spiral (see also activity with background shading in a). This dynamical behavior is similar to the one observed in the chaotic benchmark systems, where locally divergent activity the AL-RNN propelled back into the center of an unstable manifold within another subregion.
Cycle
Amax AL-RNN
0.02
Rössler
Lorenz-63
ECG
0.01
fMRI
0.00
0.2
0.5 Topo ogical entropy from symbolic sequence
0.7
Figure 16: Topological entropy computed from symbolic sequences (Figs. 14 & 15) versus Amazi calculated from corresponding topologically minimal AL-RNNs (Figs. & 7).
26

---

Ground Truth Data
BOLD Signal
0.0
-2.5
AL-RNN Activity
2
500D Sign
0
-2
200
400
600
800
1000
Time [s]
BA-10-Le
BA-40-Le BA-45-Le
BA-46-Le BA-47-Le
BA-6-Le
BA-32-Le
B4-7-Le
Figure 17: Generated fMRI activity using AL-RNN with M 100 total units and P = PWL units, with the readout unit states replaced by observations every 7 time steps.
Readout Units
Readout
Piecewise Linear
10
20
40
60
80
Wwights
Lincar Units
20
8.0
0.3
0.4
0.5
Weight Megnituee
0.76
Data
Readout Unit Weights
PWLL Units
10
-0.4
-0.2
0.0
0.2
0.4
0.6
15
15
Weight Magnitude
10
15
10
15
Figure 18: Weights of the reconstructed AL-RNN. Histogram of the absolute weight distributions for the different types of AL-RNN units. On average, weight magnitudes of the PWL units are much higher than those of the other unit types. e: The correlation structure among the weights of N = 20 readout units (rows in top) reflects the correlation structure within the observed time series variables (correlation between both matrices r x 0.76).
27

---

2-
1-
1000
1500
2000
2500
3000
-1.0
-0.5
0.5
1.0
Time
200
80
steps
150
steps
80
50
20
iooratio
iteration
Symbols
Nomule
1000
2000
3000
4000
Staps
Time Steps
Figure 19: Top row: Activity of the PWL units for the topologically minimal representations of the Rössler (a) and Lorenz-63 attractor (b) from Fig. 5. Center row: Time histogram of discrete symbols of the symbolic trajectory. Bottom row: Time series of the symbolic trajectory.
801
0.0
-2.
22
-50
rooc
Figure 20: a: Variation the location of the analytically computed real and virtual fixed points of the linear subregions aligns with the first PC of the data (generated trajectories in bluish, color-coded according linear subregion). b: Fixed point location along the first principal component (with corresponding dynamics within (z1,22)-plang of observation space top) at different characteristic stages of training. At the early stages of training. fixed points of the linear subregions are distributed along the data manifold within the subspace of readout units. Around epoch 20, the maximum absolute eigenvalues max of the Jacobians in many subregions become larger than one, inducing local divergence necessary for producing the observed chaotic dynamics.
28

---

Ground Truth
Fixed
BOLD Sigai
(dats)
0.1
0
0.0
N.
0.1
-0.2
Freely Generatee (short termn)
-0:004
Sigaal
2.5
PC (data)
-2.5
200
400
600
800
1000
Time [5]
Freely Generated (long term)
Signal
0.0
-2.5
-5.0
Time [4]
84- 04-10-Le
BA-40-14
BA-466LL
BA-6-Le
BA-9
B4-32-Le
BA L4
BA
BA BA-10
Figure 21: a: Analytically computed real and virtual fixed points the linear subregions a geometrically minimal AL-RNN (M 100, P 10) align with the first PC of the data within the subspace of readout units. The BOLD time series for different brain regions were highly correlated, so PCI accounted for approximately 80% of the data variance. b: Geometrical graph representation with relative frequency of transitions between linear subregions indicated by line thickness of edges, showing central highly connected subgraph of frequently visited (bluish) dominant subregions, as in Fig. 4. e: Example freely generated activity from ten simulated brain regions.
2.0
0.10
1
1.5
$ 1.0
0.03
10
0.00
10-6
Rössler
Lorenz-63
Rössler
Lorenz-63
Rossier
Lorenz-63
Figure 22: Same as Fig. 5d-f, but showing absolute instead of relative deviations (for D Dsteps values < 1.0 indicate tight agreement).
Rössler
Lorenz-63
ECG
10
10
10
10-1
2
2
100
110
10
-
10
ALRNN
PLRNN
ALRNN
PLRNN
ALRNN
PLRNN
Figure 23: Pairwise differences in sorted (from lowest highest probability) cumulative trajectory point distributions across linear subregions across all valid pairs from 20 training runs (quantified by the Haanogorov-Smiroo distance, Dks) for the AL-RNN vs. PLRNN, revealing much higher consistency for AL-RNN. Note the log-scale on the y-axis.
29

---

10.0
2.
10.0
5.0
Figure 24: Linear subregions (color-coded) mapped to observation space the Rössler system, showing robust representation of the individual subregions across multiple training runs/ models.
00o000
Top row: Robust placing of linear subregions (color coded) mapped to observation space across training runs using the AL-RNN. Model recovery experiments further confirmed the robustness of the model solutions, with very similar overall performance measures across different experiments (original: Datep = 3.14, DH 0.28; recovered: D Datep 3.38 + 0.18, DH 0.28 0.03: linear subregions in all cases). Bottom row: In contrast, for the PLRNN linear subregions are differently assigned (with varying boundaries) on each run.
Figure 25:
30

---

B Proofs of Theorems
Define shift space of finite type (Ax, o) such that each symbol of the alphabet of A is associated with exactly one set Us of the topological partition of S, i.c. we have a total of [24] symbols in A. Further define for a E Ay the sets [54]
Di(a) n x (Ua))
(16)
where 04 is the kth symbol in the sequence a: Think of this as building the intersection of Ua. with k-times forward iterates of subsets associated with a- and k-times backward iterates of subsets associated with ag. such that in the limit / + oo hopefully we end up with single point corresponding to unique trajectory in S, i.e. considering D(a) = N Di(a) (see Lind and Mareus [54] for details). We now define [54]
Definition 1. A symbolic representation of an invertible DS (5, Φ) with topological partition U = (Uo U..... UUn.. is a shift space (Af,o) with alphabet AA = {0... .......7 1}. such that each symbol ag E A is associated with exactly one subset Ug E U, and D(a) = Ni=o Di(a) contains exactly one point S for each AA. If AF is finite shift, we call this Markov partition.
Ideally, we would like our symbolic coding of the DS to be symbolic representation or Markov partition according this definition, but in practice this may entail other unfavorable properties (e.g.. too (ie-grrined) and we contend here with the properties given by Theorems 3.
In the statement of our theorems, we further used the terms 'eventually periodic' and 'asymptoticalll periodic'. Let us now strictly define them (according to Alligood et al. [3]): Definition 2.
An orbit {zi...., . .I of the map & is said to be asymptotically periodic if converges to a periodic orbit as n + CO. This implies that there is a periodic orbit F' {y1, y2...., Vk. V1, .2....) such that lim d(z,. [k) 0. n-+00
For instance, any orbit that is attracted to a stable fixed point or to a saddle fixed point (evolving on its stable manifold) is asymptotically periodic (fixed).
Definition 3. A point is called eventually periodic with period p for the map de if some positive integer N, (z) = 6² (z) for all n 2 N, and p is the smallest positive integer with this exact property. This means that the orbit of eventually maps exactly onto a periodic orbit.
The term 'eventually periodic' describes the extreme case where an orbit coincides precisely with a periodic orbit. Thus, any eventually periodic orbit is also asymptotically periodic, but the reverse is not always true: An asymptotically periodic orbit comes arbitrarily close to periodic orbit, but may not land precisely on it.
Note:
With these definitions we are now ready to prove the theorems.
Proof of Theorem
Proof. "="" Suppose that the orbit Us = (21...., ........ is asymptotically fixed. Hence, there exists fixed point z" Ua- of the AL-RNN Fe (i.e., x" Fe(z")) such that lim on 2".
Let (aj02a3... ... be the corresponding symbolic sequence of the orbit Is with on E Uo,, Vn. Since lim 2, = zz, so there exists some N E N such that for every n Z N the points 2, will ==000
remain arbitrarily close to z". This means the points 2, eventually enter the same linear subregion that contains z" and will remain there for all future iterations. Thus, a (a1a2as... ...0a........ (ax=))))..a (a))) eventually fixed.
"=": Assume a = (a1azas... ... av aa")) is an eventually fixed sequence of the symbolic encoding. This means for all the corresponding orbits Ug fzI...., Z.....} of Fe, there exists an index N such that for every n N the orbit points 2n must remain in the same subregion, say U... Since by assumption the map Fg is non-globally-diverginn and hyperbolic, the system cannot be expanding in all directions in any of the subregions. Thus, there must be at least one contracting or stable direction in U.-. Consequently, there must be at least one corresponding orbit
31

---

{Z1,..., En....} } that converges toward some stable structure in Ua- as n + oc. Since Fe is linear hyperbolic map in each subregion, there cannot be any k-cycle, 2 2, with all periodic points contained within single subregion. Therefore, the corresponding orhit converges to (saddle) fixed point 2" E Ua- in the stable manifold along stable directions. If all directions in Ua- are contracting or stable, then all corresponding orbits will converge to stable fixed point within that subregion. L
Proof of Theorem
Proof. "=": Let flg = {z1..... En....} be an asymptotically p-periodic orbit of Fg. Then there is periodic orbit lls {2î ....... xp) (i.c., such that all points z E Als,p distinct and 2ktp (2 2/ k 1... .......... p) and
lim d(zn; Qs.p)) = 0. n 20000
(17)
Since Fe is linear and hyperbolic map in each subregion, there cannot be any p-periodic orbit with p 2 2 where all periodic points are contained within a single subregion. On the other hand, due to the definition of the symbolic coding for each trajectory, each point 2; at time step t is assigned its own symbol Gr depending on its associated linear subregion. Hence, the corresponding symbolic sequence of the periodic orbit Sgg is (a az ... ())) with # Uai+ k 1...P Moreover, due to eq. 17, for some large enough index N E N, the points žn of the orbit S2 become arbitrarily close to the orbit S2s.p- Thus, for 2 N, the itinerary of Ug will follow the same repeating pattern as the periodic orbit's 's itinerary and will revisit the same subregions as the periodic orbit points z k = P...p. P. Therefore, the corresponding symbolic sequence of the orbit Ug is a (a102 1)) (a] o which is an eventually p-periodic orbit of the shift map a.
"<": Assume that (a102.... u.- (a ..)))..).))))).))))))))))) Au, is an eventually p-periodic orbit of the shift map a. Consequently.
og))
EN E A n N an-p- an.
(18)
which says that the symbolic sequence is repeating every p steps. Therefore, for all the corresponding of Fe, there exists an index N such that for every n 2 N the ..., Uag, and revisit each
orbits S2g = {z1...., "2,....,Zn......................
orbit points Zn will stay in the same linear subregions, say Ua; Uuj...,
Un; (i = 1, ..p after exactly p time steps:
UNi,,p = Ung-
(19)
Since by assumption the map Fe is non non- non-globally-diee giiveily and hyperbolic, the system cannot be expanding in all directions in any of the subregions. Thus, there exists at least one contracting direction in each of the subregions Ua;. Ua; .... Uaz. and therefore at least one corresponding orbit Ug (zi,-..; ......- } that converges toward some stable structure within Ua; ..... U,; as n + co. Furthermore, as F' is linear and hyperbolic map in each subregion there cannot be any k- cycle. k 2 2, with all periodic points contained within only one of the subregions. Similarly, it cannot be chaotic or quasi-periodic within just one subregion. Now, due to eq. 19, for the corresponding orbit 2gg
Vn N Vzn E U2; = 1...)) : Zutp = (Zn) = Wazn + ha E Uaf
(20)
with fixed parameters Wa = W Wayt... ...W. Since Fo is strictly affine, for N, any sub-sequence (2,+ ] m=1 of Ωg cannot be convergent to an aperiodic (i.e., chaotic or quasi-periodic) orbit. Therefore, the corresponding orbit 2s converges to a (saddle) p-periodic orbit, with all periodic points within the sub-regions Uoj Uog ..... Uaz. in the stable manifold along stable directions. If all directions in U.- are contracting or stable, then all corresponding orbits will converge to a stable p-periodic orbit with all periodic points within the subregions Ua; Ung.... ...,, Unz.
Was rrs Wrsi and ha : Ei I1; Wa h + h.
Proof of Theorem 3
Proof. "==": Let Ωs (21, 21,... 2n.... of Fo be an asymptotically aperiodic orbit of Fa. Then, there is an aperiodic orbit Ω = {21, ...) (i.e., with 2 & (Za))kk > 0) and
lim d(zn, ?) = O.
(21)
n 100
32

---

According to the proof of Theorem 2 (second part), this orbit cannot have an eventually periodic symbolic representation (a 02 ... aN- ( ap) because if it had, would need be asymptotically periodic as well.
"<": Assume an aperiodic symbolic sequence = (01 (a1,-..: an. where there is no p > 0 such that 04 o² (az jvk. This will correspond to an aperiodic succession Ua.... Ua, ... of linear subregions Ua, visited, since each subregion has its unique symbol. However, from the proof of Theorem (first part) we know that any asymptotically periodic orbit of F9 must have an eventually periodic symbolic encoding, the orbit f2g corresponding to cannot be asymptotically periodic. Consequently, cannot be (eventually) periodic either, which implies that must aperiodic. O
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The results illustrated in the text and figures properly address the presented claims, including formal proof for the theorems and hyperparameter settings and code for reproducing the results of the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations addressed in the conclusion or, c.g. in the case of theoretical results, directly in the respective section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- speciffiiiii,,nn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
34

---

Answer:
[Yes]
Justification:
The paper includes formal proofs of all theorems Appx. B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Detailed hyperparameter settings for the results in the paper given in Sect. A.2, which combined with the codebase allow for replication of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
35

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: A link to github repository is provided that contains the implementation of the network architecture that was used for the main results in the paper (https: //github. com htn/Durstewr wtt.uttwwiitaab//o/tt AALLLNNDAAAAN-AA-AA ALRNN-DSR) DSR).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
guddes//CoeeSumii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss... wwtppcccc/ppaaccccccccccccccceee haabll//guiddssodduuubiiiiii /CodeSubmissseeeeeee onnooiccy)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All hyperparameter settings are provided in Sect. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are reported in figures and with numerical results, i.c. Figs. 3, 4 and 5. The settings for the random initialization of the network architecture are specified in Sect.
A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
36

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on computational resources are provided in Sect. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:////.eu...eeeeeee wwwublic/EhissGuideeee
Answer: [Yes]
Justification: All datasets considered here were publically available. The research is aimed at facilitating interpretability of machine learning models. We believe our approach has primarily positive ethical implications. Although we have not identified specific ethical concerns, the wide range of potential applications means that possible misuses cannot be entirely ruled out.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work constitutes foundational research. As such, we do not foresee any direct negative societal impact.
37

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not foresee a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All code and results were created by us. Datasets were publically sourced and the respective authors cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
38

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
fully documented version the code is published on github.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
39

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "eXnobQxJeA", "reviewer": "Reviewer_nsFA", "summary": "This paper addresses the broad problem of learning interpretable dynamical systems from data; it builds upon existing approaches that use piecewise linear RNNs (PLRNNs), with one interesting twist: constraining the number of linear subregions to be much smaller than the \"usual\" $2^N$. This is achieved by allocating a small number P of units to have a ReLu activation function, every other unit being linear. This leads to $2^P$ linear subregions. The paper begins by proving a couple of theorems showing that these dynamical systems are amenable to interpretable symbolic analysis (though these results are generic to PLRNNs, not specific to the new version). The authors then fit AL-RNNs (this new breed of PLRNNs) to chaotic attractors, demonstrating that good dynamical reconstructions can indeed be obtained with relatively small P, providing a post-hoc justification for the approach. Finally, the approach is applied to ECG and fMRI data; in the ECG case, symbolic analysis returns a highly interpretable graph linking the different linear subregions.", "questions": "- You omitted to say that $\\phi(\\cdot)$ in Equation 1 is the ReLu function -- this is pretty critical!\n\n- In equation 4, why are you over-parameterizing the linear part? It seems that the first $M-P$ columns of $W$ can be absorbed into the first $M-P$ columns of $A$. So you really just have the last $P$ columns of $W$ to learn.\n\n- In Figure 2, why did you write \"$P^2$ subregions / symbols\"? Did you mean $2^P$? (and accordingly, should $P^4$ in fact read $2^{2P}$)?\n\n- l.185: can you please define \"hyperbolic AL-RNNs\" (AL-RNNs in which fixed points are all hyperbolic?) and what that implies concretely for A and W?\n\n- Theorems 1-3 make intuitive sense but appear a little ill-phrased to me -- for example, in Theorem 1, the specific $z^\\star$ that appears in the first clause of the iif statement is not even referred to in the other clause (and indeed it cannot be uniquely identified from knowing “the corresponding symbolic sequence $a^\\star$” -- you don't really say what you mean by “corresponding”, btw; do you mean the symbolic sequence associated with any state space trajectory that contains $z^\\star$?). Perhaps this theorem could be rephrased as \"if there exists a fixed point $z^\\star$ in $U_e$, then $e^\\infty$ is a fixed point of the shift map; conversely, if $e^\\infty$ is a fixed point of the shift map for some $e$, $U_e$ must contain a fixed point of the $F_\\theta$ map.” (?)\nSame concern in theorem 2 and 3.\nIn theorem 2, I think the $k+p$ subscript should be modulo $p$?\n\n- Should we be concerned by the fact that your measures of DSR accuracy in Figure 3 do not decrease mononotonically with the number of PWL units? Local minima in the teacher-forcing loss?\n\n- For the ECG and fMRI datasets, I couldn't see a quantitative assessment of model performance; in particular, for the fMRI dataset, I think the authors should discuss whether and how much the addition of a categorical task-stage decoder impairs relevant DSR performance metrics (or perhaps even improves consistency across training runs?).", "limitations": "Perhaps the authors could discuss the extent to which it really is easier to analyze $2^P$ subregions rather than $2^N$ subregions -- quantitatively I understand that this is a lot fewer subregions, but when $2^P$ is beyond a handful, whether it's 50 or 5 million, it's unclear to me how \"easy\" it is to analyze/understand these things (or indeed what that even means...).\n\n*post-rebuttal EDIT*:\n\nHaving read the rebuttals to all 3 revieweres, I am raising my score to an 8; strong paper likely to have an impact.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core idea is neat; although the difference between existing PLRNNs and these new AL-RNNs is more a difference of degree than a difference of nature (just changing the number of units having a ReLu activation), it's great to have thought of taking PLRNNs into that regime and to show empirically that (i) it doesn't compromise DSR quality too much, yet (ii) it gives dynamics that are computationally more amenable to symbolic analysis.\n\nAnother strength (for me at least) is that this paper contributes to exposing an audience that has historically primarily cared about dynamical systems reconstruction (e.g. me) to the concept of symbolic dynamics -- indeed I'm glad I reviewed this paper and thus got a useful (even if rudimentary) primer on SD.", "weaknesses": "I would very much like to see model recovery experiments; how easy are those amost-linear RNNs to fit, and to fit consistently? I could imagine that the model might settle into a suboptimal set of linear subregions early on during training and then have a hard time snapping out of it. I have to admit I don't have a good intuition for this, but this is something the authors could substantiate numerically by running simple model recovery experiments. On this note, what hard degeneracies do we expect here due to a majority of the state dimensions being unobserved? Can the authors use the symbolic dynamics grounding of section 3 (currently a little disconnected from the rest, I have to say) to derive meaningful measures of how well the ground truth system's topology / symbolic dynamics are recovered despite those degeneracies? (e.g. for small enough P it might be possible to look at all permutations of the transition matrix between linear subregions and conclude that the ground truth has been recovered?).\n\nRe consistency, could the authors comment on whether the error bars (across training runs) in e.g. Figure 5d-f are to be considered small or large? The figure caption says \"shows close agreement among different training runs\" but judging from these whisker plots, the underlying coefficient of variation seems quite high (which actually triggered the concern I articulated above concerning potentially inconsistent recovery of ground truth PWL dynamics).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the referee for the enthusiastic support and appreciation of our work!\n\n**Weaknesses**\n\n**W1 (consistency of fits/ model recovery)**: One crucial advantage of AL-RNNs is that they indeed consistently deliver the same model over many training repetitions. The errors in Figs. 5d-f are in fact very small: To put these numbers into context, we now normalized all 3 graphs by proper reference values (Fig. R1 in rebuttal PDF), with normalized numbers substantially below 1. Moreover, we now compared the consistency across training runs to those obtained with vanilla PLRNNs by evaluating the agreement in trajectory point distributions across linear subregions, revealing a much higher agreement among AL-RNN solutions (Fig. R2). The parsimony of AL-RNNs compared to other models leaves them with much less ‘wiggle room’ in finding different solutions, as can also be appreciated from the direct comparison in Fig. R3 (see also Figs. 20 and 21). Finally, as suggested, we also did model recovery experiments, finding that the recovered solutions are virtually identical across repetitions (original: $D_{stsp}=3.14$, $D_H=0.28$; recovered: $D_{stsp}=3.38+/-0.18$, $D_H=0.28+/-0.03$; 3 linear subregions in all cases).\n\n**W2 (degeneracies)**: While these models indeed do have the capacity to capture unobserved dimensions in their latent space (e.g. Brenner et al. 2024), we usually still would use delay embedding for this (as for the ECG). This, both, eases the training process itself and enables evaluation of agreement in attractor geometry. While it is possible to harvest the symbolic representation to measure the similarity in 2 reconstructions (overlap in symbolic codes and their graphs, e.g. agreement in adjacency matrices), for empirical data the topologically minimal representation is usually unknown, of course, and is precisely what we would like to infer through the AL-RNN (see Discussion). Hence, empirically, measures like $D_{stsp}$ or $D_H$ remain the methods of choice for initially evaluating the quality of DSR in delay-embedding spaces. But we checked this idea now for evaluating the consistency across training runs, and the symbolic graphs in fact remain identical across different runs.\n\n**Questions**\n\n*Q1:* Absolutely, thanks for pointing out this oversight!\n\n*Q2:* Yes, true, the full $\\mathbf{W}$ matrix is kind of a ‘historical quirk’ from previous formulations of the PLRNN and the resp. codebase. For model training and performance it makes no difference ($D_{stsp}$: $t(19)=-1.58, p=0.13$, $D_{H}$: $t(19)=0.68, p=0.51$), but for parsimony the diagonal of $\\mathbf{W}$ should be removed for the linear units. We will comment on this in the revision.\n\n*Q3:* Yes, thanks for catching!\n\n*Q4:* With “hyperbolic AL-RNN” we mean the system is hyperbolic in each of its linear subregions, implying that none of the Jacobian matrices $\\mathbf{A}+\\mathbf{W} \\mathbf{D}_{\\Omega}$ has eigenvalues exactly on the unit circle (a measure-0 set in parameter space). Will be clarified!\n\n*Q5:* Thanks for pointing out this source of misunderstandings, we will rephrase the theorems for clarity. The key is the term “corresponding” by which we meant the mapping from trajectories of the original system onto symbolic seq.. We will precisely define this mapping and the term “corresponding” in our rev..\n\nRe Th.2: The notation here is correct, this is the standard def. for a p-cycle (otherwise it would just read $\\mathbf{z}_k=\\mathbf{z}_k$, but we wish to indicate the map returns after p iterations).\n\n*Q6:* Some of the wiggling up and down of some of the curves after the initial kink (which we are looking for) is likely to be just noise across different training runs with different minima achieved. Also note that we are not directly optimizing for $D_{stsp}$ and $D_{H}$, so while in STF based algorithms MSE remains a good proxy in general (e.g. Hess et al. 2023), there is no guarantee the relation is strictly monotonic.\nHowever, the hump in the figure for the Lorenz-63 is a bit more suspicious, and so we dug a bit deeper: It seems that in this case the first minimum at P=2 indeed indicates the topologically minimal representation (see also Fig. R6), and that when surpassing this optimal point in fact performance first decays again. It is thus more a feature rather than a bug, and would make identifying the optimal representation even easier.\n\n*Q7:* For the ECG data, the quantitative results were not included in Fig. 7 but in Fig. 3, so perhaps easy to miss, will be made clearer. For the fMRI data, as requested we now have produced the same type of graph, included as Fig. R4 in the rebuttal PDF. That an additional categ. decoder significantly improves DSR quality on these same fMRI data has been shown before in (Kramer et al. 2022; Brenner et al. 2024).\n\n**Limitations**\n\nSince the number of subregions grows exponentially with $P$ and $M$, $P<<M$ eases model analysis profoundly. One major purpose of DSR in our minds is to provide mechanistic insight into system dynamics (rather than just forecasting). The thorough understanding of the dynamical mechanisms of chaos in Fig.5, e.g., is only possible because we have less than a handful of linear subregions (likewise for the empirical examples). Besides this important visualization aspect facilitating human interpretation, there are also clear numerical benefits: For instance, in a brute-force search algorithm for fixed points and cycles, the number of iterations we would need to find such objects also increases exponentially with $M$. Hence reducing this to $P<<M$ is always a huge benefit that enables to dig much deeper into model mechanisms. Of course it is difficult to put an exact threshold on this. However, that we were able to capture even rather complex ECG and fMRI dynamics with just a few linear subregions ($\\leq 8$) we find encouraging. Whether this will always or mostly be the case in empirical settings, is an interesting and open question, which will be discussed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LoWmmWM9hg", "reviewer": "Reviewer_82gv", "summary": "The paper proposes to limit the number of non-linear units in a RNN to facilitate the analysis and hence understanding of inferred dynamical systems. The authors show that even with a limited number of non-linear units, the model is able to explain a large portion of the data for the Rössler and Lorentz systems. Furthermore the proposed model is related to the notion of symbolic codes and it is shown theoretically how these can be used to further analyse properties of the dynamical system under study. Finally, the authors analyse insights obtained by applying the model to real-world data.", "questions": "- What is $\\phi$ in eq (1)?\n- I don't quite understand the notation in equation (2): where does the \\phi from eq (1) go? What does the subscript $\\Omega(t)$ mean, and how can there be $2^M$ configurations for $D_{\\Omega(t)}$ if $D_{\\Omega(t)}$ is a diagonal matrix? Doesn't the fact that $D_{\\Omega(t)}$ is a diagonal matrix imply that there is only a single configuration (modulo different values that they diagonal may take)?\n- In equation (6) as well as the lines of text just above it, there is a dot, which I believe may be a decimal point. Is this a common notation? I was at first thrown of by this (but I was also not familiar with the symbolic codes idea before.) If it is not standard (or maybe in general to facilitate understanding), it might be useful to add a note on this notation.", "limitations": "Limitations are well described in a dedicated section in the main text.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(Disclaimer: this is not really my area of expertise and although I like the paper a lot and have found no obvious objections in the method or the evaluation, I can not judge the novelty of the paper.)\n\n\n- Exceptionally well written\n- Well motivated and comprehensive introduction and problem motivation\n- Evaluation on both simulated data (eg Lorenz 63 and Rössler system) as well as real-world measurements (ECG, fMRI data)\n- Theoretical contribution that allows one to infer properties of the underlying dynamical system from the model via symbolic codes", "weaknesses": "- A potential weakness of the paper is the lack of comparison to other methods. On the other hand, the paper addresses a relatively niche topic so that I am not sure whether (accessible) baseline methods to compare to are available?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the referee for the supportive and positive feedback, we are happy to hear the referee liked our work!\n\n**Weaknesses**\n\nIn a sense this is the first study of its kind. We are not aware of any other work in the DSR field (and beyond) making this link to symbolic dynamics, and attempting to reduce model complexity in a way that allows for easy translations into topologically minimal representations. This approach to model interpretability is the major contribution of this work, so it is hard to compare to other methods.\n\nHowever, one may still ask whether AL-RNNs can at least compete with other DSR methods in terms of the quality of the DS reconstructions they achieve (our examples demonstrate that they are good enough even with a very low number of linear subregions, which in itself is a major advantage for obtaining mechanistic insight into dynamics). New Table R1 which we now added to the rebuttal PDF confirms AL-RNNs are at least on par with - in fact even outperform - most other SOTA methods (which lack our method’s interpretability). We also added new benchmarks to this Table.\n\n**Questions**\n\n**Q1:** $\\phi$ is the ReLU nonlinearity, sorry for this oversight. We will clarify this in the updated manuscript.\n\n**Q2:** The ReLU ($\\phi$) was absorbed into the $D_{\\Omega(t)}$ matrix: Note that we can rewrite eq. 1 equivalently into eq. 2 by placing a 1 on the diagonal of $D_{\\Omega(t)}$ for each state $z_{p,t}>0$ at time $t$, and a 0 otherwise (just by definition of the ReLU as $\\max[0,z_{p,t}] $). This matrix depends on time $t$, however, because it is determined by the values of all the states at that time. $\\Omega(t)$ denotes the subset of states for which we have $z_{p,t}>0$ at time $t$. This should also answer the referee’s last question: Since each entry on the diagonal of the $M \\times M$ matrix $D_{\\Omega(t)}$ can be either 0 or 1 at any time $t$, we have a total of $2^M$ possible configurations for $D_{\\Omega}$.\nThis may admittedly have been a bit hard to follow, especially as we didn’t make clear the meaning of $\\phi$ in the equation above! We will clarify this whole section accordingly.\n\n**Q3:** Yes, its interpretation is a bit similar to a decimal point (but note it’s not a decimal system of course): The symbolic sequences correspond to theoretically infinite length trajectories, and the point “separates past from future”, i.e. indicates which symbol corresponds to the current time $t$ along the trajectory. We will clarify this type of notation in the revision (it's indeed standard in symbolic dynamics), valid point, also for the parts above, thank you for your feedback on this!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EH0v783xGn", "reviewer": "Reviewer_5E4V", "summary": "This paper introduces a new recurrent neural network (RNN) architecture called Almost-Linear (AL)-RNN for reconstructing nonlinear dynamical systems from time-series data. The key innovation of AL-RNN is training parsimonious piecewise linear (PWL) representations of dynamical systems. By combining linear units with a small number of rectified linear units (ReLUs), AL-RNNs can effectively capture the dynamics of complex systems while maintaining interpretability. The authors demonstrate the effectiveness of AL-RNNs on benchmark datasets (Lorenz and Rössler systems) and real-world data (ECG and fMRI), showing that they can discover minimal PWL representations that accurately capture the dynamics of these systems.", "questions": "How does the performance of AL-RNNs compare to other state-of-the-art DSR methods on a wider range of benchmark and real-world datasets?\n\nHow does the choice of the number of linear units (M) affect the performance and interpretability of the AL-RNN model?\n\nIs there a principled way to determine the optimal number of ReLU units (P) for a given dataset?\n\nCan the symbolic dynamics approach be modified to better handle deterministic systems, avoiding the misleading probabilistic representation of transitions?", "limitations": "The paper does mention some limitations, such as the challenge of determining whether a topologically minimal and valid reconstruction has been achieved from empirical data.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Novelty: The AL-RNN architecture is a novel contribution to the field of dynamical systems reconstruction. It addresses the limitations of existing methods (PLRNNs and SLDS) that often result in overly complex models.\n\nInterpretability: The AL-RNN's structure, with its minimal use of ReLU units, naturally leads to a symbolic encoding of the dynamics, making the model more interpretable and facilitating mathematical analysis.\n\nEmpirical Effectiveness: The paper demonstrates the effectiveness of AL-RNNs on both benchmark and real-world datasets, showing that they can accurately capture the dynamics of these systems.", "weaknesses": "Usefulness of symbolic dynamics seems to be limited and can be misleading: Even when the underlying dynamics is deterministic, the extracted symbolic transition dynamics is probabilistic, which can be misleading (Figure 6d).\n\nLack of guidance on model size: There is no discussion on how to determine the minimum number of ReLU units (P) needed, as well as the number of linear units (M). The authors explored a range of (P) and chose an arbitrary value. The method would be much more useful in practice if it could be regularized to automatically find the minimal (P).\n\nIt is crucial that the model dynamics is partly driven by the observation data, e.g. teacher forcing, but this is not mentioned in the main text equations (eq1~5). Teacher forcing is mentioned only as a training method, not for testing.\n\nThe paper lacks details on the training process and hyperparameter selection, which could hinder reproducibility.\n\nTheorem 1 seems to be incorrect. As a counterexample, consider a linear dynamical system with a stable orbit (i.e., with 1 subregion, 0 ReLU units). In this case, the symbolic state remains constant, but not all states are fixed points.\n\nWriting needs to be improved. Section 3.2 on symbolic dynamics seems unnecessary. The symbolic partitioning is intuitive to understand in terms of the activation of ReLU units, but the formal definitions of Section 3.2 do not seem to add any further understanding. The theory section also seems unnecessary and could be moved to the appendix. Many of concepts introduced in these sections don't seem to be mentioned afterwards, e.g. shift operator.\n\nUndefined Terms: The paper does not properly define (N), which makes it confusing to understand. Additionally, the term \"hyperbolic AL-RNN\" is used without a clear definition.\n\nExcessive Use of Acronyms: The paper uses too many acronyms, which can hinder smooth reading. It would be good to reduce their usage. Here is a list of the acronyms used in the paper: AL-RNN: Almost-Linear Recurrent Neural Network BPTT: Backpropagation Through Time DH: Hellinger Distance DS: Dynamical System DSR: Dynamical Systems Reconstruction DST: Dynamical Systems Theory ECG: Electrocardiogram fMRI: Functional Magnetic Resonance Imaging FP: Fixed Point id-TF: Identity Teacher Forcing KL: Kullback-Leibler LDS: Linear Dynamical System MSE: Mean Squared Error ODE: Ordinary Differential Equation PDE: Partial Differential Equation PLRNN: Piecewise-Linear Recurrent Neural Network PWL: Piecewise Linear RADAM: Rectified Adaptive Moment Estimation ReLU: Rectified Linear Unit RC: Reservoir Computer SEM: Standard Error of the Mean SINDy: Sparse Identification of Nonlinear Dynamics SLDS: Switching Linear Dynamical System SOTA: State-of-the-Art STF: Sparse Teacher Forcing TF: Teacher Forcing STSP: ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the referee’s overall positive assessment and the valuable feedback provided!\n\n**Weaknesses**\n\n**W1 (usefulness of symbolic dynamics):** First, please note that the symbolic encoding itself is not probabilistic, i.e. the symbolic seq. (as shown in Figs. 13, 14 or 17) are as deterministic as the underlying system itself. As new Fig. R5 in the provided PDF now further highlights, quantities like the topological entropy obtained from the symbolic encoding correlate highly with quantities obtained directly from the system dynamics, like the max. Lyapunov exp., but are much easier to compute. This further confirms that important topological properties of the underlying system can be inferred from the symbolic encoding we used. In general, symbolic dynamics has led to many powerful insights and results about the dynamics of certain systems (certain proofs about chaos or the number of unstable periodic orbits could only be derived symbolically, for instance; see Wiggins 1988, Guckenheimer \\& Holmes, 1983).\n\nHence, we think the referee’s point concerns more the representation of symbolic dynamics in the form of *transition graphs*. This type of graph represent. is quite standard in symbolic dynamics (see, e.g., textbook by Lind & Marcus 2021), where arrows between nodes usually represent *admissible* transitions, just like in graph representations of finite state machines or formal languages, for example. The graphs are meant to represent the set of all possible seq. (or ‘syntactically correct’ sentences). We agree, however, that one needs to be clear about the semantics of these graphs and their interpretation, which we will clarify.\n\n**W2 (guidance on model size):** We chose $P$ according to a grid search as the min. value at which performance started to plateau (because we wanted to obtain topologically minimal representations), so its choice is not arbitrary but related to the kinks (or humps) in the curves in Fig. 3. Likewise, $M$ was determined by systematic grid search, see Appx. Fig. 9. However, we like the referee’s idea of determining an optimal number of linear subregions by regularization. We now implemented this, and find that the numbers of subregions determined this way agree well with those obtained by our prev. crit., see Fig. R6 in PDF.\n\n**W3 (teacher forcing):** Sparse teacher forcing is indeed *only applied during training*, not during testing. Hence it is, correctly, not included as part of those equations. This is in fact crucial: DSR models are supposed to be generative models that, after training, can generate new data with the same geometrical and topological structure as those produced by the observed system. During test time, therefore, the once trained model cannot rely on actual observations, but follows solely its own dynamics. While TF is a broad term, *sparse TF* is different (Brenner et al. 2022, Mikhaeil et al. 2022) and has been introduced specifically in the context of DSR where it is SOTA (see also Tab. R1). We will clarify this.\n\n**W4 (reproducibility):** Please note that we provided code for reproducibility with the org. submission here: https://anonymous.4open.science/r/SymbolicDSR-1955/README.md\n\nThe hyperpar. settings for all runs were further listed under A2: Training Method. In the rev. we will further clarify how exactly these optimal hyperpar. were determined.\n\n**W5 (Th.1 incorrect):** The th. is correct, but we see where this misunderstanding comes from: For the referee’s example, the system would need to be non-hyperbolic, i.e. would need to have a center subspace. However, this case (req. conjugate pairs of eigenvalues to lie exactly on the unit circle) we explicitly ruled out in our theorems (it is a 0-measure set in par. space). We mentioned this in the pg. above the th. (since common to all of them), but for clarity will make explicit in the theorems themselves.\n\n**W6 (theoretical sects.):** We are happy to reduce sect. 3.2 and move it partly to the Appx., the ref. is right that some of the concepts are not followed up upon. Others, like that of a shift operator, shift space, or topolog. partition, are however directly used in the theorems in sect. 4 (for the sake of the theorems, we prefer to be formally precise about certain concepts even if they may be intuitively clear).\n\n**W7 (undefined terms):** N is the observation dim. of the data, and by ‘hyperbolic AL-RNN’ we mean an AL-RNN which is hyperbolic in each of its linear subregions, i.e., such that the transition matrices defined in eq. 2 have no eigenvalues on the unit circle. Will be clarified.\n\n**W8 (use of acronyms):** Agreed, we will remove all acronyms which are not standard or are only rarely used.\n\n**Questions**\n\n**Q1:** Please note that our goal was not to introduce a novel SOTA method for DSR, but rather to introduce an approach for retrieving topologically minimal and symbolically interpretable representations from data. Still, one may ask whether the AL-RNN is at least on par with other SOTA methods for DSR. In Table R1 in the rebuttal PDF we answer this Q. We also included human EEG data \\& Lorenz-96 as further benchmarks. As can be seen, the AL-RNN even outperforms most current SOTA methods, which may be due to its simple design making training much more stable and robust (cf. also new Figs. R1-R3).\n\n**Q2:** The number of linear units makes no difference to interpretability, since it does not affect the total number of linear subregions (hence neither the symbolic encoding nor the computation of fixed points or cycles). As shown in Fig. 9, more linear units can, however, still improve performance up to a certain level.\n\n**Q3:** See W2 above.\n\n**Q4:** See response to W1 above. We also would like to note that esp. in math. chaos theory probabilistic approaches to deterministic systems are indeed commonplace, e.g. in the def. of invariant measures (which are probability ms.) and ergodic theory (see, e.g., textbooks by Katok & Hasselblatt 1995, Alligood et al. 1996).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General reply**

We thank all three referees for their thorough reading and the constructive and helpful feedback on our manuscript. We are happy to see that all referees provided a generally supportive and positive assessment of our work. We hope we could address the remaining concerns in the detailed point-by-point replies below in the individual rebuttals, and the additional new material provided in the rebuttal PDF.

In brief, the rebuttal PDF contains the following new results and figures:

* Table R1 provides a systematic comparison of AL-RNN performance to many other state-of-the-art DS reconstruction models, incl. additional benchmark systems for testing (human EEG and high-dim. chaotic Lorenz-96 system). While the idea behind the AL-RNN was not to provide a new SOTA for DSR, it does indeed outperform most other techniques when trained with sparse teacher forcing (potentially due to its simple and parsimonious design).  

* Figs. R1-R3 highlight that a particular feature of the AL-RNN, especially in comparison to the standard PLRNN, is the consistency in inferred models across multiple training runs, i.e. very similar or same model solutions are obtained across many different parameter intializations. 

* Fig. R4 is the same as Fig. 3 from the paper for the fMRI data.

* Fig. R5 shows that topological properties can be cheaply computed from the symbolic encoding.

* Fig. R6 illustrates selection of the optimal number of piecewise-linear units through a regularization approach.

**Cited References**

K. T. Alligood, T. D. Sauer, and J. A. Yorke, Chaos: An Introduction to Dynamical Systems, Springer-Verlag, New York, 1996.

M. Brenner et al. 2022, Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems, Proceedings of the 39th International Conference on Machine Learning (ICML 2022)

M. Brenner et al. 2024, Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics, Proceedings of the 41st International Conference on Machine Learning (ICML 2024)

J. Guckenheimer and P. Holmes, Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields, Springer-Verlag, New York, 1983.

F. Hess et al. 2023, Generalized Teacher Forcing for Learning Chaotic Dynamics, Proceedings of the 40th International Conference on Machine Learning (ICML 2023)

Katok, A., & Hasselblatt, B. (1995). Introduction to the Modern Theory of Dynamical Systems. Cambridge: Cambridge University Press

D. Kramer et al. 2022, Reconstructing Nonlinear Dynamical Systems from Multimodal Time Series, Proceedings of the 39thth International Conference on Machine Learning (ICML 2022)

Lind & Marcus, 2021, An Introduction to Symbolic Dynamics and Coding.. H. Cambridge University Press, 2nd edition

J. Mikhaeil et al. 2022, On the difficulty of learning chaotic dynamics with RNNs, Advances in Neural Information Processing Systems 35 (NeurIPS 2022)

S. Wiggins, Global Bifurcation and Chaos, Springer-Verlag, New York, 1988.

Meta-review: The reviewers agreed on the premise of the paper of introducing a new architecture for discovering interpretable partitions on dynamical systems. The supporting results included applications to both toy analytical systems and real world data. The reviewers all concur with an accept recommendation, all mentioning the method’s simplicity and capacity for broader impact.

The authors attached one extra page of result tables, figures, and plots, which convinced one reviewer to raise their score. The AC thinks that making the suggested corrections and stylistic edits and including the additional results would be feasible within the revision, and not represent a dramatic change from the submitted version. The AC agrees with the reviewers that the paper is an accept.
Decision: Accept (poster)