Relative Geometry of Neural Forecasters: Linking Accuracy and Alignment in Learned Latent Geometry
Authors: Deniz Kucukahmetler
Maximilian Jean Hemmann
Julian Mosig von Aehrenfeld
Maximilian Amthor
Christian Deubel
Nico Scherf
Diaaeldin Taha
Venue: Accepted by TMLR ()

Abstract: Neural networks can accurately forecast complex dynamical systems, yet how they internally represent underlying latent geometry remains poorly understood. We study neural forecasters through the lens of representational alignment, introducing anchor-based, geometry-agnostic relative embeddings that remove rotational and scaling ambiguities in latent spaces. Applying this framework across seven canonical dynamical systems—ranging from periodic to chaotic—we reveal reproducible family-level structure: multilayer perceptrons align with other MLPs, recurrent networks with RNNs, while transformers and echo-state networks achieve strong forecasts despite weaker alignment. Alignment generally correlates with forecasting accuracy, yet high accuracy can coexist with low alignment.
Relative geometry thus provides a simple, reproducible foundation for comparing how model families internalize and represent dynamical structure.

Published in Transactions on Machine Learning Research (02/2026)
Relative Geometry of Neural Forecasters: Linking Accuracy and Alignment in Learned Latent Geometry
Deniz Kucukahmetler
kucukahm@chs.mpp mpg.
Mar Planck Institute for Humon Cognätive and Brain Sciences, Leipaig, Germany School of Embedded Composite Artificial Intelligence (SECAI), Dresden presden/Leipzig, Leipzig, Germany
Maximilian Jean Hemmann
muazimiiann@jearam...
Leipzig University, Leipzig, Germany
*Julian Mosig von Aehrenfeld Leipzig Universily, Leipzig, Germany
julinnoomoosig@gmaall.cmm
Maximmiian Amthor
amthorrmacimilaaan@@maaaaccmmmm
Leipzig Universily, Leipzig, Germany
Christian Deubel
christian.denhell@mma..c
Leipzig Universily, Leipzig, Germany
Nico Scherf
nscherf@ccs.mpp
Mar Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Dresden/Leipaig, Germany
Diaaeldin Taha
taha Caluummmimmpp..coo
Max Planck Institute for Mathematics in the Sciences, Leipzig, Germany
Reviewed OpenReview: https: openreview. net/ forum? id= t43tf5Gafz
Abstract
Neural networks can accurately forecast complex dynamical systems, yet how they internally represent underlying latent geometry remains poorly understood. We study neural fore- casters through the lens of representational alignment, introducing anchor-basod, geometry- agnostic relative embeddings that remove rotational and scaling ambiguities in latent spaces. Applying this framework across seven canonical dynamical systems- ranging from periodic to chaotic- we reveal reproducible family-level structure: multilayer perceptrons align with other MLPs, recurrent networks with RNNs, while transformers and ocho-state networks achieve strong forecasts despite weaker alignment. Aligument, generally correlates with fore- casting accuracy, yet high accuracy can coexist with low aligmment. Relative geometry thus provides simple, reproducible foundation for comparing how model families internalize and represent dynamical structure.
sestems-ranging
1 Introduction
Neural forecasters- recurrent noural networks (RNNs), transformers, and reservoirs are now routinely de- ployed to model complex, time-evolving phenomena across science and engineering. While forecasting per- formance is well studied, the geometry of learned propagated latent states and how it varies across model
'Equal contribution.
E'uul supervision.
IA shorter companion version of this work appears the GTML (Geometry, Topology, and Machine Learning) 2025 workshop.

---

Published in Transactions on Machine Learning Research (02/2026)
families-remains underexplored. As their use widens, it becomes essential to understand how these fore- casters internally represent dynamical systems and whether those internal mechanisms align with human goals such as stability, interpretability, and transfer. A persistent obstacle is that latent spaces learned by different runs or model families are not directly comparable: coordinates can rotate, scale, shear, or even undergo more subtle geometric shifts with negligible effect on task loss but large effects on representational geometry. As a result, naive cross-model comparisons can be unstable and inconclusive (Figure absolute latents).
Existing alignment tools only partially address this issue. Representational Similarity Analysis (RSA) (Kriegsskorte et al. 2008) captures pairwise relational structure but remains sensitive to the geometry of the distance matrix and sampling effects; Procrustes aligument assumes an approximately isometric map between spaces and often requires careful pairing; Centered kernel aligument (CKA) Kornblith et al. (2019) improves robustness to some transformations, but can still depend on dataset sampling, layer scaling, and kernel choices. Collectively, these limitations complicate systematic studies of how different model families encode dynamics and how those encodings relate to forecasting performance.
We reuse geometzy-agnostic alternative based on relative embedddigs Moschella et al. 2023 anchor- based, extrinsic representations that index each point by its vector of similarities to fixed set of anchors. By construction, these representations ions quot.ient out global rotations and scalings, are straightforra to compute, and yield a common coordinate system in which latent spaces from different seeds, layers, and model families can be compared directly. We apply this approach for an empirical analysis encoder propagator decoder neural forecasters for dynamical systems.
This perspective matters for two reasons. First, it enables us to quantify representational families of neu- ral forecasters that is, which models converge similar relational structures even when their raw latent geometries differ. Second, it links representation to utility: we find systematic patterns in how multilayer perceptrons, recurrent networks, transformers, and echo-state networks organize dynamical information, and show that our aligument signal carries practical information about forecasting accuracy. Notably, high pre- transformers highlighting a
dictive accuracy can coexist with low cross-forecaster aligument-especialy, gap between performance and representational agreement that standard metrics overlook. By aligning latent spaces through anchor-basee based relative embeddings, we expose reproducible family-level geometry across forc- casters and offer a reproducible framework for studying how neural nelworks internalize dynamical structure.
We evaluate three neural model families multilayer perceptrons (MLPs), recurrent neural networks (RNNs), and transformers togoheo with their Koopman- (K-) and Neural Ordinary Differential Equation (NODE, augmented variants, and an Echo State Network (ESN) as a no no badddrppppaaggiin--rrru.u. time (nTT) reference model. Code available at felaaieereeeeettyyoooorororrrarr
Contributions.
We reuse the relative-embedding aligmment. framework of Moschella et al. 2023 to neural forecast- ing of dynamical systems, yielding geometry-agnostic, anchor-based latent representations that are directly comparable across forecasters. Within this framework, we train forecasters end-to-end on relative representations and demonstrate cross-family latent stitching between MLP and transformer encoders and decoders.
We conduct an extensive empirical study spanning seven canonical systems (continuous and discrete; periodic, quasi-periodic and chaotic) and three model families and a no-BPTT baseline.
We uncover consistent family-level alignment patterns and characterize their relationship to forecast- ing error. We show that high predictive accuracy can cocxist with low alignment- most prominently in transformers and ESNs highlighting the limits of task loss alone and motivating representation- aware evaluation.
Together, these results suggest that anchor-based relative embeddings provide simple, scalable basis for reproducible representation science in neural forecasting, enabling more faithful comparisons across seeds,
2

---

Published in Transactions on Machine Learning Research (02/2026)
layers, and model families and offering new insights into how different model families internalize dynamical structure.
Scope clarification. Throughout this work, aligmment with the "true system" refers exclusively to align- ment with the relative representation of observed trajectories under a shared anchor set, not to recovery of the system's governing equations, physical state variables, or dynamical invariants.
Learning objective and interpretation. All models in this study are trained solely to minimize forccast- ing loss. Representational alignment is used as an analysis lool, not as an assumed or enforced consequence of the training objective. Observed aligument- or lack thereof with the ground-truth relative representation reflects architectural inductive biases and task-induced representations, rather than evidence that minimiz- ing forecasting loss recovers the underlying system dynamics. central empirical finding of this work is precisely the divergence between forecasting accuracy and representational alignment, most. prominently in transformers and ESNs.
2 Related Work
Dynamical systems. Dynamical systems theory, from Poincaré's recurrence to modern hyperbolic dy- namics, provides the mathematical backbone for modeling time-evolving processes, (Arnold 1989 Katok
& Hasselblatt 1995 Strogatz 2018 Compact, low-dimensional models such as the Lorenz-63 attractor
Lorenz 1963 and the logistic map (May 1976) famously revealed sensitive dependence on initial con- ditions and the geometry of strange attractors (Ruelle, 1978h. Variants, including the higher-dimensional Lorenz-96 system (Lorenz, 1996), the Hamiltonian double pendulum, and the Hopf normal form, have since become canonical benchmarks for lesting data-driven approaches. In fluid mechanics, proper orthogonal de- composition (POD) reductions of the cylinder wake (Brunton et al. 2016) serve tractable proxy for the Navier-Stokes equations. These systems are now present in most neural forecasting benchmarks: reservoir computers Pathak et al. 2017; Matzner & Mráz 2025 back-propagating RNNs (Vlachas et al. 2020
physics-informed latent ODEs RRaisii et al. 2019). and Koopman autoencoders (Lusch et al.] 2018) are all routinely evaluated on one or two of them. We adopt the full suite Lorenz-63 63, logistic map, Hopf oscillator, double pendulum, and POD-wake thereby spanning periodic, quasi-periodic and chaotic regimes in both continuous and discrete time. This variety allows us study how representation alignment behaves under qualitatively different underlying flows.
Neural forecasting. Modeling and forecasting the evolution of dynamical systems is a cornerstone of scientific inquiry. Methods for dynamical system forecasting cover a wide spectrum, with first-principles modeling (Strogatz 2018 Arnold 1989 and data-driven modeling as two extremes. In the latter approach, which has gained popularity due to the availability of data and computing resources, the learned latent geometry of the system are learned directly from observations. Foundational work in nonlinear lime-series analysis demonstrated this possibility by reconstructing system dynamics from data (Takens) [981 Kantz & Schreiber 2004). Today, this tradition is dominated by a diverse family of 'ncural forecasters, inchuding RNNs (Hochreiter & Schmidhuber 1997). transformers (Vaswani al. 2017), Neural Ordinary Differential Equations (Chen et al.] (2018), and forecasters inspired by Koopman operator theory (Lusch et. al. 2018
Our study is situated within this data-driven context.
Represennational alignment. Foundational work in neuroscience representational similarity analysis (RSA) provided framework by comparing activity patterns by analyzing their distance matrices (Kriegesko- rte et al. 2008). In machine learning, similar methods are used, like Procrustes analysis, which socks an optimal rotational alignment between two sets of points (Gowerr 1975
Schönemann 1966). and, more re-
cently, centered kernel alignment (CKA), which has become a standard for comparing neural representations across different initializations and model families (Kornblith et al.
2019
Ding et al.) 2021p. Other methods
aim create structured mappings between latent spaces using techniques like topological conjugation BBzzi
et al.
2025
3

---

Published in Transactions on Machine Learning Research (02/2026)
Moan Squared Error (MSE)
Impul Rp-2432
0.1
0.2
Encocer Au.
(RNN)
Representational
RNN
Simitrrty Score
MLP
Encoder (MLP)
Mee MJ
Lorenu-83
Absolute
Relative Latent
Cross-mode sigrment
Afigiment performance
MSE (olliaatoo))
POAS
PCR
PCA
PCA
RSS
Double Pendulum
Random
Limit Cycle
2000
02
24
PCA
PCA
Figure 1: Relative embeddings reveal consistent geometric structure across model families while removing rotational and scaling ambiguities. (a) Encoder E propagator decoder forecasters lake input window of L past states X;-1+1d, embed it into a latent vector 2, and decode prediction of the next H states & To compare different forecasters, we compute absolute latent embeddings from data, transform them into anchor-basee relative embeddings using Moschella et al.
2023 and quantify alignment between forecasters using representational similarity scores. (b) Alignment, performance endpoints after training for RNNs (blue) and MLPs (green). RNNs achieve higher representational similarity and prediction accuracy (MSE), while MLPs show a clearer correlation between alignment and performance across seeds. (c- f) Example systems: Lorenz-63 (c), double pendulum (d), random skew (e), limit cycle (f). Columns display system trajectories, absolute embeddings (PCA; two or three principal components depending on dimension- ality), relative embeddings (PCA), cross-forecaster similarity heatmaps averaged over five seeds ordered as True System, MLP, Koopman MLP, NODE MLP, RNN, Autoregressive RNN, Koopman RNN, NODE RNN, Transformer, NODE Transformer, Koopman Transformer, and ESN; and alignment performance scatter plots across hyperparameter settings. Additional systems are shown in Appendix Figure and

---

Published in Transactions on Machine Learning Research (02/2026)
Relative representations. In this work, we adopt related but more direct approach that was first ap- plied to computer vision models: anchor-based relative embeddings, which establish standardized relational coordinate system to make latent spaces directly comparable (Moschella et al. 2023 Instead of defining a point's identity by its absolute coordinates, this technique represents it relationally through its vector of similarities a fixed set of anchor points-Ut thus overcoming geometric ambiguities latent. spaces.
Building on this foundation, recent works have generalized relative representations. Anchor- based methods have been used to merge multiple latent spaces into single aggregated that preserves each space's geometry, akin to fusing several maps into unified atlas (Crisostomi et al. 2023 This principle of latent- space stitching extends to other domains: unimodal vision models can be stitched into multimodal model without additional training (Norelli et al.) 2023), while RL agent policies can be stitched to form new agents for unseen visual-task combinations (Ricciardi et al. 2024p. Further refinements add topological and geometric stability for zero-shot stitching (Garcfa-Castellanos et al. 2024). and even show that simple lincar
transformations can rival anchor- basod methods in latent space alignment [Lähner Moeller 2024).
Building these insights, Latent Functional Maps (Fumero al.
2025
introduce spectral formulation
that enables robust cross-space transfer. Similarly, Maiorca et al.
(2023
estimate direct transformations
between latent spaces without training decoders on relative representations.
Lastly, Cannistraci et al. (2024)
propose constructing product latent spaces composed of multiple invariant. components, each induced by distinct similarity functions.
Anchor-based relative representations are closely related to landmark-based methods-long used in dimen- sionality reduction, clustering, and kernel learning Faloutsos & Lin (1955) De Silva. & Tenenbaum (2004)
Oglic & Gärtner (2017) Chen & Cai (2011 Liu et al. (2010). Using landmarks, point is represented the distance or similarity to fixed set of landmarks. Anchor-based approaches extend this to neural latent spaces.
This study considers such relational and anchor-basod techniques within the domain of dynamical systems lorecasting, where comparable latent spaces are essential for analyzing, aligning, and transferring represen- tations across contexts.
3 Method
3.1
Representational alignment experiment design
The representational alignment framework. Following Sucholutsky al. (2023 representational aligmmmnt experiment. consists of data, systems (models, in our case), measurements, embeddings and similarity metric. We spell out these ingredients for our study:
Data: simulated trajectories from seven dynamical systems (Sections 3.2 and
Neural forecasters: each trained encoder propagator decoder model instance (seed, model) (Sec- tions
3.3
and 4.2
Measurement operator the encoder's latent vector z de. (xi-1+1it) (Section 3.4
Embeddings: anchor-basee relative embeddings r(z) obtained by z-scored distances (Section 3.5
Similarity metric: cosine, rank and T1 similarity of two relative embeddings (Section 3.6
We provide summary of the notation used in this section in Table 1
Representational alignment task. In the sense of Sucholutsky et al. (2023). this work primarily ad- dresses the measuring regresentational aligrmmen lask: we quantily pairwise similarity across encoder ini- tializations and model families and examine how that similarity relates to forecasting loss. We also explore aspects of bridging through fross-family latent stitching, though alignment remains substantially stronger within than across families. Developing effective bridging mappings and alignment-driven training interven- tions is left to future work.
5

---

Published in Transactions on Machine Learning Research (02/2026)
3.2 Data: Trajectories of dynamical systems
A dynammical system is triple (T, X, +) in which T is an addilive monoid that plays the role of time (e.g.) T = IR for continuous time, or = E for discrete time), X is non-empty state space, and & : x X + X
is the evolution map (also called the flow) satisfying $(0, 2)) = x and l(t2, #(),,=)) = (t1 + 12, x)) for all admissible t1,t2 E and z E X. For a fixed initial state x E X, the curve Dz T + X, + (tt, x) is the
trajectory (or orbit) through z; image Tr = (+(,,)) x) t E T) is the set of states visited over time.
3.3 Model: Neural forecasters
Given a window of past states X--++1: LLxd the aim of the forecasting task is to predict the next H steps X++1++ R In this study, employ prcopeerproopgatoor-- decoder neural networks 9 the Pe de as forecasters: the encoder de, RLxdd + B maps the input slice to a latent vector, the propagator Pe R + R* evolves that. latent, and the decoder tha, R - prediction. Parameters are trained to minimise a forecasting loss Lpred (we use mean-squared error (MSE)) over trajectories drawn from the unknown dynamical system. We use the term forecaster for a trained model instance (archilecture, hyperparameters, and learned weights), and model for the corresponding rained architecture or configuration.
Hxd produces the H-step
3.4 Measurements: Latent representations
The encoder Φe. R. + & maps an input segment latent vector R Training the same model with different random seeds, or swapping to different model, yields family of encoders (o (+) 1 latent space aligument the subject of this study.
( s==1 whose
3.5 Embeddings:
Anchor-based relative embeddings
Let V = {2 C I denote the set of latent representations obtained by applying the encoder to input windows:
Zj = Φe_ (Xt, -++11t+ - 1g.
We select subset A {a 1m C V as anchors. Let sim R x R E be similarity function.
Relative embeddings via z-scoring. Each encoder produces latent vectors 2j de (Xe, -++ 1:t,) E R* which are first z-scored feature-wist across the dataset. fixed subset {a C {z, j== serves as anchors, and cach normalized latent yields relative embedding
1N
z' = Trei (sim(z, a.)..,,...................
sim am)).
where sim(-.-) denotes a similarity function introduced in Section 3.6 This produces, for each forecaster, a matrix Ryel E Nm whose rows correspond to data points and columns to anchors.
3.6 Similarity metric: Similarity of two encoders
We quantify the similarity between two encoders @f (1) metrics:
(( and (2) dataset V using three complementary の(a)
cosine similarity, rank similarity, and T1 score. Each of these captures different aspects of agreement between the encoders' relative embeddings x111 and x(2)
Cosine similarity. The representational similarity score (RSS) is defined as the mean cosine similarity of the relative embeddings:
Oceos (00) (00 0
(1) 2 ((2))
2 ||2
where z((x) rel (z) denotes the relative embedding induced by encoder s E (1. 2), and acos denotes the RSS used throughout experiments. Rank similarity and T1 score defined in Appendix B
(z)

---

Published in Transactions on Machine Learning Research (02/2026)
3.7 Stitching
We define a stitched model as the composition of an encoder that produces a latent, fixed relative transfor- mation with a global anchor set A (Moschella et al. 2023 Crisostomi et al. 2023), and a task-specific prop- agator-decoder that operates in the [Al-dimensional relative space. Concretely, the encoder Φ9. :R Lxd RR* maps an imput window to a latent vector, which is mapped to a relative representation via cosine similar- ities to (z-scored per anchor) (Moschella al. 2023). The propagator Pe TR b RIAI and decoder Veg IR [g #xd are trained end-to-end in this relative space.
Crucially, because all decoders consume the same relative representation, any trained decoder can be stitched to any trained encoder without additional training (Moschella et al. 2023 Norelli et al. 2023 Ricciardi et al. 2024b. We evaluate stitching by swapping encoders and decoders across families and reporting H-step MSE For comparison, we also train absolule variants that omit the relative Iransform; such models can only be stitched when latent dimensions match and are generally less stable (Lähner & Moeller 2024 Maiorca et al. 2023 Recurrent model familics (c.g., RNN/ESN) are excluded from cross-family stitching due their dependence on hidden state, which is not provided by non-recurrent encoders.
4
Experimental setup
4.1 Dynamical systems
Dynamical systems considered. We evaluate neural forecasters on collection of canonical dynamical systems spanning discrete and continuous time, dissipative and conservative dynamics, and low- to moder- ately high-dimensional state spaces (details in Appendix D For clarily, briefly summarize the qualitative dynamical regime represented by each system.
Lorenz 63 (chaotic, dissipative). A three-dimensional continuous- system with a strange attractor, characterized by sensitive dependence on initial conditions and strong nonlinear coupling. represents a classical example of con-dimeniional dissipative chaos.
Stable limit cycle system (periodic). A two-dimensional radial-spiral flow whose trajectories converge to a closed orbit. This system provides a simple nonlinear periodic regime with smooth and predictable long-term behavior.
Double pendulum (Hamiltonian chaos). / 'A four-dimensional energy-conserving mechanical eys- tem exhibiting chaotic motion duc to nonlinear interactions. Unlike dissipative chaotic systems, trajectories evolve on a conserved-energy manifold.
Hopf normal form (nonlinear periodic). A two-dinensional system undergoing a supercritical Hopf bifurcation, producing dingleffrquence stable limit cycle. It represents weakly nonlinear periodic dynamics near the onset of oscillations.
Logistic map (discrete chaos). A one-dimensional discrete-time system at a parameter value yielding chaotic behavior. Its stretching-and-fool dynamics provide a canonical example of discrete-time chaos distinct from continuous flows.
POD wake (reduced spatiotemporal dynamics). A three-mode Proper Orthogonal Decompo- sition of a fluid wake, capturing coherent structures of an underlying high-dimensional spatiotem- poral flow. The resulting reduced-order system exhibits multi-scale temporal variability inherited from turbulent dynamies.
Skew-product system (high-dimennsional coupled chaos). A six-dimensional system formed by weakly coupling multiple chaotic subsystems Lai et al. (0025). This construction introduces interacting but partially separable chaotic modes, increasing effective dynamical complexity while retaining interpretable structure.
I qquasi-periodic for low amplitude

---

Published in Transactions on Machine Learning Research (02/2026)
iEEG recordings (real neural dynamics; external test case). Intracranial EEG (iEEG) time series from a human subject Ghosh (2024) providing a high-dimensional, noisy, and partially observed real-world dynamical system. We use this dataset as an external validation to assess whether the relative geometric trends observed on synthetic systems transfer to empirical neural data, locusing on representational geomelry rather than neuroscientific interprelation (details Appendix
Unless noted otherwise, all synthetic dynamical systems provide trajectories from 30 distinct initial condi- tions, each of length 7-500 time steps. These trajectories are equally split into training, validation, and test sets, and a sliding window is used for data augmentation. All channels are z-scored using statistics computed on the training split, and no external noise is added. Data generation scripts for the synthetic systems are provided in the GitHub repository.
4.2 Neural forecasters
Given an input window of L past states X Ia E BRixd the forecasting lask is to predict the next H steps X1-1+# E xxx All decoerr models share the factorization
R411+H = Vo,(Pe (x: (x:-1+1=)),,
with encoder őœ, RRd + Rt, optional latent propagator Pe : RR* + R² and decoder the IR* + #xd We instantiate this framework with MLP, RNN, and transformer families, together with their K-, N-, and A- variants defined in Section 1, and include an ESN baseline described at the end of this section.
Latent state propagation. To impose temporal structure in the latent space, the encoder maps the input window to initial latent state 2o 60, (x)),+ which is then evolved forward for H steps through latent propagator Pe. The terminal latent state z is decoded to produce the forecast, & (o.(zH) We consider the following choices for Pe:
Identity:
ZH = 20:
Neural-ODE: Se(2, 1),
2H = RK45 (34455(e) So, Zo, HDt):
Koopman (linear): Zb+1 K Zk. k= H - 1. R4XA
In the identity case, the model reduces to standard one-shot encoder decoder forecaster, 8441:411 (do (x++)+1))). A summary of the propagators is provided in Table
Transformer forecaster. Our transformer forecaster follows a standard encoder-decodee decoder architecture with multi-head self-attention, feed-forward layers, residual connections, and sinusoidal positional encodings (Vaswani et al. 2017). In contrast to recurrent models, the transformer does not maintain or propagate an explicit latent state across forecast steps. Instead, performs block (one-shot) multi-step prediction: the encoder summarizes the input window into latent representation, and the decoder predicts the entire H-step forecast in single forward pass. Causal masking is applied in the decoder to preserve temporal ordering within the prediction horizon, but this masking does not induce recurrent hidden-state evolution. As result, the transformer's internal representations need not form smooth latent trajectories over forecast time, which distinguishes it from RNN- and propagator-based forecasters.
Reservoir baseline. The echo-state network does not use an encodor-docoder split. A fixed sparse reser- voir updales via r tanh(Wr: + Uxx); all inputs are retained (no wash-out). Only the linear read-out Wout is fitted by ridge regression, providing a no- co-BPTT reference.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Summary of notation used for trajectories, latent states, anchors, and representational similarity.
Symbol
Description
Trajectories and dynamical systems (Section 3.2
T
Time index set (discrete or continuous). State space of the dynamical system. Evolution map of the dynamical system. Initial condition or system state. Trajectory starting from I. Image of the trajectory 2x.
X
4 T x X
x NEX (n)
Latent states and forecasting model (Section 3.3 3.4
L
Input (context) window length.
H
Forecast horizon.
Tt-L+1a
Input window of observed states.
Latent state produced by the encoder.
Φ0.
Encoder mapping inputs to latent space.
Pe
Latent propagator (identity, NODE, or Koopman).
z8
Latent state after H propagation steps.
Decoder mapping latent states to predictions.
Anchors and relative latent representations (Section 3.5 Section 3.6
N
Number of encoded input windows used construct the latent set V. Set of latent states obtained by encoding N input windows for model m. Anchor set, subset of V. (per model; anchors correspond to the same sampled input windows across models) i-th anchor latent vector.
V
{z 1 A {ai) (=
ai
m
Number of anchors.
z'
Relative embedding of latent a with respect to anchors.
z!
i-th coordinate of z'.
sim(:,)
Similarity function between latent vectors.
Occos
Mean cosine similarity between two models relative embeddings
Table 2: Encoder Propiiearrrraaatrrrreeeee Decoder decompasition across model families.
Model
Encoder
Propagator
Decoder
MLP
MLP forwar))rward)
Identity (P(x) =x)
MLP foed--frwwad))
RNN
RNN (GRU)
Identity
RNN (GRU)
A-RNN
RNN (GRU, autoregressive)
Identity
RNN (GRU, autoregressive)
Transformer (TF)
Transformer (causal attention)
Identity
Transformer (causal attention)
N-MLP, RNN, TF
Same base model
NODE: (z(2
Same base model
MLP, RNN,
Same as base model
Linear: 2s+1 Kzz
Same as base model
None (random reservoir)
Th+1 = tanh Wry + Ux+)
Linear readout
4.3 Training details
Optimisation. Adam optimiser, step size 10-3, exponential decay factor 0.95. Early stopping (pa- tience 20) monitors validation MSE. The hyperparameter tuning settings and results are reported at compiled_
and the tuned parameters used the reported experiments (i.c., cross-forccastor alignment, benchmarking, and perturbation experiments) are provided best cirtt_mdee__ppeee poodel_parameters.cst for all trainable models, and at esn leseennhaaaaa hyperparams.....mmm csv for the ESN.

---

Published in Transactions on Machine Learning Research (02/2026)
Key hyperparameters. For each dynamical system and each model variant (excluding ESNs), selected the best configuration from the hyperparameter search based on validation performance. Across all MLP, Koopman-MLP, and NODE-MLP models, selected learning rates lay in the range 5 x 10-4- Batch size was typically 64, with NODE-MLP models consistently using batch size 32 across systems. Latent dimensionality ranged from to 256, while hidden layer widths varied between 128 and 1024, depending on system complexity.
RNN, Koopman-RNy RNN, and NODE-RNN models used encoder widths between 64 and 512 (most frequently 256), with 2-5 layers in both encoder and decoder. Latent dimensions for these models ranged from 32 to 128 across systems.
Transformer, Koopman- sions between 128 and 384, using 2-5 attention heads. Batch size was 64 for Transformer and Koopman- Transformer models and 128 for NODE-Transforee Transformer models, with learning rates of either 10-3 or 10-4 depending on the specific system model combination. Dropout was applied selectively and most often set to 0.1.
Transformer, Koopman-TTansfornee,
and NODE- Transforurr models consistently selected model dimen-
No single architcctural variant (identity, Koopman, or NODE) dominated uniformly across all systems; instead, different variants were preferred for different system-mode model combinations. Full hyperparameter ranges and corresponding validation performances are reported in compiled results. csv and the best combinations producing the lowest validation MSE are reported in best_eodeele Deet_model_pprraneters.. www.ppaaaaaa.ccmm
4.4 Evaluation metrics
Test performance (five random seeds) reported using (i) mean-squared error (MSE), (ii) root-mean-squared error (RMSE), and (iii) mean absolute error (MAE). Each metric computed per step and then averaged over the 50-step forecast.
5 Experimental Results
Table 3: Performance and Alignment for lorenz
Performance (4)
Similarity (t)
Model
MSE
MAE
RMSE
Cosine
Top-1
Spearman /
MLP
0.3828 0.0080
0.2804
0.0050
0.6187 1. 0,0064
0.7148 + 0.0193
0.3786 4: 0.0025
0.7121 1 0.0187
Koopman
0.0268
0.0179
NODE MLP
0.3684 0.0061
0.2830
0.0176
0.6070 1 0.0067
0.7489
: 0.0088
0.3028
1 0.0008
0.7476 1 0.0115
RNN
0.0096 + 0.0018
0.0570
0.0073
0.0979 + 0.0092
0.9125
+ 0.0128
0.5100
+ 0.0162
0.9031 = 0.0117
Autorogresiive RNN
0.D422 0.0382
0.1172
0.0106
0.2045
0.0203
0.9084
0.0124
0.5084 + 0.0124
0.8090 + 0.0150
Koopman RNN
0.1259 0.0364
0.0282
0.3320 0.0503
0.0367
0.4962
0.8824
NODE RNN
0.2629
0.0340
0.3931 0.0824
0.9033
+ 0.0113
Transfomer
0.0044 0.0014
0.0547
0.9085
0.0683 1 0.0102
0. 0.7263
1: 0.0473
0.4316 1 0.0167
0.7295 1 0.0417
Koopman Transformer
0.0129 1 0.0019
0.0682
0.0056
0.1 1 0.0085
0.7447
1 0 0.0585
0.1208 1 0.0082
0.7467 1 0.0337
NODE Transformer
0.0244 0.0350
0.16
0.0111
0.1556 + 0.0153
0.6375
0.1201
0.3398
+ 0.0322
0.6192 4 0.0800
P8N
0.0050
0.0256
0.0026
0.3124
We next evaluate how relative embeddings capture representational geometry across model families, systems, and training conditions, focusing on (i) cross-family alignment, (ii) its relationship to forecasting accuracy, and (iii) its stability under perturbations.
Relative embeddings establish a shared representational space across model families. Figure 1 illustrates that anchor-based relative embeddings reduce geometric arbitrariness (rotations, scalings) in latent spaces, making forecsorre comparisons more interpretable. With colors indicating distinct forecaster labels, the relative space reveals similarities and differences across forecasters in common coordinate system. For completenoss, we also quantitatively assessed cross-forecaster alignment in the original latent spaces, confirming substantial misaligument (Appendix Figure
Models form reproducible family-level aligument patterns. Cross-forecaster similarity in Figure !I (pairwise alignment heatmaps; cosine similarity of relative embeddings) reveals consistent family structure across systems: (i) in all systems, the MLP family (MLP, Koopman-Wy -MP, NODE-MLP) forms a cluster; (ii) the RNN family (RNN, autoregressive RNN, Koopman- RNN, NODE- RNN) is well-aligned in all systems
10

---

Published in Transactions on Machine Learning Research (02/2026)
except the Logistic Map (Appendix Figure 50. where alignment weakens; (iii) the ESN bascline exhibits noticeably lower alignment in Lorenz-63, double pendulum, and the random skew product; (iv) the trans- former family tends to align less with other familiee prominently in double pendulum, Lorenz-63 and random skew- suggesting a different inductive bias in how context is summarized for forecasting. Over- all, these patterns indicate that architectural choices induce reproducible representational geometries within familics, while some dynamics (e.g., Logistic Map) challenge specific families (RNNs). As an external vali- dation, we additionally report preliminary results on a high-dimensional real-world iEEG forecasting task in Appendix F
where we observe qualitatively similar family-level alignment patterns across architectures.
Forecast accuracy and representational alignment diverge across model families.
To assess the alignment with the true system for each forecaster family more systematically, we trained multiple forecasters per dynamical system (Figure | Aligument vs. Performance column; single forecaster results Appendix Figure [0] and plotted the final MSE against the alignment with the truc system. For MLPs, performance is more strongly related with aligument. Transformers, on the other hand, exhibit higher variability: they achieve both the best and worst scores across seeds, and strong performance does not always coincide with strong aligmment. However, they rarely appear in the bottom-left quartile of the plot, indicating that transformers typically do not show low performance and low alignment jointly. RNNs mostly cluster in the top-right quadrant, suggesting that they consistently attain both high performance and high aligmment. Overall, across model families, we observe a general positive relationship between representational similarity and forecasting accuracy, although its strength varies by forecaster family and dynamical system (Figure I
Next we studied performance to alignment with the true system during training. We observe family-specific training trajectories (First column in Figure 2 RNNs begin with comparatively high alignment and remain stable through training across systems (except for Logistic Maps), while their test error decreases steadily, MLPs either show a similar pattern to RNNs (high aligmment from the beginning, seen in all systems except Lorenz-63 and double pendulum) or start with lower alignment that increases as training procoeds (seen in Lorenz-63 and double pendulum), tracking improvements in error. Transformers display lower and more variable alignment across seeds, yet often achieve competitive or superior performance- frequently surpassing the MLP family and often rivaling RNN variants. This underscores that high alignment is helpful but not strictly necessary for strong forecasting: transformers can achieve good accuracy with representational geometry that aligns less to the ground-truth relative space.
Noise and input length differently affect representational stability across forecasters. To evaluate the effects of practically relevant parameters such as imput noise or available (input) sequence length (I.), we measured both predictive performance and representational aligument. with the ground- truth dynamics under varying noise levels and sequence lengths. For these experiments, we selected one representative forecaster from each major family and report results for MLP, transformer, and autoregressive RNN (Figure 2 see Appendix t for the remaining systems).
Inercasing the noise level consistently degraded both alignment and forecasting accuracy, but impacts the forecasters differently. Across all dynamical systems, RNNs tended to lose representational similarity more rapidly than predictive performance with a nearly linear trend. In contrast, transformers show a more nonlinear behavior, where performance decreases steeply with noise in the Lorenz-63 and double pendulum systems. A similar pattern can be seen for MLPs for the limit cycle. The random skew system shows a qualitatively similar picture but the patterns are overall less clear in this case.
The effect of imput length (L) varied across both forecasters and dynamical systems. In many cases, neither performance nor alignment changed dramatically with L (like for transformers in Lorenz, random skew or limit cycle), though notable exceptions exist. RNN and transformers show a similar pattern: performance remained largely stable across systems, but alignment exhibited high variability for some systems (double pendulum and logistic-map for RNNs and double pendulum and POD-wake for transformers). By compar- ison, MLPs were more sensitive to longer input windows- their performance and alignment degraded with increasing L in the Lorenz, double pendulum, limit cycle, POD-wake, and Hopf systems, while remaining stable in the random skew and logistic-map settings.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Training-lime evolution
Input noise perturbation
Sequence length (L)
Test performance
a
Lorenz-63
b
d
MSE (valldation)
0.01
MSE (validation)
0.01
0.05
MSE (validation)
144
192
0.05
MSE
0.1
0.1
0.4
RSS
RSS
0.9
RSS
KLP
N: MLP
A RNN
RNN
N-
ESN
K K/MUP
RNN
RNN
MSE (validation) 6
Double Pendulum
f
h
MSE (validation)
MSE (validaation)
0.01
0.2
0.1
MSE
0.1
0.4
RSS
1.0
0.3
RSS
0.8
0.2
RSS
1.0
MUP
N4-MP
A-IMN
Random Skew
k
KWRUP
0.3
MSE (valiciation)
MSE (validation)
0.001
MSE (validetioo)
0.1
0.01
0.06
MSE
1.0
1.1
RSS
RSS
0.2
RSS
MLP
N-MLP
A-IVVN
K-TEEN
N-TF
ESN
Limit Cycle
n
p
K-KUP
FNNN
I-TF
MSE (validation)
MSE (validation)
MSE (validation)
0.0001
0.001
0.001
0
0.001
MSE
0.01
0.1
0.1
0.01
0.007
0.75
RSS
0.86
RSS
RSS
MUP
K K.MPP
NUPP
ANN
A RAN
IN RAN
RPN
NTF
ESN
Performance alignment trade-offs across training, noise, and input conditions. Columns show (a-m) training time evolution, (b-n) effects of input noise, (c-o) effects of sequence length L, and (d-p) test performance across model families (MLP, K-MLP: Koopman MLP, N-MLP: NODE MLP. RNN, A-RNN: Autoregressive RNN, K-RNN: Koopman RNN, N-RNN: NODE RNN, TF: Transformer, N- TF: NODE Transformer, K-TF: Koopman Transformer, ESN). Each point represents the mean squared error (MSE) and the representational similarity score (RSS) of a given a forecaster trained with a different random soed (color-coded by forccaster family; same-colored lines/points denote different initializations of the same forecaster). MLPs and RNNs exhibit consistent performance aligmment relationships, while transformers (no-BPTT) train- ing. Increasing input noise consistently degrades both alignment and accuracy, whereas varying L produces system-dependent effects, highlighting differences in robustness across model families. Test results (d-p) indicate that no single family dominates across all dynamical systems. Results for additional systems in Appendix D
Figure 2:
show larger variability; FSNNs are excluded due to their no- backpropagatioo--troog------
These results highlight that rescarchers interested in geometric or representational stability, rather than ac- curacy alone, should consider noise levels and input-length choices when selecting forecasting model families.
Alignment estimates stabilize with increasing number of anchors. We empirically assess how the number of anchors affects relative-ropresentaae alignment between our pretrained MLP-MLP forccaster and the ground-truth Lorenz-63 system by correlating their anchor sample similarity matrices using the met hod Moschella et al. (202)). We swept the number of anchors K E (1,2,3,4, 4,5, 6, 8, 16, 32, 64, 128, 512, S00, 999} and, for each K, repeated the estimation 30 times with independently sampled anchors (without replace- ment). The alignment estimates display an approximately constant mean once K 2 16 (around r zs 0.74), while the across-repeat variability decreases markedly with increasing K (Appendix Figure This vari-
12

---

Published in Transactions on Machine Learning Research (02/2026)
Transfomees
ESN
Temporal Similarity
Lorenz-63 63
M
KAML
VAP
ASS
PAN
b Double Pendulum
Time
PSS
Random Skew
Time
Time
Limit Cycle
RSS
676
Time
Figure 3: Temporal evolution of representational alignment across dynamical systems. Each row shows the true system (left), reconstructed trajectories from different model families (MLPs, RNNs, transformers, ESN; same colour coding as in Figure 2 and their temporal similarity profiles (right; line thickness encodes time, T 300). For visualisation purposes we use relative coordinates z with respect to three anchor points (axes z1, z =/$)- For the Lorenz 63, double pendulum, and random skew systems, MIPs and RNNs maintain representations closely aligned with the true dynamics, whereas transformers and ESNs diverge. For the limit cycle and other periodic systems (see Appendix 7 all families capture similarly structured representations. The Logistic map is omitted due to its one-dimensional, contractive behavior.
ance reduction indicates convergence of the estimator more anchors are used. Balancing stability and computation, fixed K = 80 for all experiments except in the stitching experiment, where K = 32
used.
As random baseline (orange line in Appendix Figure g we drew distinct anchor sets for the forecaster and the true system. Under this mismatch, alignment was near zero across all K (r x 0), confirming that the observed nonzero alignment with shared anchors reflects genuine representational correspondence rather than sampling artifacts.
As three anchors already yield a relatively reliable similarity estimate, this suggests that the relative coor- dinates directly be used for (a randomized) low-dimensional visualization of embedded latent geometry,
Alignment varies along trajectorics and across model familics. So far we have computed represen- lational alignment over all latent points. To assess how alignment with the true system locally evolves along a given trajectory, we computed representational similarity along an embedded trajectory of length 300. As outlined above, we use three anchor points to create a dircct visualization of the relative latent spaces. Fig- ure 3 illustrates the temporal evolution of alignment along given trajectory. Consistent with our broader findings, forecasters within the same family tend to form similar temporal representations, whereas trans-
13

---

Published in Transactions on Machine Learning Research (02/2026)
formers and ESNs display distinct alignment catterns-particinn in the Lorenz-63, double pendulum, and random skew systems.
Latent stitching reveals family-specific representational compatibility
Table $ summarizes absolute and relative stitching losses on the Lorenz-63 dataset. Within model families MLPs with MLPs and transformers with consffrmers-rrlattee stitching outperforms absolute stitching. Transformer decoders act as strong universal decoders, achieving low losses even with absolute represen- tations. However, relative stitching offers no benefit across families, as seen when mapping transformer representations to MLP decoders. RNNs were excluded because their reliance on hidden states makes both in-family and cross- family stitching incompatible under our current setup, leaving hidden-state stitching for future work. Overall, these results show that representational compatibility and hence the ability to "stitch" encoders and decoders largely confined to model families that share similar latent geometries.
enc/doc
MLP
N-MLP
K-MLP
TF
N-TF
K-TF
Abs.
Rel.
Abs.
Rel.
Abs.
Rel.
Abs.
Rel.
Abs.
Rel.
Abs.
Rel.
MLP
1.655
0.383
2.334
0.479
2.459
2.818
0.293
0.825
1.181
1.067
0.923
0.988
N-MLP
2.195
0.404
3.916
0.491
3.511
3.078
0.233
0.813
0.545
1.040
1.235
0.925
K-MLP
1.621
0.753
2.224
0.759
2.523
0.891
0.290
0.587
1.233
0.974
0.835
0.679
TF
1.538
2.019
2.389
1.754
2.118
9.517
0.265
0.043
1.383
0.587
0.599
0.076
N-TF
1.514
1.780
2.003
1.580
2.039
7.112
0.184
0.061
1.017
0.757
0.689
0.256
K-TF
1.590
2.011
2.095
1.750
2.129
9.466
0.254
0.042
1.325
0.586
0.840
0.075
Table 4: Cross-architecture average stitching loss (MSE) over encoder decoder pairs for absolute (Abs.) and relative (Rel.) stitching. Each decoder column is independently normalized; darkest cell shows highest MSE and lightest shows lowest MSE respectively. Lower value per pair in bold.
Similarity metrics and robustness. Our primary measure of representational alignment is cosine simi- larity computed anchor-based relative embeddings, which provides a geometry-agnoste comparison across architectures and seeds. To assess robustness to the choice of similarity metric, we additionally evaluate sev- eral standard representational similarity measures on the same models and dynamical systems. Specifically, we compare against representational similarity analysis (RSA), Procrustes-based alignment, and centered kernel aligmment (CKA), each applied to the absolute latent representations following standard practice. Across all metrics, we observe consistent qualitative trends: in particular, strong within-family alignment for RNN- and MLP-based forecasters, systematically weaker alignment for transformers and ESNs, and clear dissociation between forecasting accuracy and representational aligmment for attention-basee and reservoir models. While absolute similarity values vary across metrics, the family-level structure and relative ordering of architectures remain stable (Appendix Figure 15
6 Discussion and Conclusions
This study examined neural forecasters for dynamical systems through the lens of relative embeddings (Moschelaa al. 2023 : Across periodic, quasi-periodic, and chaotic regimes, we observed reproducible, family-specific aligument patterns, alongside cases where strong forecasting performance cocxisted with com- paratively low cross-forecaster and system alignment most notably in transformers and ESNs. These find- ings suggest that task loss alone does not fully capture how forecasters internalize latent geometry, echoing prior observations that similar task performance in neural networks can arise from distinct representational organizations (Kriegeskorte et al. 2008 Kornblith et al.] 2019). Together, these results position representa- tional alignment as a complementary dimension for understanding and evaluating neural forecasters.
To interpret these findings, we first clarify what relative alignment measures reveal about learned represen- tations. Relative embeddings do not require estimating an explicit alignment or mapping between latent spaces. In contrast to Procrustes-based approaches, they do not assume linear isometric correspondence
14

---

Published in Transactions on Machine Learning Research (02/2026)
between representations (oower 1975
Schönemann 1966) Unlike CKA, which compares representations
through similarity matrices computed on matched inputs, relative embeddings define a shared relational coordinate system via similarities to a fixed set of anchors (ornbllih et al. 2019 Moschella et al. 2023
Aligument often increased with forccasting quality, but not universally. Model familics with different induc- live biases appear to summarize temporal context in distinct ways, achieving accurate predictions despite divergent relational gcometries. In practice, representational alignment analysis thus complements forecast- ing loss functions when stability, interpretability, or transferability are priorities. Although not serving as evidence of learned physical dynamics 1, it exposes architectural inductive biases and task-induced geometry.
To further interpret the observed family-level aligmment patterns, it is instructive to relate them to archi- tectural inductive biases within the shared encoder propagator decoder framework. RNN-based forecasters maintain a recursively updated hidden state, which induces temporally coherent latent-state evolution and re- sults consistently high representational alignment across sceds and architectural variants. MLP forecasters compress each input window into single global latent representation via a fixed feedforward mapping, yield- ing a different, but. relatively stable within-family geomelry. In contrast, Transformer encoders construct foken-wise contextual representations in parallel through self-attention, without architectural pressure to form smooth or trajectory-like latent reprosentations. As a result, the representational regime for Trans- formers seems to focus on contextual summarization rather than latent-state evolution, which helps explain why strong forecasting accuracy can coexist with comparatively weaker geometric alignment. Finally, ESSNs rely on fixed random reservoirs with only a trained readout layer, so reservoir trajectories primarily reflect internal reservoir dynamics rather than task-induced structure, accounting for their systematically lower alignment with the ground-truth relative representation.
A practical advantage of relative embeddings is more stable and interpretable visualization of learned la- tents. Standard projections of absolute embeddings (e.g., PCA) are sensitive to arbitrary rotations and scalings across seeds. Anchor-based relative spaces define a shared reference frame, making low-dimensional projections and neighborhood relations comparable across modols. This facilitates diagnostics analogous to representattioaal dissimilarity matrices in RSA (Kriegeskorle el al. 2008) and population "hyperalignmenu" in neuroimaging (Haxby et al.] 2011). Combined with PCA, t-distributed Stochastic Neighbor Embedding
(t-SNE) (van der Maaten & Hinton 2008 ; or Uniform Manifold Approximation and Projection (UMAP)
McInnes et al.) 2018) such spaces enable tracking of training trajectories, identifying attractor-ppeeffic regimes, and monitoring representational drift. In short, relative embeddings turn visualization from an exploratory tool into a quantitative diagnostic of representational geometry.
Aligument may serve several practical roles. It can serve as an auxiliary selection criterion during model devel- goment-favoring configurations that jointly achieve low forecast error and high representational agreement, particularly when downstream stitching or transfer is anticipated. Alignment trajectories during training may provide early warnings for overfitting instability, for example when the relative space fragments. Fi- nally, stitching encoders and decoders is more feasible when embeddings are computed relative to common anchor set (Moschella et al., 2023). Together, these roles highlight alignment as a lightweight yet informative signal for model selection, monitoring, and interoperability.
Our analysis relies on a finite anchor set and a chosen similarity function. Too few anchors reduce dis- criminability; too many incrcase computational cost. While we empirically found stable behavior beyond a moderate anchor budget, adaptive anchor selection (e-g., parthest-poinl sampling or clustering) could improve robustness in higher dimensions. Relative embeddings are also less sensitive to certain non-isometric deforma- tions (e.g., local shear). Complementary approaches based geodesic/transport-ee ttta/stoortttaaarr comparisons e.g., OT-b (optimal transport, OT) anchor bootstrapping of Cannistraci et al. (2023) and latent-space trans- lation (Maooraa et al. 2023) may capture finer structure. Finally, we focused on simulated benchmarks with controlled noise; assessments on high- high-dimensional and real- world systems will be necessary to test scalability and domain nosustness. These limitations delineate a clear path toward more adaptive and geometry-aware alignment frameworks.
Several directions appear promising. (i) Adaptive anchor selection and bootstrapped ensembling of relative spaces could further stabilize estimates under limited data. In the context of dynamica systems, anchors
Prreliinaay experiments with a simple linear readout probe are shown
15

---

Published in Transactions on Machine Learning Research (02/2026)
could be selected more informatively by targeting representative regions of the attractor or dynamically salient states. (ii) Aligmment-aware training- for instance through auxiliary losses or early-stopping crite- ria might promote generalizable latent representations. (iii) Richer comparators, including OTT-based or pnoctral/fucctional-mag techniques, could link alignment more tightly to long- hog-horizon accuracy (Garcia- Castellanos et al. 2024 Fumeeo et al. 2025 (iv) Disenlangling architeclural and algorithmic effects, for aware objectives or alternative optimization schemes, may clarify which aspects of aligmment are architecture-driven.. (v) Extended evaluations, including long-term statistics, spectral properties, topological features, could reveal whether alignment better pre- diets faithful dynamical behavior beyond short-horizon MSE. (vi) Applications to scientific forecasting and control may benefit from aligament-guided ensembling and forecaster monitoring. More broadly, integrating represenlational aligument into training and evaluation may help unify geometrie, stalistical, and dynamical perspectives on learning in neural systems.
example by comparing standard training with long-horizon aware
Our findings show that neural forccasters develop reproducible, family-specific representational geometrics that can diverge despite similar forecasting accuracy. This dissociation underscores the need for evaluation metrics that go beyond task performance and capture the geometry of learned latent geometry. By aligning latent spaces through anchor-based relative emboddings, we provide simple and reproducible approach to study how different model families internalize structure in time-evolving systems. Relative geometry offers a compact, interpretable, and reproducible lens on learned representations- one that may help bridge analyses of artificial and biological neural systems.
7 Broader Impact Statement
This work methodological and focuses on analyzing learned representations in neural forecasters rather than doveloping deployable prodiction systems. In addition canonical dynamical benchmarks, we include preliminary analysis on open, de-identified intracranial EEG (iEEG) recordings, using the dataset solely testbed for representation analysis in sigh-dimensional real-world setting. The study does not aim to perform clinical inference, diagnosis, intervention, and all human-related data are observational and ethically released.
A key limitation is that representational alignment should not be interpreted as evidence of model correct- ness, causal validity, or recovery of true neural dynamics. Overall, the work presents low societal risk and contributes tools for understanding and comparing internal representations neural and scientific time-series models beyond task performance alone.
8 Funding
Computational resources were provided by the Max Planck Computing and Data Facility (MPCDF). D.K. supported by the DAAD project SECAI (project no. 57616814), funded by the German Federal Ministry of Research, Technology and Space (BMFTR). N.S. was supported by BMFTR through ACONITE (grant no. 16IS22065) and the Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Leipzig, as well as by the European Union and the Free State of Saxony through BIOWIN.
References
Vladimir I Arnold. Mathematicul melhods of classical mechanics, volume 60. Springer-Verlag, 1989.
Arthur Bizzi, Lucas Nissenbaum, and João M Pereira. Neural conjugate flows: A physics-informed archi- tecture with flow structure. Proceedings of the AAAI Conference Artificial Intelligence, 2025. To appear.
Steven Brunton, Joshua Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113 (15):3932 -337, 2016.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Irene Cannistraci, Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, and Emanuele Rodolà. Bootstrapping parallel anchors for relative representations. orXiv preprint or orXin:2003.00721, 2023.
Trene Cannistraci, Luca Moschella, Marco Fumero, Valentino Maiorca, and Emanuele Rodolà. From bricks to bridges: Product of invariances to enhance latent space communication. In The Twelfth International Conference Leurning Represenlations, 2024. URL http:://ppaeeeiiss...e//oorrrrrp www.:e/orrrodddwwgyoolllloo
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential
equations. In Advances in Neural Information Processing Systems, volume 31, 2018.
Xinlei Chen and Deng Cai. Large scale spect ral clustering with landmark-based representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp.
2011.
Donato Crisostomi, Trene Cannistraci, Luca Moschella, Pietro Barbiero, Marco Ciccone, Pietro Liò, and Emanuele Rodolå. From charts to atlas: Merging latent spaces into onc. arXiv preprint 2rX.0:2311.06547, 2023.
Vin De Silva and Joshua B Tenenbaum. Sparse multidimensional scaling using landmark points. Technical report, technical report, Stanford University, 2004.
Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding Representation Similarity Through Statistical Testing. In Advances in Neural Information Processing Systems, volume 34, pp. 1556-1568. Curran Associates, Inc., 2021.
Christos Faloutsos and King-Tp Lin. Fastmap: A fast algorithm for indexing, data-mining and visualization of traditional and multimedia datasets. In Proceedings of the 1995 ACM SIGMOD international conference on Monagement of data, pp. 163-174, 1995.
Marco Fumero, Marco Pegoraro, Valentino Maiorca, Francesco Locatello, and Emanuele Rodolå. Latent funational maps: spectral framework for representation alignment, 2025. URL https://arxiv... org/
abs/2406.r 14183
Alejandro Castel-,s Giovanni Luca Marchetti, Danica. Kragic, and Martina Scolamiero. Relative representations: Topological and geometric perspectives. In UniReps: 2nd Edition of the Workshop on Uni- fying Represenlations in Newral Models, 2024. URL
Saptarshi Ghosh. Swec-ethz iceg seizure detection datasct for bangalore neuromorphic bnew workshop 2025, December 2024. URL https://doo..org/00.. http://1..281/eeodoo. 14578383
John C Gower. Generalized procrustes analysis. Psychometriku, 40(1):33 51, 1975.
James V Haxby, Jyoti S Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Brian R Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2):404-414 2011. doi: 1..006/..eeuron22011.00 2001.08.026......
Sepp Hochreiter and Jürgen Schmidluber. Long short-term memory. Neural computation, 98)) 1780, 1997.
Holger Kantz and Thomas Schreiber. Nonlinear time series analysis, volume Cambridge university press, 2004.
Anatole Katok and Boris Hasselblatt. Introduction to the modern theory of dynamical systems, volume 54. Cambridge university press, 1995.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519-3529. PMLR, 2019.
Nikolaus Kriegeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity analysis-conneeting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008
17

---

Published in Transactions on Machine Learning Research (02/2026)
Zorah Lähner and Michael Moeller. On the direct alignment of latent spaces. In Proceedings of UniReps: the Pirst. Workshop on Unifying Represenlations in Neural Models, pp. 158-169 169. PMLR, 2024.
Jeffrey Lai, Anthony Bao, and William Gilpin. Panda: A pretrained forecast model for universal represen- tation of chaotic dynamics, 2025. URL attp:::/arxx...orgaab////// 13755
Wei Liu, Junfeng He, and Shih-Fu Chang- Large graph construction for scalable semi-supervised learning. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 679 686, 2010.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of the atmospheric sciences, (0(2):100-14:) 1963.
Edward N Lorenz. Predictability: problem partly solved. In Proceedings of the Seminar on Predictability, volume 1, pp. 1-18. ECMWF, 1996.
Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9(1): 2018.
Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. In Thirty-seventh Conference on Neural Infor- mation Processing Systems, 2023.
Filip Matzner and Františck Mráz. Locally connected echo state networks for time series forecasting. In The Thirteenth International Conference on Learning Representations, 2025. URL. https://openrevier/
net/forummiddKKRRLLLaZza
Robert M May. Simple mathematical models with very complicated dynamics. Nature, 261 2616560)::559-417 1976.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03/ 03426, 2018. URL https://arxiv, org/abs//0/aa http:://aaxiii.oorg/ebe///22 03426
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In The Eleventh Interna- tional Conference on Learning Representalions, 2023.
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodola, and Francesco Lo- catello. Asif: Coupled data turns unimodal models to multimodal without training. Advances in Neural Information Processing Systems, 36:15303-15314 2023.
Dino Oglic and Thomas Gärtner. Nyström method with kernel k-means++ samples as landmarks. In International Conference on Machine Learning, pp. 2652-2660. PMLR, 2017.
Jaideep Pathak, Zhixin Lu, Brian R. Hunt, Michelle Girvan, and Edward Ott. Using machine learning to replicate chaotic altractors and calculate Iyapunov exponents from data. Chaos: An Interdiscéplinary Journal of Nonlinear Science, 27()),, December 2017. ISSN 1089-7682. doi: 10.1063//.0010300, URL http:://xx.do..org///// 1tt..oorrro...0511555000000
attp://dx.c doi.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:656-700 707, 2019.
Antonio Pio Ricciardi, Valentino Maiorca, Luca Moschella, Riccardo Marin, and Emanuele Rodolå. R31: Relative representations for reinforcement learning. arXiv preprint arXiu:2404.12013 2024.
David Ruelle. What are the measures that describe turbulence? Progress of Theoretical Physics Supplement, 14:339-345, 1978.
Peter H Schönemann. generalized solution of the orthogonal procrustes problem. Psychometriko, 31(1): 1-10, 1966.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Steven H Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018.
Ila Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley Love, Erin Grant, Tris Groen, Jascha Achterberg, et al. Getting aligned representational alignment. arXiv preprint ar Xiv:2310. 130188 2023.
Floris Takens. Detecting strange attractors in turbulence. Dynamical systems and turbulence, Warwick 1980, pp. 366-381. Springer, 1981.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579 2605, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Autention is all you need. In Advances in Newral Information Processing Systems, volume 30, 2017.
Pantelis R Vlachas, Jaideep Pathak, Brian R Hunt, Themistoklis Sapsis, Michelle Girvan, Edward Ott, and Petros Koumoutsakos. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiolemporal dynamics. Neural Networks, 126:191 217, 2020.
19

---

Published in Transactions on Machine Learning Research (02/2026)
A
Remaining Experimental Results
Validatioo MSE
b Hopf
Similarity
Vallcation MSE
Similarity
Figure Forecasting and representational alignment. (a, b) Example systems: proper orthogonal decomposition (POD)-wake (a), Hopf (b). Columns show time series trajectorics, absolute emboddings (visualized with principal component analysis (PCA); we plot the first 2 components for 2-dimensional systems and 3 components for the rest of the systems), relative embeddings (PCA), cross-forecase similarity heatmaps (averaged over five sceds) with the order of True System, MLP, Koopman MLP, NODE MLP, RNN, Autoregressive RNN, Koopman RNN, NODE RNN, Transformer, NODE Transformer, Koopman Transformer, and ESN; and alignment perforrmnc of forecasters with different hyperparameter settings.
Table 5: Results for Double_penduue
(4)
Sianilarity (1)
Model
MSE
MAE
RMSE
Cosine
Top-1
Spoazman
MLP
0.4122 0.0098
0.4573
0.0049
0.6420
0.0075
0.6745 : 0.0106
0.4262 + 0.0065
0.8754 1 0.0115
Koopman MLP
0.4314 0.0267
0.4871
0.0202
0J6565 + 0.0202
0:6097 + 0.0241
0.4112 + 0.0076
0.6033 + 0.0287
NODE MLP
0.3573 0.0102
0.4364
0.0033
0.5977 + 0.0085
0.6212
0.0076
0.4186 + 0..025
0.6211 + 0.0100
RNN
0.8197
Autoragressivn
0.3984 0.0494
0.4450
0.6407
0.6310
0.0397
0.8791
0.0241
0.5526
0.0434
0.0258
Koopman RNN
0.4851 0.0943
0.5148
D.0381
0.6966 + 0.0500
0.8608
0.1458
0.6928 + 0.0279
0.5622 + 0.1272
NODE RNN
0.3799 + 0.0332
0.4612
D.0233
0.6159 + 0.0270
0.8631 = D.D348
0.5274 +: 0.0210
0.8628 :: 0,0382
Transformr
0.0072
0.0685
0.0846 0.0057
0.4360
0.0150
0.6485
Keopman Transformer
0.9023
NODE Transformer
0.0178 0.0040
0.1072 0.0098
0.1329 1: 0.0144
0. 3836
0.0485
0.3886
0.0173
0.4068
0.0378
F8N
1.3081 + 0.0863
0.7915 0.0342
1.1432 + 0.0379
0. 1601
+ 0.0014
0.3808
+ 0.0010
0.1501 + 0.0015
Table 6: Results for random. skew
Performance (1)
Similarity (1)
Model
RMSE
Coeine
Top-1
Spearman P
MLP
0.9778 +: 0.0532
0.6652 # 0.0174
DJ888G 4: 0.0271
0.7533 d: D.U222
0.8138 +: 0.0034
0.7724 #: 0.0159
Koopman MLP
0.5137 0.1571
0.3918
0.0409
0.0837
0.8523
0.0229
0.8304
4: 0.0038
0.8440
0.0279
0.0960
0.7536
RNN
0.6815 0.0283
0.6481
0.0081
0.9388
0.0151
0.8155
0.0284
0,7840
0.0091
0.8058 : 0.0209
Antoragressive RNN
0.7683 0.0171
0.5823
0.0089
0.8765
+ 0.0097
0.8090
0.D153
0.8178
+ 0.0134
0.8059 + 0.0153
Koopean RNN
0.7356 0.0369
0.5891
0.0196
0.857G
0.0213
0.8329
D.D40G
0.8394
0..2280
0.8338 + 0.0362
NODE RNN
0.7211 0.0172
0.0055
0.0202
0.8638
Transformer
0.4878
0.4458
0.0083
0.6817
0.7417
0.0855
0.5000
0.7426
0.0321
Koopman Transforer
0.4630 + 0.0150
0.4600
0.9073
0.6848 + 0.0108
0.7288
0.0211
0.8174 + 0.0093
0.7842 + 0.0196
NODE Teansformer
0.4429 + 0.0105
0.4425
0.0121
0.0055 + 0.0079
0.3904 - 0.0091
0.0410 +: 0.0433
0.3957 4: 0.D482
ESN
0.6436 0.0436
0.3803
0.0206
0.0268
0.9074
0.0211
0,8340
0,9008
0.0185
20

---

Published in Transactions on Machine Learning Research (02/2026)
1.0
1.00
0.93
0.92
0.99
0.71
0.69
0.71
0.68
1.00
1.00
0.99
0.93
0.95
1:00
0.99
0.98
0.82
0.83
0.85
0.82
6.96
0.96
0.96
0.98
0.93
0:93
0.94
0.8
Average similarity
0.6
0.71
0.85
0.00
0.77
0.85
0.06
1.00
0.82
0.73
0.73
0.74
0.04
0.4
0.68
0.82
0.85
0.74
0.76
0.79
0.82
1.00
0.71
0.71
0.72
0.82
1.00
0.95
0.93
0.99
0.73
0.72
0.73
0.71
1.00
1.00
1:00
0:94
1.00
0.95
0.93
0.99
0.73
0.72
0.73
0.71
1.00
1.00
1.00
0.94
0..
0.99
0.95
0.94
0.74
0.74
0.74
0.72
1.00
1.00
1.00
0.94
0.94
0:54
0:94
1.00
True System
Koopman RNN
MLP
NODE
Kaopman
Transformer
NODE NLP
Koopman Transformer
Iotoonnggeiive RNN
ESN
Figure 5:
Cross-model similarity Logistic Maps.
POD
Sindlartyy
sandiiity
Hopf
Palcoaaa 1922
Logistic Map
Figure 6:
Perturbation experiments POD-wake, Hopf and Logistic Maps. (X) noise experiment could not
be performed because the data were obtained Brunton 2016), only principal components not original data were available.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Simtee COO
POD
Time
Hopt
Figure 7: Temporal alignment visualization for POD-wake and Hopf.
-0.8
Anchor
System
-0.9
..6
Anchor 1
-0.8
-0.4
-0.3
-02
-0.7
0.1
-0.
-05
-0.4
Figure 8: Temporal alignment rotated visualization of the True System Lorenz.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Table 7: Results for spiral
Performance (1)
Simflurity (1)
Model
MSE
MAE
RMSE
Cosine
Top-1
Spearman
MLP
0.0002 + 0.0000
0.0059 = 0.5004
0.0128 + 0.0010
0.8548 # 0.0322
0.4020 = 0.0029
0.8788 # 0.0104
Koopman MLP
0.0002 1 0.0301
0.0049 # 0.0036
0,0136 4: 0.0035
0.8700 4: 0.0220
0.4200 1: 0.0033
0.5573 4: 0.02232
NODE MLP
0.0003 0.0001
0.0067
0.0012
0.0178 0,0028
0.8796
0.0127
0.4004 1: 0.0067
0.5698
1 0.0121
RNN
0,0002 0.0001
0.0017
0.0153
0.0027
0.0039
0.6222 1: 0.0442
0.0906
0.0038
Autorogression RNN
0.0017 0.0004
0.0298
0.0044
0.0407 0.0044
0.9036
0.0040
0.5424 + 0.0452
0.9921 + 0.0033
Koopman RNN
0.0061 + 0.030a
0.0633
0.0044
0.0782 + 0.0065
0.9952
# 0.0033
0.5570 + 0.0229
0.9938 + 0.0026
NODE RNN
0.0043 0.0306
0.0527
0.0057
0,0658 + 0.0045
0.9913 : 0.0036
0.5330 :: 0.0208
0.0680 1 0.0040
Transformer
0.0013 0.0003
0.0254
0.0033
0.0364 0.0035
0.9622 0.0040
0.4324
0.9774
Koopman Transformer
0.0025 0.0300
0.0381
0.9066
0.0493
0.0095
0.9792
0.0125
0.4344 + 0.0074
0.9743
0.0114
NODE Transformer
0.0044 + 0,03009
0.0495 # 0.0042
0.0664
+ 0.00G8
0.98224
# 0.0054
0.421G
= 0.0105
0.9760
# 0.0055
ESN
0.0003 1 0,0301
0.0041
- 0.0010
0.0151
1 0.0053
0.9702
E 0.0027
0,3874 1 0.0058
0.9637
1 0.0026
Table 8: Results for pod
Performancn (1)
Similarity (1)
Model
MSE
MAE
RMSE
Cosine
Top 1
Spearman
MLP
0,0270 + 0.0046
0.0862 # 0.0071
0.1637 + 0.0141
0.9611 + 0.0307
0.8856 + 0.0079
0.9024 + 0.0176
Koopman MLP
D.0331 + 0.0111
0.0931 # 0.0125
0,1800 +: 0.0303
0.9277 :: 0.0179
0.8810 +: 00.0129
0.8079 # 0.0338
NODE MLP
0.0317 0.00755
0.0942
0.0115
0.1770
0,0227
0:9367
0.0063
0.8758
000118
0.8783 +: 0.0052
RNN
0.0242 0.0063
0.0110
0.0586
0.8876
0.0558
0.9170
Autorugressive RNN
0.0422 0.0097
0.1305
0.0096
0,2043
+ 0.0239
0.9781 : 0.0086
0,9328 : 0.0188
0.9450 + 0.0197
Koopman RNN
0.0626 + 0.0201
0.2120
0.0297
0.2857
+ 0.0347
0.9612 + 0.0622
0.9280 + 0.0457
0.9482 + 0.0570
NODE RNN
0.0022 0.0011
0.1697
0.0019
0.2285
0.0025
0.9953 = 0.0022
0.9662 + 0.0077
0.9741 = 0.0137
Transformer
0.0227 0.00143
0.0852 0.9074
0,0142
0.D4US
0.8506 0.0176
0.8521
Koopman Transformer
0.0387 0.0058
0.1310
0.9097
0.1910
0.0155
0.9149
0.0101
0.8700 +:
0.8767 + 0.0135
NODE Transformer
0.0680 0.0268
0.1645 1 0.0173
0,2571 1. 0.0491
0.9019 4: 0.0490
0,8326
1 0.0479
0.8574 1 0 0.0668
ESN
0.0127 + 0.0041
0.0483 = 0.0076
0.1113
+ 0.0198
0.9262
: 0.0210
0.5636
1 0.0145
0.8774 1 0.0384
Table 9: Results for hopf
Performance (4)
Similarity (t)
Model
MSE
MAE
RMSE
Cosine
Top-1
Spearminn
MLP
0.0002 1. 0.0001
0,0080 1 0.9036
0.0146 1 0.0028
0.9672 4: 0.0058
0.5190 1: 0,0078
0.9596 1: 0.0119
Koopman MLP
0.0002 1 0.0301
0.0061 1 0.0036
0.0135 1 0.0039
0.9463 :: 0.0119
0.5440
1 0.0175
0.9300 1 0.0227
NODE MLP
0.0003 0.0301
0.0096
0.0007
0.0169 + 0.0027
0.9550 : 0.0082
0.5160 + 0.0162
0.9463 4 0.0107
RNN
0.0001 0.0000
0.0022
0.0070
0.9507 0.0040
0.5474
0.0297
0.9005 0.0044
Autoregressive RNN
0.0015 0.0001
0.0332 1 0.0009
0.0386 1 0.0012
0. 0.9888 1 0.0088
0.5638 1 0.0358
0.9904 1 0.0052
Koopman RNN
0.0130 + 0.0016
0.1010 0.0060
0.1137 + 0.0069
0. 0,9038- : 0.0020
0.5876 1 0. 0.0284
0.9937 : 0.0020
NODE RNN
0.0108 0.0306
0.0919
0.0026
0.1038
0.0029
0.9063 0.0014
0.6192 + 0.0310
0.9958 + 0.0014
Transformer
0.0061 0.0012
0.0673 0.0064
0.0776 0.0076
0.0093
0.0218
0.9871 0.0074
Koopman Transformer
0.0056 0.0011
0.0645
0.0044
0.0748
0.0069
0.9895
0.0064
0.5806
0.0200
0.9891 0.0061
NODE Trunsformer
0.0130 1 0.0075
0.0893
1 0.0290
0.1085
1 0.0355
0.9001
1 0.0061
0.5486
1 0.0239
0.9883
1 0.0062
ESN
0.0000 1 0.0000
0.0004 - 0,0000
0.0011
+: 0.0002
0.9978
1 0.0003
0.5216
1 0.0038
0.9949
1 0.0005
Table 10: Results for logistic_maat map
Performance (4)
Similurity (1)
Model
MSE
MAE
RMSE
Cosine
Top-1
Spearman
MLP
0.0002 4: 0.0000
0.0118 =: D.0036
0.0157 +: 0.0008
0.3489 :: 0.0074
0.02322 +: 0.0040
U.7468 #: 0.0035
Koopman MLP
0.3713 1. 0.4895
0.4102 4 0.4452
0.4505
1 0.4588
0.9806
4: 0.0047
0.0232
1 0.0051
0.7468
+: 0.0044
NODE MLP
0.6986 0.0191
0.8939
0.0109
0.9479
1 0.0101
0.9165
0.0085
0.0228 1 0.0046
0.7446
4 0.0044
RNN
0.0025 0.0038
0.0325
0.0245
0.0302
0.7056
0.1526
0.0302
4: 0.0059
0.6182
Autoragression RNN
0.0065 0.0041
0.0648
0.0197
0.0771
0.0255
0.6875 0.2004
0.0240
0.0074
0.5479
0.2504
Koopman RNN
0.D453 + 0.0427
0.1801 = D.0678
0.2074
+ 0.0517
0.6835
+ 0.2248
0.0264 + 0.00G4
0.5198 + 0.2583
NODE RNN
D.02289 +: 0.00/29
0.142) = 0.0082
0.1545 +: 0.0051
0. 0.7063 =: D.0921
0.0198 + 0.0037
0.6261 +: 0.1181
Transformr
0.0088 0.0015
0.0867 0.0085
0.0820
0.0092
0.9003 0.0023
0.0230
0.0074
0.7413 1 0.0013
Koapman Transformer
0.0142 0.0043
0.0098
0.0168
0.0023
1 0.0034
1
NODE Transformer
0.2153 0.4374
0.2631 0.3738
0.3117 1 0.3844
0.9964 = 0. 0.0008
0.0208 1 0. 0.0077
0.7413
4 0.0019
F8N
0.0000 + 0.0000
0.0007 + 0.0011
0.0013 + 0.0024
0.9307 + 0.0093
0.0162
+ 0.0004
0.7401
+ 0.0008
23

---

Published in Transactions on Machine Learning Research (02/2026)
0.8
0.6
044
Smiuny
-02
16
64
128
512
800
999
K (f of anchans)
Figure 9: Anchor ablation and baseline. (Blue) Alignment number anchors K; lines show mean repeats. (vertical marker) for the main experiments. (Orange) Random baseline
Stabilization occurs for K we disjoint anchor sets across spaces, yickdnn near-zero alignment.
Figure 10: Hyperparameter Tuning for All Systems and Models. ESN is exluded since it needed additional manual hyperparameter tuning due to its sensitivity to hyperparameters and unstable nature.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Lorenz-63
Double-penduliee
Random-skew
POD
Limit-cycle
Hopf
Logistic Map
Figure 11: Cross-model similarity using absolute embeddings.
25

---

Published in Transactions on Machine Learning Research (02/2026)
B Similarity Metrics
T1 similarity. The T1 score measures the agreement in the identity of the most similar anchor across encoders. For each latent E V, we check whether the anchor with highest similarity under encoder 1 coincides with that under encoder 2:
1 1 0T1
1 V
Σ REV
arg max r0)) (z) arg max r rel. (2)
Rank similarity. The rank similarity evaluates how similarly two encoders order the set of anchors for each latent For each encoder E {1, [}, let (ankj(r,a)) (z)) denote the vector of descending ranks (with 1 assigned to the largest component) of the relative similarity vector ((e) (z), with ties resolved using a stable sort order. The average Spearman correlation between these rank vectors defines the rank similarity:
Oraakk (0 ():
V)
Σ BEV
E ranky (rlal (z)), rank, (r (r (z)))
MV
where denotes Spearman's rank correlation coefficient, implemented as the Pearson correlation between the transfrresf relative similarity vectors.
C Stitching Details
0.51
0.39
87:
Banalmer
0.54
0.3
0.0
0.33
072
183
rodetanloomerr
0.22
Average cosine similarity over all each forecaster pair, excluding exact same forecaster and seed. Models were trained on relative latent spaces rather than absolute spaces.
Figure 12:
26

---

Published in Transactions on Machine Learning Research (02/2026)
D Dynamical Systems Details
Lorenz-61 63 (3-D chaotic ODE). - oly- 1 = ==<(p=))--) & == ==zy zy--y-- 32, with 10. p= = 28, 3 8/3. Initial states are sampled from [-20, 20 and integrated with Runge Kutta 45 (RK45) at Av 0.01. Its compact phase space and positive Lyapunov exponent (se 0.91) make it a classical culti-step-foreccan benchmark.
Stable limit cycle (2-D rpdial-spiral ODE). == p(R - r), á = w, (ar, y) = cr 0, rsin 0), with # = I, == w = 1. Trajectories start from To ~0 20) and 0g ~ ~210 27|; integration uses RK45 with AL = 0.01.
Double pendulum (4-D Hamiltonian chaos). Two unit-mast unit-length links evolve under gravity 9-9.81. Writing the state (01, 02, w1, w2) with A = 82 01, the equations of motion
01 W1,
02 = W2,
W1 wP sin A cos A g gsin 02 cos A + w/ sin A 2g sin 0g
cos
W2 = -w3 sin A cos A + 2g sin 01 cos A 2w sin A 2g sin 02
2 cos A
Initial angles are sampled from [-200, 20% and angular velocities from [-1, Trajectories are integrated with RK45 at At=0.01. The system exhibits strongly chaotic, nearly energy conserving motion, with a positive Lyapunov exponent of = 1.5.
Hopf normal form (2-D near-critical oscillation). == pa. - wy - (a² + +y²),, D = war + py (x² + y² yr),. with μ 0,
Starting points (zo, (0)=2([22,2)] spiral onto unit-radius limit cycle; At 0.01 with
RK45.
Logistic map (1-D near-onset discrete chaos). T++1 = 3.57 xx) with xo ~M 1): sequences of length T-500 are recorded at an effective step At = 0.1.
3). We adopt the three leading Proper- coefficients from Brunton et al. (2016) (Re 100, Strouhal x 0.16). We supply 10 trajectories per split, each of 550 snapshots sampled at At 0.2; only z-score normalisation is applied.
Fluid wake behind a cylinder (POD-wake coefficients; d orthogonal-Deecmppsiiit
Skew-product of 3-D chaotic founders (6-D weakly coupled ODE). Following Lai et al. (2025 select two founders from (Lorenz-63, Rössler, Chen), jitter parameters by multiplicative log-normal noise (log ~ (O,0 0.15),, sign preserved), and couple them in a skew-product: the first 3-D system x R drives the second y E IR3 via a weak injection into the first response coordinate. Writing & = fa(x;Pa) and ÿ = fo():ps) for the founders with jittered parameters,
4 = (u;;
ý = fo(v +89 0.05, e1 = (1,0,0)
Founder templates and nominal seeds:
Lorenz -3: à = ( - x), ; x(p
: - (a,P,
xo == 00==1,10 (1,1,1),
Rössler: å y 2, ý x ay, == bb z(x c): (a, b..b.c) c) (0.2, 2g (0.1,0,0).
Chen: & aly z), ý (c a)æ
cy, 1 = zy bz; (a, c) = 335,3.28 28), zo (-10,0,37)
splits then differ only by initial conditions. Initial states jitter the concatenated founder seeds än = (=o: yo] yo] with i.i.d. Gaussian noise of scale 0.1. Trajectories are integrated with DOP853 at the dataset step At (absolute tolerance 10-8, 8 relative 10 9). We discard an initial warm-up fraction (default 10%) and keep the next T stops. Runs are rejected if any state is non-finite, the radius excceds 10 or the summed channel variance falls below 10 6, on rejection we resample once.
A single skew system is sampled once per dataset; train/val
27

---

Published in Transactions on Machine Learning Research (02/2026)
E
Probing state information in latent representations.
To assess the extent to which learned latent representations preserve information about the underlying system state, we perform a simple linear probing analysis on a representative dynamical system (Lorenz- 63). For each trained model, we freeze the encoder and fit a single linear ridge regressor to decode the current observable state x(() from the corresponding latent representation z(), training on the training split and evaluating held-out test data. When probing absolute latent representations, decoding performance is near-perfect across all model families, indicating that the encoder latents retain almost complete information about the instantancous system state. When applying the same probe to anchor-based relative embeddings, decoding performance remains high but exhibits a modest, architecture-deeendeei reduction. This behavior is expected, as relative embeddings are designed to quotient out certain geometric degrees of freedom in order enable cross-model comparison, rather than to preserve all lincarly decodable structure. We emphasize that this probing analysis is intended sanity check on representational content rather than evidence of system identification recovery of governing dynamics. We report these results in Figure 13 and interpreted bounding the information retained by the representations, rather than claim about learning the true dynamical system.
Near n' (Mbte derice, abrolue))
1.0
0.8
0.4
0.2
0.0
Kaeermn HLP
NODE MLP
Autoregnastive Hay
RaW
NODE RAN
Raansormtt
Konoman Ransforer
NODE Trersffrmer
Karamen
B
Moan of (sate decode relevve
0.8
0.6
IRP
Koopman MLP
RMN
Koseman RMN
RoDE AWW
Tranfomer
Kosprnnn Transfomer
MaaE Blanstorreer
NODE
Autongrmssiv
Figure 13: Mean lest-set R2 of linear ridge probe decoding the current system state x(() from absolute latent representations 2(t) for the Lorenz 63 system, averaged over three random seeds (error bars denote standard deviation).
28

---

Published in Transactions on Machine Learning Research (02/2020)
A
CKA
RSA
Procrustes
Figure 14: Cross model alignment using centered kernel alignment (CKA), representational similarity analysis (RSA), and phocrustes-basce alignment. Values indicate the average of three seeds.
F Preliminary Results on iEEG Data.
We inchude this experiment as a preliminary external validation of the representational alignment analysis a high-dimensional, real-world setting, using human intracranial EEG recordings from the first. participant (ID1) of the SWEC ETHZ dataset [Ghosh] (2024)). ). In contrast the simulated systems, where representa- tional similarity can be measured directly against known ground-truth dynamics, such comparison is not possible in the iEEG setting, as the underlying generative system is unknown.
The goal of this experiment is not to benchmark forecasting accuracy or to draw neuroscientific conclusions, but to assess whether the family-level aligument patterns observed in controlled dynamical systems persist when models are trained on real neural time series under otherwise comparable experimental conditions.
A
B
0:0
6.55
0.45
400
104
6-4
040
630
335
X71
6.54
332
432
635
140
Readona
6:00
3.43
0:01
613
140
Recer RCP
Auiing
Figure 15: A. Cross model alignment in forecasters trained on forecasting iEEG dataset. B. Performances of forecasters.
29


Reviews:
[{"review_id": "y8hHkea9Vp", "reviewer": "Reviewer_6aaX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates representational alignment in neural networks that forecast dynamics.\nThis is achieved using anchor-based relative embeddings.\nThe authors systematically compare latent space geometries across model families (MLPs, RNNs, Transformers, ESNs) on seven canonical dynamical systems spanning periodic, quasi-periodic, and chaotic regimes.\nThe key finding is that architectural families exhibit reproducible internal alignment patterns, and importantly, that high forecasting accuracy can coexist with low representational alignment, particularly for Transformers and ESNs.\nThe work adapts the relative embedding framework of Moschella et al. (2023) to the setting of dynamical systems forecasting and provides extensive empirical analysis.\n\n**Strengths**\n- Addresses an important and underexplored question: when do forecasting nets learn structurally similar representations of dynamics?\n- Systematic evaluation across diverse dynamical regimes and architectures with strong reproducibility\n- The finding that accuracy and alignment can diverge has practical implications for assessing the robustness of models.\n- Clear experimental framework with thorough analyses\n- Well-executed, reproducible empirical study\n\n**Weaknesses**\n- Limited exploration of when alignment matters for downstream tasks (generalization, robustness)\n- Anchor selection is not deeply investigated\n- The evaluation relies solely on short-horizon MSE, missing richer dynamical metrics (attractor geometry, Lyapunov exponents, long-term statistics) that could reveal deeper connections between alignment and faithful dynamical reconstruction.\n- No theoretical grounding\n\n**Overall:**\nA valuable empirical contribution that opens an important research direction.\nThe framework may not tell us whether models learn the underlying true dynamics, but it provides a principled tool for assessing whether models learn consistent dynamics, which is crucial for robustness, model comparison, and scientific understanding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The paper shows that model families exhibit reproducible internal alignment patterns (MLPs align with MLPs, RNNs with RNNs) while also demonstrating that high forecasting accuracy can coexist with low alignment to other models or to the ground-truth trajectory structure, particularly for Transformers and ESNs.\nThese findings are consistent across seven dynamical systems, multiple random seeds, and various metrics.\nThe experiments are thorough, and the conclusions are appropriately cautious.", "audience_yn": "Yes", "audience_explanation": "Yes. The paper may not introduce any new representation learning method, but the findings are genuinely interesting for anyone working on forecasting, whether in dynamical systems, weather, fluid flows, or similar fields.\nThe observation that models can forecast accurately while learning very different internal representations raises important questions about what these models actually learn.\nThe analysis is a solid starting point for understanding how different architectures organize their latent space and could pave the way for further work that connects representational structure to generalization, robustness, etc.\n\n**Important and Timely Question**:\nThe paper addresses a fundamental question that is often overlooked in the neural forecasting literature: when do two models learn structurally similar representations of dynamics?\nThis is distinct from and complementary to the standard question of forecasting accuracy.\nThe community has focused heavily on prediction error metrics, but understanding *how* models internally represent temporal structure is crucial for interpretability, robustness, and trustworthiness.\n\n**Practical Relevance Beyond Academia**: The finding that models can achieve similar forecasting accuracy while learning structurally different latent representations has significant implications for industrial applications. In a wide range of applications where neural forecasters are applied (from weather and climate prediction, energy grid management, and financial markets to fluid dynamics, structural health monitoring, and digital twins, just to name some), practitioners need to understand not just whether a model works, but also how it works and whether its learned representations are robust. The proposed framework provides a tool for:\n   - Detecting when model updates or retraining lead to structurally different solutions\n   - Evaluating robustness: models with similar alignment patterns may generalize more consistently\n   - Model selection beyond accuracy: preferring models whose representations align with physical intuition or other trusted models", "requested_changes": "The authors have performed a thorough revision addressing the main concerns raised by reviewers, and I find the paper adequate for publication.\nI offer below some optional discussion points for the authors that, in my opinion, would strengthen the work.", "broader_impact_concerns": "No significant concerns.", "additional_comments": "**Optional Discussion Points**\n\nThe following are optional changes/points that the authors are encouraged to add in their discussion. \nI raise them in the spirit of scientific dialogue, as I find the paper particularly interesting.\n\n**On Loss Functions and Dynamical Systems Evaluation Metrics**:\nThe paper focuses on MSE as the forecasting loss, which is standard and reasonable. However, the dynamical systems community has developed a rich set of evaluation metrics that go beyond short-horizon MSE and capture different aspects of learned dynamics:\n- **Attractor geometry**: Do the predicted trajectories lie on/near the true attractor? (e.g., correlation dimension, fractal dimension)\n- **Spectral properties**: Do power spectra or Lyapunov exponents match?\n- **Long-term statistics**: Climate/invariant measure metrics (do long rollouts reproduce the correct statistical distribution over state space?)\n- **Topological properties**: Are qualitative features (fixed points, limit cycles, basins of attraction) preserved?\n\nIt would be interesting to explore whether representational alignment correlates differently with these alternative evaluation metrics.\nFor example, a model might achieve a low MSE but poor long-term climate performance.\nDoes alignment predict this failure mode?\nConversely, models with similar alignment might share similar long-term statistical behavior even if short-term errors differ.\nThis could connect the representational analysis to the broader goals of faithful dynamical reconstruction, though I recognize this would be a substantial extension.\n\n**On Architecture vs. Training Algorithm Effects**:\nThe current study attributes the observed alignment patterns primarily to architectural inductive biases (MLPs vs. RNNs vs. Transformers vs. ESNs), and all models are trained with the same optimization procedure (Adam, MSE loss, early stopping).\nThis raises a natural question: to what extent are the alignment patterns a property of the architecture alone, or a joint effect of the architecture and the training algorithm?\n\nFor instance:\n- Would different optimizers (SGD, second-order methods) or learning rate schedules produce different alignment patterns for the same architecture? (probably not)\n- Training strategies designed for long-term forecasting (such as curriculum learning over prediction horizons, scheduled sampling, or backpropagation through longer rollouts) might encourage latent representations that better capture long-term dynamics. Would such training schemes lead to higher alignment with the true system? (no intuition here)\n- Regularization techniques (e.g., noise injection, spectral normalization) might implicitly shape the geometry of learned representations.\n\nIt is plausible that alignment is primarily architecture-driven, but it would be valuable to disentangle architectural from algorithmic contributions. Even a preliminary experiment comparing, say, standard training versus a long-horizon-aware training scheme on the same architecture could shed light on this. This might also connect to practical recommendations: if alignment can be improved through training modifications, it becomes an actionable target rather than a passive diagnostic.\n\n**On Anchor Selection and Optimal Experimental Design**:\nThe current approach selects anchors randomly from the dataset, and the ablation shows stability beyond a moderate number. However, for dynamical systems, one wonders whether there is a principled way to select anchors that would be more informative or efficient. For instance:\n- Should anchors be chosen to sparsely but optimally cover the latent attractor or state-space region, e.g., clustering on the attractor, or selecting points near dynamically significant regions (saddle points, attractor boundaries, transient vs. steady-state regions)?\n- Is there a notion of \"anchor efficiency\", i.e., achieving stable alignment estimates with fewer, more strategically chosen anchors?\nThis is likely beyond the scope of the current work, but could be a fruitful direction for follow-up studies, particularly for high-dimensional systems where anchor count affects computational cost.\n\n**On the relation with the Platonic representation hypothesis [1]**\nThe Platonic Representation Hypothesis argues that as models scale and become more capable, their representations converge toward a shared statistical model of reality.\nThe hypothesis was formulated based on large-scale foundation models in vision and NLP trained on internet-scale data for general-purpose tasks, a setting quite different from the task-specific neural forecasters studied here.\nNonetheless, the connection is interesting: while the Platonic hypothesis predicts convergence with scale, this paper shows that in forecasting, architectures (of much smaller scale) can achieve similar accuracy while their representations diverge, suggesting that task performance alone does not guarantee representational alignment.\n\n[1] Minyoung Huh et. al. \"The Platonic Representation Hypothesis\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "4T33iCt4WU", "reviewer": "Reviewer_t9fv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis paper studies representational alignment in neural forecasters for dynamical systems. The authors use a common encoder, propagator , decoder forecasting framework, and instantiate it with several model families such as multilayer perceptrons, recurrent networks, Transformers and echo state networks. They adopt an anchor-based relative embedding method to transform latent states and ground truth trajectories into geometry aware representations, and define similarity metrics between models on this basis. Experiments on seven canonical dynamical systems compare model families, evaluate how alignment relates to forecasting error under different noise levels and context lengths, and analyze how well encoders and decoders can be stitched across models. The paper reports consistent family level structure in the learned latent spaces, shows that high forecasting accuracy can appear together with low alignment especially for Transformers and echo state networks, and argues that relative geometry offers a stable tool to compare neural forecasters beyond standard loss based evaluation.\n\n**Main Strengths**\n\n1. The paper provides a systematic empirical study of representational alignment in neural forecasters across multiple model families and dynamical systems. The experimental design is thorough with a shared framework, comparable training setups, and a rich set of visualizations. This gives the community a consolidated view of how different architectures organize their latent spaces in this setting.\n\n2. The use of anchor-based relative embeddings offers a clear and unified lens to compare representations from heterogeneous models. By mapping both latent states and ground truth trajectories into a common relative space, the paper avoids trivial invariances and allows model to model comparisons. \n\n3. The work reveals several interesting empirical phenomena that can inspire further research. Examples include family level structure in the learned geometries and the observation that high forecasting accuracy can coexist with low alignment for some architectures.  \nThese findings raise concrete questions about inductive biases and optimization that go beyond standard loss based evaluation.\n\n**Main Weaknesses**\n\n1. The forecasting architectures are standard and the relative embedding framework closely follows prior work with only light adaptation to dynamical systems.  \n\n2. The evaluation relies almost entirely on cosine similarity in the relative embedding space. Other similarity measures and a more systematic robustness analysis are missing, so the choice of metric is not fully justified.  \n\n3. The analysis of differences between model families is mostly descriptive. The paper reports patterns for MLPs, RNNs, Transformers, and ESNs but offers little mechanistic explanation or theory to support these observations.  \n\n4. Latent states are treated as black box vectors, with no probing experiments or strong links to physical quantities or dynamical properties, which limits the depth of the conclusions about representation geometry.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims are supported by systematic experiments on several canonical dynamical systems with multiple neural forecaster families. The methodology for relative embeddings and alignment is described in sufficient detail and is implemented consistently across models. Some broader interpretive statements, such as the generality of relative geometry as a tool for representation science, are more qualitative, but they remain plausible.", "audience_yn": "Yes", "audience_explanation": "I think the work speaks to readers who care about neural forecasting for dynamical systems. The study offers a systematic comparison of common forecaster architectures and highlights family level structure in their latent spaces. Researchers interested in neural ODEs, Koopman operators, and model interpretability may find the relative geometry analysis useful as a diagnostic tool.", "requested_changes": "1. The paper relies almost entirely on cosine similarity in the relative embedding space. I recommend adding comparisons with other similarity measures such as CKA, Procrustes based metrics, or RSA on the same setups. Please also study robustness to the choice and number of anchors in a more systematic way, for example through ablations and statistical summaries.\n\n2. The current analysis mainly describes that different architectures show different alignment and accuracy patterns. It would be important to provide a more mechanistic explanation that links these patterns to properties of MLPs, RNNs, Transformers, and ESNs. \n\n3. At present the latent states are treated as black box vectors. I suggest adding probing experiments that connect representations to physical quantities, dynamical invariants, or local stability indicators. Such analyses would provide solid evidence about what information the latent geometry actually encodes and would greatly strengthen the main claims.\n\n4. The paper argues that relative geometry can help evaluate and compare forecasters, yet it does not show concrete decision scenarios.  \nI recommend adding at least one experiment where alignment metrics are used for early stopping, model selection, or failure detection.  \nDemonstrating that these metrics change a practical choice would make the contribution more compelling for practitioners.\n\n5. Several statements present the framework as a simple and general tool for representation science, which may feel stronger than what the experiments support. Please better situate the work relative to prior alignment methods and clarify the specific regimes where the proposed approach is most appropriate. A more explicit discussion of limitations and of settings where the method may fail would improve the balance and credibility of the paper.", "broader_impact_concerns": "1. The study focuses on representation analysis for neural forecasters on synthetic or standard dynamical systems.  \nIt would still be useful to add a short Broader Impact statement that clarifies this low risk profile.\n\n2. In those settings misinterpretation of model reliability or alignment metrics could influence high impact decisions. The authors could briefly discuss the need for domain specific validation before using these tools in real-world decision-making.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GoTr5lCrmh", "reviewer": "Reviewer_pBzn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Neural forecasters (MLPs, RNNs, Transformers, and ESNs) take a window of past states and predict future states, but their internal hidden-states or latent representations evolve differently w.r.t the architecture. This paper examines how these internal latent trajectories reflect the underlying dynamical system, and how similar the internal latent representations across different neural forecasters. To achieve this, the authors select the subject of anchor-based relative embeddings: they select a shared set of anchor points from the dataset and represent each model’s latent vectors by their similarities to these anchors, producing geometry-agnostic representations that can be compared across architectures. The evaluation focuses on periodic, quasi-periodic, and chaotic systems. The authors observe that both RNNs and MLPs learn similar latent geometry within their respective families even under different seeds, while ESNs and Transformers show weaker and less stable alignment. Across architectures, they observe that Transformers (and sometimes ESNs) result in higher prediction accuracy even when their alignment with the underlying dynamics is low, indicating a disconnection between forecasting performance and representational similarity. Further, the authors interpret that the temporal evaluation of alignment is shown to be sensitive to noise and input-sequence length, with effects varying across neural forecasters.\n\n**Contributions:**\n1. *Introduction of anchor-based relative embeddings to neural forecasters*: This study adapts the relative-embedding alignment approach to neural forecasters, making them geometric-agnostic and directly comparable across different architectures.\n2. *Uncovering the alignment patterns across architectures*: This study interprets how representational alignment relates to forecasting accuracy across architectures and provides architecture-specific behaviors.\n3. *Comprehensive evaluation*: The study presents an extensive assessment of seven canonical systems including, continuous, discrete, periodic, quasi-periodic, and chaotic, and four different architectures. Further, the authors compute the underlying latent alignment similarity within their model families, cross-modal families, and perform robustness studies including noise input and longer sequence lengths.\n\n**Technical summary:**\nThis is primarily an empirical study, and its methodology involves the following components:\n1. *Representational alignment framework:* The authors present the alignment framework with components including data (trajectory of data from seven canonical systems), neural forecasters (model architecture), encoder’s latent representation (applying neural forecaster on input windows), anchor-based relative embeddings (obtained from a fixed subset), and similarity metrics (cosine, T1 and rank similarity) between two embeddings is well explained. \n2. *Neural forecasters and propagated dynamics:* The authors present encoder -> propagator -> decoder architecture: (i) Encoder takes a window of past states and produces a latent vector, (ii) Evolves (or “propagates”) this latent state forward for H forecasting steps (z0​->z1​->z2​->....->zH​) and (iii) Decoder converts the final latent state zH​ into future predictions. Further, the authors discuss three types of propagators: (i) Identity, used in standard MLP, RNN and Transformer models, (ii) Neural ODE, used for continuous-latent dynamics, and (iii) Koopman, which enforces linear latent-time evolution.\n\n**Experimental design/evaluation:**\nThe authors focus on (i) cross-family alignment, (ii) alignment and its relation to forecasting accuracy, and (iii) perturbation experiments.\n1. *Cross-family alignment*: This analysis evaluates the representational alignment using anchor-based relative embeddings and measures the similarity between architectures by making them geometry-agnostic. Across seven canonical systems and four different architectures, this analysis provides cross-forecaster alignment beyond what can be observed in the original latent spaces.\n2. *Alignment relation to forecasting accuracy*: This analysis provides whether the prediction accuracy of a true system relates to its underlying latent representation changes and how this relation varies across different neural forecasters. Specifically, the authors present the Alignment vs. Performance column for each forecaster to illustrate how representational similarity correlates with forecasting error.\n3. Perturbation experiments: This analysis evaluates the effect of different parameters, such as input noise and sequence length, on both prediction accuracy and representational alignment across architectures.\n\n**Strengths:**\n\nI found this work to have the following strengths:\n* *Clarity:* The main motivation for understanding the geometry of learned latent dynamics across different model families is well explained, with a clear limitation of prior work. The paper also highlights the limitations of current representational similarities metrics and identifies a gap in comparing forecasting systems. Later, the introduction of anchor-based relative embeddings, how they selected subset from a dataset and use the same set for computing similarity with latent representation from each model is clearly presented.\n* *Originality:* The idea of using anchor-based relative embeddings to neural forecasters, is quite novel conceptually. Since, the forecasting accuracies may vary with respect to different architectures and their underlying latent state representation may differ, this paper examines the representational alignment across architectures using relative embeddings and provides architecture-specific behaviours.\n* *Significance:* This work is significant in that it contributes to a better understanding of the underlying changes in latent representations in neural dynamics while forecasting trajectories. It helps clarify how different architectures behave internally and whether their representation alignment is similar or different when achieving strong forecasting accuracy. Overall, the adaptation of the anchor-based approach is quite novel and meaningful for comparing different architectures and interpreting architectural behaviors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provides strong empirical evidence for its main claims through experiments on seven canonical dynamical systems and four neural forecaster families. However, from my perspective, the primary weaknesses of this study arise from the lack of empirical validation on real world datasets.\n\n* *Lack of real-world datasets:* The neural forecasters in this study are primarily validated on seven canonical systems \tbut does not test real-world forecasting datasets. This implies that the findings may not generalize to noisy, high-dimensional real-world time series. I strongly recommend that the authors evaluate the neural forecasters on existing real-world datasets and compare their forecasting performance on established benchmarks. This would strengthen the empirical validity of the work and demonstrate whether the representational alignment findings generalize beyond synthetic dynamical systems.\n* *Anchors are dataset specific*: Although the anchor-based relative embedding idea is interesting, all the anchors are selected from a subset of dataset and used as fixed reference vectors to measure similarity with model latents. The paper does not discuss why a fixed subset of anchors is required, how results might change when anchors are selected from different datasets, or whether anchors can be learned optimally. The authors do not provide criteria or justification for their anchor selection strategy.\n* *Cross-modal comparison is indirect*: Although authors mentioned that their primary goal is to compare similarities between internal representations of different models, the method compares models only indirectly through their relative similarities to a shared set of anchors, rather than through direct latent-space alignment. While this design is intentionally geometry-agnostic, the resulting similarity measures depend heavily on anchor-point relationships. Therefore, the authors should further interpret what these anchor points represent, how they influence alignment conclusions, and whether anchor selection affects the observed cross-model similarity patterns.\n* *Limited discussion on results*: Although the paper reports some interesting findings such as RNNs and MLPs showing strong within-family alignment, Transformers and ESNs result in weak alignment despite high accuracy. However, the authors have not discussed why these patterns occur. In particular, because Transformer models dominate current research trends, a deeper interpretation of why their latent representations diverge from other architectures, and why they achieve higher forecasting accuracy despite low alignment, is missing. The absence of such analysis leaves an important gap in understanding architecture-specific behaviors.\n\nFor a complete and detailed account of both major and minor issues, please refer to the “Requested Changes” section.", "audience_yn": "Yes", "audience_explanation": "Yes.\nThe findings of this paper would be of interest to a meaningful portion of TMLR’s journal.", "requested_changes": "Specifically, there are several points that I believe require further attention/work. I have divided these into major issues, which should be prioritized, and minor ones, which should be addressed for a strong version of current work.\n\n**Major Comments/Questions:**\n1. *The Transformer architecture is under-specified:* The authors clearly specified in the Table 1 and Table 2 that RNNs are autoregressive while Transformer uses casual attention. However, it is unclear whether the Transformer in this paper is functionally autoregressive in the forecasting sense (i.e., producing future steps sequentially) or whether it performs one-shot multi-step prediction from a single latent vector. Autoregressive Transformers typically include self-attention, feed-forward layers, and positional encodings that preserve temporal order, but the paper does not fully describe how these components are used for forecasting in this setup. This raises an important question: Which specific components of the Transformer architecture lead to high forecasting accuracy despite extremely low latent alignment? Furthermore, it is unclear whether this mismatch arises from a limitation of the relative-embedding method (e.g., anchor-based comparisons not capturing attention-based representations) or from the representational geometry of Transformers themselves, which may not form smooth latent trajectories. Clarifying these points is essential to interpret the alignment results.\n2. All Figures (e.g., Fig. 1, Fig. 2, and Fig. 3) use very small font sizes for axis ticks and labels, making them hard to read. Also, none of the Figures have proper x-axis, y-axis and color legends, making it difficult to understand the results without repeatedly referring back to the text.\n\n**Minor Comments/Typos**:\n* While addressing the following points may not be critical to the paper’s core contributions, doing so would enhance the overall quality. \nIn Section 4.3, the authors report parameters and experimental details by referring directly to a .csv file. I recommend that all key hyperparameters and training details be described explicitly in the text rather than requiring readers to inspect external files. Since the authors already provide a GitHub repository, including a clear summary of experimental configurations within the paper would improve clarity and self-containment.\n* I recommend that the authors provide a notation table summarizing the symbols used for trajectories, relative embeddings, anchor points, and other key mathematical elements. This would greatly improve readability and help readers follow the methodological components of the paper more easily.\n* Although the paper evaluates models on seven canonical dynamical systems, the descriptions provided are minimal. I recommend that the authors clearly explain the differences between these systems and outline the key dynamical properties of each (e.g., periodic, quasi-periodic, chaotic, spatiotemporal chaos). This would help readers understand why these particular systems were chosen, what regimes they represent, and how they relate to the reported alignment behavior.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "w2kB4FX9Lc", "reviewer": "Reviewer_yRE7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\n\nThis paper conducts an empirical study of representational alignment in neural forecasting models applied to synthetic dynamical systems (e.g., Lorenz-63, double pendulum, skew-product maps). It adapts previously proposed anchor-based relative representation methods to compare latent spaces of different architectures. The authors report structure in representational similarity within and across model families and present preliminary evidence of encoder–decoder stitching across architectures.\n\n---\n\n**Key strengths:**\n\n- Systematic empirical study across multiple dynamical systems and architecture families; well-executed evaluation methodology.\n- Clear experimental framework and high reproducibility (simulation code, hyperparameters, and implementation details are provided).\n- Some insightful findings, such as divergence between representational alignment and forecasting performance for architectures like transformers and ESNs, and structure in family-wise alignment.\n\n---\n\n**Key weaknesses**\n\n- The learning objective shaping the latent representations is not clearly articulated. It is unclear why optimizing for forecasting loss should be expected to yield latent variables that reflect underlying system dynamics rather than generic short-horizon predictive encodings.\n- Some claims risk overclaim: to my understanding, use of terms such as \"learned dynamics\" or \"training dynamics\" suggests recovery of true dynamical structure, but the experiments do not provide evidence to support that interpretation.\n- Empirical evaluation focuses on low-dimensional synthetic systems, limiting the ability to assess generalization to real-world or higher-complexity settings.\n- The anchor-based representational comparison closely follows prior work, with modest methodological extension beyond applying the technique to a new domain (though the authors clearly acknowledge this prior work).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The evidence is convincing with respect to representational alignment and forecasting comparisons. However, it does not adequately support claims that the models learn or recover underlying system dynamics. Without theoretical grounding or diagnostics linking latent structure to true dynamics, some interpretations appear overstated relative to the results.", "audience_yn": "Yes", "audience_explanation": "The topic aligns with areas including representation learning, neural forecasting, and latent space analysis. Although the contribution is modest, the insights on representational alignment across forecasting architectures are likely to benefit some readers.", "requested_changes": "I would suggest the authors consider the following revisions:\n\n- Clarify the contribution relative to prior anchor-based representation alignment and stitching work (e.g., Moschella et al., 2023).\nPlease articulate which components are reused, which are adapted to the dynamical forecasting setting, and what new insights the present work contributes. As written, the methodological novelty is unclear.\n\n- Clarify the learning objective and its connection to \"dynamics\".\nState explicitly whether the goal is: (a) forecasting performance only, (b) learning representations that reflect underlying system dynamics, or (c) both. If (b) or (c), provide theoretical or empirical justification (e.g., diagnostics showing that latent spaces preserve dynamical invariants or structure) rather than assuming forecasting loss implies learning dynamics.\n\n- Expand evaluation beyond low-dimensional synthetic systems.\nIncluding at least one higher-dimensional or real-world dataset (e.g., realistic fluid/CFD systems, spatiotemporal climate/traffic/energy forecasting) would improve external validity and strengthen claims about the generality of the findings.", "broader_impact_concerns": "The work uses fully synthetic data and focuses on representation analysis. No significant ethical concerns are identified.", "additional_comments": "The figures are strong and contribute meaningfully to the interpretability of the results. The work has the potential to serve as a reproducible empirical benchmark for studying representational alignment in neural forecasting models, particularly if the authors expand the experimental coverage and provide a more comprehensive and carefully packaged release of code, data, and evaluation tools.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)