Navigating the Latent Space Dynamics of Neural Models
Authors: Marco Fumero
Luca Moschella
Emanuele Rodolà
Francesco Locatello
Venue: ICLR 2026 Oral (2026)
Keywords: Representation learning
latent vector field
autoencoders
memorization and generalization
attractor
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Neural networks transform high-dimensional data into compact, structured representations, often modeled as elements of a lower dimensional latent space. In this paper, we present an alternative interpretation of neural models as dynamical systems acting on the latent manifold. Specifically, we show that autoencoder models implicitly define a _latent vector field_ on the manifold, derived by iteratively applying the encoding-decoding map, without any additional training. We observe that standard training procedures introduce inductive biases that lead to the emergence of attractor points within this vector field.
Drawing on this insight, we propose to leverage the vector field as a _representation_ for the network, providing a novel tool to analyze the properties of the model and the data. This representation enables to: $(i)$ analyze the generalization and memorization regimes of neural models, even throughout training; $(ii)$ extract prior knowledge encoded in the network's parameters from the attractors, without requiring any input data; $(iii)$ identify out-of-distribution samples from their trajectories in the vector field.
We further validate our approach on vision foundation models, showcasing the applicability and effectiveness of our method in real-world scenarios.

Published as a conference paper at ICLR 2026
NAVIGATING THE LATENT SPACE DYNAMICS OF NEURAL MODELS
Marco Fumero IST Austria
Luca Moschella Sapienza
Emanuele Rodolà Sapienza Paradigma
Francesco Locatello IST Austria
ABSTRACT
Neural networks transform high-dimensional data into compact, structured repre- sentations, often modeled as elements of lower dimensional latent space. In this paper, we present an alternative interpretation of neural models as dynamical sys- tems acting on the latent manifold. Specifically, we show that autoencoder models implicitly define latent vector field on the manifold, derived by iteratively apply- ing the mncoding-decoding standard training procedures introduce inductive biases that lead to the emergence of attractor points within this vector field. Drawing this insight, we propose leverage the vector field as a representation for the network, providing a novel tool analyze the properties of the model and the data. This representation enables to: (i) analyze the generalization and memorization regimes of neural models, even throughout training; (fi) extract prior knowledge encoded in the network's parameters from the attractors, without requiring any input data; (ii1) identify out-of-distribution samples from their trajectories in the vector field. We further validate our approach on vision foundation models, showcasing the applicability and effectiveness of our method in real-wort scenarios.
map, without any additional training. We observe that
INTRODUCTION
Figure 1: Latent dynamics of AEs. Latent vector fields induced by autoencoders with bottleneck = 2, trained on MNIST, with Zn ~ U[-8,8) 8). Models with different initializations are shown. Colors (viridis colormap) represent vector norms ranging from violet (low) yellow (high). The shape of the latent manifold identifies with the encoder's support. White regions indicate where the vector field vanishes, revealing attractors aligned with bigh-density areas of the data distribution.
Neural networks are powerful function approximators, capable of solving complex tasks across a wide range of domains (Jumper et al., 2021; Radford et al., 2021). A core component of their success lies in their ability to transform high-dimensional imputs into compact, structured representations (Bengio al., 2013), typically modeled as points in dower-dimensionall latent space.
In this work, we propose a novel perspective: for any given autoencoder (AE) architecture, there exists an associated latent vector field (see Figure 1) that characterizes the model's behavior in
"Corresponding email: marco.fumeroeist.cc.. 'Curently at Apple.
'Equally advising

---

Published as a conference paper at ICLR 2026
representation space. This field arises naturally by iterating the mncoder-decoder map in the latent space, requiring no additional training. Intuitively, such dynamics often settle into attractors: stable states toward which nearby trajectories converge. summarizing the long-term behavior of the system.
While prior work has shown that overparameterized autoencoders can memorize training data under strong assumptions (Radhakrishnan et al., 2020; Jiang and Pehlevan, 2020), we consider a broader setting. We show that memorization emerges as just one type of attractor in the latent vector field, and that the structure of this field reflects deeper properties of the network, such generalization behavior and sensitivity to distribution shifts.
Our contributions can be summarized as follows:
We show that every AE implicitly defines a latent vector field, whose trajectories and fixed points encode properties of both the model and the data.
We demonstrate that most neural mappings are contractive, leading naturally to the emer- gence of fixed points and attractors in the latent space.
We empirically connect attractors to the network's 's memorization and generalization regimes, showing how they evolve during training.
We show experimentally that vision foundation models can be probed in a data-free manner: initializing with noise, recover attractors that reveal semantic information embedded in the model's s weights.
Finally, we show that trajectories in the latent vector field can capture the learned data distribution and serve as a signal for detecting distribution shifts.
2 METHOD
Notation and background We consider neural models as compositions of encoder-decoder modules, defined as Fe Do, Es, parametrized by é [0, 02). The encoder Es, maps inputs x p(x) supported on X' C Rm to a typically lower-dimensionll space C R² and the decoder Do, reconstructs the input. The model is trained to minimize the mean squared error LMSE:
(xSE(x)
Σ
| - (x) [2 XR(O),
where R is either an explicit or implicit regularization term encouraging contractive behavior in F.
We posit that minimizing this objective leads to reduction in the spectral norm of the Jacobian, [|Jr(x)llo for p(x). Examples of regularization include weight decay, where R ||O|| or data augmentation by sampling transformations ~ p(T) and minimizing:
LMSE(x) = Σ x 1(x) + Σ F(T))
(2)
XEX
TEP(T)
Such augmentations include additive Gaussian noise in denoising AEs (Vincent et al., 2008) and input masking in masked AEs (He et al., 2022). Another form of contractive pressure is the bottleneck dimension A: dim 2), which places hard upper bound the rank of the encoder Jacobian, rank(Jg) <= & and, by the chain rule, constrains the full model Jacobian (-) = JJ(.) (.) @ JD(.). Table 3 (Appendix) lists many AE variants, including denoising AEs (DAEs) (Vincent et al., 2008), sparse AEs (SAEs) (Ng et al., 2011), 1), variational AES (VAEs) (Kingma et al., 2013), and others (Rifai al., 2011; I; Alain and Bengio, 2014; Gao al., 2024), highlighting how their objectives promote local contractive behavior around training data.
Given a possibly pretrained AE model, we define the map f(z) E D(z) and study its repeated application f(. fz)())) which can be modeled as differential equation.
2.1 THE LATENT DYNAMICS OF NEURAL MODELS
Given sample z E Z C A we study the effect of repeatedly applying the map f. i.e., fo fz)). By introducing discrete time parameter t, this iterative process defines the discrete ODE:
{
Z
f(z)
(3)
2

---

Published as a conference paper at ICLR 2026
which discretizes the following continuous differential equation:
Oz Ōt
f(z) -
In Eq. 3, the map f defines the pushforward of a latent vector field V RR + R tracing nonlinear trajectories in latent space (Fig. 1). A natural question is whether the ODE has well-defined and unique solutions. By the Banach fixed-point theorem, this holds if and only if f is hipschit-coottinuon with Lipschitz constant C<1.
Definition 1. A function Z + Z is Lipschite-continuuus if there exists constant C s.t. for every pair of points Z1: Zz
((f(),), f(2))) SC 22)1,
When C' < 1, f is called contractive, for a given metric d on Z.
For any contractive map. Eq. 3 admits fixed-point solutions, i.e., repeatedly applying the map f will converge to unique solution z' satisfying z' f(z"). The fixed points z can act attractors, capturing and summarizing the system's long-term dynamies.
Definition 2. A fixed point z fz) is attractor of differentiable map f if all eigenvalues of the Jacobian J of f at z are strictly less than one in absolute value.
When is nonlinear, the ODE Eq. 3 can have multiple solutions depending on the initial conditions The previous definition allows for definition of Lipschitz continuity, which is inherently local:
Definition 3. Let f + Z be differentiable and C-Lipschitz continuous. Then the Lipschitz constant C is given by: C = supxez ( [z)(2) where (z) is the Jacobian of f evaluated at z and ||- lla is the spectral norm.
The set of initial conditions Zg leading to the same attractor z", denoted as basin of attraction.
Why are neural mappings contractive in practice? We argue that mappings learned by neural AEs and their variants tend to be locally contractive, i.e.,their Jacobians have small eigenvalues near training examples. This behavior emerges naturally from several explicit and implicit inductive biases present in modern training pipelines. Below, we outline the main factors promoting contractive behavior:
Initialization bias. Standard initialization schemes (LeCun et al., 2002; Glorot and Bengio, 2010; He al., 2015) are designed to preserve activation variance the start of training avoid vanishing or exploding gradients. Theoretical arguments in support assume i.i.d. inputs and weights, and are often tied to specific activation functions (e.g., ReLU in (He et al., 2015)). However, real-world training data is typically correlated, and architectural features like residual connections (He et al., 2016) break weight independence. As result, networks often exhibit bias toward mappings that are globally expansive or contractive, empirically skewed toward the latter (Poole et al., 2016).
We illustrate this empirically in Figure 6 (Appendix), showing that various vision backbones exhibit contractive behavior at initialization. Figure 3a (left) shows a 2D latent vector field which is globally contractive at initialization towards single attractor at the origin. This contractive behavior also holds in higher dimensions (see the number of attractors at epoch 0 in Figure 3c).
Explicit regularization. Common regularization methods like weight decay (D' Angelo et al., 2024) encourage contraction by penalizing the norm of model parameters. In the linear case, minimizing parameter norms directly reduces the Jacobian's 's spectral norm, making the map contractive. While this link is less direct for nonlinear models, the effect due weight decay persists in practice. Regularization is often integrated into optimizers used in large-scale models (Loshchilov, 201 7). Additional architectural constraints, such small bottleneck dimensions, also limit the rank of (z) Soft constraints include KL divergence in VAEs or sparsity penalties in SAES (see Table in the Appendix for a complete overview).
Implicit regularization. Data augmentations introduce local perturbations around training exam- ples, effectively defining a neighborhood structure in input space. For instance, Gaussian noise in DAEs or masking in MAEs perturbs inputs along specific directions. These augmentations implicitly regularize the Jacobian (x) by penalizing sensitivity to those perturbations. Unlike parametric regularization, this effect is inherently local and nonparametric.
3

---

Published as a conference paper at ICLR 2026
3 THEORETICAL REMARKS
In this section, we study the behavior of the latent vector field, focusing on its trajectories (Section 3.1) as well its fixed points and attractors (Section 3.2), when they exist. Formal statements and proofs of all theorems and propositions are provided in Appendix A.
3.1 TRAJECTORIES OF THE LATENT VECTOR FIELD
Prior work (Miyasawa al., 1961; Robbins, 1992; Alain and Bengio, 2014) has shown that, under ideal conditions, the residual of a denoising AE trained with noise variance o² approximates the score function, i.e., the gradient of the log-density of the data, as + 0. Interestingly, Alain and Bengio (2014) shows a connection between denoising and contractive AEs, where the Jacobian norm of the input-output mapping explicitly penalized.
We build on this by observing a more general phenomenon: when an autoencoding map is locally contractive relative to a chosen neighborhood structure (c.g.. Gaussian noise, masking) and sufficiently approximates the input distribution p(x), the induced latent vector field pushes points in the direction of the score of the corresponding prior in latent space.
Informally, this implies that the vector field acts to nonlinearly project samples toward regions of high probability on the data manifold.
Theorem 1 (informal, proof in Appendix A.1). Let F be a trained autoencoder and let q(z) = (zxx)x(z)x)dx be the marginal distribution induced in latent space. Assume q(z) smooth and that there exists an open neighborhood Ω 2 supp and a constant
such that supzeal || [z) S
L. Then, latent dynamics f(z) in is locally proportional to the score function V, log (z)).
This result establishes a general link between the vector field's trajectories and the score function, under the assumption of local contractivity. In practice, neural networks often strike balance between the reconstruction loss (Eq. 1) and regularization the Jacobian, which contributes to the emergence of attractors in the latent space.
An important implication of Theorem 1 is that integrating the vector field f -z effectively estimates the log-de density, i.e., f V log q dz = log q(z) + C' in unbounded domains. If the Jacobian J is symmetric (e.g., in AÉS with tied weights (Alain and Olivier, 2013)), then the latent vector field is conservative and corresponds to the gradient of a potential V = V. In this case, the AE defines an energy-based model with ((z) = e B()) (Zhai et al., 2016; Song and Kingma, 2021). In the general mnon-conservative case, we show empirically in Section 4.2 that the trajectories still reflect the learned prior distribution.
However, attractors are generally not accessible solving the fixed-poine equation with first order methods, unless initialized very close to them. The following result formalizes this:
Proposition 3.1 (informal, proof in Appendix A.2). Iterations of the map f in Eq. 3 correspond to gradient descent potential T(z) (i.e., I s z - aT((z) if and only if f is an isometry (its eigenvalues are near 1), or the dynamics occur near attractors, where Jy vanishes.
This underscores the role of hider dynamics in the vector field: repeated applications of f trace complex, nonlinear trajectories that can escape spurious or unstable fixed points.
3.2 CHARACTERIZING THE ATTRACTORS OF THE LATENT VECTOR FIELD
In this section, we aim to characterize what the fixed point solutions of Eq. 3 represent. In Ap- pendix A.4 we consider the two cases of linear and homogencous networks as examples, to build intuition on the characterization of attractors and latent vector field.
Between memorization and generalization. Our goal is to characterize the properties of the latent vector field and the leamned attractors in the general setting. We argue that different models can fit training points reaching similar low loss, but interpolate differently outside the training support, depending the regularization strength and the amount of overparamettrization.
In this context, a key question arises: what information do the attractors z" encode? We argue that neural models lie in spectrum between memorization and generalization, depending on the strength
4

---

Published as a conference paper at ICLR 2026
Memorization
2.3
1.
CRATS Hosd
8
3
0
4
5
34
5
9
Tee Drtt
32
3
80
S-2566
Gemeralization
DetieneccDimasionn
(a) Attractors
Figure 2: Memorization vs Generolization. Attractors memorize the training data as a function the
rank of y (z) by adjusting the bottleneck dimension k (leff) which is inversely proportional to the amount of generalization attained by the model (center); On the right we show example decoded attractors transitioning from a strong memorization model (first row) to good generalization (last row).
of the regularization term in Eq. 1. We hypothesize that attractors fully characterize where model falls on this spectrum. To do so, we first connect attractors to the notion of generalization via the following proposition:
Proposition 3.2 (Informal, proof in Appendix A.3). Let Z" be a dictionary of attractors of f E in neighborood Ω of the latent space, and let I1z)) denote the projection onto the nearest altractor latent code z Exx). If D LD-ischhhtzz on Q then for any test point x:
|x 12 S
X -DII(EEX))II + 1E - (I(E
error prototype
coverage
In short, attractors define dictionary for the data: when they coincide with training points, prototype error on these points vanishes but coverage is narrow (memorization regime). whereas generalization requires attractors that both cover the latent space and serve good prototypes for unseen data.
We consider attractors as representations summarizing the information stored in the weights of the network. This interpretation is in line with the convergence of paths in the vector field to modes of the learned distribution Theorem 1, and can be easily seen in the case of memorization.
Extreme overparametrization case. When the capacity of a network exceeds the number of training examples by far, AEs enter an overfitting regime leading to data memorization (Zhang et al., 2019; Radhakrishnan al., 2020; Jiang and Pehlevan, 2020; Kadkhodaie al., 2023). The network in this case learns a constant function or an approximation thereof, which can be retrieved through the iterations of Eq. 3.
3.2.1 THE ROLE OF REGULARIZATION
We empirically demonstrate that the balance between memorization and generalization is governed by the strength of regularization. In Figure 2, we show how increasing regularization on the Jacobian (z()) drives the model from memorization regime, where many training examples are stored as attractors, toward generalization. We remark that this memorization arises in an over-regularized regime, as opposed the overfitting regime of extremely overparametrized networks.
Setting. We trained 30 convolutional AEs on the CIFAR, MNIST, and FashionMNISS datasets, varying the bottleneck dimension from 512. This acts as hard regularizer on JJ (z"),, by constraining its rank to be < k. We compute attractors Z from elements of the training set Xerain by iterating f till convergence. We measure the degree of memorization by defining memorization coefficient, which is given by the cosine similarity of each decoded attractor x" D(z") ) to its closest point in the training set, mem(z") = mil standard deviation over attractors. We measure generalization by simply reporting the error on the test set. In Figure 13 in the Appendix, we also report the rank explaining 90% of the variance of the matrix of decoded attractors X showing that attractors corresponding to good generalization models span more directions in the input space. For additional information on the model, hyperparameters, and settings, refer to Appendix D).
cDs(D(z)) x) and we report the mean and
5

---

Published as a conference paper at ICLR 2026
(a) Evolution of latent vector field over training iterations.
Noaraa Sualian confidert
Train Atrrectirs
Teat derrccora
Navice Aetrartart
352
Epoch
(b) Memorization to generalization.
(c) Number of attractors.
(d) Similarity Zooise and Zirain
Figure 3: Latent vector field dynamics. (a) The 2D vector field (k 2) expands from a single attractor, eventually stabilizing and over-fitting because of capacity limits. Bottom: Evolution of larger capacity AEs (k: = 128) across training. (b) Throughout training, the network first memorizes the data with high memorization coefficient (in blue) and then generalizes, achieving low test error (red). (c): Evolution of attractor count for training (blue), test (red), and noise (yellow) samples; (d) Attractors computed from training and from gaussian noise converge during training (green), while the separability of the trajectories (measured as FPR95, the lower the better) increase (purple).
Analysis of results. In Figure 2 on the left, we observe that the more regularized networks from to 16) tend to memorize data, trading off generalization performance (middle plot). We remark that this kind of memorization not due overfitting, as was shown in (Kadkhodaie et al., 2023) for diffusion models, but it happens in the underfit regime, due the strong regularization constraint on (x)). We show Figure 14 the Appendix results in the overfitting regime, by training the models with different sample sizes and observing similar pattern.
Takeaway. Attractors capture the interplay between generalization and memorization of neu- ral models, which corresponds the trade-off between the reconstruction performance and regularization term of the AE model.
3.2.2 MEMORIZATION AND GENERALIZATION ACROSS TRAINING
In the experiment in Figure 3 show that similar transition from memorization regime generalization occurs across training.
Setting. We monitor the latent vector field and attractors statistics across the training dynamics of convolutional AE trained on MNIST with bottleneck dimension & = 128 (bottom row plots in the Figure). To show qualitatively the evolution of latent vector field, also plot across training epochs for AEs with bottleneck dimension & = 2 in Figure 3a.
Analysis of results. In Figure 3b show the transition from memorization, occurring the first epochs of training, to generalization, by plotting the memorization coefficient and the test error across training, observing trade-off between the two. the center plot, we plot the fraction of distinct attractors, computed from 5000 random elements respectively from the Ptrain. Ptest and (0, I), where we consider two attractors equal if cos zz) 0.99. Initially, the model converges to single attractor (as also seen in the 2D example in Figure 3a). Over time, the number of attractors increases, stabilizing for training and test data tandem with the test loss, while attractors from noise inputs converge more slowly. The right plot shows the Chamfer symmetric similarity between attractors from the training and noise distributions, which increases over training as the two distributions of attractors match.

---

Published as a conference paper at ICLR 2026
Laion28
EuT
Places365
grhe Alricton
Input
MSE
MSE
02
Orhe
Ortho
Sparsity
Sparsty
Sparsity
Imagenet-1k
PatchComelyon
Cifar100
Altractors
0.4
MS
(a) Reconstructions with 5% atoms. First
0.1
row: Input; second row: Orthogonal basis;
last row: Attractors
0.3
Sparsity
Sparskl
Sparsity
Figure 4: Data-free weight information probing of Stable Diffusion model. We plot the error (MSE) vs sparsity (number of atoms) used to reconstruct samples from diverse dataset respectively from (i) an orthonormal random basis of the latent space (blue); (ii) attractors computed from gaussian noise (red), showing that attractors consistently reconstructs samples better on all datasets. (Right) Reconstructions using 5% of the atoms ImageNet
Importantly, while attractors derived from noise and training data become increasingly similar over the course of training, the trajectories leading to them differ significantly. To quantify this, we compute the False Positive Rate 95% True Positive Rate (FPR95) scores for distinguishing trajectories originating from noise versus those from training data: FPR95 measures what percentage of noise latent trajectories are falsely classified as training trajectories, setting a threshold such that classified correctly. We define the trajectory score score(z)
95% of training trajectories N Ezer(()) d(z,, Zivain) where TT = Zoo nN) is the trajectory. The FPR95 decreases sharply during training, showing that the network learns to separate the two types of trajectories.
1
Takeaway. As the latent vector field evolves during training, the model transitions from mem- orization to generalization, forming similar attractors from different input distributions, while retaining information of the source distribution in the latent trajectories.
4 EXPERIMENTS ON VISION FOUNDATION MODELS
In this section we demonstrate the existence and applicability of the latent dynamics on vision foun- dation models, including the AE backbones the Stable Diffusion model (Rombach et al., 2022) and vision transformer masked AEs (He et al., 2022), showing; (i) how information stored in the weights of a pretrained model can be recovered by computing the attractors from noise (ii) how trajectories the latent dynamics are informative to characterize the learned distribution and detect distribution shifts.
4.1 DATA FREE WEIGHT PROBING
Setting. In this experiment, we investigate how much information stored in neural network's weights can be recovered purely from attractors computed on Gaussian noise, without access to any input data. We focus on AE component of Stable Diffusion (Rombach et al., 2022) pretrained on the large scale Lai cn2B dataset (Schuhmann et al., 2022).
We sample Z, ~ (0, 7) and compute the corresponding attractors as solutions to fZZ;) = Z We generate NN 4096 such points, matching the latent dimensionality & of the model. We evaluate the reconstruction performance on 6 diverse datasets Laion2B (Schuhmann et al., 2022), Imagenet (Deng al., 2009), EuroSAT (Helber et al., 2019), CIFAR1 00 (Krizhevsky al., 2009), Phw (Litjens et al., 2018), and Places365 (Zhou and Paffenroth, 2017). These datasets span general, medical, and satellite image domains.
For each dataset, we randomly sample 500 test examples and reconstruct them in latent space using Orthogonal Matching Pursuit (OMP) (Mallat and Zhang, 1993), with varying sparsity levels (i.e.,
7

---

Published as a conference paper at ICLR 2026
SEN3997
Placee365
Tomber
Methoe
AUROCTT
19R 1
AUROCT
HPX
AUROCT
FVK
AUROC
29.95
(a) FPR and AUROC scores
(b) KNN baseline
(c) Latent trajectories
Figure 5: Trajectories in the latent vector field characterize distribution shifts We measure out-of- distribution detection performance ViTMAE: On the left report scores for 4 different datasets, highly outperforming the KNN baseline. On the right, histograms of scores on the INaturalist dataset, demonstrating much better separability between in-distribution and out-of-distribution when probing latent trajectories distances to in-distribution attractors (c), as opposed to measure distances to in-distribution features (b).
number of atoms used). Reconstructions are decoded via D, and performance is compared against a baseline reconstruction using the initial orthogonal samples Zū, which, by design, fully span the latent space Z.
Analysis of results. In Figure we plot the error function of the number of elements (atoms) chosen to reconstruct the test samples using OMP. We compare by building random orthogonal basis sampled in the latent space. On all considered datasets, noise attractors recover test samples with lower reconstruction error, representing better dictionary of signals. In Figure 4a, we show qualitative reconstruction using only 5 % of the atoms. Figures 7, 8, 9 in the Appendix, we include additional qualitative evidence of this phenomenon, visualizing reconstructions of random samples of the datasets as a function of the number of atoms used. In Appendix B, we report additional results on variants of the AE of different sizes.
Takeaway. Attractors of foundation models computed from noise can serve dictionary of signals to represent diverse datasets, demonstrating that is possible to probe the information stored in the weights of foundation models in black box way, without requiring any input data.
4.2 LATENT TRAJECTORIES CHARACTERIZE THE LEARNED DISTRIBUTION
In the following experiment. we test the hypothesis that trajectories informative on the distribution learned by the model, testing how well Theorem 1 holds in practice. Our goal is to evaluate whether these trajectories can be used to detect distribution shifts in the input data. To classify a sample as out-o--distribution (OOD), we focus on two key questions: (i) does the sample trajectory converge to one of the attractors of the training data, i.e., does it share the same basin of attraction? (ii) If so, how fast does it converge?
Notably, OOD sample may still lie within shared attractor basin. To capture both scenarios, track the distance from each point along test sample's 's trajectory to the nearest attractor from the training set. In the former case, we expect that an OOD sample converges faster, while in the latter case, the distance term to the attractors will dominate the score.
Setting. We test this hypothesis using the ViT-MAE (He et al., 2022) architecture pretrained on ImageNet. We sample 2000 training images and compute their attractors, stopping when convergence reaches tolerance of 10-5 or maximum number of iterations. We test on samples from SUN397 (Xiao et al., 2016), Pl 3cces336 65 (Zhou and Paffenroth, 2017), Texture (Cimpoi et al., 2014), and i Natural st (Van Horn et al., 2018), 8), standard benchmarks for OOD detection (Yang et al., 2024). We report two metrics: FPR95, and Area Under the Receiver Operating Characteristic Curve (AUROC), FPR95 here measures what percentage of OOD data we falsely classify as ID, setting threshold s.t. 95% of ID data is correctly classified.
Analysis of results. In Figure 5 we report OOD detection performance by using the follow- ing score function: for test sample Ztest compute its trajectory T [Zo. .... z towards attractors and compute the distance of Tz... to the set of training attractors Zirain Le score(z) d( ( Z.rain). ). As distance function, employ Fuclidean distance, and we aggre-
8

---

Published as a conference paper at ICLR 2026
gate the score by computing the mean distance over the trajectory score(z) Σ, d(zz, Zirain) We compare with a K-Nearest neighbor baseline (adapted from Sun et al. (2022)), where the score for a test sample is obtained by taking the mean distance over the K-NN on the training dataset, where K = 2000 in the experiments. The score proposed demonstrates how informative the latent vector field is on the training distribution. In Figure 5 the right, we show histograms of scores for the distance to attractors and the KNN, showing again that the former method is able to tell apart in-distribution and out-of-distribution data correctly.
Takeaway. Trajectories in the latent vector field characterize the source distribution and are informative to detect distribution shifts.
RELATED WORK
Memorization and generalization in neural networks (NNs). NNs exhibit a rich spectrum of behaviors between memorization and generalization, depending model capacity, regularization, and data availability (Arpit et al., 2017; Zhang et al., 2021; Power et al., 2022). In the case of extreme overparametriizatoon, namely networks trained on few data points, it has been shown experimentally in (Radhakrishnan et al., 2020; Zhang et al., 2019) and theoretically for sigmoid shallow AEs in (Jiang and Pehlevan, 2020) that AEs memorize examples and implement associative memory mechanisms. (Alain and Bengio, 2014; Vincent, 2011) 1), A similar phenomenon has been observed in diffusion models in (Somepalli al., 2023; Dar al., 2023) and analyzed in (Kadkhodaie et al., 2023) Similarly, non gradient-based approaches such as Hopfield networks and their modern variants (Hopfield, 1982; Ramsauer et al., 2020) extend classical attractor dynamics to neural systems that interpolate between memory-based and generalizing regimes. In our work, we show that AEs fall in general in the spectrum between memorization and generalization, depending on inductive biases that enforce contraction.
Contractive neural models. Different approaches have been proposed regularize NNs in order to make them smoother and more robust to input perturbations and less prone to overfitting. Many of these regularization techniques either implicitly or explicitly promote contractive mappings in AEs: for example, sparse AEs (Ng et al., 2011: Gao et al., 2024), their denoising variant (Vincent et al., 2008), and contractive AEs (Rifai et al., 2011: Alain and Bengio, 2014) losses enforce learned maps to be contractive through the loss. Regularization strategies such weight decay (Krogh and Hertz, 1991) favor as well contractive solutions for neural models, and they are incorporated in standardized optimizers such as Adam (Loshchilov, 2017). All these approaches enforce either directly or indirectly the existence of fixed points and attractors in the proposed latent vector field representation.
Neural networks as dynamical systems. Distinet lines of work have interpreted neural networks as dynamical systems. Neural ODEs (Chen et al., 2018) view depth as continuous time and model hidden states via differential equations, while deep equilibrium models (Bai al., 2019) characterize predictions as fixed points of implicit dynamics. Closer to our work, Radhakrishnan et al. (2020) in- terpret overparameterized autoencoders as dynamical systems acting the input space, implementing associative memory. In contrast, we show that any autoencoder induces latent vector field, and we link its properties to generalization, memorization, and the characterization of the learned distribution.
Nonlinear operators spectral analysis. In the context of image processing and 3D graphics, previous work has inspected generalization of spectral decompositions nonlinear operators (Bungert et al., 2021; Gilboa et al., 2016; Fumero et al., 2020), focusing one homogencous operator. In our work, fixed points of Eq. can be interpreted as the decomposition of the NN into a dictionary of signals.
CONCLUSIONS AND DISCUSSION
In this work we proposed to represent neural AES as vectors fields, implicitly defined by iterating the autoencoding map in the latent space. We showed that (i) attractors in the latent vectors field exists in practice due to inductive biases in the training regime which enforce local contractions; (ii) they retain key properties of the model and the data, linking to memorization and generalization regimes of the model; (iii) knowledge stored in the weights can be retrieved without access to input data in vision foundation models; (iv) paths in the vector field inform the learned distribution and its shifts.

---

Published as a conference paper at ICLR 2026
6.1 LIMITATIONS AND FUTURE WORKS
Generalizing to arbitrary models. Eq. 3 cannot be directly generalized to model trained with discriminative objectives such deep elassifiers, self supervised models, as the network is invertible. However we note that (i) our theory holds in general for any self map which is locally contractive (ii) the vector field still defined the output space, and can be simply obtained by computing the residual F(x) - and in the neighborhood of an attractor, the relation in proposition 3.1 can still hold for different objectives. One idea train decoder on top of the frozen encoder-only model. We give preliminary evidence of the existence of latent vector fields in self-supervised models and in next token predictors (e.g. LLMs) in Appendix B.8, showing that extending our framework to arbitrary models holds promise. An alternative intriguing idea is the one to train surrogate AE model in the latent space of the model of interest, which would be agnostic from the pretraining objective. Sparse AEs for mechanistic interpretability of large language models (LLMs) (Gao et al., 2024) in this category. Analyzing the associated latent vector field can shed light on features learned by SAEs and biases stored in their weights.
Learning dynamics. Characterizing how attractors forms during training, under which conditions noise attractors converge the training attractors, holds promise the proposed representation to study the learning dynamics of neural models to inspect finetuning of AE modules, such as low-rank adapters (Hu et al., 2022) and double descent (Nakkiran al., 2021).
Alignment of latent vector fields. Finally, following recent findings in representation align- ment (Moschella et al., 2022; Fumero al., 2024; Huh al., 2024) inspecting how latent vector fields of networks trained are related is an open question for future works, to possibly use this representations to compare different neural models.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the MUR FIS2 grant FIS-2023-00942 "NEXUS" (cup B53C250010300011, and partly by Sapienza University of Rome via the Seed of ERC grant "MINT.AI" (cup Austrian Science Fund (FWF) 10.55776/COE12. For open access purposes, the author has applied CC BY public copyright license to any author accepted manuscript version arising from this submission. MF is supported as well by the MSCA IST-Bridge fellowship which has received funding from the European Union's 's Horizon 2020 research and innovation program under the Marie Sklodowska- grant agreement No 101034413. M.F. thanks C.Domine, V.Maiorca, I.Canistraci, R.Cadei, B.Demirel, S. Vedula for insightful discussions.
research was also funded in whole or in part by the
ETHICS STATEMENT
This work is primarily methodological and theoretical, focusing the dynamics of autoencoder networks. We do not foresee any direct negative societal impacts. All datasets used are publicly available benchmark datasets (e.g.. MNIST, CIFAR, Imagenet, Laion2B) or openly released via HuggingFace (Lhoest et al., 2021). No sensitive, private, or personally identifiable information involved. Our experiments are designed for scientific analysis only, and no human subjects or protected groups are included. While the methodology in Section 4.1 provides tools to probe information about trained data autoencoders foundation models, it does not directly improve fairmess, robustness, safety. In principle, insights into memorization could be misused to develop algorithms to recover information from pretrained models unintended ways. However, our study is limited to publicly available models and datasets, and emphasize that the proposed techniques should be applied responsibly in research contexts. We hope our findings contribute to a better understanding of neural representations and their generalizaaion//emorizzz inform the design of more transparent and robust models. We provide details and hyperparameters for each experiment for reproducibility and formal statements and proof of our theoretical results in the Appendix.
properties, which may
REPRODUCIBILITY STATEMENT
All formal statements and proofs of theorems are reported in Appendix A. We implement all our experiments using PyTorch (Paszke et al., 2019) and HuggingFace libraries Lhoest Wolf et al. (2020). We experiment solely with openly available models and datasets available on HuggingFace Hub. We will open-source our codebase for all the experiments upon acceptance. Unless otherwise noted in the experiment-specific sections, all hyperparameters are listed in Section D.
(2021);
REFERENCES
Droniou Alain and Sigaud Olivier. Gated autoencoders with tied input weights. In International Conference on Machine Learning, pages 154-162. PMLR, 2013.
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data lata-generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593, 2014.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big- big-data-centric training of small language model. arXiv preprint arXiv:2502.00737, 2025.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks, In International conference on machine learning, pages 233-242.
PMLR, 2017.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances neural information processing systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2026
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning; A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence. 35(8):1798-1822 2013.
Leon Bungert, Ester Hait-Fraenkel, Nicolas Papadakis, and Guy Gilboa. Nonlinear power method for computing cigenvectors of proximal operators and neural networks. SIAM Journal on Imaging Sciences, (113::114-1148, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
Francesco D' Angelo, Maksym Andriushchenko, Aditya Vardhan Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? Advances in Neural Information Processing Systems,
2024.
Salman UI Hassan Dar, Arman Ghanaat, Jannik Kahmann, Isabelle Ayx, Theano Papavassiliu, Stefan Schoenberg. and Sandy Engelhardt. Investigating data memorization 3d latent diffusion models for medical image synthesis. In International Conference on Medical Image Computing Intervention, pages 56-65. Springer, 2023.
and Computer-Assee
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 255. leee, 2009.
Marco Fumero, Michael Möller, and Emanuele Rodolà. Nonlinear spectral geometry processing via the tv transform. ACM Transactions on Graphics (TOG), 39(6):1-16, 2020.
Marco Fumero, Marco Pegoraro, Valentino Maiorca, Francesco Locatello, and Emanuele Rodolà. Latent functional maps: a spectral framework for representation aligmment. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems, volume 37, pages 66178-666 76886603. Curran Asso- ciates, Inc., 2024. URL https: httpttts///ccceeiinnn///////// wtwp:ediiiiiisu.niiieeeeee neurips ce /paper files/papes 2024 l 7007499445444 htt.:/eee//bb1118888111999966553 Core
pdf.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.006 2020.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04099 04093, 2024.
Guy Gilboa, Michael Moeller, and Martin Burger. Nonlinear spectral analysis via one-homogeneous functionals: overview and future prospects. Journal of Mathematical Imaging and Vision, 56: 300-319, 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
12

---

Published as a conference paper at ICLR 2026
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022. 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8)22554-255 2558, 1982.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987 2024.
Yibo Jiang and Cengiz Pehlevan. Associative memory in iterated overparameterized sigmoid autoen- coders. International conference on machine learning, pages 4828-4833 PMLR, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunkkool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-58) 2021.
Zahra Kadkhodaie, Florentin Guth, Ecro P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptiv harmonic representations. arXiv preprint arXiv:2310.02557, 2023.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.000 2014.
Diederik Kingma, Max Welling, al. Auto-encoding variational bayes, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. UofT, 2009.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
Yann LeCun, Léon Bottou, Genevieve Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Majpe Majoor, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac. Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. URL https: / ac aclanthhloggy..o.ooooooooooolola org/ 2021. monnlpddenoo 21.
Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob Van de Loo, Rob Vogels, et al. 1399 h&e-stained sentinel lymph node sections breast cancer patients: the camelyon dataset. GigaScience, 7(6): giy065, 2018.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint
2017.
S.G. Mallat and Zhifeng Zhang. Matching pursuits with time fime-freuynncy dictionaries. IEEE Transac- fions Signal Processing, -113333397--341, 1993. doi: 10.
13

---

Published as a conference paper at ICLR 2026
Koichi Miyasawa et al. An empirical bayes estimator of the mean of normal population. Bull. Inst. Internat. Statist, (91181-188)11-1 1961.
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication, arXiv preprint arXiv:2209. 2544: 2022.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 2021(12):12000) 2021.
Andrew Ng al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19, 2011.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071100 193, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. URL https: arxiv org abs 1912. 01703.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02117 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterized neural networks implement associative memory. Proceedings of the National Academy of Sciences, 117 (44):27162-22177, 2020.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleimnr, Milena Pavlović, Geir Kjetil Sandve, al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02211 2020.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In Proceedings of the 28th international conference on international conference machine learning, pages 833 843-840,12 2011.
Herbert Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pages 388-394, Springer, 1992.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confee ence on computer vision and pattern recognition, pages 10684-10695, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-2522 2022.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication diffusion models. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 6048-6058, 2023.
14

---

Published as a conference paper at ICLR 2026
Yang Song and Diederik P Kingma. How to train your energy-basee based models. arXiv preprint arXiv:2101.03288, 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputting. 0668:127066 2024.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International conference on machine learming, pages 20827-20840. PMLR, 2022.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia. Basil Mustafa, et al. Siglip 2: Multilingual language encoders with improved semantic understanding, localization, and dense features. arXiv preprini 2502 14786, 2025.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- fion, 23(7):1661-1665 2011.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Associa- tion for Computational Linguistics. URL https: /ww. acl web.. org/ anthology/2022
emn 1 p-ddemo.ss 6.
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 1 119: 3-22, 2016.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv.2505.09388 2025.
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized -ittoo-diitrrbuution detection: A survey. International Journal of Computer Vision 132((2):5635-566) 2024.
Shuangfei Zhai, Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In International conference machine learning, pages 1100-1109. PMLR, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameteriizaton. arXiv preprint 190.009902.44988 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115. 2021.
Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with represen- tation autoencoders. arXiv preprint arXiv:2510. 1690, 2025.
Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 665-674, 2017.
15

---

Published as a conference paper at ICLR 2026
Rui-Jie Zhu, Tianhao Peng, Tianhao Cheng, Xingwei Qu, Jinfa Huang, Dawei Zhu, Hao Wang, Kaiwen Xue, Xuanliang Zhang, Yong Shan, et al. A survey on latent reasoning. arXiv preprint 0rXi1:2507..6203,, 2025.
16

---

Published as a conference paper at ICLR 2026
A PROOFS AND DERIVATION
A. THEOREM
We report here detailed formulation of Theorem 1 alongside its proof.
Assumption A.1 (Latent marginal), Let (x) be the data distribution and go(z) x) an encoder mapping imputs x E II" to latent codes z E Rd. The latent marginal density is defined as:
((2) f |Paata(x (x) go(z x) dx.
We assume that g(z) is continuously differentiable, and that its gradient V log q(z) and Hessian 2 log (z)) are well-defined and continuous on an open domain containing Ω2 C R2
Assumption A.2 (Fixed-pa Frixed-point manifold). Let E IR IR be an encoder and D Rd Rd decoder, both continuously differentiable. Define the composite map
f(z) = F(((z)) E Rd.
We define the fixed-point manifold f as
M = {z Rd f(z) = z].
Assumption A.3 (Local contraction). There exists an open, convex set Ω C Rd, with M C Ω and constant 0 L such that:
sup
[1]ا( S L, zE
where Jz(z)= Raxd denotes the Jacobian of and || llo denotes the spectral norm (i.e., largest singular value). Therefore, f is contraction mapping on Ω.
Assumption A.4 (Training optimality with Jacobian regularization). Let f(z) = E(D(z) be as above, with E, D trained to minimize the objective:
Excppatta
[ID(E(() x12 + ([3)(E(x)) for À > 0.
For example, assume the encoder is deterministic, i.e., 20 (2 = 8(z E(x)), so the latent marginal satisfies:
(zz) = Pduta (x) 8(2 - E(x))
Then (z) is supported and concentrated on the fixed-point manifold M, and f is locally contractive around M.
Lemma A.5 (Directional ascent of the residual field). Under the above assumptions, for every z E Q M, define the residual field:
=2)
Then the directional derivative of log in direction v(z) is strictly positive:
(V log ((2), vzz)) > 0.
Proof. Let z E Ω MM be arbitrary. Then f(z) * so v(z) f(z) 2 O. By contractiveness and training optimality, f(z) is closer to the fixed-point set M than z is. Since ((2) is concentrated on M, we have:
(((())
((z)
log (ogg(((z)) > logg(z).
We now Taylor-expand log q at z in direction v(z):
log q( (z)) = log qzz) + (V log q(z), v(z)) + R.
where R= v (2) log (Z), for some between z and fzz). Since f contractive, [v(z)] is small and R (z)]l) Therefore:
V logg v(z)) - = (Vlog v(()) >
17

---

Published as a conference paper at ICLR 2026
Theorem A.6 (Convergence to latent-space modes). Let Assumptions 1-4 hold. Let 2y Q, and define the iterative sequence:
21+1 == f(z)) E(D(z)) for all t 0.
Then:
1. The sequence {zr} converges exponentially fast to a unique fixed point z² M, satisfying [z") =z".
2. At the limit point v", we have V log q(z") =.
3. Moreover, z² is local maximum of the density q: the Hessian satisfies
q(²)) < 0,
meaning that the Hessian is negative definite.
Proof. Step 1: Contraction and convergence. By Assumption 3, / Ω - Ω is a contraction
mapping with contraction constant I. < 1. Since Ω is convex and hence complete under Banach's 's fixed-point theorem applies. Therefore:
There exists unique fixed point z' E l such that f(z") z²"
For any initial Zo E Ωt the iterates satisfy:
1; z - L' L' - 2 +0 as + Co.
Step 2:
Stationarity of the limit. At the fixed point z", we have:
v(z") f(z") - z 0.
Assume for contradiction that V log (2") 7 0. By continuity of V log q. there exists neighborhood U e zz where V log (z) stays close to V log ((²).. Then for nearby z, Lemma A.5 implies:
v log (Z), v(z)) 0.
+ 0 implies the ascent direction vanishes at z", which contradicts the assumption that log (22") O. Hence, we conclude:
But continuity of f and the residual v(z)
v log g(z") = 0.
Step 3: Local maximality. By Lemma A.5 and Assumption 4, the sequence ((z;) is strictly increasing and converges to q(z"). ). If z were a saddle point or local minimum, there would exist direction Rd such that the second-order Taylor expansion yields:
d²
log q(z tu)
u
T
2 log gozg(zz)) )u 0,
dt
t=0
which contradicts the fact that z through ascent directions. Therefore, all second directional derivatives must be negative, implying:
2 log q(z") 0.
That is, z is strict local maximum of g.
A.2 PROPOSITION 3.1
We report here detailed formulation of Proposition 3.1 alongside its proof.
Proposition A.7. Let f = F : Rd - d be the composition of an autoencoder's 's decoder D and encoder E. and define the residual vector field v(z) := f(z) Consider the reconstruction loss
L(z) lz) z||²
Then, the iteration 2,+1 = f (zz) corresponds locally to gradient descent on L (i.e., v(z) o -V LZ)() either the following conditions holds:
18

---

Published as a conference paper at ICLR 2026
1. The Jacobian Jy(z) Id (i.e., f is locally an isometry).
2. z is near attractor point z' such that f(z") zz" and J = 0.
Proof. We begin by expanding the loss:
L(z) [z z|1² = (f(z) 2. f(z) 2).
Let (() = f() - 2. Then,
L(z) = l[v(2)]??
Now compute the gradient of
V [z) V. (v(z) v(z))
(z) v(z), (v)),,
where (z(z)= = (z Id is the Jacobian the residual field.
So we obtain:
V2I(z)=2(Jj(e) Id) v(z).
Therefore, if ask whether v(z) vz),zz), we need:
v(z) x (J)(z)) Id) v(z),
which simplifies to:
(z) - Id) + ald]v(z)= = 0 for some > 0. This holds approximately in the two special cases:
Isometry: If Jy(z) Id, then (L(z) 0, v(z) is nearly stationary-i.e., we at near a local minimum.
Attractor: If ~ zz with f(z") = z and Jf (z') x 0, then:
V ~ 2v
and thus z" is a fixed point and local minimum of the loss.
In these cases cases, the dynamics of z f(z) follow the direction of steepest descent of L, up to scaling. Higher order terms dominate influence the dynamics in the general case.
A.3 ATTRACTORS CONNECTS TO GENERALIZATION ERROR
We report here a detailed formulation of Proposition 3.2 alongside its proof.
Proposition A.8 (Attractors prototypes for generalization). Let Z' c. Q C Hd be (finite) dictionary of attractors of f = in a neighborhood Q of latent space, and let II : Ω + Z denote (measurable) nearest-atrracto map [[(z) arg min ||z ull2 (with any fixed tie- breaking rule). If the decoder D LD-Lipschitz on 9, then for any test point x with z = E(x) E Ω:
(x-F(x))l - D(I(( + L. Ilz)
prototype evoor Moreover, if Z* is an E-cover of supp q (the latent marginal), then for all x with E(x) E supp4.
coverage error
|x - F(x) < x D(T(((()))1 I.
and the same bound holds in expectation over x Peest. whenever E(x) supp almost surely.
We make the following mild assumptions:
(A1) Z Ω is (finite) set of attracting fixed points of f E o D contained in an open neighborhood Ω c d
(A2) D is LD -iiLschhhtz Ω, i.e., [[D(z) D(u)ll < Lpliz u/12 for all Q. (A3) II Ω + Z* is any measurable selection nearest points, e.g. II(z) E arg m ||z u||2
19

---

Published as a conference paper at ICLR 2026
Proof of Proposition A.8. Let x be any test point with z = E(x S2 Add and subtract (II()))):
x ((E(x)) = x ((T(([(z))) D(II())) (x
prototype ervor
decoder distortion
Taking norms and using Sauchy-Schwar inequality gives:
|xx (E I |xx (II( + [D(II())) D((()))
By (A2), [[)) - (x) ) || - Lp [[[(z) z|22, which yields:
x - F 12 < ||x ((=((())) + L - 1122
This proves the first inequality. Z* an c-cover of supp q, then for any z E suppy we have |z [z) I hence
x| F(x) < |x 1 e
for all x with F(() E suppy. Taking expectations (when E(x) E supp almost surely) gives the stated bound.
Remarks. (i) The bound decomposes test error into a prototype term, ||x D(II and coverage term, 1 [l (ii) Under the assumptions of Theorem 1, Z are attractors with basins; then aligns with the destination of latent dynamics, tying the algebraic bound to the memorizzationnggeeralizzatoooooi
memorization generalization picture.
A.4 ATTRACTORS IN SIMPLE NETWORKS
In this section we characterize attractors linear and homogeneous networks, showing that this simple setting is possible to prove converge speed to attractors of the iteration in Eq 3.
Linear maps. In the linear case, the encoding map E and decoding D are parametrized by matrices W1 RN x and W2 E R*XN and the only fixed point of the map corresponds to the origin if the network is bias free or to a shift it. The rate of convergence by the spectrum of W W1, W?W,, the eigenvectors with associated eigenvalue À < 1. In case the eigenvectors of the trained AEs are aligned with the optimal solution given by the Principal Component Analysis (PCA) decomposition of the data X $ then the latent vector field vanished as the mapping is isometric and no contraction occurs.
the iteration in Eq is established and the iteration is equivalent to shrinking the input in the direction of
Homogeneous maps. In the case of homogeneous neural networks, the network satisfies F(cx). = ca F F(x) with e e IR for some a. For example, this holds for ReLU networks without biases with 1, which learn piecewise linear mapping. The input-output mapping can be rewritten as:
F(x) = Jx(x)x
A similar observation made in for denoising networks in (Kadkhodaie et al., 2023), remark here its generality.
This equality implies that we can rewrite Equation 3 as
2++1 Jy(2))z Σ A1o.2
(7)
Where E, Aids is the eigendecomposition of (z). Since in the proximity of an attractor marX(Jr(ee <= 1, the iterations shrink directions corresponding to the eigenvectors of the Jacobian by the corresponding eigenvalue. This allows us to derive the following result on the speed of convergence.
Proposition A.9 (informal, proof in Appendix A.5). The error e, |zz -z=l= converge exponentially log( T
to an e depending on the spectral norm (|J, )|,, according to the formula log Ill)
that
provides an estimate for the number of iterations T to converge to the attractor.
In Figure 1I in the Appendix, we measure how well the convergence formula predicts the measured number of iterations to converge to an attractor, on a fully non linear AE with biases, where the assumptions of the current section are less likely hold. The error in the estimate will be higher, when the initial condition 2, is far from the attractor, as higher order terms dominate the dynamics, and the first order Taylor approximation accumulates more error.
20

---

Published as a conference paper at ICLR 2026
A.5 PROOF OF PROPOSITION A.9
We report here a detailed formulation of Proposition A.9 alongside its proof.
point Proposition A.10. Let f RR - R be a differentiable function with a fixed point x E R such that f(x") z². Assume the Jacobian Jz(z²) has spectral norm [[J = p Then, for initial point Zo sufficiently close to zi, the sequence {ze} defined by the iteration Z = f(z)) satisfies:
1z - z & l|col| p²:
and the number of iterations T needed to reach an error | - 21 S is bounded by:
log(e/ //ll///llol))
log(p)
Proof. Let = z" denote the error at iteration l. We are given that is differentiable around z", and that f(z") = z². Applying the first-order Taylor expansion of f around z', we obtain:
f(z)) = f(x') ) + J (2: - 2²) + R(zz),
where R((z)) is the Taylor remainder satisfying || [[(2)]l o(l2z - 2² I]). Since f(z*) zz, this becomes:
2++1 = f(z,) = =2" + - z²) + R(zz).
Thus, the error evolves as:
e++1 24+1 2" J + R(z.).
For zy sufficiently close to z", we can neglect the higher-order term R(z;) in comparison to the leading linear term. Hence, the error evolves approximately
||c+||| [ (1= .. ||e,]| + oll
By continuity of there exists neighborhood 1/ of z" where (z)r( 5 < 1 for some
> Choosing Zo e U and letting R(z;) = (first-order approximation), the error satisfics:
|
|l S | p².
We now ask: for which T does ||rl|| S c? We solve:
||eal| pT < E.
Dividing both sides by |eol| and taking logarithms yields:
T log(p) s log
E
ep
Tea
Since log(p) < 0, reverse the inequality when dividing:
log(e/[leoll)
T
log(p)
This proves that the error converges exponentially and that the number of iterations required reach emror c is lower bounded by the expression above.
B ADDITIONAL EXPERIMENTS
B. BIAS AT INITIALIZATION
As stated in the method section of the main paper, To empirically measure to what extent model are contractive at initialization sample N 1000 samples from (0 /), and map them trough 12 different vision backbones, initialized randomly across 10 seeds. We measure the ratio between the input variance om 1 and output variance Cont and report in Figure 6.
21

---

Published as a conference paper at ICLR 2026
Figure 6: Contraction at initialization Variance preserving ratio at initialization of torchvision models:
all models considered have a ratio < 1, indicating that the map at initialization is contractive.
(a) 256 atoms
(b) 1024 atoms
(c) 2048 atoms
Figure 7: Visualization of reconstructions from the data-free sample recovery experiment of Figure 4: Visualizing on Ln.onBB reconstructions of five random samples. First row: input samples; second reconstructions from orthogonal basis; third row reconstructions from attractors of noise.
(a) 256 atoms
(b) 1024 atoms
(c) 2048 atoms
Figure 8: Visualization of reconstructions from the data-free sample recovery experiment of Figure Visualizing on Imagenet 1 reconstructions of five random samples. First row: input samples; second row: reconstructions from orthogonal basis; third row reconstructions from attractors of noise.
(a) 256 atoms
(b) 1024 atoms
(c) 2048 atoms
Figure 9: Visualization of reconstructions from the data-free sample recovery experiment of Figure Visualizing on Camelyon17 reconstructions of five random samples. First row: input samples; second row: reconstructions from orthogonal basis; third row reconstructions from attractors of noise.
22

---

Published as a conference paper at ICLR 2026
B.2 QUALITATIVE RESULTS: DATA-FREE WEIGHT PROBING
We report additional qualitative results from the data free weight probing experiment in section 4.1, in Figures 7,8,9, observing that the reconstructions from attractors functions of the number of atoms used, are superior with respect to the orthogonal basis baseline.
B.3 ADDITIONAL RESULTS ON LATENT TRAJECTORIES
In Table I extend results of the experiment in Fig. 5, adding two additional baselines as OOD detection score: Mahanabolis distance to the on in-distribution features and reconstruction error. In both cases statistics using the latent trajectories are more discriminative than using scores based on features.
SUN397
Places365
Texture
iNaturalist
Method FPR
AUROC
FPR
AUROC1
FPR
AUROC
FPR
AUROC t
KNN 100.00
42.60
100.00
32.40
34.50
89.40
86.40
68.60
Mahalanobis
82.00
58.00
88.00
45.00
43.00
90.00
31.00
87.00
Reconstruction
91.80
49.50
91.60
50.90
98.40
49.10
99.20
23.60
(AAtractors) 29.60
91.20
29.90
91.00
25.90
92.60
29.90
91.30
Table 1: FPR and AUROC scores extended: results of the experiment in Figure extended, adding reconstruction eттоr baseline and Mahalanobis distance on the features space.
B.3.1
ADDITIONAL BASELINES
In Figure 5
B.3.2 HISTOGRAMS OOD DETECTION
We report in Figure 10 additional histograms In-Distribution vs Out-Of-Dissrriution scores, corresponding to the experiment in Section 4.2.
Roa
200
20.
60
Figure 10: Histograms from OOD detections
B.4 ANALYSIS SPEED OF CONVERGENCE
In figure we report an empirical validation of proposition A.9, for 2 dimensional bottleneck convolution autoencoder trained on MNI ST, by sampling points in the training set and measuring their convergence speed in terms of number of iteration. We remark that assumptions for the proposition don't hold for this model, the farer an initial condition is from attractor the more higher order terms will dominate in the dynamics, making the evaluation of the spectrum of the Jacobian at the attractor insufficient. We observe nevertheless, that the convergence speed bounds still correlates, providing a looser estimate for the number of iteration nceded to converge towards an attractors with error
23

---

Published as a conference paper at ICLR 2026
Measured
10a
Theory
10
10-3
10
7
Figure 11: Converge speed analysis We plot the convergence speed in terms of number of iterations the error (distance to the attractors) of the initial condition on a convolutional autoencoders trained on MNIST. The estimated convergence has Pearson correlation coefficient of 0.67 to the true number of iterations to converge. Higher error in the estimates occur for farer initial conditions.
B.4.1 CONVERGENCE RATES IN PRETRAINED MODELS
In Figure 12 we plot the rate of convergence in terms of latent residuals norms of the latent trajectories in VIT MAE autoencoders from 2000 samples drawn from the Imagenet dataset. We plot (a) the latent residual norm function of the number od the iteration, showing that it approaches exponentially fast; (b) the norm of the embedding at each iteration. showing that they are bounded and the trajectory don't diverge.
Imagenst
12.00
Imagenst
3.5
11.75
2.5
11:25
11:00
W 1.5
10.75
0.5
10:25
10:00
50
150
250
300
Iterations
Rerations
(a) Latent residuals
(b) Embedding norms
Figure 12: Latent trajectories in masked autoencoders are contractive.: we plot norms of the latent residuals across iterations in ViTMAE model (a) and norms of embeddings across iterations (b).
B.5 RANK OF ATTRACTOR MATRIX
In Figure 13 we report an alternative measure of generalization of the attractors computed in the experiment in Figure 2 and in Figure 3. Specifically, we consider the matrix X D(Z)*, and we compute its singular value decomposition U* diag(s*) = X We define the generalization entropy as the number of eigenvectors needed to explain 90% of the variance of the matrix. We report this measure for both the models employed in the experiment in Figure2 and in Figure 3, showing that attractors are more expressive the model generalized better, as well during training.
B.6 MEMORIZATION OVERFITTING REGIME
In this section, we test the memorization capabilities models in different (strong) overparametriza- tion regimes, similarly as tested in (Kadkhodaie et al., 2023) for diffusion models and in (Radhakrish- nan et al., 2020; Zhang et al., 2019) for the extremely overparametrized case (network trained on few samples). We remark that this case corresponds to overfitting regime of the network, opposed
24

---

Published as a conference paper at ICLR 2026
Kank expiaal iepiaiinco verrance
MNIST FashiorMMNIST
Hark expreined varrance
Figure 13: Generalization increase ranks of attractors matrix: entropy rank as a function of the bottleneck dimension (left) and during training, as function of the number of epochs (right). Transitioning from memorization to generalization
as underfitting (over-regularized) regime showed the main paper. We train a convolutional autoen- coder subsamples of the CIFAR, MNIST, Fashi ONMNIST FnSNIO datasets, with bottleneck dimension 128 and weight decay le - 4. Figure 14 we plot the memorization coefficient as a function of the dataset size, showing that networks trained on less data( strong overparametrizatton) tend to memorize more the training data.
1.0
CIARIO
contficions
0.9
FasNiONNNIST
0.8
0.7
128
522
1624
2043
Figure 14: Strong overpuramelrization favors memorization: Memorization coefficient as function of the dataset size on CIFAR, MNIST, Fashi oNMNIST, showing that networks trained on less data (strongly overparametriiee) tend to memorize more the training data.
B.7 DATA-FREE WEIGHT PROBING: ADDITIONAL RESULTS
We replicate the experiment in 4.1, on the larger XL version of the Stable diffusion architecture (Rombach et al., 2022), observing that attractors from noise still form an informative dictionary of signals which scales better w.r.t, random orthogonal basis.
B.8 BEYOND AUTOENCODERS MODEL
In this section preliminary explore the extent to which latent vector exist beyond pretrained autoencoder models.
B.8.1 LATENT VECTOR FIELDS IN SELF SUPERVISED MODELS
We first consider the case discriminative self-supervised (SSL) models such DINOv2 Oquab et al. (2023) and SigLIP2 Tschannen et al. (2025). straightforward idea to extend our framework to encoder-only models, is the one of training decoder top of the frozen encoder model. To this end we employ the decoders trained Zheng et al. (2025), which provide pretrained decoders models for DINOv2 and IPP
25

---

Published as a conference paper at ICLR 2026
La
EuroSAT
Places365
Ortho Altrctors
Orthe Mitrctors
0.3
MSE
MSE
MSE
Ortha Attractors
0.0
2045
1211
013
e0e
193 112
6M 2041 100
Sparsity
Sparsity
Sparsity
Imagenet-1k
PatchCamelyon
Cifar100
0.4
Attractrr
MSE
MSE
MSE
0.2
Sparsity
Sparsity
Sparsity
Figure 15: Data-free weight probing Stable diffiusion XL AE the results on the larger version of the table diffusion autoencoder confirm that attractors from noise form informative dictionary of signals to reconstruct different datasets.
To assess how the latent vector field behaves, we sample 500 samples from the Imagenet 1 k training dataset, and compute 300 iterations Eq 3. We monitor (a) the norm of the residuals in the output space (i.e. reconstruction error with respect to previous iteration) (b) the norm of the residuals in the latent space (i.e. the elements vector field at current time step) (c) the average norm of the embeddings at every step. We show the results in Figure 16 observe that the reconstruction residuals and the latent space residuals both monotonically decrease at the beginning, with the former approaching O, demonstrating a contraction behavior and then stabilize. This together with the observation that the norm of the embeddings is bounded and does not increase across iterations makes us conclude that the latent vector field exists, does not diverge, and is well behaved. The latent space residuals stabilizing around iteration 150 highlight that the eigenvalues of (z,)) are approaching 1, indicating either that the iteration has reached an attractor region (as opposed to point) that the the map is still contractive but with a constant very close to 1.
0.025
24
CENOV3
24
23
013
1.5
DIVONS
1.4
12
2.00
2000
220
1.50
252
Iterations
Iberationss
1bar aoraaion
(a) Output space residuals
(b) Latent space residuals
(c) Embeddings norms
Figure 16: SSL models induce well- rell-behaved latent vector fields: field::: For latent space iterations in DINOv2 and LiP2 we plot norm of residuals in output space (a):latent residuals norms (b); and norms of embeddings across iterations (c).
B.8.2 LATENT VECTORS FIELDS IN LLMS
We now consider the case of next token predictors models, in particular case light weight large language models. In this case we consider the input-output mapping in representation space, by iterating representation computed after the embedding layer to the final layer before the softmax, therefore corresponds to the entire residual stream of the model. As models consider Qwen3-0.6 Yang et al. (2025) and Smol-LM2 Allal et al. (2025). To test whether the latent vectors field are sample 1000 sample from the Pi e dataset Gao et al. (2020) and iterate Eq. for 300 iteration
26

---

Published as a conference paper at ICLR 2026
monitoring residuals in the latent space (final layer) (a) and the norm of the final layer embeddings (b). In Figure 17 we plot the results, observing that both models show contractive behavior (a) ithout diverging, given the bounded norms (b).
Taken together these results give us preliminary evidence that latent vector fields cxist and are well defined beyond the case of autoencoder model, but stress that further analysis should be performed in order to study them, and analyze their attractor modes whenever they exists. Also the impact of different architectural elements, for example for LLMs the role of positional embeddings, (e.g. rotary position embeddings Su et al. (2024) as opposed to absolute ones) pose questions on the resulting dynamics. We believe that studying properties of the latent vector field induced by models beyond autoencoders hold promise to understand mechanistically the properties of the models, and we find promising study recent phenomenon such latent space reasoning Zhu et (2025).
Qwen T-C.T--C.000 6B
5malLM2-1175
140
Owen3 M2-1.88
60
40
80
(a) Latent space residuals
(h) Emheddings norms
Figure 17: LLM models induce well-behaved latent vector fields: For latent space iterations in Qwen3-0.6B and SmolL.M2-1.76 we plot the latent residuals norms(a); and norms embeddings across iterations (b).
B.9 QUALITATIVE VISUALIZATION OF ATTRACTORS
In this section provide more visualizations from decoded attractors other than the ones in Figure (c). In Figure 18 we report examples of decoded attractors from the latent space of the Stable Diffusion autoencoder. Attractors don't show semantically meaningful patterns but rather geometrical and texture patterms,resembere elements of a basis, similarly to the last rows of Figure 2 (c).
Figure 18: Visualization of decoded attractors from Stable Diffusion AE,
27

---

Published as a conference paper at ICLR 2026
Table 2: Architecture of the AutoencoderKL in Stable Diffusion
Details
Output Shape
Encoder
Inpur
3 256 256 image
3.x 256
216
Comv2D
3 x stride
256
256
Block
Canv2D
128 256, 3 x stride
256
128 128
ResNet Block
258 channels
256 128 128
Conv21)
256
64
Conv2
512 = 612, 3 x stride 2
612 x 52
ResNet Block
512 channels
612
Com22
512 512. stride
RosNet Block
512 chumnels
612
16
Comv21>
512 512, x 3, stride 2
6122
ResNet Block
512 chamnels
512
Log
Sampling
Regarameterization trick
x
Decoder
ResNet Block
512 channels
512 x.8
Upsample
Scale fastor 2
612
16 16
ResNet Block
512 channels
16
Upsample
ReaNet Block
512 chunnels
Upsample
Scale factor 2
812 64 64
ResNet Block
512 channels
61
Upsample
ResNet Block
258 channels
256
128
Upsample
Scale factor 2
128
236 256
ResNet Block
128 channels
256
128 3.3
256
Output: Recomstruettd Image
C REGULARIZED AUTOENCODER ENFORCE CONTRACTIVENESS
In Table 3, we list many AE variants including denoising AEs (DAEs) (Vincent al., 2008), sparse AES (SAEs) (Ng et al., 2011), variational AEs (VAFs) (Kingma et al., 2013) and other variants (Rifai al., 2011; Alain and Bengio, 2014; Gao al., 2024) and show how their objectives enforce local contractive solutions around training points.
We provide below a precise connection for the case of weight decay and feed-forward networks.
Example: weight decay Consider an L-layer feed-forward autoencoder Fg = Dg o Eg, where each layer defined h4 = de by) and the activations Φe are 1-Lipschitz (e.g., ReLU, GELU, SiLU, tanh). The overall map can be written as
Fo(x) = WLOL-IWWL-1 ... di (W1z)).
Let Dz(x) di (x))) be the diagonal Jacobian of the activation at layer f. By the chain rule,
Jo (x) = WeDD-1(sW W1
Since each activation is 1-Lipschitz, Da(()]]] < for all e, and therefore
[>o(>)l1a II ||al
A sufficient condition for local contraction is thus
II Wellle <1
[|o(x))12 <
Training with weight decay minimizes L(0) + X > [[&l and since ||We|22 & [[e]] this suppresses the product of spectral norms appearing above:
L
II W
II
[Wele.
E=1
biasing the solution toward
[=) l|2 even without explicit Jacobian penalty.
28

---

Published as a conference paper at ICLR 2026
Autoeneuder
Regularizer Rx(X)
Description
Effect on Contractiveness
Standard AE (rank S
(bottlencck dimennsio
No explicit regularization, but dimen- sionality constraint induces compres- sion
Botteneck limits rank (x))
Jacobian
rank:
Plain AE with Weight Decay
[Weli [Wall
L2 penalty encoder and decoder weights
Reduces LA WollWel by weight norms
[[,x((]]]
shrinking
Deep AE with Weight Decay
Σ, [WF 11
Weight decay across mullipli layers of deep encoode/deeoker
Layerwise
shrinkage enforces
smoother, more continctive composi- J
R- R-Contractive AE (Alain and Olivier, 2013)
Penalizes Jacobian of the full map fo De Penalizes eneoder Jacobion norm
Encourages stability of reconsirue- tions: fo(x) fo(x
Contractive AE (Ritai et ah,, 2011)
[K,, EL (GN1
Expliiily enforees local fltnesss of en- coder
Sparse AE (KI, sigmold) (Ng et al., 2011)
E Kli A (B.(x))) ()Eo(
Enforces low average activation under signoid Promotes spasity activarions
Saturated units + Jn, (X) = (x)]l small
Sparse (L1, ReLU) (Ng at. 2011)
have
tives =y sparse Jacobian Ja, (x)
Denoising AE (Vincent 2008; Alain and Bengio, 2014)
E M
Reconstructs from noisy input
For small noise: fe(x) o logp()) contractive vector Belld (x) (X) Smooths p(x), 503); penalizes sharp variations cacoder (u(x), Ju(x)
Variational
AK
KLdgv(s)x) pie)),
where
Encourages latent distribution to match
IVAEXKingma al., 2013)
q(a)x) N(plx). L(x))
prior Learns reconstract from partial input
Masked AE (MAEXHe et al., 2022)
Es [[IM @ (x Jo(M x))[[]
Promooes invariance to missing cotrics [| (x)) low
Table 3: Unified formulation of autoencoder variants as minimizing reconstruction error plus a regularizer that encourages local contractiveness.
D ADDITIONAL IMPLEMENTATION DETAILS
Table 4: Architecture of the autoencoder employed in experiments on CIFAR, MNIST, FashionMNIST datasets
Layer
Output Shape
Encoder
Input
C x image
x dx &
Conv2D
C 3 x 3, stride 2, pad
Conv2D)
d - 2d, 3 x 3, stride 2, pad
2d x 1
Cony2D
pad
Conv2D
Sd, 3 3, strid pad
8d 11th
Flatten
Sd
Bottleneck
Lincar layer projection
(lasent code)
Decoder
8d d !V
x
Conv 2DD
8d - del, 3 stride pad ousput pad
4d
Comv Transpos:2D
4d - 22,, x 3, stride 2, pad 1, ourput pad I
2d x
TransposcDD Con/Trannea TransposeDD
stride pad output pad
d - C. x 3, stride 2, pad I, output pad
C x B
Oulput: Reconstructed Image
For the experiments in Figures 2, 3, 14 we considered convolutional autoencoders with architecture specified in Table 4. We train the models with Adam (Kingma, 2014) with learning rate 5e linear step learning rate scheduler, and weight decay le - 4 for 500 epochs. For the experiements in section 4.1, we use the pretrained autoencoder of (Rombachet al., 2022). We report in Table 2 the architectural details and we refer to the paper for training details. For the experiments in section 4.2, we use the pretrained autoencoder of eet al., 2022), considering the base model. We refer to the paper for training details. For computing attractors in experiments in Figures 2,3, 14, we compute the iterations Z++1 = f(z,) until [zf())))) (z,) < le - 6 or reaching t = 3000. Similarly in experiments in Sections 1, 4.2, compute the iterations 2,+1 f(z,) until [l fz)(2))) < le 5 or reaching t = 500. All experiments are performed on a GPU NVIDIA 3080TI in Python
code, using the PyTorch library (Paszke al., 2019).
29

---

Published as a conference paper at ICLR 2026
E USE OF LARGE LANGUAGE MODELS (LLMs)
In this work, we used LLMs for occasionally polishing and improving the readability of the paper. All substantive research contributions, analysis, and interpretations were carried out by the authors.
30


Reviews:
[{"review_id": "4BjIKD7SuN", "reviewer": "Reviewer_KKEL", "summary": "This paper shows that iterating the autoencoder‑induced map $f(z)=E \\circ D(z)$ implicitly defines a vector field in latent space. It then exploits the field's dynamics and attractor structure to diagnose memorization versus generalization and to detect out‑of‑distribution (OOD) inputs.", "questions": "- **Numerical validation of Theorem 2.** Can you empirically validate Theorem 2? Such evidence would help assess the plausibility of the assumptions underlying its derivation and test the theorem's robustness in realistic settings.\n- **Iteration complexity under weaker assumptions.** Is it possible to analyze (or bound) the number of iterations required to reach a fixed point under assumptions weaker than those currently stated?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is a novel and distinctive observation that iteratively applying $f$ induces the residual vector field $V(z) = f(z)-z$, whose fixed points serve as attractors toward which nearby trajectories converge.\n- The claim that this vector field is proportional to the score of the latent prior $q(z)$ is highly intriguing; it effectively generalizes the small‑noise limit result for denoising autoencoders to the latent space.\n- Proposition 2 is particularly insightful: when training biases the model toward memorization, the prototype term approaches zero while the coverage term narrows, yielding a clear, interpretable criterion for judging memorization versus generalization from the proposed error decomposition.\n- The paper also establishes a lower bound on the number of iterations required to converge in simple linear settings, grounding the dynamics with an interpretable complexity estimate.", "weaknesses": "- The explanation for why contraction emerges *naturally*  via initialization bias, explicit regularization, and implicit regularization would benefit from a stronger theoretical foundation or, at least, a more formal set of sufficient conditions.\n- Several assumptions, e.g., smoothness of the induced latent distribution and related regularity, are stated, but the extent to which they hold for large‑scale models in practice remains unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l0RNrqAuYZ", "reviewer": "Reviewer_b56C", "summary": "The paper presents an alternative representation of autoencoder (AE) models as dynamical systems acting on the latent manifold. The authors introduce the theoretical framework required for this interpretation and establish results linking the dynamical system induced by the iterative application of an AE to the gradients of the underlying data distribution. They also theoretically characterize the attractors of the latent vector field and link them to memorization and generalization.\n\nThis framework is then used to analyze how regularization affects the phenomena of memorization and generalization in AEs, showing that as memorization decreases and generalization increases, the attractors of the AEs evolve from latent points corresponding to training examples toward more general attractors. The paper further shows that a transition from memorization to generalization occurs during training, with an increasing number of attractors being learned and the similarity of attractors found using different data converging.\n\nFinally, the authors extract AEs from common pre-trained models and show that noisy inputs can be used to find the attractors of the induced dynamical system, and that these attractors can serve as a dictionary helping reconstruct data points from diverse distributions (as compared with a random orthogonal basis). The trajectories of examples can also be used for OOD detection.", "questions": "- Theorem 1 strength and scope: The result relies on uniform contractivity and latent concentration on fixed points, which are strong assumptions rarely satisfied by general AEs. The empirical section provides motivation for approximate contractivity, but the theorem should be reframed as a local or heuristic alignment with the score, not strict proportionality.\n- Banach fixed-point misstatement: The text says well-posedness “holds iff f is Lipschitz-continuous.” Banach’s theorem requires a contraction (Lipschitz constant < 1), not mere Lipschitz continuity. Please correct.\n- Definition 3 typo: There appears to be an extra f after the Jacobian when defining the Lipschitz constant in Definition 3.\n- Detail: The numbering of the Theorems / Propositions is inconsistent across the main paper and appendix, which is confusing. Either match the numbering exactly or link the appendix theorem/proposition in the main paper.\n- Section 4.1: The claim that “OMP = PCA” when using a random orthogonal basis is incorrect. PCA involves the data-covariance eigenbasis; OMP on a random orthobasis is simply sparse projection in that basis, not PCA. Please fix the description and, ideally, compare against true PCA (top-k principal components learned from data) as a stronger baseline.\n- From the definition of the trajectory score in Sec. 3.2.2, it is not directly clear whether the distance is to all training attractors at once, and which exact point-cluster distance is used (this likely affects results since different point-cluster distances capture different notions of similarity).\n- OOD baselines are too weak. The proposed trajectory-distance score is only compared to K-NN (with K = 2000). Modern OOD detection for vision backbones includes MSP/energy scores, Mahalanobis, ViM, ODIN, KLM, etc. Adding these would materially strengthen the claim that trajectories convey additional signal beyond embeddings. Also, which neighbors are considered? Since it is trajectories that are analyzed, and each embedding along the trajectory may have different K-NNs, the reference points are moving if you recompute the K-NN for each point in the trajectory.\n- The distinction between “aggressive regularization” (1) and “over-parameterization” (2) as two forms of memorization is very interesting and would warrant further analysis. Perhaps this framework could allow the characterization of different forms of memorization in NNs. Currently, these are disjoint and hard to compare since (1) is presented in the main text as a function of k (latent-space dimension) and (2) is presented in the appendix as a function of dataset size. Unifying these observations would be valuable.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This new perspective on AEs is simple and intuitive. It is somewhat surprising that this type of analysis has not been done sooner.\n- The links to regularization, memorization, and generalization are interesting, and the proposed framework could be a useful analysis tool.\n- The theoretical framework is well presented and clear, and the theoretical results appear correct.\n- The paper is well written; the dynamical-systems terminology is clear and intuitive.\n- The experiments using AEs extracted from pre-trained models are particularly strong, without these, the toy settings described earlier would not have been sufficient.", "weaknesses": "- The scope of the paper is somewhat limited since the theory only holds for AEs.\n- While the proposed framework is well justified and interesting in its own right, its impact is difficult to gauge. There is no immediate practical impact for practitioners, nor any strikingly new finding that this framework helps uncover. However, the work has clear potential as a future analysis tool.\n\nSee the Questions section for more precise comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gz0h3WFDhI", "reviewer": "Reviewer_LZxN", "summary": "This work introduced a method to interpret autoencoder neural networks as dynamical systems defined by a latent vector field on their manifold. This vector field on their manifold is derived by iteratively applying their encoding-decoding map.\n\nThe paper claims that inductive biases introduced by standard training can be seen as emerging attractor points in their latent vector fields and propose to leverage this vector fields as representations of the neural network for downstream tasks such as (i) the analysis of the neural network with respect to generalization and memorization, (ii) the extraction of knowledge encoded in the weights of the neural network, and (iii) as tool to identify out-of-distribution samples.\n\nThe paper presents three experiments. The first experiment investigates the relationship between generalization and memorization and the role of regularization on 30 convolutional AE trained on small-scale datasets such as CIFAR10, MNIST, and FashionMNIST. The second experiment aims to investigate vision foundation models and probe the recovery of information about the data encoded in the models' weights. This is done on Stable diffusion AE and vision transformer masked AEs. The third experiment aims to demonstrate the method's expressiveness to detect distribution shifts of input data from the latent trajectories of the vector field.\n\nI like the idea and think that this paper does well in motivating the proposed methods, providing a theoretical foundation for it, and demonstrating the method's utility through the set of downstream tasks. Unfortunately, this approach is limited to encoder-decoder models, which is mentioned in the limitation section of the papers.\n\nThere is one open question that I would appreciate getting answered by the authors. There exists another method that learns a lower-dimensional manifold of neural network models using an autoencoder architecture. The embeddings on this manifold are then used for several downstream tasks, revealing the encoded information of the neural network's weights. This paper and the method I have mentioned sound similar, and I would like to make sure that they are different, as I understand it. I do provide mode details in the questions section.", "questions": "- **(Q1)**: As mentioned above, I would like to make sure I understand the presented approach properly and do not confuse it with another method. In [1], a lower-dimensional manifold of neural network weights is learned by an encoder-decoder setup using a reconstruction loss and a self-supervised loss. This encoder-decoder bottleneck is interpreted as the representation of a neural network, which itself can be used to reveal encoded information of the neural network. I think that this submission is different from the work mentioned, but I am not sure since the methods are similar in their terminology and ideas.\n\n[1] Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction\nSchuerholt et al, NeurIPS, 2021", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **(S1)**: I appreciate the paper's motivation and theoretical foundation. It provides an interesting view of (AE) neural networks and provides a novel tool for analysis.\n\n- **(S2)**: I think that the experimental section is honestly aiming to demonstrate the method's utility with respect to different downstream tasks and different datasets. I also appreciate the details and additional results listed in the appendix.", "weaknesses": "- **(W1)**: The proposed method is limited to reconstruction-based autoencoder neural networks. The authors are aware of this as they do mention this in the imitation section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YKkrMwaasu", "reviewer": "Reviewer_XuAT", "summary": "This paper proposes viewing autoencoder models as latent dynamical systems, where iterating the mapping $f=E\\circ D$ defines a latent vector field and reveals attractors that capture the model’s memorization and generalization behavior.\n\nThe authors connect the local contractivity of this mapping to the emergence of attractors and use them for practical analyses such as (1) distinguishing memorization vs. generalization regimes, and (2) performing data-free probing and out-of-distribution (OOD) detection by analyzing trajectories toward these attractors.\n\nIn addition to theoretical connections and remarks, empirical results are shown using autoencoders, a diffusion-model autoencoder, and a large-scale vision model utilizing masked autoencoders.", "questions": "- For a pretrained model, is it possible to generate attractors from random noise and gain intuition on whether the model is operating in the generalization or memorization regime?\n- Do the reconstructions of attractors for the pretrained models carry interpretable or semantically meaningful information?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of treating $E\\circ D$ as a dynamical system in latent space is novel and intuitive, providing a unifying perspective on autoencoders.\n- The framework is validated across various architectures, including autoencoders, a pretrained diffusion AE, and a large-scale vision model, showing that attractors can indeed be identified even in large-scale, complex models.\n- Using attractors derived from noise, without requiring access to source training data, to reconstruct meaningful images is an interesting way to explore what information the model stores in its weights, opening up interpretability and compression directions.\n- The paper provides clear and well-motivated definitions of concepts such as contractivity and attractors, and links them intuitively to properties of the model’s Jacobian, which helps make the overall framework more interpretable and understandable.", "weaknesses": "- As the authors acknowledged in the discussion, it remains uncertain whether the approach applies to widely used forecasting or next-token prediction models, or encoder-only architectures. Since such models dominate modern representation learning, discussing whether attractors exist or can be defined meaningfully in these settings would strengthen the impact. To examine this, would training a lightweight decoder on top of a frozen encoder (without modifying the encoder weights) help reveal similar attractor dynamics?\nThis could clarify whether the attractor framework extends beyond autencoder-based models.\n- While the authors show that attractors inferred from noise can reconstruct the actual inputs, it is not entirely clear whether these attractors correspond to actual training examples, or how one can infer generalization capacity without explicitly comparing attractors to the training data.\n- The theoretical analysis assumes local contractivity, which potentially does not hold globally. Empirically, as long as stable attractors can be identified, the proposed approach appears to remain valid. Nevertheless, it would be good to quantify how much of the latent space exhibits convergent dynamics, characterize the stability of these attractors, and report how long or how many iterations are typically required to discover them beyond the MNIST example. \n- Is the KNN analysis performed using latent embeddings of training data or the attractors identified by training data? How sensitive are the KNN results in Figure 5a to the choice of the number of neighbors K? Would similar results hold for smaller values of K or with adaptive neighborhood sizes? Also, for the proposed attractor trajectory-based scoring, is the distance computed with the nearest training attractor's trajectory or averaged across all training attractors?\n- Following up on the previous item, how does the proposed attractor trajectory-based scoring compare with other standard OOD detection metrics, such as the Mahalanobis distance in latent space or the reconstruction loss (MSE) in input space?\n- Similarly, would OOD detection performance remain similar if attractors were computed from Gaussian noise?\n- Please provide the definition of FPR95 where it first appears. Also, the definition in L415 can be improved by stating that it uses a threshold such that 95% of ID samples are correctly classified.\n- The caption of Figure 5 needs improvement; it is currently unclear which histogram corresponds to which method. Similar clarification should be added for Figure 2 for the attractor reconstructions, by specifying whether those are latent attractors or decoded outputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)