# Deep Neural Networks as Iterated Function Systems and a Generalization Bound

Jonathan Vacher

► To cite this version:

Jonathan Vacher. Deep Neural Networks as Iterated Function Systems and a Generalization Bound.
2026. hal-05475862

HAL Id: hal-05475862

https://hal.science/hal-05475862v1

Preprint submitted on 26 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Deep Neural Networks as Iterated Function Systems and a Generalization Bound

Jonathan Vacher*<fnref n="1" />

## Abstract

Deep neural networks (DNNs) achieve remarkable performance on a wide range of tasks, yet their mathematical analysis remains fragmented: stability and generalization are typically studied in disparate frameworks and on a case-by-case basis. Architecturally, DNNs rely on the recursive application of parametrized functions, a mechanism that can be unstable and difficult to train, making stability a primary concern. Even when training succeeds, there are few rigorous results on how well such models generalize beyond the observed data, especially in the generative setting. In this work, we leverage the theory of stochastic Iterated Function Systems (IFS) and show that two important deep architectures can be viewed as, or canonically associated with, place-dependent IFS. This connection allows us to import results from random dynamical systems to (i) establish the existence and uniqueness of invariant measures under suitable contractivity assumptions, and (ii) derive a Wasserstein generalization bound for generative modeling. The bound naturally leads to a new training objective that directly controls the collage-type approximation error between the data distribution and its image under the learned transfer operator. We illustrate the theory on a controlled 2D example and empirically evaluate the proposed objective on standard image datasets (MNIST, CelebA, CIFAR-10).

## 1 Introduction

Deep neural networks (DNNs) are now the default modeling tool for high-dimensional prediction and generation, yet their theoretical understanding remains fragmented. Many successful architectures, e.g. ResNets, Transformers, and Mixture-of-Experts (MoE) layers, are built from the repeated composition of parametrized maps, sometimes with

stochastic routing. This recursive structure naturally invites a dynamical-systems viewpoint, but most existing analyses either linearize locally, pass to continuous-time limits, or focus on optimization dynamics (e.g. stochastic gradient descent) rather than on the action of the trained network itself. As a result, we still lack a unified framework that simultaneously (i) captures the depth-wise dynamics of modern architectures, (ii) yields stability and convergence guarantees, and (iii) provides task-relevant generalization bounds, in particular for generative models.

**Random Dynamical Systems** Iterated function systems (IFSs) offer a compact language for contractive dynamical systems whose attractors are fractal sets or invariant measures. Deterministic IFSs (Barnsley & Hutchinson, 1989) yield existence and uniqueness of attractors via the Banach fixed point theorem and the associated “collage” principle, while stochastic variants e.g. independent IFS, place-dependent IFS, and random matrix products extend these guarantees to Markovian compositions of maps (Barnsley et al., 1988; Diaconis & Freedman, 1999; Stenflo, 2003). At the level of measures, contraction in Wasserstein distance leads to invariant laws and quantitative control of approximation error. Early neural-network work already hinted at this connection (Stark, 1991; Bressloff & Stark, 1991), but the IFS viewpoint has remained largely underexploited in the context of modern deep architectures and large-scale generative modeling. We note two exceptions: one in which IFSs are used to analyze the dynamic of stochastic gradient descent (Camuto et al., 2021); a second in which “collage” is used as an objective (Poli et al., 2022) without leveraging the stochastic IFS theory.

**Deep Learning and Generative Models** Modern deep generative models are dominated by deep architectures: diffusion and score-based models rely on iterated denoising networks (Song et al., 2021); GANs use deep discriminators and generators trained in adversarial games (Goodfellow et al., 2014); VAEs and normalizing flows combine deep encoders/decoders or invertible blocks (Papamakarios et al., 2021); and Transformers underpin autoregressive and masked-generation models (Vaswani et al., 2017). Despite impressive empirical performance, these families typically offer only partial guarantees: diffusion models trade sample quality for slow iterative sampling (Karras et al., 2022); GANs can be unstable and mode-collapse-

<footnote n="1">¹Université Paris Cité, CNRS, MAP5, F-75006 Paris, France. Correspondence to: Jonathan Vacher <jonathan.vacher@u-paris.fr>.</footnote>
prone (Cobbinah et al., 2025); VAEs blur high frequencies (Bredell et al., 2023); normalizing flows restrict expressivity for tractable Jacobians (Papamakarios et al., 2021); and Transformers, even when their depth dynamics have been analyzed (Geshkovski et al., 2023a,b), come with no general guarantee that the empirical training distribution is close to an invariant attractor of the learned dynamics. More generally, standard architectural choices (residual connections, self-attention, normalization layers, drop-out etc.) are still largely guided by empirical observations rather than explicit stability criteria, which leads to ad-hoc design “recipes”. In contrast, several works show that explicit control of the Lipschitz constant or spectral norm is crucial for training stability (Miyato et al., 2018; Delattre et al., 2023).

In this work, we revisit these questions through the lens of stochastic IFSs and random dynamical systems. At a qualitative level, this viewpoint turns the depth evolution of a network into a random dynamical system acting on measures, so that stability becomes contractivity of an associated transfer operator and generalization in generative modeling becomes the proximity between its invariant law and the data distribution. This perspective suggests a collage-type training principle: instead of matching a generic normal distribution to the empirical distribution using deep architecture (GANs, VAEs, normalizing flows, diffusion models/flow matching), one directly controls how far a single application of the learned operator moves the empirical distribution.

**Contributions.** Our contributions are threefold: (i) We provide a unified stochastic-IFS interpretation of several standard deep architectures by making explicit the branch maps and selector kernels of ResNets, Transformers, and Mixture-of-Experts layers, and we identify conditions under which their depth dynamics admit unique invariant measures and well-defined attractors. (ii) We derive a generalization bound for generative modeling in Wasserstein distance, showing that, under a contractivity assumption on the transfer operator, the distance between the model's invariant law and the data distribution is controlled by an empirical “collage error” plus a purely statistical sampling term; this bound directly motivates a new training objective that penalizes the discrepancy between the empirical measure and its image under the learned operator. (iii) We instantiate this objective with spectrally-normalized MoE/IFS models, approximate the collage error with regularized optimal transport (Peyré et al., 2019), and show on both a controlled 2D example and on latent representations of MNIST, CelebA, and CIFAR-10 that minimizing the collage error yields meaningful attractors and promising qualitative results without diffusion steps or adversarial training<fnref n="1" />.

**Notation.** Throughout the manuscript, $(\mathcal{X}, d)$ denotes a complete separable metric space with Borel $\sigma$-algebra and

<figure><img src="image_2.png" /></figure>

Hutchinson iterates : $T_{n+1} = \mathcal{H}(T_n)$

Attractor set : Sierpiński triangle

<figure><img src="image_3.png" /></figure>

For all source patch $i$, find $(j_i, k_i)$ such that $\mathcal{H}(I) \approx I$.

Fixed point of $\mathcal{H}$ for IFS $W = \{\tau_{i \to j_i} \circ w_{k_i}\}_{i}$ is close to $I$.

<figure><img src="image_4.png" /><figcaption>**Figure 1.** Schematic illustration of fractal generation with IFSs. Top: classical fractal attractor construction. Bottom: fractal encoding of an image, where contractive maps learned from patch correspondences define an IFS whose invariant set approximates the target image.</figcaption></figure>

$\mathcal{P}(\mathcal{X})$ is the set of probability measures on $\mathcal{X}$ with second-order finite moments. The $K$-simplex is $\Delta^{K-1}$ and for any $p \in \Delta^{K-1}$, $\mathcal{C}(p)$ is the categorical distribution. Finally, we denote $d_H$ the Hausdorff distance and $W_2$ the 2-Wasserstein distance. Definitions and proofs are given in supplementary Section A.

## Preliminary Result

A linear operator $T: \mathcal{X} \to \mathcal{X}$ is $c$-contractive when its Lipschitz constant $c$ belong to $[0, 1[$. In this case, the Banach fixed point theorem applies (see supplementary Section A). A key corollary states that if one has a vector that is $\varepsilon$-invariant by $T$ then this vector is close enough to the fixed point of $T$. This formalizes as it follows.

**Corollary 1** (“Collage” in Metric Space). Let $T: \mathcal{X} \to \mathcal{X}$ a contraction with constant $c \in [0, 1]$. Let $\tilde{x} \in \mathcal{X}$ be the (unique) fixed point of $T$, i.e. $T\tilde{x} = \tilde{x}$.

<footnote n="1">¹See online code: https://github.com/xxx/xxx</footnote>
Then, for any $y \in \mathcal{X}$, we have the following implication

$$d(Ty, y) \le \varepsilon \implies d(\tilde{x}, y) \le \frac{\varepsilon}{1-c}. \quad (1)$$

This result will be used along this manuscript for different metric spaces. It is the key result of fractal image compression (Barnsley et al., 1996).

# 2 Iterated Function Systems

## 2.1 Deterministic IFS

**Definition 1 (IFS).** Let $\mathcal{I} = \{1, \dots, m\}$ be a finite index set and $\mathcal{W} = \{w_\xi : \mathcal{X} \to \mathcal{X}\}_{\xi \in \mathcal{I}}$ a finite family of functions. We call $\mathcal{W}$ an iterated function system (IFS). We say that an IFS is c-contractive when all its functions are at most c-contractive.

Equip $\mathcal{K}(\mathcal{X})$, the set of nonempty compact subsets of $\mathcal{X}$, with the Hausdorff distance $d_H$. The space $(\mathcal{K}(\mathcal{X}), d_H)$ is a complete metric space (Henrikson, 1999). Define the Hutchinson operator

$$\mathcal{H}(K) = \bigcup_{\xi \in \mathcal{I}} w_{\xi}(K).$$

**Theorem 1 (Original Collage Theorem).** Let $\mathcal{W}$ be a c-contractive IFS. Then $\mathcal{H}$ is c-contractive on $(\mathcal{K}(\mathcal{X}), d_H)$ and Corollary 1 applies.

Theorem 1 is the original “collage” theorem. It states that if a compact set $K$ is $\varepsilon$-invariant $(d_H(K, \mathcal{H}(K)) \le \varepsilon)$, then it is close enough to the fixed point $\tilde{K}$ of the Hutchinson operator $(d_H(K, \tilde{K}) \le \varepsilon/(1-c))$. The fixed point can be recovered using the recursive iteration of the operator. Classical applications of this result include line contractions, the Sierpiński gasket (Figure 1-top), and fractal image compression where the maps encode coarse-to-fine patch mappings (Barnsley et al., 1996) (Figure 1-bottom). More than that, IFSs are widely studied in the field of random dynamical systems.

## 2.2 Stochastic IFSs

Stochastic IFSs generalize the deterministic application of an operator depending on a family of functions by defining a recursion with a Markovian function selection. The most general type of stochastic IFSs is when the probability of choosing one function depends on the previous iterate. This defines place-dependent IFSs. A simpler case of stochastic IFS is when the probability of choosing one function does not depend on the previous iterate. This defines independent IFSs.

**Definition 2 (Stochastic IFS).** Let $\mathcal{W} = \{w_\xi : \mathcal{X} \to \mathcal{X}\}_{\xi \in \mathcal{I}}$ be an IFS and $p : \mathcal{X} \to \Delta^{|\mathcal{I}|-1}$ a selector kernel. Then $(\mathcal{W}, p)$ is a place-dependent IFS (P-IFS) iff its Markov

recursion is

$$X_{t+1} = w_{\Xi_t}(X_t), \quad \Xi_t | X_t = x_t \sim \mathcal{C}(p(x_t)).$$

When the selector kernel $p$ does not depend on $x$ i.e. for all $x \in \mathcal{X}$,

$$p(x) = q \in \Delta^{|\mathcal{I}|-1},$$

the pair $(\mathcal{W}, q)$ is called an Independent IFS (I-IFS).

Both I- and P-IFSs induce a Markov operator on bounded functions and an adjoint operator on measures. These operators allow to make the connection with the Hutchinson operator defined in Section 2.1.

**Definition 3 (Markov operator, transfer operator and invariant measure).** For $f \in C_b(\mathcal{X})$ the Markov operator is

$$\begin{aligned} (Tf)(x) &= \mathbb{E}_{\Xi|X=x \sim \mathcal{C}(p(x))}[f(w_{\Xi}(x))] \\ &= \sum_{\xi \in \mathcal{I}} p_{\xi}(x) f(w_{\xi}(x)). \end{aligned}$$

Its adjoint, the transfer operator is $T^*$ acting on $\mathcal{P}(\mathcal{X})$, satisfies

$$\begin{aligned} T^* \mu &= \mathbb{E}_{X \sim \mu, \Xi|X \sim \mathcal{C}(p(X))}[\delta_{w_{\Xi}(X)}] \\ &= \sum_{\xi \in \mathcal{I}} (w_{\xi}) \#(p_{\xi} \cdot \mu), \end{aligned}$$

where $(w_\xi) \# \mu$ is the pushforward of $\mu$ by $w_\xi$ and $p_\xi \cdot \mu$ has density $p_\xi$ against $\mu$. An invariant measure satisfies $T^* \tilde{\mu} = \tilde{\mu}$.

Intuitively $T^* \mu$ represents the distribution of the IFS output after one step when the input is drawn from $\mu$.

An important question related to stochastic IFSs is whether their transfer operator admit an invariant measure. Several theorems exist for I-IFS and P-IFS (Barnsley et al., 1988; Stenflo, 2002; 2003). The hypotheses of these theorems are generally based on the contractive nature of the functions as presented in the theorem below.

**Theorem 2 (Invariant measure for P-IFS).** Let $(\mathcal{W}, p)$ be a P-IFS whose functions $(w_\xi)$ are Lipschitz with constants $(c_\xi)$. If $\sup_x \sum_{\xi \in \mathcal{I}} p_\xi(x) c_\xi^2 < 1$ (average contraction), then there exists a unique invariant measure $\tilde{\mu}$ for $T^*$, and the associated Markov recursion converges in law to $\tilde{\mu}$

As mentioned earlier, the invariant measure of the transfer operator can be related to the fixed point of the Hutchinson operator as being its support.

**Proposition 1 (Support of invariant measure and Hutchinson fixed-point).** Let $(\mathcal{W}, q)$ be a c-contractive I-IFS with all $q_\xi > 0$. Let $\tilde{\mu}$ be the (unique) invariant measure of the transfer operator $T^*$ and let $\tilde{K}$ be the fixed point of the Hutchinson set operator $\mathcal{H}$. Then, $\tilde{K} = \text{supp}(\tilde{\mu})$.
Finally, in the context of measures we also have collage theorems. For I-IFS, contraction of the branch functions yields contraction of the transfer operator in Wasserstein distance (Dionakis & Freedman, 1999; Villani et al., 2008).

**Theorem 3 (Collage Theorem on measure).** Let $(W, q)$ be a c-contractive I-IFS. Then, its transfer operator $T^*$ is contractive on $(P(\mathcal{X}), W_2)$ and Corollary 1 applies.

For general P-IFS, contractivity of each branch does not suffice to guarantee Wasserstein contraction of the operator. A Lipschitz condition on the selector kernel itself is required (Meyn & Tweedie, 2012; Villani et al., 2008).

**Consequence for generative modeling** This measure collage theorem has an important consequence for generative modeling. Given an empirical measure $\mu$, it means that if we are able to find a contractive transfer operator $T^*$ (or its corresponding stochastic IFS) such that $W_2(T^*\mu, \mu) \le \epsilon$ then the attractor (or fixed-point) $\tilde{\mu}$ of $T^*$ will be close enough to $\mu$. We derive a bound on the generalization error in Section 4.

The deterministic Hutchinson operator, I-IFS and P-IFS are all versions of the same thing : the first applies all branches in parallel, the second averages over a fixed categorical draw, and the third selects branches in a state-dependent manner. Average contraction recovers the Banach fixed point intuition in each case. In the following section, we formalize how ResNet and Transformer architectures together with MoE fit in the IFS framework.

# 3 Deep Neural Networks as IFS

Each subsection below states a formal definition of the considered architecture, a formal definition of the function set and selector kernel pair $(W, p)$ and a proposition establishing that it is a well-defined IFS. In this section, we assume that $\mathcal{X} = \mathbb{R}^d$.

## 3.1 ResNet as (degenerate) P-IFS

First, we consider the case of ResNet with ReLU activation function composed of the same residual block.

**Definition 4 (ReLU residual block).** Let $m \in \mathbb{N}$ be the width of a hidden layer, and let $A \in \mathbb{R}^{m \times d}$, $b \in \mathbb{R}^m$, $B \in \mathbb{R}^{d \times m}$, $c \in \mathbb{R}^d$ be parameters. With ReLU activation $\sigma(z) = \max\{z, 0\}$ applied coordinate-wise, the residual block is the function $F_\theta: \mathcal{X} \to \mathcal{X}$ defined for all $x \in \mathcal{X}$ by

$$F_{\theta}(x) = x + B \sigma(Ax + b) + c$$

where $\theta = (A, B, b, c)$.

Iterations across depth are denoted $x^{(t+1)} = F_\theta(x^{(t)})$. Previous works have established this recursion as the Euler discretization of a so-called Neural ODE (Chen et al., 2018) which allows to study theoretically the limit of infinitely

deep neural networks using dynamical system theory. Here we keep the iterations discrete in depth. It is possible to consider a most general case where the parameter $\theta$ depends on depth but we prefer to keep it constant for simplicity. Indeed, considering the time dependence of $\theta$ only drastically increases the set of possible functions which stays finite in finite discrete time. In the following definition, given a ResNet we define an associated set of functions and a selector kernel.

**Definition 5 (Residual branch family and selector kernel).** Let $\mathcal{I} = \{0, 1\}^m$ index activation patterns of the hidden layer. For $\xi = (\xi_1, \dots, \xi_m) \in \mathcal{I}$ define the diagonal matrix $D(\xi) = \text{diag}(\xi)$ and the branch map $w_\xi^\theta: \mathcal{X} \to \mathcal{X}$ by

$$w_{\xi}^{\theta}(x) = x + B D(\xi)(Ax + b) + c.$$

Set $W_{\text{ResNet}} = \{w_\xi^\theta\}_{\xi \in \mathcal{I}}$. For $x \in \mathcal{X}$, let $\xi(x) \in \mathcal{I}$ be the indicator of positive pre-activations, for all $j \in \{1, \dots, m\}$

$$\xi_j(x) = 1\{(Ax + b)_j > 0\}.$$

We define the (degenerate) selector $p_{\text{ResNet}}: x \in \mathcal{X} \mapsto p(x) \in \Delta^{|\mathcal{I}|-1}$ by

$$p_{\xi}(x) = \delta_{\xi(x)}(\xi) = \begin{cases} 1, & \text{if } \xi = \xi(x), \\ 0, & \text{otherwise.} \end{cases}$$

**Proposition 2 (ReLU residual block as place-dependent IFS).** The pair $(W_{\text{ResNet}}, p_{\text{ResNet}})$ defines a P-IFS on $\mathcal{X}$. The deterministic residual map $F_\theta$ is the following conditional expectation

$$F_{\theta}(x) = \mathbb{E}_{\Xi|X=x} \mathcal{C}(p_{\text{ResNet}}(x)) [w_{\Xi}^{\theta}(x)].$$

The above construction is not limited to ReLU activation, see supplementary Section B for a construction with softplus activation. Given an activation function, it is in fact often possible to build an IFS and there are often multiple possible constructions.

The selector kernel $p_{\text{ResNet}}$ depends on $x$ only through the sign pattern of $Ax + b$, that is, through the ReLU activation pattern at the first linear layer. Each pattern $\xi$ selects a subset of the columns of the subsequent linear map $B$ and therefore determines a specific affine map $w_\xi$ at each layer. In fact, the selection is deterministic, so that a ReLU residual block corresponds to a degenerate (Dirac) selector kernel. This activation-pattern viewpoint has been used by Mohan et al. (2020) to show that a bias-free ReLU network is a positively homogeneous linear function of order 1. However, no further dynamical analysis was developed.

Within the present IFS framework we can go further. A ReLU residual block is a piecewise-affine P-IFS whose stability is governed by the average contraction Theorem 2.
Hence, imposing average contractivity guarantees stability at infinite depth. When such networks are trained for denoising, clean images are approximate fixed points; hence the empirical distribution of the training set is approximately invariant under the associated transfer operator. Since each branch map is affine (or linear in the bias-free case), the asymptotic behavior of the network as depth tends to infinity can be analyzed using the classical theory of products of random matrices (Furstenberg & Kesten, 1960; Bougerol & Lacroix, 1985) and affine IFS (Barnsley et al., 1988). In the linear (positively homogeneous) case the attractor of the induced dynamics is constrained to be a single point, a cone, or a finite union of cones, in agreement with the observations of Mohan et al. (2020). In the affine case the attractor may be a nontrivial fractal set, which is fully compatible with the empirically observed tendency of biased networks to encode the scale of the noise level present during training.

## 3.2 Transformers as P-IFS

Second, we consider the case of the Transformer architecture. Transformers depth dynamics have been recently studied and our interpretation corroborates previous findings about convergence towards degenerate measures (Geshkovski et al., 2023a;b). Yet, our approach slightly contrasts as it is valid on the context space $\mathcal{X}^n$, not the token space $\mathcal{X}$. The case of Transformers is also more interesting than the ResNet as the selector kernel is not degenerated and is explicitly related to the attention weights.

**Definition 6** (Transformer block). Let $n$ be the context length and $\mathbf{x} = (x_1, \dots, x_n) \in \mathcal{X}^n$ a context. Let $\ell_{\text{norm}}$ denote layer normalization. Queries, keys and values are

$$q_i(\mathbf{x}) = W_q x_i, \quad k_j(\mathbf{x}) = W_k x_j, \quad v_j(\mathbf{x}) = W_v x_j,$$

where $(W_q, W_k, W_v)$ are matrices. Attention weights are

$$\alpha_{ij}(\mathbf{x}) = \frac{\exp(q_i(\mathbf{x})^{\top} k_j(\mathbf{x}))}{\sum_{\ell=1}^{n} \exp(q_i(\mathbf{x})^{\top} k_{\ell}(\mathbf{x}))}.$$

The attention map is

$$f_{\theta}^{\text{SA}}(\mathbf{x})_i = \sum_{j=1}^{n} \alpha_{ij}(\mathbf{x}) v_j(\mathbf{x}),$$

and a two-layer MLP is

$$g_{\theta}(\mathbf{x}) = W_2 \sigma(W_1 \mathbf{x} + b_1) + b_2,$$

with activation $\sigma$ (e.g. ReLU GELU or tanh) applied coordinate-wise. The residual transformer block is the function $G_{\theta}: \mathcal{X}^n \to \mathcal{X}^n$ defined by for all $\mathbf{x} \in \mathcal{X}^n$ by

$$G_{\theta}(\mathbf{x}) = \mathbf{x} + f_{\theta}^{\text{SA}}(\ell_{\text{norm}}(\mathbf{x})) + g_{\theta}(\ell_{\text{norm}}(\mathbf{x})). \quad (2)$$

In practice, there are additional affine embedding and projector matrices / vectors $(W_{\text{in}}, b_{\text{in}})$ and $(W_{\text{out}}, b_{\text{out}})$, though

we omit these as they are factoring other parameters. Similarly, we do not define Multi-Head attention as it only consists in concatenating multiple single head and projecting back to the desired output space.

Again, iterations across depth on the context space $\mathcal{X}^n$ are denoted $\mathbf{x}^{(t+1)} = G_{\theta}(\mathbf{x}^{(t)})$, we do not need to consider the continuous limit as it usually done to analyze the behavior at infinite depth (Geshkovski et al., 2023b). We can now define the set of functions and the selector kernel to establish the link with IFS.

**Definition 7** (Transformer branch family and selector kernel). Let $\mathcal{I} = \{1, \dots, n\}^n$. For $\xi = (\xi_1, \dots, \xi_n) \in \mathcal{I}$ define

$$(w_{\xi}^{\theta}(\mathbf{x}))_i = x_i + v_{\xi_i}(\ell_{\text{norm}}(\mathbf{x})) + g_{\theta}(\ell_{\text{norm}}(\mathbf{x}))_i, \quad i \in \{1, \dots, n\},$$

and set $W_{\text{TF}} = \{w_{\xi}^{\theta}\}_{\xi \in \mathcal{I}}$. We define the selector kernel $p_{\text{TF}}: \mathbf{x} \in \mathcal{X}^n \mapsto p(\mathbf{x}) \in \Delta^{|\mathcal{I}|-1}$ by

$$p_{\xi}(\mathbf{x}) = \prod_{i=1}^{n} \alpha_{i, \xi_i}(\mathbf{x}),$$

with attention weights $\alpha_{i,j}(\mathbf{x})$ from Definition 6.

The following proposition establishes definition 7 as a well-defined P-IFS.

**Proposition 3** (Transformer as place-dependent random IFS). The pair $(W_{\text{TF}}, p_{\text{TF}})$ defines a P-IFS on $\mathcal{X}^n$. The deterministic Transformer block is the following conditional expectation

$$G_{\theta}(\mathbf{x}) = \mathbb{E}_{\Xi|\mathbf{X}=\mathbf{x} \sim C(p_{\text{TF}}(\mathbf{x}))} [w_{\Xi}^{\theta}(\mathbf{x})].$$

Basically, this proposition says that a Transformer block can be viewed as randomly picking an attention pattern $\Xi$ according to the softmax weights and applying the corresponding update. In expectation this recovers the usual transformer output.

Contrary to the ResNet, the Transformer is not a degenerate IFS. In fact, the transformer block $G_{\theta}$ is only canonically associated to a distinct P-IFS as its conditional expectation. The associated P-IFS has a selector kernel defined by attention weights and non-linear branch functions $w_{\xi}^{\theta}$. The two-layer MLP $g_{\theta}$ is shared across all functions and only the linear part is selected $(v_{\xi_i})$. When the MLP is removed ($g_{\theta} = 0$), the family $\{w_{\xi}^{\theta}\}$ reduces to an affine IFS and its asymptotic behavior can be characterized by the classical theory of products of random matrices (Bougerol & Lacroix, 1985) and affine iterated function systems (Barnsley et al., 1988). In this linear or affine setting, the attractor of the associated P-IFS is constrained to be a single point, a cone, a finite union of cones, or a fractal affine set.
<figure><img src="image_5.png" /><figcaption>Figure 2. Left: Training collage error $\epsilon(n, \theta)$ over epochs. Center: Training data and samples of the attractor measure of the trained MoE IFS. Right: Estimated generalization error and its bound (Theorem 4) for different contraction constants.</figcaption></figure>

**Practical precedents of IFS-like sampling.** Stochastic or hard routing (i.e. branch sampling) is already used to approximate dense attention: mixture-of-experts with sampled top-k gating (Shazeer et al., 2017; Lepikhin et al., 2021; Fendus et al., 2022), content-based sparse routing of keys (Child et al., 2019; Roy et al., 2021), hash-based neighbor sampling (Kitaev et al., 2020), and randomized feature estimators of softmax attention (Choromanski et al., 2021). These variants replace, conditionally on the input, the deterministic convex combination by sampled maps, matching the IFS formalism above.

Other deep learning architectures can be viewed as IFS and it might be interesting to complete an exhaustive classification. Though, now we would like to highlight that MoE can be canonically associated to an IFS.

### 3.3 Mixture-of-Experts as P-IFS

To complete we consider the case of MoE which have a very generic formulation. Using the right notations from the beginning will simplify the analysis.

**Definition 8 (Mixture-of-Experts).** Let $\mathcal{I} = \{1, \dots, K\}$ be an index set, and $\Theta$ a parameter space. Let $\theta \in \Theta$ and $W_{MoE} = \{w_{\xi}^{\theta}\}_{\xi \in \mathcal{I}}$ be expert functions on $\mathcal{X}$. Let $p_{MoE} = p^{\theta}: \mathcal{X} \to \Delta^{K-1}$ be a gating network. For all $x \in \mathcal{X}$, the dense MoE function is

$$H_{\theta}(x) = \sum_{\xi \in \mathcal{I}} p_{\xi}^{\theta}(x) w_{\xi}^{\theta}(x).$$

At this stage, it is clear that we have directly defined the expectation of some P-IFS. This is made explicit in the following proposition.

**Proposition 4 (MoE as place-dependent IFS).** The pair $(W_{MoE}, p_{MoE})$ defines a P-IFS on $\mathcal{X}$ with recursion

$$x_{t+1} = w_{\Xi_t}^{\theta}(x_t), \quad \Xi_t | X = x_t \sim C(p^{\theta}(x_t)).$$

Moreover, the dense MoE output is the conditional expectation

$$H_{\theta}(x) = \mathbb{E}_{\Xi|X=x \sim C(p^{\theta}(x))}[w_{\Xi}(x)].$$

We can cascade MoE with different parameters at each layer as defined in the supplementary Section D.

Supplementary Table 1 summarizes the IFS associated to ReLU-ResNet, Transformers and MoE.

### 3.4 Consequences

**Convergence and Stability** Viewing ResNets, Transformers, and MoE layers as stochastic IFS enables the import of classical convergence and stability results: existence and uniqueness of invariant measures and characterization of attractors. For a P-IFS, the stochastic recursion $X_{t+1} = w_{\Xi_t}(X_t)$ and the deterministic update $x_{t+1} = F(x_t)$ with the conditional expected function $F(x) = \mathbb{E}_{\Xi|X=x \sim C(p(x))}[w_{\Xi}(x)]$ (the neural network) are in general distinct dynamical systems. Under the strong average Lipschitz condition of Theorem 2, the function $F$ inherits contractivity and therefore has a unique global point attractor (see supplementary Section C). However, for linear/affine stochastic IFS, weaker Lyapunov-type “average contractivity” conditions (based on $\mathbb{E}[\log c_{\Xi}] < 0$) (Furstenberg & Kesten, 1960; Bougerol & Lacroix, 1985) control only the stochastic recursion and do not guarantee stability of $F$. A simple one-dimensional example illustrating this distinction is provided in Appendix C. In the non-linear case, up to our knowledge there is no better hypotheses than the strong average Lipschitz condition. Lyapunov-type conditions can be imported to analyze gradients and to guarantee local stability of random dynamical systems (Oseledets, 1968; Ruelle, 1979).

**Deep Learning in Practice** The IFS viewpoint brings interesting insight to the design and practice of deep learning. The IFS stability results require contractivity (either uniform or in the averaged sense of Theorem 2), these conditions are too strong for deterministic networks and their depth dynamic would collapse to zero. Yet, some existing practices have come closer to this regime e.g., Lipschitz networks, spectral normalization (Miyato et al., 2018), or OT entropy-regularized attention (Sander et al., 2022). Sampling in deep neural networks is also a common practice as mentioned
<figure><img src="image_6.png" /><figcaption>Figure 3. From left to right: attractors obtained with ResNet with zero bias, ResNet with learned bias and Transformer architectures trained with collage error minimization.</figcaption></figure>

in Section 3.2 and noise injection (Bishop, 1995) is known to regularize. The main takeaway of the IFS viewpoint is that contractivity combined with sampling will guarantees stability while preserving expressivity.

# 4 Collage Error Bounds Generalization Error

**Definition 9 (Generalization, statistical and empirical collage error).** Let $\mu$ be the data distribution on $\mathcal{X}$ and $\mu_n \stackrel{\text{def.}}{=} \frac{1}{n} \sum_{i=1}^n \delta_{x_i}$ the empirical measure associated to samples $\{x_i\}_{i=1}^n \subset \mathcal{X}$. Let $T_\theta^*: \mathcal{P}(\mathcal{X}) \to \mathcal{P}(\mathcal{X})$ a $c_\theta$-contractive operator on $(\mathcal{P}(\mathcal{X}), W_2)$ parametrized by $\theta \in \Theta$ and let $\mu_\theta$ be its fixed point.

The empirical collage error is $\varepsilon(n, \theta) \stackrel{\text{def.}}{=} W_2(T_\theta^* \mu_n, \mu_n)$.

The generalization error is $W_2(\mu_\theta, \mu)$.

The statistical error is $W_2(\mu, \mu_n)$.

**Theorem 4 (Generalization bound).** Use the notation and hypotheses of Definition 9. Then, for any $(n, \theta)$ we have

$$W_2(\mu_\theta, \mu) \le \frac{\varepsilon(n, \theta)}{1 - c_\theta} + W_2(\mu, \mu_n).$$

The first term $\varepsilon(n, \theta)$ is the empirical collage error, an objective that can be minimized in practice. The second term $W_2(\mu, \mu_n)$ is purely statistical and depends only on sampling fluctuations of the empirical measure; under mild regularity assumptions on $\mu$ (finite q-th moment, bounded support, or subgaussian tails) it converges to 0 at classical empirical OT rates (Fournier & Guillin, 2015). Thus, whenever $T_\theta^*$ is contractive, minimizing $\varepsilon(n, \theta)$ controls the Wasserstein distance $W_2(\mu_\theta, \mu)$ between the model's invariant law and the true data distribution. This bound can justify heuristically the use of the collage error as an evaluation metric.

# 5 Numerical Experiments

The IFS view on neural networks brings several stability/convergence results and a generalization bound. The

contractivity condition is somehow strong and there is a tradeoff as the generalization bound explodes for $c_\theta$ close to 1. In addition, the new collage error minimization objective might not be useful in practice for high-dimensional datasets. To answer these questions, we first design a simple MoE/I-IFS following Proposition 4. We use spectral normalization of the linear layers used to define the IFS functions to ensure contractivity. Contrary to a standard Transformer we do use branch sampling during the forward pass (+ additional noise). We train this MoE/I-IFS on a simple 2D dataset supported on two separated half-circles plus Gaussian noise (two-moons datasets). Second, we compare the attractor of our MoE/I-IFS to the attractor of a small ResNet and a single-head Transformer (both standard non-stochastic and no imposed contractivity). We chose the hyper-parameters so that the total number of parameters is approximately 1300 (see supplementary Table 2). Lastly, we train bigger MoE/I-IFS on larger datasets (MNIST, CelebA, CIFAR-10) to show that minimizing the collage error is a valid and promising objective function for generative modeling. We refer to the supplementary Section E for implementation details.

## 5.1 Feasibility, Convergence and Error Bound

We approximate the Wasserstein collage error using regularized OT (geomloss (Feydy et al., 2019)) between samples of $T_\theta^* \mu_n$ and $\mu_n$. Figure 2 shows the successfully minimized collage error (left, $c_\theta = 0.9$), the training set and the attractor of our trained MoE/I-IFS (center, $c_\theta = 0.9$) and the generalization error and its bound for multiple contraction values $c_\theta \in \{0.3, 0.5, 0.7, 0.9\}$. The attractor set matches the training set very well except where it is discontinuous. This is not a surprise as the functions of the IFS are continuous. Regarding the generalization bound, it is indeed tighter when $c_\theta$ is small enough. In low-dimension the statistical error is negligible, and, consistent with the theory, there is a scale factor proportional to $1/1-c_\theta$ between the generalization error and its bound as $c_\theta$ increases. Finally, the $c_\theta$-error bound tradeoff is empirically mitigated (at least in our example). Indeed, the lower $c_\theta$ the tighter the bound, yet
<figure><img src="image_7.png" /><figcaption>Figure 4. Samples from the learned MoE IFS on latent MNIST, CelebA, and CIFAR-10 representations (left to right).</figcaption></figure>

we observe that the generalization error increases and the bound decreases. In contrast, for larger $c_θ$ the generalization error gets smaller while the bound worsens. In fact, the attractor is getting more constrained with lower contractivity constant (see supplementary Figure 5) but it remains close to the data distribution for moderate contractivity constants.

## 5.2 Attractors

ResNets are degenerate affine/linear P-IFS, their theoretical attractors are points/cones (when linear) and fractals (when affine). This is exactly what we get numerically in Figure 3 (left-center). In both cases the attractors remain poorly aligned with the training set. Transformers are non-linear P-IFS, their attractors are more complex and they are in general more expressive. Hence, the attractor set matches partially the training set. Yet, we note that after 2000 epochs the objective collage error is still decreasing until 5000 epochs (supplementary Figure 6-left) and the displayed attractor (Figure 3-right) evolve with more training (supplementary Figure 6-right). In addition, we note that constraining the embedding dimension to the one of the dataset (i.e. 2) prevent the transformer from learning non-linear structures, it collapses to a pair of Dirac (see supplementary Figure 7-right). Such results are expected in theory (Geshkovski et al., 2023b).

Together, the attractors of Figure 3 must be compared to the MoE/I-IFS attractor presented in Figure 2-center. To complete the comparison, we train a deterministic version of the MoE/I-IFS i.e. a standard MoE without branch sampling (see Definition 8). In this case the attractor is a Dirac at zero (even if we reactivate sampling to generate samples), see supplementary Figure 8.

## 5.3 Larger Datasets

Finally to test whether the collage objective can be useful to train generative models of real data, we train cascaded MoE/I-IFS on MNIST, CelebA and CIFAR-10. To this purpose, we first train an auto-encoder and then train the cascaded MoE on the latent distribution (see details in

supplementary Section E). Finally, we can generate new samples by decoding power iterates of the IFS from any random latent initialization. The results displayed in Figure 4 demonstrate that minimizing the collage objective leads to meaningful samples in higher dimension (32 to 256). Yet, the samples are worsening with the increasing complexity of the dataset (from MNIST to CIFAR-10). When trained on MNIST and CelebA, auto-encoder latent spaces distributions are known to have a manifold structure. For MNIST, the learned manifold captures variations in digit identity, stroke thickness, and slant. For CelebA, it organizes semantically meaningful factors such as pose, skin tone, and facial expression. This is less true for CIFAR-10 which is more diverse and has less systematic variability.

# 6 Discussion and Conclusion

We introduced a stochastic-IFS perspective on depth dynamics that makes the branch maps and selector kernels explicit for standard architectures, yielding existence/uniqueness of invariant measures under average contractivity and a Wasserstein generalization bound via the collage error. Empirically, the collage objective produces meaningful attractors for MoE/I-IFS and nontrivial samples in latent spaces, while highlighting the expressivity-stability tradeoff in high dimensions. More broadly, combining IFS branches with projection operators offers a way to study stability in other settings, such as DNN classifiers or AEs, where the dynamics act on constrained manifolds or feature subspaces.

Two limitations are intrinsic: (i) contractivity assumptions are strong, and deterministic networks without sampling typically collapse; (ii) the collage objective scales with OT estimation. Recent analyses of smooth attention and Lipschitz control in Transformers (Castin et al., 2024) suggest that data-dependent kernels can be regularized directly, and contraction arguments used in diffusion dynamics (Gao et al., 2025) point to a broader connection between generative flows and stochastic IFS. Future work should emphasize empirical extensions, including alternative discrepancies for the collage objective (sliced Wasserstein, MMD, energy
distances) and training directly in data space without AE.

## References

Barnsley, M. F. and Hutchinson, J. E. Fractals and self-similarity in probability theory. In *Probability Theory on Vector Spaces III*, volume 1362 of *Lecture Notes in Mathematics*, pp. 36–55. Springer, 1989. doi: 10.1007/BFb0083591.

Barnsley, M. F., Demko, S., Elton, J., and Geronimo, J. Invariant measures for markov processes arising from iterated function systems. *Journal of Approximation Theory*, 51(4):331–345, 1988. doi: 10.1016/0021-9045(88) 90156-X.

Barnsley, M. F. et al. Fractal image compression. *Notices of the AMS*, 43(6):657–662, 1996.

Bishop, C. M. Training with noise is equivalent to tikhonov regularization. *Neural computation*, 7(1):108–116, 1995.

Bougerol, P. and Lacroix, J. *Products of Random Matrices with Applications to Schrödinger Operators*, volume 8 of *Progress in Probability and Statistics*. Birkhäuser, 1985. doi: 10.1007/978-1-4684-9329-1.

Bredell, G., Flouris, K., Chaitanya, K., Erdil, E., and Konukoglu, E. Explicitly minimizing the blur error of variational autoencoders. In *The Eleventh International Conference on Learning Representations*, 2023. URL https://openreview.net/forum?id=9krnQ-ue9M.

Bressloff, P. C. and Stark, J. *Neural Networks, Learning Automata and Iterated Function Systems*, pp. 145–190. Springer New York, New York, NY, 1991. ISBN 978-1-4612-3034-2. doi: 10.1007/978-1-4612-3034-2_8. URL https://doi.org/10.1007/978-1-4612-3034-2_8.

Camuto, A., Wang, G., Hayou, S., Gretton, A., Doucet, A., and Sejdinovic, D. Fractal structure of stochastic gradient descent in deep neural networks. In *Advances in Neural Information Processing Systems*, 2021. URL https://arxiv.org/abs/2106.02657.

Castin, V., Ablin, P., and Peyré, G. How smooth is attention? In *Proceedings of the 41st International Conference on Machine Learning*, pp. 5817–5840, 2024.

Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. *Advances in neural information processing systems*, 31, 2018.

Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. *arXiv preprint arXiv:1904.10509*, 2019.

Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, L., Belanger, D. B., Colwell, L. J., and Weller, A. Rethinking attention with performers. In *International Conference on Learning Representations*, 2021. URL https://openreview.net/forum?id=Ua6zuk0WRH.

Cobbinah, M., Nunoo-Mensah, H., Ebenezer Adjei, P., Adoma Acheampong, F., Acquah, I., Tutu Tchao, E., Selasi Agbemenu, A., John Kponyo, J., and Abaidoo, E. Diversity in stable gans: A systematic review of mode collapse mitigation strategies. *Engineering Reports*, 7(6):e70209, 2025. doi: https://doi.org/10.1002/eng2.70209. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/eng2.70209.

Delattre, B., Barthélemy, Q., Araujo, A., and Allauzen, A. Efficient bound of lipschitz constant for convolutional layers by gram iteration. In *Proceedings of the 40th International Conference on Machine Learning*, pp. 7513–7532, 2023.

Diaconis, P. and Freedman, D. Iterated random functions. *SIAM Review*, 41(1):45–76, 1999. doi: 10.1137/S0036144598338446.

Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. *Journal of Machine Learning Research*, 23(120):1–39, 2022. URL http://jmlr.org/papers/v23/21-0998.html.

Feydy, J., Séjourné, T., Vialard, F.-X., Amari, S.-i., Trouve, A., and Peyré, G. Interpolating between optimal transport and mmd using sinkhorn divergences. In *The 22nd International Conference on Artificial Intelligence and Statistics*, pp. 2681–2690, 2019.

Fournier, N. and Guillin, A. On the rate of convergence in wasserstein distance of the empirical measure. *Probability theory and related fields*, 162(3):707–738, 2015.

Furstenberg, H. and Kesten, H. Products of random matrices. *The Annals of Mathematical Statistics*, 31(2):457–469, 1960.

Gao, X., Nguyen, H. M., and Zhu, L. Wasserstein convergence guarantees for a general class of score-based generative models. *Journal of machine learning research*, 26(43):1–54, 2025.

Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. The emergence of clusters in self-attention dynamics. *arXiv preprint arXiv:2305.05465*, 2023a.

Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. A mathematical perspective on transformers. *arXiv preprint arXiv:2312.10794*, 2023b.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. *Advances in neural information processing systems*, 27, 2014.

Henrikson, J. Completeness and total boundedness of the hausdorff metric. *MIT Undergraduate Journal of Mathematics*, 1(69-80):10, 1999.

Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. *Advances in neural information processing systems*, 35: 26565–26577, 2022.

Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In *International Conference on Learning Representations*, 2020. URL https://openreview.net/forum?id=rkgNKkHtvB.

Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. {GS}hard: Scaling giant models with conditional computation and automatic sharding. In *International Conference on Learning Representations*, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb.

Meyn, S. P. and Tweedie, R. L. *Markov chains and stochastic stability*. Springer Science & Business Media, 2012.

Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. In *International Conference on Learning Representations*, 2018.

Mohan, S., Kadkhodaie, Z., Simoncelli, E. P., and Fernandez-Granda, C. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In *International Conference on Learning Representations*, 2020. URL https://openreview.net/forum?id=HJ1SmC4FPS.

Oseledets, V. I. A multiplicative ergodic theorem. characteristic ljapunov, exponents of dynamical systems. *Trudy Moskovskogo Matematicheskogo Obshchestva*, 19:179–210, 1968.

Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference. *Journal of Machine Learning Research*, 22(57):1–64, 2021.

Peyré, G., Cuturi, M., et al. Computational optimal transport: With applications to data science. *Foundations and Trends® in Machine Learning*, 11(5-6):355–607, 2019.

Poli, M., Xu, W., Massaroli, S., Meng, C., Kim, K., and Ermon, S. Self-similarity priors: Neural collages as differentiable fractal representations. In Oh, A. H., Agarwal,

A., Belgrave, D., and Cho, K. (eds.), *Advances in Neural Information Processing Systems*, 2022. URL https://openreview.net/forum?id=U2bAR6qzF9E.

Roy, A., Saffar, M., Vaswani, A., and Grangier, D. Efficient content-based sparse attention with routing transformers. *Transactions of the Association for Computational Linguistics*, 9:53–68, 2021.

Ruelle, D. Ergodic theory of differentiable dynamical systems. *Publications Mathématiques de l'Institut des Hautes Études Scientifiques*, 50(1):27–58, 1979.

Sander, M. E., Ablin, P., Blondel, M., and Peyré, G. Sinkformers: Transformers with doubly stochastic attention. In *International Conference on Artificial Intelligence and Statistics*, pp. 3515–3530. PMLR, 2022.

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In *Proceedings of the 5th International Conference on Learning Representations (ICLR)*, 2017. URL https://arxiv.org/abs/1701.06538.

Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In *International Conference on Learning Representations*, 2021. URL https://openreview.net/forum?id=PdTIG12RRHS.

Stark, J. Iterated function systems as neural networks. *Neural Networks*, 4(5):679–690, 1991.

Stenflo, Ö. Uniqueness of invariant measures for place-dependent random iterations of functions. In Barnsley, M. F., Saupe, D., and Vrscay, E. R. (eds.), *Fractals in Multimedia*, pp. 13–32, New York, NY, 2002. Springer New York. ISBN 978-1-4684-9244-6.

Stenflo, Ö. Ergodic theory of iterated function systems. *Journal of Mathematical Analysis and Applications*, 279(2): 779–792, 2003. doi: 10.1016/S0022-247X(03)00264-3.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. *Advances in neural information processing systems*, 30, 2017.

Villani, C. et al. *Optimal transport: old and new*, volume 338. Springer, 2008.
A Supplementary Definitions and Proofs

**Definition 10** (c-contraction). Let $T: \mathcal{X} \to \mathcal{X}$. We say that $T$ is a c-contraction (or c-contractive) iff there exists a constant $c \in [0, 1)$ such that for all $x, y \in \mathcal{X}$,

$$d(Tx, Ty) \leq cd(x, y)$$

**Theorem 5** (Banach Fixed Point Theorem). Let $T: \mathcal{X} \to \mathcal{X}$ be c-contraction. Then:

1. There exists a unique fixed point $\tilde{x} \in \mathcal{X}$ such that $T\tilde{x} = \tilde{x}$.

2. For any $x_0 \in \mathcal{X}$, the sequence defined by $x_{n+1} = Tx_n$ converges to $\tilde{x}$.

3. Moreover, for all $n \geq 0$,

$$d(x_n, \tilde{x}) \leq \frac{c^n}{1-c} d(x_1, x_0). \quad (3)$$

**Proof of Banach Fixed Point Theorem 5.** Fix $x_0 \in \mathcal{X}$ and define the Picard iterates $x_{n+1} = Tx_n$ for $n \geq 0$. Using (10),

$$d(x_{n+1}, x_n) = d(Tx_n, Tx_{n-1}) \leq c d(x_n, x_{n-1}).$$

By induction,

$$d(x_{n+1}, x_n) \leq c^n d(x_1, x_0), \quad \forall n \geq 0.$$

For $m > n$, the triangle inequality yields

$$d(x_m, x_n) \leq \sum_{k=n}^{m-1} d(x_{k+1}, x_k) \leq \sum_{k=n}^{\infty} c^k d(x_1, x_0) = \frac{c^n}{1-c} d(x_1, x_0).$$

Hence $(x_n)_{n \geq 0}$ is a Cauchy sequence. Since $(\mathcal{X}, d)$ is complete, there exists $\tilde{x} \in \mathcal{X}$ such that $x_n \to \tilde{x}$.

We now show that $\tilde{x}$ is a fixed point. By continuity of $T$ (which follows from (10)),

$$T\tilde{x} = T\left(\lim_{n \to \infty} x_n\right) = \lim_{n \to \infty} Tx_n = \lim_{n \to \infty} x_{n+1} = \tilde{x}.$$

Thus $\tilde{x}$ is a fixed point.

To prove uniqueness, suppose $y^*$ is another fixed point, $Ty^* = y^*$. Then

$$d(\tilde{x}, y^*) = d(T\tilde{x}, Ty^*) \leq cd(\tilde{x}, y^*).$$

Since $c < 1$, this implies $(1-c)d(\tilde{x}, y^*) \leq 0$, hence $d(\tilde{x}, y^*) = 0$ and $\tilde{x} = y^*$.

Finally, the estimate (3) follows by letting $m \to \infty$ in the previous bound:

$$d(x_n, \tilde{x}) \leq \frac{c^n}{1-c} d(x_1, x_0).$$

This completes the proof. □

**Proof of Banach Fixed Point Corollary 1.** Using $T\tilde{x} = \tilde{x}$ and the triangle inequality,

$$d(\tilde{x}, y) = d(T\tilde{x}, y) \leq d(T\tilde{x}, Ty) + d(Ty, y).$$

By the contraction property (10),

$$d(T\tilde{x}, Ty) \leq cd(\tilde{x}, y),$$

and by definition of $\varepsilon$ we have $d(Ty, y) = \varepsilon$. Thus

$$d(\tilde{x}, y) \leq cd(\tilde{x}, y) + \varepsilon.$$

Rearranging gives

$$(1-c)d(\tilde{x}, y) \leq \varepsilon,$$

which is equivalent to (1). □

**Proof of Proposition 1.** Let $K_{\tilde{\mu}} \stackrel{\text{def.}}{=} \text{supp}(\tilde{\mu})$. We have

$$\begin{aligned} K_{\tilde{\mu}} &= \text{supp}(T^* \tilde{\mu}) \\ &= \text{supp} \left( \sum_{\xi \in \mathcal{I}} (w_{\xi}) \#(q_{\xi} \cdot \tilde{\mu}) \right) \\ &= \overline{\bigcup_{\xi \in \mathcal{I}} w_{\xi} (\text{supp}(q_{\xi} \cdot \tilde{\mu}))} \\ &= \overline{\bigcup_{\xi \in \mathcal{I}} w_{\xi} (\text{supp}(\tilde{\mu})).} \end{aligned}$$

The equality from the second to the third line holds because the support of the sum is the closure of the union of the individual supports. The last equality holds because for all $\xi \in \mathcal{I}$, $q_{\xi} > 0$. Closed-ness of support and continuity for the functions gives

$$K_{\tilde{\mu}} = \mathcal{H}(K_{\tilde{\mu}}),$$

i.e. $K_{\tilde{\mu}}$ is a fixed point of the Hutchinson operator on compacts. Contractivity of the $w_{\xi}$ makes that operator a contraction on $(\mathcal{K}(\mathcal{X}), d_H)$ with unique fixed point $\tilde{K}$. Hence $K_{\tilde{\mu}} = \tilde{K}$. □

**Proof of Proposition 2.** We verify easily that the Markov recursion is conform to Definition 2. □

**Proof of Proposition 3.** Conditioned on $\mathbf{x}$, the indices $(\Xi_i)$ are independent with marginals $(\alpha_{i, \cdot}(\mathbf{x}))$. Sampling

$$\Xi_i | \mathbf{X} = \mathbf{x} \sim C(p_{\text{TF}}(\mathbf{x}))$$

and applying $w_{\Xi}^0$ yields a well-defined Markov recursion and therefore a place-dependent IFS on $\mathcal{X}^n$. Taking conditional expectation over $\Xi$ produces $G_{\theta}$ as claimed. □

**Proof of Proposition 4.** We verify easily that the Markov recursion is conform to Definition 2. □
**Proof of Theorem 2.** See Stenflo (2002) Section 4 and references therein. □

**Proof of Theorem 3.** Let $(W, q)$ be a $c$-contractive I-IFS, so each $w_\xi$ is $c$-Lipschitz. For any $\mu, \nu \in \mathcal{P}(\mathcal{X})$,

$$ 
\begin{aligned}
W_2^2(T^*\mu, T^*\nu) &= W_2^2\left(\sum_{\xi \in \mathcal{I}} q_\xi(w_\xi)\#\mu, \sum_{\xi \in \mathcal{I}} q_\xi(w_\xi)\#\nu\right) \\
&\le \sum_{\xi \in \mathcal{I}} q_\xi W_2^2((w_\xi)\#\mu, (w_\xi)\#\nu) \\
&\le \sum_{\xi \in \mathcal{I}} q_\xi c^2 W_2^2(\mu, \nu) = c^2 W_2^2(\mu, \nu).
\end{aligned}
$$

The first inequality uses convexity of $W_2^2$ and the second uses the pushforward stability $W_2((w_\xi)\#\mu, (w_\xi)\#\nu) \le$ Lip$(w_\xi) W_2(\mu, \nu)$; see Villani et al. (2008). Thus $T^*$ is $c$-contractive in $W_2$, and Corollary 1 applies. □

**Proof of Theorem 4.** Let $\mu_\theta$ be the fixed point of $T_\theta^*$. By the triangle inequality,

$$ W_2(\mu_\theta, \mu) \le W_2(\mu_\theta, \mu_n) + W_2(\mu_n, \mu). $$

Since $T_\theta^*$ is $c_\theta$-contractive and $\varepsilon(n, \theta) = W_2(T_\theta^*\mu_n, \mu_n)$, we have

$$ 
\begin{aligned}
W_2(\mu_\theta, \mu_n) &\le W_2(\mu_\theta, T_\theta^* \mu_n) + W_2(T_\theta^* \mu_n, \mu_n) \\
&\le c_\theta W_2(\mu_\theta, \mu_n) + \varepsilon(n, \theta).
\end{aligned}
$$

Rearranging yields

$$ W_2(\mu_\theta, \mu_n) \le \frac{\varepsilon(n, \theta)}{1 - c_\theta}, $$

which gives the claimed bound. □

<table><caption>Table 1. Summary of IFS interpretations across architectures.</caption><thead><tr><th>Model</th><th>Maps w<sub>i</sub></th><th>Kernel p<sub>ξ</sub>(x)</th><th>IFS type</th></tr></thead><tbody><tr><td>ReLU residual block (sampled)</td><td>Definition 5</td><td>δ<sub>ξ(x)</sub></td><td>P-IFS (degenerate)</td></tr><tr><td>Transformer block (sampled)</td><td>Definition 7</td><td>Π<sub>i</sub> α<sub>i,ξ<sub>i</sub></sub>(x)</td><td>P-IFS</td></tr><tr><td>MoE (sampled)</td><td>Generic</td><td>Generic</td><td>P-IFS</td></tr></tbody></table>

## B Softplus ResNet as an I-IFS

Construction of IFSs associated to neural networks is not only feasible for ReLU activations. It is in fact always feasible. Below we make a construction for a ResNet with Softplus activation.

**Definition 11 (Softplus residual block).** Let $\varphi(z) = \log(1+e^z)$ be the softplus activation applied coordinatewise. Given $A \in \mathbb{R}^{m \times d}$, $b \in \mathbb{R}^m$, $B \in \mathbb{R}^{d \times m}$, $c \in \mathbb{R}^d$, define the residual block $F_\theta: \mathcal{X} \to \mathcal{X}$ by

$$ F_{\theta}(x) = x + B \varphi(Ax + b) + c, \quad \theta = (A, B, b, c). $$

**Definition 12 (Softplus branch family and kernel).** Let $\rho$ be the logistic density. Let $\mathcal{I} = \mathbb{R}^m$ and for $\tau \in \mathcal{I}$ define the branch map

$$ w_{\tau}(x) = x + B \operatorname{ReLU}(Ax + b - \tau) + c, $$

where $\operatorname{ReLU}$ is applied coordinatewise. Let $p$ be the product measure with density $\prod_{j=1}^m \rho(\tau_j)$ on $\mathcal{I}$.

**Proposition 5 (Softplus residual block as I-IFS).** The pair $(W_{sp}, p)$ with $W_{sp} = \{w_\tau\}_{\tau \in \mathcal{I}}$ defines a place-independent IFS on $\mathcal{X}$, and the softplus residual block satisfies

$$ F_{\theta}(x) = \mathbb{E}_{\tau \sim p}[w_{\tau}(x)]. $$

*Proof.* For a scalar $z$, $\varphi(z) = \mathbb{E}_{T \sim \rho}[\operatorname{ReLU}(z - T)]$ where $\rho$ is the logistic density. Applying this coordinatewise and using independence of the components of $\tau$ yields the stated expectation identity. □

## C Stability of IFS vs Expected IFS ?

We distinguish two notions of average contractivity for a place-dependent P-IFS ($\{w_\xi\}_{\xi \in \mathcal{I}}$, $p$) on $(\mathcal{X}, d)$ with selector kernel $(p_\xi(x))_{\xi \in \mathcal{I}}$ and Lipschitz constants $c_\xi$ for each $w_\xi$.

**Definition 13 (Strong average Lipschitz contractivity).** We say that the P-IFS is strongly average-contractive if

$$ \sup_{x \in \mathcal{X}} \sum_{\xi \in \mathcal{I}} p_{\xi}(x) c_{\xi} \le c < 1, \quad (4) $$

and the selector kernel $p: \mathcal{X} \to \Delta^{|\mathcal{I}|-1}$ depends on $x$ in a Lipschitz way (e.g. in total variation or Wasserstein distance), as in Theorem 2.

**Definition 14 (Lyapunov (log-average) contractivity).** We say that the P-IFS is Lyapunov-average-contractive if, for the random Lipschitz constants $c_{\Xi_k}$ along a trajectory, one has

$$ \limsup_{t \to \infty} \frac{1}{t} \sum_{k=0}^{t-1} \log c_{\Xi_k} < 0, \quad \text{a.s.} \quad (5) $$

In the place-independent case, this reduces to $\mathbb{E}[\log c_\Xi] < 0$. This is the condition classically used in the theory of products of random matrices.

In our framework the deterministic map associated with the P-IFS is the barycentric map

$$ F(x) = \mathbb{E}[w_{\Xi}(x) | X = x] = \sum_{\xi \in \mathcal{I}} p_{\xi}(x) w_{\xi}(x). \quad (6) $$

The following proposition clarifies when $F$ inherits contractivity and when it does not.

**Proposition 6 (Inheritance under strong average Lipschitz).** Assume the strong average Lipschitz condition (4) and that each $w_\xi$ is $c_\xi$-Lipschitz on $(\mathcal{X}, d)$. Then the barycentric map
F is a contraction with Lipschitz constant at most $c < 1$, i.e.

$$d(F(x), F(y)) \leq c d(x, y) \quad \forall x, y \in \mathcal{X}.$$

In particular, F has a unique fixed point and the deterministic recursion $x_{t+1} = F(x_t)$ converges to it for any initial condition.

*Proof.* For any $x, y \in \mathcal{X}$,

$$d(F(x), F(y)) = d\left(\sum_{\xi} p_{\xi}(x) w_{\xi}(x), \sum_{\xi} p_{\xi}(y) w_{\xi}(y)\right) \\ \leq d\left(\sum_{\xi} p_{\xi}(x) w_{\xi}(x), \sum_{\xi} p_{\xi}(x) w_{\xi}(y)\right) \\ + d\left(\sum_{\xi} p_{\xi}(x) w_{\xi}(y), \sum_{\xi} p_{\xi}(y) w_{\xi}(y)\right).$$

Under the regularity assumptions on $p(\cdot | x)$, the second term can be bounded by a multiple of $d(x, y)$ (as in Theorem 2), while the first term satisfies

$$d\left(\sum_{\xi} p_{\xi}(x) w_{\xi}(x), \sum_{\xi} p_{\xi}(x) w_{\xi}(y)\right) \\ \leq \sum_{\xi} p_{\xi}(x) d(w_{\xi}(x), w_{\xi}(y)) \\ \leq \left(\sum_{\xi} p_{\xi}(x) c_{\xi}\right) d(x, y).$$

Taking the supremum over $x$ and using (4) yields the claimed contraction bound for $F$ (up to adjusting constants to absorb the kernel regularity term). □

*Example 1* (Failure under Lyapunov-average contractivity). Consider $\mathcal{X} = \mathbb{R}$ with two maps

$$w_1(x) = \frac{1}{2}x, \quad w_2(x) = 2x,$$

and a place-independent selector kernel $P(\Xi = 1) = p$, $P(\Xi = 2) = 1 - p$. Each $w_i$ is linear with Lipschitz constants $c_1 = \frac{1}{2}$, $c_2 = 2$, so

$$\mathbb{E}[\log c_{\Xi}] = p \log \frac{1}{2} + (1 - p) \log 2 = (1 - 2p) \log 2.$$

For $p \in (\frac{1}{2}, 1)$ this Lyapunov exponent is negative and the random recursion $X_{t+1} = w_{\Xi_t}(X_t)$ is contracting on average in the sense of (5). However the barycentric map is

$$F(x) = \mathbb{E}[w_{\Xi}(x)] = p \frac{1}{2}x + (1 - p) 2x = (2 - \frac{3p}{2})x,$$

with Lipschitz constant $|2 - \frac{3p}{2}| > 1$ whenever $p \in (\frac{1}{2}, \frac{2}{3})$. In this regime the random IFS is Lyapunov-average-contractive, but the deterministic map $F$ is expanding and its iterates diverge for all $x \neq 0$. Thus Lyapunov-average-contractivity of the random P-IFS does not imply contractivity or stability of its deterministic barycentric map.

This example shows that our strong average Lipschitz condition (4) is precisely the regime in which stability and convergence are inherited by the deterministic update, whereas weaker Lyapunov-type conditions only control the stochastic recursion and have no direct implication for $F$.

## D Deep MoE

**Definition 15** (Deep Mixture-of-Experts as a P-IFS). Let each layer $d \in \{1, \dots, D\}$ consist of a selector kernel $p_{\text{MoE}}^{(d)} = p^{(d)}: \mathcal{X} \to \Delta^{K-1}$ on a finite index set $\mathcal{I}^{(d)}$ and branch maps $\{w_{\xi}^{(d)}\}_{\xi \in \mathcal{I}^{(d)}}$. A depth-D MoE defines a place-dependent IFS on $\mathcal{X}$ through the random cascade

$$x_{t+1} = w_{\Xi_t^{(D)}}^{(D)} \circ \dots \circ w_{\Xi_t^{(1)}}^{(1)}(x_t) \\ \Xi_t^{(d)}|X = x_t \sim C(p^{(d)}(x_t)).$$

*The deterministic update is the conditional expectation*

$$H(x) = \mathbb{E}[w_{\Xi_t^{(D)}}^{(D)} \circ \dots \circ w_{\Xi_t^{(1)}}^{(1)}(x)],$$

*and the associated transfer operator is the composition*

$$T^* = T_D^* \circ \dots \circ T_1^*, \quad T_d^* \mu = \sum_{\xi \in \mathcal{I}^{(d)}} (w_{\xi}^{(d)}) \#(p_{\xi}^{(d)} \cdot \mu).$$

This amounts to consider depth-dependent recursion and it increases the number of possible maps. Overall, it can be seen as a one-layer IFS with a multi-index set to account for all possible branches.

## E Implementation Details

**Two-moons.** We generate 2D point clouds with 2,048 training points and a 50k point reference set (noise = 0.1, radius = 2.0). Models follow Table 2(MoE/ResNet/Transformer) and are trained with the stochastic collage objective using Sinkhorn loss (blur = 0.05), batch size 256, 2k epochs, Adam (lr = 1e-3). For MoE runs we use contraction ($c < 1$) and ($\sigma = 0.04$); ResNet/Transformer are non-contractive unless spectral normalization is enabled. Dynamics/attractor plots use 50 Markov steps; when computing the bound we use burn-in = 100 and Monte-Carlo estimates with 8 batches of size 512.
<table><caption>*Table 2.* Key hyperparameters for the two-moons experiments. The width of the transformers is the dimension of the input embedding.</caption><thead><tr><th></th><th>MoE IFS</th><th>ResNet</th><th>Transformer</th></tr></thead><tbody><tr><th>Base map</th><td>Nonlinear<br>MLP branches</td><td>Repeated<br>residual block</td><td>Self-attention<br>on contexts</td></tr><tr><th>Width</th><td>32</td><td>32</td><td>10</td></tr><tr><th>Depth</th><td>1</td><td>8</td><td>1</td></tr><tr><th>Routing</th><td>K = 8</td><td>—</td><td>1 head</td></tr><tr><th>Contraction c</th><td>c &lt; 1</td><td>None</td><td>None</td></tr><tr><th>Noise σ</th><td>0.04</td><td>0</td><td>0</td></tr><tr><th>Context size</th><td>—</td><td>—</td><td>n = 2</td></tr><tr><th># params</th><td>1304</td><td>1296</td><td>1382</td></tr></tbody></table>

**MNIST/CIFAR-10/CelebA.** We train in latent space using a ConvResNetAE (MSE reconstruction) and then fit a latent IFS on encoded samples. Datasets are resized to 32 × 32 for MNIST/ CIFAR-10 and center-cropped (148) then resized to 64 × 64 for CelebA; we use latent dims/base channels of 32/64 (MNIST) and 256/128 (CIFAR-10, CelebA). The latent IFS is a stacked NonlinearContractiveIFS (see Definition 15) with $D = 6$ stages, $K = 128$ maps, branch depth 16 (depth of individual maps $w_i$), hidden dim 256, contraction ($c = 0.99$) (warm-up from ($c_{init} = 10$) over 12 epochs), and ($\sigma = 0.01$); routing uses constant-IFS with learned logits (temperature = 1.0). We fit on 40k latents with Sinkhorn loss in latent space using Adam (lr = 1e-5), batch size 512, 30 epochs with ReduceLROnPlateau, and generate samples by iterating 50 IFS steps and decoding 64 latents.

## F Supplementary Figures
<figure><img src="image_8.png" /><figcaption>Figure 5. Training data and samples of the attractor measure of the trained IFS for contraction constants $c \in \{0.3, 0.5, 0.7\}$ (left to right).</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 6. Empirical collage error $\epsilon(n, \theta)$ over 10k epoch for transformer with embedding dimension 10 (left) and its attractor (right).</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 7. Empirical collage error $\epsilon(n, \theta)$ for transformer with embedding dimension 2 (left) and its attractor (right).</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 8. Attractors of deterministic MoE/I-IFS. Left: deterministic sampling. Right: stochastic sampling.</figcaption></figure>