When Bias Meets Trainability: Connecting Theories of Initialization
Authors: Alberto Bassi
Marco Baity-Jesi
Aurelien Lucchi
Carlo Albert
Emanuele Francazi
Venue: ICLR 2026 Poster (2026)
Keywords: Trainability
initial guessing bias
mean field regime
phase diagrams
Primary area: learning theory
TL;DR: We prove theoretically that the optimal trainable state is necessarily biased in a wide range of models.

Abstract: The statistical properties of deep neural networks (DNNs) at initialization play an important role to comprehend their trainability and the intrinsic architectural biases they possess before data exposure. Well established mean-field (MF) theories have uncovered that the distribution of parameters of randomly initialized networks strongly influences the behavior of the gradients, dictating whether they explode or vanish. Recent work has showed that untrained DNNs also manifest an initial-guessing bias (IGB), in which large regions of the input space are assigned to a single class. In this work, we provide a theoretical proof that links IGB to previous MF theories for a vast class of DNNs, showing that efficient learning is tightly connected to a network’s prejudice towards a specific class. This connection leads to a counterintuitive conclusion: the initialization that optimizes trainability is systematically biased rather than neutral.

Published as a conference paper at ICLR 2026
WHEN BIAS MEETS TRAINABILITY: CONNECTING THEORIES OF INITIALIZATION
Alberto Bassi ETH Zurich
Marco Baity-Jesi
Aurelien Lucchi University of Basel
Eawag (ETH)
Carlo Albert Fawag (ETH)
Emanuele Francazi EPFL
ABSTRACT
The statistical properties of deep neural networks (DNNs) at initialization play important role to comprehend their trainability and the intrinsic architectural biases they possess before data exposure. Well established mean-field (MF) the- ories have uncovered that the distribution of parameters of randomly initialized networks strongly influences the behavior of the gradients, dictating whether they explode or vanish. Recent work has showed that untrained DNNs also manifest an initial-guessing bias (IGB), in which large regions of the input space are assigned to a single class. In this work, we provide a theoretical proof that links IGB to previous MF theories for a vast class of DNNs, showing that efficient learning is tightly connected to a network's prejudice towards a specific class. This con- nection leads to counterintuitive conclusion: the initialization that optimizes trainability is systematically biased rather than neutral.
INTRODUCTION
In recent years, deep neural networks have achieved remarkable empirical success across diverse domains (Jumper et al. [222]] Brrwn et al., 22000; Ramesh et al.) [022]. However, understanding their properties theoretically, especially regarding their trainability, remains challenging. central difficulty consists in explaining how the choice of variances - governs the network's ability propagate signals and gradients through depth. Improper initialization typically leads to gradient-related issues: vanishing gradients, causing persistent initial conditions and learning stagnation; or exploding gradients, causing instability in the early stages of training.
- such as weights and biases
Mean-fich field (MF) theories of wide networks have provided a systematic framework to analyze how these initial parameters shape trainability (Schoenholz et al. 2016 Gilboa al. 2019 Chen et al.
2018 Ángel Poc-López & Aguilera 2024
Yang al.
2019 Yang & Schoenholz, 2017| Xiao al.
2018 Les et a 201 8) Hayou et al. 2019
Jacot et al. 2022 Poole et al. 2016]] Yang & Hu 2021)
Depending the initialization state, network exhibits either an ordered phase, where gradients vanish, or a chaotic phase, where gradients explode. The optimal boundary - the so-called edge of chaos (EOC)- is characterized by infinite depth scale in both the forward and backward pass, making the network effectively trainable. This highlights the initial state's crucial role in determining network's subsequent learning dynamics.
Beyond their seemingly simplistic theoretical assumption - that is of infinite width - MF theories have led surprising results valid for of very deep CNN from scratch (Xiao et al., 2018) and hyper-paramen parameters transferability across depth and width through the Tensor Program approach (Yang) 2019: 2020 Yang & Hu/ (02T). In particular, Yang & Hu (2021) showed that MF networks are capable of feature learning, effectively exiting the lazy learning regime predicted by the Neural Tangent Kernel (NTK) parametrization. These results underscore the importance of MF theories in effectively describing production architectures, which are in many cases sufficiently large to lay in the MF description.
models. For the first time, they allowed training
1 Corresponding authors. Contact: aba i @ethe. ch and emanuele e. francazi @gmai com

---

Published as a conference paper at ICLR 2026
While previous MF theories specialized on the trainability conditions, recent insights also derived in the infinite-width limit showed that architectural choices also impact the initial predictive states (Francazi et al. 2024). significantly changing the classification behaviour of neural networks even before being exposed to data. Specifically, an untrained network may exhibit a prejudice initial guessing bias (IGB) or it may remain neutral, assigning equal frequency to all classes. Surprisingly, IGB solely depends on factors such as network architecture and the initialization of weights. The impact these initial predictive states have on learnability, however, remains unclear. Since IGB is itself theory of wide- untraindd neural networks, begs the question: how does connect to previous MF theories of initialization?
toward certain classes referred
Main Contributions In this work, we bridge this gap between MF-based trainability insights and IGB-based predictive state characterizations. Specifically, our contributions are:
We elucidate the link between predictive initial behaviours (IGB states) and trainabil- ity conditions (MF phases), clarifying how initialization hyperparameters and architec- tural choices jointly determine initial predictive behaviours and shape subsequent training dynamics.
We show that trainability in wide architectures coincides with state of transient deep prejudice at initialization, challenging the intuitive assumption (Francazi et al., 2024) that the optimal trainability state must be unbiased.
We verify empirically conclusions a wealth of architectures trained on binary and multi-classification tasks, showing that at the EOC models exhibit strong bias, which is absorbed during the initial phase of the learning dynamics.
We generalize the IGB framework to accommodate non-zero bias terms and multi-node tivation functions (such as maxpool layers), further expanding its applicability, and correct existing MF phase diagram inaccuracies (e.g. for ReLU).
We emphasize the role of per-class gradients, showing that the vanishing/exploding gradi- ent behaviour is class-dependent.
Our results are valid for a general wide architecture and only assume its output to be normally distributed, simpler and more general setting than the original, full IGB analytical framework.
Practical Consequences of Our Theory Aside from the theoretical value of linking bias with trainability, there are some practical takeaways which become obvious in light of our theory:
Gradient exploding. Due to initialization biases, gradient exploding only involves a subset of classes (Fig. This causes an imbalance in per-class gradients, which can drastically slow down learning (Francazi et al. 2023).
Hyperparameter tuning. Assessing model performances based on short runs is at risk of privileging specific classes due to residual bias (Fig. 5)
Knowledge transfer. By linking IGB to previous MF theories, which do not impose any prior data distribution, we relax the assumptions of IGB theory, thus extending its validity. On the converse, the IGB formalism complements the MF phase diagrams, revealing novel phases and unveiling unexpected biases (Fig. 3).
BACKGROUND
2.1 SETUP
We consider generic neural network architecture A composed by L layers (depth) of width Ni. for 1 = 1, 1,...,I, L. We define y!"h (a) the pre-activation (signal) of each layer I and each neuron i = ,,..,, N₁. and wil the set of all network's weights at layer l. The architecture A processes sequentially the pre-activations according to the recursive rule:
(0)) = F(ol)) (ya-z)) m)))))
(1)
2

---

Published as a conference paper at ICLR 2026
where (.0) (.) RN - RN: is the 1-th layer generic non-linear activation function and FF is linear function. We consider the problem of processing ND samples belonging to a dataset D = (E(a))o=1,....N through A, and we denote with yi (1) (a) the pre-activation computed with the a- th data sample as input. Thus, in the input layer we have by definition that ( (a) = (a) and No denotes the input dimension. We consider the aarge-width limit taken before that of the dataset samples and depth. Formally, our results are valid when the limiting order is: for any function f. we take lim lim limming (N)) f.
For the sake of simplicity, we will specialize the discussion to multi-layer perceptrons (MLPs), which still play an important role in modern machine learning as they are the building blocks of most complex architectures. For instance, Transformers (Vaswani et al. 2017) are composed by several stacked MLPs and attention layers. For MLPs, Eq. (1) reads W (W))b())) and FF
is the affine transformation: F(x; W b(0)= ( x + ((0, where (((()) is the weight matrix and b (()) the bias vector.
2.2 ORDER/CHAOS PHASE TRANSITION: INITIALIZATION CONDITIONS FOR TRAINABILITY
One kind of average When the datapoints are fixed, the pre-activations of an untrained MLP random functions of one source of randomness, coming from the joint set of all weights and biases, shortly denoted with W. One usually considers the random ensemble corresponding to initializing weights and biases i.i.d. with Gaussian distributions (Schoenholz et al.) 2016 Poole
et al. neurons 1,J I,..., Ni and for every layer 1 = 1,..., L. The scaling of the weight variance by the square root of the network's width is necessary in order maintain the signal O(1) as it transverses the network.
2016 Hayou et al. 2019), i.c w!
4. ~ N (0 (ot /Ni and 6!" ~ N (0, )o))) for every
In this setup, only one kind of average naturally arises: the average over W at fixed dataset D, which is denoted with an overbar, ā = Ew (x|D). For fixed inputs, when performing the limit of infinite width before that of depth, the pre-activations become i.i.d. Gaussian variables with mean (1
and signal variance og = Toa- with qau 0ig = yi (1) (a)y," ( (a) (Lee et al. 2018). The indinite-width limit is commonly referred to as mean-field regime, since correlations among neurons vanish and the activaait distributions are fully characterized by the signal variance. The pre-activation distributions defined in MF theory permit the study of the propagation of the signal through the network; however, they do not yield any insight into interactions among distinct data samples. To such end, one has to define a correlation coefficient between inputs as
(1) cab = Lab V You lbb
(1)
(2)
where (1) (a)y, (b)
Lab deg. and Lab is the signal covariance between inputs a and b. The reader referred to App. where we report the recursive relations for the signal variance [Eq. 7 and
(8)
(1)
covariance [Eq. 8 first derived by Poole et al. (2016).
Phase transition in bounded activation functions
The authors of [Schoenholz al. (016))
tensively analyzed bounded activation functions, such as Tanh. By defining X1 = 0e /de c-
X1 = 1 separates an ordered phase (x1 < 1) where the correlation coefficient converges to one, i.e.
c lim c 1, and a chaotic phase (x1 > 1) where the correlation coefficient converges to lower value. Additionally, the value of X1 determines the transition from vanishing gradients
X1 < 1), to exploding gradients (X1 > 1). These two phases have well-known consequences for training: vanishing gradients hinder learning by causing long persistence of the initial conditions, while exploding gradients lead to instability in the training dynamics (Bengio et al.] [194] Pascanu al. 2013). At the transition point, both the gradients are stable and the depth-scale of signal prop- agation diverges exponentially. This is the optimal setting for training, as it allows all layers in the network to be trained from the start.
Phase diagram The MF theory constructs a phase diagram by looking at the convergence be- haviour of the correlation coefficient, in terms of the weight and bias variances at initialization that is, the (of, o plane. of and o² are referred to as control parameters, whereas the quanti- ties identifying different phases are termed order parameters. Consequently, for bounded activation
3

---

Published as a conference paper at ICLR 2026
functions, the correlation coefficient constitutes order parameter, as its asymptotic value alone suffices to distinguish between phases.
Unbounded activation functions In this case, the signal variance not guaranteed to converge, and one has to account for unbounded signals when defining the order/chaos phase transition. For example, it is possible for the correlation coefficient to converge always to one in the whole phase diagram, as we will later demonstrate for ReLU; hence, c does not always serve as an effective order parameter for discriminating between phases. In particular, in App. [ we prove that the quantity X1 dg (1+1) /a le=1 can discriminate the ordered from the chaotic phase and is equal X1 in the domain of convergence of the variance. Thus, for unbounded activation functions, X1 = separates the region in the phase diagram with exploding gradients from the one where gradients vanish, acting as a discriminative order parameter. Following Hayou et al. (2019), it is therefore appropriate to define the edge of chaos (EOC) as the set of points in the phase diagram where 1) the signal variance converges and 2) X1 X1 1. Additionally, Hayou et al. (2019) provided simple algorithm (Algorithm I of the main paper) to compute the EOC for a generic single-node activation function. This way, it is also possible to analyze unbounded activation functions in regions of the phase diagram characterized by convergent signals.
2.3 INITIAL GUESSING BIAS: PREDICTIVE BEHAVIOUR INITIALIZATION
Two kinds of averages For randomly initialized deep neural networks processing inputs drawn from dataset distribution, two distinct sources of randomness naturally arise: randomness from network W and randomness from D. The MF approaches typically fix the input and average over the ensemble of random weights to analyze signal propagation. In contrast, recent studies (Francazi ct al. 2024 2025) introduced an alternative approach the IGB framework - where, for a fixed initialization, the entire input distribution is propagated through the network. Coherently with these works, here we suppose each data component to be i.i.d. according to a standard Gaussian distribution, ie. {a(a) (O, 1), Va E Interestingly, when averages over the dataset are performed first, the pre-activation distributions change, being not centred around zero (Francazi (1) ct al. 2024 : y; (a) ~ ~ (14 oyo)) with u NV (0.0
Neutrality vs prejudice Within the IGB framework, a predictive bias arises as a consequence of a systematic drift in signal activations. In the presence of IGB, the pre-activation signals at each node are still Gaussian distributed in the midtt limit, with variance of (0 but each node
centred around a different point, ((), which is generally different from zero. The centers only vary with initialization, and are Gaussian-distribune the signals related to different nodes in the same layer are distributed differently. This causes misalignment between the decision boundary - initially positioned near the origin and the data distribution (Francazi al. 2024). Consequently, most input points are assigned to a single class, defining predictive state which we term as prejudice, to emphasize that this is a statistically skewed assignment of classes before any data is observed or evaluated.
too, with zero mean and variance -20- Now
Conversely, when the drift is negligible and activations remain symmetrically distributed around zero, predictions remain balanced across classes; call it neutral state. Prejudice can manifest at different levels, depending on the strength of the classification bias. The extent of activation drift and corresponding predictive bias can be quantified by the activation drift ratio (²))
Definition 2.1 (Activation Drift Ratio). We define the activation drift ratio at layer as:
70 0 /20//0
where 0² 2 is the variance across random initializations of node activation centers (averaged over the dataset), and oy0 is the variance of activations due to input data variability (at fixed initialization..
If the variances around the node centers are much larger than the variances of the centers them- selves, the signals of different nodes become indistinguishable. Thus, large (diverging) values of
4

---

Published as a conference paper at ICLR 2026
xf 0.05
7/4 1.0
70 10.0
91
91
91
Pi5"'
Figure 1: Example of pre-activation distributions for neutrality (left) and moderate prejudice (right) computed by sampling Gaussian variables with synthetic data. The inset plots show the distribution of the dataset elements classified into the reference class Gg- In the neutral phase, Go is centred around 0.5, while with moderate prejudice, Go concentrates at the extremes. At the transition be- tween these two phases, Gg is uniformly distributed (middle).
y()) indicate significant drift (strong prejudice), whereas small (vanishing) values reflect minimal drift (neutrality).
Classification fractions In classification tasks, predictive bias can be quantified by measuring the fraction of inputs, G.. classified into each class at initialization. For illustrative clarity, we con- sider the binary setting, where the predictive imbalance fully captured by the fraction of inputs assigned one reference class. Here, we derive an implicit formula to compute the distribution over W of the fraction of points classified as a reference class in the infinite-widd width regime, where all pre-
activations are normally distributed: Go
(11 y2 (L) 8(W) 4
(1)
where (-)
is the Gaussian cumulative function, and 5 is standard Gaussian variable. The proof is straightfor- ward upon utilizing the IGB pre- activviivtt distributions (see App. D).
The value of (()) permits to distinguish between three phases and connects them to the preju- dice/neutrality framework provided before. When x(L))
the distribution of Go converges, in the distribution sense, to Dirac-delta centred in 0.5, whereas for 7(()) >> 1 its distribution con- verges to a mixture of two Dirac-deltas centred in and 1, respectively. Remarkably, for 7(()) (Fig. D- middle). Go is uniformly distributed in (0, 1); this critical threshold delineates phase where the fraction of points classified to a reference class exhibits a Gaussian-like shape centred in 0.5 (Fig. left), from one where the distribution deviates markedly from Gaussian behaviour and it is bi-modal (Fig. [ right). Consequently, neutrality emerges for y(4) < 1, whereas for 7,(4) >> the network exhibits prejudice. Moreover, prejudice can compound with depth we call this deep prejudice- when y lim (L) = oo, resulting in a network with strongly-biased predictions.
The IGB framework offers a rigorous theoretical characterization specifically in settings involving random, unstructured data identically distributed across classes. This demonstrates how predictive imbalances at initialization can emerge purely from architectural choices, independently of any in- trinsic data structure. Moreover, it provides quantitative tools to distinguish unbiased (neutrality) from biased (prejudice) initial states.
CONNECTING CLASSIFICATION BIAS TO THE ORDERED PHASE
In this section, we establish a direct link between the IGB framework and the standard MF theory. The phase diagram in MF is typically expressed in terms of the initialization parameters (o 2 whereas the original formulation of IGB was limited to the case of = O. To bridge this gap and lay the groundwork for a unified understanding, we extend the IGB framework to include non-zero bias variances (App. D). This extension allows to reinterpret the MF phase diagram in terms of the IGB phases, revealing the connection between initial predictive behaviour and trainability. Here, we show that all the quantities of interest in MF have equivalent counterpart in the IGB framework.
5

---

Published as a conference paper at ICLR 2026
ReLU
Tanh
11
1/8
05
10
00
155
100
layer
3.0
Figure 2: Convergence behaviour of the correlation coefficient of ReLU and Tanh for a single MLP with width equal to 10 000 and depth 100. of 0.1 and o varies uniformly from the ordered phase (blue) to the chaotic phase (red). The transition point is o = 2.0 for ReI.U and close to it for Tanh. Scatter points indicate the asymptotic values. The inset plots show the convergence rate for the correlation coefficient to ity asymptotic value c, always exponential for Tanh and power law for ReLU in the chaotic phase. Solid lines are computed using the IGB approach [Eq. (6),, while shaded areas represent the 90 % central confidence interval computed using the MF approach (Eq. 22).
Theorem 3.1 (Equivalence between MF and IGB, informal). Consider a generic architecture
(0) A (Eq. (1) in the mean-field regime. Let suppose that qaa 1, Va D and 4ah O, Va, D, & Then in the infinite data limit, Va E and VI 0, the total variance in the IGB approach is equal to the signal variance in the MF approach:
(0)
02 (e0) + o² get
44)
Moreover, Ya.b E D with a + b, the centers variance the IGB approach is equal to the imput covariance the MF approach:
Jab = o
(5)
and the correlation coefficient is related to through
700
(6)
1+ 1+70)
We report the proof of this theorem App. The key result of Thm. establishes corre- spondence between the MF formulation and IGB, enabling signal propagation in wide networks to be described interchangeably using either framework. Intriguingly, this result makes no further as- sumption than the MF regime, and it remains valid for every architecture defined by Eq. O In MF
(1) theory, qae and qab are generally functions of the dataset D, and therefore become random variables upon imposing a distribution over D, done in the IGB approach. Remarkably, these two quantities concentrate around their mean in the infinite-width and then -data limit, allowing their treatment as deterministic variables (equivalently through o² and o² o.a)). On the one hand, this formal con- nection between MF and IGB frameworks enables unified view, where predictive behaviour initialization and trainability conditions are jointly entangled, enriching the classical MF picture, as will be discussed Secs. 4 and 5 On the other, extends the IGB framework from Francazi et al. 2024) to settings not previously analyzed (c.g., identifying prejudice and neutrality phases for Tanh activations directly from the MF phase diagram (Schoenholz al. 2016)).
In Fig. 2, we test the validity of Thm. 3.1 by plotting the correlation coefficient in function of the depth for ReLU and Tanh MLPs with of 0.1 and oA uniformly varying from the ordered to the chaotic phase analyzed in MF theory. We show similar results on realistic architectures and datasets in App.
In all cases observe good agreement between the curves obtained with the IGB approach (solid lines computed via Eq. (6)) and the 90 % central confidence interval computed using the MF approach (shaded areas computed via Eq. 2 The distribution of MF

---

Published as a conference paper at ICLR 2026
is very narrow around the (GB-computed values, corroborating the treatment of the signal variance and covariance as deterministic variables. As network's ' width increases, the MF distributions of
qaa and qah become progressively more concentrated (see App. E For RELU, we observe that the correlation coefficient always converges to one, but the convergence rate is exponential in the ordered phase (o < 2) and follows a power-law in the chaotic phase (o > 2). For Tanh, the correlation coefficient converges one in the ordered phase and to a lower value in the chaotic phase; in this case, we always observe an exponential convergence behaviour.
4 BEST TRAINABILITY CONDITIONS
In Sec. we proved the connection between IGB and MF frameworks. We will now see how this association allows us to connect predictive behaviour at initialization with dynamic behaviour, specifically in terms of the network's trainability conditions, rooted in gradient stability.
Gradients at initialization have extensively been analyzed in the MF literature, so the reader is re- ferred to, for example, [Schoenholz al. (2016) Hayou al. (2019) for an extensive discussion.
For our purposes, is sufficient to note that Xi = 04,.6 (2q_ttt)/// /ag.tt/@gall c= is a key quantity separating the ordered phase, where gradients vanish, from the chaotic phase, where gradients explode. We show examples of this transition from gradient vanishing to exploding in Fig. left and in App. G
When the signal variance is non-divergent, X1 = X1 measures the stability of the fixed point c
where its dynamic counterpart cahh is connected to 7.")) through Eq, (6) In both the ordered phase stable fixed point. Consequently, both cases, we observe an
and at the EOC, the state asymptotic oo, indicating a state of deep prejudice at initialization.
However, the dynamical behaviour differs significantly between ordered and chaotic phases. In the ordered phase, gradients vanish exponentially. resulting in a state of persistency of the initial condi- tions characterized by remain stable, enabling trainability and facilitating the gradual absorption of the initial bias, result- ing in a condition of transiency of deep prejudice. Conversely, the chaotic phase mmase-in in which training is precluded by gradient instability (x1 > 1) is generally characterized either by prejudice (i.e.,
and X1 < (see Tab. [ At the EOC (x = 1), by contrast, gradients
1 1>c> 0.5), or neutrality (i.e. c < 0.5).
In Francazi et al. (2024), one of the main open questions concerned the distinction in dynamical be- haviour between neutral and prejudiced phases. The present results not only clarify this distinction by linking it to gradient stability properties, but also reveal finer structure within the prejudiced phase, identifying conditions that govern the persistence of predictive bias. Moreover, these findings
lead to the following conclusion (see proof in App. E which counters the suggestion of Francazi et al. 2024). that neutral initializations lead to the fastest dynamics.
Proposition 4.1.
From trainability perspective, the optimal initial condition (EOC) is not one of neutrality, but rather a state of transient deep prejudice.
We validate this proposition by performing training experiments of different architectures across multiple datasets (see Sec. (
How to access the EOC in practice In real scenarios, when the EOC is not known analytically, one can compute the initialization gradients for different values of oZ and of and choose the com- bination that shows stable gradients across the depth. In App. G we report initialization gradients for an array of architectures and datasets, showing how the BOC is reached. We also compare the propagation of signals in pretrained vs untrained networks, showing that also in pretrained models one has a similar phenomenology (App. G.4.I).
5 DETAILED PHASE DIAGRAMS
Due to the equivalence between IGB and MF, all MF results remain valid in the IGB framework. Therefore, for a comprehensive analysis of generic single-node activation functions, we refer to the work of Hayou et al. (2019). Beyond that, in App. F we extend the MF/IGB theory to multi-node activation functions, broadening the range of applicability of these theories to e.g. max- and average-pont pool layers.
Here, example, compare the differences between two widely utilized activation functions:

---

Published as a conference paper at ICLR 2026
ReLU
Taah
8-
Chaotic-dowe prejudice
Neutrality
Prejudice
Orderooi -lep projuctios
Ordered deep projudice
600
0.50
8100
0.55
0
6.65
0.30
of
Figure 3: Extensive phase diagrams of infinitely wide MLPs, where we can observe some phases The EOC is indicated with a continuous red line and it becomes a single point
described in Tab. T for ReLU (unbounded). In general, red lines indicate the transition between vanishing/exploding gradients.
Table Phase descriptions with IGB and MF order parameters.
IGB
MF
Phase
81 <
Ordered-deep prejudice
7 00
c=1
X1 =1
Transient-deep prejudice (EOC)
X1
Chaotic-deep prejudice
1 1<7 < 00
0.5 <e<c
X1>1
(chaotic) Prejudice
7 751
0.5
81 11
(chaotic) Neutrality
Tanh (bounded) and ReLU (unbounded). This analysis enables the construction of comprehensive phase diagram for these two illustrative cases, thereby broadening the range of phases examined in the preceding sections. A summary of these phases is reported in Tab. [
Bounded activation functions In this case, the signal variance is also bounded and the value of
fully delineates the ordered and the chaotic phases. analyzed in [Schoenholz al.] (2016), the chaotic phase of Tanh - characterized by gradient explosion and training instability induces shift the correlation fixed point to e 1, which is shown Fig. (right plot). Remarkably, the EOC exists for every of IR with non-trivial shape (Hayou al. 2019 (right plot of Fig.
Unbounded activation functions This case has been extensively analyzed by HHaoou et al. 2019
However, prior work has overlooked that, for ReLU networks, the correlation coefficient e() con- verges to c = 1 across the entire phase diagram (see Fig. 2 left plot), revealing persistent deep prejudice initialization. In App. H derive explicit recursive relations for the IGB metrics in ReLU networks, demonstrating that lim c!)) MF phases remain distinct, as in the bounded activation case: in the ordered phase, gradients van- ish; in the chaotic phase, gradients explode. Crucially, these two phases differ in their depth-scaling behaviour: in the ordered phase, the total signal variance converges and 7((1) diverges exponentially with depth, whereas in the chaotic phase, the signal variance diverges and (() follows power-law divergence (Lemma H.T Hence, persistent deep prejudice may arise via two distinct mechanisms.
while (1)) diverges. Nevertheless, the two
In the first, the total signal variance (o²,.) + o remains bounded while oyn tends toward zero; denote this phase ordered-des deep prejudice, owing its link with vanishing gradients. In the second mechanism, at least one of o² ( or oz diverges, causing network outputs to blow up and gradients to explode. We refer to this as chaotic-deey prejudice. Two independent order parameters are required to distinguish between these regimes.
Per-class gradient exploding. The behaviour of the signal in the chaotic phase unbounded activation functions has direct consequences for training with cross entropy loss. Indeed, large separation of output distributions makes the softmax to concentrate entirely to a single class (which depends the weights initialization), therefore causing strong per-class dependence of gradients vanishing/exploding (see Fig. and App. G).
8

---

Published as a conference paper at ICLR 2026
Bel.U
Clobal
Pascered
Unboorted cless
Figure 4: Layerwise gradients 4ab a function of the network layer, of a ReLU MLP in the MF regime computed on batch of CIFARIO 10. Left: we show the total gradients. Center: we show the most favored class (the towards which there is most prejudice). Right: the least favored class. In the chaotic phase, the gradients of the favored class are numerically zero. In App. Gie we explain the notation Lab for the gradients, and show similar figures different datasets and architectures.
,(()
,(0)
Therefore, depending the MLP architecture design and the initialization hyper-parameters (such as weight and bias variances), different behaviours can emerge at initialization. Specifically, network can become untrainable either by entering a persistent-deep prejudice phase (either characterized by vanishing or exploding gradients) or purely chaotic phase, in which the signal variance finite and gradients explode. The nature of the chaotic phase itself depends critically on the activation function. For bounded activations, the chaotic phase can give rise to either
prejudiced phase (0.5 < c < 1) or a neutral phase (c < 0.5), depending on the initialization parameters. In contrast, for ReLU, the chaotic phase leads exclusively to chaotic-deep prejudice, where biased initial predictions are coupled with dynamical instability due to gradient explosion. Therefore, successful training requires finely tuning the initialization precisely sit at the transition between these phases - the transient-deeep prejudice phase (equivalent to EOC) - where gradients are stable and both persistence and instability avoided.
6 TRAINING DYNAMICS
we train several vanilla architectures (MLP, residual MLP, Vision Transformer) on binary (binarized Fashion MNIST, binarized CIFAR1O) and multi- class classification tasks (CIFAR10) across the phases described in Tab. Extensive details on the training and the models employed are reported in App. B order evaluate the evolution of bias and overall performance during training, we report the global accuracies, and the maximum classi- fication frequency at each training step. For a Tanh MLP. we can observe the dynamical behaviour described in this work (Fig. [5 top). Indeed, the EOC corresponds to the maximally biased state, but this bias is rapidly absorbed at the beginning of the dynamics. As expected, the unbiased state (neutrality) performs poorly and it cannot retrieve high accuracy.
In order to experimentally verify Prop. 4.1
As an example of more complex architecture, we study empirically a vanilla Vision Trans- former (Dosovitskiy al. 2020), where remove all batch- and layer-norm and skip connections, which may destroy the distinction between phases (indeed, it is easy to show that e.g. residual MLP has only critical phase of optimal learning (Yang Schoenholz (017)). Our simplified
Vision Transformer posseses only convolutional, linear and attention layers, and it shows the same transition behaviour of the gradients as the MI.P (Fig. (27
Finally, we validate our assumption by fine-tuning out modifications) pre-trained on ImageNet (Deng et al.) 2009) (Fig. - bottom). By multiplying all linear and convolutional layers' weights by a factor owe we trigger the phase behaviour which we observe in vanilla models. The optimal training state, corresponding to the original non-scaled model, exhibits weak IGB (i.e. maximum classification frequency is significantly greater than 1/ng, where n, is the number of classes). However, slightly reducing all weights by factor of o² = 0.5 triggers strong IGB phase, while larger scaling factor (o² 1.5) reduces IGB, yet hindering the
CTFAR100 large Vision Transformer (with-

---

Published as a conference paper at ICLR 2026
Tanh MLP trained on binarized fashion MNIST
Ontirdl Dopp Projudioe (o) 1.50)
Edge Ciaace [e] 1.00)
Projiaation (e2 2.48)
Nestrality (o) 4.97)
Triiiin
Test
Max Broq. Aucunicy
1.0
0.5
1.0
10P
10
10P
10
L0
10
102
10a
10*
Vision Transformer fine-tuned on CIFARIO0
Ordered Deep Prejedim 0.5)
Falge Of Choow (e] 1.0)
Chostic Deep Prejudion (n] 1.5)
Max Preq. Accucecy
0.5
0.0
100
10-1
10P
100
10
Number of troining of steps
Figure 5: Accuracies and max classification frequency for Tanh MLPs trained on binarized fash- ion MNIST, and the large Vision Transformer fine-tuned on CIFARI00. The EOC corresponds to both the initial state with the fastest learning dynamics and maximally biased state. The green line corresponds to 1/ne, where ne is the number of classes.
training dynamics. In both cases, the dynamical instabilities encountered either due vanishing or exploding gradients cannot be absorbed in the early phase of learning.
7 CONCLUSIONS AND OUTLOOK
We established an equivalence between two independent frameworks for the analysis of wide net- works at initializaton: mean field theory (MF) and initial guessing bias (IGB). We showed that the fundamental quantities in the two approaches can be mapped onto one another. This connection offered us the possibility to reinterpret the order/chaos phase transition in light of classification bias; the ordered phase is characterized by persistent-deep prejudice, while the chaotic phase is either characterized by persistent-deep prejudice, prejudice or neutrality. Furthermore, our categorization of the edge of chaos (EOC) as a state with deep prejudice reveals that the best trainable model nec- essarily exhibits bias, which, however, rapidly disappears in the learning dynamics.
Our findings have important implications for understanding the role of architectural choices and choices in shaping the onset behaviour of deep networks. They suggest that even before training begins, design decisions can inject systematic biases that impact signal propagation, gradient stability. and ultimately trainability.
Overall, our work provides a new lens to understand how dataset randomness at initialization, cou- pled with architectural design, shapes the phase diagram of large MLPs at initialization. We hope this connection between MF and IGB inspires further exploration of the subtle interplay between structure, randomness, and learning dynamics in modern neural networks, well of the condi- tions for reabsorption of initialization biases.
Practical Takewaways Finally, we highlight two practical takeaways of our work. First: neutral initializations are inefficient, so typical initializations are prejudiced. This means that, order to be meaningful, hyperparameter tuning runs need to be long enough to reabsorb IGB. As our runs show, the EOC not only optimal from the point of view of stability of the gradients, but also in terms of how fast the biases are reabsorbed. A consequence is that, if one tunes the weights' variances in order to be at the edge chaos (as explained Sec. 4. shorter HP tuning runs are needed. Second: since initializations are biased, the gradients of the favored classes will typically be smaller than those of the unfavored. This is especially true with gradient exploding in the chaotic deep prejudice phase, where one class has exploding, while the other(s) have zero gradients (Fig. 4.. Such disparities in the per-class gradients can have drastical impacts on the learning speed and quality, as demonstrated in Francazi et al. 2023).
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide the full code used to replicate all the simulations and training experiments at the link: bttpss:/gg trg..hhb.coo com/ abassi 98/1gb. and tra www.naaballlg.... We declare the us- age of Large Language Models for finding related work.
ACKNOWLEDGEMENTS
This work was supported by the Swiss National Science Foundation, SNSF grants # 208249 and 196902.
REFERENCES
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpps://prrcceeddnggss neurips. wttp:/rrrppsccppppprrp.ppppp httppeer_filleepppprrree Eil Flle2258.cccrr000511116662222222
pdf
Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and mean field theory RNNs: Gating enables signal propagation in recurrent neural networks. Jennifer Dy and Andreas Krause (eds.). Proceedings the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 873-882, PMLR, 10-15 Jul 2018. URL WtRLLtttps.:/prceeddii
mlr press v80 cwwnc8i. html
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Emanuele Francazi, Marco Baity-Jesi, and Aurelien Lucchi. A theoretical analysis of the learning dynamics under class imbalance. In International Conference on Machine Learning, pp. 10285- 10322. PMLR, 2023.
Emanuele Francazi, Aurelien Lucchi, and Marco Baity-Jesi. Initial guessing bias: How untrained networks favor some classes. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 13783-13839, PMLR, 21-27 Jul 2024. URL https: :ettss//pree h ptttps://proceeedngs.or m... STess//225/Errncett
html
Emanuele Francazi, Francesco Pinto, Aurelien Lucchi, and Marco Baity-Jesi. Where you place the norm matters: From prejudiced to neutral initializations. arXiv preprint urXiv:2505. 1312, 2025.
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, and Jeffrey Pennington. Dynamical isometry and a mean field theory of Istms and grus, 2019. URL http::
//arxiv.org/atl /001 08987
11

---

Published as a conference paper at ICLR 2026
Soufiane Hayou, Araad Doucet, and Judith Rousseau. On the impact of the activation function deep neural networks training. In International conference on machine learning, pp. 2672-2680, PMLR. 2019.
Arthur Jacot, Franck Gabriel, Francois Ged, and Clement Hongler. Freeze and chaos: Ntk views on dnn normalization, checkerboard and boundary artifacts. In Mathematical and Scientific Machine Learning. pp. 257-270. PMLR, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 589, Aug 2021. ISSN 1476-4687. doi: 10. 1038//41186-021--5550 URL (tLLhtts::/c/om /ww: org 10. 10 1038/ 44158 021 03819-2
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https://arxiv.. org/ abs 1412. 6980
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes, 2018 URL hhttps://arrat
org/aba/1717 00165
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. Pmlr, 2013.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray. Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831 PMLR, 18-24 Jul 2021. URL htt.:/eeiiiiisa.mmm..reeeee/eees
ml www.spressssee wttpppsssss..eeeeeeeeepeppp http:/33/rrmmam222aam1 html
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprint arXiv: 1ol:::611..233 2016.
Charles Stein. Inadmissibility of the usual estimator for the mean of a multivariate normal distribu- tion. In Proceedings of the Third Berkeley Symposium on Mathematical Statisties and Probability. volume 1, pp. 197-206, Berkeley, CA, 1956. University of California Press.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st Inter- national Conference Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Ross Wightman. Pytorch image models. pytorch- wwwgimmmdeeeeeeeeee 2019.
https://githil htwp:/tttthhhoo..oae.eoommooottt wwt.wwigttaaaa//////p/pp.oo..aat
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and mean field theory of CNNs: How to train 10,000-layer vanilla con- volutional neural networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learn- ing Research, pp. 5393-5403 PMLR, 10-15 Jul 2018. URL Actps: bttps::/ppoceeee//ee http::///cceeeeen..../m mmlr. areas/v80/ss x
8a html
12

---

Published as a conference paper at ICLR 2026
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:ttts:///rrceedinng new. hatpsaaseoparrrso...p.pprprrrrrr files/
Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. In Н. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpps://procesddnng neur nwwrips.rcccmm cttporrrpsccpppperrrr_eee firrffiless Pappe
Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture, 2020. URL https:
www.:/rggabb///0/e/ 14548
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural net- works. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737 PMLR, 18-24 Jul 2021. URL httph::////// httpp::/ppoceedee ngs m r...ss
v139/yang2lc..ham html
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. A mean field theory of batch normalization, 2019. URL https::/arxiv..ttt wwwpwwwwwwwww/////..2.2.. 08129
Ángel Poc-López and Miguel Aguilera. Dynamical mean-field theory of self-attention neural net- works, 2024. URL https: arxiv. wwtp://///a.bbb.....b./ 07247
13

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIALS
Extensive notation
15
Training experiments
17
MLP
17
B.2
Residual MLP
19
B.3
Vanilla Vision Transformer
20
Large Vision Transformer
C Mean field results
22
D IGB extension to explicit initialization biases
25
The equivalence between MF and TGB in the large-width limit
28
MF/IGB equivalence for production architectures
34
F
MF/GB extension to multi-node activation functions: the effect of pooling layers
36
MaxPool
38
Average Pooling
40
Phase diagrams
Gradients at initialization
43
MLP
43
G.2
Residual MLP
45
Vanilla Vision Transformer
45
Large Vision Transformer]
Example of an explicit IGB calculation for the ReLU
48
14

---

Published as a conference paper at ICLR 2026
A EXTENSIVE NOTATION
Px (z'): Given random variable X, Px (2') denotes the probability density function (p.d.f) evaluated at X = x'. Formally, Px (a') - fx (x)
For variables with multiple sources of randomness, if some of these sources either fixed marginalized, we will specify the active sources of randomness as subscripts.
erf (-): Error function. erf (-) /े So e- dt
Fx (z): Given a K... X, we denote its cumulative distribution function (c.d..) as Fx (I), i.e., Fx (x)) TP (X < x). Considering the X(D, W), which is function of two independent sets random variables 'D and W, when one these sources of randomness is fixed, we explicitly indicate the active source in the notation. For example, FX (x) P (X < x WW)
(z: (e))..²):: d Given a Gaussian E. X, we indicate with N (zi 0?) the p.d.f. computed 252 (m-p)2
X z, i.e. N (z; o²) Px (x)
<2x0²
ēp: the Dirac-delta function.
Oz: the Heaviside-theta function.
+ (x) 1 crf())): Gaussian cumulative function.
Dy
v== e /2 denotes the standard Gaussian measure.
A: generic neural network archiecture.
L: number of layers.
Ny: number node of layer
No: network's input dimension.
{a) E No; a-th data sample.
Np: number of dataset samples.
D: dataset, i.e. collection of ND data samples, i.e. {{(4)]u=1....N>>
(x): average of x w.r.t. the input dataset, i.c. (x) = Ep (x|W).
x: average of x w.r.t. the weights W, i.e. = Ew (r/D).
Varæ (-): Indicate the variance of the argument. Since we have æus with multiple sources randomness where necessary we will specify in the subscript the source of randomness used to compute the expectation. For example Varp (+) = (1))²
Cov (s+): similar to Var () but for the covariance among different variables.
( weight matrix at layer I, i.e. matrix of elements w!)) connecting neuron at layer / with neuron i at layer [ + 1.
b(O) bias vector at layer 1, i.e. vector of elements 1,", corresponding to neuron
o variance of the weights. Each weight W!(h) is i.i.d. as Normal with zero mean and variance o
of variance of the biases. Each bias b(" isi.d. as Normal with zero mean and variance of.
W: shorthand notation for the set of all network weights and biases, i.c. for the MLP: (W(II) b()))_,,....
Ní: number of nodes in the I-th layer; No = d indicates the dimension of the input data (number of input layer nodes) while N₂ the number of classes (number of output layer nodes).
G.: fraction of dataset elements classified as belonging to class c. The argument M indi- cates the total number of output nodes for the variable definition, i.e. the number of classes considered. For binary problems we omit this argument (G₂) as there only one non-trine trivial possibility, iee. M 2.
15

---

Published as a conference paper at ICLR 2026
ne: number of classes.
V/ (a): pre-activation neuron at layer 1, computed by propagating the a-th data sample.
lab byyy
(1) (b): the covariance of the pre-activation signals the /-th hidden layer nodes, corresponding to inputs and b, computed before applying the activation function.
( (8) (a)y,
Qao 6,yy, ( (a)u, (a): the variance of the pre-activation signals the 1-th hidden layer nodes, corresponding to imputs computed before applying the activation function.
(1)
(!)
correlation coefficient between the pre-activation signals layer' corre-
V (2) the
sponding to inputs and computed before applying the activation function.
X1 = =d c=1 derivative of the correlation coefficient at layer + w.r.t. layer I, at the fixed point с = 1. This value determines the gradient vanishing/cxplosion for bounded activation functions.
(1+1) / derivative of the covariance among two inputs at layer / + w.r.f. layer 1, at the fixed point e 1. This value determines the gradient vanishing/explosion for unbounded activation functions.
X1 = =
2 4 VarD (vs (v!")) signal (pre-activation) variance w.r.t. the dataset layer I. In the Supplementary Material, we show that this quantity does not depend on neuron i.
by you = Varw (( (0) )): signal (pre-activation) variance of the signal centers, i.e. the
signal averaged w.r.t. dataset.
((()) = / the ratio between the signal variance w.r.t the dataset and the variance of the signal centers w.r.t. the weights ensemble.
( (a) = oh!
(a): derivative of the loss w.r.t. the pre-activation.
lab = 5, (a)o (b): correlation between the gradient computed with inputs a and b.
16

---

Published as a conference paper at ICLR 2026
B TRAINING EXPERIMENTS
In this section, we provide detailed information the training experiments we performed, reported in Tab. 2 We employ the following datasets: binarized Fashion MNIST (BFMNIST), binarized CI (BCIFAR), CIFARIO, CIFARI00. We binarized Fashion MNIST and CIFARIO by clas- sifying odd versus even classes
Architecture
Depth
Width
Act. Function
lr
Dataset
MLP
100
1000
Tanh, ReU
10 10-7
BFMNIST
MLP
100
1000
Tanh, ReLU
10
CIFARIO
Residual MLP
100
1000
Tanh, ReLU
10-7
BEMNIST
Vanilla VIT
100
300
ReLU
10-5
CIFAR10
Vanilla VIT
100
300
ReLU
10-5
CIFARIO
Large VIT (~ 300 M params.)
97
5000
GeLU
10-3
CIFARI00
Table 2: Details on the training experiments performed in this work. The learning rates for each
rows (Ir) refer different datasets.
We train all the models employing the Adam optimizer (Kingma Ba 2017) without any regular- ization. The training and test batch sizes are both set to 100. The learning rate is chosen according to the model and dataset, as reported in Tab. 2 In the next subsections, we report the global ac- curacy, the accuracy of the most favoured and the most unfavoured classes during training, and the maximum classification frequency, all computed at each training step. In particular, we observe the phase behaviour claimed in Prop. 4.. phases that are the fastest to train correspond to maximally biased state, with deep prejudice.
B. MLP
We train vanilla MLP (Eq. (II) with ReLU and Tanh activation functions binarized Fashion MNIST and CIFARIO. On BFMNIST, we can reach high accuracies (Fig. 6. Fig. 7. while the vanilla MLP reaches most around 50% test accuracy when trained CIFARIO (Fig. and Fig. D. We argue that this is due to the simplicity of the architecture, and training would require aedeeadjuuentt the phase distinctions in the early phase of learning, with the EOC being the fastest to absorb the bias. Clearly, in the chaotic phases (chaotic-deep prejudice, prejudice and neutrality), learning slowest where the initial ignorance of the class (i.e. neutrality in class assignment) hinders training. The ordered-deep prejudice phase seems to reach eventually the same performance as the BOC, but with sudden performance jump. We argue that this is combined effect of the optimizer employed and the convexity of the loss landscape, typical of over weer-paametrrzed models. In particular, when gradients vanishing (as it happens in the ordered-deep prejudice phase), momentum-bart based optimizers like Adam can quickly navigate almost-fia loss landscapes, eventually reaching a (sub)- optimal plateau.
more adjustments perform well even for more complex datasets. However, we can still observe
17

---

Published as a conference paper at ICLR 2026
Ordered Prejudica Of
Chastic Projudicn
Training
Test
Glabel
1.0
0.5
1.0
Fav.
0.5
Unta:
Max Peeg
1.0
0.0
training steps
Figure 6: Global, favoured and unfavoured class train and test accuracies for ReLU MLP trained on binarized fashion MNIST.
Ordied
Clace
Prejuciior
Nestrality
Training
Tedt
Global
1.0
0.0
1.0
Rav.
0.8
Max Freg.
1.0
0.0
training steps
Figure 7: Global, favoured and unfavoured class train and test accuracies for Tanh ML. trained on binarized fashion MNIST.
Ordored Deep Prejudice (o) LM)
Of Clooe (e]
Chactic Deep Prejudice io? 3.0)
Training
Test
Globa
0.0
1.0
0.5
Unba::
0.5
Max Froo.
100
10-1
10P
101
13P
10P
104
Number iraining of steps
Figure 8: Global, favoured and unfavoured class train and test accuracies for ReLU MLP trained on CIFAR10.
18

---

Published as a conference paper at ICLR 2026
Onlieed Dwep
Prejuction
Nestrality
Training
Tist
Global
0.5
1.0
Untar.
Max Peep
100
10-1
training steps
Figure 9: Global, favoured and unfavoured class train and test accuracies for Tanh MLP trained on CIFAR10.
B.2 RESIDUAL MIP
We employ residual MLP studied in the MF regime by Yang & Schoenholz (007) where the residual branch rescaled by the total network's 's length in order to avoid trivial signal explosion. In particular, Yang Schoenholz (2017) showed that residual MLP are always critical, i.e. there is no order/chaos phase transition; the correlation coefficient always converges to one at initialization and gradients are always stable (see Fig. 266. In this work, we corroborated this result by training Residual Tanh and ReLU MLPs on binarized Fashion MNIST (Fig. I0 and II respectively). Since these models are always critical, the experiments show deep prejudice at beginning of learning, which is absorbed at the same time in all the models, producing high classification accuracies.
Ordored Deep Prejudice (o) L.
Of Close (e]
Chactic Deep Prejudice io? 2.5)
Truining
Tost
Global
1.0
0.5
1.00
Fat.
Max Freg:
1.0
0.5
10P
10a
Number iraining of steps
Figure 10: Global, favoured and unfavoured class train and test accuracies for a ReLU Residual MLP trained on binarized Fashion MNIST.
19

---

Published as a conference paper at ICLR 2026
Onliend Deep Projucios
Edge Clace
Prejuctton
Nestrality
4.37)
Training
Tost
Glabel
1.0
1,00
Fay.
0.70
Max Peeg
1.0
0.0
training of steps
Figure 11: Global, favoured and unfavoured class train and test accuracies for a Tanh Residual MLP trained on binarized Fashion MNIST.
B.3 VANILLA VISION TRANSFORMER
We employ a vanilla Vision Transformer (VIT) Dosovitskiy al. 2020) for classification trained
on binarized CIFARIO (Fig. [2) and CIFARIO (Fig. 13. We climinate all batch- and layer-norm layers, since they not included in our theory and could potentially destroy the phase distinctions. We preserve only linear and convolutional layers, which we initialize with the prescription this work. In particular, this assumption justified since the MF theory of convolutional layer identical to that of MLP (Xiao et al. (018)). We initialize the attention matrices as the linear layer, with the gain depending on the activation function placed before the activation. This way, we show that gradients manifest the same transition behaviour observed in vanilla models, where an exact theory available (see for instance Fig. 27).
Ordored Deep Prejudice (o) L.0) Traiing
Edge Of Chose (e]
Chactic Deep Prejudice io? 2.0)
Test
Global
1.01
Far.
0.3
Max Btog. Uafax.
0.5
1.0
0.5
10P
10
10P
10P
1nt
102
107
104
Number iraining of steps
Figure 12: Global, favoured and unfavoured class train and test accuracies for a ReLU Vision Trans- former trained on binarized CIFAR10.
20

---

Published as a conference paper at ICLR 2026
Ordered Prejudica
Of Clase Chastic Projudico
Training
Tist
Glabal
0.50
0.20
1.0
Max Peg. Unfac.
0.25
0.00
100
10-1
training of steps
Figure 13: Global, favoured and unfavoured class train and test accuracies for a ReLU Vision Trans- former trained CIFARIO.
B.4 LARGE VISION TRANSFORMER
Finally, we validate our assumption by fine-tuning a large model, without any architectural modifi- cation. We download pre-train large Vision Transformer (vit large patch16 384) from the library Timm [Wightman] (019). This model was pre-trained on ImagenNet-21k, and consists of about 300 million parameters. We fine tune this model on CIFAR100. We multiply the weights of all linear and convolutional layers by factor o² with the goal of triggering the order/chaos phase transition studied in this paper. This way, the pre-trained model behaves like an untrained model when eval- uated on statistically different dataset. Indeed, we show that gradients behave like at the phase boundary between order and chaos (Fig. (28). We report the training dynamics in Fig. [4] where we observe that the original unscaled state (at EOC red line) manifests the fastest learning dynamics. In particular, it begins with moderate prejudice and the only state significantly improving the classification accuracy of the most unfavoured class. Instead, the state rescaled with of 1.5, ideally in the chaotic phase, begins with lower amount of prejudice, but cannot recover. Finally, the state rescaled by of 0.5 shows strong initialization prejudice, typical of the ordered phase, but smaller gradients (Fig. 28 which hinder training.
Ordeted Deep Prejuuicr (e] 0.5)
Dige Of Clase (o! L.)) Chuotic Deep Projudice to? 1.)
Trainnn
Test
Global
0.5
0.0
Fex.
Under
0.5
Max Freq.
100
10
10P
10"
10'
10a
Number of training of stops
Figure 14: Global, favoured and unfavoured class train and test accuracies for a large pre- traiindeed Vision Transformer fine-tuned on CIFAR100.
21

---

Published as a conference paper at ICLR 2026
C MEAN FIELD RESULTS
In this section, we report some results of previous MF theory, starting from wide MLPs (Poole al. 2016 Schooenholz et al. 2016 Hayou et al., 2019). We report the derivation of the following recursive equations for the signal variance and covariance
! d / Dy @ (ui-1))² 2 + of
/ Dyy Di o (u(t-i)) o (ul0-11) + of
Aab = dm
(8)
where Dy
dy e v² denotes the standard Gaussian measure, and V2n
(1) w0) = V Yaa y
(9)
a(()) V
( 4bb
Cab y
(
V
(ca)) y2y
(10)
To prove them, let us consider the signal propagation through a MLP, which reads
N;
v (a(())
10O0
(a)= + 6,00
(11)
where for the first layer we have
d
v (a)
(12)
where Ey(a) is the j-th component of the a-th data instance. We consider the ensemble of MLPs over weights and biases (W) initialized according to the following scheme:
Pway (x) = N x;: 0,
wil
Vi,j I....,N
(13)
NI
Pogo (x) = N (2; 0, )3)))
1.......,, N.
(14)
The signal variance is defined as
N Σ N
(1)
M (a))
(15)
In the limit of large width (N + oo), the distributions of the pre-activations y/") converge i.i.d. zero mean Gaussians, since weights and biases are all independent across neurons and layers, and (1) yi is a weighted sum of large number of uncorrelated random variables. This treatment is valid as long as we do not impose any distribution on the input dataset D, but consider only averages over W. By applying Eq. (TT) Eq. (15) and using the definition of the weights and biases distribution
(Eqs. (13 and 14), we easily get
N Σ Φ ( (a))
2 + +
Pas o
(16)
Since the empirical distribution across the layer / - 1 is zero mean Gaussian with variance given substitute the empirical distribution with an integral
by qoa ((-11) in the large-width limit, over a Gaussian variable. In this regime, the distribution of signals across neurons of a single MLP converges to the distribution of signals of single neuron across the random ensemble; this is known as self-averaging assumption from statistical physics of disordered systems (which is formally true in the large-width width limit). This Gaussian variable can be re-parametrized and finally we get an integral over standard Gaussian variable
/ Dy o (ull11) + 00.
(17)
22

---

Published as a conference paper at ICLR 2026
The covariance among inputs is defined
N
1 N
Σ
n (o)v: (b). 9i
(18)
The joint empirical distribution of y; (1) and ys converges at large N to 2-dimensional Gaussian with covariance Lab Similarly as for the signal variance, we can find a recursive equation for Lab as
(1)
[1] Jab
d / Dy Dij @ (ull- @ (u/(-1)) + of.
(19)
Let us denote with q the limiting variance in its domain of convergence (Hayou et al. 2019).
We
can show (Schoenholz et al. 2016) that:
Dcubt (1+1)
fDy ø' (Vqy)² =
X1
11+1)
(20)
cab
c=1
and
aqa (1+1)
X1 + dn / (vQ((V (VVTu)
a =
=
(21)
Qqui
In previous MF works, it not clear the role played by X1, because it is usually derived by assum- ing the variance to converge faster than the correlation coefficient (Poole et al.) [016]] Schoenholz al., 2016)) Here, we prove that when the variance is not assumed to be constant, the result slightly changes, suggesting that for some activation f'unctions, the asymptotic correlation coefficient is not able to discriminate between phases. First, we prove the following Lemma.
Lemma C.1. If the variance does not converge, we can compute
(())
f Dy Øf (VaTh))
(22)
q[(+3]
where (1) = q²), Va D.
Proof. Let us compute 86t5 For generic activation functions, Yea may diverge with depth and
a (++1)
(1)
thus cannot safely kept constant as done in the MF literature for bounded activation functions
Schoenholz et al., 2016). Moreover, due to main result (reported Thm. E.T. for large N, (1)
qaa 4sh Va, b E D, VI; thus, we can write qaa = q0), Va E D. From Eq. we can directly
(1) (1)
calculate:
(111)
2 Dy Dy o (w) (w) v 9
/
ab
ah
00
((++1)
(23)
V
(c )
where ( ( have been defined in Eqs. and 10
Next, we proceed using the following key identity known as Stein's lemma (teein 1956):
f [Du F(y)== / Dy F'(y).
(24)
which holds for any function Fly), where y is a standard Gaussian variable (zero mean and unit standard variance).
Using this key identity and the definition of ( and (1 in Eqs. 9 and 10 I- 1-dependency for simplicity)
get (omitting their
f Dy Dy o (u) o' (u') = vo / Dy Dej of (w) (v)) (o) (0 (1)))
(25)
I Dy Dij (1) o' (1)))) = -1- (c(8)) (2 D Dy ((w)d"(v))
(26)
23

---

Published as a conference paper at ICLR 2026
Therefore, by combining Eqs. 25
and (26 with Eq. (23). we get
acab g1))
I Dy Di of (w)
q(++) o
(27)
(1) ob
At the critical point c the former expression further simplifies
(e(+11)
q0) g X1
Xi
(28)
dcll)
c=1
where
ag 11+1)
= o f (Dy Dy o' (u())
x1 =
(29)
dg
c=1
L
Specifically, for ReLU activations we can prove the following Lemma.
Lemma C.2. For ReLU, it holds that
of
X1 =
(1+1)
(30)
Proof. For ReLU we get Dy 16' (u!))}² as long as the variance g0) is finite, and ((++1)
0 ( + oof which implies that
?
Xi
(1)
q ((+
(31)
and the fixed point c= = 1 is never repelling. Only for bounded activation function, the variance q() always converges, therefore X1 = o2 f Dy (Øl (u)]² and the definition agrees with MF theory. Hence
if of = or o 2 2, 2, Vo
lim Xi 1000
(32)
else.
By using the IGB framework. in App. H we show that this implies that for ReLU the correlation coefficient (respte. 7(0) converges exponentially (respet. diverges) in the order phase, while the chaotic phase (and the line of = 0) is all critical and the correlation coefficient converges sub- exponentially.
In App. we find explicit recursion relations for quantities of interest for ReLU by using the IGB approach, corroborating the divergence behaviour of ()) different phases.
24

---

Published as a conference paper at ICLR 2026
D IGB EXTENSION TO EXPLICIT INITIALIZATION BIASES
The standard MF approach takes into account only source of randomness, coming from the network ensemble, whereas the input is fixed. Here, we extend the analysis to the case where the dataset randomly distributed. For simplicity, we suppose that each datapoint follows a standard Gaussian distribution, i.e. {a(a) ~ (O 1). For random datasets, it is meaningful to define an averaging operator over fixed weights and biases.
Definition D.1 (Averages over the dataset), The average over data D fixed weights and biases W is denoted with (x) = Ep (x:l).
Ref. Francazi et al. 2024) derived the pre-activation distributions of a MLP processing a random dataset in case of zero explicit initialization biases (Thm D.2, Appendix). Here, generalize it to accomplish non-zero initialization biases.
Theorem D.2 (IGB pre-activation distributions). When averages over the dataset are taken first, in the limit of infinite width and then data, the pre-activation yi are independently Gaussian distributed as:
(D) p (x) = N (a;11) 1 o vi -i= 1, 1,..,,,,
(33)
(1) where Og the node variance. {u Ji=1....N are independent random variables which de- pend on W only and are distributed according to zero mean Gaussian distribution:
(W) 00 (x) = (2; (z:0, 0, -ii=
(34)
where oy is the variance of the centers of the node signals.
Proof. The result in Thm extends directly from prior work that considered the same setting but with zero bias terms. In particular, Ref. Francazi et al. (2024) proved that under the assumption of i.dd. Gaussian data, fixed weights and large layer, the pre-activations
(1) 0i
Σ
W
i.
U3
(35)
i.i.d. normally distributed with mean n and variance a2 (n When considering the variability over the weights, p! is also normally distributed with mean zero and variance 2000- Moreover, 220 self-averaging with respect to the weights, i.e. oz = - To extend this to the setting with non-zero bias terms, observe that the bias enters as an additive random variable that is itself Gaussian and independent of the weighted sum Eq. 35 Therefore, the resulting pre-activations remain Gaussian, as the of independent Gaussian varialles is Gaussian.
The key difference compared to the MF approach is that, in the IGB approach, the pre-activation distributions are not centred around zero. Moreover, the variability of the dataset can be captured solely by the variance of the nodes of whereas the ensemble variability is fully characterized by the variance of the centers o2
25

---

Published as a conference paper at ICLR 2026
Lemma D.3 (IGB recursion formulas, informal). For a general MLP (Eq. ), the signal variance and the centers variance satisfy the following recursive equations:
oy yu+) = o. VarD (v(v)))
(36)
o mll+1) o (o (g())) + to
(37)
with initial values o g00
0. Moreover, VarD (o (y()) is self-averaging with respect to W, that is VarD (o (y())) VarD (o ()y"))).
02 4(0)
Proof. We now want to prove the recursive relations for Lg and 2n0, iee. Eq. (36) and Eq. (37)
respectively. By defining d & (u)")) we compute the covariance (with respect the input data) of the generic layer as:
N Σ
COVD
Vi
(1+1)
131 ,Bg
(111))
w
w
(d) (2)
W
W
(1)
6
W
j.h
(1)
k,p=1
+ E
(11)) (0x0)
Σ
wilh
(1)
(1)
W (1)
(1)
(1)
(6
i,k
0k
5.k
9k
R,p=1
1=
Ar
CovD
COVD
0/ : 600
k
(38)
Ref. Francazi al. (2024) proved that in the large width limit VarD (v.²) is self-averaging (as distribution of the weights) and does not depend on i, ie.
lim VarD (0(")
VarD (2(1)
(39)
N-500
and that Cov (of), . 00 = 6p VarD (o()).
Self-averaging implies also that
lim Et.p=1 w(l w:( 1,k i.p
= oZ, which together with Eq.
38 yields Eq. (36).
Now let us consider the calculation for o² m From Eq.
we easily see that (v. ((11)) and
Vi
Σ
w (000)
+ -09))
Σ W (1) ( + 6000) )
(40)
N
W inj i.k
will)
(09) ( (000
+ (0) )2 =
(41)
1Ak=1
= 0 (0(1)) 2
+ oh
(42)
which is Eq. 37
26

---

Published as a conference paper at ICLR 2026
Lemma D.4 (Fraction of Inputs Classified to Reference Class), Given a fixed initialization W, the fraction of inputs classified into reference class 0 is given by:
(V
Go = (v) 1 ((((1))))))))))))))))) =
(43)
(L) (1)
where V1 1/2 are the two oulput node pre-activations at layer L, is the Gaussian cumu-- lative function, and 5 is standard Gaussian variable.
(L) (L) follows Normal distribution with mean #1 12 and
(1) (L)
Proof. From Lemma D.2
y1
(L) (L) follows Normal distribution centred around zero and with
variance 20 ( Moreover, Pi u2
variance 2o² atz). Therefore
8(W)) = -1- 1 (11 122 (L)
M1
(1)
ub
(E)
(44)
20 y(2)
where is the Gaussian cumulative function. By reparametrizaatoon 11th
22a
w 6,
where 8 is standard Gaussian variable. By definition ((x) [1 erf(x))) where erf is the error
function. Since crf(-x) crf(2r) and 7(t)
finally get
1/(0)/ )) (VoL
1 1 erf Go 2
(45)
27

---

Published as a conference paper at ICLR 2026
E THE EQUIVALENCE BETWEEN MF AND IGB IN THE LARGE-WIDTH LIMIT
Theorem E.1. Consider a generic architecture A (Eq. (T) in the mean-field regime. Let us suppose to fix the initial conditions of IGB and MF to be equal, i.e. qao = 1, Va E 'D and (0) lab = 0, Va, b E D, with + b. Then in the infinite width and then data limit, and for every layer > O, the total variance in the IGB approach equal to the signal variance in the MF approach:
(0)
d =
ogo Ya D
(46)
Moreover, the centers variance the IGB approach is equal to the input covariance in the MF approach:
qab o , Va, D. bb.
(47)
Finally, the correlation coefficient is related to y as:
700 ab 1+70) Va b #b
(1)
(48)
Proof. We begin by relating the IGB parameters o² mno and o² g(t) to the signal covariance 9ab and (1)
(1)
variance qae From Eq. 33 the pre-activation for neuron at layer and data point a can be written
as
(a) #4 + gg 5, (a),
(49)
where & (a) are i.dd standard Gaussian variables for each neuron and data point. This parametriza- tion ensures
(1) (3 4 0,
(50)
(1)
(51)
( Since c (a) is i..dd. for each neuron, its expectation over the ensemble, e; (a), can consistently estimated, in the wide layers limit, by an average over neurons within the layer, i.e.
(a)
lim N N
(a).
(52)
As consequence of the central limit theorem, the random variable E (
e is distributed according to the density p, (x) N (ar; 0, 1/N) for large N, therefore self-averaging to zero in the infinite-width limit. With the help of a little Algebra we can write
Lub nu: + o s(,
(1) 2
(53)
((0) oa s(,
where we define
N
8 1 (), b)
n
(a)c (b).
(54)
N
(0) Rab
(1) A.) qua 426
therefore (!) ce meaningfully
is easy to prove that for every dataset element a, defines the correlation coefficient between inputs.
V
Lemma E.2.
( Cab < for every a. b e D.
28

---

Published as a conference paper at ICLR 2026
cab - 1, which is true if and only (2 Σ
(0)
(1) qua Lad which is
Proof. We want to prove that
equivalent to
20 als o² oos (a, + o (s) s o2 nn o (b
(55)
+ 09 g0ss (a, (a) (b,
From the Cauchy-Schwarz inequality gl a)s 2 (s (a, b]].. Moreover, with little Al- gebra we get
N N
3hh (a,,a)+s - (a,
Σ f
& 19 (a)
[ 2 0,
(56)
which together with the previous Cauchy-Schwarz inequality implies Ineq.
55
Now, we are interested in the distributions of ()) (1)
Lab and laa with respect to the dataset. 1 (a) and
(1(b) should be thought as two independent random samples from ( Consequently, < (a)e:))) (b) should be thought the product of two independent standard Gaussian variables, whose mean is zero and variance is one. From a computational point of view, this product is obtained by inde- pendently varying the imputs and b. Accordingly, e, e/" (a))("(a) (a) is the square of standard Gaus- sian, which follows a chi-squared distribution with one degree of freedom. Its mean is one and its variance is two. Therefore, random variables in function of a Gaussian distributed dataset. For large N, when a * b we have (a.M) (I) R N (z; 0, L/N), while for b, (a.)a
fully characterized the variance and covariance in MF as
(x) A (æ; 1,2/N)
It follows that in the infinite-widhe limit, the signal variance and covariance self- averagnrgg with respect to the dataset, i.e qab = ( (1)
(9ab and (1)
( qna
(())
(1) We plot the absolute percentage error for the ReLU of (1) gaa, and Lab as the network sizes in- creases in Figs. [5] 16 and 17 respectively. Moreover, we report the absolute percentage error of (1) (1) You. and Lab for Tanh activation (Figs. 18 Fig. 19 respectively).
(
Proposition E.3. From a trainability perspective, the optimal initial condition (stable gradi- ents) is not one of neutrality, but rather slute of transient deep prejudice.
Proof. Thm. E.I establishes the equivalence between MF and IGB in the infinte-width limit. In MF, optimal training conditions are to be found at the edge of chaos (EOC). In the IGB framework, the EOC corresponds to the deep prejudice state, since the asymptotic value of the correlation coefficient one. Moreover, this prejudiced state is transient since the EOC, gradients are stable and so the network can absorb the bias rapidly HHyouu al. 2019).
29

---

Published as a conference paper at ICLR 2026
Width
1300
2000
0000
10000
0 3.0
of 0.1
o
0.2
1.5
166
1.72
1.83
1.94
2
2.06
2.17.50
2.39
2.5
layer l
Figure 15: Absolute percentage eπrоr of the experimental versus theoretical values of 7(()) obtained for ReLU with different values of o close to the critical point o 2.0. The width of the network varies from 1000 to 10000. We observe a reduction of the relative error as the network size increases, corroborating the theoretical curves shown Fig. 31
30

---

Published as a conference paper at ICLR 2026
Width
10:0
2000
00.0
10000
of
0.0
0 0.1
of
0.2
1.5 50
50
1.72 50
1,83 50
1.94
6 2
2.05
2.17
2.28
2.30
2.5
100
100
layer 1
Figure 16: MF 90% confidence interval of the signal variance 2 percentage of the median for ReLU activation. We compute for single MLP with increasing width inputted with 100 random data samples. We observe that the percentage deviation from the median decreases as the network width increases, corroborating the results of Thm. B.
31

---

Published as a conference paper at ICLR 2026
Width
10.0
2010
00.0
10000
0.0
0 D.1
of
0.2
50
1.5
50
50
1.72
1.83
1.94
6
2.05
2.17
2.28
2:30
2.5
100
layer [
Lab percentage of the median for ReLU activation. We compute for single MLP with increasing width inputted with 100 random data samples. We observe that the percentage deviation from the median decreases as the network width increases, corroborating the results of Thm. E.
Figure 17: MF 90% confidence interval of the signal covariance (t)
32

---

Published as a conference paper at ICLR 2026
Width
10.0
2010
0000
10000
0.0
0 D.1
of
0.2
1.5
1.67
1.83
25
2.0
25
2.17
0
2
25
2.33
2.5
2.67
2.83
75
100
layer 1
Figure 18: MF 90% confidence interval the signal variance qaa in percentage of the median for Tanh activation. We compute it for single MLP with increasing width inputted with 100 random data samples. We observe that the percentage deviation from the median decreases as the network width increases, corroborating the results of Thm. B.
(1)
33

---

Published as a conference paper at ICLR 2026
Width
10.0
2010
00.0
10000
0.0
0 D.1
of
0.2
1.5
1.83
2.0
2.17
0
2.33
2.5 50
2.67 50
2.83 50
3.0 50
100
75
100
layer I
Figure 19: MF 90% confidence interval of the signal covariance Lab percentage of the median for Tanh activation. We compute it for single MLP with increasing width inputted with 100 random data samples. We observe that the percentage deviation from the median decreases as the network width increases, corroborating the results of Thm. E.I
E. MF/IGB EQUIVALENCE FOR PRODUCTION ARCHITECTURES
To validate our theory in further realistic settings, we compute the correlation coefficient with both the IGB and MF theories for two production architectures downloaded from the Timm Python li- brary: a RESNET with 18 layers (https: /ttps:://uuugtt htt.gggggg......ccoeooocooa co ct/cc///oooocoooo /wtp//oc//////cooccoooco htttttimmmennn//ee /t///////oooeee ss/ resnet Fig. 20 and MLP mixer with 100 layers (https:/ //hggg agface. co/t i mm/
in1k Fig. 2Tp. These figures should be interpreted in the same way Fig. 2 the equivalence between the two frameworks is demonstrated by the fact that do, has the same value independently of whether it is calculated through using the MF [Eq. (2) the IGB approach [Eq. (6)]. We employ the CIFAR 10 dataset for these experiments.
mixer_ble 6_166_2200 224 mi _e____n2k________
34

---

Published as a conference paper at ICLR 2026
RESNET
10
(I)
ab
0.0
-0.4
5.0
10.0
12.5
15.0
17.5
layer /
Figure 20: Behaviour the correlation coefficient of a production RESNET. Solid lines are computed using the IGB approach, while shaded areas represent the 90 % central confidence interval computed using the mean-field approach. The IGB solid lines are all within the corresponding MF interval, demonstrating the agreement between the two theories. Moreover, coherently with residual MLPs (see the Appendix B.2), we do not observe distinction in the predicted correlation coefficient at varying "..
MLPMIXER
10
0.2
(1) ab
0.0
-0.2
layer /
Figure 21: Same as Fig. 20 for a production MLP-MIXER with 100 layers at initialization. The agreement between IGB and MF good across the entire phase diagram, being stronger as we increase the scaling factor of the weights o²
35

---

Published as a conference paper at ICLR 2026
F MF/IGB EXTENSION TO MULTI-NODE ACTIVATION FUNCTIONS: THE EFFECT OF POOLING LAYERS
Within the MF literature, common architectural components such batch normalization Yang al. (019) and dropout [Schoenholz et al. (2016) have been extensively studied. In this work, however, investigate the impact of multi-node pooling layers on the phase diagrams a topic that has received comparatively little attention.
To this end, we leverage the IGB framework, which naturally extends the general case of multi-node activation functions. We start by deriving recursive equations for generic activation function Φ RR > RT, involving n nodes. Once these general recursive relations are established, the theoretical results from (Hayou et al., 2019) remain applicable. In particular, we can directly employ Algorithm from (Hayou et al. 2019) compute the EOC.
Lemma F.I. Let f R" + R" generic activation function of n. nodes. Then the MF recursive equations read:
(att) = o²
((+1)
II Dy; f(u)² +
(57)
lab
(L+1)
a²
Dy, Dy, f(u) + +o
(58)
where
(1)
u 9aa
(59)
(1) Cab + V yaby')
u
qea
(60)
Proof. For a generic function multi variable function R - R R we have
R
/
II
du, N (u.:0,07) / dy., N (Usip,, o2) f(y)
i=1
dy:
n
dup
42 (x-
20
20
f(y)2=
e
2
/200
i=1
(61)
(a, J,.
dy
v² 24
dp
II
20 10
f(y)²
2TTT²
im1
22TT3
72
dy
f
/
II Dy, f(vgy)2
n
2mq
where in the second equality we swapped the integrals over y and n4 in the third we completed the square exponent, in the fourth we performed the integration over pe and in the last parametrized the Gaussian integrals. Moreover
In this proof, we omit the /-dependency for better readability.
36

---

Published as a conference paper at ICLR 2026
/ II ==1
/
/Ilda
dy, N (Bri f(y) f
I1 dy N (Biip) f(y')
(fy))) =
du, N (u;:0, 02))
1=1
dy.
dy
dp
1 2
19
f(y)((r))
20
20
2
2m
2
dy:
dyj
N2 +u 2
207 (2-- (u tott
[(y)s(y))
22
+
v 2
(
(16-11)) 73. VITT!
42/2
dy
dy.
f(y)s(r))
V2y
+1
2
dij
di.
f(i)f(cao + V1 c2077"
E
2mqq
V 2
Dy, Dy fl + + (2-2))))
622)
where the second equality we applied the definition of Gaussian measure, in the third we com- pleted the square and integrated over μ. in the fourth we completed the squares on yis y/ and change the integration variables to
y 3'
(63)
+1
y
(64)
(2 +1 /2y +
Finally, we renamed the dummy integration variables and appreciate standard Gaussian integrals.
The result follows from the definition of 71),, e0 14700 conclude by recalling Lemma D.3
700
and 22
V
We
7041
Lemma F.2. Let f. RR - R. a generic activation function of n nodes. Then:
de (++1)
q())
| (vs(vve)))
g(0)
X1
de (1)
ob
((+1)
(65)
c=1
(1) 0q (-1)
t f DI / (Vowg) (
(Vgll))
(1)
(66)
(1(()
where 11-113 is the I,2 norm and A = Ei= oe is the Laplacian operator.
Proof. The proof is similar of that of Lemma C.1 where for generic multi-node function f R -> RR, Stein's Lemma reads
/ -=1
Du. f(y) u / II E=1
Dv, a.f(y)
(67)
We now prove some Lemmas regarding generic single node activation function ó () followed by 2-dimensional max- and average- pool layers. This analysis will allow us to draw the phase diagram for ReL.U and Tanh enriched with these pooling layers.
37

---

Published as a conference paper at ICLR 2026
F.1 MAXPOOL
Lemma F.3. Let f R² R22 2-node activation function that can be written the com-
position of MaxPool IR² + R2² with a single-node activation function & R - R. Then e satisfies the following conditions
1. All conditions Proposition 2 of the main paper of Hayou et al. 2019
2. (x) is either odd or even
ø (a:) is either odd or even
Then f MaxPool & exhibit the same EOC of o (z).
To prove this Lemma, we need to compute the following operator, which is defined for any multi- node activation function.
Definition F.4 (V operator). Let f R + I Then we define the following operator V,, acting on f, for any a E R as
vif(() o / / DI.DA.(VEY)
(68)
Note the the V operator can be used to compute the recursive equation of the variance for a multi- node activation function as gua = o V [f2]
V
qea
(I)
- oh
Lemma F,5 (V operator for 2-d MaxPool), Let () R I. a generic single-node activation function. Then the V operator of f = MaxPool can be computed as:
vif((>) = du / Duo (Voy) (y)
(69)
where ((x) = [ + erf(x)) is the Gaussian cumulative function.
Proof. From Def. F4
we have
v[ʃ((x) dz f Dyi Duz Max( (Voy)),((Vz))))
o (Vruz))
d'n / "uuiDus On (y1 - 62) (Vryi) + 0n (12 - -11) @ (Vayz) (1)((vzy))) = (70)
20 / DDD (Vayr) 1") Dyz= = ou / Dyo (Vzy) $ (y)
where in the third equations we exploited the symmetry between y/1 and and in the fourth we used the definition of Gaussian cumulative function. Finally, we renamed the dummy integration variable.
We can prove Lemma
Proof. From condition 1, we can use algorithm Algorithm of HHyou al. (2019) compute the EOC. Then fxx) exhibits the same EOC of & (z) if and only if V [
Vi (œ) and
Vl (z). Vr R. The operator defined in Def. F.4 It immediate verify
that condition implies V[f²((x) = [x))] and condition 2 implies (2) = (xx). O
We compute the EOC for ReLU and Tanh enriched with MaxPool layers. In particular, Tanh + MaxPool satisfies the hypothesis Lemma F3 so exhibits the same EOC as Tanh. For ReLU, first prove the following Lemma.
38

---

Published as a conference paper at ICLR 2026
Lemma F.6. Let ReLU and f MaxPook ReLU, we have
3π +
(71)
4x
The signal variance satisfies the following recursion
lão
(1+1)
aged
+ ob
(72)
Moreover compute
30
(1
o (1+1)
of [1+1)
Xi
!
0.82
<
<1,
72 <1, 0.
(73)
4a
qan
Jaa
Therefore across the entire phase diagram we have
lim 1-000
(1) Cab
(74)
and the convergence rate is exponential.
Proof. In this case
= f Du,, Dyz px U2)
3
0
(75)
4
where xx (V1 1/2) is the characteristic function of the set X = Xi
U X2, where X;
{1/1 1/2
R M, 2 0}. Indeed, is easy to verify that X the set points where
Vf
1. Let
us now compute the second term of Eq. (66); for standard ReLU,, this term is zero, since the second derivative of ReLU is (a) non-zero (distribution) only for y = O, where ReLU is zero. Instead, for ReLU+MaxPool we have:
2% Dy, o (Vaug) Af (Vawx))
)
/ Dyy, Max
Voweer Va On (vn On (00))))))((() 52)) (76)
22
220 (nx dy v² y
1
2x ow
2n
where & is the Dirac-delta and 0μ is the Heaviside-ther theta function. Therefore
3 + 2m
3T + 2
a=
(77)
4x
is immediate to verify that this is also the value the V operator, by using Lemma F. the signal variance satisfies Eq. (80). which, when combined with Eq. 665 yields
Therefore
3
of (1+1)
3m
of (1+1) Jua
1
1,>0,
(78)
4a
3T + 2
Jao
with 2+
0.82.
The convergence rate is thus always exponential (see also the experimental
curves in Fig. 23
Lemma F.7 (Phase diagram of ReLU+MaxPPol), The phase diagram of ReLU+MaxPool is qualitatively similar that of ReLU. In particular, the EOC collapses to the singleton (0
1.10, 0f = =0). while in general gradients vanish for o2 below this threshold and explode above it, independently of of.
4m 3n+2
39

---

Published as a conference paper at ICLR 2026
Proof. Since the correlation coefficient converges exponentially fast across the whole phase dia-
gram, the signal covariance is rapidly equal to the signal variance and the value of a = o +x++2
dictates where gradients explode or vanish (see Fig. ??). Across this line, the variance converges only for of = 0 and so the EOC collapses to this point.
F.2 AVERAGE POOLING
For Tanh, directly Algorithm 1 of ref Hayou et al. (2019). For ReLU, have the following Lemma.
Lemma F.8. Let ő ReLU and f AveragePool o ReLU, we have
++1
(79)
4
The signal variance satisfies the following recursion
(1+
aqua + to
(80)
Moreover we can compute
03
of
of
(1)
1
~ 0.76
<1,4>> 0.
< <1,
(81)
4a
(I+1)
(++1)
Yoa
Qua
Therefore across the entire phase diagram we have
lim 1-000
Cab
(1)
(82)
and the convergence rate is exponential.
Proof. We compute
x(0 oz Dy,
4f (Va0y)
)(p
02
(83)
since
||
=1
(84)
in the set X = (1 3/2 E ERly > 0.32 > 0}, which has measure 3. and otherwise. Moreover
Du,Dys f (glly Af
V Vglly)
0²
/ Duy
(Vallll
qua 81
(1)
+
(Vall
(V
gaay/2
(1)
8D
V (V quay)
d0 (Vaitd qqud
]
(85)
0 /
Dy, [Dy Dyz 2
8D V qua
U/2
4x
Therefore
π
o
(86)
4m
which can be directly obtained by using Eq. (57). Similarly to what done for ReLU+MaxPool, we get
π
<
<1,000
(87)
(I+1)
Yaa
and therefore we also have that the correlation coefficient converges exponentially to one across the entire phase diagram.
40

---

Published as a conference paper at ICLR 2026
Finally, compute the phase diagram for this activation function.
Lemma F.9 (Phase
diagram of ReLU+AveragePoal), The phase diagram of ReLu+AveragePPol is qualitatively similar to that of ReLU. In particular, the EOC col-
( -n = 3.03, of =0) while in general gradients vanish for o below this threshold and explode above it, independently of of;
lapses to the singleton
Proof. The proof is similar that of Lemma F7
F.3 PHASE DIAGRAMS
In Fig. 22 we report the phase diagram of ReLU and Tanh applied before these pooling layers. In general, the presence pooling layers has the effect of shifting the EOC. MaxPool generally shifts the phase diagram toward lower o values; this is intuitive, since o globally scales the recursive equations and MaxPool preserves only the larger signal. Tanh with MaxPool show the same phase diagram without MaxPool; this holds
generally for symmetric single-node activation functions (Lemma [F3).. Conversely, for Average Pool the effect is the opposite and the phase diagram is shifted toward larger values of o
Max
MaxPoll
Prejudice
Chaotie-doep prejudice
Ordered-doop projudice
Orderod-docp prejudice
ReLU Avergeeoo
Ganh--Avrraell
Nentrality
Prejudice
5-
Chaotic-deep prejudice
3
Ordered-deep prejudice
Ordered-deep prejudice
0.00
0.3G
0.15
0.06
0.15
of
Figure 22: Extensive phase diagrams for ReLU and Tanh enriched with some 2-dimensional pooling layers. These phase diagrams qualitative equivalent to those without pooling layers, but in general we observe a shift of the EOC and the neutrality/prejudic transition line.
we study the behaviour the correlation coefficient across the phase transition. In par- ticular, report the convergence with depth for MLP with ReLU and Tanh activation functions enriched with 2-dimensional average and max pool layers. We can observe the same qualitative behaviour manifested plain ReLU and Tanh (Fig. 2b. but with shifted values of o?
In Fig. 23
41

---

Published as a conference paper at ICLR 2026
MaxPool
Tanh+ MaxPool
4.0
1.000
1.0
0.290
0.8
3.
0.6
0.900
2.0
10
0.4
10-3
10-9
0.8
10
10-33
0.980
0.0
3.5
20
40
60
90
100
60
50
100
layer
MeLL-AAveraeee
Tanh + AveragePool
6.0
1.000
1.0
3.4
0.995
0.900
0.8
5.
3.2
0.901
0.0
0.902
4.0
0.900
0.4
10-3
0.985
10-9
10
3.5
0.986
1n-13
10
0.981
60
layer
Figure 23: Convergence behaviour the correlation coefficient of ReLU and Tanh with 2-dimensional
Max and Average pooling layers for single MLP with width equal to 10 000 and depth 100. of 0.1 and a varies uniformly from the ordered phase (blue) the chaotic phase (red). The transition points are o² ~ 1.10 (ReLU+MaxPool), of ~ 3.03 (ReLU+AveragePPoo). of = 2.00 (Tanh+MaxPool), and 2 As 3.96 (Tanh+AveragePPoo), Scatter points indicate the asymptotic val- ues. The inset plots show the convergence rate for the correlation coefficient to its asymptotic value e. Solid lines are computed using the IGB approach, while shaded areas represent the 90 central confidence interval computed using the MF approach.
42

---

Published as a conference paper at ICLR 2026
G GRADIENTS AT INITIALIZATION
Let L be the generie loss want optimize. The gradient compute for data sample obeys the following equations:
aw (1) (a) 6%¹² (a)Φ
0C
(a))
Vj
(88)
(a) =
(a)
0C
(a) = & Vi (a)
Σ
N
(a)
dy.
==1
We define the mean square gradients computed with inputs a and b as:
dab = (a)o (b).
(89)
Practically, 90 is computed by propagating inputs and b and computing the empirically
relation, over the layer I, of 6 (a) and 6(9 (b).
By assuming the forward weights be independent from the backward ones, Hayou al. (2019)
proved that (see their Supplementary Materials)
1 [1++1) du Dy Dy o' ( (w')
Lab qah
(90)
Therefore, at the critical point 1, from Eq. 29
gradients satisfy the following recursion
(1+1) (1) Lab X1
Lab
(91)
Gradients are thus stable if x19
In the next sections, we report the initialization gradients of the models empirically studied this work different datasets. We employ the cross entropy loss function. For review of the models and datasets employed we refer Tab. 2 The value of a(l) depends on the inputs and b and becomes distribution upon imposing distribution over the input dataset D, though not Gaussian. Thus, in the following we report the median of lob computed over the dataset. Additionally, we report the same quantity, but computed by restricting the samples to the most favoured and most unfavoured classes.
G.1 MLP
We report the initialization gradients for a vanilla MLP (Eq. 11 for both ReLU and Tanh activa- tions binarized Fashion MNIST (BFMNIST Fig. 24 and CIFARIO (Fig. 25 We observe the gradient vanishing/exploding behaviour very neatly across the phase transition. The same be- haviour is present for the most unfavoured class for both datasets and activations. This is expected since the gradient recursion (Eq. ( does not distinguish between classes. Instead, for the most favoured class of ReLU models, we observe numerically zero gradients of the most favoured class in the chaotic phase. This is due to the fact that the classification loss is numerically zero in the chaotic phase, and so is in every layer due to Eq. (I). easy to explain this by analysing what happens to the cross entropy loss with diverging signals. Indeed, the ReLU chaotic phase is char- acterized by output signal whose distribution moves further and further from the origin. Therefore, since computing the probability that the output belongs to certain class involves computing the exponential the output, all the probability mass concentrates to a single class, yielding practically zero classification loss.
43

---

Published as a conference paper at ICLR 2026
Clobal
Pascured
Unboorted cless
Tinh MLP
Rasseral
Figure 24: Initialization gradients computed on a batch of binarized Fashion MNIST for a vanilla MLP with ReLU and Tanh activation functions. We set of = 0.1.
RelU MLP
Parerned
Tarh MLP
200e
Figure 25: Initialization gradients computed on batch of CIFARIO for a vanilla MLP with ReLU and Tanh activation functions. We set of 0.1.
44

---

Published as a conference paper at ICLR 2026
G.2 RESIDUAL MLP
We report the initialization gradients for Residual MLP Yang & Schoenholz 2017 for both ReLU and Tanh activations on binarized CIFARIO (BCIFAR Fig. 26) For ReLU (Fig. 26 top), the gradients remain perfectly constant across layers. For Tanh (Fig. 26 bottom). we observe mild exponential does not impede training the model's ability separate phases (Figs. 10 (0aad 11
smaller than in the non-residual MLP (Fig. Fig..22 25 However, this decay
ReLU RESMLP
Tanh RESMLP
Cl.be
Fowword das
Unloooured clase
2000
Figure 26: Initialization gradients computed on batch of binarized CIFARIO for a vanilla Residual MLP with ReLU and Tanh activation functions. We set of 0.1.
G.3 VANILLA VISION TRANSFORMER
We compute the initialization gradients of a vanilla Vision Transformer (Dosovisskiy et al. 2020 with ReLU activation function binarized CIFAR10 (BCIFAR, Fig. top) and CIFARIO (Fig.
27 bottom). By removing all batch- and noryer-oommaaoo layers, we trigger the phase transition observed for instance in the MLP.
45

---

Published as a conference paper at ICLR 2026
G0obal
Unfoowrrd daos
ReLU VIT
Unforrannd
Figure 27: Initialization gradients computed on batch of binarized CIFAR10 (top) and CIFARIO
(bottom) for vanilla Vision Transformer (VIT) with ReLU activation function. We set of = 0.1.
G.4 LARGE VISION TRANSFORMER
Finally, we compute the gradients on CIFAR100 of a large Vision Tranformer pre-trained on Ima- genet. The weights are rescaled as explained Sec. B.4 while biases are not modified. We observe similar transition behaviour as in vanilla models (Fig. 27 Due to computational limits, we are able to compute the full gradients only on a batch of size 100 for this model.
Glolel
Fascaund
Unieonanel dos
Figure 28: Initialization gradients computed on batch of CIFAR 100 for a large Vision Transformer
(VIT) with GeLU activation function. We do not modify the pre-trained biases.
G.4.1 THE RFFECT OF THE PRE-TRAINING PRIOR ON INITIALIZATION GRADIENTS
One question that might arise naturally is whether the phenomenology analyzed in this paper appears in the same way in pretrained and untrained models. We address this question by comparing the propagation of gradients in pretrained untrained ViT.
46

---

Published as a conference paper at ICLR 2026
Pretrained
Untrained
107
10
107
20
100
leyer
(a)
Pretrained
Untrained
101
1.2
(
1.0
Lah
801
0.5
1072
0.5
layer
(b)
Initialization gradients of the large VIT on CIFAR100, pre-trained and untrained models. In (a) we average the gradients over blocks of four layers to improve clarity, while the original gradients are reported in (b). For easier comparison, curves related to a different value of of are shifted in such way to be equal to at the last layer.
Figure 29:
We compare the rescaled gradients of the pretrained large ViT with those of same model -this time untrained-, both computed on batch of CIFARI00 (Fig. 29 Each hidden layer is organized as block composed of multiple units (or elements). The regular oscillations observed within each layer reveal both the blockwise modular structure and the contribution of individual elements within
each block (Fig. b). b). To more clearly illustrate propagation effects across layers, we additionally report the gradient norms averaged over each layer block (Fig. a)) This aggregation suppresses within-layer oscillations and makes the inter-layer propagation behavior more apparent.
expect that the presence of residual connections suppresses the gradient instability, putting the models closer to an EOC situation. This indeed what we observe in Fig, 29 b. Surprisingly, however,
As shown in Sec. G.2
do observe gradient exploding in the pretrained netowrk (Fig 29
This suggests that the prior induced by pretrained weights makes gradient stability substantially more fragile than in the untrained case, leading noticeably stronger gradient explosion under the same range of o
47

---

Published as a conference paper at ICLR 2026
H EXAMPLE OF AN EXPLICIT IGB CALCULATION FOR THE RELU
In this section, we provide an explicit calculation with the IGB approach. In particular, upon com- puting the explicit recursive relations for oyas and of in case of ReLU MLP, we are able to prove the following Lemma.
Lemma H.1 (Convergence of ( for ReLU). The correlation coefficient for the ReLU always converge to one; convergence rate is exponential for of > 0, o < 2, where X1 < 1, and quadratic otherwise (X1 = 1))
Proof. Let consider the ReLU activation function. We want explicitly compute the objects that appear on the RHS of Eq. (36). The expectation value of á (y) over the dataset (i.e. the distribution defined in Eq. 333 easy to obtain. For better readability, in the next calculations we drop the layer label, since every quantity, if not explicitly declared, refer to layer l. We thus have for the linear term:
L dy max(0. y)
(w-w) 20
1
(u-a)²
(b))
J
dy y
20
/ 2
v
2TT
(92)
1
erf
11
+
+
y
e
2
2
20
2T
In the same way we can compute the expectation value & (y)² over data as:
(0(U)) V f dy y'e
(w-n
20
122 + o
erf
#
+
+
e
2ng
2
V203
2T
(93)
We can now compute the expectation values over network ensemble (i.e. distributions given by Eq. 34 for the quadratic term as
(10)))
1.2
+
crf
11
+1
9
e
20
2²
0
0
(7
+ 11
(94)
20
2x
2
2
(95)
For the expectation (over weights and biases) of the square linear term, we get:
( (v))
1
erf
11
2erf
14
+
1
d²
of
20
20 20
2
2
-0
n²2
0
+ /1
erf
2
+
dupp erf
20
/203
4
4 22T02
/203
V
By + 1
V
1
3r +
I(t)
+
+
2T (T + 1)
22n +1
2
ক( + 1)
/2 +1
ViT
(96)
where the integral function I(7) = S dx x² erf 2-270 defined for every 7 > 0, is not trivial. Note that T(>) is smooth in (0, but is not defined for 7 O. We can thus analytically
48

---

Published as a conference paper at ICLR 2026
prolonged it y 0 by defining [(0) = lim [(r) 0. By taking the derivative with respect 7, we easily get [(t) 7- 2 h(7)). where we introduce the
auxiliary integral function h(?) s dx erf (a)² 6-0017 Moreover, by repeatedly integrating by parts I(7), we have
1(7) - ? L dx x er fxx) d
= dx erf(x)
---2/= 7 (
2 -0
4 V
I dæ erf(z)e
2
dx
r' ツ 7 h(7) 77 (
[ de erf(x)
d e-o
=
= 3 h(7) +
277
--2111"
dæ
7 -00
h(7) + 2 Van" 2 h(7) +
27 1
V 7 + 1 27 +1
(97)
By putting everything together we can write the following differential equation for h(y):
dh(r)
[h(r)
7
+
(98)
dy 27
Vir 7 + V +1
whose solution is
h(>) 4V7
arctan / +1 VTy
(99)
where the integration constant has be fixed by analytically computing h(1) = / thus obtain:
By derivation we
I(7)
2
72
aretan 2 +1 +1+
(100)
TT π
(7 + 1)V27+ +I
2
By defining
I(7)
3y +
24
g(7) = 2 IT
+
arctan 2 +
(101)
2/
7 +
f(7) = = -
(102)
we can write also
V
(0 (v)))
2 (((1))
(103)
and
o² y0
Varp (0))
(v)))) - (b1))
f(7)
(104)
2
Therefore we write the recursive relations for you and 22 (by restoring the I-dependency):
02 y(t 1))
0 gatt)
(105)
0² will
(()
o² (1(+1)
+ +o
(106)
2
700
Since f(7) 9(7) 1 + 7. by summing together Eq. 105 with Eq. 106) we get recursion
relation for (1)) =02 o
((1) a which has been already discussed by Hayou et al. 2019)
o2g900 2
+++
to
(107)
In particular, 91) converges exponentially to zero for og < and diverges exponentially for o > 2, while for o² = 2 it is constant when of = 0 and diverges linearly when of > 0. For 7 can write:
(1+1) -( +
of [(1(0) ((+1)
(108)
49

---

Published as a conference paper at ICLR 2026
fix] goxl/x
10"
10
10
10a
10a
10a
10a
10*
Figure 30:
Log-log plot g(x)/x / and f()) defined in Eq. 1001) and Eq. (02).. respetively. We observe that they converge one as x tends toward infinity. Therefore, g/l is asymptotically linear.
The function (g) has positive derivative and asymptotically converges to the identity function for Let us start analyzing the case of Interestingly. in such case Eq. (108) does depend on o The convergence rate depends on the sub-leading term and to find it we can expand f and g for large
(see Fig. 30 Therefore, 7 always diverges with the depth. That is, <0) always converges
ReLU, contrary to what claimed by e.g. Schoenholz et al. (2016) and Hayou et al. (2019).
y. We get
2v2
g()) = +
+ 0(7 007-)))),
(109)
3 /
2/2
1(7) 3x /y
+ 0(7 0073)
(110)
and thus
g/f)()) - 2v2 / 11
(111)
Therefore y always diverges quadratically when of 0. When of > 0 we have to distinguish the case base on the value o² For o2 < 2, (()) converges exponentially fast to zero, therefore y(t) diverges exponentially, For o² 2, (0)) diverges, and so the the divergence rate of ((()) is quadratic in the of case. similar discussion applies for o
To summarize, we obtained:
if a 2,
lim 2 1+000
y00
400
if o2 2, Yo
(112)
+00
if 0² 2, >0
lim o2
to0
if a > 2, Vo
(113)
10000
finite else
exxonentially is o 2 22,03 of 0 quadratically
lim 70) 1-000
to0
(114)
else
50

---

Published as a conference paper at ICLR 2026
b
2
0.0
0.2
1.5
10-6
10P
10a
10a
10
10
1.1
10
10
10a
10
10
1.72
10
109
1.83
10"
100
109
10a
10a
1.94
10-1
100
10
o
w
10
10a
2.06
10a
100
10a
10
10
10
10
10
10
109
2.28
102
10a
10a
108
108
2.39
103
10a
10a
1010
1010
2.5
104
104
104
100
100
50
layer
Figure 31: Theoretical (dashed) and experimental (dots) lines obtained for ReLU with different val- of o close to the critical point o 2.0. The width of the network is 10000. The initial theoretical values are adjusted to take into account finite datasets effects. We observe good agree- ment between the theory and the experiments.
51


Reviews:
[{"review_id": "ote2fWfXns", "reviewer": "Reviewer_2DTJ", "summary": "The paper studies initial guessing bias (IGB)—when a network exhibits a bias towards certain classes. It mathematically links IGB to mean-field theory of wide networks, thus connecting learnability properties to IGB. Using this, the authors show that initializing networks to be optimized for trainability and have stable gradients also yields class bias at initialization.", "questions": "What is the difference between “chaotic-deep prejudice” and “(chaotic) prejudice”? Why is it called “deep”?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Understanding how and why neural networks are biased is an important topic for fairness in AI systems. This paper considers the impact of weight initialization, independent of data, which could help us understand how learning dynamics affect bias. By connecting IGB to established mean-field theory of wide networks, it helps build a framework for this. The authors support their theoretical claims with experiments in several architectures and datasets.", "weaknesses": "I found the paper difficult to follow and understand. I think it could have more strongly motivated why the link to MF theory is an important one to make and why the contribution made is important. It was also unclear to me at times what was background on previous work versus a novel contribution of the paper.\n\n“Prejudice” is a more loaded term than “bias” and I find the use of it here somewhat inappropriate for what is being referred to. Bias is used to refer to behavior a network might be more prone to do (e.g., inductive bias) and I think would be more apt.\n\nThe paper briefly mentions it, but by assuming the infinite-width limit, the paper focuses on the so-called lazy learning regime. While the lazy learning regime may be valuable to study, it seems like a limitation for a study motivated by fairness to not focus on the rich feature learning regime optimized for in practice. Furthermore, there is substantial literature on feature learning that discusses how different initializations impact learning and representations, which should probably be reviewed and discussed more in this paper.\n\nI think the contribution of the paper is limited. In particular, the main finding concerns how different initializations associated with trainability also reflect different bias. However, although they show that stable initializations are initially biased, this effect goes away with training. It’s unclear whether this specific initialization effect would persist in other settings and actually affect fairness. For example, this result would be strengthened by showing that it persists or is amplified by certain properties of the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V3gwS9pncf", "reviewer": "Reviewer_cZgs", "summary": "The paper links two views of randomly initialized networks: initial guessing bias, which is when untrained models over-predict one class, with the mean field theory describing trainability at initialisation. The core result is that there is a mapping between both views, such that the same initialisation that can yield a good gradient flow also produces the most biases starting predictions. The paper extends this across architectural choices, including an experiment to perturb a pre-trained ViT to demonstrate these effects.", "questions": "1. Is it possible to link the theory to regimes beyond the infinite width setup? \n2. How does this theory work under harder settings e.g. the vanilla MLPs are not the strongest baselines for performance?\n3. Is it possible to measure bias level in practice and across training?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Clean conceptual claim that ties IGB statistics with network trainability\n2. Clear testable claim forhow models which start with maximal bias learn fastest\n3. Robust checks beyond toy MLPs to ViTs, both training and perturbing them", "weaknesses": "1. The infinite width setting is somewhat limiting on the theory side\n2. It's unclear how much the choice of norms matters or whether this result is idiosyncratic\n3. The pooling for the 2-dimensional case is similarly limited", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jFJHcHLkRD", "reviewer": "Reviewer_h1rA", "summary": "This paper investigates the relationship between initialization bias and trainability of DNNs, reconciling two theoretical perspectives:\n- Mean-Field (MF) theory — which studies gradient propagation and identifies the “edge of chaos” (EOC) as the optimal initialization boundary for stable training;\n- Initial Guessing Bias (IGB) framework — which describes how untrained networks can exhibit predictive bias (favoring one class) even before training.\n\nThe authors establish a formal equivalence between the MF and IGB frameworks, showing that the trainability boundary in MF corresponds to a biased initialization state in IGB. Contrary to the intuitive belief that the most trainable initializations should be neutral, they demonstrate theoretically and empirically that the optimal initialization is systematically biased rather than unbiased — a state they call transient deep prejudice.", "questions": "There can be some terminological confusion: It would be helpful to clarify that “bias,” “prejudice,” and “neutrality” refer to statistical asymmetry / symmetry in prediction space. Otherwise, readers may think about fairness or ethical bias.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Theoretical contribution: The paper establishes a clean mathematical connection between two previously distinct frameworks (MF and IGB), enriching both perspectives.\n2. Insight: It introduces a novel idea: bias at initialization can improve trainability; The new “prejudice-neutrality” phase view offers an intuitive explanation for initialization effects, linking bias to the dynamical stability of gradient flow.", "weaknesses": "1. The theory is derived in the infinite-width limit and validated on small- to mid-scale settings. Its applicability to practical, large-scale deep networks (e.g., transformers with normalization and attention) is not demonstrated. Additionally, empirical evaluation focuses on synthetic and small vision datasets. It is of readers' interest to learn results on simple language tasks. \n2. Ambiguous practical relevance: While the “transient bias” insight is conceptually interesting, there is no clear recipe for practitioners (e.g., how to initialize weights to achieve the right level of prejudice). It would be helpful to add some executable takeaways.\n3. It would be helpful to introduce and compare with alternative trainability-enhancing initializations (e.g., orthogonal, LSUV, scaled ReLU, or NTK-based initializations).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EgG1DX3WbM", "reviewer": "Reviewer_YDAS", "summary": "Develops a theoretical link between mean-field (MF) trainability and initial guessing bias (IGB) at random initialization. Shows an equivalence between MF quantities and IGB statistics, argues that the edge of chaos corresponds to a state of transient deep prejudice that enables fast learning, and provides supporting experiments on MLPs, a simplified ViT, and a large ImageNet-pretrained ViT fine-tuned on CIFAR-100.", "questions": "- Representativeness of experiments. Can the empirical section include standard ImageNet-scale or at least non-binarized, multi-class tasks with production architectures (with normalization and residuals) to assess whether the MF–IGB link and the “transient deep prejudice” claim persist beyond toy settings?\n\n- Input-distribution assumptions. How sensitive are the theoretical conclusions to deviating from i.i.d. Gaussian inputs, for example, to real image statistics or correlated features? Any finite-width theory or experiments targeted at non-Gaussian inputs?\n\n- Fairness measurements. Since the paper motivates fairness, could group metrics on a dataset with sensitive attributes be provided to test whether initial “transient deep prejudice” impacts disparities during early training and at convergence?\n\n- Pretraining biases. For the large ViT, is it possible to disentangle the role of ImageNet pretraining from the weight-scaling operation, and measure whether class-prior skew or spurious correlations from pretraining amplify or dampen IGB on the target data?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Clear statement of the MF-IGB connection, including a formal mapping between MF covariances and IGB drift ratio, and phase-diagram reinterpretations for bounded and unbounded activations.\n- Reproducibility note and code availability claim.\n- Empirical plus theoretical curves illustrating $c^{(l)}\\rightarrow$1 for ReLU, with different rates in ordered vs. chaotic regimes.", "weaknesses": "- Empirical scope is limited relative to broad claims. Experiments focus on binarized Fashion-MNIST, binarized CIFAR-10, CIFAR-10 with simplified MLP/ViT, and a single large ImageNet-pretrained ViT on CIFAR-100.\n\n- Assumptions narrow external validity. Core analysis relies on the infinite-width mean-field regime and an i.i.d. Gaussian input model in the IGB framework.\n\n- Fairness framing without fairness evaluation. The introduction and contributions connect learnability and fairness from initialization onward, but the experiments report only accuracy and maximum class fractions, with no group-attribute metrics.\n\n- Pretraining-bias angle is underdeveloped. For the large ViT, weights are uniformly rescaled to traverse phases, but there is no analysis disentangling ImageNet pretraining priors from weight-scaling dynamics.\n\n- Clarity/calibration: extrapolating MF to practice. Theoretical mapping claims that “best trainability” corresponds to a state of transient deep prejudice (Proposition 4.1), yet validation remains on toy or simplified settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)