Gaussian Pre-Activations in Neural Networks: Myth or Reality?
Authors: Pierre Wolinski
Julyan Arbel
Venue: Accepted by TMLR ()

Abstract: The study of feature propagation at initialization in neural networks lies at the root of numerous initialization designs. A very common assumption is that the pre-activations are Gaussian. Although this convenient *Gaussian hypothesis* can be justified when the number of neurons per layer tends to infinity, it is challenged by both theoretical and experimental work for finite-width neural networks. Our main contribution is to construct a family of pairs of activation functions and initialization distributions that ensure that the pre-activations remain Gaussian throughout the network depth, even in narrow neural networks, under the assumption that the pre-activations are independent. In the process, we discover a set of constraints that a neural network should satisfy to ensure Gaussian pre-activations. In addition, we provide a critical review of the claims of the Edge of Chaos line of work and construct a non-asymptotic Edge of Chaos analysis. We also propose a unified view on the propagation of pre-activations, encompassing the framework of several well-known initialization procedures. More generally, our work provides a principled framework for addressing the much-debated question: is it desirable to initialize the training of a neural network whose pre-activations are guaranteed to be Gaussian? Our code is available on GitHub: https://github.com/p-wol/gaussian-preact/ .

Published in Transactions on Machine Learning Research (04/2025)
Gaussian Pre-Activations in Neural Networks: Myth or Reality?
Pierre Wolinski
pierre. serree.woinnkk@ddnappieee.e
LAMSADE, Paris- DruphhDeee University, PSL University, CNRS, 75016 Paris, Pronce
Julyan Arbel
julyon.arbell@inng
Unio. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France
Reviewed on OpenReview: https: // openreview. net/ forum id- goe6fu615h
Abstract
The study of feature propagation at initialization in neural networks lies at the root of numerous initialization designs. very common assumption is that the pre-activations are Gaussian. Although this convenient Gaussian hypothesis can be justified when the number of neurons per layer tends to infinity, it is challenged by both theoretical and experimental work for finite-width neural networks. Our main contribution is to construct family of pairs of activation functions and initialization distributions that ensure that the pre-activations remain Gaussian throughout the network depth, even in narrow neural networks, under the assumption that the pre-activations are independent. In the process, we discover a set of constraints that a neural network should satisfy to ensure Gaussian pre-activations. In addition, we provide a critical review of the claims of the Edge of Chaos line of work and construct non-asymplolic Edge of Chaos analysis. We also propose a unified view the propagation of pre-activations, encompassing the framework of several well-known initialization procedures. More generally, addressing the much-debated question: is it desirable to initialize the training of neural network whose pre-activations are guaranteed to be Gaussian? Our code is available on GitHub: https://g wwwppwwom///ecooooooocotttt.ooot wwwwwo/gaussaa--peeeeee
work provides principled framework for
Notations and vocabulary
Bold letters such as Z, W, B, or x, represent tensors of order larger or equal to For a tensor W E Raxp, we denote by Wi its component
the intersection the i-th row and j-th column, W, RIxp i-th row and W.y E R its j-th column. Upper-case letters such as W, X, Y. Z, or G, represent random variables. For random variable Z, the function fz represents its density, Fz its cumulative distribution function (CDF), Sz its survival function, and vzz its characteristic function. The depth of neural network is its number of layers. The width of one layer is its number of neurons or convolutional units. The infinite-width limit of a neural network is the limiting case where the width of each layer tends to infinity.
1 Introduction
Let us take a neural network with L layers, in which every layer I [1, L] performs the following operation:
x 1+1 == (1)
with: z
w'x +B'
(2)
TTU
where X E IR: 1 its activation, z 1 its pre-activation, is the coordinate- wise activation function, Wi E Roo4i xn; is the weight matria of the layer, B¹ BR its vector of biases, and X! RR its

---

Published in Transactions on Machine Learning Research (04/2025)
imput (also the preceding layer activation). This paper focuses on the distribution of the pre-activations Z' as I grows, for fixed input x, and weights w and biases B' randomly sampled from known distributions.
Recurring questions arise in both Bayesian deep learning and in parameter initialization procedures: How to choose the distribution of the parameters W W' and according to which criteria, and how should the distribution of the pre-activations Z' look like? Answering these questions is fundamental to finding efficient ways of initializing neural networks, that is, appropriate distributions for W and B' at initialization. In Bayesian deep learning, this question is related to the search for suitable prior, which is still a topic of intense research (Wenzel et al., 2020, Fortuin et al, 2022, Fortuin, 2022, Arbel et al,, 2024, Papamarkou et al., 2024).
Initialization strategics. whole line of work the field of initialization strategies for neural networks is based on preserving the statistical characteristies of the pre-activations as they propagale into a network. In short, the input of the neural network is assumed to be fixed, while all the parameters are considered to be randomly drawn, according to candidate initialization distribution. Then, by using heuristics, some statistical characteristics of the pre-activations are deemed desirable. Finally, these statistical characteristics are propagated to the initialization distribution, which indicates how to choose it.
For instance, one of the first results of this kind, proposed by Glorot & Bengio (2010), is based on the preservation of the variance of both the pre-activations and the backpropagated gradients across the layers of the neural network. The resulting constraint. on the initialization distribution of the weights W is about its variance: /ar(WW//mmi) = 2/(m) + + Then, He et al. (2015) have refined this idea by taking into account the nonlinear deformation of the pre-activations by the activation function. They also showed that the inverted arithmetical average 2/(m: + n441), resulting from a compromise between the preservation of variance during both propagation and backpropagation, can be changed into 1/n, or 1/m141 with negligible impact on the neural network. 2 Notably, with Ø(x) = ReLU(x) == max(0, x),, they obtaïn Var(W?)) = 2/m,, where the factor 2 is meant to compensate for the loss of information due to the fact that ReI.U zero on We summarize and compare several initialization strategies in Table (Section 3.4)
R
Edge of Chaos (EOC). After the studies revolving around initialization, Poole et al. (2016) and Schoen- holz et al. (2017) focused on the correlation between the pre-activations of two data points x, and xg- By preserving this correlation when propagaling the inputs into the neural network, information about the ge- ometry of the input space is meant to be preserved. Thus, training the weights meaningful at initialization, regardless of their positions in the network. This heuristic is finer than the previous ones, since attention is paid to the correlation between pre-activations and their variance (a joint criterion is used instead of a marginal one). The range of valid initialization distributions is modified accordingly, with relation between 2 = Var(W!!)) and of = Var(B)) that should be ensured. This specific relationship is referred to as the Edge of Chaos (EOC).
Finally, it is worth mentioning the work of Hayou et al. (2019), in which the usual claims about the EOC initialization are tested with several choices of activation functions. Notably, the authors have run a large series of experiments in order to check whether the intuition behind the EOC initialization leads to better performance after training. Also, at the opposite of finding an initialization distribution for the parameters, Klambauer et al. (2017) focused on tuning the parameters of the activation function (leading to the SELU activation fun)tion). As in the preceding techniques, they aim for variance-preservingg layers.
In the following, the term "Edge of Chaos" is used in two different manners: "EOC framework", "EOC formalism", or "EOC thcory" refer to a setup where input data points are deterministic and weights and biases are random, whereas in the context of initialization of weights and biases, "EOC" alone refers to a specific set of pairs (o of) matching given theoretical condition (Point 2, see Section 3.1).
Bayesian prior and initialization distribution. There exists a close relationship between the initializa- tion distribution in deterministic neural networks and the prior distribution in Bayesian neural networks. For
IIn the original paper, the considered weight matrix is W = W /vmi, so Var(W = 2/(ny + m441). More generally, any choice of the form 1/(nf m17)) with a E [o, 1] is valid, as long as the same is used for all layers.
2

---

Published in Transactions on Machine Learning Research (04/2025)
instance, let us use variational inference to approximate the Bayesian posterior of the parameters of neural network (Graves, 2011). In this case, the Bayesian posterior is approximated sequentially by performing a gradient descent over the so-called "variational parameters" (Hoffman et al., 2013). This technique requires to backpropagate the gradient of the loss through the network, as when training deterministic networks. Therefore, as with the initialization distribution, the prior distribution must be constructed in such a way that the input and the gradient of the loss propagate and backpropagate correctly (Sec. 2.2, Ollivier, 2018; Chap. 2, Neal, 1996).
Gaussian hypothesis for the pro-activations. In the context of random weights and biases, we call the Gaussian hypothesis the assumption that all the pre-activations Z are Gaussian random variables, at any layer 1 and for any neuron i. This hypothesis is common in the theoretical analysis of the properties of neural networks at initialization. Specifically, this is a fundamental assumption when studying the "Neural Tangent Kernels" (NTK) (Jacot et al., 2018) or Edge of Chaos (Poole et al., 2016). In nutshell, the NTK is an operator describing the optimization trajectory of an infinilely wide neural network (NN), which is believed to help understand the optimization of ordinary NNs. On one side, this Gaussian hypothesis can be justified in the case of "infinitely wide" NNs (i.e., when the widths ny of the layers tend to infinity), by application of the Central Limit Theorem (Matthews et al., 2018). On the other side, this Gaussian hypothesis is apparently necessary to get the results of the EOC and NTK lines of work. However, it remains debated for both theoretical and practical reasons.
First, from a strictly theoretical point of view, it has been shown that, for finite-width NNs (finite ni), the distribution of z has heavier tails as I increases, that is, as information flows from the input to the output (Vladimirova et al., 2019; 2021). Second, a scries of experiments tend to show that pushing the distribution of the pre-activations towards a Gaussian (e.g., through a specific Bayesian prior) leads to worse performances than pushing it towards distributions with heavier tails, e.g., Laplace distribution (Fortuin et al., 2022).
Besides, the condition under which the Gaussian hypothesis remains valid is an important source of confusion. As an example, Sitzmann et al. (2020) state that: "for a uniform input in [-1,1], the [pre-Jactivations throughout SIREN" are standard normal distributed [...], irrespective of the depth of the network, if the weights are distributed uniformly in the interval [-c, c] with e = v6/m in each layer [[]." (Theorem 1.8, Appendix 1.3). Though this formal statement seems to hold for all layers and whatever their widths, it is only an asymptotic result, since it uses the Central Limit Theorem in its proof. Consoquently, this theorem is not usable in practical SIRENs, since it does not provide any speed of convergence of the distribution of the pre-activations to Gaussian, as each ny tends to infinity.*
Deviation from the Gaussian distribution in the finite-widtt case. More recently, several attempts have been made to characterize the distribution of the pre-activations in the finite-width case. In particular, Yaida (2020), Balasubramanian et al. (2024), Trevisan (2023), Basteri & Trevisan (2024), Favaro et al. (2025) provide bounds on the discrepancy between the actual distribution of the pre-activations in the finite-width case and the Gaussian limit in the infinite-width case. One may also refer to Roberts et al. (2022) for a complete study. Although these works provide results for finito-widths networks, they only apply when the widths are large. Additionally, Vladimirova et al. (2019; 2021) show that for finite-width NNs, the pre-activation distribution gets heavier-tailed when getting deeper into the neural network. From another point of view, Noci et al. (2021) provides an analytical expression is provided for the distribution of the pro-activations, but only in terms of Meijer-G functions (Erdélyi et al., 1953), which are difficult to manipulate.
Addressing the limitations of the EOC analysis. Preserving the shape of the distribution of the pre-activations is key assumption of the EOC works. This assumption is usually justified if we assume that the considered network is in the NTK regime, which means that the layer widths are large enough to ensure that the Neural Tangent Kernel is constant throughout training. However, this assumption is generally not
Sinusoidal representation network.
4 "According to Matthews et al. (Th. 4, 2018), the pro-activations tend become Gaussian irrespective of the growth rates of each ny 80 Theorem 1.8 of Sitzmann al. (2020) is asymptotioally true for all layers and all growth rates of each ny. But this result still does not provide any convergence speed.
3

---

Published in Transactions on Machine Learning Research (04/2025)
true and has been discussed in several papers. For instance, Hanin & Nica (2020) showed that if the ratio depth/width does not tend to zero, the NTK varies significantly during training. This illustrates that the NTK is unlikely to be constant during training for narrow and deep neural networks. This phenomenon has also been observed empirically by Seleznova & Kutyniok (2022).
In parallel, another limitation of the EOC has been spotted: with the EOC initialization scheme, the corre- lation between the pre-activations given by two different outputs tends to unique real number, independent of the imput data. Therefore, the EOC initialization inevitably leads to degenerate correlations in deeper layers, which means that the geometry of the input space is progressively forgotten during propagation. This phenomenon has been studied extensively by Martens et al. (2021), and Martens et al. (2021), Zhang et al. (2022) solve this problem with an innovative solution: transform the activation function in order to "shape" the kernel.
This idea of "kernel shaping" through transformation of the activation function has then been used by Li et al. (2022) to solve both issues presented above: the NTK regine does not hold in practice, and the EOC initialization leads to degenerate correlations. So, Li et al. (2022) have proposed to take into account the OLL/ Vn) deviation from the infinite-width limit + oo. This results in set of modifications of the activation functions that are adapted to n and, therefore, are also dependent on n.
Motivation. In the works cited above, the general problem is control the distribution of pre-activations, mostly in order to design good initialization strategics. To this end, two approaches have been used: 1. de- scribing analytically this distribution with given activation functions (e.g., Noci et. al., 2021, Favaro et al., 2025); 2. tuming the activation function to make the pre-activation match a given property (e.g.. Zhang et al., 2022, Li et al., 2022). In the current work, we take approach 2. by constructing activation functions and initialization distributions to provide Gaussian pre-activation at initialization, without relying on the wifth limit hypothesis or any asymptotic exponsion around that limit.
The starting point of our study is the original EOC works (Schoenholz et al., 2017, Poole et al., 2016), which rely only on the preservation of the shape of the pre-activations. This behavior is usually justified by the central limit theorem in the infinite-width limit, which is one of the main assumptions in the works cited above. The direction we explore in this paper is different: we study the shape of the distributions of the pre-activations themselves.
So, we first measure the Gaussianity of the pre-activations. Since the EOC relies on the assumption that the pro-activations are Gaussian, it is necessary to check it experimentally and directly, regardless of how close our neural network is to the NTK regime (i.e., infinite-width limit). Second, Martens et al. (2021), Zhang et al. (2022), Li et al. (2022), we propose to construct new activation functions. tions is preserved. But, unlike these works, we focus on the Gaussianity of the pro-activations, so we do not rely on the infinite-width limit hypothesis or any asymptotic expansion around that limit. To do so, we study individual weights, and, unlike Li et al. (2022), the resulting activation functions do not depend on the width of the neural network. This approach explicitly allows the study of very narrow neural networks, regardless of the depth.
More generally, the goal of the series of articles presented above, of which this one is a part, to propose a principled framework for studying the neural networks at initialization, while proposing initialization schemes and activation functions to solve problems we may encounter at initialization. To study the distribution of the pre-activations in detail, several strategies can be used. As explained in Section 2, such a study is very difficult to achieve for arbitrary activation functions. Moreover, these studies are based on deviations from the infinito-width limit (Yaida, 2020, Balasubramanian et al., 2024, Favaro ct al., 2025). The strategy we adopt this paper is the opposite: we construct activation functions that, by design, should lead to pre-activations whose distribution is known.
Contributions. Therefore, we present in this paper a new framework for studying neural networks at initialization and propose new families of initialization distributions and activation functions. In particular, to obtain results that are independent of how close we are to the NTK regime, we focus on the distribution of the pre-activations during propagation. First, we experimentally test their Gaussianity, i..,, whether the Gaussian hypothesis holds or not. We show that it does not hold in many cases. Second, we construct
4

---

Published in Transactions on Machine Learning Research (04/2025)
families of activation functions and initialization distributions make this Gaussian hypothesis hold, and we show their construction process. Since this process is flexible and can be used to construct other activation functions, it is a contribution in itself.
To be more specific:
we experimentally demonstrate that the Gaussian hypothesis mostly invalid in multilayer percep- trons with finite width (Section 3.1 and Section 3.2);
contrary to a claim of Poole et al. (2016) and usual practical results in the EOC framework, we show in Proposition 2 that the variance of the pre-activations does not always have at most one nonzero attraction point; we provide an example of an activation function for which the number of such attraction points infinite (Section 3.3);
we propose a unified view on pre-activations propagation, encompassing the framework of several well-known initialization procedures (Section 3.4);
deduce set of constraints that the activation function and the initialization distribution of weights and biases must fulfill to guarantee Gaussian pre-activations at initialization under a pre- activations' independence assumption including with finito-width layers (Section 4.1 and Sec- lion 4.2);
we propose new families of activation functions, denoted by (0g) and (oE)o, and initialization distributions designed to achieve this goal of Gaussian pre-activations at initialization (Section 4.3 and Section 4.4);
empirically demonstrate that, with our activation functions (o )e the distribution of the pre- activations remains Gaussian during propagation, while it drifts away from the standard Gaussian when using tanh and ReLU (Sections 5.1 and 5.2);
show that, overall, when training very narrow and deep networks, the usual setups with tanh ReLU activation functions and EOC initialization lead to worse performance than with our activation functions (6%) (Section 5.4).
Additionally, we train, evaluate and compare neural networks built according to our family of activation functions and initialization distributions, and usual ones (tanh or ReLU, Gaussian EOC initialization).
Summary of the paper. We begin by reviewing the widely used assumptions about pre-activations in Section 2. Then, we make in Section 3 a critical review of several results about pre-activations propagation in a neural network: the discussion, additional experiments, and the criticism we are proposing, particularly in the EOC line of works, are the foundations of our contributions. In Section 4, we propose a new family of activation functions, along with a family of initialization distributions. They are defined so as to ensure that the pre-activations distribution propagates without deformation across the layers, including with networks that are far from the "infinitewwidth limit". More specifically, ensure that the pro-activations remain Gaussian at any layer, and we provide set of constraints that the activation function and the inilialization distribution of the parameters should match to attain this goal, under pre-activations' independence assumption. Finally, we propose in Scction a scries of simulations in order to check whether our propositions meet the requirement of maintaining Gaussian pre-activations across neural networks. We also show the performance of trained neural networks in different. setups, including standard ones and the one we are proposing.
2 Common assumptions about pre-activations
Several attempts have been made to provide accurate description of the pre-activations at initialization.
5

---

Published in Transactions on Machine Learning Research (04/2025)
Neural networks as Gaussian processes. A line of works, initiated by Neal (1996), has focused the study of the pre-activations of neural networks on their infinite-width limit, that is, the limit of infinite number of neurons per layer. Notably, it has been proven (Matthews et al., 2018, Lee et al., 2018) that the outputs and the pro-activations of a randomly initialized neural network tend to a Gaussian process with known mean and covariance. The proof can be achieved by using the fact that the pre-activations of the same layer are exchangeable random variables, making it possible to use a variation of the central limit theorem at each layer. These first results have led to the development of the Neural Tangent Kernel framework (Jacot et al., 2018, Lee et al., 2019, Arora et al., 2019c), which allows to study the full optimization trajectory of a neural network trained by gradient descent (Du et al., 2019, Allen-Zhu et al., 2019, Arora et al., 2019b).
Despite the significance of these convergence results, the discussion about the practicality of their hypotheses is still active, notably about the infinite number of neurons per layer and the linear training dynamic. Besides, two essential by-products of these hypotheses are subject to discussion: the fact that the pre-activations are Gaussian and independent in the infinite-widhh limit.
Assumption of independent pre-activations. The assumption that pre-activations of the same layer are independent is very common in the literature neural networks' initialization. For instance, this assumption is explicitly made in He et al. (2015). In the Edge of Chaos literature, the pre-activations of the same layer are implicitly assumed to be independent (Poole et al., 2016, Schoenholz et al., 2017): at a given layer, the vector of pro-activations assumed to be a Gaussian vector with a diagonal covariance, which implies independence. More broadly, as soon as the infinite-width limit is laken, the pre-activations tend to Gaussian processes with zero covariance between pre-activations of the same layer (Matthews al., 2018, Jacot et al., 2018), that is independence.
3 Propagating pre-activations
In this section, we propose a critical review of several aspects of the Edge of Chaos framework. We recall the fundamental ideas of the EOC in Section 3.1. In Section 3.2, we perform some experiments at the initialization of a multilayer perceptron, in which we propagate data points sampled from CIFAR-10. These results illustrate a limitation the EOC framework when using neural networks with a small number of neurons per layer. Then, we build in Section 3.3 an activation function such that the variance of the propagated pre-activations admits an infinite number of stable fixed points, which counterexample to a claim of Poole al. (2016). Finally, we propose in Section 3.4 a unified representation of several initialization procedures.
3.1 Propagation of the correlation between data points
Edge of Chaos (EOC) framework. In the EOC line of work, the inputs of the neural network are supposed to be fixed, while the weights and biases are random. In order to study the propagation of the distribution of the pre-activations Z', Poole et al. (2016) and Schoenholz et al. (2017) propose to study two quantities:
J/ = E[(Z),a. j:a )²),
(3)
L0 1
ElZ j:a 2/u01
(4)
V oo.
where Z/Jm is the j-th coordinate of the vector Z of pre-activations before layer 1, when the input of the neural nelwork a data point Xa- The expectation is computed over the full set of the parameters, i.e., weights and biases. So, we can interpret of as the variance of the pre-activations of a data point xa and d as the correlation between the pre-activations of two data points x, and Xs, over random initializations of the parameters, distributed independently in the following way:
W ~ (ww) with B[ [ = 0 and Var(W()) 2
(5)
B/ Po(a)) with
B[B]] 0 and Var(B)) of
2
(6)
6

---

Published in Transactions on Machine Learning Research (04/2025)
Remark 1. Since the porameters are sompled independently with zero-mean, then:
E 7/210l = =i j:úz:
(7)
1 B J1:a 2/a00 3152
(8)
This why Definitions (3) and (4) do not depend on i, and the crossed terms in J1 and j2 are not worth studying (they are zero).
Remark 2. The distributions of W and B, considered by Poole et al. (2016) and Schoenholz et al. (2017) are normal with zero-mean, that is:
-olam)==((0.
and -
We loosen this assumption in (5) and (6), where we assume that these random variables are zero-mean with a variance we can control. Their theoretical claim remains valid under this broader assumption.
Theoretical analysis. Given two fixed imputs x and Xa, the goal of the EOC theory is to study the propagation of the correlation dab of Z//a and ZJ.5 which goes as follows (Poole et al., 2016, Schocnholz al., 2017):
build recurrence equations for (cab) of the form: 2+1 = s(ca)::
describe the dynamics of (ca)ti
3. provide procedure to compute the variance of the weights' and biases' distributions such that (cab)) tends to 1 with sub-exponential rate (instead of an exponential rate).
Point 1: recurrence equations. Point 1 achieved by using the Gaussian hypothesis for the pre- activations. That is, the distribution of the pre-activations Z' is assumed to be Gaussian, whatever the layer and its width ny. The recurrence equations define variance map V and correlation map C as follows:
d/t = -Vulloww, 0b) = =o² I (Vilee
Dz + of
(9)
1+1 C(cabr u/ u/loww ob):=
1
ot [ (Vovaa) (V
Dz DziDz2 Dzg + of
(10)
V 1+1 abtt1,t
2+1
22 =²) 22T (-2) dz.
where 2/2 = cab22 + =
(ch 122,
and D=: =
cxp
(11)
2
These equations are approximations of the true information propagation dynamics, which involves necessarily the number of neurons nu per layer. Actually, passing a Gaussian vector Z' through a layer with random Gaussian weights and biases produces a pre-activation Z with a distribution which is difficult to describe. On one side, as the dimension ny of the imput Z tends to infinity, the Central Limit Theorem (CLT) applies, and the output * tends to become Gaussian." 5 The assumption that the components of Z are Gaussian is referred to as the Gaussian hypothesis. On the other side, with finite ny, the tail of the distribution of z has been proven to become heavier than the Gaussian one, both theoretically (Vladimirova et al., 2019; 2021) and experimentally (see Section 3.2).
Finally, by assuming that (v)), and (vg) have already converged to the same limit e" + 0 as + 0o, it is possible to rewrite Equation (10) in a nicer way:
J=1 = C. == C(Cabr v", (ww
(12)
Now that the dynamics of (c2a)) written the form e C. (c( becomes sufficient to plot C,
study its convergence. The two hypotheses made, i.e, the Gaussian one and the instant convergence of (v
and (vf), to unique nonzero limit v², are fundamental to obtaining the simple equation of evolution (12).
Evvn if the coordinates z are dependent, the CLT is still valid, proven by Matthews et al. (2018) by using properties of exchangeable random variahles (De Finetti, 1937).

---

Published in Transactions on Machine Learning Research (04/2025)
Point 2: dynamics of the correlation through the layers. Then, Point 2 can be achieved. Now that the trajectory of (cab)t is determined only by the function C., it becomes easy to find numerically its limit e' and its rate of convergence. Specifically, we can distinguish three possible cases:
chaotic phose: oo cah - <11 The correlation between 2/ and 2ith tends to constant that strictly less than 1. So, even if Z/ and 21 are highly correlated (which means that Xg and x are close to each other), they tend to decorrelate when going deeper in the network;
ordered phose: limy des = c' =1 with (1(1) < (the prime here denotes the derivative of a function). The correlation between Z//a and Zi, tends to 1 with an exponential rate, including when j/a and 25b are almost fully decorrelated;
edge of chaos: lom ca = c' = = with C₁ (1) =1. The correlation between Zj and Zjz tends to 1 with a sub-exponential rate, including when 2),, and 2),0 are almost fully decorrelated.
Point 3: best choices for the initialization distribution. Poole et al. (2016) and Schoenholz et al. (2017) claim that pairs (o of) which lead either to the chaotic phase or the ordered phase should be avoided. In both cases, we expect that information contained in the propagated data (or the backpropagated gradients) would vanish at an exponential rate. So, we want to find pairs (o of) lying "at the edge of chaos", that is, making the sequence (caa)) converge to at a sub-exponential rate. The Edge of Chaos inilializations are the initialization distributions of the weights and the biases such that the pair of variances (o lies at the Edge of Chaos.
Remark 3. Even in the favorable edge of chaos configuration, the sequence of correlations (cab)t tends to 1, whatever the data points Xa and xs- So a loss of information at initialization seems unavoidable in very deep nelworks.
If one wants to create an initialization procedure with a smaller information loss, it becomes reasonable to consider data-dependent initialization schemes (i.e., a worm-up phase before training). Such an initialization strategy has been sketched by Mao et al. (2021), who make use of the "Information Bottleneck" formalism (Tishby, 1999, Aawartz-Ziv & Tishby, 2017 7, Sare et al., 2019).
Strengths and weaknesses of the Edge of Chaos framework. One key feature of the Edge of Chaos framework is the simplicity of the recurrence equation (12): it involves only the correlation ce as a variable, and all other parameters (such as uy and vn) are assumed to be fixed once and for all. Notably, in Equa- tion (10), the computation of cab involves the distribution of the pre-activations outputted by layer i, which
is assumed to be DD = (O, 1). In other words, the distribution D' of the pre-activations z is assumed to be constant and equal to D. However, in neural networks with finite widths, D is not constant and evolves according to a propagation equation:
DI+1 is the distribution of 1 W (Z') + B', where Z' ~ D'.
(13)
Such equation involves a sum of products of random variables, which is usually difficult to keep track of. Though Noci el al. (2021) have proposed an analytical procedure lo compute the D' explicitly, it works only for networks with ReLU or linear activation functions, and involves Meijer G-functions (Erdélyi et al., 1953), which are difficult to handle numerically.
3.2 Results on realistic datasets
As far as we know, there does not exist any experimental result about the propagation of the correlations with a non-synthetic dataset and finite-width neural network. We propose to visualize Figure the propagation of correlation d, with dataset CIFAR-10 (results on MNIST are reported in Appendix H.1), in the case of the multilayer perceptron with various numbers of neurons per layer ny (i.e., widths). Then, we show in Figure2 2 the distance between the standardized distribution of the pre-activations and the standard Gaussian (0, 1).
For instance, product of two Gaussian random variables is not Gaussian.
8

---

Published in Transactions on Machine Learning Research (04/2025)
Propagation of the correlations. First, we have sampled randomly 10 data points in each of the 10 classes of the CIFAR-10 dataset, that is 100 in total for each dataset. Then, for each tested neural network (NN) architecture, we repeated Rinit = 1000 times the following operation: (i) sample the parameters according to the EOC; (ii) propagate the 100 data points in the NN. Thereafter, for each pair (Xa; of the selected 100 data points, we have computed the empirical correlation dy between the obtained pre- activations, averaged over the Dinit samples. Finally, we have averaged the results over the classes: the matrix Oly plotted in Figure 1 shows the mean of the correlation dab. for data points X, and Xs, belonging respectively to classes p and q in {0, .9. R Only the experiments with CIFAR-10 are reported in Figure 1; the results on MNIST, which are similar, are reported in Figure 14 in Appendix H.1.
In accordance with the theory of the EOC, we observe in Figure 1 that the average correlation between pre-activations tends to 1, except in the case o = ReLU and n, = 10. In this case, it is not even clear that the sequences of correlations (Cr,), converge at all, since some inter-class correlations are lower at 30 than / 10, while we expectod them to grow until There also a difference between activation functions (anh and ReLU: the convergence to 1 seems to be much quicker with ReLU than with Φ tanh, when ny 100.
We observe that the rate of convergence of (C)) towards 1 not only varies with the NN width ny but also varies in different directions depending on the activation function. When ny grows from 10 to 100, the convergence of (C% (Ggg)) to 1 slows down with é = tanh, while it accelerates with é = ReLU. Since this striking inconsistency with the EOC theory is related to ny, it due to the "infinitewwidth" approximation, precisely made to eliminate the dependency on n, in the recurrence Equations (9) and (10), and consequently simplify them.
Remark 4. According to the framework of the EOC, the inputs xa and xs are assumed to be fized. So, it is improper to define a correlation cat between Xa and X6. However, when considering the correlation ca of the pre-activations right after the first layer, the empirical correlation between imputs Xa and Xb. appears naturally:
x
2 cab d
ob
2
(14)
where C2 : x2_ xp/n plays the role of on empirical correlation bebween X, and X5, assuming that the empirical mean and variance of both Xa and xs, are respectively 0 and 1.9
Propagation of the distances to the Gaussian distribution. We test in Figure 2 the Gaussian hypothesis in a multilayer perceptron of L = 100 layers, with a constant width ny E {10, 100, 1000), and an activation function o E (ReLU, tanh). We propagate a single point sampled from the CIFAR-10 datasot, and we compute the empirical distribution of the pre-activations by drawing 10000 samples of the parameters of the neural network.
In Figure 2, we plot the Kolmogorov-Sminng Smirnov statistic of the standardized pre-activations for each layer, and we compare it to a threshold corresponding to a p-value of 0.05. Thus, according to Figure 2, the Gaussian hypothesis is rejected with a p-value of 0.05 for the ReLU activation function in all considered setups (nj E {10, 100, 1000)), and it is rejected too for the tanh activation function in the narrow network setup (n+ = 10).
We provide all the details about the Bolmogorov-Sminno test in Appendix G, and additional details and experiments in the multilayer perceptron setup in Section 5.2.
For the tanh activation function, a study of the EOC can be found in Poole et al. (2016). For ReLU, the EOC study is more subtle and can be found in Hayou et al. (2019).
&Correltions dy with b have been excluded from the computation show the intra-class correlation between different samples.
Usually, this assumption does not hold exactly: it is common to normalize the entire datasct in such way that the whole of the features of all training data points has empirical mean 0 and variance 1, but not each data point individually. See also Definition 5.
9

---

Published in Transactions on Machine Learning Research (04/2025)
input
1 = 10
= 30
= 50
10 neurons per layer
ReLU
m
tanh
my = 100 neurons per layer
ReLU
tanh
-1.0
0.5
0.0
0.5
1.0
Correlation
Figure 1: Propagation of correlations d, in multilayer perceptron with activation function e (tanh, ReLU) and inputs sampled from the CIFAR-10 dataset. According to the EOC claims, they should tend to as + co. Each plot displays 10 x 10 matrix C₁ whose entries the average correlation between the pro-activations propagated by samples from classes p.q E {0...., .9, at the input and right after layers 1 (10,30 30, 50}. See Fig. 14, App. H.l, for results on MNIST.
10

---

Published in Transactions on Machine Learning Research (04/2025)
layer wirhh 10
layer widhh 100
layer width 1000
ReLU
Lanti sic.
0.4
0.20
0.20
ReLU + atd
KS, vol. 0.35
Tanh
0.2
0.10
0.10
0.1
0.06
0.06
4-
0.0
0.00
20
40
100
20
40
80
100
20
60
100
ligst 1
liyst 1
Layer
Figure 2: Evolution of the -dsttinnncee between the standard Gaussian ((0, 1) and: (solid line) the empirical CDF of the ire-activations (dotted line) the the empirical CDF of the standardized pre-activations. This distance should remain close to zero if the pre-activations are Gaussian. The empirical CDF D' have been computed with 10000 samples. The green dotted line corresponds to the threshold given by the Kolmogorov- Smirnov Lest. with p-value of 0.05: any point above it corresponds distribution for which the Gaussian hypothesis should be rejected with p-value of 0.05.
3.3 Convergence of the sequence of variances (vr))
Multiple stable limits. As reminded in Point 1, Section 3.1, it is key assumption of the EOC formalism to assume that, whatever the starting point na the sequence (v,), converges to the same limit v² as + co. As far as we know, no configuration where the map V has two nonzero stable points or more has been encountered past works. Moreover, it believed that such configurations do not exist, as stated for example by Poole al. (2016): "for monotonic nonlinearities [ø], this length map [V] is a monotonically increasing, concave function whose intersoctions with the unity line determine its fixed points."
In this subsection, we build monotonic activation function for which the V map is not concave and admits infinite number of stable fixed points.
Definition 1 (Activation function Ps.)). For 5 E [0 and > O two real numbers, define:
i ()) = xexp sin(w In lel))
(15)
w
It is casy to prove that for all E [0, 1] and w > 0:
1. (o (0) 0, by continuity;
2. (6, is odd;
3. Pa, strictly increasing;
4. the map2" + (Pa (P (voz)² Dz is e³ and strictly increasing.
Definition 2 (Stable fixed points). Let (un). be a sequence defined by recurrence:
un+1 = f(un) with R.
(16)
For any starting point a, we denote by (un (a)), the sequence defined as above, with un a.
We say that u" is stable fixed point of if f(u)) u² and if there exists open ball B(u" e) centered in x' of radius > 0 such that:
Va E B(u", e), un un(a) - n²
(17)
Notation Dz defined in Eqn. (11)-
11

---

Published in Transactions on Machine Learning Research (04/2025)
Proposition 1. If f((")= =- and f is C¹ in neighborhood of u² with f'(u*) E (-1,1), then 2² is stable fined point of f.
Proposition 2. For any 8 (0, 1] and w 0, let us use the activation function Psw of Definition 1. We consider the sequence (v²), defined by:
AI 2 0, 0/43 oi paw
paw (Vola) Dz ob: with v² E IR...
(18)
Then there crist ow > 0, ob 2 0, and a strictly increasing sequence of stable fixed points (vE)ezz of the recurrence equation (18).
The proof can be found in Appendix A.1.
Remark 5. In short, Proposition 2 ensures that there evists an infinite number of possible (nonzero) limits for the sequence (wa),, depending on v/a Consequently, the proposed activation functions F6ow are counterex- amples to the claim of Poole et al. (2016).
Figure 3 shows the shape of several activation functions fāw for various w, along with their V maps. We have chosen 8 = 0.99 <1 to ensure that 488 is a strictly increasing function. 11 We have chosen of 0 and of = o computed as indicated in Appendix A.2.
Plots.
In Figure 3a, the proposed activation functions exhibit reasonable properties: they are non-linear, differen- tiable at each point (excluding 0), and remain dominated by linear function. However, we expect that as w grows, Pā should become closer and closer to the identity function, 12 which is not desirable for the activation function of a NN.
In Figure 3b, it is clear that the function sPs with w (2016): two nonzero stable points appear. So, in that case, depending on the square norm dQ of the input, the variance vf of the pre-activations may converge to different values. For instance, for w 6 and an input with square norm oy around the unstable point at R 2.3, it may converge either to v* x 0.8 or o" s 6.5.
a counterexample to the claim of Poole et al.
Also, we observe that for w = 6, the variance map V tends to be closer to the idcntity function than for smaller W. Thus, we expect the sequence (w))z to converge at a slower rale with w = than with w 2.
In Figure 3d, all the configurations lie in the chaotic phase. Since all the correlation maps C are below the identity function, the sequence of correlations (c/s) always tends to O. However, the plots are close to the identity, (cas) varies very slowly, and we expect that the correlation between data points propagates into the NN with little deformation. Despite being not perfect and lying in the chaotic phase, this configuration roughly preserves the input correlations between data points (without performing pre-training phase), which is a desirable property at initialization: information propagates with little deformation to the output, and the error can be backpropagated to the first layers.
3.4 Maintaining property of the pre-activations during propagation
To conclude this section and introduce the next one, we propose a common representation of the various methods used to build initialization distributions for the weights (W'), and biases (B'). )-
Several initialization methods (Glorot & Bengio, 2010, He et al, 2015, Poole et al., 2016, Schoenholz et al., 2017) are based on the same principle: initialization should be done in such a way that some characteristic x of the distribution of z' is preserved during propagation (e.g., x² = Var(Z')). Intuitively, any change between se and *t+1 reflects a loss of information between Z' and Z which impacts propagation or backpropagation. For instance, when Var(Z') + O, the network output tends to become deterministic, and when Var + oo, the output tends forget the operations made by the first layers (i.e., the gradients vanish during backpropagation).
We have chasen 8 close to 1 to obtaín a function P that is strangly nonlinear, but a bit lower than to ensure that 4/ remains strictly positive, order to prevent the training process from being stuck. 12 As + Do, Pha, converges pointwise to the identity function..
12

---

Published in Transactions on Machine Learning Research (04/2025)
10
2
2
8
2
6
4-
2-
-2
2
0
2
4
6
8
10
(a) Activation function 905.w-
(b) Variance map V(-lows Tb = 0), linear scale.
10a
1.0
2
2
10
3
0.8
0.6
10
0.4
10
0.2
10
0..0
10
10
10
0.2
0.4
0.6
0.8
1.0
(c) Variance map V(-law, 0b 0), log-log scale.
(d) Correlation map C(s v*, 0b).
Figure 3: Properties of the proposed counterexamples f8w represented in Fig. 3a for (2,3,6) and 8 0.99. Their variance map V has an infinite number of fixed points.
In Fig. 3b, stable points are marked by crosses (+), and unstable points by bullets (o), when they are away from O: two stable points appear for w = 6 (in red). As established in Proposition 2, aw is tuned for every w in such a way that crosses the identity function an infinite number times (not visible on the figure). In Fig. 3e (log-log scale), it is clearer that has an infinite number of fixed points, due regular oscillations (in log-log scale) below and above the identity.
In Fig. 3d, we show that w grows, the correlation map becomes closer to the identity function, which means that the correlation between data points tends to propagate perfectly.
Note: since an infinite number of stable fixed points are available, we have arbitrarily picked one for each w, denoted by v². This choice does not affect the plot of the correlation map C, due to the very specific structure of Paw.
13

---

Published in Transactions on Machine Learning Research (04/2025)
More generally, denote by D' the distribution associated to the pre-activations Z', by P(, Ti(P:]()) the transformation of D' performed by layer 1, that is DD+1 = (D') (where Pj is the initialization distribution of (W', (') and ø' is the activation function at layer (), and by == xDD)) the characteristic of the distribution D' we are interested in. Then, according to a hcuristic of "information preservation", it is assumeed that the sequence (x²)) must. remain constant, and the initialization distributions (Pı), are built. accordingly. In some cases, is possible to build a map PP), so that each x/+1 can be built out of its predecessor xt, without using all the information may have on the (D)),-
We summarize this way of building initialization procedures in Figure 4, and we show how it applies to well-known initialization procodures in Table 1.
Table 1: Examples of D', of and X in various setups. Notations: for any vector E R", its empirical mean Σ ii=1 (a;; Rx)²
is E x = 1,2 Σi and is empirical variance is Varx n-I
Method
D
T((P)](K)
Assumption
Glorot & Bengio
distr. of Z' distr. of Z'
Id
Var
ii + jo of
He et al.
ReLU
Var
j:at Z Gaussian,
Poole, Schocnholz
distr. of (Z,, z,)
corr
(k(
u V/ 0²
Ours
distr. of Z'
P
o/p
Id
T(Pp)(x)
E X Xb 0,
Var Xa = Var Xb =
Ti-1 P D
T(Pd > Di+1 Ti+1|Pt+1]
Ti 111-1] kcl
[(P)
T:41
I+1
Figure 4: Building process of the initialization distributions Pz of the parameters (W!, B'): (i) the pre-
activations-related distribution D' passes through a map Ti[P:] and becomes DI+1, (ii) some statistical charactoristic re of D' can be computed with a function X: x² = x(D"); (iii) we tune the (Pt); in order to make the sequence (N²)), constant.
Remark 6. We can use Figure 4 to build new initialization distributions: first, we choose a statistical property of Z', which determines D' and x; then, we bwild framework in which x(D)) can be easily computed for every 1 (e.g., we choose a specific aclivation function, or we make simplifying assumplions).
In the following section, we aim to impose Gaussian pre-activations through a specífic activation function go/p and initialization distribution Po- It implies that we would preserve perfectly the distribution D' itself: our characteristic is x(D) D'. That way, all the statistical properties of D are preserved during propagation.
4 Imposing Gaussian pre-activations
In this section, propose family of pairs (Po. 0g do where Pg is the distribution of the weights initialization, /o/p is the activation function, and 0 E (2, 00) is parameter, such that the pre-activations z/ are ((0 1) at any layer I. Imposing such pre-activations is a way to meet two goals.
First, in Soction 3.2, we have shown that the Gaussian hypothesis not fulfilled in the case of realistic datasets propagated into simple multilayer perceptron, and we have recalled in the Introduction that the tails of the pre-activations tend to become heavier when information propagates in neural network. By
14

---

Published in Transactions on Machine Learning Research (04/2025)
imposing Gaussian pre-activations, we ensure that the Gaussian hypothesis is true, which reconciles the results provided in the EOC setup (see Eqn. (9) and (10)) and the experiments."
Second, as we recalled in the Introduction, many initialization procedures are based on the preservation of some characteristic of the distribution of the pre-activations (see Table and Figure 4). Usual characteristics are the variance and the correlation between data points. By imposing Gaussian pre-activations, we would ensure that the whole distribution is propagated, and not only one of its characteristics.
Besides, provide a set of constraints, Constraints 1, 2, 3, and 4, that the activation function and the initialization procedure should fulfill in order to maintain Gaussian pre-activations at each layer.
Summary. Formally, we aim to find family of pairs (Po, 08 PP such that:
z ~ (o,
i..d ii.d. 7 w
W ~ Po
00 / / (Z)) + +B' 1O0,1,
Vnu
B =0
where W is the i-th row of the matrix W In other words, the pre-activations 2+1 remain Gaussian for all I.
As result of the present section, we make the following proposition for (Po, (Po.0) 0%/P):
Pg is the symmetric Weibull distribution W(O, 1) with scale parameter 1 and shape (or tail) parame- ter 0, which is obtained by symmetrizing a standard Weibull distribution from R+ TR (Generalizod Weibull-Tail distributions are detailed in Definition 4). The symmetric Weibull distribution W(0, 1) has the following CDE:
1 + sgn(t) exp (-1t)9) exp
(t) = 2 2
(19)
o/P is computed to ensure that 7/+1 W o/p (Z') % family of functions o0/p contains (Fig. 5):
B is Gaussian (00, In short, the
va 1.99
a sub-family (og)e odd activation functions, spanning a range of functions from a tanh-like function (as 8- 2+) to the identity function (as o0),
sub-family (x))) of posilive activation functions, spanning range of functions from sigmoid- like function (as + 2+) to a softplus-like function (as 0 + oc).
We shall see that of preserves the Gaussianity of pre-activations better than 6g-
In order to obtain this result, we:
reduce and decompose the initial problem (Section 4.1);
2. find constraints the initialization distribution of the parameters to justify our choice Po W(0,1) (Section 4.2);
compute the distribution Qo of (Z)) we must choose to ensure Gaussian pre-activations 2 given initialization distribution Pe (Section 4.3):
4. build og and of from Q9 (Section 4.4).
13eheree exists another way to solve this problem: use propagation equstions which would take into account the sequence (n)) of layer widths, that adopt non-asymptotic sctup, contrary the process leading to Eqn. (9) and (10). However, taking into account the whole sequence (m)) would lead to recurrence equations that far less casy Morcover, a precise characterization of the distributions (D*); of the pro-activations (2)): may be very difficult since they would not be Gaussian anymore.
than Eqn. (12),
14e softplus function defined by: x + log(1 + exp(x:)).
15

---

Published in Transactions on Machine Learning Research (04/2025)
4.1 Decomposing the problem
By combining Equations (1) and (2), the operation performed by each layer is:
1 w
z
(20)
V
In this subsection, show that finding the distribution of the weights w and the activation function e in order to have:
W/ E [1, [L, ~7~,100,1).
(21)
can be done if manage to get:
= = Wx(X) ~ (O with X ~ NO,,
(22)
by tuning the distribution of W and the activation function Φ.
7 as a sum of Gaussian random variables. First, we focus on the operation made by one layer: if each layer transforms Gaussian imputs ZJ into pre-activations Z that are Gaussian too, then we can ensure that the pre-activations remain Gaussian after each layer. Thus, it is sufficient to solve the problem for one layer. After renaming the variables Z t Z W3 W, + Z), B + B n + nt, we have:
1 Vn 1=1
2 =
W (X) + B.
(23)
In the rest of this subsection, we assume that the X, N(0, 1) are independent. We discuss the independence hypothesis in Remark 7 and Appendix B. We want to build an activation function o, and distributions for (W,), and B such that ~ 10,,
Second, we narrow our search space. According to Equation (23), Z is the sum of a random variable B and a number of ii.d. random variables W,)(X,)/VR, Since Z must be NN,, 1) whatever the value of n, it is both convenient and sufficient to check that each summand in the right-hand side of Equation (23) is Gaussian, that is:
B B~,, W,O(X,)~ ~ N( -ff)
(24)
with σb (0, 1). In that case, we have Z ~~ -0 1). For the sake of simplicity, we assume that. B 0 with probability 1, that we just have to ensure that, for all j, Wj((X,) (O 1). If one wants to deal with nonzero bias B o0 it is sufficient to scale the random variables Wyo(X,) accordingly.
To summarize, we have chosen to build Z ~ NO0 1) by ensuring that Wz)(X,) ~ N((01) 1). With B = 0, this choice formally imposed by this straightforward proposition.
Proposition 3 (Lévy-Cramér Theorem). Let (Zj); be sequence of i.i.d. random variables. Let Z If Z is N(O, 1), then the distribution of each Z, is also 1).
EE= Zz.
Proof. Let (z((x) = E[eeZz] be the characteristic function of the distribution of Z. Besides, the (Zz), are i.idd. and Z 1 Eグ= Z, so:
x² and v2(x1) = 1/ 2, (ar)
bz(z)) = exp
(25)
2
via
This proves that vz (x) 12, So, for all j in [1, n), Zz ~ NN 1).
16

---

Published in Transactions on Machine Learning Research (04/2025)
As result, we obtain the first constraint.
Constraint 1. If, for alll, the weights (Wi,), are i.i.d. and independent from the pre-activations (X))s, which are also supposed to be i.i.d. then we must ensure that:
v1,2, W1 N(0.
Remark 7. The hypothesis of independent imputs (X)), truly holds only for the second layer. Although the hypothesis of independent (X)), is common (see Section 2), it is mostly unrealislic praclice. So, we propose Appendir B an empirical study of this hypothesis, order identify in which cases the dependence between the inpuls of one layer damages the Gaussianity of its outputted pre-activations.
New formulation of the problem. We have proven that, to ensure that Z N 1), it sufficient to solve the following problem:
find P and ø such that: X ~~ 1(0, and W ~ P
(26)
In the following subsections, we build family P of initialization distributions (Section 4.2) such that, for any P E P, there exists function ø such that (P, is a solution to (26). We decompose the remaining problem into two parts, by introducing an intermediary random variable Y = ((X):
for a distribution P, deduce Q s.t.: N ~ Q + WY = G NO, 1) (Section 4.3);
for a distribution Q, find a function o s.t.: X NN(10)) = Y x(X) ~ Q (Section 4.4).
4.2 Why initializing the weights W according to a symmetric Weibull distribution?
We are looking for a family P of distributions such that, for any P E P, there exists Q such that:
W~PPY~QQ = WY =: G~
(27)
Therefore, the family P is subject to several constraints. In this subsection, present two results indicating that subset of the family of Weibull distributions is good choice for P;
the density of W at 0 should be 0;
2. W should be a generalized Weibull-tail random variable (see Section 4.2.2 or Vladimirova el al., 2021) with parameter 0 E (2,oo).
In the process, we are able to gather information about the distribution of (Y], namely its density 0 and the leading power of the log of its survival function infinily, respectively:
2 π (r)
fwi(()
dt
(28)
log S1y (1) x - 1/ (1)
(29)
As a conclusion of this subsection, we consider that the distribution P = Po of W should lie the following subset of the family of symmetric Weibull distributions (defined Eqn. (19)):
P = (W 1) : E 9),
θ = oo).
18rThe inputs of the first. layer are deterministic.
17

---

Published in Transactions on Machine Learning Research (04/2025)
4.2.1 Behavior near 0
Since the product = WY is meant to be distributed according to NV, 1), then we must have fia1(0) - V (0), which is impossible for several choices of distributions for W.
Proposition 4 (Density of a product of random variables at 0). Let W.Y Y be two independent non-negative random variables and Z = WY. Let f fz be their respective densities. Assuming that fv is continuous at with fy (0) O, we have:
[w(())
lim w0 w-
dt 00, then
lim fz(2) oc. 2-0
(30)
Moreover, if fy is bounded:
Sw (t) de oc, then fz(0) 5x(0)) /o fw(() dt.
(31)
The proof can be found in Appendix C.
Corollary 1. If fy and fw are continuous at 0 with fr(0) > 0 and fw (0) > 0, then:
lim fz( = oc. 2-4
(32)
According to Corollary 1, it impossible to obtain Gaussian G by multiplying two random variables W and Y whose densities are both continuous and nonzero at So, if we want to manipulate continuous densities, we must have either fw( 0 or (r( =00000 O.
Let assume that fy (0) O. We want Y to be the image of X 1)) through the function o, where (x > 0. So, in order to obtain Y with a zero density at O, it is necessary to build function & with (0) oc (sce Lemma 2 in Appendix D), which is usually not desirable for an activation function of a neural network for training stability reasons." 16 So, it is preferable to design W such that /w(0)= = 0.
Constraint avoid activation functions with verticul langent at 0, the densily of the inélialization distribution of a weight W mast be at O:
vI,ii, fwi, (0) = 0.
Remark 8. In the common case of neural networks with activation function á tanh and weights W initialized according to Gaussian distribution, if assume that the Gaussian hypothesis is true, then fy(0) > 0 and fz(0) > 0. Thus, Corollary 1 applies and the density of Z is infinite at 0.
If Φ = Id, Z is the product of two independent. (0, 1), whose densily is well-known:
fz(2) Ko(1z)) π
(33)
where Ko is the modified Bessel function of the second kind, which lends to inferiity at O, which illustrates Corollory 1.17
Finally, if Constraint 2 holds and want (ia V 2 then, according to Equation (31), the following constraint must hold.
191f (0) and & is C³ on R², then numerical instabilities occur during training: if pre-activation Z approaches too closely, o' o'(2)) can explode and damage the training. 'Thse instabilities can be handled by gradient clipping (Pascanu et al., 2013).
Ihhough, even each Wyd(Xy) has an infinite density at 0, the density at of weighted sum 2 Vn En Wio(X,)+B may be finite. instance, occurs when the Wj and d(X,) i.I.d. and Gaussian. But this case, even fz cc, Gaussian pre-activation (see Prop. 3).
impossible
18

---

Published in Transactions on Machine Learning Research (04/2025)
Constraint 3. The density of at must have a specific value depending on the distribution of W
[(y((0) V (t)
2
-1
dl
4.2.2 Behavior of the tail
We use the results of Vladimirova al. (2021) on the generalized Weibull-tail distribulions and start by recalling useful definitions and properties.
Definition 3 (Slowly varying function). A measurable function f (0) - (0) is said to be slowly varying if:
Va 0, lim f(ax) f(z)
=1
(34)
Definition 4 (Gencralized Weibull-Tail (GWT) distribution, Vladimirova et al., 2021). A random variable X is called generalized Weibull-tail with parameter @ 0, or GWT(0), if its survival function Sx is bounded in the following way:
Vz O, exp f==))((()))=))))))))))))())))f S Sx(2x)
exp (-x' fa(x))
(35)
where fi and f2 are slowly-varying functions and 0 > 0.
Proposition (Vladimirova al., 2021, Thm.
2.2). The product of two independent non-negative random variables [W] and [Y] which respectively GWT( and GWT(Or) is GWT(O), with such that:
1
1
(36)
0
Ow 0y
We recall that, in our case, |G] = [W] |Y| is the absolute value of Gaussian random variable. So G is GWT(2). Thus, if we assume that [W] and |Y] are respectively GWT(Ow) and GWT(Oy), then we have:
1
1
1
(37)
2
Ow
0y
Therefore we have the following constraint.
Constraint The weights W are GWT(() with 0 E θ o0).
4.2.3 Conclusion
Constraints 2 and 4 indicate that the distribution P of the weights W:
(i) should have a density fw such that fw(0) O;
(ii) should be GWT(() with E 20).
A simple choice for P matching these two conditions is: P Pa = W(0,1) with 0 E (2, where W(0, 1) is the symmetric Weibull distribution, defined in Equation (19). Thus, we ensure that fw (0) = 0 and W is generalized Weibull-tail with a parameter 9 easy to control (see remark below).
Remark 9. 1f W W(0, 1), then W is GWT(
19

---

Published in Transactions on Machine Learning Research (04/2025)
4.3 Obtaining the distribution of the activations Y
Now that the distribution P of W is supposed to be symmetric Weibull, that is, P Pa = W(0,1), we are able to look for odd and positive activation functions 0n such that:
W w(e, 1), ~ (O = (x))(X)~N(O.11
(38)
As first step, we look for a distribution Qo such that:
W~ PoPPy
WY =Y=
(39)
In order to "invert" this equation, it is natural to make use of the Mellin transform. A comprehensive and historical work about Fourier and Mellin transforms can be found in Titchmarsh (1937), and a simple application to the computation of the density of the product of two random variables can be found in Epstein (1948).
However, the technique involving the Mellin transform is very difficult to use in this case, both analytically and numerically. Details about the Mellin transform and these difficulties can be found in Appendix E.
Computation of fy hand-designed parameterized function. Thus, inspired by the shape of fix computed via the numerical inverse Mellin transform (see Fig. 13, App. E.2), we build an approximation of fiy from the family of functions (garndida a, y, A1, A2 > 03 with:
(x(x) (o. = ya 00-1
2
Xi
exp
X
+
T
exp 8)
A
(40)
where is the conjugate of 0: 8 + = 1,2 and A = (a, y, A1, A2)- It is clear that, whatever the param-
eters, (0) / #)] which is exactly Constraint 3. Moreover, when 0, g matches also Constraint 4.
Then, we optimize the vector of paramcters with respect the following loss:
((A) : |lAA- - -iaailo
(41)
Px(2) = 100 000
Fiwl ( ex())) (see Equ. (57)),
(42)
where FA is meant to approximate the CDF of the absolute value of a Gaussian NN(0,) 1). The integral is computed numerically. For the loss, we have chosen to compute the -di-taancce between two CDFs, order to be consistent with the Kolmogorov Smirnov test we perform in Section 5.1. Optimization details can be found in Appendix F.
4.4 Obtaining the activation function /O/P
In the preceding section, we have computed the dist of [Y]. We are restricting ourselves to respectively symmetrical and positive Y, whose distributions are respectively denoted by Q2 and QB (*o" for "odd" and "p" for "positive"). Now, we want to build the activation function d8 in order transform a pre-activation G ~ N((0 1) into an activation Y -00 o/P (G) distributed according to Q%
if G (0, then /8 Q
To compute 8/p, we will use the following proposition:
Proposition 6. Let X be a random variable such that Fxx is strictly increasing on IR Let Q be distribution
without atoms such that Fo is strictly increasing S := Fo ((0.1)). Then:
with d(x) := FF=(((((()))
have: ((X) Q
(43)
20

---

Published in Transactions on Machine Learning Research (04/2025)
Proof. Let Φ(x) Fo (Fx(2)) which is strictly increasing bijection from R to S.
For any E S, we have:
P(O(X) v) = P(X -x0
FQ(V)-
(44)
For y E S, since connocted, we have either y S inf or y 2 sup
y $ S with y S inf S; then P(o(X) - 3) s ((((X) 5) = 0;
if y & S with 2 sup S, then P(((X)= - v) 2 (X) S sup S) 2 P(O(X) E S) =1.
So, for all y R, (X)( < y) FQ(y).
Odd activation function %.- We build symmetric distribution for Y: +=25g(()
1 1 ssgn()) L/ (i) dy.
Fy (1) = ] +
(45)
Since FG and Fy are strictly increasing on R, we can use Proposition 6:
(t) = (FF))
(46)
Positive activation function of We build a distríbution for Y with support in R
(0x(0,.)
F$(t): =
(v)
(47)
Since FG is strictly increasing on and Fy is strictly increasing on S (PP)-1((0, 1)) = RR+((),, we can use Proposition 6:
-19)
(48)
Remark 10. There many other possible choices for the distribution Q Yf, resulting in various activa- tion functions other than /./p. However, among all the possible distributions Q2 we identify two "natural" usable solutions: Q symmetric (leading to an odd activation function) with support in g+ (leading to positive activation function).
4.5 Results and limiting cases
We have plotted in Figure 5 the different distributions related to the computation of og o/p and the functions so/p themselves. The family of the &8 is continuum spanning unbounded functions from the tanh-like function d2- to the identity function, while the family of the of is a continuum spanning functions from a sigmoid-like function to softplus-like function, all of which tending to in -o and to too in toc. As reminder, softplus(x)) log(1 + exp(x)).
Our construction degenerates into the following two extreme cases at the boundaries of the parameter space e = (2,
when + 20, we have Pa A R and:
d ld pointwise,
S pointwise, with: db : + 1eelf-1 + (2))
when 0 we have Po - W(2,1);
21

---

Published in Transactions on Machine Learning Research (04/2025)
2:05
1.0
2.50
3,00
0.8
0.6
10:00
0.4
0.2
0.0
(a) Initialization distribution Pa of the weights: sym-
(b) Estinated density fr of the distribution Qo of Y.
metric Weibull 1).,
10.0
10.0
7.5
7.5
5.0
5.0
2.5
2.5
0.0
2.5
-7.5
-7.5
-7.5
-5.0
-25
0.0
2.3
7.5
10.0
10.0
16.0
(c) Activation function Qg-
(d) Activation function db.
Figure 5: How to build a random variable WY = Wo =; G ~ (o(,,) 1), where X ~ NO 1)? (a) Choose the distribution Pa of W, then (b) deduce the distribution Qo of Y, and finally (c-d) find og and db
where R is the Rademacher distribution (g R + P(g = +1) = }) and Id is the identity function.
In the limiting case f + 20, we initialize the weights at =1, which corresponds to binary "weight quantiza- tion", used in neural networks compression (Pouransari et al., 2020), and we use a linear activation function, commonly used in thcoretical analyses of neural networks (Arora et al., 2019a). In the limiting case - we recover weights with Gaussian tails, with tanh-like sigmoid-like activation function.
5 Experiments
In this section, we test the Gaussian hypothesis with ReLU, tanh and our activation functions 00/p after one layer (Section 5.1) and after several layers (Section 5.2). Then, we plot the Edge of Chaos graphs (a,, aw); which are exact with
/ (Section 5.3) in the case of finite ny with independent, pre-activations. Finally, in Section 5.4, we show the training trajectories of LeNet-type networks and multilayer perceptrons, when using tanh, ReLU, and various activation functions we have proposed.
In the following subsections, when we use Φ tanh or ReLU, we initialize the weights according to a Gaussian at the EOC. This is, for tanh: of = 0.013, a² = 1.46; and for ReLU: of = 0, o² 2. When we so/p. initialize the weights according to W(0, 1).
22

---

Published in Transactions on Machine Learning Research (04/2025)
Notation for the activation functions. We recall that: & denotes an arbitrary activation function; o/p denotes the odd or the positive function verifying fá,, denotes the function defined in Definition 1; Pg Eqn. (38).
5.1 Testing the Gaussian hypothesis: synthetic data, one layer
Most importantly, we must experimentally verify that our family of initialization distributions and activation functions ((Po, / E 20)) can reliably produce Gaussian pre-activations.
Framework. First, we test our selup the one-layer neural network case with synthetic inputs. More formally, we consider a (O 1) pre-input Z E IR",, which is meant to be first transformed by the activation function / (hence the name "pre-input"), then multiplied by a matrix of weights W E Rix=. This one-neuron layer outputs a scalar Z":
1
z' - -W(Z) with Zy ~ 10) and Wiy ~ Pa for all j.
(49)
We want to check that the distribution
is equal to (O, 1).
Experimental results. For that, we use of the Kolmogorov- Smirnov (KS) test (Kolmogoroff, 1941, Smirnov, 1948) (sec Appendix G). We perform the KS test within two setups: with and without pre- liminary standardization of the sets of samples. With preliminary standardization, we perform the test on (", (Z)):
7/4 z/
Z
(Z) -²)²
(50)
8
8
For the sake of simplicity, let us denote by Fz. := samples (7,,... Z%), and let Zz. be the empirical CDF of standardized Z', computed with (Zi, Z.).
:= F3 the empirical CDF of Z', computed with the s data
We have plotted in Figure the KS statistic of the distribution of the output Z', when using our activation functions / tanh and ReLU, Our sample size is s = 10a. A small KS statistic corresponds to a configu- ration where Z' is close to being (0, 1). If a point is above the KS threshold (green line, dottod), then the Gaussian hypothesis rejected with p-value 0.05.
When we perform standardization (Fig. 6b), the neurons using og output always a pre-activation Z' that is closer to (0, 1) than with ReLU or tanh. But, despite this advantage, the Ganssian hypothesis should be rejected with / when the neuron has very small number of inputs (n < 30).
In Figure 6a, we compare directly the distribution of Z' to (0, 1). This test is harder than testing the We observe that when using & the
Gaussian hypothesis because the variance of Z' must be equal KS statistic remains above the threshold (while it is still below 10-2, and even below 6 10-3 for > 3).
This result is due to the fact that our computation of 10/p is only approximate (see Section 4.4).
Our sample size (s = 10?) is very large, which lowers the threshold of rejection of the Gaussian hypothesis. We have chosen this large 8 to reduce the noise of the KS statistics. If we had chosen s = 18000, a threshold close to 10-2 would have resulted, which higher than any of the KS statistics computed with o =6//0 One will also note that, in Section 5.2, we use only s = 10000 samples to keep reasonable computational cost.
Remark 11.
Remark 12. Since tanh and ReLU have not been designed such that Z' ~ NN(, 1), we did not plot the related non-standartized KS statistics in Figure 6a. Anyway, given the standard deviation of z' when using Φ = tanh or ReLU (see Table 2, Appendir H.3), very large KS statistics are expected in this setup.
We discuss the limits of the KS test in Appendix G.
18 Juch a pre-input plays the role of the pre-activation outputted by a hypothetical preceding layer.
23

---

Published in Transactions on Machine Learning Research (04/2025)
10"
10"
<, 0 2.05
<g.
o5, = 2.05
0g. 5.00
<8. 0 = 2.50
10.00
( = 2.50
d.. 7.00
<g. 0 3,00
tanh
10-1
(6. = 3.00
68 10.00
<g. 0 = 4.00
ReLU
<5) 4.00
KS statistic
% 8 5,00
KS, p-val 0.05
KS-statistic
10-2
10-2
10-3
10-3
10
10
10'
10a
10
10
10a
Number of input weights
Number of input weights
(a) KS-statistic of Fz..
(b) KS-statistic of Pz. (standardized samples).
Figure 6: Evolution of the KS statistic of the distribution of z' (Fig. 6a) and the standardized distribution
of Z' (Fig. 6b), with number of inputs (1,3, 10, 100).
5.2 Testing the Gaussian hypothesis: CIFAR-10, multilayer perceptron
Now, test setup multilayer perceptron with CIFAR-10, which is more realistic. We show in Figure 7 how the distribution of the pre-aclivations propagates in a multilayer perceptron, for different layer widths E (10, 1000).
Setup. Let D' be the distribution of the pre-activation Z after layer l. For all 1, let us define (Zi sequence of i..d. samples drawn from D'. The plots in Figure show the evolution of the f distance F Fa Folloo between the CDF of N(O, 1); Fa, and the empirical CDF of Di, Ezz, built with 10000 samples (E(L))
We have built the plots of Figure 7 with the same input data point. See Appendix H.4 for comparison of the propagation between different data points. Also, the propagated data point has been normalized according to the whole training dataset, that is:
Definition 5 (Input normalization over the whole dataset). We build the normalized data point &:
Raijj
Jazif - 14
Tij.
of p,d
115)2
(51)
0,
P.d XED
1=1
XED
where Tasi the j-th component of the i-th channel of the imput image Xa, Pi is the size of the i-th channel, and d is the size of the dalasel DD.
Results. First, in all cases, tanh leads to pre-activations that remain far from the standard Gaussian. Also, with 10 and 100 neurons per layer and ReLU, the pre-activations tend to diverge from the standard Gaussian.
00 o/P leads various results, depending on the layer width n, and the subfamily which it belongs, (og)s (d))g. On one hand, the positive activation functions d lead consistently to standard Gaussian pre-activations for all 0 E [2.05, 10] (Fig. 7b). On the other hand, the odd activation functions di lead consistently to Gaussian pre-activations only for small 0 (Fig. 7a).
Second, our proposition of activation functions -=
19tn the Torch implementation of the training CIFAR-10: data point #47981 (class plane). This data point has been chosen randomly.
24

---

Published in Transactions on Machine Learning Research (04/2025)
layer width 10
layer wilth 100
Layer width 1000
2.06
R 7.00
0.4
0.4
2.50
0 10.00
0.08
66.
3.00
ReLU
63.
1.00
tonh
63.
5.00
XS, p-val 0.05
0.2
0.24
0.04
0.1
0.02
0.0
20
40
60
50
100
20
40
60
80
100
20
40
60
80
100
layer
layor
layer
(a) Distance [1f2; - Fall so; odd activation function ob: inputs normalized over the dataset (Definition 5).
layer width n,
layer width nu
100
layer width xy = 1000
0.10
0.10
0.10
of. 9 2,05
of. 5,00
o5. 2,50
of. 7.00
of. = 3,00
o.. 10.00
0.06
0.06
of. 4,00
KS. 0.05
0.00
0.02
0.02
0.00
Ne0 048
949 4y
0.00
0.00
0
20
09
90
100
20
40
60
30
100
20
80
60
50
100
layer
(b) Distance || Follooi positive activation function dB: imputs normalized the dalaset (Definition 5): these figures have been zoomed around 0.
Figure 7: Evolution of the distance between D and the standard Gaussian, during propagation where varies from to 100. If the pre-activations are Gaussian, these curves should remain close to zero. The dotted green line is the KS threshold: any point above it corresponds to a distribution D' for which the Gaussian hypothesis should be rejected with value 0.05. Weight initialization is W(0, 1) when using & 8 and is Gaussian according to the EOC when using tanh ReLU.
25

---

Published in Transactions on Machine Learning Research (04/2025)
Conclusion. For all tested widths, combining positive activation function of and weights sampled from W(0,1) leads to pre-activations that are close to (N 1), and obviously closer to (V 1) than with tanh or ReLU. However, our proposition of odd activation functions % leads to a distribution of pre-activations that drifts away from (00, 1) along the propagation, at least for 8 > 2.5. These results are confirmed with various inpuls, as shown Figures 16 and
Appendix H.4.
So, in the case Φ = og D' NO, 1) is close to a stable fixed point of the recurrecce relation D (D)) with the notations of Section 3.4 (sce Fig. 4). But, in the case ő &g. D' = =Vo, 1) is not stable fixed point. This deviation from the expected result is likely to be due to the assumption of independent pre- activations, which we used in the derivation of both 6 and ΦP. This assumption is discussed in Appendix B, where we show that such dependence plays more important role with g than with op.
This search for stable fixed points in the recurrence relation D = (T() is closely related to the discovery of stable fixed points for the sequence of variances (v/) and the sequence of correlations (car) (Poole et al., 2016, Schoenholz al., 2017), and may be explored further in future works.
5.3 Non-asymptotic Edge of Chaos
In Figure 8, we show the Edge of Chaos graphs for several activation functions: tanh and ReLU on one side, and our family (d on the other side. We remind that each graph corresponds to family of initialization standard deviations (aws σb) such that the sequence of correlations (c(+) converges to at a sub- sub-eaponential rate (see Section 3.1, Point 2). Such choices ensure that the initial correlation between two inputs changes slowly so that the information contained in these inputs is lost the slowest possible rate.
Instead of assuming that the pre-activations Central Limit Theorem, claim that, with our activation functions di, for any layer widths (including narrow layers and networks with various layer widths), the Edge of Chaos is non-asymptotic. Therefore, the corresponding curves in Figure 8 hold for realistic works
Gaussian
effect of the "infinite-widthh limit" and the
3.5
1i 0 2.05
3.0
6g. 0 2.50
<ū. 0 = 3.00
2.5
Φi 4.00
0g. 5.00
2.0
b 7.00
6g 0 10.00
1.5
tanh
1.0
x
ReLU
0.5
0.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Fw
Figure 8: Edge of Chaos for several activation functions. The ordered phase (resp. chaotic) lies above (resp. below) cach EOC curve.
20e fixed points Dt+1 (D') can also stationary distributions of the Markov chain (7)), if the layer width ny and the initialization distribution P;
26

---

Published in Transactions on Machine Learning Research (04/2025)
Remark 13. For the ReLU activation function, the EOC graph reduces to one point. One can refer to Hayou et al. (2019, Section 3.1) for complete study of the EOC of RRe.U-likee funclions", ; that is, functions that are linear on Rt and on R- with possibly different factors.
Also, as in Figure 1 (see Section 3.2), we have plotted the propagation of the correlations (Cpp) with Φ = 6g and weights sampled from W(O, 1) in Appendix H.2.
5.4 Training experiments
Finally, we compare the performance of a trained neural network when using tanh and ReLU, and our activation functions. Although the EOC framework (and ours) do not provide any quantitative prediction about the training trajectories, is necessary to analyze them in order to enrich the theory.
This section starts with a basic check of the training and test performances on a common task: training LeNet on CIFAR-10. Then, challenge our activation function, along with tanh and ReLU, by training MNIST diverse set of multilayer perceptrons, some of them being extreme (narrow and deep).
In the following, we train all the neural networks with the same optimizer, Adam, and the same learning rate 17 = 0.001. We use a scheduler and an early stopping mechanism, respectively based on the training loss and the validation loss, the test loss not being used during training. We did not use data augmentation. All the technical details are provided in Appendix H.5.
LeNet-type networks. We provide here the results only for odd activation functions Φg- The results with the positive ones of provided in Appendix H.6. The results are very similar in both cases.
We consider type networks (LeCun et al., 1998). They are made of two (5 x 3)-covvolutional layers, each of them followed by 2-stride average pooling, and then three fully-comnected layers. We denote by "6 16 1206--10--00 - 84-884-8 10" a LeNet neural network with two convolutional layers outputting respectively 6 and channels, and three fully connected layers having respectively 120, 84, and 10 outputs (the final out of size 10 is the output of the network).
We have tested LeNet with several sizes (see Figure 9). In Figure 9a, have plotted the training loss. Overall, ReLU and tanh perform well, along with some og with small @ and Paa,, with (6,w) (0.99, 2). In Figure 9b, the results in terms of test accuracy are quite different: ReLU and tanh still achieve good accuracy, but the other functions achieving similar results on the training loss seem to be a bit behind.
So, this standard setup, the functions we are proposing seem make the neural network trainable and as expressive as with other activation functions, but with some overfitting. This not surprising, since we are testing long-standing activation functions, ReLU and tanh, which have been selected both for their ability to make the neural network converge quickly with good generalization, against functions have designed only according to their ability to propagate information. Therefore, according to these plots, taking into account generalization may be the missing piece of our study.
Multilayer perceptron. We provide here the results only for our odd activation functions og. The results with the positive ones of are provided in Appendix H.6. The results are very different compared to the previous experiment: training narrow and deep neural networks is much more difficult with the positive activation functions than with the odd ones.
We have trained family of multilayer perceptrons MNIST. They have a constant width ny {3, 10) and depth E {3, 10). So, extreme cases such as a narrow and deep neural network (nz = 3, L = 30) have been tested.
A series results are presented in Figure 10. The training curves have been averaged over 5 experiments. In terms of training, the strength of our activation functions de more visible in the case of narrow neural networks (ny 3): in general, the loss decreases faster and attains better optima with = de than with Φ tanh or ReLU.
27

---

Published in Transactions on Machine Learning Research (04/2025)
LeNot: 3-8-80-42-10
LeNet: 6-16-120-84-10
LeNet: 12-32-240-168-1
2.5
2.5
2.0
2.0-
2.0
1.5
1.0-
1.0-
1.0-
0.5
0.5-
0.5
50
100
150
200
0
50
100
150
200
250
300
50
100
150
(a)
Training loss (negative logelikelihood).
LoNe;: 38 42
LeNet: 6:16 120
LoNet: 12 32
0.7
0.7
0.7
0.5-
0.5-
0.5
kanb
0.4
0.4-
ReLU
0.4
46. 6
2.05
46. 0
5.00
2.00
66. 6
2.50
46. 0 7.00
0.3-
Plssu 2.00
46. 6
3,00
4% 0 10.00
Plass 6.00
68. 6 6.00
0.2
30
100
200
0.2
50
200 250
300
0.2
50
100
150
(b) Test accuracy.
Figure 9: Training curves for LeNet with 3 different numbers of neurons per layer.
We also notice that training a narrow and deep neural network with a ReLU activation function is challenging. This result is consistent with several observations we have made in Section 3.2: in narrow ReLU networks, the sequence of correlations (Opg) fails to converge to (Fig. 1), and the pre-activations are far from being Gaussian (Fig. 2).
Finally, the case of d2.05 put aside, the results regarding the activation functions de consistent bet the two runs. This is not the case with tanh and ReLU.
6 Discussion
Generality of Constraints to 4. To ensure that pre-activations follow a Gaussian (o 1) distribution and that the weights in each layer are i..d. at initialization, the four constraints outlined in Section 3 must be satisfied. Relaxing these roists requires either breaking symmetries or addressing more complex problems. Let us consider two examples.
Example 1: instead of making all the pre-activations Gaussian, one may want to impose some other distribu- tion. In this case, the problem to solve would be more difficult: we would have to decompose a non-Gaussian random variable into a weighted sum v Σÿ= W,o(X,). In this case, we cannot use Proposition 3, and we would have to make WX)(X,) belong to family of random variables stable by multiplication by constant and by sum, for arbitrary n. If we aim for non-Gaussian Z, this task is much harder. For a study of infinitely wide neural networks going beyond Gaussian pre-activations, see Peluchetti et al. (2020).
1
Example 2: instead of assuming that all the weights of a given layer are i..dd one may want to initialize them with different distributions or to introduce a dependence structure between them. If the goal remains obtain Gaussian pre-activations, this kind of generalization should be feasible without drastically changing the constraints proposing.
28

---

Published in Transactions on Machine Learning Research (04/2025)
L 3
10; =3
L 30
2.5
2.5
1.5
1.5
1.5
1.0
1.0
1.0
0.6
0.5
0.0
3.0
50
100
100
20
0
50
100
150
900
50
100
150
200
L
3.; n
L
10
10
30 m
2.5
2.5
2.5
<g.
2.05
0g 8 7.00
2.0
og.
2.50
0g.
10.00
2.0-
2.0-
<8.
3.00
tanh
1.5
<6.
4.00
ReLU
1.5
<g.
5.00
1.0
1.0
1.0
0.5-
0.5-
0.55
50
100
150
0
50
100
150
200
30
100
150 200
Figure 10: Training loss for multilayer perceptron, narrow (n E {3, 103) and of various depths (L E {3, 33,10,30), Training curves averaged over experiments.
Hypothesis of independent pre-activations. In the constraints, we have assumed that the inputs of any layer are independent. This assumption is discussed in Remark and in Appendix B. According to the experimental results presented Appendix B, we can build specific cases where the dependence ween inputs breaks the Gaussianity of the outputted pre-activations, including with our pairs (0% (0%/,, PP).
However, the results are different with 63 and op. The difference between Figures 7a and 7b is striking: with og. the dependence between pro-activations tends to damage the Gaussianity after a certain number of layers, but with og. the Gaussianity of the pre-activations is remarkably well-preserved during propagation. The same kind of result is presented in Appendix B, Figure 11.
Other families of initialization distributions and activation functions. Given the constraints we have derived, we have made a choice to obtain our family of initialization distributions and activation functions: we have decided that the weights should be sampled from a symmetric Weibull distribution W(O, 1). We have made this choice because Weibull distributions meet immediately Constraint 2, and we can modulate their Generalized Weibull Tail parameter easily (see Remark 9).
One may propose another family of initialization distributions, as long it meets the constraints. However, such family should be selected wisely: if one chooses distribution with compact support, GWT(O) with 0 > 2, then the related activation function o is likely to be almost linear. Intuitively, if we want W (o(X) to be ((, 1) with X ~ (0, 1) and very light-tailed W, then & must "reproduce" the tail of its input X, so ø must be approximately linear around infinity.
Preserving a characteristic during propagation or imposing stable fixed points? Previous works have used both ideas: Glorot & Bengio (2010) aim at preserving the variance of the pre-activations, while Poole et al. (2016), Schoenholz et al. (2017) aim at imposing specific stable fixed point for the correlation map C. According to the results we have obtained in Section 5.1, it is possible to approrimately preserve the distribution of the pre-activations when passing through one layer. However, according to the results of Section 5.2, a drift can appear after several layers. Therefore, when testing an initialization setup in the real world, with numerical errors and approximations, it necessary check the stable fixed points of the monitored characteristic. However, this not casy to do in practice: without the Gaussian hypothesis, it can be difficult to find the possible limits of (c
29

---

Published in Transactions on Machine Learning Research (04/2025)
Gaussian pre-activations and Neural Tangent Kernels. With pair of activation functions and initialization procedure, we have been able to obtain non-asympiolic Edge of Chaos, removing the infinite- width assumption. Since this infinite-width assumption is also fundamental in works on the Neural Tangent Kernels (NTKs) (Jacot et al., 2018), would it be possible to obtain the same kind of results within the NTK setup? We do not think so. On the one hand, the infinit-wwidth limit used end up with an NTK that is constant during training, in order to provide exact equations of evolution of the trained neural network. On the other hand, in our setup, we only ensure Gaussian pre-activations, and this does not imply that the NTK would be constant during training. Nevertheless, with Gaussian pre-activations, we could expect an improvement in the convergence rate of the neural network towards a stacked Gaussian process, as the widths of the layers tend to infinity. Thus, the NTK regime would be easier to achieve.
A broadly useful technique. Our proposed sctup can be readily adopted by practitioners interested in testing the Gaussian hypothesis on problems beyond those considered in this work. The code used hoap/p/wo/ggausian-rreeee
here is available GitHub; https://github train, evaluate, and compare neural networks built using our family of activation functions and initialization distribut against standard choices (such as tanh or ReLU with Gaussian EOC initialization), to assess the impact of enforcing Gaussian pre-activations.
One could use this code to
Taking into account the generalization performance. In the EOC framework and ours, common principle could be discussed and improved. Generalization performance is not taken into account at any step of the reasoning. As we have seen in the training experiments, the main difficulty in our setup is overfitting as the training loss decreases without obstacles, the test loss ends up being worse than with ReLUU and tanh activation functions. To improve these generalization results, one can introduce a separation between the training set and a validation set into the framework.
Precise characterisation of the pre-activations distributions D'. Finally, finding an accurate and useful characterization of the of the pre-activations distributions D' remains an unsolved problem. In the EOC line of work, the problem has been simplified by using the Gaussian hypothesis. However, as shown in the present work, such a simplificatton is too coarse and leads to inconsistent results. Nevertheless, our approach has its own drawbacks. Namely, we can ensure Gaussian pre-activations only when using specific initialization distributions and activation functions. Thus, we still lack characterization of the distributions D' that is applicable to widely-used networks without rough approximations, and that can be casily used to achieve practical goals, such as finding an optimal initialization scheme. From this perspective, we expect that the problem statement of Section 3.4 will lead to fruitful future research.
Is it desirable to have Gaussian pre-activations? We have presented several results that should help to answer this tough question, but it remains difficult to answer it definitively:
there is paradox regarding the ReLU activation. On the one hand, when ReLU is used in narrow and deep perceptrons, the pre-activations are far from being Gaussian, the sequence of correlations does not converge to 1, and training is difficult and unstable. On the other hand, in LeNet-type networks with ReLU, training is easy, and the resulting networks generalize well;
our activation functions (o))s perform quite differently depending on the sctup: with Φ2.05, LeNet can achieve good training losses, but the training of narrow and deep perceptrons may fail. With $10.00, we observe opposite results;
when training narrow networks, the training curves are very different with og and og. They are difficult to train when using the functions ob, as well as ReLU (whose pre-activations are far from being Gaussian). But training is feasible and leads often to relatively good results when using 49.
Therefore, a temporary answer we can give is: with Gaussian pre-activations at initialization and odd activations functions, neural network is likely to be trainable.
30

---

Published in Transactions on Machine Learning Research (04/2025)
Is the "Gaussian pre-activations hypothesis" myth or reality? Our findings reveal that, when using the ReLU activation function in several practical scenarios, it is largely myth. In contrast, with the tanh activation function, the hypothesis holds true depending on the network's width: it becomes a reality with a sufficiently large yet reasonable mumber ncurons per layer. To extend this reality to networks with any layer width, we established a set of constraints that neural network designs must satisfy and proposed solutions to meet these constraints. Consequently, we developed a family of activation functions (Ø))) and initialization distributions Pe that fulfill these requirements, providing robust foundations for the Gaussian hypothesis and validating it across all tested cases.
Acknowledgements
The project leading to this work has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No 834175) and from the French National Research Agency (ANR-21-ISTM-0000] in the framework of the "Investissements d' d'avenir" program (ANR-15-IDEX-02) This work was granted access to the HPC resources of IDRIS under the allocation 2024-AD011013762R22 made by GENCI. We thank Thomas Dupic for proposing the trick involving the Laplace transform used in Appendix A.1. We would like to thank an anonymous reviewer for pointing out an error in a previons version of the paper, and proposing an example that inspired Example 1 (see App. B).
References
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
Julyan Arbel, Konstantinos Pitas, Mariia Vladimirova, and Vincent Fortuin. A primer on Bayesian neural networks: review and debates. Statistical Science, 2024.
Sanjeov Arora, Naday Cohen, Wei Hu, and Yuping Luo. Implicit regularization deep matrix factorization. Advances Neural Information Processing Systems, 32, 2019a.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- gene-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 2323.32. PMLR, 2019b.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhuldinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019c.
Krishnakumar Balasubramanian, Larry Goldstein, Nathan Ross, and Adil Salim. Gaussian random field approximation via stein's method with applications wide random neural networks. Applied and Com- putational Harmonic Analysís, pp. 101668, 2024.
Andrea Basteri and Dario Trevisan. Quantitative Gaussian approximation of randomly initialized deep neural networks. Machine Learning, 113(9):6373 6393, 2024.
Patrick Billingsley. Probobility and Measure. John Wiley & Sons, third edition, 1995.
Bruno De Finetti. La prévision: ses lois logiques, ses sources subjectives. In Annales de l'Institut Henri Poincaré, volume 7, pp. 1-68, 1937.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of doep noural networks. In International Conference on Machine Learning, pp. 1675-1685. PMLR, 2019.
Benjamin Epstein. Some applications of the Mellin transform in statistics. Slatistics, pp. 370 379, 1948.
The Annals of Mathematical
31

---

Published in Transactions on Machine Learning Research (04/2025)
Arthur Erdélyi, (Hans Heinrich) Wilhelm Magnus, Fritz Oberhettinger, Francesco Giacomo Tricomi, David Bertin, Watson B. Fulks, A. R. Harvey, Donald L. Thomsen, Jr., Maria A. Weber, E. L. Whitney, and Rosemarie Stampfel. Higher transcendental functions, volume 1. Bateman Manuscript Project, 1953.
Stefano Favaro, Boris Hanin, Domenico Marinueci, Ivan Nourdin, and Giovanni Peccati. Quantitative CLTs in doep neural networks. Probability Theory and Related Fields, pp. 1-45, 2025.
Vincent Fortuin. Priors in Bayesian deep learning: A review. International Statistical Review, 2022.
Vincent Fortuin, Adrià Marriga-Alonso, Sebastian W. Ober, Florian Wenzel, Gunnar Ratsch, Richard E Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. In Inter- national Conference on Learning Representations, 2022.
Bruno Gabutti and Laura Sacripante, Numerical inversion of the Mellin transform by accelerated series of Laguerre polynomials. Journal of Compulational and Applied Mathematics, 34(2):191 200, 1991.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 249-256.
JMLR Workshop and Conference Proceedings, 2010.
Izrail Solomonovich Gradshleyn and losif Moiseevich Ryzhik. Table of Integrals, Series, and Producls. Academic Press, eighth edition, 2014.
Alex Graves. Practical variational inference for neural norksrks Advances in Neural Information Processing Systems, 24, 2011.
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. In International Conference on Learning Representations, 2020.
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International Conference on Machine Learning, pp. 2672-2680. PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Matthew Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 2013.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Diederik P Kingma and Jimmy L Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalixing neural networks. Advances in Neural Information Processing Systems, 30, 2017.
Andrey Kohmogoroff. Confidence limits for an unknown distribution function. The Annals of Mathematical Statistics, 12(4):4614463 1941.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to doeu- ment recognition. Proceedings of the IEFE, 86(11):2278 2324, 1998.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel $ Schoenholz, Jeffrey Pennington, and Jascha Sohl- Dickstein. Deep Neural Networks as Gaussian Processes. In International Conference on Learning Repre- sentations, 2018.
Jachoon Loe, Lechao Xiao, Samuel Schocnholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems, 32, 2019.
32

---

Published in Transactions on Machine Learning Research (04/2025)
Mufan Li, Mihai Nica, and Dan Roy. The neural covariance sde: Shaped infinite depth-and wndhhhh networks at initialization. Advances in Neural Information Processing Systems, 35:10795- 10808, 2022.
Haitao Mao, Xu Chen, Qiang Fu, Lun Du, Shi Han, and Dongmei Zhang. Neuron campaign for initialization guided by information bottleneck theory. In Proceedings of the 30th ACM International Conference Information & Knowledge Management, pp.
2021.
James Martens, Andy Ballard, Guillaume Desjardins, Grzegorz Swirszez, Valentin Dalibard, Jascha Sohl- Dickstein, and Samuel S Schoenholz. Rapid training of deep neural networks without skip connections normalization layers using deep kernel shaping. orXiv preprinl ar arXi:22110.01766 2021.
Alexander G de G Matthows, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahramani. Gaus- sian process behaviour in wide deep neural networks. In International Conference on Learning Represen- tations, 2018.
Radford M Neal. Bayesion learning for neural nelworks, volume 118 of Leclure Notes in Stalislics. Springer, New York, 1996.
Lorenzo Noci, Gregor Bachmann, Kevin Roth, Sebastian Nowozin, and Thomas Hofmann. Precise charac- terization of the prior predictive distribution of deep ReLU networks. Advances in Newral Information Processing Systems, 34:20851 20862, 2021.
Yann Ollivier. Online natural gradient as Kalman filter. Electronic Journal of Statistics, 12(2):2930 2961, 2018.
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, Aliaksandr Hubin, Alexander Immer, The- ofanis Karaletsos, Mohammad Emtiyaz Khan, Agustinus Kristiadi, Yingzhen Ti, Jose Miguel Hernandez Lobato, Stephan Mandt, Christopher Nemeth, Michael A. Osborne, Tim G. J. Rudner, David Rügamer, Yee Whye Tch, Max Welling, Andrew Gordon Wilson, and Ruqi Zhang. Position Paper: Bayesian Deep Learning is Needed in the Age of Large-Scale AI. In International Conference on Machine Learning, pp. 39556-33 39566. PMLR, 2024.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. 1310 1318. PMLR, 2013.
Stefano Peluchetti, Stefano Favaro, and Sandra Fortini. Stable behaviour of infinitely wide deep neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 1137-1146. PMLR, 2020.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential ex- pressivity in deep neural networks through transient chaos. Advances in Neural Information Processing Systems, 29, 2016.
Hadi Pouransari, Zhucheng Tu, and Oncel Tuzel. Least squares binary quantization of neural networks. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 698 699, 2020.
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks. Cambridge University Press, 2022.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning, Journal of Statistical Mechanics: Theory and Esperiment, 2019 (0):112):22400, 2019.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha tion. In International Conference on Learning Representations, 2017.
Deep information propaga-
Mariia Seleznova and Gitta Kutyniok. Analyzing finite neural networks: Can we trust neural tangent kernel theory? In Mathemalical and Scientific Machine Learning, pp. 868 895. PMI.R, 2022.
33

---

Published in Transactions on Machine Learning Research (04/2025)
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint
2017.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Newral Information Processing Systems, 33:7462-7473, 2020.
Nickolay Smirnov. Table for estimating the goodness of fit of empirical distributions. The Annals of Math- ematical Statistics, 19(2):279-28) 1948.
P S Theocaris and A C Chrysakis. Numerical inversion of the Mellin transform. IMA Journal of Applied Mathemalics, 20(1):73 83, 1977.
Naftali Tishby. The information bottleneck method. In Proc. 37th Annual Allerton Conference Commu- nicalions, Control and Compuling, 1999, pp. 368 377, 1999.
Edward Charles Titchmarsh.. Introduction to the theory of Fourier integrals. The Clarendon Press, Oxford, 2nd edition, 1937.
Dario Trevisan. Wide deep neural networks with Gaussian weights are very close to Gaussian processes. arXiv preprint arXim:2312.11799 2023.
Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. Understanding priors in Bayesian neural networks at the unit level. In International Conference on Machine Learning, pp. 6458-6467.
PMLR, 2019.
Mariia Vladimirova, Julyan Arbel, and Stéphanc Girard. Bayesian neural network unit priors and generalized Weibull-tail property. In Asion Conference Machine Learning, pp. 1397-1412. PMLR, 2021.
Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? In International Conference on Machine Learning, pp. 10248-10259. PMLR, 2020.
Sho Yaida. Non-gaussian processes and neural networks at finite widths. Mathematical and Scientific Machine Learning, pp. 165 192. PMLR, 2020.
Guodong Zhang, Aleksandar Botev, and James Martens. Deep learning without shorteuts: Shaping the kernel with tailored rectifiers. International Conference on Learning Representations, 2022.
34

---

Published in Transactions on Machine Learning Research (04/2025)
A Activation function with infinite number of stable fixed points for V
A.1 Proof that V admits an infinite number of fixed points when using = 46,,
Proposition 2. For any ó E (0, 1] and w > O, let us pose the activation function & Po.w- We consider the sequence (v"), defined by:
VI 120, 0, 0ltl = o =ou
Pow (Volz)² D= + oi
(52)
00 E E...
Then there erists ow > 0, σb 0, and strictly increasing sequence of stable fized points (ot)kez of the recurrence Equation (52).
Proof. Let us define:
V(v)= = V(ulow = 1,0, = 0),
have:
Viclo 06) oz vV()) toū
(53)
In the following, we use a simplified notation: V(u) = V(vlow: 0b)- Our goal is to find a sequence (VE)kEz, ow and σb such that:
Vk E Z, V(vE) v and V((tt) E (-1,1),
which would ensure that all v are stable fixed points V.
V. In order to understand how to build the sequence (ut)eez, let consider Let of and où 1/V(v). So we have:
(
So, any v can possibly be a fixed point we tune aw accordingly. We just have find such that: (v) (-1,1) 1), with:
1 (V(v) + vV( V(o)
V'())
V. (In())).
(54)
where D. r In(((exp()))- So, knowing that V. is e³ and periodic, it sufficient to prove that not constant to ensure that we can extract one v such that V'(a)) (-1,1). Then, by periodicity of V., we can build sequence of stable fixed points (ux)xez.
We have:
((v)= = _ Pa (V Dz = 10= &eep _ sinwwin (n Dz
_ - (ii(ww) Dz = ( exp 2 sin f Dz.
= 2 z exp
Lemma 1. V is not constant.
Proof.
2
5 sin(w 1n
exp (-
x2 dz
2² exp
2
2T
03/2
2v
1
2
2 5 sin(w In v2) exp
22T
3//2
2/2
exp
dz
1
33/2
L.
[5مممc (V2)
2m
2v
35

---

Published in Transactions on Machine Learning Research (04/2025)
where C is the Laplace transform.
Let us suppose that V is constant: Vo > 0, V(v) = C. So, if we pose v + L. 20 we have:
Ve 0, c=
3/2 /100,u (VE) (v),
V
that is:
[[p) (v) CVTT v 33322
Since the function (V2) is continuous on R+, then, almost everywhere (sce Thm. 22.2, Billingsley, 1995):
(Vz) c//T (2)) = c/2,
2
which is impossible for 8 E (0, 1]. Hence the result.
The function Ve : In((eexp())) is continuous and 4r " -erridddi::
(r) In /- 2² exp 2 sin (~" + w In D2
Ve is lower and upper bounded and reach its bounds (and, by Lemma 1, these bounds are different). We define:
DE- max Ve
rot = inf fr > 0: V.(r) Dー ro = inf (r > + Ve(r) = V.
D. min D.
By continuity, V.(rt)) = Vt and (r(r))= V.. Since D シー and V. is C21 then there exists rō E (rt,1)) such that (1() E (-2,0).
Since V. is 4 -periodic, can define sequence (rt)kez such that:
r = r + 4km
w
v.(rz) = ve(ri) =: 1000
(r 12(r))) E (-2,0).
So, by using Eqn. (53) and Eqn. (54) with of = 0 and o = 1/ exp(
o exp(rи);
V(ng) = exp() ot (1t)) = vi
V'( = (1r) E (-1, 1).
Thus, (bt)ezz a sequence of stable fixed points V--law σh) for well-choson ow and db-
A.2 Practical computation of o²
We propose practical method to ensure that V(-low, 0b 0) admits an infinite number of stable fixed points when using activation function & P8.0-
36

---

Published in Transactions on Machine Learning Research (04/2025)
In order to achieve this goal, we build ow ow in the following way:
8
now :=
x² exp
(in(wl((2)) D=
0
w
uupp :=
2² Exp
sin(w In(2)) D2,
Viow + Vupp
2
In practice, for 8 == 0.99, we obtain ou for w E (3
02 0.879, 03 0.945, Ø6 0.987.
B Discussion about the independence of the pre-activations
In Proposition 3 and Constraint 1, we assume that, for any layer I, its inputs (Z)), independent. This does not hold in full generality:
Example 1. Let X ~ 1)) be some random input of a two-layer neural network. We perform the following operalion:
= V2 WPO(W/)) + (2
where (W1, W2, WT, W3) be i.idd. random variables samples from some distribulion and d is some acti- vation function.
Let P = R, the Rademacher distribution, i.c., if W ~ R, then W =1 with probability 1/2. Let Φ = Id.
Then we have: Y1 WIX ~ 1), and Y2 = W X ~ (N 1). But they are not independent:
W
w?w, (W) W
where WiW! and W2W, are two independent Rademacher random variables. So, Z = 0 with probability 1/2. So, Z is not Gaussian.
In this example, we build a non Gaussian random variable with a minimal neural network, in which we con- struct two dependent random variables. So, we should pay attention to this phenomenon when propagating the pro-activations in a neural network.
One should note that the structure of dependence of WPWFX X and W X does not involve their correlation (which is zero), and yet breaks the Gaussianity of their sum. So, in order to obtain theoretical result about the distribution of Z, we should study finer aspects of the dependence structure.
But, on the practical side, we want to answer the question: to which extent does the relation of dependence between the pre-activations (Z/), affect the Gaussianity of (2!+)),
In order answer this question, we propose series of experiments on two-layer neural network. We consider vector of inputs X Rm0,, where the (Xj)1sısn (O, 1) and i.i.d. The scalar outputted by the network is:
(
1
Vnu
where the weights W Th Xmo and 2 E 1lmm are i.dd
We test this setup with different initialization distributions P and activation functions 6:
usual ones: o tanh or ReLU, P (N,0 ).)) where of is such that the pair (o of 0.01) lies the EOC;
37

---

Published in Transactions on Machine Learning Research (04/2025)
ours: Øg. P= Po W)
We study three cases:
unfavorable case: 1, various E [0]:
the intermediary features W X are weakly "mixed", so it is credible that they lead to an output that far from boing Gaussian;
favorable n1 2, various no E [1, 10):21
the intermediary features W are "mixed" with an increasing rate as no increases;
same-width case: no = n1.
KS-statistic for various layer sizes a 2-layer NN
70p 1, warying ni. db
Ti = 2, varying no. db
warying Bg. Tr. &
10.1
2.16
7.00
10.
2.50
10.00
10
BeLU
#= 4.00
troh
100
00"3
10r2
11g
770
Flp 1. varying 1, ob
F1 varying T4. 6
varying 10g 71, 0
30
13
10
2
10
Figure 11: Evolution of the distance of the standardized distribution of Z to the N(0, 1) according to the Kolmogorov Smirnov statistic. Weight initialization is W(0, 1) when using Ón and is Gaussian according to the EOC when using Ø = tanh or ReLU, For each point, we have computed the KS-statistic over 200 000 samples of (X, WT W In the first row, we have plotted the results for the odd activation functions dg- In the second row, we have plotted the results for the positive activation functions of.
According to Figure 11:
unfavorable case (Ist graph, no = 1): Z is far from being Gaussian for ReLU, tanh, and the Φ9, but much better for the ob. especially with small 0;
favorable case (2nd graph, n1 = 2): is closer to be Gaussian with our setups (Po, 60/P) than with tanh or ReLU, especially with larger ni;
same-width case (3rd graph, For a fixed n, the distribution of is closer to a Gaussian with smaller 0. When we use (Po, 0g 60/P), we are close the performance of tanh or better.
n1): the larger the width n, the closer Z is to being Gaussian.
21ehe case 11 is trivial: there is sum dependent, random variables in the second layer.
38

---

Published in Transactions on Machine Learning Research (04/2025)
The case no n1 is the most realistic one: usually, the sizes of the layers of neural network are of the same order of magnitude. In this case, our setup is better than or equivalent to the ones with tanh or ReL.U.
Mixing of inputs and interference phenomenon. We see in Figure 11 that, in every graph, sctups with small 0 lead to better results than the ones with large 0. This observation could be explained by Example 1. In this example, the inputs are weakly "mixed": since Xg is multiplied by a Rademacher random variable, it is possible partially reconstruct Xa after the first layer, and then build destructive interference (leading to output Z 0 half of the time).
So, if we want to avoid this "interference" phenomenon, we should use initialization distributions and ac- tivation functions such that every layer "mixes" strongly the inputs. Notably, initialization distributions should be far from being a combination of Dirac distributions. Typically, Pø=10 is close to the Rademacher distribution (see Fig. 5a).
Finally, this phenomenon can be greatly reduced by using our family (og)e of positive activation functions (see second row of Figure 11).
C Constraints on the product of two random variables
Proposition 4 (Density of a product of random variables at 0). Let W, Y be two independent. non-negative random variables and Z = WY. Let fw, fv fz be their respective density. Assuming that fy is continuous at with fy (0) > O, we have:
fw(() dt =0, then
if
lim 10+0 w
lim fz(2) oc. 2-0
(55)
Moreover, if fy is bounded:
Sw de oc,
then Sz(0) = 5r(0) / fw(() dl.
is
(56)
Proof. Let 20 > 0:
fz(2) = / 0c
5r(1) fw (*) dt
2 1 (r(t)
1 fw (7) dt 2
20 fw
00
fwr(t)
inf fr [0,20]
dt >
inf
fr
dt.
[0,20]
2/20 t
Let us take 20 = VE. We have:
fz(2) inf fr. f fw((t) dl.
10.E
Then we take the limit z + 0, hence:
if
fo fw(l) dt = co, then: lim fz(2) = 0o, which achieves (55);
if fo fwil) de < co, then: fz(0) 2 (r(0) fo fw(()) dt, which achieves one half of (56);
Let us prove the second half of (56). Let 2, 20 > 0:
f x 1r(0 (7) dt
fz(2)
f0a Fr(0)
fw (7) dt+ + I fr(t) sw (7) dt
pro 1
1 Sw((), di f Ir (zol) 1 7ow
[=] de
S sup fr
[0.20]
2/20
39

---

Published in Transactions on Machine Learning Research (04/2025)
Let 20 V2. We have:
2 ( dl, 1/= t dl +dd++/ / fr (VEl) j.w
00 fw(()
fz(z) S sup .
(0.a
where:
J
1 00 fy (VEt) fw f
V2 dt S fl 1l0 J, /,
0c fow
V2
dt S llfr prv fw (t)
t dt.
According to the hypotheses, we have, as - 0:
[re (t)
t dt I 00 fw((t)
lo v2 sw (1)
sup fr - (0).
dt,
dt +
[0.VE]
t
hence the result.
D Activation functions with vertical tangent at 0
In the following lemma, we show that if we want the activation Y to have a density that is 0 at O, then the activation function & should have a vertical tangent at O. G plays the role of pre-activation.
Let & be a function transforming a Gaussion random variable ~ NO0 1) into a symmetrical random variable with density fy such that fr 0. That is, Y ((G). Then has a vertical tangent al 0.
Lemma 2.
Proof. We have:
(<x) = Fyy=
where F& and Fy are the respective CDFs of G and Y.
Thus:
x'xx= = Fz(2) F((z=
Therefore:
6'(0) = (0)
Fy (Fr (Fo(O)))
V (Fy (1/2)) V2 FG
oc.
E
The Mellin transform
E.1
Generalities
We assume that G WY ~ (O0, 1). Let us consider the random variables [W], | and |G] |W|-|Y| Let
[w JIY and fia! be their densities. Under integrability conditions, we can express the density fia of the product |G| = |W][Y| with the product-convvlution operator
f1))(2) (f))) where ((w))siii(()== = fiwl (2) fm
We can also express the CDF of G this way:
(z)(2)= .. Fwwl (7) (t)
(57)
40

---

Published in Transactions on Machine Learning Research (04/2025)
Then, we use the following property of the Mellin transform M:
Mfa = (MSwwi) (MJiri), where (MJ(I) = [o s(x) dx.
In short, M transforms a product-convolution into product in the same manner as the Fourier transform F transforms convolution into product. We have then:
MSia
fir(()) = M-1
(v).
Mff
Then, by symmetry, can obtain fv from Jiy-- However, while Mfía and Mfiwi are easy to compute, the inverse Mellin transform M seems to be analytically untractable in this case:
(MSa)))) 28 2T3) VT
8 s-11 18-1+1 -1).
(MS(wW)(s) =T
(Mfia))(s) (Mf(w)(())
TT))
S0:
+1)
E.2 Numerical inversion of the Mellin transform
Computation of Jly by numerical inverse Mellin transform. The Mellin transform of function can be inverted by using Laguerre polynomials. Specifically, we use the method proposed by Theocaris & Chrysakis (1977) and slightly accelerated by the numerical procedure of Gabutti & Sacripante (1991):
k h: (-1)a-1 f(n) [n) Ck+1 L (5) with =
(== =
(58)
k=0
where the (La) are the Laguerre polynomials (sce Section 7.41, Gradshteyn & Ryzhik, 2014).
Experiments. A common way of computing the inverse Mellin transform consists of using Equation (58).
Specifically, the sequence (c*) must be computed.
In order to compute the density Jy of Qo with = 2.05, we have computed numerically (cx)a for [1,500]. The results are plotted in Figure 12. We have tested three methods compute the cy:
Soating-point operations using 64 bits floats;
floating-point operations using 128 bits floats;
SymPy: make the whole computation using SymPy, Python library of symbolic computation (very slow).
In all three cases, instabilities appear before the sequence (cx)s has fully converged to Moreover, the oscillations of (cx)& around have an increasing wavelength, which indicates that we may have to go far beyond k 500 get enough coefficients (cx)x to reconstruct the wanted inverse Mellin transform.
In Figure 13, have plotted two estimations of f(r): the density obtained directly by using (c*)ke(1,300) and the density obtained by using a sequence (c=))E[1.20000) where the values (cx (c=))e[301,2000 20000) have been extrapolated from (c000=[1.300)." 22
The resulting estimations of the density fix take negative values and seem to noised. So, more work is needed to obtain smooth and proper densities, especially if we want them to meet Constraints 3 and 4 (density 0 and decay rate at oc).
The extrapolation has been performed by modeling the graph of (cu)x the product decreasing function cosine with decreasing frequency.
41

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
floating-point, 64 bits
doating-poinn 128 bits
0.5
SymPy
0.0
-0.5
-10
100
200
300
100
500
Figure 12: Evolution of various numerical computations of cp. as k grows.
1.5
(cN)eeLLN)) N 300 (cu)sef)NN) extrapolated with N 20000
1.0
MAO
0.5
0.0
2
6
10
Figure 13: Numerical inverse Mellin transform with two different computations of (ck)s: direct computation of (cu)kE(L,N) with N 300; extrapolation of (c4)) from (c+))e(1,300)
Conclusion.
We observe that this computation of the inverse Mellin transform has several intrinsic prob-
lems:
the computation of the a coefficients involves a sum of terms with alternating signs, which become larger (in absolute value) as grows and which are supposed to compensate such that ck - 0 as k co. Such numerical computation, involving both large and small terms, makes the resulting Ck very unstable as k grows;
when 2, the sequence (cx)s tends extremely slowly to 0:
we approximate M- with the finite sum of the first K terms of the scries in Equation (58), we cannol guarantee the non-negativeness of the resulting function, which meant be a density.
So, this method is unpractical to compute the density of distribution in case.
42

---

Published in Transactions on Machine Learning Research (04/2025)
F
Obtaining fivt= experimental details
We optimize the vector of parameters A with respect to the following loss:
((A) == |lFx Fiaillo
PA(2) Lo- Fiwi (7) ga(t) dt.
Datasct. We build the datasct 2 of size d:
1
0, mmax d
mmax d - 1 11 mmax
In our setup, d 200 and Zmax 5.
Computing the loss. For each z in 2, we compute numerically (z) Then, we are able to compute (A)). We keep track of the computational graph with PyTorch, order backpropagate the gradient and train the parameters A by gradient descent.
Initialization of the parameters. We initialize (a, 7, A1, X2) in the following way: 3; 1:A1 -1;2==1
Optimizer. We use the Adam optimizer (Kingma & Ba, 2015) with the parameters: learning rate 0.001: 3 0.9; B2 = 0.999; weight decay 0. We train A for 100 epochs.
Learning rate scheduler. We learning rate scheduler based the reduction of the training loss. If the training loss does not decrease at least by factor 0.01 for 20 epochs, then the learning rate is multiplied by factor 1/V V10. After modification of the learning rate, we wait at least 20 epochs before any modification.
Scheduler for 0' We recall that the definition of d사 involves 0', defined by: 1/2 + 0r = It is not a parameter to train. Empirically, we found that the following schedule improves the optimization process:
from epoch 0 to epoch 49, 0' increases lincarly from 2 to its thcoretical value (2 - d
at the beginning of epoch 50, reinitialize the optimizer and the learning rate scheduler;
we finish the training normally, with 0' = ( -
G The Kolmogorov-Smirnov test
Description. We describe here the Kolmogorov Sommogorov-Smirnov (KS) test (Kolmogoroff, 1941, Smirnov, 1948). Given a sequence ".,,... Z2) of s i.i.d. random variables sampled from Pt:
we build the empirical CDF F. of this sample: F2(2) = 1 Σxm 1z55si
2. we compare F, the CDF Fg of G ~ NN(, 1) by using the ~ norm: D, ||FF, - Falloc, where
D. is the "KS statistic":
3. under the null hypothesis, ie. = NN(0,) 1), we have: VsD, $ K, where K is the Kolmogorov distribution (Smirnov, 1948). We denote by (Ka)a the quantiles of K: P(K & Ka) = - a, for all
a [0, 1]:
4. finally, reject the null hypothesis level if: VoD, S K..
43

---

Published in Transactions on Machine Learning Research (04/2025)
Limitations. In the KS test presented above, the null hypothesis Hy is P' = =O which is exactly what we intend to demonstrate when using our family {(Ps, ") (0)), while the alternative hypothesis H, is P' NV.. 1). With this test design, we face a problem: is impossible to claim that Hg holds. More precisely, we have two possible outcomes: either Hg is rejected, or it is not. Since H1 is the complementary of Ho, reject of Hig means accept of H.]. But the converse does not hold: if Hig is not rejocted, then it is impossible to conclude that Hg is true: the sample size may simply be too small, the KS test may be inadequate for our case.
So, to be able to conclude that the pre-activations are Gaussian, we should have built an alternative test, where the null hypothesis Hy is some misfit between P and (O 1). Or, at least, we should have computed the power of the current KS test.
However, experimental rosults sufficient compare the quality of fit between P and A in the tested selups (see Figures 6 and 7). It remains clear that, with ((Pe, oo) 0 E (2, x)), the Gaussian pre-activations hypothesis is far more likely to hold than in setups involving tanh, ReLU and Gaussian weights.
H Experiments
H.1 Propagation of the correlations with MNIST
Within the setup of Section 3.2, we have plotted in Figure 14 the correlations propagated in a multilayer perceptron with inputs sampled for MNIST (10 samples per class, that is, 100 samples in total). The layers of the perceptron have ni neurons each, and the activation function is Φ = ReLU.
We observe that, in this narrow NN case (nu 10), some irregularities appear as information propagates into the network: some classes seem to (de)ccrrelate in an inconsistent way with the others. Specifically, while most of the classes tend to correlate exactly (C* s 1), the third class (corresponding to the digit. "2")
tends to correlation C* = 0.7 * 1. This result is contradictory to the EOC theory.
input
= 30
= 50
ReLU (MNIST)
1.0
-0.5
0.0
0.5
1.0
Correlation
Figure 14: Propagation of correlations dg in multilayer perceptron with activation function ReLU and inputs sampled from the MNIST datasct. The neural network is initialized at the EOC. Each plot displays 10 x 10 matrix Ch whose entries are the average correlation between the pre-activations propagated by samples from classes p.g E (O.... .9. at the input and right after layers 1 E {10, 30, 503. See also Figure 1 in Section 3.2 for results on CIFAR-10.
H.2 Propagation of the correlations with Φg
Within the setup of Section 3.2, we have plotted in Figure 15 the correlations propagated in multilayer perceptron with & do. The weights have been sampled from W(0, 1).
44

---

Published in Transactions on Machine Learning Research (04/2025)
In this setup, the results consistent, and are consistent with the results for o tanh (sce Figure 1): the sequence (C converges to 1, which was not the case for & ReLU and ny = 10.
H.3 Variance of the pre-activation when using 6 = 0g
As observed in Figure 6a, the product W (xo(X) is above the KS threshold, corresponding to s = 10 samples and a p-value of 0.05. This not necessarily the case in Figure 6b, where the samples are standardized. So, we suspect that the variance of W (X) is not exactly
Therefore, we have reported in Table 2 the empirical standard deviation of the product Wo(X) (i.c., Z' with 1); computed with 10? samples, where X ~ (0, 1), W W(O. 1) if ø = d0 and W ~N 1) if @ = tanh or ReLU. We observe that the standard deviation of Z', which is expected to be 1 with Φ &g is actually a bit different. This would largely explain the differences between Figure 6a and Figure 6b.
Table 2:
Empirical standard deviation of Z' with n
68
tanh
ReLU
2.05
2.5
4
5
7
10
1.003
0.994
0.989
0.984
0.981
0.979
0.978
0.628
0.707
H.4 Propagation of the pre-activations
In this subsection, we show how the choice of the input data points affect the propagation of the pre- activations.
Let D' be the distribution of the pre-activation Zi after layer I. In Figure 16, we have plotted the _ distance between the CDFs of 'D' and (N(01) 1) for various input points, sampled from different classes to improve diversity between them. We have chosen the following setup: CIFAR-10 inputs, multilayer perceptron with 100 layers and 100 ncurons per layer, Gaussian initialization at the EOC when using the activation function Φ = tanh or ReLU, and symmetric Weibull initialization W(0, 1) when using & = Φ8 or & db.
This setup has been selected illustrate clearly the variability of D' when using various data points. This variability can be observed with 10 or 1000 neurons per layer, though it is less striking.
Input normalization over the whole dataset. We perform the usual normalization over the whole dataset:
Tojij Mi
Bogj
0i
14 =
Tij
Np: x=D
1
0f Np;
(zaj -14)²
XED j=1
where & is the normalized data point, Tazi is the j-th component of the i-th channel of the input image Xa: Pe is the size of the i-th channel, and N is the size of the dataset D.
Results. In Figure 16, we observe that, when using the activation function & = tanh, the shape of the curves the same for all data points. This is not the case for ReLU: for the input point "bird", the sequence (D'), drifts away from A/ 1) since the beginning, while for the input "car", (D'), first becomes closer (00 1), then drifts away. It is also the case for de with 0 10: for the input "deer", the distance remains high, while it starts low and increases very slowly (input "dog"), or even decreases in the first place (input
45

---

Published in Transactions on Machine Learning Research (04/2025)
input
1 = 10
1 = 30
1 = 50
11 10 neurons per layer
22.05 2.05
$9.00
T1 = 100 neurons per layer
22.0
67
1.0
-0.5
0.0
0.5
1.0
Correlation
Figure 15: Propagation of correlations do in multilayer perceptron with activation function dg with 0 E (2.05, 7.00} and inputs sampled from the CIFAR-10 dataset. The weights are sampled from W(0,) 1) and the biases are zero. Each plot displays a 10 x 10 matrix Ct whose entries are the average correlation between the pre-activations propagated by samples from classes p.q E {0, ,9), at the input and right after layers 1 110,30,50 503.
46

---

Published in Transactions on Machine Learning Research (04/2025)
"truck"). But, in general, when the curves have converged after 100 layers, it seems that the limit is the same whatever the starting data point, and depends only on the choice of the activation function.
The most striking result lies in the difference between Figures 16 and 17. With o = og the pre-activations tend always to be Gaussian, whatever & or the input data point. This is not the case with ReLU, tanh and some ob with large 0. However, ob remain "more Gaussian" than ReLU or tanh.
Data point 447981 (das planc)
0.5
65.
46.
0g.
4G. 10.00
3,00
bonh
03-
6
4.00
ReLU
63. 0 5,00
RS, p vol 6.05
0.2-
0.0
Data point #21228 (cless car)
Data point # 10725 (clase bard)
Data point #21574 (class cat)
0.5
0.5
0.5
0.4-
0.4-
0.4
0.3
0.2
0.2
0.2
0.0
0.0
0.3
100
100
Data point If 12672 (class doer)
Data point #21928 (class dog)
Data point # 19676 (class frog)
0.5
0.5
0.4-
0.4
0.1
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
20
160
0
20
100
70
80
100
Daia point. #21000 (class = barse)
Dali point #26720 (cluss ships
Data: pointl # 417229 (class truck)
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.1
0.1
0.1
20
30 100
20
60 100)
0
21
40
60
50
100
Figure 16: Activations functions tanh, ReLU and dg. Propagation of the distribution D' of the pre-activations across the layers 1 E [1, 100] when inputting various data points of CIFAR-10. Each curve represents the e distance between the CDF of D' and the CDF of the Gaussian A(0 1). The dashed green line is the threshold of rejection the Kolmogoror Smirnov test with v-vuuue 0.05: if a distribution D' represented by a point above this threshold, then the hypothesis "D' = =N 1)" is rejected with p-value 0.05.
Dolmogoror-Smmrnov
47

---

Published in Transactions on Machine Learning Research (04/2025)
Data poin: 4447881 (class = planc)
8-
66. 600
1..0-22.. 2,50
0g. 7.00
044
0- 3,00
og.
10.00
0.3-
85, p-oal 0.65
0.7
0.1
0
23
60
80
100)
Duta point #22222 (class par)
Dota point 4 10725 [elass bire))
Data point #221574 (cless pab)
0.5
0.5
0.5
0.3
0.3
0.3
0.1
0.1-
0.1
0.0
0.0
211
60
90
-30
40
00
50
3001
Data point # 12672 (class doer)
Data point #21928 (clase dog)
Data point 196670 (clase ftog)
0.5
0.5
0.5
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.0
0.0
0.0
20
80
100
Data point #21007 (claes- horse)
Data point #20720 (clase shipp
Data point #1572 (class truck)
0.5
0.5
0.5
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
01.00
20
100
20
40
60
100
Figure 17: Activation functions of, same setup as in Figure 16.
H.5 Experimental details of the training procedure
Training, validation, and test sets. For MNIST and CIFAR-10, we split randomly the initial training set into two sets: the training set, which will be actually used train the neural network, and the validation set, which will be used to stop training when the network begins to overfit.
The sizes of the different sets are as follows:
MNIST: 50000 training samples; 10000 validation samples; 10000 test samples;
CIFAR-10: 42000 training samples, 8000 validation samples; 10000 test samples.
The training sets are split into mini-batches with 200 samples each. No data augmentation is performed.
Loss. Given a classification task with P classes, let z E R be the pro-activation outputted by the last layer of the neural network. First, we perform a softmax operation:
exp(2f))
yp : s0ftmax(zt)
Σ/= exp(2
48

---

Published in Transactions on Machine Learning Research (04/2025)
where the (Up)p are the components of y E R and (zp)p the components of z Then, compute the negative log-likelihood loss. For target class p E {1,-. P), we pose:
E(y.p) == log(Up))
Optimizer. We use the Adam optimizer (Kingma & Ba, 2015) with the parameters: learning rate 0.001; 81 = 0.9; 82 = 0.999; weight decay- = 0.
Learning rate scheduler. We use a learning rate scheduler based on the reduction of the training loss. If the training loss does not decrease at least by a factor 0.01 for 10 epochs, then the learning rate is multiplied by a factor 1/vI0.
Early stopping. We add an early stopping rule based on the reduction of the validation loss. If the validation loss does not decrease at least by factor of 0.001 for 30 epochs, then we stop training.
H.6 Additional training results: o= =
We provide here additional training experiments, with the same sctup as Scction 5.4 but with our family of positive activation functions (o%)..
Results with CIFAR-10 LeNet. We have plotted in Figure 18 the evolution of the training loss and the test accuracy during training. The test loss is slightly better than with the family of odd activation functions (ø)) (see Fig. 9b) and the training loss is slightly worse (see Fig. 9a). Overall, the plots are very similar with both types of activations functions.
LeNet:: 3-8-00-42-10
LeNet: 6 18-11-0----110
LeNet: 12-32-290-18 200-1880----110
2.5
2.5
2.5
2.0
2.0
1.5
1.5
1.0
1.0
0.5
0.5
0.0
0.0
aa)
Training loss (negative logelikel)hood)
LeNet: 3-8-60-42-10
LeNet: 6 120-8844--1
LeNet: 22-32-240-168-10
0.7
0.7
0.6
0.5
0.5
0.5
0.4
0/ 8= 2.05
05, 0= 5.00
0 2.70
47, 0 7.00
0.3
tanh ReLU
05 3.00
50 100
(b) Test accuracy.
Figure 18: Training curves for CIFAR-10 + LeNet with 3 different numbers of neurons per layer and Φ db.
Results with MNIST + Multilayer perceptron. We have plotted in Figure 19 the results of the training of narrow multilayer perceptron, as in Figure 10. In this extreme setup, the positive activation
49

---

Published in Transactions on Machine Learning Research (04/2025)
functions og perform worse than the odd ones øg. In particular, many trainings failed with narrow and deep network (I. = 30).
1 = 3 Ri =3
10 By =
L 30 ni
2.0
2.0
1.5
1.5
1.5
1.0
1.0
0.0
0.0
0.0
60
100
100
X0
0
00
100
200
100
160
900
L
3: nu
10
L
10
10
30 ny
10
2.5
2.5
46
<8.
2.30
48. 7.00
68.
3.00
48. 10.00
1.5
4.00
1.5
1.5
1.0
1.0
1.0
0.5-
0.0
3.0
50
100
150
200
0
50
100
150
200
0
30
100
150
200
Figure 19: Training loss for multilayer perceptron, narrow (n4 E (0}) and of various depths (L E
{3, 10, 30}). Training curves averaged over 5 experiments.
50


Reviews:
[{"review_id": "Ntz9oCXuDk", "reviewer": "Reviewer_VZ8x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\n\n- The authors identified an inaccuracy in the original edge of chaos (EOC) work.\n- The authors developed the insight that there exists trainable networks that more closely resemble the Guassian hypothesis, via a creative proof by construction.\n- The derivation is presented in a clean and understandable way.\n\n\n*Weakness:*\n\nThe motivation and contributions of the paper could be further clarified. Some observation/suggestions:\n- As the authors pointed out, the experimental results of (Fortuin et al., 2022) shows that Gaussian pre-activations are not necessarily beneficial for training. The empirical experiments in section 5.4 appear to support this point of view.\n- If the main motivation is not empirical but theoretical, it would be helpful if the authors state how their weight distribution and activations can be a useful tool for future theoretical analysis. For example, some potential benefits of replacing the infinite-width assumption with these distributions?  \n- As this reviewer understood it, the Gaussian hypothesis is a convenient assumption that simplifies the derivation of EOS. In case this is known to the author, it would help to point out: to what extent does one expect the heavy tail to affect the main conclusion of the analysis, i.e. the convergence towards fix point and the corresponding rates?\n- Is the pitfall in the original EOS analysis (section 3.3) related to the Gaussian hypothesis?\n\nA more comprehensive engagement with recent literature could also help to refine the problem formulation and assess the contributions. For example, Li et al., 2022 and follow up works specifically analysed the propagation and distribution of correlation/covariances, of networks of finite width and proportionally large depth.\n\nA potential point of discussion: empirically, for the $\\phi_\\theta$ activation, a smaller $\\theta$ seems to improve the KS-statistics; would modifying a tanh/Relu activation be expected to have a similar effect? More generally, to what extent is the benefit attributed to the shape of the activation, v.s. the carefully selected pairing of weights and activations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors refined the edge of chaos analysis to address the possibility of multiple stable fix points.\n- The authors constructed pairs of weight distribution and activation functions that results in Gaussian pre-activations (on the assumption that the weights are independent), and in the process concludes that the standard choice of weights and activation functions cannot lead to Gaussian pre-activations at finite width.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To secure my recommendation, please refine the list of contributions:\n- Clarifying the value of each contribution and its relevance to the community.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TJ29try5zB", "reviewer": "Reviewer_Z87X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The edge of chaos regime and similar propagation of information properties are highly relevant to modern neural networks. Understanding the validity and applicability of these phenomena should be of interest to the community.\n- I found the Mathematical premise of the stated problem appealing.\n\nWeaknesses:\n- First and foremost, it is not clear to me what is the actual benefit of enforcing the pre-activations to be Gaussian. Yes, it makes the analysis more straightforward in some cases, but is there any actual practical value? The experiments in the paper at least do not suggest it.\n- The proof of the main result is ultimately based on writing the Mellin transform of the standard Gaussian as a product of two other Mellin transforms. After that step is done, one chooses the monotone rearrangement function to be the activation. The role of the activation is to take a Gaussian pre-activation to a some distribution dictated by the Mellin transform. As the authors note, there are infinitely many ways to decompose the Gaussian distribution as a product of two independent random variables. There is also no specific reason to choose the monotone rearrangement to transport between the two distributions. In that sense, it is not clear to me what is special (if any) about the specific construction the authors propose. Does it have any desirable properties other constructions do not enjoy?\n- Another weak point is that pre-activations remain Gaussian only in an idealized setting, where we can enforce independence between their different coordinates. I don't think this assumption will ever hold in practice, at least not for deeper layers. It would have been nice if there could be some way to decorrelate the pre-activiations within each layer. Is there some way to choose the weights, perhaps with less independence to have this effect?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper's main contribution is constructing a deep neural network with specially crafted random weights and activation functions, satisfying the following property: For a random input following a Gaussian distribution, the law of the pre-activation of each layer is Gaussian as well.\nSuch a construction is motivated by analysis of phenomena like the so-called 'edge of chaos' which sometimes implicitly assumes that pre-activiations are Gaussian. Since this assumption is almost always false with common architectures, it makes sense to enforce it and investigate the resulting network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'll start with the bottom line: I believe that the analysis performed in the paper and some of the experiments could be of interest to some individuals in TMLR's audience. I also agree with the proofs and I think the experiments support the claims. Therefore I am inclined to recommend acceptance of the paper.\n\nI do have some suggestions for changes:\n- First, I think some discussions regarding the 'weaknesses' I mentioned in the previous section are in order. I know the authors touched upon some of these topics already, but I feel there is room to go deeper into the details. Another question that would make sense is whether for a given set of inputs, their pre-activations can remain jointly Gaussian?\n\n- It is a stylistic preference, but I found Section 4 to be a bit awkward to read. I would say that it could be better to first state a concrete theorem with the Weibull weights and provide its proof (which will be rather short). The discussion about the different constraints and the generalized setting can come later.\n\nMinor comments:\n\n- Proposition 3 is usually known as Cramer's theorem in the literature.\n- What does slowly varying mean in the definition of the generalized Weibull distribution?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L888ML27F8", "reviewer": "Reviewer_Rovr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nI believe the results are definitely not known previously, and the edge of chaos framework broadly has several issues that deserve a more careful analysis. In this regard, I believe the authors has done helpful work towards this direction. \n\nI should also emphasize the construction the authors have found is definitely non-trivial, and the method developed here could definitely be useful in another context where converting random variables to Gaussians are interesting. \n\nWeaknesses \n\nI understand that in TMLR, the subjective evaluation of novelty or impact is not meant to prioritized. So my criticism here is not meant to be a serious argument against this paper, but rather to start a discussion with the authors next. \n\nWhile I have many criticisms of the Gaussian process limit and the EOC arguments, I don't think I understand the motivation of the authors here in this work. For example, is Gaussian pre-activation important for downstream applications? Personally I think we should do the opposite: instead of constructing neural networks that are closer to Gaussian, we should characterize the distributions that can be a good model of finite size neural networks. Because at the end of the day, neural networks as are they currently are do work very well in practice, so shouldn't theory be adapting to practice here? \n\nI don't think the results of this result will change my overall positive view of this submission, so feel free to engage in a casual manner.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyzed the signal propagation of MLPs at initialization, where the Gaussian property of the pre-activation is examined carefully. In particular, the authors find the Gaussian property generally fail at finite width, propose a method to achieve Gaussian pre-activations. There are several other smaller yet interesting contributions such as finding the variance having multiple fixed points, as well as some empirical results. \n\nI believe the main result here of constructing the desired Gaussian result is worth mentioning in the summary. Essentially the author uses the inverse CDF construction of a random variable, so that if $X,Y$ are random variables with CDFs $F_X, F_Y$, then we have that \n\n$ Y \\overset{d}{=} F_Y^{-1} ( U ) \\overset{d}{=} F_Y^{-1} ( F_X(X) )  $\n\nwhere $U$ is a uniform random variable on $[0,1]$. So if we want $Y$ to be a Gaussian, then we simply need to choose the activation function $\\phi = F_Y^{-1} \\circ F_X$ for any random variable $X$. \n\nOn top of which, the authors show that the weights could be drawn from a family of the Weibull distribution, and characterizes the constraints that the weights need to satisfy in order to achieve this result.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Only minor comments \n1. A bit of an introduction to general Weibull distributions could be helpful for the unfamiliar readers like myself. \n2. I'm not sure if the \"Gaussian hypothesis\" is a good terminology here, as I don't think the previous works really intended it to be a hypothesis, but rather a property of the infinite width limit. I would consider rephrasing it, maybe a \"property\" or a \"condition\". \n3. I have a subjective opinion that the conclusion drawn from the figure should be stated within the Figure caption to make it easier to read. For example, I wasn't sure what I should be focused on for Figure 1, which is a very large plot with a lot of information. \n4. Is there a reason that some of the text are in green?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the claims and evidence are appropriately supported and that some of the TMLR audience will find the paper interesting. The paper appears to be well-written. Authors have already responded to multiple reviewer comments in a revision, so I can recommend acceptance as is.
Decision: Accept (poster)