Year: 2018

Abstract:
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Schoenholz et al. (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the ‘edge of chaos’ can lead to good performance. We complete this analysis by providing quantitative results showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos. By further extending this analysis, we identify a class of activation functions that improve the information propagation over ReLU-like functions. This class includes the Swish activation, φswish(x) = x · sigmoid(x), used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and Ramachandran et al. (2017). This provides a theoretical grounding for the excellent empirical performance of φswish observed in these contributions. We complement those previous results by illustrating the benefit of using a random initialization on the edge of chaos in this context.

heading: 1 INTRODUCTION
text: Deep neural networks have become extremely popular as they achieve state-of-the-art performance on a variety of important applications including language processing and computer vision; see, e.g., LeCun et al. (1998). The success of these models has motivated the use of increasingly deep networks and stimulated a large body of work to understand their theoretical properties. It is impossible to provide here a comprehensive summary of the large number of contributions within this field. To cite a few results relevant to our contributions, Montufar et al. (2014) have shown that neural networks have exponential expressive power with respect to the depth while Poole et al. (2016) obtained similar results using a topological measure of expressiveness.
We follow here the approach of Poole et al. (2016) and Schoenholz et al. (2017) by investigating the behaviour of random networks in the infinite-width and finite-variance i.i.d. weights context where they can be approximated by a Gaussian process as established by Matthews et al. (2018) and Lee et al. (2018).
In this paper, our contribution is two-fold. Firstly, we provide an analysis complementing the results of Poole et al. (2016) and Schoenholz et al. (2017) and show that initializing a network with a specific choice of hyperparameters known as the ‘edge of chaos’ is linked to a deeper propagation of the information through the network. In particular, we establish that for a class of ReLU-like activation functions, the exponential depth scale introduced in Schoenholz et al. (2017) is replaced by a polynomial depth scale. This implies that the information can propagate deeper when the network is initialized on the edge of chaos. Secondly, we outline the limitations of ReLU-like activation functions by showing that, even on the edge of chaos, the limiting Gaussian Process admits a degenerate kernel as the number of layers goes to infinity. Our main result (4) gives sufficient conditions for activation functions to allow a good ‘information flow’ through the network (Proposition 4) (in addition to being non-polynomial and not suffering from the exploding/vanishing gradient problem). These conditions are satisfied by the Swish activation φswish(x) = x ·sigmoid(x) used in Hendrycks & Gimpel (2016),
Elfwing et al. (2017) and Ramachandran et al. (2017). In recent work, Ramachandran et al. (2017) used automated search techniques to identify new activation functions and found experimentally that functions of the form φ(x) = x · sigmoid(βx) appear to perform indeed better than many alternative functions, including ReLU. Our paper provides a theoretical grounding for these results. We also complement previous empirical results by illustrating the benefits of an initialization on the edge of chaos in this context. All proofs are given in the Supplementary Material.

heading: 2 ON GAUSSIAN PROCESS APPROXIMATIONS OF NEURAL NETWORKS AND THEIR STABILITY

heading: 2.1 SETUP AND NOTATIONS
text: We use similar notations to those of Poole et al. (2016) and Lee et al. (2018). Consider a fully connected random neural network of depth L, widths (Nl)1≤l≤L, weights W lij iid∼ N (0, σ 2 w Nl−1 ) and bias Bli iid∼ N (0, σ2b ), where N (µ, σ2) denotes the normal distribution of mean µ and variance σ2. For some input a ∈ Rd, the propagation of this input through the network is given for an activation function φ : R→ R by
y1i (a) = d∑ j=1 W 1ijaj +B 1 i , y l i(a) = Nl−1∑ j=1 W lijφ(y l−1 j (a)) +B l i, for l ≥ 2. (1)
Throughout the paper we assume that for all l the processes yli(.) are independent (across i) centred Gaussian processes with covariance kernels κl and write accordingly yli ind∼ GP(0, κl). This is an idealized version of the true processes corresponding to choosing Nl−1 = +∞ (which implies, using Central Limit Theorem, that yli(a) is a Gaussian variable for any input a). The approximation of yli(.) by a Gaussian process was first proposed by Neal (1995) in the single layer case and has been recently extended to the multiple layer case by Lee et al. (2018) and Matthews et al. (2018). We recall here the expressions of the limiting Gaussian process kernels. For any input a ∈ Rd, E[yli(a)] = 0 so that for any inputs a, b ∈ Rd
κl(a, b) = E[yli(a)yli(b)] = σ2b + σ2wEyl−1i ∼GP (0,κl−1)[φ(y l−1 i (a))φ(y l−1 i (b))]
= σ2b + σ 2 wFφ(κ l−1(a, a), κl−1(a, b), κl−1(b, b)),
where Fφ is a function that depends only on φ. This gives a recursion to calculate the kernel κl; see, e.g., Lee et al. (2018) for more details. We can also express the kernel κl in terms of the correlation clab in the l th layer used in the rest of this paper
qlab := κ l(a, b) = E[yli(a)yli(b)] = σ2b + σ2wE[φ( √ ql−1a Z1)φ( √ ql−1b (c l−1 ab Z1 + √ 1− (cl−1ab )2Z2))]
where ql−1a := q l−1 aa , resp. c l−1 ab := q l−1 ab / √ ql−1a q l−1 b , is the variance, resp. correlation, in the (l − 1)th layer and Z1, Z2 are independent standard Gaussian random variables. when it propagates through the network. qla is updated through the layers by the recursive formula q l a = F (q l−1 a ), where F is the ‘variance function’ given by
F (x) = σ2b + σ 2 wE[φ(
√ xZ)2], where Z ∼ N (0, 1). (2)
Throughout the paper, Z,Z1, Z2 will always denote independent standard Gaussian variables.

heading: 2.2 LIMITING BEHAVIOUR OF THE VARIANCE AND COVARIANCE OPERATORS
text: We analyze here the limiting behaviour of qLa and c L a,b as the network depth L goes to infinity under the assumption that φ has a second derivative at least in the distribution sense1. From now onwards, we will also assume without loss of generality that c1ab ≥ 0 (similar results can be obtained straightforwardly when c1ab ≤ 0). We first need to define the Domains of Convergence associated with an activation function φ.
1ReLU admits a Dirac mass in 0 as second derivative and so is covered by our developments.
Definition 1. Let φ be an activation function, (σb, σw) ∈ (R+)2. (i) (σb, σw) is in Dφ,var (domain of convergence for the variance) if there exists K > 0, q ≥ 0 such that for any input a with q1a ≤ K, liml→∞ qla = q. We denote by Kφ,var(σb, σw) the maximal K satisfying this condition. (ii) (σb, σw) is in Dφ,corr (domain of convergence for the correlation) if there exists K > 0 such that for any two inputs a, b with q1a, q 1 b ≤ K, liml→∞ clab = 1. We denote by Kφ,corr(σb, σw) the maximal K satisfying this condition.
Remark : Typically, q in Definition 1 is a fixed point of the variance function defined in equation 2. Therefore, it is easy to see that for any (σb, σw) such that F is increasing and admits at least one fixed point, we haveKφ,corr(σb, σw) ≥ q where q is the minimal fixed point; i.e. q := min{x : F (x) = x}. Thus, if we re-scale the input data to have q1a ≤ q, the variance qla converges to q. We can also re-scale the variance σw of the first layer (only) to assume that q1a ≤ q for all inputs a. The next result gives sufficient conditions on (σb, σw) to be in the domains of convergence of φ. Proposition 1. Let Mφ := supx≥0E[|φ′2(xZ) + φ′′(xZ)φ(xZ)|]. Assume Mφ < ∞, then for σ2w < 1 Mφ and any σb, we have (σb, σw) ∈ Dφ,var and Kφ,var(σb, σw) =∞ Let Cφ,δ := supx,y≥0,|x−y|≤δ,c∈[0,1]E[|φ′(xZ1)φ′(y(cZ1 + √
1− c2Z2)|]. Assume Cφ,δ < ∞ for some δ > 0, then for σ2w < min( 1 Mφ
, 1Cφ ) and any σb, we have (σb, σw) ∈ Dφ,var ∩Dφ,corr and Kφ,var(σb, σw) = Kφ,corr(σb, σw) =∞.
The proof of Proposition 1 is straightforward. We prove that supF ′(x) = σ2wMφ and then apply the Banach fixed point theorem; similar ideas are used for Cφ,δ .
Example : For ReLU activation function, we have MReLU = 2 and CReLU,δ ≤ 1 for any δ > 0.
In the domain of convergence Dφ,var ∩Dφ,corr, for all a, b ∈ Rd, y∞i (a) = y∞i (b) almost surely and the outputs of the network are constant functions. Figure 1 illustrates this behaviour for d = 2 for ReLU and Tanh using a network of depth L = 10 with Nl = 100 neurons per layer. The draws of outputs of these networks are indeed almost constant.
To refine this convergence analysis, Schoenholz et al. (2017) established the existence of q and c such that |qla−q| ∼ e−l/ q and |clab−1| ∼ e−l/ c when fixed points exist. The quantities q and c are called ‘depth scales’ since they represent the depth to which the variance and correlation can propagate without being exponentially close to their limits. More precisely, if we write χ1 = σ2wE[φ′( √ qZ)2] and α = χ1 + σ2wE[φ′′( √ qZ)φ( √ qZ)] then the depth scales are given by r = − log(α)−1 and
c = − log(χ1)−1. The equation χ1 = 1 corresponds to an infinite depth scale of the correlation. It is called the edge of chaos as it separates two phases: an ordered phase where the correlation
converges to 1 if χ1 < 1 and a chaotic phase where χ1 > 1 and the correlations do not converge to 1. In this chaotic regime, it has been observed in Schoenholz et al. (2017) that the correlations converge to some random value c < 1 when φ(x) = Tanh(x) and that c is independent of the correlation between the inputs. This means that very close inputs (in terms of correlation) lead to very different outputs. Therefore, in the chaotic phase, the output function of the neural network is non-continuous everywhere.
Definition 2. For (σb, σw) ∈ Dφ,var, let q be the limiting variance2. The Edge of Chaos, hereafter EOC, is the set of values of (σb, σw) satisfying χ1 = σ2wE[φ′( √ qZ)2] = 1.
To further study the EOC regime, the next lemma introduces a function f called the ‘correlation function’ simplifying the analysis of the correlations. It states that the correlations have the same asymptotic behaviour as the time-homogeneous dynamical system cl+1ab = f(c l ab).
Lemma 1. Let (σb, σw) ∈ Dφ,var ∩Dφ,corr such that q > 0, a, b ∈ Rd and φ an activation function such that supx∈S E[φ(xZ)2] < ∞ for all compact sets S. Define fl by cl+1ab = fl(clab) and f by f(x) = σ2b+σ 2 wE[φ( √ qZ1)φ( √ q(xZ1+ √ 1−x2Z2))
q . Then liml→∞ supx∈[0,1] |fl(x)− f(x)| = 0.
The condition on φ in Lemma 1 is violated only by activation functions with exponential growth (which are not used in practice), so from now onwards, we use this approximation in our analysis. Note that being on the EOC is equivalent to (σb, σw) satisfying f ′(1) = 1. In the next section, we analyze this phase transition carefully for a large class of activation functions.
3 EDGE OF CHAOS
To illustrate the effect of the initialization on the EOC, we plot in Figure 2 the output of a ReLU neural network with 20 layers and 100 neurons per layer with parameters (σ2b , σ 2 w) = (0, 2)
(as we will see later EOC = {(0, √
2)} for ReLU). Unlike the output in Figure 1, this output displays much more variability. However, we will prove here that the correlations still converges to 1 even in the EOC regime, albeit at a slower rate.

heading: 3.1 RELU-LIKE ACTIVATION FUNCTIONS
text: We consider activation functions φ of the form: φ(x) = λx if x > 0 and φ(x) = βx if x ≤ 0. ReLU corresponds to λ = 1 and β = 0. For this class of activation functions, we see (Proposition 2) that the variance is unchanged (qla = q 1 a) on the EOC, so that q does not formally exist in the sense that the limit of qla depends on a. However, this does not impact the analysis of the correlations.
Proposition 2. Let φ be a ReLU-like function with λ and β defined above. Then for any σw <
√ 2
λ2+β2 and σb ≥ 0, we have (σb, σw) ∈ Dφ,var with Kφ,var(σb, σw) = ∞. Moreover EOC = {(0, 1√
E[φ′(Z)2] )} and, on the EOC, F (x) = x
for any x ≥ 0.
This class of activation functions has the interesting property of preserving the variance across layers when the network is initialized on the EOC. However, we show in Proposition 3 below that, even in the EOC regime, the correlations converge to 1 but at a slower rate. We only present the result for ReLU but the generalization to the whole class is straightforward.
Example : ReLU: The EOC is reduced to the singleton (σ2b , σ2w) = (0, 2), which means we should initialize a ReLU network with the parameters (σ2b , σ 2 w) = (0, 2). This result coincides with the
2The limiting variance is a function of (σb, σw) but we do not emphasize it notationally.
recommendation of He et al. (2015) whose objective was to make the variance constant as the input propagates but did not analyze the propagation of the correlations. Klambauer et al. (2017) also performed a similar analysis by using the "Scaled Exponential Linear Unit" activation (SELU) that makes it possible to center the mean and normalize the variance of the post-activation φ(y). The propagation of the correlation was not discussed therein either. In the next result, we present the correlation function corresponding to ReLU networks. This was first obtained in Cho & Saul (2009). We present an alternative derivation of this result and further show that the correlations converge to 1 at a polynomial rate of 1/l2 instead of an exponential rate. Proposition 3 (ReLU kernel). Consider a ReLU network with parameters (σ2b , σ2w) = (0, 2) on the EOC.We have (i) for x ∈ [0, 1], f(x) = 1πx arcsin(x) + 1 π √ 1− x2 + 12x, ii) for any (a, b), liml→∞ clab = 1 and 1− clab ∼ 9π 2 2l2 as l→∞.
Figure 3 displays the correlation function f with two different sets of parameters (σb, σw). The red graph corresponds to the EOC (σ2b , σ 2 w) = (0, 2), and the blue one corresponds to an ordered phase (σb, σw) = (1, 1). In unreported experiments, we observed that numerical convergence towards 1 for l ≥ 50 on the EOC. As the variance qla is preserved by the network (qla = q1a = 2‖a‖2/d) and the correlations clab converge to 1 as l increases, the output function is of the form C · ‖a‖ for a constant C (notice that in Figure 2, we start observing this effect for depth 20).

heading: 3.2 A BETTER CLASS OF ACTIVATION FUNCTIONS
text: We now introduce a set of sufficient conditions for activation functions which ensures that it is then possible to tune (σb, σw) to slow the convergence of the correlations to 1. This is achieved by making the correlation function f sufficiently close to the identity function. Proposition 4 (Main Result). Let φ be an activation function. Assume that (i) φ(0) = 0, and φ has right and left derivatives in zero and φ′(0+) 6= 0 or φ′(0−) 6= 0, and there exists k > 0 such that ∣∣φ(x) x
∣∣ ≤ k. (ii) There exists A > 0 such that for any σb ∈ [0, A], there exists σw > 0 such that (σb, σw) ∈ EOC. (iii) For any σb ∈ [0, A], the function F with parameters (σb, σw) ∈ EOC is non-decreasing and limσb→0 q = 0 where q is the minimal fixed point of F , q := inf{x : F (x) = x}. (iv) For any σb ∈ [0, A], the correlation function f with parameters (σb, σw) ∈ EOC introduced in Lemma 1 is convex.
Then, for any σb ∈ [0, A], we have Kφ,var(σb, σw) ≥ q, and lim σb→0
(σb,σw)∈EOC
sup x∈[0,1]
|f(x)− x| = 0.
Note that ReLU does not satisfy the condition (ii) since the EOC in this case is the singleton (σ2b , σ 2 w) = (0, 2). The result of Proposition 4 states that we can make f(x) close to x by considering σb → 0. However, this is under condition (iii) which states that limσb→0 q = 0. Therefore, practically, we cannot take σb too small. One might wonder whether condition (iii) is necessary for this result to hold. The next lemma shows that removing this condition results in a useless class of activation functions.
(a) The correlation function f of a Swish network for different values of σb
Lemma 2. Under the conditions of Proposition 4, the only change being limσb→0 q > 0, the result of Proposition 4 holds if only if the activation function is linear.
The next proposition gives sufficient conditions for bounded activation functions to satisfy all the conditions of Proposition 4.
Proposition 5. Let φ be a bounded function such that φ(0) = 0, φ′(0) > 0, φ′(x) ≥ 0, φ(−x) = −φ(x), xφ(x) > 0 and xφ′′(x) < 0 for x 6= 0, and φ satisfies (ii) in Proposition 4. Then, φ satisfies all the conditions of Proposition 4.
The conditions in Proposition 5 are easy to verify and are, for example, satisfied by Tanh and Arctan. We can also replace the assumption "φ satisfies (ii) in Proposition 4" by a sufficient condition (see Proposition 7 in the Supplementary Material). Tanh-like activation functions provide better information flow in deep networks compared to ReLU-like functions. However, these functions suffer from the vanishing gradient problem during back-propagation; see, e.g., Pascanu et al. (2013) and Kolen & Kremer (2001). Thus, an activation function that satisfies the conditions of Proposition 4 (in order to have a good ’information flow’) and does not suffer from the vanishing gradient issue is expected to perform better than ReLU. Swish is a good candidate.
Proposition 6. The Swish activation function φswish(x) = x · sigmoid(x) = x1+e−x satisfies all the conditions of Proposition 4.
It is clear that Swish does not suffer from the vanishing gradient problem as it has a gradient close to 1 for large inputs like ReLU. Figure 4 (a) displays f for Swish for different values of σb. We see that f is indeed approaching the identity function when σb is small, preventing the correlations from converging to 1. Figure 4(b) displays a draw of the output of a neural network of depth 30 and width 100 with Swish activation, and σb = 0.2. The outputs displays much more variability than the ones of the ReLU network with the same architecture. We present in Table 1 some values of (σb, σw) on the EOC as well as the corresponding limiting variance for Swish. As condition (iii) of Proposition 4 is satisfied, the limiting variance q decreases with σb.
Other activation functions that have been shown to outperform empirically ReLU such as ELU (Clevert et al. (2016)), SELU (Klambauer et al. (2017)) and Softplus also satisfy the conditions of Proposition 4 (see Supplementary Material for ELU). The comparison of activation functions satisfying the conditions of Proposition 4 remains an open question.

heading: 4 EXPERIMENTAL RESULTS
text: We demonstrate empirically our results on the MNIST dataset. In all the figures below, we compare the learning speed (test accuracy with respect to the number of epochs/iterations) for different activation functions and initialization parameters. We use the Adam optimizer with learning rate lr = 0.001. The Python code to reproduce all the experiments will be made available on-line.
Initialization on the Edge of Chaos We initialize randomly the deep network by sampling W lij iid∼ N (0, σ2w/Nl−1) andBli iid∼ N (0, σ2b ). In Figure 5, we compare the learning speed of a Swish network for different choices of random initialization. Any initialization other than on the edge of chaos results in the optimization algorithm being stuck eventually at a very poor test accuracy of ∼ 0.1 as the depth L increases (equivalent to selecting the output uniformly at random). To understand what is happening in this case, let us recall how the optimization algorithm works. Let {(Xi, Yi), 1 ≤ i ≤ N} be the MNIST dataset. The loss we optimize is given by L(w, b) = ∑N i=1 `(y
L(Xi), Yi)/N where yL(x) is the output of the network, and ` is the categorical cross-entropy loss. In the ordered phase, we know that the output converges exponentially to a fixed value (same value for all Xi), thus a small change in w and b will not change significantly the value of the loss function, therefore the gradient is approximately zero and the gradient descent algorithm will be stuck around the initial value.
ReLU versus Tanh We proved in Section 3.2 that the Tanh activation guarantees better information propagation through the network when initialized on the EOC. However, Tanh suffers
from the vanishing gradient problem. Consequently, we expect Tanh to perform better than ReLU for shallow networks as opposed to deep networks, where the problem of the vanishing gradient is not encountered. Numerical results confirm this fact. Figure 6 shows curves of validation accuracy with confidence interval 90% (30 simulations). For depth 5, the learning algorithm converges faster for Tanh compared to ReLu. However, for deeper networks (L ≥ 40), Tanh is stuck at a very low test accuracy, this is due to the fact that a lot of parameters remain essentially unchanged because the gradient is very small.
ReLU versus Swish As established in Section 3.2, Swish, like Tanh, propagates the information better than ReLU and, contrary to Tanh, it does not suffer from the vanishing gradient problem. Hence our results suggest that Swish should perform better than ReLU, especially for deep architectures. Numerical results confirm this fact. Figure 7 shows curves of validation accuracy with confidence interval 90% (30 simulations). Swish performs clearly better than ReLU especially for depth 40. A comparative study of final accuracy is shown in Table 2. We observe a clear advantage for Swish, especially for large depths. Additional simulations results on diverse datasets demonstrating better performance of Swish over many other activation functions can be found in Ramachandran et al. (2017) (Notice that these authors have already implemented Swish in Tensorflow).

heading: 5 CONCLUSION AND DISCUSSION
text: We have complemented here the analysis of Schoenholz et al. (2017) which shows that initializing networks on the EOC provides a better propagation of information across layers. In the ReLU case, such an initialization corresponds to the popular approach proposed in He et al. (2015). However, even on the EOC, the correlations still converge to 1 at a polynomial rate for ReLU networks. We have obtained a set of sufficient conditions for activation functions which further improve information propagation when the parameters (σb, σw) are on the EOC. The Tanh activation satisfied those conditions but, more interestingly, other functions which do not suffer from the vanishing/exploding gradient problems also verify them. This includes the Swish function used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and promoted in Ramachandran et al. (2017) but also ELU Clevert et al. (2016).
Our results have also interesting implications for Bayesian neural networks which have received renewed attention lately; see, e.g., Hernandez-Lobato & Adams (2015) and Lee et al. (2018). They show that if one assigns i.i.d. Gaussian prior distributions to the weights and biases, the resulting prior distribution will be concentrated on close to constant functions even on the EOC for ReLU-like activation functions. To obtain much richer priors, our results indicate that we need to select not only parameters (σb, σw) on the EOC but also an activation function satisfying Proposition 4.

heading: A PROOFS
text: We provide in the supplementary material the proofs of the propositions presented in the main document, and we give additive theoretical and experimental results. For the sake of clarity we recall the propositions before giving their proofs.

heading: A.1 CONVERGENCE TO THE FIXED POINT: PROPOSITION 1
text: Proposition 1. Let Mφ := supx≥0E[|φ′2(xZ) + φ′′(xZ)φ(xZ)|]. Suppose Mφ < ∞, then for σ2w < 1 Mφ and any σb, we have (σb, σw) ∈ Dφ,var and Kφ,var(σb, σw) =∞ Moreover, let Cφ,δ := supx,y≥0,|x−y|≤δ,c∈[0,1]E[|φ′(xZ1)φ′(y(cZ1 + √
1− c2Z2)|]. Suppose Cφ,δ < ∞ for some positive δ, then for σ2w < min( 1Mφ , 1 Cφ
) and any σb, we have (σb, σw) ∈ Dφ,var ∩Dφ,corr and Kφ,var(σb, σw) = Kφ,corr(σb, σw) =∞.
Proof. To abbreviate the notation, we use ql := qla for some fixed input a.
Convergence of the variances: We first consider the asymptotic behaviour of ql = qla. Recall that ql = F (ql−1) where,
F (x) = σ2b + σ 2 wE[φ(
√ xZ)2].
The first derivative of this function is given by:
F ′(x) = σ2wE[ Z√ x φ′( √ xZ)φ( √ xZ)] = σ2wE[φ′( √ xZ)2 + φ′′( √ xZ)φ( √ xZ)] (3)
where we used Gaussian integration by parts E[ZG(Z)] = E[G′(Z)], an identify satisfied by any function G such that E[|G′(Z)|] <∞. Using the condition on φ, we see that for σ2w < 1 Mφ
, the function F is a contraction mapping, and the Banach fixed-point theorem guarantees the existence of a unique fixed point q of F , with liml→+∞ q
l = q. Note that this fixed point depends only on F , therefore, this is true for any input a, and Kφ,var(σb, σw) =∞.
Convergence of the covariances: Since Mφ < ∞, then for all a, b ∈ Rd there exists l0 such that, for all l > l0, | √ qla − √ qlb| < δ. Let l > l0, using Gaussian integration by parts, we have
dcl+1ab dclab
= σ2wE[|φ′( √ qlaZ1)φ ′( √ qlb(c l abZ1 + √ 1− (clab)2Z2)|].
We cannot use the Banach fixed point theorem directly because the integrated function here depends on l through ql. For ease of notation, we write cl := clab, we have
|cl+1 − cl| = | ∫ cl cl−1 dcl+1 dcl (x)dx| ≤ σ2wCφ|cl − cl−1|.
Therefore, for σ2w < min( 1 Mφ , 1Cφ ), c l is a Cauchy sequence and it converges to a limit c ∈ [0, 1] . At the limit
c = f(c) = σ2b + σ 2 wE[φ(
√ qz1)φ( √ q(cz1 + √ 1− c2z2)))]
q ,
The derivative of this function is given by
f ′(x) = σ2wE[φ′( √ qZ1)φ ′( √ q(xZ1 + √ 1− xZ2)]
By assumption on φ and the choice of σw, we have supx|f ′(x)| < 1, so that f is a contraction, and has a unique fixed point. Since f(1) = 1, c = 1. The above result is true for any a, b, therefore, Kφ,var(σb, σw) = Kφ,corr(σb, σw) =∞.
As an illustration we plot in Figure 12 the variance for three different inputs with (σb, σw) = (1, 1), as a function of the layer l. In this example, the convergence for Tanh is faster than that of ReLU.
Lemma 1. Let (σb, σw) ∈ Dφ,var ∩Dφ,corr such that q > 0, a, b ∈ Rd and φ an activation function such that supx∈K E[φ(xZ)2] < ∞ for all compact sets K. Define fl by cl+1a,b = fl(cla,b) and f by f(x) = σ2b+σ 2 wE[φ( √ qZ1)φ( √ q(xZ1+ √ 1−x2Z2))
q . Then liml→∞ supx∈[0,1] |fl(x)− f(x)| = 0.
Proof. For x ∈ [0, 1], we have
fl(x)− f(x) = ( 1√ qlaq l b − 1 q )(σ2b + σ 2 wE[φ(
√ qlaZ1)φ( √ qlbu2(x))])
+ σ2w q
(E[φ( √ qlaZ1)φ( √ qlbu2(x))]− E[φ( √ qZ1)φ( √ qu2(x))]),
where u2(x) := xZ1 + √
1− x2Z2. The first term goes to zero uniformly in x using the condition on φ and Cauchy-Schwartz inequality. As for the second term, it can be written as
E[(φ( √ qlaZ1)− φ( √ qZ1))φ( √ qlbu2(x))] + E[φ( √ qZ1)(φ( √ qlbu2(x))− φ( √ qu2(x)))]
again, using Cauchy-Schwartz and the condition on φ, both terms can be controlled uniformly in x by an integrable upper bound. We conclude using the Dominated convergence.

heading: A.2 RESULTS FOR RELU-LIKE ACTIVATION FUNCTIONS: PROOF OF PROPOSITIONS 2 AND 3
text: Proposition 2. Let φ be a ReLU-like function with λ and β defined above. Then for any σw <√ 2
λ2+β2 and σb ≥ 0, we have (σb, σw) ∈ Dφ,var with Kφ,var(σb, σw) = ∞. Moreover EOC = {(0, 1√
E[φ′(Z)2] )} and, on the EOC, F (x) = x for any x ≥ 0.
Proof. We write ql = qla throughout the proof. Note first that the variance satisfies the recursion:
ql+1 = σ2b + σ 2 wE[φ(Z)2]ql = σ2b + σ2w
λ2 + β2
2 ql. (4)
For all σw < √ 2 λ2+β2 , q = σ 2 b ( 1− σ2w(λ2 + β2)/2 )−1 is a fixed point. This is true for any input, therefore Kφ,var(σb, σw) =∞ and (i) is proved.
Now, the EOC equation is given by χ1 = σ2wE[φ′(Z)2] = σ2w λ2+β2 2 . Therefore, σ 2 w = 2 λ2+β2 . Replacing σ2w by its critical value in equation 4 yields
ql+1 = σ2b + q l.
Thus q = σ2b + q if and only if σb = 0, otherwise q l diverges to infinity. So the frontier is reduced to a single point (σ2b , σ 2 w) = (0,E[φ′(Z)2]−1), and the variance does not depend on l.
Proposition 3 (ReLU kernel). Consider a ReLU network with parameters (σ2b , σ2w) = (0, 2) on the EOC.We have (i) for x ∈ [0, 1], f(x) = 1πx arcsin(x) + 1 π √ 1− x2 + 12x, ii) for any (a, b), liml→∞ clab = 1 and 1− clab ∼ 9π 2 2l2 as l→∞.
Proof. In this case the correlation function f is given by f(x) = 2E[(Z1)+(xZ1 + √
1− x2Z2)+] where (x)+ := x1x>0.
• Let x ∈ [0, 1], note that f is differentiable and satisfies, f ′(x) = 2E[1Z1>01xZ1+ √ 1−x2Z2>0],
which is also differentiable. Simple algebra leads to
f”(x) = 1
π √ 1− x2 .
Since arcsin′(x) = 1√ 1−x2 and f ′(0) = 1/2,
f ′(x) = 1
π arcsin(x) +
1 2 .
We conclude using the fact that ∫ arcsin = x arcsin + √ 1− x2 and f(1) = 1.
• We first derive a Taylor expansion of f near 1. Consider the change of variable x = 1− t2 with t close to 0, then
arcsin(1− t2) = π 2 − √
2t− √ 2
12 t3 +O(t5),
so that
arcsin(x) = π 2 − √
2(1− x)1/2 − √ 2
12 (1− x)3/2 +O((1− x)5/2),
and
x arcsin(x) = π 2 − √
2(1− x)1/2 + 11 √ 2
12 (1− x)3/2 +O((1− x)5/2).
Since √ 1− x2 = √ 2(1− x)1/2 − √ 2
4 (1− x)3/2 +O((1− x)5/2),
we obtain that
f(x) = x→1−
x+ 2 √ 2
3π (1− x)3/2 +O((1− x)5/2). (5)
Since (f(x) − x)′ = 1π (arcsin(x) − π 2 ) < 0 and f(1) = 1, for all x ∈ [0, 1[, f(x) > x. If cl < cl+1 then by taking the image by f (which is increasing because f ′ ≥ 0) we have that cl+1 < cl+2, and we know that c1 = f(c0) ≥ c0, so by induction the sequence cl is increasing, and therefore it converges (because it is bounded) to the fixed point of f which is 1. Now let γl := 1− clab for a, b fixed. We note s = 2 √ 2
3π , from the series expansion we have that γl+1 = γl − sγ3/2l +O(γ 5/2 l ) so that
γ −1/2 l+1 = γ −1/2 l (1− sγ 1/2 l +O(γ 3/2 l )) −1/2 = γ −1/2 l (1 +
s 2 γ 1/2 l +O(γ 3/2 l ))
= γ −1/2 l +
s 2 +O(γl).
Thus, as l goes to infinity γ −1/2 l+1 − γ −1/2 l ∼ s 2 and by summing and equivalence of positive divergent series
γ −1/2 l ∼
s 2 l,
which terminates the proof.

heading: A.3 A BETTER CLASS OF ACTIVATION FUNCTIONS: PROOFS OF PROPOSITIONS 4, 5, 6 AND
text: LEMMA 2
Proposition 4 (main result). Let φ be an activation function. Recall the definition of the variance function F (x) := σ2b + σ 2 wE[φ( √ xZ)2]. Assume that (i) φ(0) = 0, and φ has right and left derivatives in zero and at least one of them is different from zero (φ′(0+) 6= 0 or φ′(0−) 6= 0), and there exists K > 0 such that ∣∣φ(x) x
∣∣ ≤ K. (ii) There existsA > 0 such that for any σb ∈ [0, A], there exists σw,EOC > 0 such that (σb, σw,EOC) ∈ EOC. (iii) For any σb ∈ [0, A], the function F with parameters (σb, σw,EOC) is non-decreasing and limσb→0 q = 0 where q is the minimal fixed point of F , q := inf{x : F (x) = x}). (iv) For any σb ∈ [0, A], the correlation function f with parameters (σb, σw,EOC) introduced in Lemma 1 is convex.
Then, for any σb ∈ [0, A], we have Kφ,var(σb, σw) ≥ q, and
lim σb→0
(σb,σw)∈EOC
sup x∈[0,1]
|f(x)− x| = 0.
Proof. We first prove that Kφ,var(σb, σw) ≥ q. We assume that σb > 0, the case σb = 0 is trivial since in this case q = 0 (the output of the network is zero in this case).
Since F is continuous and F (0) = σ2b > 0, we have x ≤ F (x) ≤ q for all x ∈ [0, q]. Using the fact that F is non-decreasing for any input a such that q1a ≤ q, we have ql is increasing and converges to the fixed point q. Therefore Kφ,var(σb, σw) ≥ q. Now we prove that on the edge of chaos, we have
lim σb→0 σ2b q = 0. (6)
The EOC equation is given by σ2wE[φ′( √ qZ)2] = 1. By taking the limit σb → 0 on the edge of chaos, and using the fact that limσb→0 q = 0, we have σ 2 w φ′(0+)2+φ′(0−)2 2 = 1. Moreover q verifies
F (q)
q = 1 = σ2b q + σ2wE[( φ( √ qZ) √ q )2],
so that by taking the limit σb → 0, and using the dominated convergence theorem, we have that 1 = limσb→0 σ2b q + σ 2 w φ′(0+)2+φ′(0−)2 2 = limσb→0 σ2b q + 1 and equation 6 holds.
Finally since f is strictly convex, for all x ∈ [0, 1] f ′(x) ≤ f ′(1) = 1 if (σb, σw) ∈ EOC. Therefore 0 ≤ f(x)− x ≤ f(0) = σ 2 b
q , we conclude using the fact that limσb→0 σ2b q = 0.
Note however that for all σb > 0, if (σb, σw) ∈ EOC, for any inputs a, b, we have liml→∞ cla,b = 1. Indeed, since f is usually strictly convex (otherwise, f would be equal to identity on at least a segment of [0, 1]) and f ′(1) = 1, we have that f is a contraction (because f ′ ≥ 0), therefore the correlation converges to the unique fixed point of f which is 1. Therefore, in most of the cases, the result of Proposition 4 should be seen as a way of slowing down the convergence of the correlation to 1.
Lemma 2. Under the conditions of Proposition 4, the only change being limσb→0 q > 0, the result of Proposition 4 holds only if the activation function is linear.
Proof. Using the convexity of f and the result of Proposition 4, we have in the limit σb → 0, f ′(0) = f ′(1) = 1, which is equivalent to E[φ′(√qZ)2] = E[φ′(√qZ)]2 which implies that var(φ′( √ qZ)) = 0. Therefore there exists a constant a1 such that φ′( √ qZ) = a1 almost surely. This implies φ′ = a1 almost everywhere.
Proposition 5. Let φ be a bounded function such that φ(0) = 0, φ′(0) > 0, φ′(x) ≥ 0, φ(−x) = −φ(x), xφ(x) > 0 and xφ′′(x) < 0 for x 6= 0, and φ satisfies (ii) in Proposition 4. Then, φ satisfies all the conditions of Proposition 4.
Proof. Let φ be an activation function that satisfies the conditions of Proposition 5.
(i) we have φ(0) = 0 and φ′(0) > 0. Since φ is bounded and 0 < φ′(0) <∞, then there exists K such that ∣∣φ(x) x ∣∣ ≤ K. (ii) The condition (ii) is satisfied by assumption.
(iii) Let σb > 0 and σw > 0. Using equation 3 together with φ′ > 0, we have F ′(x) ≥ 0 so that F is non-decreasing. Moreover, we have F (R+) ⊂ [B,C] := [σ2b , σ2b + σ2wM2], therefore any fixed point of F should be in [B,C]. We have F (B) ≥ B and F (C) ≤ C and F is strictly increasing, therefore, there exists a fixed point q of F in [B,C]. Now we prove that limσb→0 q = 0. Using the EOC equation, q satisfies the equation q = σ2b + E[φ(√qZ)2] E[φ′(√qZ)2] . Now let’s prove that the function e(x) := x− E[φ( √ xZ)2]
E[φ′( √ xZ)2] is increasing near 0 which means it is an injection near 0, this is sufficient to conclude (because we take q to be the minimal fixed point). After some calculus, we have
e′(x) = − E[φ′′(
√ xZ)(φ( √ xZ)E[φ′( √ xZ)2]− Z√ x φ′( √ xZ)E[φ( √ xZ)2])]
E[φ′( √ xZ)2]
Using Taylor expansion near 0, after a detailed but unenlightening calculation the numerator is equal to −2φ′(0)2φ′′′(0)2x √ x+O(x2), therefore the function e is increasing near 0.
(iv) Finally, using the notations U1 := √ qZ1 and U2(x) = √ q(xZ1 + √ 1− x2Z2), the first and second derivatives of the correlation function are given by
f ′(x) = σ2wE[φ′(U1)φ′(U2(x))], f ′′(x) = σ2wqE[φ′′(U1)φ′′(U2(x))]
where we used Gaussian integration by parts. Let x > 0, we have that
E[φ′′(U1)φ′′(U2(x))] = E[1{U1≥0}φ ′′(U1)φ ′′(U2(x))] + E[1{U1≤0}φ ′′(U1)φ ′′(U2(x))]
= 2E[1{U1≥0}φ ′′(U1)φ ′′(U2(x))]
where we used the fact that (Z1, Z2) = (−Z1,−Z2) (in distribution) and φ′′(−y) = −φ′′(y) for any y. Using xφ′′(x) ≤ 0, we have 1{u1≥0}φ′′(u1) ≤ 0. We also have for all y > 0, E[φ′′(U2(x))|U1 = y] < 0, this is a consequence of the fact that φ′′ is an odd function and that for x > 0 and y > 0, the mapping z2 → xy + √ 1− x2z2 moves the center of the Gaussian distribution to a strictly positive number, we conclude that f ′′(x) > 0 almost everywhere and assumption (iii) of Proposition 4 is verified.
Proposition 6. The Swish activation function φswish(x) = x · sigmoid(x) = x1+e−x satisfies all the conditions of Proposition 4.
Proof. To abbreviate notation, we note φ := φSwish = xex/(1 + ex) and h := ex/(1 + ex) is the Sigmoid function. This proof should be seen as a sketch of the ideas and not a rigourous proof.
• we have φ(0) = 0 and φ′(0) = 12 and ∣∣φ(x) x ∣∣ ≤ 1 • As illustrated in Table 1 in the main text, it is easy to see numerically that (ii) is satisfied.
Moreover, we observe that limσb→0 q = 0, which proves the second part of the (iii).
• Now we prove that F ′ > 0, we note g(x) := xφ′(x)φ(x). We have
g(x) = x2 (1 + e−x + xe−x)
(1 + e−x)3
Define G by
G(x) = { g(x) if x ≤ 0 −g(−x) if x > 0
so that G(−x) = −G(x) for all x ∈ R and g(x) ≥ G(x) for all x ≤ 0. Let x > 0, then
g(x) > G(x)⇐⇒ 1 + e−x + xe−x ≥ e−3x(−1− ex + xex)
which holds true for any positive number x. We thus have g(x) > G(x) for all real numbers x. Therefore E[g( √ xZ)] > 0 almost everywhere and F ′ > 0. The second part of (iii) was already proven above.
• Let σb > 0 and σw > 0 such that q exists. Recall that
f ′′(x) = σ2wqE[φ′′(U1)φ′′(U2(x))]
In Figure 10, we show the graph of E[φ′′(U1)φ′′(U2(x))] for different values of q (from 0.1 to 10, the darkest line being for q = 10). A rigorous proof can be done but is omitted here.
We observe that f ′′ has very small values when q is large, this is a result of the fact that φ′′ is concentrated around 0.
Remark : On the edge of chaos, we have σ2wE[φ′( √ qZ)2] = 1. Recall that F ′ can also be expressed as F ′(x) = σ2w(E[φ′( √ xZ)2] + E[φ′′( √ xZ)φ( √ xZ)]),
this yields F ′(q) = 1 + σ2wE[φ′′( √ qZ)φ( √ qZ)]. (7) The term E[φ′′(√qZ)φ(√qZ)] is very small compared to 1 (∼ 0.01), therefore F ′(q) ≈ 1. Notice also that the theoretical results corresponds to the equivalent Gaussian process, which is just an approximation of the neural network. Thus, using a value of (σb, σw) close to the EOC should not essentially change the quality of the result.

heading: B SUPPLEMENTARY THEORETICAL RESULTS

heading: B.1 SUFFICIENT CONDITIONS FOR BOUNDED ACTIVATION FUNCTIONS
text: We can replace the conditions "φ satisfies (ii)" in Proposition 5 by a sufficient condition. However, this condition is not satisfied by Tanh. Proposition 7. Let φ be a bounded function such that φ(0) = 0, φ′(0) > 0, φ′(x) ≥ 0, φ(−x) = −φ(x), xφ(x) > 0 and xφ′′(x) < 0 for x 6= 0, and |Eφ′(xZ)2| & |x|−2β for large x and some β ∈ (0, 1). Then, φ satisfies all the conditions of Proposition 4.
Proof. Let φ be an activation function that satisfies the conditions of Proposition 7. The proof is similar to the one of 5, we only need to show that having |Eφ′(xZ)2| & |x|−2β for large x and some β ∈ (0, 1) implies that (ii) of 4 is verified. We have that σ2w|Eφ′( √ qZ)2| & σ2wq−2β = σ2w(σ2b + σ2wE[φ( √ qZ)2])−β , so that we can make the term σ2w|Eφ′( √ qZ)2| take any value between 0 and ∞. Therefore, there exists σw such that (σb, σw) ∈ EOC, and assumption (ii) of Proposition 4 holds.
B.2 IMPACT OF SMOOTHNESS
In the proof of Proposition 5, we used the condition on φ′′ (odd function) to prove that f ′′ > 0, however, in some cases when we can explicitly calculate f , we do not need φ′′ to be defined. This is the case for Hard-Tanh, which is a piecewise-linear version of Tanh. We give an explicit calculation
of f ′′ for the Hard-Tanh activation function which we note HT in what follows. We compare the performance of HT and Tanh based on a metric which we will define later.
HT is given by
HT (x) =  −1 if x < −1 x if −1 ≤ x ≤ 1 1 if x > 1
Recall the propagation of the variance ql
ql+1 = σ2b + σ 2 wE(HT (
√ qlZ)2)
where HT is the Hard-Tanh function. We have E(HT ( √ qlZ)2) = E(1Z<− 1√
ql ) + E(1−1/ √ ql<Z<1/ √ ql Z2) + E(1 Z>1/ √ ql )
= 1− 2√ ql exp(− 1 2ql ) √ 2π
This yields ql+1 = g(ql)
where
g(x) = σ2b + σ 2 w(1− 2√ x exp(− 1x )√ 2π )
- EDGE OF CHAOS :
To study the correlation behaviour, we will assume that the variance converges to q. We have E(HT ′(√qZ)2) = E(1− 1√q<Z< 1√q ) = 2Ψ( 1√ q )−1 (where Ψ is the cumulative distribution function of a standard normal variable). The edge of chaos is then given by the equation σ2w(2Ψ( 1√ q )−1) = 1. We fix σw to its value on the edge.
Now let a and b be any two inputs. We have
cl+1 = f(cl)
where
f(x) = σ2b + σ 2 wE[φ(U1)φ(U2(x))]
q
with U1 := √ qZ1, and U2(x) := √ q(xZ1 + √ 1− x2Z2).
Lemma 3. Suppose ql converges to a fixed point q > 0. Then,
∀x ∈ [0, 1], f ′′(x) = σ 2 w
π √ 1− x2 (e− 1 q(1+x) − e− 1 q(1−x) )
Proof. We note α := 1/ √ q. For x ∈ [0, 1[, we have that :
f ′(x) = σ2wE[1−α<Z1<α × 1−α<xZ1+√1−x2Z2<α] = σ2wE[1−α<Z1<α × (1−α<xZ1+√1−x2Z2 − 1α<xZ1+√1−x2Z2)]
We deal with the first part f1(x) = σ2wE[1−α<Z1<α × 1−α<xZ1+√1−x2Z2 ], we have that :
f ′1(x) = σ 2 wE[1−α<Z1<α × 1√ 1− x2 (Z1 − x√ 1− x2 Z2)δ−α=xZ1+ √ 1−x2Z2 ]
= σ2w√ 2π E[1−α<Z1<α × 1√ 1− x2 Z1 + xα 1− x2 exp(− (xZ1 + α) 2 2(1− x2) )]
= σ2w
2π √ 1− x2 ∫ α −α z1 + xα 1− x2 exp(−z 2 1 + 2xαz1 + α 2 2(1− x2) )dz1
= σ2w
2π √ 1− x2 (e−
α2 1+x − e− α2 1−x )
we show a similar result for the second part and we conclude.
We proved that f ′′ > 0 for Hard-Tanh, all other conditions of proposition 5 (excluding the conditions on φ′′ since those were only used to prove f ′′ > 0) are verified, therefore the result of Proposition 4 holds for Hard-Tanh. we want to compare Tanh and Hard-Tanh when σb is small since this is the important case. The proof of Proposition 4 gives us an idea on how to compare them, the ratio σ 2 b
q controls the quality of approximation of f by the identity function, so a smaller ratio means a better approximation. Figure 2 shows that Tanh outperforms Hard-Tanh in this sense. This also means that for the same quality of approximation, we have bigger q (bigger output variance) with Tanh compared to Hard-Tanh. This can particularly be due to the nonsmoothness of Hard-Tanh, which slows down the dominated convergence in the proof of Proposition 4.

heading: C SUPPLEMENTARY EXPERIMENTAL RESULTS

heading: C.1 ELU ACTIVATION
text: We show numerically that the activation function ELU defined by φ(x) = (ex − 1)1x<0 + x1x≥0 satisfies the conditions of Proposition 4. We have φ(x) = 0, φ′(0+) = 1, φ′(0−) = 1 and ∣∣φ(x) x ∣∣ ≤ 1. Other conditions of Proposition 4 are shown numerically in graphs below.
Figure 12a shows the EOC curve (condition (ii) is satisfied). Figure 12b shows that is non-decreasing and Figure 12d illustrates the fact that limσb→0 q = 0. Finally, Figure 12c shows that function f is convex. Although the figures of F and f are shown just for one value of (σb, σw), the results are true for any value of (σb, σw) on the EOC.

heading: C.2 WHAT HAPPENS WHEN THE DEPTH IS LARGER THAN THE WIDTH?
text: Table 2 presents a comparative analysis of the validation accuracy of ReLU and Swish when the depth is larger than the width, in which case the approximation by a Gaussian process is not accurate (notice that in the approximation of a neural network by a Gaussian process, we first let Nl →∞, then we consider the limit of large L). ReLU tends to outperforms Swish when the width is smaller than the depth and both are small, however, we still observe a clear advantage of Swish for deeper architectures.

Review summaries:
The authors prove some theoretical results under the mean field regime and support their conclusions with a small number of experiments. Their central argument is that a correlation curve that leads to sub-exponential correlation convergence (edge of chaos) can still lead to rapid convergence if the rate is e.g. quadratic. They show that this is the case for ReLU and argue that we must ensure not only sub-exponential convergence, but also have a correlation curve that is close to the identity everywhere. They suggest activation functions that attain conditions as laid out in propositions 4/5 as an alternative.