Year: 2021

Abstract:
Using a mean-field theory of signal propagation, we analyze the evolution of correlations between two signals propagating forward through a deep ReLU network with correlated weights. Signals become highly correlated in deep ReLU networks with uncorrelated weights. We show that ReLU networks with anti-correlated weights can avoid this fate and have a chaotic phase where the signal correlations saturate below unity. Consistent with this analysis, we find that networks initialized with anti-correlated weights can train faster by taking advantage of the increased expressivity in the chaotic phase. An intiialization scheme combining this with a previously proposed strategy of using an asymmetric initialization to reduce dead node probability shows consistently lower training times compared to various other initializations on synthetic and real-world datasets. Our study suggests that use of initial distributions with correlations in them can help in reducing training time.

heading: 1 INTRODUCTION
text: Rectified Linear Unit (ReLU) Fukushima (1969); Fukushima & Miyake (1982) is the most widely used non-linear activation function in Deep Neural Networks (DNNs) LeCun et al. (2015); Ramachandran et al. (2018); Nair & Hinton (2010), applied to various tasks like computer vision Glorot et al. (2011b); Krizhevsky et al. (2012); He et al. (2015), speech recognition Maas et al. (2013); Tóth (2013); Hinton et al. (2012), intelligent gaming Silver et al. (2016), and solving scientific problems Seif et al. (2019). ReLU, φ(x) = max(0, x), outperforms most of the other activation functions proposed Glorot et al. (2011a). It has several advantages over other activations. ReLU activation function is computationally simple as it essentially involves only a comparison operation. ReLU suffers less from the vanishing gradients, a major problem in training networks with sigmoid-type activations that saturate at both ends Glorot et al. (2011b). They generalize well even in the overly parameterized regime Maennel et al. (2018).
Despite its success, ReLU also has a few drawbacks, one of which is the dying ReLU problem He et al. (2015); Trottier et al. (2017). The dying ReLU is a type of vanishing gradient problem in which the network outputs zero for all inputs and is dead. There is no gradient flow in this state. ReLU also suffers from exploding gradient problem, which occurs when backpropagating gradients become large Hanin (2018).
Several methods are proposed to overcome the vanishing/exploding gradient problem; these can be classified into three categories Lu et al. (2020). The first approach modifies the architecture, which includes using modified activation functions Ramachandran et al. (2018); He et al. (2015); Trottier et al. (2017); Clevert et al. (2016); Klambauer et al. (2017); Hendrycks & Gimpel (2016), adding connections between non-consecutive layers (residual connections)He et al. (2016), and optimizing network depth and width. The proposed activations are often computationally less efficient and require a fine-tuned parameter Lu et al. (2020). The second approach relies on normalization techniques Ba et al. (2016); Ioffe & Szegedy (2015); Salimans & Kingma (2016); Ulyanov et al. (2016); Wu & He (2018), the most popular one being batch normalization Ioffe & Szegedy (2015). Batch normalization prevents the vanishing and exploding gradients by normalizing the output at each layer but with an additional computational cost of up to 30% Mishkin & Matas (2016). A related strategy involves using the self-normalizing activation (SeLU), which by construction ensures output with zero mean and unit variance Klambauer et al. (2017). The third approach focuses on the
initialization of the weights and biases. As local (gradient-based) algorithms are used for optimization Kingma & Ba (2015); Zeiler (2012); Duchi et al. (2011), it is challenging to train deep networks with millions of parameters Du et al. (2019); Srivastava et al. (2015), and optimal initialization is essential for efficient training Nesterov (2014). He-initialization He et al. (2015) is a commonly used strategy that uses uncorrelated Gaussian weights with variance 2N , where N is the width of the network. Recently, Lu et al. (2020) proposed Random asymmetric initialization (RAI), which reduces the probability of dead ReLU at the initialization. In this paper, we aim to further improve the initialization scheme for ReLU networks.
A growing body of work has analyzed signal propagation in infinitely wide networks to understand the phase diagram of forward-propagation in DNNs Saxe et al. (2014); Poole et al. (2016); Raghu et al. (2017); Schoenholz et al. (2017); Lee et al. (2018); Hayou et al. (2019a); Li & Saad (2018; 2020); Bahri et al. (2020). We mention a few results for ReLU networks. Hayou et al. (2019a) showed that correlations in input signals propagating through a ReLU network always converge to one. Many other works found that ReLU networks are in general biased towards computing simpler functions De Palma et al. (2019); Rahaman et al. (2019); He et al. (2020); Valle-Perez et al. (2019); Hanin & Rolnick (2019), which may account for their better generalization properties even in the overly parameterized regime. However, from their successful application in different domains, one may guess that they should be capable of computing more complex functions. There might be a subspace of the parameters where the network can represent complex functions.
Li & Saad (2018; 2020) applied weight and input perturbations to analyze the function space of ReLU networks. They found that ReLU networks with anti-correlated weights compute richer functions than uncorrelated/positively correlated weights. Consistent with this, Shang et al. (2016) found that ReLU CNN’s produce anti-correlated feature matrices after training. These studies motivated us to analyze the phase diagram of signal propagation in ReLU networks with anti-correlated weights.
Following the mean-field theory of signal propagation proposed by Poole et al. (2016), we found that ReLU networks with anti-correlated weights have a chaotic phase, which implies higher expressivity. In contrast, ReLU networks with uncorrelated weights do not have a chaotic phase. Furthermore, we find that initializing ReLU networks with anti-correlated weights results in faster training. We call it Anti-correlated initialization (ACI). Additional improvement in performance is achieved by incorporating RAI, which reduces the dead node probability. This combined scheme, which we call Random asymmetric anti-correlated initialization (RAAI), is the main result of this work and is defined as follows. We pick weights and bias incoming to each node from anti-correlated Gaussian distribution and replace one randomly picked weight/bias with a random variable drawn from a beta distribution. The code to generate weights drawn from the RAAI distribution is given in Appendix G. We analyze the correlation properties of RAAI and show that it performs better than the best-known initialization schemes on tasks of varying complexity. It may be of concern that initialization in an expressive space may lead to overfitting, and we do observe the same for ACI for deeper networks and complex tasks. In contrast, RAAI shows no signs of overfitting and performs consistently better than all other initialization schemes.
We organize the article as follows. First, we contrast the mean-field analysis of ReLU networks with correlated weights with uncorrelated in Section 2. Next, Section 3 analyzes the critical properties of correlations in input signals for RAI and RAAI. Then, in Section 4, we describe the various tasks used to validate the performance of different initialization schemes in Section 5. Lastly, Section 6 concludes the article.

heading: 2 MEAN-FIELD ANALYSIS OF SIGNAL PROPAGATION WITH CORRELATED WEIGHTS
text: This section presents the mean-field theory of signal propagation (proposed by Ref. Poole et al. (2016)) in ReLU networks with correlated weights and compares it with uncorrelated weights. Unlike Ref. Li & Saad (2018; 2020), which study perturbation to a ReLU network, we aim to understand the phase diagram of the signal propagation. Furthermore, we provide numerical results to corroborate the mean-field results.
Consider a fully connected neural network with L layers (in addition to the input layer) and Nl nodes in layer l. The layer index ranges between 0 and L. For an input signal s0 = x, we denote
the pre-activation at node i in layer l by hli(x) and activation by s l i(x). A signal (s l−1 1 , . . , s l−1 Nl
) at layer l − 1 propagates to layer l by the rule
hli(x) =
Nl−1∑
j=1
wlijs l−1 j (x) + b l i where l ∈ {1, L}
sli(x) = φ(h l i(x)),
where φ is the non-linear activation function and wlij , b l i are the weights and biases. We consider correlations within the set of weights (wli) incoming to each node i. The correlated Gaussian distribution is
P (wl1,w l 2,w l 3 . . . ) =
Nl∏
i
e(− 1 2 (w l i) TA−1wli)
√ (2π)Nl−1 |A|
with A = σ2w Nl−1
( I− k
1 + k
J
Nl−1
) , (1)
where I is the identity matrix, J is an all-ones matrix, and k parameterizes the correlation strength. Positively correlated and anti-correlated regimes correspond to the regions −1 < k < 0 and k > 0, respectively, whereas k = 0 generates uncorrelated weights. The overall scaling by 1/Nl−1 in the covariance matrix ensures that the input contribution from the last layer to each node is O(1). The bias is drawn from a Gaussian distribution bli ∼ N (0, σ2b ). Note that weights reaching two different nodes are uncorrelated, and also the bias is uncorrelated with the weights.
To track the layer-wise information flow, consider the squared length and overlap of the preactivations for two input signals, s0 = x1 and s0 = x2, after propagating to layer l
qlh(xa) = 1
Nl
Nl∑
i=1
( hli(xa) )2 where a ∈ {1, 2} and 1 ≤ l ≤ L
qlh(x1, x2) = 1
Nl
Nl∑
i=1
hli(x1)h l i(x2).
Assuming self averaging, consider an average over the weights and the bias incoming to layer l. For simplicity of notations later, we use the same symbol for averaged qlh.
qlh(xa) = σ2w Nl−1
Nl−1∑
j,m=1
( δj,m − k
1 + k
1
Nl−1
) φ(hl−1j (xa))φ(h l−1 m (xa)) + σ 2 b
qlh(x1, x2) = σ2w Nl−1
Nl−1∑
j,m=1
( δj,m − k
1 + k
1
Nl−1
) φ(hl−1j (x1))φ(h l−1 m (x2)) + σ 2 b ,
For large width, each hl−1i is a weighted sum of a large number of zero-mean random variables. Thus, we expect the joint distribution of hl−1i (x1) and h l−1 i (x2) to converge to a zero-mean Gaussian with a covariance matrix with diagonal entries ql−1h (x1), q l−1 h (x2) and off-diagonal entries ql−1h (x1, x2). On replacing the average over h l−1 i (this is equivalent to considering an average over all previous layers) in the last layer with an average over this Gaussian distribution, we obtain iterative maps for the length and overlap. Specializing to ReLU activation yeilds the equations
qlh(x) = σ2w 2
( 1− k
1 + k
1
π
) ql−1h (x) + σ 2 b
qlh(x1, x2) = σ2w 2 ( f(cl−1h )− k 1 + k 1 π )√ ql−1h (x1)q l−1 h (x2) + σ 2 b
f(cl−1h ) = cl−1h 2 + cl−1h π sin−1(cl−1h ) + 1 π √ 1− (cl−1h )2,
(2)
where clh = qlh(x1,x2)√ qlh(x1)q l h(x2) is the correlation coefficient between the two signals reaching layer l (for details of the derivation, see Appendix A).
Poole et al. (2016) found that the signal’s length reaches its fixed point within a few layers, and the fixed point of the correlation coefficient, c∗h, can be estimated with the assumption that q l h(x) has reached its fixed point q∗h. We can check that c ∗ h = 1 is always a fixed point of the recursive map (Eqn. 2) under this assumption. The stability of the fixed point c∗h = 1 is determined by
χ1 ≡ ∂clh ∂cl−1h ∣∣∣ cl−1h =1 ,
which evaluates to σ 2 w
2 . χ1 separates the parameter space into two phases —first, an ordered phase with χ1 < 1, where the c∗h = 1 fixed point is stable; and second, a chaotic phase with χ1 > 1, where the c∗h = 1 fixed point is unstable. χ1 = 1 defines the phase boundary line. In the ordered phase, two distinct signals will become perfectly correlated asymptotically. In the chaotic phase, the correlations converge to a stable fixed point below unity. Two closely related signals will eventually lose correlations in this phase. This suggests that initializing the network with parameters (σ2w, σ 2 b ) at the phase transition boundary (corresponding to an infinite depth of correlations) allows for an optimal information flow through the network Poole et al. (2016); Schoenholz et al. (2017)
ReLU networks with uncorrelated weights (k = 0) The above analysis is applied assuming q∗h is finite shows that ReLU networks with uncorrelated weights do not have a chaotic phase, and any two signals propagating through a ReLU network become asymptotically correlated for all values of (σ2w, σ 2 b ). In other words, c ∗ h = 1 is always a stable fixed point. However, the parameter space can be classified into two phases based on the boundedness of the fixed point q∗h of the length map (Eqn. 2) - first, a bounded phase where q∗h is finite and non-zero; second, an unbounded phase, where q ∗ h is either zero or infinite Lee et al. (2018); Hayou et al. (2019b). The two phases are separated by the boundary σ2w = 2. Figure 1a depicts the phase diagram for ReLU networks with uncorrelated weights. Note that the analysis of the stability of c∗h = 1 fixed point in ReLU networks is valid only in the bounded phase. However, numerical results presented in Fig. 2 indicate that the fixed point remains stable even in the unbounded phase.
ReLU networks with correlated weights The phase diagram for ReLU networks with correlated weights can be analyzed similarly. The length is bounded if σ2w < gk = 2
(1− k1+k 1 π )
. Thus, for anti-
correlated weights (k > 0), the boundary gk moves upwards relative to the k = 0 case (see Fig. 1a). The c∗h = 1 fixed point of the correlations is unstable in this region of the bounded phase.
In summary, anti-correlations induce a bounded chaotic phase in 2 < σ2w < gk (see Fig. 1b). We demonstrate these results numerically in Fig. 3 for a correlation strength of k = 100. As predicted by the above equations, the stability of the fixed point c∗h = 1 changes at σ 2 w = 2, and the length diverges at gk=100 = 2.92. In contrast, for positively correlated weights, the length’s fixed point boundary shifts downward resulting in a similar phase diagram as uncorrelated weights.
As a result, a ReLU network with anti-correlated weights can be more expressive by taking advantage of a chaotic phase, and it may be beneficial for a ReLU network to remain in this subspace. Thus, we propose initializing ReLU networks with anti-correlated weights at the order to
chaos boundary (σ2w, σ 2 b ) = (2, 0). We call it Anti-Correlated Initialization (ACI). Appendix B demonstrates that ReLU networks initialized with anti-correlated weights give an advantage over He initializations for a range of tasks.
Many alternatives are proposed to improve ReLU networks Trottier et al. (2017); Lu et al. (2020); Clevert et al. (2016). Of particular interest is Random asymmetric initialization (RAI), which aims to increase expressivity through an independent strategy of reducing the dead node probability. In the next section, we analyze the correlation properties of RAI and then combine it with ACI to propose a new initialization scheme RAAI, which has both a chaotic phase and low dead node probability.

heading: 3 RANDOM ASYMMETRIC ANTI-CORRELATED INITIALIZATION
text: We begin by analyzing critical properties of Random asymmetric initialization (RAI) proposed in Lu et al. (2020) to reduce the dead node probability. For ReLU networks with symmetric distributions for weights and biases, the dead node probability is half. RAI reduces it by initializing one of the weights/the bias incoming to each node from a distribution with positive support (like the beta distribution), resulting in a positive mean for the pre-activations. Lu et al. (2020) proposes initializing RAI with a variance σ2w = 0.36 to ensure that the signal’s length is bounded. We analyzed the correlation properties of RAI and found that c∗h = 1 is always a fixed point of the recursive maps (see Appendix C). Deriving the stability condition for the fixed point c∗h = 1 even with the meanfield assumptions is difficult. However, numerical results presented in Figure 4 show that c∗h = 1 is always a stable fixed point (right panel), and the length remains finite for σ2w up to 0.72 (left panel). A qualitative picture of the phase diagram can be captured with additional assumptions over the mean-field approximation (see Appendix D).
RAI focuses on decreasing the dead node probability to increase expressive power, whereas ACI uses anti-correlated weights to improve the expressivity. As RAI and ACI increase the expressivity using different mechanisms, we explore the possibility of combining the two. We call it Random asymmetric anti-correlated initialization (RAAI). To prepare weights drawn from RAAI, we consider anti-correlated Gaussian weights and bias incoming to each node (like Eqn. 3) and replace one randomly picked weight/bias with a random number drawn from a beta distribution. Note that the weights and biases reaching different nodes are uncorrelated. Like ACI, we observe three phases for RAAI. Numerical results presented in Fig. 5 suggest that the order to chaos boundary is around σ2w = 0.9, and the length diverges for σ 2 w > 1.2. Again, a qualitative picture of the phase diagram can be captured with additional assumptions over the mean-field approximation (see Appendix E).
In summary, RAAI has a chaotic phase like ACI, and a lower dead node probability like RAI, as can be checked numerically. As RAAI inherits the advantages of both strategies, we expect it to be a strong candidate for initializing ReLU networks. Table 1 summarizes and compares different
initialization schemes. In the following sections, we analyze the training dynamics and performance of RAAI and compare it with other initialization schemes.

heading: 4 TRAINING TASKS
text: This section describes various tasks used to analyze the dynamics and performance of different initialization schemes. We consider a variant of teacher-student setup Seung et al. (1992), in which a student ReLU network is trained with examples generated by an untrained teacher network. We consider three different tasks with varying complexities.
1. First, a standard teacher task, in which the training data is generated by a ReLU network of the same size as the student network, initialized with He initialization.
2. Next, we consider a simple teacher task in which the capacity of the teacher network is much lower than the student network. In many real data sets, the high-dimensional inputs lie in a lowdimensional manifold Goldt et al. (2020), which motivates us to consider a simple teacher task. We consider a single-layer ReLU network with N = 10 nodes, initialized with He initialization.
3. Lastly, we consider a complex teacher task, in which the complexity of the teacher network is more than the student network. We consider a teacher network with tanh activation of the same size as the student network initialized in the chaotic regime, (σ2w, σ 2 b ) = (1.5, 0) Poole et al. (2016). A ReLU network initialized with symmetric distributions has half of the nodes dead. Therefore, it has a lower capacity than a tanh network of the same size.
We consider an L = 10 layered (in addition to input layer) student network with a constant width N = 100, trained using SGD and Adam algorithms (for further details, see 6).

heading: 5 COMPARISON OF LEARNING DYNAMICS FOR DIFFERENT INITIALIZATION SCHEMES
text: This section compares the performance of RAAI with other initialization schemes listed in Table 1 on tasks described in Section 4.
Standard teacher task Figure 6 shows the average validation loss for the standard tasks trained with SGD and Adam algorithms. We observe that RAAI performs better than all other schemes.
Simple teacher task Figure 7 shows the average validation loss for the simple teacher task. Similar to the standard teacher task, RAAI performs better than or on par with other initialization schemes.
Complex teacher task Figure 8 shows the average validation loss for the complex teacher task. We observe that for a complex teacher, ACI starts to perform worse when trained with SGD algorithm, whereas, RAAI faces no such problem and performs better or on par with other initializations.
In various scenarios, RAI performs comparable to RAAI, however, we find that RAAI performs better RAI on real-world datasets. We present different intitialization schemes on three different real-world datasets —MNIST, Fashion-MNIST and CIFAR-10. We find that RAAI outperforms all other intialization schemes at early training steps and all initialization schemes perform equally well after a few epochs. For further details, see Appendix F.

heading: 6 DISCUSSION AND CONCLUSION
text: In this article, we analyzed the evolution of correlation between signals propagating through a ReLU network with correlated weights using the mean-field theory of signal propagation. Multiple studies show that ReLU networks with uncorrelated weights are biased towards computing simpler functions, but ReLU networks do perform complex tasks in practice. Unlike ReLU networks with uncorrelated weights, ReLU networks with anti-correlated weights reaching a node have a chaotic phase where correlation saturates below unity. This suggests that such networks can exhibit higher expressivity. Although we have focused on the ReLU networks in this study, anti-correlation in weights may be useful in general. Networks with other non-linear activation functions like tanh, SELU, and sigmoid have a chaotic phase even with uncorrelated weights. In these cases, the weight correlations may still help to tune the phase boundaries and expressivity of the networks.
We further investigated the possibility that ReLU networks with the enhanced expressivity may prove beneficial in faster learning. Comparison of training and test performance of networks in a range of teacher-student setups clearly showed that networks with anti-correlated weights learn faster. While ACI shows better learning performance in general, it shows poor performance with SGD during an intermediate learning stage when the teacher network has a relatively higher capacity. We believe that this may be due to the system getting stuck in local minima. This is consistent with the absence of a similar regime on training with Adam optimizer. On training deeper networks with ACI, we found that it overfits, but this can be avoided by fine-tuning correlation strength k. We also investigated a possible improvement in training time from adding a regularization term in the loss function that favors anti-correlated weights, but our attempts did not show any systematic results.
We compared ACI with a recently proposed initialization scheme called RAI, which introduces a systematic asymmetry (around 0) in the weights to decrease dead node probability. We find that the relative performance between RAI and ACI depends on the task and the optimization algorithm. RAI improves expressivity by reducing the dead node probability, whereas ACI achieves the same by inducing a chaotic phase. As RAI and ACI rely on different mechanisms, we explored a strategy of combining the two initialization schemes. We analyzed the correlation properties of the combined scheme, which we call RAAI and found that it has a chaotic phase like ACI. We demonstrated that RAAI leads to faster training and learning than commonly-used methods on various teacher tasks of a range of complexity. For different initialization schemes, the behavior of the training dynamics at large epochs may depend on the optimizer and training data, however RAAI shows a definite advantage over other schemes when using the SGD optimizer, especially in early training epochs. In addition to faster training, RAAI also shows no sign of overfitting and thus improves on the simpler strategy that relies only on anti-correlations. Our study has focused on adding simple two point correlation in the initial distributions motivated by a richer phase space for ReLU networks with anti-correlated weights. This simplest deviation from the uncorrelated Gaussian distribution showed a consistent advantage in terms of training time, suggesting that initialization with more complex and tailored correlations may lead to better performance.

heading: ETHICS STATEMENT
text: We do not think that there are any direct negative impacts of this work, however, faster training using RAAI may lead to easier training of algorithms with negative societal impact.

heading: REPRODUCIBILITY STATEMENT
text: We implemented feedforward networks in Tensorflow (version 2.2.0) Abadi et al. (2015) and train them using 105 training examples with a mean squared error loss, a mini-batch size of 32, and default parameters for the optimizers. The validation set contains 103 examples.

heading: A DERIVATION OF THE LENGTH AND CORRELATION MAPS FOR CORRELATED WEIGHTS
text: This section derives the length and covariance maps for ReLU networks with correlated weights given by
P (wl1,w l 2,w l 3 . . . ) =
Nl∏
i
exp ( − 12 (wli)TA−1wli ) √
(2π)Nl−1 |A| , (3)
where the covariance matrix given by
A = σ2w Nl−1
( I− k
1 + k
J
Nl−1
) .
Here I is the identity matrix, J is an all-ones matrix, and k parameterizes the correlation strength. Positively correlated and anti-correlated regimes correspond to the regions −1 < k < 0 and k > 0, respectively, whereas, k = 0 generates uncorrelated weights. For simplicity, we consider Nl = N in all layers, but the results hold for all Nl, as long as it is large.

heading: A.1 DERIVATION OF LENGTH MAP
text: To derive the length map, we follow the approach introduced by 36. Assuming self-averaging, we obtain the average value of the squared length of a signal, s0 = x, after propagating to layer l by considering an average over weights and biases between layer l and l − 1
qlh(x) = 1
N
〈 N∑
i=1
(hli(x)) 2
〉 = 1
N
N∑
i=1
N∑
j,m=1
〈 wlijw l im 〉 φ(hl−1j (x))φ(h l−1 m (x)) + 〈 (bli) 2 〉
qlh(x) = σ2w N
N∑
j,m=1
( δj,m − k
1 + k
1
N
) φ(hl−1j (x))φ(h l−1 m (x)) + σ 2 b ,
(4)
where we have used 〈 wlijw l im 〉 = σ2w N ( δj,m − k1+k 1N ) and 〈 (bli) 2 〉 = σ2b . For large N , each hl−1i (x) is a weighted sum of a large number of correlated random variables, which converges to a zero-mean Gaussian with a variance ql−1h (x). Replacing the average over h l i at layer l − 1 by a Gaussian distribution to get the general form of the recursive map. This average corresponds to averaging over all the weights and biases upto layer l − 1.
qlh(x) = σ 2 w
∫ Dz φ (√ ql−1h (x) z )2 − σ2w k
1 + k
[∫ Dz φ (√ ql−1h (x) z )]2 + σ2b ,
where Dz is the standard normal distribution. For the second term in the Eqn. 4, we have used the fact that for different nodesm 6= j, hlj(x) and hlm(x) are uncorrelated random variables and ignored O(1/N) terms. Note that the weights and biases reaching two different nodes are uncorrelated. For a ReLU activation, we can perform the integrals to get the exact form of the recursive relation between qlh(x) and q l−1 h (x)
qlh(x) = σ2w 2
( 1− k
1 + k
1
π
) ql−1h (x) + σ 2 b . (5)
.

heading: A.2 DERIVATION OF THE COVARIANCE MAP
text: The covariance map can be derived similarly by considering an average over the weights and biases
qlh(x1, x2) = 1
N
〈 N∑
i=1
hli(x1)h l i(x2)
〉 = N∑
j,m=1
〈 wlijw l im 〉 φ(hl−1j (x1))φ(h l−1 m (x2)) + 〈 (bli) 2 〉
qlh(x1, x2) = σ2w N
N∑
j,m=1
( δj,m − k
1 + k
1
N
) φ(hl−1j (x1))φ(h l−1 m (x2)) + σ 2 b ,
and then replacing the sum over neurons in the previous layer with an integral with a Gaussian measure. For large N , the joint distribution of hlj(x1) and h l m(x2) will converge to a two-dimensional Gaussian distribution with a covariance matrix
Σl−1 =
[ ql−1h (x1) q l−1 h (x1, x2)
ql−1h (x1, x2) q l−1 h (x2)
] .
The correlations among hlj(x1) and h l m(x2) are induced as the two signals are propagating through the same network. Propagating this joint distribution across one layer, we obtain the iterative map
qlh(x1, x2) = σ 2 w ∫ Dz1Dz2φ(u1)φ(u12)− σ2w k
k + 1
∫ Dz1Dz2 φ(u1)φ(u2) + σ 2 b
u1 = √ ql−1h (x1)z1, u12 = √ ql−1h (x2) [ cl−1h z1 + √ 1− (cl−1h )2 z2 ] u2 = √ ql−1h (x2)z2,
where clh = qlh(x1,x2)√ qlh(x1)q l h(x2) is the correlation coefficient, andDz1, Dz2 are standard normal Gaussian distributions. Again, in the second part of the above equation, we have used the fact that for j 6= m, hlj(x1) and h l m(x2) are uncorrelated random variables and have ignored O(1/N) terms. Further, we can perform the integrals for ReLU networks to get the exact recursive map
qlh(x1, x2) = σ2w 2 ( f(cl−1h )− k 1 + k 1 π )√ ql−1h (x1)q l−1 h (x2) + σ 2 b (6)
f(cl−1h ) = cl−1h 2 + cl−1h π sin−1(cl−1h ) + 1 π √ 1− (cl−1h )2. (7)

heading: B TRAINING WITH ANTI-CORRELATED VS. POSITIVELY CORRELATED INITIALIZATION
text: This section compares the training dynamics and performance of ReLU networks initialized with different weight correlation strengths on tasks described in Section 4. We utilize the increased expressivity in ReLU networks with anti-correlated weights (ACI) at the initial training phase and compare its training dynamics with He initialization and positively correlated weight initialization. We observe that ACI provides a definite advantage over the other two initialization schemes. We also find that positively correlated weight ReLU networks train slower than He initialization, suggesting that anti-correlation may develop in weights during training. We choose three different correlation strengths; k = 100 induces anti-correlated weights, k = −0.5 produces positively correlated weights, and lastly, k = 0 corresponds to uncorrelated weights (He initialization). We train networks with two different optimization algorithms, SGD and Adam. For SGD, we train for 104 epochs, and for Adam, we train for 103 epochs.
Standard teacher task Figure 9 shows the average validation loss for different correlation strengths trained using SGD and Adam algorithms. We observe that ReLU networks initialized with ACI train faster than He initialization. In contrast, ReLU networks with positively correlated weights train slower than He initialization.
Simple teacher task Figure 10 shows the average validation loss for the simple teacher task. We observe similar qualitative results as in the standard teacher task. For SGD, we observe an initial linear region in which all initialization schemes perform equally; however, at large epochs, ACI shows a definite advantage over other initializations.
Complex teacher task Figure 11 shows the average validation loss for a complex teacher task. For some intermediate regions, ACI performs worse than other initializations on training with SGD. The regions where ACI performs poorly shift depending on the complexity of the task.

heading: C DERIVATION OF THE LENGTH AND COVARIANCE MAP FOR RAI
text: To draw weights from the RAI distribution, we first initialize the weights and bias incoming to each node with a Gaussian distribution N (0, σ 2 w
N ). Next, we replace one weight or the bias incoming to each neuron by a random variable from beta distribution (see 18 for details). The weights and bias are treated on an equal footing. Thus, to simplify the notations, we incorporate the bias in the weight matrix by introducing a fictitious additional node with a constant value of one, i.e.,
sl(x) = [φ(hl(x)), 1].
The evolution equation is now given by
hl(x) = Wl · sl−1(x).
It is easier to track the evolution using the activation instead of the pre-activations. So we define a few covariance matrices which will come in handy
qls(x1, x2) = 1
N + 1
N∑
i=0
sli(x1)s l i(x2)
ql−klj (x1, x2) =
1
N
N∑
t 6=klj
slt(x1)s l t(x2) ,
where klj tags variables associated with the special weight. We will use the notations q l s(x) = qls(x, x) and q l −k(x) = q l −k(x, x). The corresponding correlation coefficients are given by,
cls = qls(x1, x2)√ qls(x1) q l s(x2)
cl−klj =
ql−klj (x1, x2)
√ ql−klj (x1) ql−klt (x2)

heading: C.1 DERIVATION OF THE LENGTH MAP FOR RAI
text: Given hl−1(x) and considering weights between layers l and l− 1, we can view hlj(x) as a random variable
hlj(x) = σw √ ql−1 −kl−1j (x) z + sl−1 kl−1j (x) u,
where z ∼ N (0, 1) and u ∼ β(2, 1). By applying the activation function and squaring it, we obtain
φ(hlj(x)) 2 = φ ( σw √ ql−1 −kl−1j (x) z + sl−1 kl−1j (x) u )2 .
Next, we take an average over the weights and the special weight ( average denoted by < . >) to get
〈 φ(hlj(x)) 2|hl−1(x) 〉 = N∑
kl−1j =0
1
N + 1
∫ dz du f(z) g(u) φ ( σw √ ql−1 −kl−1j (x) z + sl−1 kl−1j (x) u )2 ,
where g(u) ∼ β(2, 1) distribution, and f(z) ∼ N (0, 1). We can take a sum over all nodes and re-write the equation in terms of the overlap
〈 qls(x)|hl−1(x) 〉 = 1
N + 1
 1 + N∑
j=1
N∑
kl−1j =0
1
N + 1
∫ dzdu f(z)g(u) φ ( σw √ ql−1 −kl−1j (x) z + sl−1 kl−1j (x) u
)2   .
(8)

heading: C.2 DERIVATION OF THE COVARIANCE MAP FOR RAI
text: The covariance map can be derived similarly, with a key difference of covariance between the preactivations. For two input signals s0 = x1 and s0 = x2, the covariance map reads
〈 qls(x1, x2)|hl−1(x1), hl−1(x2) 〉 = 1
N + 1
 1 + N∑
j=1
N∑
kl−1j =0
1
N + 1
∫ dy1dy2du f(y1, y2) g(u) ×
× φ ( σw √ ql−1 −kl−1j (x1) y1 + s l−1 kl−1j (x1) u ) φ ( σw √ ql−1 −kl−1j (x2) y2 + s l−1 kl−1j (x2) u )] ,
(9)
where f(y1, y2) is the joint Gaussian distribution of y1 and y2, with a covariance matrix given by
Σl−1 klj =
  ql−1 −kl−1j (x1) q l−1 −kl−1j (x1, x2)
ql−1 −kl−1j (x1, x2) q l−1 −kl−1j (x2)
  .
We can re-write Eqn. 9 in terms of cl−klj
〈 qls(x1, x2)|hl−1(x1), hl−1(x2) 〉 = 1
N + 1
 1 + ∑
j
∑
kl−1j
1
N + 1
∫ dz1 dz2 du f(z1)f(z2) g(u) ×
× φ ( σw √ ql−1 −kl−1j (x1) z1 + s l−1 kl−1j (x1) u ) × × φ ( σw
√ ql−1 −kl−1j (x2) [ cl−klj z1 + √ 1− (cl−klj ) 2 z2 ] + sl−1 kl−1j (x2) u )] ,
where f(z1) ∼ f(z2) ∼ N (0, 1) are standard Gaussian distributions. As suggested by 36, we can find the fixed point of the correlation map under the assumption that the length qlh(x) has reached its fixed point. It can be checked that clh = 1 is a fixed point of the correlation map.

heading: D STABILITY OF THE FIXED POINTS FOR THE LENGTH AND CORRELATION MAPS FOR RAI

heading: D.1 STABILITY OF THE FIXED POINT FOR THE LENGTH MAP FOR RAI
text: The derivation of the analytical form of the length map (Eqn. 8) is difficult, and only bounds to the map have been derived (see 18). Inspired by the analytical form of the length map for the anticorrelated initialization and the analysis done by 18, we assume that the length map has a linear dependence on ql−1s (x). Under this assumption, we can find the stability of the fixed point of the length map by taking a derivative with respect to ql−1s (x). Yet another problem exists. While taking a derivative, we have to encounter derivatives of the form
∂sl−1 −kl−1j (x)
∂ql−1h (x) .
To simplify the calculations further, we employ a mean-field type approach by approximating ql−1−klj (x) and sl−1 kl−1j by ql−1s (x). Note that we can also approximate s l−1 klj by its mean value, giving the same qualitative results. This simplifies Eqn. 8 to
〈 qls(x)|hl−1(x) 〉 = 1
N + 1
[ 1 + (N + 1) ∫ dz du f(z) g(u) φ (√ ql−1s (x)(σwz + u) )2] .
To find the fixed point of the length map, we take a derivative wrt ql−1s (x) to get the condition for stability of the fixed point q∗. We denote this derivative by ζq∗ . It separates the dynamics into two phases —a bounded phase when ζq∗ < 1, and an unbounded phase when ζq∗ > 1.
ζq∗ = ∂qls(x)
∂ql−1s (x) ∣∣∣ ql−1s (x)=q∗
ζq∗ = ∂
∂ql−1s (x)
∫ dz du f(z) g(u) φ (√ ql−1s (x)(σwz + u) )2
ζq∗ = 1√
ql−1s (x)
∫ dz du f(z) g(u) (σwz + u)φ ′ (√ ql−1s (x)(σwz + u) ) φ (√ ql−1s (x)(σwz + u) )
ζq∗ = σ 2 w ∫ dz du f(z) g(u) [φ′ (σwz + u)] 2 + σw ∫ dz du f(z) g(u)φ′ (σwz + u)φ (σwz + u)
(10) where we have used the fact that for a > 0, φ(ax) = a φ(x). On evaluating the integral, we find that ζq∗ = 1 when σ2w = 0.56. This critical value underestimates the numerical value obtained in Fig. 4.

heading: D.2 STABILITY OF THE FIXED POINT FOR THE CORRELATION MAP FOR RAI
text: Under the assumption, qls(x) → q∗, the correlation map has a fixed point c∗s = 1, and its stability is given by χ1 =
∂cls ∂cl−1s evaluated at cl−1s = 1. But again, we get into the difficulties mentioned in the previous section, and we employ the same assumptions to arrive at a tractable equation for the correlation map
〈 cls|hl−1(x1), hl−1(x2) 〉 = 1
q∗s (x)
1
N + 1
[ 1 +N ∫ dz1 dz2 du f(z1)f(z2) g(u) ×
× φ (√ q∗s (x)(σw z1 + u) ) φ (√ q∗s (x) [ cl−1s σw z1 + √ 1− (cl−1s )2 σw z2 + u ])] ,
Next, we take a derivative to get the condition for the stability of the fixed point c∗h = 1
χ1 = ∂clh ∂cl−1h ∣∣∣ cl−1h =1
χ1 = 1
q∗h(x)
∂
∂cl−1h
∫ dz1dz2duf(z1)f(z2)g(u)φ (√ q∗s (x)(σw z1 + u) ) ×
× φ (√
q∗s (x) [ cl−1s σw z1 + √ 1− (cl−1s )2 σw z2 + u ])∣∣∣ cl−1h =1
χ1 = σ 2 w ∫ dz du f(z) g(u) [φ′(σwz + u)] 2 . (11)
The above equation is the same as the first term we obtained in the condition for the stability of the length map (Eqn. 10). We obtain a critical value of σ2w = 1.41 by solving for χ1 = 1. We observe that the critical point for the length is smaller than the critical point of the correlation coefficient, and from our experience with ReLU networks calculations, we expect RAI to have an ordered phase only, which is confirmed by numerical results shown in Fig. 4.

heading: E DERIVATION OF LENGTH AND CORRELATION MAP FOR RAAI AND STABILITY CONDITIONS

heading: E.1 DERIVATION FOR THE LENGTH MAP FOR RAAI AND THE STABILITY CONDITION
text: Similar to the previous section, we can view hlj(x) as a random variable
hlj(x) = σw
√ q̃l−1(x) z + sl−1
−kl−1j (x) u,
where q̃l−1(x) = ql−1 −kl−1j
(x) ( 1− k1+k 1π ) . Then, we can re-define σw as
σ̃2 = σ2w
( 1− k
1 + k
1
π
) ,
which yields,
hlj(x) = σ̃ √ ql−1 −kl−1j (x) z + sl−1 −kl−1j (x) u,
Now, the entire analysis goes through as Appendix C.1, just with a re-definition of the variance. Now, we can read off the stability condition for the fixed point of the length map
ζq∗ = σ̃ 2 w
∫ dz du f(z) g(u) [φ′ (σ̃z + u)] 2 + σ̃ ∫ dz du f(z) g(u)φ′ (σ̃z + u)φ (σ̃z + u) (12)
On solving the equations numerically, we observe that the length is bounded when σ2w < 1.75, which overestimates the numerical value observed in Fig. 5.

heading: E.2 DERIVATION FOR THE CORRELATION MAP FOR RAAI AND THE STABILITY CONDITION
text: The correlation map for RAAI can be derived similar to RAI (Appendix C.2), with a key difference being the covariance matrix. The covariance matrix, in this case, is
Σl−1 klj (x1, x2) =  
ql−1 −kl−1j (x1)− k1+k ( ml−kl−1j (x1) )2 ql−1 −kl−1j (x1, x2)− k1+kml−kl−1j (x1)m l −kl−1j (x2)
ql−1 −kl−1j (x1, x2)− k1+kml−kl−1j (x1)m l −kl−1j (x2) q l−1 −kl−1j (x2)− k1+k ( ml−kl−1j (x2) )2
  .
Again, we can check that c∗s = 1 is a fixed point of the dynamics. The stability of the fixed point under the assumptions considered in Appendix D.2 determines the order to chaos boundary at σ2w = 1.41. The critical value overestimates the numerical results presented in Fig. 5.
Note that instead of approximating sl−1 kl−1j by its RMS value ql−1s (x), we can also approximate it by its mean value. In this case, we observe similar qualitative results. In Table 2, we compare the boundaries predicted by the RMS and mean approximations.

heading: F COMPARISON OF THE PERFORMANCE OF DIFFERENT INITIALIZATION SCHEMES ON REAL-WORLD DATASETS
text: In this section, we compare the performance of different initializiation schemes on three different real-world datasets. We consider MNIST, Fashion-MNIST and CIFAR-10 datasets, and train them with feedforward networks with depth L = 10 (in addition to input layer) and a constant width of N = 100 for all hidden layers. We implemented feedforward networks in Tensorflow and train them using the complete training set with a cross entropy loss, a mini-batch size of 32, and default parameters for the optimizers. As high performance for these datasets is achieved quickly (in terms of epochs), we observe the training accuracy as a function of the number of steps (and not epochs). It is noteworthy that we used training accuracy for demonstration purposes and validation accuracy shows similar behaviour for the trends.

heading: F.1 MNIST TASK
text: Figure 12 shows the average training accuracy for the MNIST task trained with SGD. We observe that RAAI performs better than all other schemes, however, the advantage is only observed at early time steps.

heading: F.2 FASHION-MNIST TASK
text: Figure 13 shows the average training accuracy for the MNIST task trained with SGD. We observe that RAAI performs better than all other schemes, however, the advantage is only observed at early time steps.
F.3 CIFAR-10
Figure 14 shows the average training accuracy for the CIFAR-10 task trained with SGD. We observe that RAAI performs better than all other schemes, however, the advantage is only observed at early time steps.

heading: G CODE TO GENERATE WEIGHTS DRAWN FROM RAAI DISTRIBUTION
text: 1 import numpy as np 2 3 def RAAI(fan_in, fan_out, k = 100, variance_weights = 0.9): 4 """Randomized Asymmetric Anti-correlated Initializer (RAAI) 5 Arguments: 6 fan_in -- the number of neurons in the previous layer 7 fan_out -- the number of neurons in the next layer 8 corr -- correlation strength for the Gaussian weights 9 variance_weights -- variance of the weights
10 Returns: 11 W, b -- weight and bias matrices with shape(fan_in, fan_out), and ( fan_out, ) 12 """ 13 corr = k/(1+k) 14 mean = np.zeros(fan_in + 1) 15 J = np.ones((fan_in + 1, fan_in + 1)) 16 cov = (np.identity(fan_in + 1) - J*(corr/(fan_in +1)) )* variance_weights/fan_in 17 P = np.random.multivariate_normal(mean = mean, cov = cov, size = ( fan_out)) 18 for j in range(P.shape[0]): 19 k = np.random.randint(0, high = fan_in + 1) 20 P[j, k] = np.random.beta(2, 1) 21 W = P[:, :-1].T 22 b = P[:, -1] 23 return W.astype(np.float32), b.astype(np.float32)

Review summaries:
The authors propose an initialization method called random asymmetric anti-correlated initialization (RAAI), which initializes residual networks in an expressive subspace of weights. RAAI is a combined initialization of anti-correlated initialization (ACI) and random asymmetric initialization (RAI), where the last initialization method is already proposed. Through both theoretical analysis and empirical evaluations, the authors show that RAAI has both a chaotic phase and low dead node probability, which accelerates the rate of convergence.

The paper studies the signal propagation properties of deep ReLU networks at initialization time. For a standard ReLU network, the correlation between activations of two training examples converges to one as the depth goes to infinity. To resolve that, the authors proposed to use correlated weights at initialization time to avoid this pathology. With correlated weights, the correlation could instead converge to a value smaller than one (so-called chaotic phase). Empirically, the authors found the networks initialized with correlated weights train faster.

This paper proposes and studies the initialization of ReLU networks with correlated Gaussian random variables. Using the language of mean field theory, the authors define an ordered phase as two distinct signals becoming perfectly correlated asymptotically, and a chaotic phase as two signals losing all correlations. They find that while He init (uncorrelated Gaussians) only results in ordered phases, there exists an initialization with negative correlations that results in a bounded chaotic phase. They suggest that the use of anti-correlated initializations can speed up training.