## 5 Weight Initialization
### [PERSON] Initialization  
The assumption that \(\phi\) is approximately linear around 0 implies that \(x_{i}=\phi(y_{i-1})\approx y_{i-1}\) has a mean approximately equal to 0. Therefore \(\mathbb{E}\big{(}\phi^{2}(y_{i-1})\big{)}{\approx}\text{Var}(y_{i-1})\), and by inserting this approximation into (64), the following recurrence is obtained:  
\[\text{Var}(y_{i})=\overrightarrow{N}_{i}\text{Var}(w_{i})\text{Var}(y_{i-1}). \tag{65}\]  
Expanding it over \(L\) layers leads to the following product:  
\[\text{Var}(y_{L})=\text{Var}(y_{1})\prod_{i=2}^{L}\Big{(}\overrightarrow{N}_ {i}\text{Var}(w_{i})\Big{)} \tag{66}\]  
which is crucial to the design of weight initialization. Clearly, an incorrect design may cause the signals to magnify or vanish exponentially. Therefore, as [PERSON] [17] rightly remarks,to keep the information flowing normally during the forward computations, an obvious sufficient condition is to keep all factors of product (66) equal to 1:  
\[\overrightarrow{N}_{i}\text{Var}(w_{i})=1,\quad\forall i. \tag{67}\]  
This suggests that one could draw initial weights, for example, from a zero-mean Gaussian distribution with the standard deviation \(\sqrt{1/\overrightarrow{N}_{i}}\). However, this conclusion is premature because the backward flows of signals must also be taken into account. Analogously to (62) and (63), during the _backward_ pass, we can think of the following operation:  
\[\delta_{X_{i}}=\phi^{\prime}(X_{i})\circ\overleftarrow{W}_{i}\cdot\delta_{Y_{ i}}, \tag{68}\]  
where \(\overleftarrow{W}_{i}\) denotes a suitable backward transformation matrix (e.g., a transposed matrix of weights for dense layers, or a flipped and rearranged matrix of filter weights for convolutional; see, e.g., (54), (39)). This time, \(\overleftarrow{N}_{i}\) stands for the number of \"backward inputs\" incoming to each neuron from the right. For convolutional layers, \(\overleftarrow{N}_{i}=F_{i}^{2}\cdot K_{i}\), where \(K_{i}\) is the number of output channels (kernels). Similarly to before, we look at the variance of any single element, \(\delta_{x_{i}}\), in \(\delta_{X_{i}}\). That variance is equal to the sum of variances of \(\overleftarrow{N}_{i}\) independent summands, yielding  
\[\text{Var}(\delta_{x_{i}})=\overleftarrow{N}_{i}\text{Var}\big{(}\phi^{\prime }(x_{i})w_{i}\delta_{y_{i}}\big{)}=\overleftarrow{N}_{i}\text{Var}(w_{i}) \mathbb{E}\Big{(}(\phi^{\prime}(x_{i})\delta_{y_{i}})^{2}\Big{)} \tag{69}\]  
-- a counterpart of (64). This time, the assumption that \(\phi\) is approximately linear around 0 implies that \(\phi^{\prime}(x_{i})\approx 1\) can be inserted into (69), and the following backward recurrence for the variance of \(\delta\)s is achieved:  
\[\text{Var}(\delta_{x_{i}})=\overleftarrow{N}_{i}\text{Var}(w_{i})\text{Var} \big{(}\delta_{x_{i+1}}\big{)}, \tag{70}\]  
where \(\delta_{x_{i+1}}=\phi^{\prime}(x_{i})\delta_{y_{i}}\) defines the recursive quantity. Expanding the recurrence yields the product  
\[\text{Var}(\delta_{x_{1}})=\text{Var}(\delta_{x_{L}})\prod_{i=2}^{L}\Big{(} \overleftarrow{N}_{L-i+1}\text{Var}(w_{L-i+1})\Big{)}, \tag{71}\]  
which, from the backward computations point of view, suggests the following sufficient condition:  
\[\overleftarrow{N}_{i}\text{Var}(w_{i})=1,\quad\forall i. \tag{72}\]  
Please note that, in general, one is unable to ensure that conditions (67) and (72) are satisfied simultaneously. Therefore, as a compromise, [PERSON] [17] proposed the application of an _average_, namely  
\[\text{Var}(w_{i})=2/(\overrightarrow{N}_{i}+\overleftarrow{N}_{i}),\quad \forall i. \tag{73}\]  
This proposition implies the following two recipes for a random initialization of weights, based on normal and uniform distribution, respectively.