## 5 Weight Initialization
### Randomness and Variance Analysis Setup  
Consider a network with \(L\) layers. In the analyses below, we use layer indexes, \(i=1,2,\ldots,L\), instead of names. Let \(\overrightarrow{N}_{i}\) denote the number of inputs incoming to each neuron in the \(i\)-th layer during the _forward_ pass--hence, the symbolic left-right arrow. We take into account only layers with tunable weights (dense, convolutional) and assume that the following unified linear expression (one should think here of a standard matrix multiplication for dense layers, and a GEMM-based approach for convolutional layers) (62)  
Figure 10: Fake forward pass leading to a stable numerical behavior due to initial weights drawn from a properly scaled normal distribution with standard deviation \(1/\sqrt{N}\).  
Figure 9: Fake forward pass leading to vanishing signals due to initial weights drawn from a normal distribution with standard deviation scaled by \(10^{-2}\), regardless of \(N\).  
describes the way a layer operates. In other words, multiplication by a transformation matrix \(\overrightarrow{W}_{i}\) with weights suitably arranged takes place:  
\[Y_{i} =\overrightarrow{W}_{i}\cdot X_{i}+W_{0,i}, \tag{62}\] \[X_{i+1} =\phi(Y_{i}). \tag{63}\]  
Rows of \(\overrightarrow{W}_{i}\) and column(s) of \(X_{i}\) are both of length \(\overrightarrow{N}_{i}\), in particular, for convolutional layers \(\overrightarrow{N}_{i}=F_{i}^{2}\cdot C_{i}\), where \(F_{i}\times F_{i}\) represents the filter shape, and \(C_{i}\) the number of input channels. The following randomness-related conditions are satisfied:  
1. Random elements \(w_{i}\) in each \(\overrightarrow{W}_{i}\) are mutually independent and share the same distribution (i.i.d.--independent, identically distributed);
2. Likewise, elements \(x_{i}\) in each \(X_{i}\) are i.i.d.;
3. \(X_{i}\) and \(\overrightarrow{W}_{i}\) are independent of each other;
4. Biases are initialized with zeros.  
Knowing that any element \(y_{i}\) of \(Y_{i}\) in (62) becomes computed as a sum (of products) with \(\overrightarrow{N}_{i}\) mutually independent summands having the same distribution, we have  
\[\text{Var}(y_{i}) =\overrightarrow{N}_{i}\text{Var}(w_{i}x_{i})=\overrightarrow{N} _{i}\Big{(}\mathbb{E}\Big{(}w_{i}^{2}x_{i}^{2}\Big{)}-\mathbb{E}^{2}(w_{i}x_{ i})\Big{)}\] \[=\overrightarrow{N}_{i}\big{(}\mathbb{E}\Big{(}w_{i}^{2}\Big{)} \mathbb{E}\Big{(}x_{i}^{2}\Big{)}-\underbrace{\mathbb{E}^{2}(w_{i})}_{0} \mathbb{E}^{2}(x_{i})\big{)}\] \[=\overrightarrow{N}_{i}\text{Var}(w_{i})\mathbb{E}\Big{(}\phi^{2} (y_{i-1})\Big{)}, \tag{64}\]  
where \(w_{i}\), \(x_{i}\) stand for single elements in \(\overrightarrow{W}_{i}\) and \(X_{i}\). Note that \(\mathbb{E}\big{(}x_{i}^{2}\big{)}\
eq\text{Var}(x_{i})\), unless \(x_{i}\) has a zero mean. For example, for ReLU \(x_{i}=\phi(y_{i-1}){=}[y_{i-1}\!>\!0]y_{i-1}\), thus clearly \(\mathbb{E}(x_{i})\
eq 0\).  
To make use of Equation (64), one needs to transform it into a recurrence, binding \(\text{Var}(y_{i})\) with \(\text{Var}(y_{i-1})\). Both [PERSON] [17] and [PERSON] [29] achieve this, however, via different means. [PERSON] assumes symmetric and approximately linear activation functions \(\phi\) around 0 (true, e.g., for a sigmoid or hyperbolic tangent but not necessarily in general), whereas [PERSON] focuses on convolutional layers activated by ReLU, takes advantage of its properties, and achieves a more elegant result, though restricted specifically to ReLU (or a parametric family of ReLUs).