## 5 Weight Initialization
### He Initialization  
With the ReLU function in mind, [29] also studied the variance in signals described by (64) and noticed that the following interesting property holds:  
\[\mathbb{E}\Big{(}x_{i}^{2}\Big{)} =\mathbb{E}\Big{(}([y_{i-1}>0]y_{i-1})^{2}\Big{)}\] \[=\int_{0}^{\infty}y_{i-1}^{2}\,p_{i-1}(y_{i-1})\,dy_{i-1}=\frac{1} {2}\text{Var}(y_{i-1}). \tag{76}\]  
The last equality is true because the distribution of \(y_{i-1}\) is symmetric at around 0 due to the symmetric distribution of \(\overrightarrow{W}_{i-1}\). Thus, (64) can be transformed into the following elegant recurrence:  
\[\text{Var}(y_{i})=\frac{1}{2}\overrightarrow{N}_{i}\text{Var}(w_{i})\text{ Var}(y_{i-1}), \tag{77}\]  
which, expanded over \(L\) layers, yields the product  
\[\text{Var}(y_{L})=\text{Var}(y_{1})\prod_{i=2}^{L}\biggl{(}\frac{1}{2} \overrightarrow{N}_{i}\text{Var}(w_{i})\biggr{)}. \tag{78}\]  
Therefore, a sufficient condition to avoid the product from growing or vanishing exponentially is  
\[\frac{1}{2}\overrightarrow{N}_{i}\text{Var}(w_{i})=1,\quad\forall i. \tag{79}\]  
Note the presence of factor \(1/2\) in contrast to (67). In fact, for the first layer (\(i=1\)), one could have \(N_{1}\text{Var}(w_{1})=1\) because there is no ReLU applied to the input signals. But a redundant factor, \(1/2\), existing only on one layer does not matter; thus, He [29] keeps \"for all \(i\)\" in (79) for simplicity. The derived condition suggests that one could draw initial weights from a zero-mean Gaussian distribution with standard deviation \(\sqrt{2/\overrightarrow{N}_{i}}\), or a uniform distribution, \(U(-\sqrt{6/\overrightarrow{N}_{i}},\sqrt{6/\overrightarrow{N}_{i}})\). However, similarly as before, to make the analysis complete, one has to take backward computations into account as well.  
The backward recurrence of interest includes the \(1/2\) factor:  
\[\text{Var}(\delta_{x_{i}})=\frac{1}{2}\overline{N}_{i}\text{Var}(w_{i})\text {Var}\bigl{(}\delta_{x_{i+1}}\bigr{)}, \tag{80}\]  
and, when expanded, implies the product  
\[\text{Var}(\delta_{x_{1}})=\text{Var}(\delta_{x_{L}})\prod_{i=2}^{L}\biggl{(} \frac{1}{2}\overleftarrow{N}_{L-i+1}\text{Var}(w_{L-i+1})\biggr{)} \tag{81}\]  
and the following sufficient condition for the backward flow:  
\[\frac{1}{2}\overleftarrow{N}_{i}\text{Var}(w_{i})=1,\quad\forall i. \tag{82}\]  
Instead of looking for a compromise between (79) and (82), [29] noticed something more interesting--namely, for convolutional layers, it is sufficient to apply (79) _or_ (82) alone. Recall that \(\overrightarrow{N}_{i}=F_{i}^{2}\cdot C_{i}\) and \(\overleftarrow{N}_{i}=F_{i}^{2}\cdot K_{i}\). Therefore, if, for example, (82) is applied in (78), then the product becomes  
\[\text{Var}(y_{1})\prod_{i=2}^{L}\biggl{(}\frac{1}{2}\overrightarrow{N}_{i} \text{Var}(w_{i})\biggr{)}=\text{Var}(y_{1})\prod_{i=2}^{L}\frac{\overrightarrow {N}_{i}}{\overleftarrow{N}_{i}}=\text{Var}(y_{1})\prod_{i=2}^{L}\frac{F_{i}^ {2}C_{i}}{F_{i}^{2}K_{i}}=\text{Var}(y_{1})C_{2}/K_{L}. \tag{83}\]  
The last pass is true, as successive ratios cancel out, owing to \(C_{i+1}=K_{i}\) for all \(i\geqslant 2\). The final constant means that, if weight initialization properly scales the forward signals in theforward flow, then it simultaneously does so in the backward flow, or vice versa. Most libraries pick (79) and offer the following initialization for convolutional layers activated via ReLUs.  
**He normal initialization:**:  
\[w_{i}\sim N\!\!\left(0,\sqrt{2/\overrightarrow{N}_{i}}\right)=N\!\!\left(0, \sqrt{2/(F_{i}^{2}C_{i})}\right), \tag{84}\]
**He uniform initialization:**:  
\[w_{i}\sim U\!\!\left(\!-\sqrt{6/\overrightarrow{N}_{i}},\sqrt{6/ \overrightarrow{N}_{i}}\right)=U\!\!\left(\!-\sqrt{6/(F_{i}^{2}C_{i})},\sqrt{ 6/(F_{i}^{2}C_{i})}\right)\!. \tag{85}\]