## 5 Weight Initialization  
A cautious initialization of weights in deep networks is of great importance. Its goal is to prevent signals that travel through many layers from exploding or vanishing. Without proper initialization, such problems are likely to occur, and their probability increases with the number of layers. They are commonly referred to as _exploding gradients_ or _vanishing gradients_ to indicate that the learning process no longer converges, but despite the names, the problem pertains not only to the backward flows--it is equally valid for the forward ones. To provide a simple and convincing numerical argument, below, we repeat one after [50].  
Think of a network consisting of 100 dense layers, each with 512 neurons (without activations). Let x be the input vector also of length 512, and suppose its entries are drawn from a standard normal distribution, \(N(0,1)\). We are going to check what happens if initial weights for all layers are also drawn from \(N(0,1)\). The forward pass, represented by the following Python code (Figure 8), consists of 100 consecutive multiplications:  
Figure 8: Forward pass leading to exploding signals: a fake network consisting of 100 dense layers with 512 neurons each (no activations) and intial weights drawn from a standard normal distribution.  
Its execution produces the printout \"nan, nan\" which informs us about the numerically unwanted behavior. In fact, it is easy to check that, after 28 iterations, all elements inside x are at least of order \(10^{35}\) and some of order \(10^{38}\)--close to the maximum non-infinity value allowed for floating-point numbers of single precision. One iteration later, there are already 503 nan values and 9 inf values inside x.  
On the other hand, it is easy to check that weights must not be too small either. The following carefree correction (Figure 9) scales weights by a factor of \(10^{-2}\):  
And the program execution produces the printout \"0.0, 0.0\"--the signals have vanished.  
The mathematical rationale behind the above observations is that, for all layers, any element of the vector resulting from W. dot(x) arises as a sum of 512 statistically independent summands. Therefore, the variance of such an element is equal to the _sum_ of component variances. Note that the fact that each summand in itself is a product (a weight times an input) can be neglected because weights and inputs are mutually independent. Strictly speaking, this is true for the considered simplified example due to zero-mean distributions (for both weights and inputs) and due to the lack of activation functions. More general analysis is provided in the subsequent sections.  
Since each element of the resulting vector arises as a sum of 512 summands, a sensible first guess could be to scale weights by \(1/\sqrt{512}\)--the inverse of the standard deviation for the mentioned sum. More generally, when input sizes differ, a plausible scaling factor seems to be the inverse of the square root of the number of inputs. In the following code (Figure 10) we apply the guess:  
And the obtained printout is \"-0.016219143 0.81295455\"--the signals have neither exploded nor vanished.  
The accurate analysis in subsequent sections shows that the above first guess is not far off from the truth. The analysis follows the works of [PERSON] [17] and [PERSON] [29] that resulted in two state-of-the-art methods for weight initialization.