text: 224,1,1104075634593095681,4870078413,Sam Schoenholz,"1/3 Our new paper analyzing batch normalization in neural networks at initialization is out (and will be at @iclr2019). We find that batch norm + MLPs always feature exploding gradients for any choice of nonlinearity and batch size. <LINK> <LINK> 2/3 We propose several schemes to ameliorate this by careful parameter tuning. The formalism here also opens the door to performing Bayesian inference on Gaussian Process that correspond to neural networks with batch normalization. 3/3 As always, this was a really fun collaboration with @TheGregYang, Jeffrey Pennington, @vinaysrao, @jaschasd.",https://arxiv.org/abs/1902.08129,"We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization. Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function. Indeed, batch normalization itself is the cause of gradient explosion. As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations. While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections. Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range. Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest. ",A Mean Field Theory of Batch Normalization,3,"['1/3 Our new paper analyzing batch normalization in neural networks at initialization is out (and will be at @iclr2019). We find that batch norm + MLPs always feature exploding gradients for any choice of nonlinearity and batch size. <LINK> <LINK>', '2/3 We propose several schemes to ameliorate this by careful parameter tuning. The formalism here also opens the door to performing Bayesian inference on Gaussian Process that correspond to neural networks with batch normalization.', '3/3 As always, this was a really fun collaboration with @TheGregYang, Jeffrey Pennington, @vinaysrao, @jaschasd.']",19,02,591