text: 180,1,1017373588033298432,927837253,Emtiyaz Khan,"I will talk about our new work on ""Bayesian deep learning using weight-perturbation in Adam"" at #icml2018 in ""Deep Learning (Bayesian) 2"" session at 4:50pm in room A4. Paper here <LINK> Slides here <LINK> Code here <LINK> 1/6 <LINK> Short summary: Gaussian mean-field variational inference by running Adam on the MLE objective and making the following changes: perturb the weights. Second, add a contribution from the prior, and use a small minibatch size. 2/6 This result is a direct consequence of using natural-gradients instead of gradients. The mean is equal to the parameter returned by Adam, and the variance can be obtained from the scale vector. Perturbation is due to the sampling from variational distribution. 3/6 Small minibatches are due to 'a square of sum of gradients' approximation in Adam for the second-order information. See theorem 1 in the paper. 4/6 <LINK> We also propose VadaGrad and Variational Adaptive Newton (VAN) method for variational optimization (or what @beenwrekt calls Random search). This work is cool because the variance of the search distribution is automatically adapted. Also see <LINK> 5/6 <LINK> Also check out a very similar work by @Guodzh @DavidDuvenaud @RogerGrosse <LINK> They have done some interesting things with KFAC. 6/6 7/6 Tweeting is hard.",https://arxiv.org/abs/1806.04854,"Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms can be implemented within the Adam optimizer by perturbing the network weights during gradient evaluations, and uncertainty estimates can be cheaply obtained by using the vector that adapts the learning rate. This requires lower memory, computation, and implementation effort than existing VI methods, while obtaining uncertainty estimates of comparable quality. Our empirical results confirm this and further suggest that the weight-perturbation in our algorithm could be useful for exploration in reinforcement learning and stochastic optimization. ",Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam,7,"['I will talk about our new work on ""Bayesian deep learning using weight-perturbation in Adam"" at #icml2018 in ""Deep Learning (Bayesian) 2"" session at 4:50pm in room A4. Paper here <LINK> Slides here <LINK> Code here <LINK> 1/6 <LINK>', 'Short summary: Gaussian mean-field variational inference by running Adam on the MLE objective and making the following changes: perturb the weights. Second, add a contribution from the prior, and use a small minibatch size. 2/6', 'This result is a direct consequence of using natural-gradients instead of gradients. The mean is equal to the parameter returned by Adam, and the variance can be obtained from the scale vector. Perturbation is due to the sampling from variational distribution. 3/6', ""Small minibatches are due to 'a square of sum of gradients' approximation in Adam for the second-order information. See theorem 1 in the paper. 4/6 https://t.co/Bff8Grqprd"", 'We also propose VadaGrad and Variational Adaptive Newton (VAN) method for variational optimization (or what @beenwrekt calls Random search). This work is cool because the variance of the search distribution is automatically adapted. Also see https://t.co/0Izj1HsmKI 5/6 https://t.co/axKWYyHgnH', 'Also check out a very similar work by @Guodzh @DavidDuvenaud @RogerGrosse https://t.co/vuPtmVm26o They have done some interesting things with KFAC. 6/6', '7/6 Tweeting is hard.']",18,06,1297