text: 467,167,1387929898099134473,2800204849,Andrew Gordon Wilson,"What are Bayesian neural network posteriors really like? With high fidelity HMC, we study approximate inference quality, generalization, cold posteriors, priors, and more. <LINK> With @Pavel_Izmailov, @sharadvikram, and Matthew D. Hoffman. 1/10 <LINK> We show that Bayesian neural networks reassuringly provide good generalization, outperforming deep ensembles, standard training, and many approximate inference procedures, even with a single chain. 2/10 <LINK> However, we find that BNNs are surprisingly poor at OOD generalization, even worse than SGD, despite the popularity of approximate inference in this setting, and the relatively good performance of BNNs for OOD detection. 3/10 <LINK> Even though deep ensembles are often talked about as a ""non-Bayesian"" alternative to standard approximate inference, we find they approximate the HMC predictive distribution better than MFVI, and about as well as standard SGLD. 4/10 <LINK> There has been much attention lately on ""cold posteriors"" in BDL, where the posterior raised to a power 1/T with T&lt;1 can lead to better results. We see little evidence for a general cold posterior effect, which we find is largely due to data augmentation. 5/10 <LINK> We explored Gaussian, mixture of Gaussian, and heavy-tailed logistic priors, which performed similarly, although the heavy-tailed priors did slightly better. We also found performance relatively insensitive to the scale of the Gaussian prior... 6/10 <LINK> ...these results highlight the relative importance of the architecture compared to the distribution over weights in defining the induced prior over functions. Indeed, other work shows that even standard Gaussian priors have many useful properties: <LINK>. 7/10 <LINK> We present many other results, including mixing in function space vs. weight space, posterior geometry and mode connecting paths, single chain vs. multi-chain...! 8/10 <LINK> Many of the results, both positive and negative for BDL, are contrary to conventional wisdom. 9/10 <LINK> We worked hard to obtain these HMC samples, which we plan to release as a public resource, as a reference for evaluating more practical alternatives to HMC, and for researchers to explore their own questions around approximate inference in BDL. 10/10",https://arxiv.org/abs/2104.14421,"The posterior over Bayesian neural network (BNN) parameters is extremely high-dimensional and non-convex. For computational reasons, researchers approximate this posterior using inexpensive mini-batch methods such as mean-field variational inference or stochastic-gradient Markov chain Monte Carlo (SGMCMC). To investigate foundational questions in Bayesian deep learning, we instead use full-batch Hamiltonian Monte Carlo (HMC) on modern architectures. We show that (1) BNNs can achieve significant performance gains over standard training and deep ensembles; (2) a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains; (3) in contrast to recent studies, we find posterior tempering is not needed for near-optimal performance, with little evidence for a ""cold posterior"" effect, which we show is largely an artifact of data augmentation; (4) BMA performance is robust to the choice of prior scale, and relatively similar for diagonal Gaussian, mixture of Gaussian, and logistic priors; (5) Bayesian neural networks show surprisingly poor generalization under domain shift; (6) while cheaper alternatives such as deep ensembles and SGMCMC methods can provide good generalization, they provide distinct predictive distributions from HMC. Notably, deep ensemble predictive distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference. ",What Are Bayesian Neural Network Posteriors Really Like?,10,"['What are Bayesian neural network posteriors really like? With high fidelity HMC, we study approximate inference quality, generalization, cold posteriors, priors, and more. \n<LINK>\nWith @Pavel_Izmailov, @sharadvikram, and Matthew D. Hoffman. 1/10 <LINK>', 'We show that Bayesian neural networks reassuringly provide good generalization, outperforming deep ensembles, standard training, and many approximate inference procedures, even with a single chain. 2/10 https://t.co/sSeW6NGSRX', 'However, we find that BNNs are surprisingly poor at OOD generalization, even worse than SGD, despite the popularity of approximate inference in this setting, and the relatively good performance of BNNs for OOD detection. 3/10 https://t.co/7EYqt1JIDe', 'Even though deep ensembles are often talked about as a ""non-Bayesian"" alternative to standard approximate inference, we find they approximate the HMC predictive distribution better than MFVI, and about as well as standard SGLD. 4/10 https://t.co/Wmbp4rFCnJ', 'There has been much attention lately on ""cold posteriors"" in BDL, where the posterior raised to a power 1/T with T&lt;1 can lead to better results. We see little evidence for a general cold posterior effect, which we find is largely due to data augmentation. 5/10 https://t.co/eXkRopCpRs', 'We explored Gaussian, mixture of Gaussian, and heavy-tailed logistic priors, which performed similarly, although the heavy-tailed priors did slightly better. We also found performance relatively insensitive to the scale of the Gaussian prior... 6/10 https://t.co/REonWDwHvJ', '...these results highlight the relative importance of the architecture compared to the distribution over weights in defining the induced prior over functions. Indeed, other work shows that even standard Gaussian priors have many useful properties: https://t.co/midasGNPYn. 7/10 https://t.co/yrDjvkO20m', 'We present many other results, including mixing in function space vs. weight space, posterior geometry and mode connecting paths, single chain vs. multi-chain...!  8/10 https://t.co/efyU00gjIz', 'Many of the results, both positive and negative for BDL, are contrary to conventional wisdom. 9/10 https://t.co/8naizY9p35', 'We worked hard to obtain these HMC samples, which we plan to release as a public resource, as a reference for evaluating more practical alternatives to HMC, and for researchers to explore their own questions around  approximate inference in BDL. 10/10']",21,04,2262