Bayesian Neural Networks: 1 Why Bother? | by Adam Woolf | Towards Data Science

This is the first chapter in a series on Bayesian Deep Learning. The next chapter is available here while the third is here.
I’m often asked to explain my quirky interest in Bayesian Neural Networks. So this is the start of a series about them. Therefore it’s safe for me to say this is the start of a series about the future of deep learning — a future that’s available right now. Over the weeks you’ll learn how to use one of the new libraries built for the Bayesian-type neural network. I’ll include code and discuss work with both TensorFlow-Probability and Pytorches Pyro including the installation, training and prediction with various different network architectures. In this chapter we’ll cover the following objectives:
• Learn why Bayesian Neural networks are so useful and exciting • Understand how they’re different from normal neural networks • Appreciate how the uncertainty metrics you can obtain are a major advantage • Install a library to start work with all things random
Many developers of deep learning today didn’t begin with a computer-science or academic background. Trial and error, patience with a book and some blogs go a long way. And unsurprisingly Bayesian deep learning doesn’t require anything different.
But the advantage of some basic theoretical knowledge is greater than usual. Reassuringly this can be compressed, expressed and understood fairly simply. Primarily, it’s the unusual terminology that causes confusion. So we’ll have a talk about them with a few chips of code before getting stuck in.
We’re interested in random variables. That means most our variables change values whenever they’re requested. So informally random variables are slippery things (and more formally we call them ‘stochastic variables’, but who benefits from formality?). Working with either TensorFlow-Probability (TFP) or Pytorches Pyro you can call:
rv_normal = dist.Normal(loc=0., scale=3.)rv_normal.sample([1])
This will return a single value which will be different every time we run the second line. That’s the case even when we don’t re-run the first line, because rv_normal is a random variable defined by the normal (gaussian) distribution.
You might therefore plug these random variables into a neural network and use them in the place of the point values we normally use for weights. Why’d you want to bother? Because after the network is trained each prediction we make will be with different random variable weights. Therefore, the whole model will be different. Yet — before I loose you, the importance of our neural networks built with random variables is the variables are constrained as though on a dog leash to only return values over a certain range. That’s because our neural network training has trained our normal distribution to be placed with its bell shaped peak around the very best value.
Three points come up from all this: • The distributions own the random variables that return our random weight values every time we make a prediction.  • Our neural network training moves the a whole distribution, not just a point weight value. • This is similar to the process of using a distribution to initialise weights in a conventional neural network (for the starting values for the weights). Except we carry on using the distribution throughout training and for prediction, not just for initialisation.
Training the distributions is more formally called conditioning the distributions (similar in meaning to Pavlov conditioning his dogs. (I can neither escape formality nor dogs it seems)). So all these fiddly random variables and distributions add up to one thing.., different predictions. But if we’ve trained the Bayesian Neural network well our predictions will be very similar (for continuous, floating point, regression outputs) and the same for (categorical outputs.). And if we train our neural network badly we’ll get very different results each time. If the Bayesian network could speak it would say the difference between it’s predictions being similar or its predictions being very different is its uncertainty. Think of uncertainty as confidence (only Bayesian neural networks don’t like to talk of their confidence so call it something different). If we’ve built a good network architecture, and most importantly if we’ve provided good data, the neural network will give confident predictions, and if everything we do is bad we’ll get unconfident predictions. In Bayesian wizardry terms the neural network will be more uncertain when we give bad and less uncertain when we give good. The wizardry difference in terms because Bayesian neural networks are never certain or anything, they’re only less uncertain. While our conventional neural networks are always certain of everything! You’ve may seen something similar already when you first began working with neural networks. You finished training your first model and have moderate performance, then you train another without making any changes and still have moderately good performance on average, only you’re getting entirely different results returned. Bayesian neural networks are like that, only instead of relying on your frustration to keep training neural networks with minor differences, with a Bayesian approach you train once and have an infinite number of models built-in.
Wonderful. Applause. So? So... we can switch that all around and see how suitable our data (predominantly) is for the task by looking at the variability of the results. But much much better than that with a Bayesian model we’re usually able to train a model with less data than we’d conventionally need for a neural network. The Bayesian model will generalise well. Its also less troubled by imbalances in the amount of data for each training class — a much neglected reason for poor performance.
Enough of the chatter my good man, show us how to do-something. Right-oh.
First you’ll need to install one of the two main probabilistic-programming libraries to follow the series. While you can build Bayesian neural networks with just the core libraries you’ll need to implement the training code manually which isn’t shy on math and hidden errors. So we’ll use either Pyro (built on top of Pytorch) or TensorFlow-Probability (perhaps obviously built on top of TensorFlow)1. If you have experience with one of the core libraries it’s probably best to continue with it specifically and benefit from your familiarity rather than changing over. Scroll bellow to one of the relevant library sections you prefer, or if you’re a masochist go ahead and implement them both then email me3.
We’ll be working with TensorFlow 2 as a base library which has some similar implementations to PyTorch, though benefits from many features we’ll use in later chapters2. TensorFlow in general also benefits from language and platform portability. You can call a model written in Python TensorFlow from almost any language from Java to C# and deploy models quite easily to mobile devices.
TensorFlow and TensorFlow-Probability are easy to install. Ensure you have Python installed (Python 3, if you haven’t this or any Python distribution I strongly recommend downloading the Anaconda distribution from: www.anaconda.com which contains everything you need to get going here on in.)
If you want to use Anaconda/Miniconda for installation of TensorFlow then use the conda package manager and type:
conda install tensorflow
or for GPU support:
conda install tensorflow-gpu
At the time of writing there are some benefits to installing with conda rather than pip if you want GPU support. The conda version automatically installs math libraries that significantly improve CPU training performance. GPU installation (tensorflow-gpu) is also easier with conda as by default it installs CUDA and cuDNN which require independent installation with pip. GPU versoins require a CUDA (i.e. NVIDIA) GPU on the machine. However conda doesn’t contain as many different libraries as pip does and breaks easily when using pip along side it.
To install the compute (non-GPU) version with pip:
pip install — upgrade tensorflow==2.*
or if you want to use a GPU version with pip:
pip install — upgrade tensorflow-gpu==2.*)
If you’re unsure choose the first one. The 2.* part ensures installation of the latest 2 version. The series can mostly be worked through with TensorFlow versions >1.13 but requires activation of eager execution (activated by default in TF2) with: tf.enable_eager_execution() after importing TensorFlow in addition to some other changes in latter chapters of the series.
Note: you must install the core TensorFlow library before installing TensorFlow-Probability and surprisingly TensorFlow-Probability installation wont install the core library for you as a dependency.
Install TFP via conda with:
conda install -c conda-forge tensorflow-probability
and with pip:
pip install — upgrade tensorflow-probability
Then to check that everythings working create a new Jupyter Notebook, IPython instance or a Python script and add:
import tensorflow as tfimport tensorflow_probability as tfpdist = tfp.distributionsrv_normal = dist.Normal(loc=0., scale=3.)print(rv_normal.sample([1]))
You’ll recognise the final two lines from the start of the chapter. You’re sampling a random variable so as output you should get a tensor containing a different random value every time you execute the final line.
[Out]: tf.Tensor([2.2621622], shape=(1,), dtype=float32)
For Pyro you first require a PyTorch installation. Typically this can be easiest performed through conda with:
conda install pytorch-cpu torchvision-cpu -c pytorch
or if a NIVIDA GPU is available for you to use:
conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
Pip installation is also possible but unfortunately differs significantly by platform and by CUDA version therefore it’s recommended you refer to the installation guide at pytorch.org if you only want to use pip.
Pyro is then installed with:
conda install -c gwerbin pyro-ppl
or
pip install pyro-ppl 
Note of caution: The package ‘pyro’ (without the -ppl) is entirely different software and unrelated to Pytorch-Pyro.
To get going with Pyro create a Jupyter notebook, Ipython console or Python script and type:
import torchimport pyroimport pyro.distributions as distrv_normal = dist.Normal(loc=0., scale=3.)print(rv_normal.sample([1]))
You’ll recognise the final two lines from the start of the chapter. You’re sampling a random variable so as output you should get a tensor containing a different random value every time you execute the final line.
[Out]: tensor([0.0514])
You’ve learnt about the relationship between a random variable and a distribution and seen it in action. You implemented a normal distribution located at 0 with a scale of 3. The location is another word for the distributions mean, while the scale is another word for the standard deviation. We’ll stick with the terms location and scale however as these terms help us to be consistent when working with other (non-normal) distributions.
Next week we’ll learn how to implement a Bayesian neural network and find out a bit more about where these distributions and random variables fit in. I bet you can’t wait! Subscribe to the blog for notifications about the next release.
[1]: You might have heard of PyMC3 too another good library that you can build probabilistic models with. However PyMC3 leans specifically towards statistical modelling in general and isn’t as friendly to people working with neural networks (Autograd and GPU support more challenging for instance).
[2]: TensorFlow 2 for instance removes the ’session’ context needed to setup graphs and instead works with functions.
[3]: If you can’t choose which library to use you’ll find TensorFlow-Probability is considerably simpler and easier than Pyro to both use and understand. However, that said documentation for Pyro is excellent while it’s lighter on explanation for TFP from the perspective of neural networks.