When your Neural Net doesn’t know: a bayesian approach with Keras | by Marco Cerliani | Towards Data Science

Everyone who has tried to fit a classification model and checked its performance has faced the problem of verifying not only KPI (like accuracy, precision and recall) but also how confident the model is in what it says. The most intuitive instrument to use to verify the reliability of a prediction is one that looks for the probabilities of the various classes.
The higher the probabilities, the higher the confidence. This does not imply higher accuracy. It’s typical to also have misclassifications with high probabilities.
What we do now is to extract the best results from our fitted model, studying the probability distributions and trying to limit mistakes when our neural network is forced to make a decision. We carry out this task in two ways:
a classical study of probabilities on validation data, in order to establish a threshold to avoid misclassifications.
a recent method based on the inference of probabilities from bayesian theories with a ‘permutation’ inside Neural Network framework known as Monte Carlo Dropout, which provides uncertainty estimation almost for free, as shown in this paper.
I found the data for this experiment on Kaggle. I chose a funny dataset containing images of 10 Monkey Species. The dataset consists of two files, training and validation. Each folder contains 10 subfolders labeled as n0~n9, each corresponding a species form Wikipedia’s monkey cladogram.
The images are of good quality and balanced among classes.
We load them with Keras ‘ImageDataGenerator’ performing data augmentation on train. During this process, we store 10% of our train set as validation, this will help us when we’ll try to build thresholds on probabilities following a standard approach. According to the scope of this post, we limit the target classes, only considering the first five species of monkeys.
The neural network structure we want to use is made by simple convolutional layers, max-pooling blocks and dropouts. The last is fundamental to regularize training and will come in handy later when we’ll account for neural network uncertainty with bayesian procedures.
inp = Input(shape=SHAPE+tuple([3]))x = Conv2D(32, (3, 3), activation='relu')(inp)x = MaxPooling2D(pool_size=(2, 2))(x)x = Conv2D(32, (3, 3), activation='relu')(x)x = MaxPooling2D(pool_size=(2, 2))(x)x = Dropout(0.5)(x, training = True)x = Conv2D(64, (3, 3), activation='relu')(x)x = MaxPooling2D(pool_size=(2, 2))(x)x = Conv2D(64, (3, 3), activation='relu')(x)x = MaxPooling2D(pool_size=(2, 2))(x)x = Dropout(0.5)(x, training = True)x = Flatten()(x)x = Dense(512, activation='relu')(x)x = Dropout(0.3)(x, training = True)out = Dense(5, activation='softmax')(x)model = Model(inp, out)model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
After training, accuracy on test is around 0.79, forcing our model to classifies all.
The first approach we introduce is based on simple studies of probabilities computed on a validation set. This procedure enables us to know when our neural network fails and the confidences of mistakes for every class. In this way we create thresholds which we use in conjunction with the final predictions of the model: if the predicted label is below the threshold of the relative class, we refuse to make a prediction.
Our validation is composed of 10% of train images. In order to have an adequate distribution of probabilities to build significative thresholds, we operate data augmentation on validation properly: in the phase of prediction, every image is augmented 100 times, i.e. 100 more probabilities for every sample.
At the end of the prediction step with our augmented data, we have 3 different distributions of scores: Probability scores for every class, probability score of misclassified samples (in each class), probability score of correct classified samples (in each class).
We compute thresholds on the first of the three cited distribution for every class as the 10th percentile. Test images with a predicted probability below the competence threshold are marked as ‘not classified’. Suppressing the ‘not classified’ images (20 in total), accuracy increases from 0.79 to 0.82.
Bayesian probability theory offers mathematically grounded tools to reason about model uncertainty, but these usually come with a prohibitive computational cost. It is surprising that it is possible to cast recent deep learning tools as Bayesian models without changing anything! The solution is the usage of dropout in NNs as a Bayesian approximation.
Specifically, stochastic dropouts are applied after each hidden layer, so the model output can be approximately viewed as a random sample generated from the posterior predictive distribution. As a result, the model uncertainty can be estimated by positional indexes or other statistics taken from predictions in a few repetitions. This procedure is particularly appealing because it is easy to implement, and directly applicable to any existing neural networks without the loss in performances.
Given a new input image, we activate dropout, setting it at 0.5 (turned off by Keras at the end of the training) and compute predictions. When we reactivate dropout we are permuting our neural network structure making also results stochastic. It is clear that if we iterate predictions 100 times for each test sample, we will be able to build a distribution of probabilities for every sample in each class. All can be clarified with some colorful plots.
At this point evaluation is easy... We want the neural network to output a monkey species as a recommendation if out of multiple samples of probability, the median probability for that image is, at the same time, the higher among other medians (red dashed lines in plots above) and at least 0.5 (green dashed line in plots above). Otherwise, we mark this image as ‘not classified’. Suppressing the ‘not classified’ images (16 in total), accuracy increases from 0.79 to 0.83
In this post, we evaluate two different methods which estimate a Neural Network’s confidence. A standard way imposes to hold part of our data as validation in order to study probability distributions and set thresholds. This is a common procedure for every kind of model. Another way suggests applying stochastic dropouts in order to build probabilities distribution and study their differences. Both techniques are useful to avoid misclassification, relaxing our neural network to make a prediction when there’s not so much confidence.
CHECK MY GITHUB REPO
Keep in touch: Linkedin
REFERENCES
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning: Yarin Gal, Zoubin Ghahramani