Implementing a Bayesian CNN in PyTorch

username: blessen
cooked: <p>I wanted to implement a bayesian CNN from scratch using the PyTorch framework. I searched for the implementation of dropout in order to write a custom modification. I searched for the function given in the definition of dropout3d and could not find the apply() function?<br>
Could anyone help me? Also I’m new to this forum. Thanks.</p>
<p>Link:<a href="http://pytorch.org/docs/master/_modules/torch/nn/functional.html#dropout3d" rel="nofollow noopener">http://pytorch.org/docs/master/_modules/torch/nn/functional.html#dropout3d</a></p>
<p>def dropout3d(input, p=0.5, training=False, inplace=False):<br>
return _functions.dropout.FeatureDropout.apply(input, p, training, inplace)</p>

username: jpeg729
cooked: <p><code>dropout3d</code> calls a method of <a href="https://github.com/pytorch/pytorch/blob/9f980b179506489d226051259eacec8639c00c5f/torch/nn/_functions/dropout.py#L55">_functions.dropout.FeatureDropout</a> which inherits from <a href="https://github.com/pytorch/pytorch/blob/9f980b179506489d226051259eacec8639c00c5f/torch/nn/_functions/dropout.py#L7">Dropout</a> which has a <code>forward</code> method.</p>
<p>According to the <a href="http://pytorch.org/docs/master/notes/extending.html">docs on extending PyTorch</a> you implement a custom function by creating a class with a <code>forward</code> method, and you use it by calling the <code>apply</code> method. Therefore, when <code>dropout3d</code> calls _functions.dropout.FeatureDropout.apply, we can assume that <code>Dropout.forward</code> gets run.</p>
<p>Therefore to write a custom version of dropout3d, you could copy the Dropout class from the above link and modify its <code>forward</code> method.</p>

username: blessen
cooked: <p>That’s right. But the thing is I want to implement dropout after each kernel filter. Currently the dropout layer works over the entire previous layer, but according to the paper "Bayesian CNN with approximate bernouilli variational inference, we’ll need to implement dropout after each filter during convolution.</p>
<p>So for that purpose I’ll need to modify the dropout implementation itself which is why I wanted to know where the apply() function would be available in order to change that.</p>
<p>Thanks!</p>

username: jpeg729
cooked: <p>I thought that each filter of a CNN produced its own channel in the output. Which means that doing dropout after each individual filter is equivalent to doing dropout after all of the filters.</p>
<p>The paper itself seems to support my interpretation…<br>
From page 2 of that paper…</p>
<blockquote>
<p>Our model is implemented by performing dropout after convolution layers.</p>
</blockquote>
<p>This sentence appears as quoted here with no mention of any specificity in implementation of the dropout.</p>
<p>In section 2.2 the description of the dropout they use seems pretty standard to me.</p>
<p>From page 5</p>
<blockquote>
<p>Implementing our Bayesian CNN is therefore as simple as using dropout after every convolution layer before pooling.</p>
</blockquote>
<p>Again, no mention of any particular need to implement a custom dropout for this case.</p>
<p>As far as I can tell, model training is done as usual, but they say that predictions at test time should be produced using “Monte-Carlo dropout” which I haven’t yet managed to understand. But then again Bayesian stuff isn’t my thing and maybe I have understood it all backwards.</p>

username: blessen
cooked: <p>Thanks but the following is taken from the paper in section 5:</p>
<p>“In existing literature, however, dropout is used in CNNs only after inner-product layers – equivalent<br>
to approximately integrating these alone. Here we wish to integrate over the kernels of the CNN as<br>
well. Thus implementing a Bayesian CNN we apply dropout after all convolution layers as well as<br>
inner-product layers.”</p>
<p>As stated, integration is done over the kernel of the CNN. Do I have the right idea? If so, how would I go about implementing?</p>

username: jpeg729
cooked: <p>I understood that to mean that dropout is usually applied only after fully connected layers but not directly after CNN layers because test error increases in that case.</p>

username: jpeg729
cooked: <p>As I understand it, the output of each kernel of a CNN is equivalent to exactly one channel of the CNN layer output. The paper wants to integrate over the kernels of the CNN, and they suggest using dropout as an approximate method. So I suppose you could loop over the channels of the output applying dropout to each channel individually - that would apply dropout to each kernel of the CNN individually, but that approach is mathematically equivalent to applying dropout to the entire layer output in one go. The last but one paragraph of section 5 that explains how to apply dropout to the CNN layer seems to support my interpretation.</p>
<p>This is my summary of the paper as I understand it. I could be wrong, but I am fairly sure I have understood it correctly.</p>
<p>In training apply standard dropout after every CNN and fully connected layer of the network. Train with the usual softmax loss combined with L2 regularisation.<br>
When making predictions at test time, don’t turn off dropout as we usually do, instead make several runs through the network with dropout turned on, thus collecting several different predictions for the same input. The average of the predictions can be used as the predicted value. The spread of the predictions can be used as a measure of uncertainty. This is what they call Monte-Carlo dropout.<br>
The number of runs needed for optimal test performance must be determined by testing. Their tests suggest that 20 runs through the network generally produces a significant decrease in test error, and that more than 100 runs is generally unnecessary.<br>
The prediction runs can be efficiently run in parallel by making a batch containing the same image repeated many times.</p>

username: Ismail_Elezi
cooked: <p>Isn’t Monte-Carlo dropout just applying dropout in forward pass during the inference time? Something like for each object, run the algorithm n times while applying dropout, and then averaging the results.</p>

username: jpeg729
cooked: <p>I think that is basically what I said.</p>

username: Ismail_Elezi
cooked: <p>Yep. Sorry, missed your last post when you said that, and replied on your earlier post.</p>

username: MERAH_Samia
cooked: <p>Hi!<br>
What libraries I could use to implement Bayesian CNN?</p>

username: ptrblck
cooked: <p>I’m not sure what Bayesian CNNs are, but have a look at <a href="https://botorch.org/">BoTorch</a>, which provides a library for Bayesian Optimization. Would that fit your use case or are you looking for some specific model architectures?</p>
<p>EDIT:<br>
Seems to have been answered in <a href="https://discuss.pytorch.org/t/implementing-bayesian-cnn/88689/2">the double post</a>.</p>

username: Shisho_Sama
cooked: <p>This is a great recommendation. Thanks a lot!</p>

username: MERAH_Samia
cooked: <p>Hi !<br>
I am searching dor an approach to implement Bayesian Deep learning, i found two methode either by bayes by backprop or by dropout, I’ve read that Optimising any neural network with dropout is equivalent to a form of approximate Bayesian inference and a network trained with dropout already is a Bayesian neural network,<br>
Could you please confirm these statements if you have any idea ?<br>
Thank youu for your time,</p>

username: tom
cooked: <p>One of the results on dropout as a approximate Bayesian inference is Gal and Ghahramani. Yarin Gal has a <a href="http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa532c1ce.html" rel="nofollow noopener">blog post</a> on it with interactive demo and all.</p>