Measuring uncertainty using MC Dropout

Ka_Hina: <p>I am trying to implement Bayesian CNN using Mc Dropout on Pytorch,<br>
the main idea is that by applying dropout at test time and running over many forward passes , you get predictions from a variety of different models.<br>
I’ve found an application of the Mc Dropout and I really did not get how they applied this method  and how exactly they did choose the correct prediction from the list of predictions</p>
<p>here is the code</p>
<blockquote>
<p>def mcdropout_test(model):<br>
model.train()<br>
test_loss = 0<br>
correct = 0<br>
T = 100<br>
for data, target in test_loader:<br>
if args.cuda:<br>
data, target = data.cuda(), target.cuda()<br>
data, target = Variable(data, volatile=True), Variable(target)<br>
output_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in xrange(T):<br>
output_list.append(torch.unsqueeze(model(data), 0))<br>
output_mean = torch.cat(output_list, 0).mean(0)<br>
test_loss += F.nll_loss(F.log_softmax(output_mean,dim=1), labels, reduction=‘sum’).data  # sum up batch loss<br>
pred = output_mean.data.max(1, keepdim=True)[1]  # get the index of the max log-probability<br>
correct += pred.eq(target.data.view_as(pred)).cpu().sum()</p>
<pre><code>test_loss /= len(test_loader.dataset)
print('\nMC Dropout Test set: Average loss: {:.4f}, Accuracy: {}/{} ({:.2f}%)\n'.format(
    test_loss, correct, len(test_loader.dataset),
    100. * correct / len(test_loader.dataset)))
</code></pre>
</blockquote>
<blockquote>
<p>train()<br>
mcdropout_test()</p>
</blockquote>
<p>I have replaced</p>
<blockquote>
<p>data, target = Variable(data, volatile=True), Variable(target)</p>
</blockquote>
<p>by adding</p>
<blockquote>
<p>with torch.no_grad()    at the beginning</p>
</blockquote>
<p>:<br>
And this is how I have defined my CNN</p>
<blockquote>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 6, 5)<br>
self.pool = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(6, 16, 5)<br>
self.fc1 = nn.Linear(16 * 5 * 5, 120)<br>
self.fc2 = nn.Linear(120, 84)<br>
self.fc3 = nn.Linear(84, 10)<br>
self.dropout = nn.Dropout(p=0.3)</p>
<pre><code>    nn.init.xavier_uniform_(self.conv1.weight)
    nn.init.constant_(self.conv1.bias, 0.0)
    nn.init.xavier_uniform_(self.conv2.weight)
    nn.init.constant_(self.conv2.bias, 0.0)
    nn.init.xavier_uniform_(self.fc1.weight)
    nn.init.constant_(self.fc1.bias, 0.0)
    nn.init.xavier_uniform_(self.fc2.weight)
    nn.init.constant_(self.fc2.bias, 0.0)
    nn.init.xavier_uniform_(self.fc3.weight)
    nn.init.constant_(self.fc3.bias, 0.0)


def forward(self, x):
    x = self.pool(F.relu(self.dropout(self.conv1(x))))  # recommended to add the relu
    x = self.pool(F.relu(self.dropout(self.conv2(x))))  # recommended to add the relu
    x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(self.dropout(x)))
    x = self.fc3(self.dropout(x)) 
    return x
</code></pre>
</blockquote>
<p>Can anyone help me to get the right implementation of the Monte Carlo Dropout method on CNN?</p>

ptrblck: <p>After 100 iterations, the authors are calculating the mean prediction in:</p>
<pre><code class="lang-python">output_mean = torch.cat(output_list, 0).mean(0)
</code></pre>
<p>and use this tensor to calculate the <code>test_loss</code> as well as the predictions.</p>
<p>PS: You can post code snippets by wrapping the complete code block into three backticks ```, which makes debugging a bit easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Ka_Hina: <p>thank you so much for taking time  debugging the code. I would like to know if this process is applied for each image apart or by taking a batch of images ?</p>

ptrblck: <p>Based on the current code, I assume that the input batch will be used <code>T</code> times.<br>
However, I don’t know, where the mean is calculated, as the code is not formatted properly (i.e. where are <code>output_mean</code> and the following operations used).</p>

Ka_Hina: <p>Actually, the mean is calculated after each T iterations</p>
<pre><code class="lang-auto">for i in xrange(T):
    output_list.append(torch.unsqueeze(model(data), 0))
output_mean = torch.cat(output_list, 0).mean(0)
</code></pre>
<p>but by doing this for each image apart it really takes to much to run, so i was wondering if i could calculate the mean by batches ,will that make it run faster?</p>

ptrblck: <p>Yes, you could use a batched input, if that’s not already the case for <code>data</code>.<br>
If <code>test_loader</code> uses a batch size with is larger than 1, your code would already use batches instead of single samples.</p>

Ka_Hina: <p>I’m sorry for asking so many questions, but I’m a bit confused about this method. I would like to know why they are calculating the mean in:</p>
<pre><code class="lang-auto">output_mean = torch.cat(output_list, 0).mean(0)
</code></pre>
<p>for just the first element of the <code>output_list </code> and considering it as the correct prediction from  all the T tensors found in the <code>output_list </code></p>

ptrblck: <p>The mean is calculated in dim0, not the first element.<br>
Here is a code snippet to show, how this operation works:</p>
<pre><code class="lang-python">batch_size = 2
nb_classes = 3

# initialize empty list
output_list = []

# append predictions to list
for _ in range(10):
    # Here you would call the model and create the outputs
    output_list.append(torch.randn(batch_size, nb_classes))

# Concatenate in dim0 and create tensor
output_list = torch.cat(output_list, 0)
print(output_list.shape) # [20, 3] = [batch_size * 10, nb_classes]

# Calculate mean in dim0 (over all samples)
output_mean = output_list.mean(0)
print(output_mean.shape) # [3]
</code></pre>

Ka_Hina: <p>Now it’s clear ! thank you so much .</p>

vetsa.sai: <aside class="quote quote-modified" data-post="8" data-topic="91753">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/measuring-uncertainty-using-mc-dropout/91753/8">Measuring uncertainty using MC Dropout</a> 
  </div>
  <blockquote>
    The mean is calculated in dim0, not the first element. 
Here is a code snippet to show, how this operation works: 
batch_size = 2
nb_classes = 3

# initialize empty list
output_list = []

# append predictions to list
for _ in range(10):
    # Here you would call the model and create the outputs
    output_list.append(torch.randn(batch_size, nb_classes))

# Concatenate in dim0 and create tensor
output_list = torch.cat(output_list, 0)
print(output_list.shape) # [20, 3] = [batch_size * 10, nb_class…
  </blockquote>
</aside>

<p>Shouldn’t the mean be taken per image [i.e., for 10 samples at once and then again for next 10 samples]?<br>
In that way, output_mean.shape should be [batch_size, nb_classes], here it should be [2,3]. So each value in that tensor gives the mean of 10 class_predictions {since there are 10 MC_Dropout iterations} for one image.</p>