Using nn.Dropout2d at eval() time (for modelling uncertainty)

mrdbourke: <p>Hello,</p>
<p><strong>Backstory:</strong></p>
<p>I’ve taken some inspiration from this post on the fast.ai forums:</p>
<aside class="onebox discoursetopic" data-onebox-src="https://forums.fast.ai/t/uncertainty-in-deep-learning-bayesian-networks-gaussian-processes/5551">
  <header class="source">
      <img src="https://forums.fast.ai/uploads/default/optimized/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5_2_32x32.png" class="site-icon" width="" height="">

      <a href="https://forums.fast.ai/t/uncertainty-in-deep-learning-bayesian-networks-gaussian-processes/5551" target="_blank" rel="noopener nofollow ugc" title="06:54AM - 18 September 2017">fast.ai Course Forums – 18 Sep 17</a>
  </header>

  <article class="onebox-body">
    <img src="https://forums.fast.ai/uploads/default/original/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5.png" class="thumbnail" width="" height="">

<div class="title-wrapper">
  <h3><a href="https://forums.fast.ai/t/uncertainty-in-deep-learning-bayesian-networks-gaussian-processes/5551" target="_blank" rel="noopener nofollow ugc">Uncertainty in Deep Learning (Bayesian networks, Gaussian processes)</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #AB9364;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Deep Learning</span>
        </span>
      </span>
  </div>
</div>

  <p>I am really enjoying reading Yarin Gal’s work drawing connections between deep learning techniques and Bayesian Inference / Gaussian Processes.  Particularly fascinating is the idea of producing useful uncertainty metrics from deep neural networks...</p>

  <p>
    <span class="label1">Reading time: 1 mins 🕑</span>
      <span class="label2">Likes: 20 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>to build in dropout at evaluation time as a way of attempting to measure the uncertainty of a prediction.</p>
<p>I also used this post as a basis for <code>.apply()</code>-ing a function at <code>.eval()</code> time:</p>
<aside class="quote" data-post="1" data-topic="6738">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gullal/48/12135_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dropout-at-test-time-in-densenet/6738">Dropout at test time in densenet</a> 
  </div>
  <blockquote>
    I have fine-tuned the pre-trained densenet121 pytorch model with dropout rate of 0.2. 
Now, is there any way I can use dropout while testing an individual image? 
The purpose is to pass a single image multiple times through the learned network (with dropout) and calculate mean/variance on the outputs and do further analysis.
  </blockquote>
</aside>

<p><strong>The way I understand these techniques:</strong></p>
<p>By applying dropout at evaluation time and running over many forward passes (10-100+), you get predictions from a variety of different models.</p>
<p>What you can then do with these predictions is measure how much they differ (get the <code>.var()</code> of your 100 different samples).</p>
<p>With this difference, you can then see what samples the model is ‘uncertain’ about (the ones with high variance).</p>
<p><strong>Use case example:</strong></p>
<p>Uber seems to be using this technique for some of their predictions: <a href="https://eng.uber.com/tag/monte-carlo-dropout/" rel="noopener nofollow ugc">https://eng.uber.com/tag/monte-carlo-dropout/</a></p>
<p><strong>My main question (more of a sound check…):</strong></p>
<p>I’ve put together an example pipeline using MNIST but I’m unsure of some of the custom functions I’ve created/taken from code examples online.</p>
<p>Has anyone had experience with Monte Carlo Dropout or another method of measuring uncertainty they can share?</p>
<p><strong>My code (critiques/advice welcome):</strong></p>
<pre><code class="lang-auto"># Create function to apply to model at eval() time
def apply_dropout(m):
    if type(m) == nn.Dropout2d:
        m.train()

# Func to predict MNIST class
def predict_class(model, X):
    model = model.eval()
    model.apply(apply_dropout) # apply dropout at pred time (see func above)
    outputs = model(Variable(X))
    #print(outputs)
    _, pred = torch.max(outputs.data, 1)
    return pred.numpy()

# Run for T times and get list_of_preds for measuring variance
def predict(model, X, T=100):
    list_of_preds = []
    standard_pred=predict_class(model, X)
    y1 = []
    y2 = []
    for _ in range(T):
        _y1 = model(Variable(X))
        _y2 = F.softmax(_y1, dim=1)
        y1.append(_y1.data.numpy())
        y2.append(_y2.data.numpy())
        list_of_preds.append(predict_class(model, X)) # predict T times
    return standard_pred, np.array(y1), np.array(y2), np.array(list_of_preds)
</code></pre>

LMA: <p>Your code looks fine to me. I tried a similar implementation on a different data set. I did see some variation in the prediction for in-class samples, but virtually nothing in the prediction for out-of-class samples. My network also used batch norm. However, when I put the whole <code>model</code> in the training mode, I observed significant variations in the predictions for both in-class samples and out-of-class samples, but the prediction accuracy for in-class samples were very low. What was your observation from your experiments – even if just on in-class samples will be interesting to me?</p>

rzhang63: <p>I think you can just call model.train() (instead of model.eval()) everytime you want to leave dropout turned on in testing time. This will produce different predicted probabilities each time even with the same input image since dropout is working.</p>

oezguensi: <p>I think it is better to use the above shown approach with model.eval() as eval() not only affects Dropout.<br>
As <code>model.apply()</code> uses <code>children()</code> to iterate through the models, it may not affect nested layers.<br>
Therefore it is better to use <code>modules()</code>. Also, if you like to set all Dropout layers and not only <code>nn.Dropout2d</code> to training mode you can use:</p>
<pre><code class="lang-auto">def enable_eval_dropout(model):
	for module in model.modules():
		if 'Dropout' in type(module).__name__:
			module.train()
</code></pre>

Zhou_Peiyan: <p>Hey, thank you for the insights! I was wondering if you were able to apply Monte Carlo Dropout to a pre-trained faster rcnn model, instead of your custom model?</p>