Dropout at test time in densenet

Gullal: <p>I have fine-tuned the pre-trained densenet121 pytorch model with dropout rate of 0.2.</p>
<p>Now, is there any way I can use dropout while testing an individual image?</p>
<p>The purpose is to pass a single image multiple times through the learned network (with dropout) and calculate mean/variance on the outputs and do further analysis.</p>

smth: <p>you can set your whole network to <code>.eval()</code> mode, but then set your <code>dropout</code> layers to <code>.train()</code> mode.</p>
<p>You can use the <code>apply</code> function to achieve this for example:<br>
<a href="http://pytorch.org/docs/master/nn.html#torch.nn.Module.apply" class="onebox" target="_blank">http://pytorch.org/docs/master/nn.html#torch.nn.Module.apply</a></p>

Gullal: <p>Thanks. apply is a very handy function.</p>

dylanthomas: <p>Related to this, I have another question.</p>
<p>To achieve the same goal – that is to use dropout during testing–<br>
I thought</p>
<ol>
<li>train a net with dropout and then</li>
<li>during testing I could just set the net to .train(mode=True) and get the output  with the same input for multiple runs without updating the network params after each run.</li>
</ol>
<p>But even when I set the net to .train(), I got the same output during multiple runs with a single image.</p>
<p>Can you explain to me why my approach did not work ?</p>

isalirezag: <p>can you provide a simple example on how to do it via <code>apply</code> to have dropout in the testing time?</p>

Gullal: <p>You can do something like this:<br>
Suppose model has dropout layers.</p>
<pre><code class="lang-auto">def apply_dropout(m):
    if type(m) == nn.Dropout:
        m.train()

model.eval()

model.apply(apply_dropout)

</code></pre>

panaali: <p>Or more generaly:</p>
<pre><code class="lang-auto">def enable_dropout(m):
  for m in model.modules():
    if m.__class__.__name__.startswith('Dropout'):
      m.train()
model.eval()
enable_dropout(model)
</code></pre>

sharifza: <p>Clean solution!</p>
<p>A typo edit suggestion:</p>
<pre><code class="lang-auto">def enable_dropout(m):
  for each_module in m.modules():
    if each_module.__class__.__name__.startswith('Dropout'):
      each_module.train()```</code></pre>

rayryeng: <p>Using <code>apply</code> is the recommended approach as setting the whole network to <code>train</code> mode will modify the statistics of any batch normalization layers you have in the network. Thanks for showing how to do it with that method!</p>

Moore: <p>Just to add my two cents:</p>
<p>it only works when you have registered dropout layer during initialization with <code>nn.Dropout</code> layers. In case one uses functional dropout <code>F.dropout(x,training = self.training)</code>  in the <code>forwad()</code> method as <a href="https://github.com/pytorch/vision/blob/e95a3d22590645243fc31fe2ab4a3f2f5f4b1fcf/torchvision/models/densenet.py#L96" rel="noopener nofollow ugc">it is in densenet</a>, such tuning off will not work. The only way to turn on the dropout during evaluation for me currently is to define the <code>forward()</code> method again by replacing the <code>F.dropout(x,training = self.training)</code> with <code>F.dropout(x,training = True)</code>.</p>

Peng_Li1: <p>clear and good solution</p>