Inject dropout into resnet (or any other network)

fiendfish: <p>So i want to inject dropout into a (pretrained) resnet, as i get pretty bad over-fitting. (for example add a dropout layer after each residual step)</p>
<p>I guess that i could simply monkey patch the resnet base class. However i wonder if there is easy way to achieve this?</p>

ptrblck: <p>I think the cleanest approach would be to create your custom resnet class by deriving from the <code>torchvision</code> implementation and overriding the <a href="https://github.com/pytorch/vision/blob/61763fa955ef74077a1d3e1aa5da36f7c606943a/torchvision/models/resnet.py#L197-L213"><code>forward</code> method</a>.</p>

fiendfish: <p>With Res net in particular the derivation is even easier as the actual layers are already implemented in a separate class that gets passed to a basic resnet.</p>
<p>I was hopping that there is a general approach, that i could apply to multiple models. The standard models typically don’t contain drop out, as they are usually trained with big datasets. But for small Training datasets which are pretty common in practice, dropout helps a lot. So having a function that would adds dropout before/after each relu would be very useful.</p>
<p>model_with_dropout = add_dropout(model, after=“relu”)</p>

ptrblck: <p>Alternatively to my proposed approach you could also use forward hooks and add dropout at some layers. I’m not sure, if it’s the best approach, but it might be similar to your mentioned approach.</p>

crypdick: <p>Wouldn’t that also make the network apply dropout during testing/inference? Edit: maybe adding <a href="https://discuss.pytorch.org/t/how-do-you-create-a-layer-that-behaves-differently-during-training-and-evaluation-like-dropout/38675/2">something like this</a>.</p>

ptrblck: <p>It depends of course on your implementation, but you could use the linked <code>training</code> argument as:</p>
<pre><code class="lang-python">model = models.resnet18()
model.fc.register_forward_hook(lambda m, inp, out: F.dropout(out, p=0.5, training=m.training))
</code></pre>

sudonto: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, could you tell me where I should put that hook? In <code>model.__init__</code> or <code>model.forward</code> or in training loop or somewhere else?</p>

ptrblck: <p>You could register the hook once after the model was initialized, which would be before the training loop.<br>
Inside the <code>__init__</code> would also work. I wouldn’t register it in the <code>forward</code>, as you would re-register a hook in each iteration.</p>

crypdick: <p>EDIT: NVM, found <a href="https://discuss.pytorch.org/t/unclear-behaviour-of-dropout/22890/4">this discussion</a>.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> I went <a href="https://discuss.pytorch.org/t/resnet-last-layer-modification/33530/2">this solution</a> you posted elsewhere. However, if I am reading <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#dropout" rel="noopener nofollow ugc"><code>torch.nn.functional dropout()</code></a> correctly, this doesn’t apply the weight scaling inference rule:</p>
<p>From Ref: <a href="https://www.deeplearningbook.org/contents/regularization.html" rel="noopener nofollow ugc">“Deep Learning” Section 7.12 - Dropout</a>:</p>
<blockquote>
<p>A key insight (Hinton et al., 2012c) involved in dropout is that we can approximate p_ensemble by evaluating p(y | x) in one model: the model with all units, <em>but with the weights going out of unit i multiplied by the probability of including unit i</em>. The motivation for this modiﬁcation is to capture the right expected value of the output from that unit. We call this approach the <strong>weight scaling inference rule</strong>. […]</p>
<p>Because we usually use an inclusion probability of 1/2, the weight scaling rule usually amounts to dividing the weights by 2 at the end of training, and then using the model as usual. Another way to achieve the same result is to multiply the states of the units by 2 during training. Either way, the goal is to make sure that the expected total input to a unit at test time is roughly the same as the expected total input to that unit at train time, even though half the units at train time are missing on average.</p>
</blockquote>
<p>Or am I missing something?</p>

ptrblck: <p>The scaling is applied internally during training as seen here:</p>
<pre><code class="lang-python">x = torch.ones(1, 10)
print(F.dropout(x, p=0.5, training=True))
&gt; tensor([[2., 2., 0., 2., 2., 0., 2., 0., 2., 0.]])
print(F.dropout(x, p=0.5, training=False))
&gt; tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])
</code></pre>

M_M: <p>what are your parameters m and F please?</p>

ptrblck: <p><code>m</code> would be the first argument to the <code>lambda</code> function, so the module, while <code>F</code> is the functional API imported as <code>import torch.nn.functional as F</code>.</p>

Nana1: <p><a class="mention" href="/u/fiendfish">@fiendfish</a><br>
I know this is an old post, but I need to do a similar thing for a trained Resnet18 model to mitigate overfitting or premature convergence.<br>
(will continue training as it is a Federated learning training, but need to do it at certain points of training as a regularization),<br>
Could you please share how you ended up doing it and how useful it was?<br>
Thank you!</p>

NilsB: <p>Although this is an older thread I wanted to add the info that the resnets in the <a href="https://github.com/huggingface/pytorch-image-models" rel="noopener nofollow ugc">timm library</a> have support for Dropout and <a href="https://arxiv.org/pdf/1810.12890.pdf" rel="noopener nofollow ugc">DropBlock</a> already implemented.<br>
As it is not that well documented I thought it might save others some time if they are searching for this as well.<br>
With <code>timm.create_model(..., drop_rate=, drop_block_rate=)</code> the droupout can be configured.</p>