Random initialisation of densenet layers

SU801T: <p>What is the best way of randomly initialising but freezing the last layers of a densenet?</p>
<p>I have the following code, where I am using a pretrained model but unfreezing the last denseblock4 and norm5 blocks for fine-tuning:</p>
<pre><code class="lang-auto">model = models.densenet161(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
submodules = model.features[-2:]  
for param in submodules.parameters():
    param.requires_grad = True
</code></pre>
<p>However, instead of training these submodule layers, I would like to randomly intialise them and freeze them. What would be the best way of doing this?</p>

ptrblck: <p>You could remove the second loop, which “unfreezes” the parameters and call <code>.reset_parameters()</code> on the desired modules instead:</p>
<pre><code class="lang-python">for module in submodules.modules():
    if hasattr(module, "reset_parameters"):
        module.reset_parameters()
</code></pre>
<p>or use a custom weight initialization via <code>submodules.apply</code>.</p>

SU801T: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="167974">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">for module in submodules.modules():
    if hasattr(module, "reset_parameters"):
        module.reset_parameters()
</code></pre>
</blockquote>
</aside>
<p>ok great, I think that’s exactly what I was looking for. Is there a way of checking if they are frozen/unfrozen?</p>

ptrblck: <p>Yes, you can check the <code>.requires_grad</code> of the corresponding parameters and verify it’s set to <code>False</code>:</p>
<pre><code class="lang-python">for name, param in submodules.named_parameters():
    print("{}.requires_grad == {}".format(name, param.requires_grad))
</code></pre>

SU801T: <p>Let’s say I want to use batch normalisation stats provided by ImageNet and so not randomly initialise them. Would the following be appropriate when defining the model before running the training loop?:</p>
<pre><code class="lang-auto">with torch.no_grad():  # allows to re-initialize the parameters
    submodules = model.features[-2:] 
    for submodule in submodules.modules():
        if isinstance(submodule, torch.nn.Conv2d):
            # randomly re-initialize the weights
            torch.nn.init.kaiming_normal_(submodule.weight)
            if submodule.bias is not None:
                # reset the bias to zero
                torch.nn.init.zeros_(submodule.bias)
        elif isinstance(submodule, torch.nn.BatchNorm2d):
            torch.nn.init.ones_(submodule.weight)
            torch.nn.init.zeros_(submodule.bias)
            # also reset running mean and running_var
            torch.nn.init.zeros_(submodule.running_mean)
            torch.nn.init.ones_(submodule.running_var)
</code></pre>

ptrblck: <p>No, since you are resetting the batchnorm parameters and buffers to their original values while it seems you want to use the pretrained values or am I misunderstanding your use case?</p>

SU801T: <p>Well, I’m trying to randomly initialise (or set) the last few conv layers whilst keeping them frozen. However, I am also considering keeping the batch norm layers frozen with the previous ImageNet values rather than also randomly generating those values. I’m just curious to see if this would have an impact during training.</p>
<p>Also, this has made me think, wen resetting the parameters, how would I keep them frozen. I’m not sure if the above solution does that <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20">.</p>

ptrblck: <aside class="quote no-group" data-username="SU801T" data-post="7" data-topic="167974">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar"> SU801T:</div>
<blockquote>
<p>However, I am also considering keeping the batch norm layers frozen with the previous ImageNet values rather than also randomly generating those values. I’m just curious to see if this would have an impact during training.</p>
</blockquote>
</aside>
<p>You are resetting the batchnorm parameters and buffers to their initial value as seen <a href="https://github.com/pytorch/pytorch/blob/4372dbb89f457a0aa90f22d27f1468442a2a51bd/torch/nn/modules/batchnorm.py#L68-L80">here</a> and are not keeping the pretrained values.</p>
<aside class="quote no-group" data-username="SU801T" data-post="7" data-topic="167974">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar"> SU801T:</div>
<blockquote>
<p>Also, this has made me think, wen resetting the parameters, how would I keep them frozen.</p>
</blockquote>
</aside>
<p>Set their <code>.requires_grad</code> attribute to <code>False</code> as already explained.</p>

SU801T: <p>Ah ok, so would this be sufficient to keep the later modified layers “frozen” at a random initialisation rather than ImageNet values?:</p>
<pre data-code-wrap="model"><code class="lang-nohighlight">    for param in model.parameters():
        param.requires_grad = False
    submodules = model.features[-2:]  
    for module in submodules.modules():
        if hasattr(module, "reset_parameters"):
            module.reset_parameters()
        
    num_ftrs = model.classifier.in_features
    model.classifier = torch.nn.Linear(num_ftrs,2)
</code></pre>

ptrblck: <p>No, you would need to freeze <code>model.classifier</code> after replacing it with a new linear layer:</p>
<pre><code class="lang-python">for param in model.classifier.parameters():
    param.requires_grad = False
</code></pre>

SU801T: <p>Ok, apologies I don’t think I made myself clear here.</p>
<p>I want to create a feature extractor and so only train the classifier layers, and freeze the preceding feature extractor layers. However, I would like to randomly initialise and freeze the layers in the last two blocks of the feature extractor and only keep ImageNet values for the layers before these blocks.</p>
<p>I’m just wondering if adding this code after freezing all layers, unfreezes the last two blocks:</p>
<pre><code class="lang-auto">for module in submodules.modules():
        if hasattr(module, "reset_parameters"):
            module.reset_parameters()

</code></pre>
<p>I’m assuming that despite resetting parameters, they are still frozen apart from the classifier.</p>

ptrblck: <p>Calling <code>reset_parameters()</code> will manipulate the parameters inplace and will thus not change any attributes. If you’ve frozen these parameters before, they should still be frozen.<br>
However, you can use my code snippets to easily verify it by printing the <code>.requires_grad</code> attribute afterwards.</p>

SU801T: <p>Great, thank you for being patient with me!</p>