Freezing layers issue for parallel GPU

username: Han_Brian_Lee
cooked: <p>Hi,</p>
<p>So I think I found out what the cause for having weird error (see below) when trying to freeze layers using the following the following method (context manager also fails to freeze layers) is using DataParallel (model = nn.DataParallel(model)) across multiple GPUs. I’ve been running my model on 2 identical GPUs (gtx1080) and when I tried to freeze weights, I got the error shown below. When I don’t apply DataParallel and just use single GPU, freezing layers works like a charm as shown in countless examples across the internet and on this forum, but when DataParallel is used, it gives out a weird error (doesn’t make sense to me… the layers it should try to freeze are not leaf variables)</p>
<p>Could anyone help me understand what is happening and <strong>if this is a bug in pytorch?</strong> (regardless of whether it is or not, I’d like to know <strong>if there is a workaround</strong>)</p>
<pre><code class="lang-auto">for param in model.parameters():
  param.requires_grad = False
</code></pre>
<p><strong>RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn’t require differentiation use var_no_grad = var.detach().</strong></p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>Could you post a code snippet to reproduce this issue?<br>
This dummy code snippet works:</p>
<pre><code class="lang-python">model = models.resnet18()

for param in model.parameters():
    param.requires_grad = False

model = nn.DataParallel(model)
model.cuda()

out = model(torch.randn(8, 3, 224, 224))
print(out.shape)
</code></pre>

username: Han_Brian_Lee
cooked: <p>Here it is. If you comment out the model = nn.DataParallel(model) line, no error occurs, otherwise the error occurs (on a computer with multiple GPUs):</p>
<pre><code class="lang-auto">from __future__ import print_function
from __future__ import division
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

class mynet(torch.nn.Module):
    def __init__(self):
        super(mynet, self).__init__()
        self.fc = nn.Linear(2, 1, bias=False)

    def forward(self, x):        
        for param in self.fc.parameters():
            param.requires_grad = False

        return self.fc(x)

model = mynet()
model = nn.DataParallel(model)  # comment out this line to make the error go away
model.cuda()

data = np.array([0.0, 1.0])
x = torch.from_numpy(data).float().unsqueeze(0).unsqueeze(0).unsqueeze(0)
gt = torch.from_numpy(np.array([0.0])).float().unsqueeze(0).unsqueeze(0).unsqueeze(0)

# model.train()
result = model(x=x.cuda())


</code></pre>

username: ptrblck
cooked: <p>Thanks for the code.<br>
The error is most likely thrown, as you are trying to manipulate the <code>require_grad</code> attribute of the model replica parameters.<br>
If you freeze the parameters outside of the <code>forward</code> method, it’ll work.</p>
<p>What’s your workflow that you want to freeze them in each forward pass?</p>

username: Han_Brian_Lee
cooked: <p>Hm… I see. I ideally wanted to train some iterations with a part of my network frozen, and some iterations after that with it not frozen, and alternate.</p>
<p>I have 4 sub-networks(a,b,c,d). And there’s a big wrapper network (say N) that contains those 4 sub-networks, where the wrapper network’s flow goes  (a,b separately)-&gt;combined into c-&gt;then d. And I wanted to freeze subnetwork a’s weights (load pickled trained weights for subnetwork a and don’t train them).</p>
<ol>
<li>Not considering alternate freezing/unfreezing, what’s the best way to achieve freezing weights in this case?<br>
Would it be like the following? <strong>But Once DataParallel wraps the model, the “a” subnetwork is no longer accessible from model.</strong> What’s the best way of freezing subnetwork a’s weights for multi-GPU setting? Some way of maybe model.modules()? or model._modules[‘module’].a.parameters() ?</li>
</ol>
<pre><code class="lang-auto">model = N()
model = nn.DataParallel(model)  
for parameter in model.a.parameters():
    parameter.requires_grad = False
</code></pre>
<ol start="2">
<li>If I wanted to alternate freezing/unfreezing, what’s the ideal way?</li>
</ol>

username: ptrblck
cooked: <p>You should still be able to freeze the submodules via <code>model.module.a</code> after you’ve wrapped them in <code>nn.DataParallel</code>. As long as your changes are done outside of the forward pass, which is executed on each replica, it should be fine.</p>