Torch.no_grad() with DDP

username: be7f984b2f2e66ba7969
cooked: <p>Hi,</p>
<p>I tried to use torch.no_grad() with DDP, but it would throw</p>
<pre><code class="lang-auto">This error indicates that your
module has parameters that were not used in producing its output (the
return value of forward). You can enable unused parameter detection
by passing the keyword argument find_unused_parameters=True to
torch.nn.parallel.DistributedDataParallel
</code></pre>
<p>the pseudo code is as following</p>
<pre><code class="lang-auto">
class MyModel(nn.Module):
    def forward(self, x):
        with torch.no_grad():
            self.layers(x)
        return x

class WholeFlow(nn.Module):
    def __init__(self):
        self.f=MyModel()
        self.g=nn.Linear(256, 256)
    def forward(self, x):
        x=self.f(x)
        x=self.g(x)
        return x

SGD=(WholeFlow.g.parameters(),...)
</code></pre>
<p>There is a similar issue here: <a href="https://github.com/PyTorchLightning/pytorch-lightning/issues/6087" class="inline-onebox" rel="noopener nofollow ugc">DDP does not work well with `torch.no_grad()` in 1.2 · Issue #6087 · PyTorchLightning/pytorch-lightning · GitHub</a></p>
<p>It works with DataParallell, but can’t work with DDP.</p>
<p>Any idea?</p>

username: ptrblck
cooked: <p>Did you try to add the suggested <code>find_unused_parameters=True</code> argument and if so, did you get any other error?</p>

username: be7f984b2f2e66ba7969
cooked: <p>add <code>find_unused_parameters=True</code> works. but is this a bug for DDP?</p>

username: ooohh
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I meet a similar problem, I have a code like follows, only last frame will do bp and calculate loss. But all frames use the same parameters.<br>
It works with find_unused_parameters=True, but the weights cannot be updated. I need the weights to be updated by last frame’s bp. So I can only use <strong>detach</strong> for this case?</p>
<pre><code class="lang-auto">class DummyDataset(torch.utils.data.Dataset):

    def __init__(self, seq_len=5):
        self.seq_len = seq_len
        

    def __len__(self):
        return 200

    def __getitem__(self, idx):
        return np.random.rand(self.seq_len, 10).astype(np.float32), np.random.rand(1).astype(np.float32)
        
class DummyNet(nn.Module):

    def __init__(self):
        super().__init__()

        self.mlp_fea = nn.Linear(10, 10)
        self.mlp_out = nn.Linear(10, 1)
    
    def forward(self, inputs, labels, training=False):
        inputs = inputs.cuda()
        labels = labels.cuda()

        B, SEQ, C = inputs.shape
        print(self.mlp_fea.weight)
        for i in range(SEQ):
            # print('seq: ', i)
            x = inputs[:, i]
            if i == 0:
                fea_prev = torch.zeros_like(x)

            if i &lt; SEQ - 1:
                self.eval()
                with torch.no_grad():
                    fea_prev = self.mlp_fea(x + fea_prev)
                self.train()
            else:
                fea_prev = self.mlp_fea(x + fea_prev)
                out = self.mlp_out(fea_prev)
            if i == SEQ - 1:
                loss = nn.L1Loss(reduction='mean')(out, labels)
                return loss
</code></pre>

username: ptrblck
cooked: <p>Could you explain how your use case is related to DDP as I don’t see the connection, please?</p>

username: ooohh
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I use ddp to wrap the DummyModel, same problem with log</p>
<p>‘This error indicates that your<br>
module has parameters that were not used in producing its output (the<br>
return value of forward)’</p>

username: ptrblck
cooked: <p>If you don’t want to synchronize the gradients, wrap the forward pass into a <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync"><code>no_sync()</code></a> context.</p>