Custom Layer, RunTime error when using values of a previous layer

Theo_Voillemin: <p>Hello !</p>
<p>I’m currently developing a custom layer and having trouble getting it to work, specially when I’m using the results of a previous layer in my network as input of my custom layer.</p>
<p>Here’s a code snippet isolating the problem I’m currently having :</p>
<pre><code class="lang-auto">    def forward(self, y):

        a = torch.unsqueeze(y, 0)
        b = a.expand(self.output_resolution[0] * self.output_resolution[1], -1, -1)
        c = torch.permute(b, (1, 0, 2))
        d = torch.sum(c, dim=-1)
        end = torch.reshape(d, (-1, self.output_resolution[0], self.output_resolution[1]))

        output = end[:, None, :, :]

        out = output.expand(-1, self.in_c, -1, -1).float()
        return out
</code></pre>
<p>It’s not what my custom layer will do but it isolates the last error that I can’t resolve.</p>
<p><em>y</em> is the output of a previous Linear Layer (with shape [batch_size, num_features])<br>
<em>output_resolution</em> and <em>in_c</em> are just some dimensions I want as output to my custom layer</p>
<p>The problem is, from the moment I use <em>y</em>, I get the <strong>RunTime error : one of the variables needed for gradient computation has been modified by an inplace operation</strong></p>
<p>I try everything, copy pasting every value of the <em>y</em> tensor manually, use <em>y.clone()</em>, <em>torch.clone(y)</em>, even <em>torch.clone(y.clone())</em>, nothing works, I get the RunTime error even though I only do <em>unsqueeze</em> and <em>expand</em> operation on <em>y</em>.</p>
<p>On my full Custom Layer, the only way I found to make it run, is by using <em><strong>y.detach()</strong></em>, but I’m afraid that, a part of my custom layer that is directly using value of <em>y</em> will not be considered in the backward and gradient computation operation.</p>
<p>I have two questions :</p>
<ul>
<li>Is there a problem using <em>y.detach()</em> in my final Custom Layer, specially in the gradient computation ?</li>
<li>Is there a solution to my problem, is this a problem with some upper code (I don’t know, maybe by using <em>DistributedDataParallel</em> or other) ?</li>
</ul>
<p>Thank you in advance and have a nice day !</p>
<p>TV</p>

ptrblck: <p>Could you post a minimal, executable code snippet which would reproduce the issue, please?<br>
I don’t see any obvious inplace operation in your code so don’t know what’s causing the issue.</p>

Theo_Voillemin: <p>Thank you for the answer !</p>
<p>The code is currently massive and from a large GitHub project, I’ll try to provide you with a minimal snippet as soon as I manage to recreate the error in a small environment. Probably early next week.</p>
<p>However, I noticed that the error does indeed occur when I use DistributedDataParallel on the network :</p>
<pre><code class="lang-auto">network = DistributedDataParallel(network, device_ids=[torch.cuda.current_device()], find_unused_parameters=find_unused_parameters)
</code></pre>
<p>If I only use</p>
<pre><code class="lang-auto">network = DataParallel(network)
</code></pre>
<p>my custom layer work well without using <em>y.detach()</em>, just the execution seems to be longer.</p>
<p>I found other issues with a similar problem of runtime error, gradient and inplace operation when using DistributedDatParallel, but with BatchNorm (solved by using SyncBatchNorm) :</p>
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/66165366/trying-to-use-distributed-data-parallel-on-gans-but-getting-runtime-error-about" rel="noopener nofollow ugc">neural network - Trying to use Distributed data parallel on GANs but getting runtime error about an inplace operation - Stack Overflow</a></p>
</li>
<li>
<p><a href="https://discuss.pytorch.org/t/distributeddataparallel-gradient-computation-has-been-modifed-by-an-inplace-operation/133682">DistributedDataParallel: Gradient computation has been modifed by an inplace operation - distributed - PyTorch Forums</a></p>
</li>
</ul>
<p>Any idea why it happen with my Custom Layer ?</p>
<p>I just call my layers like that :</p>
<pre><code class="lang-auto">x = nn.Conv2d(num_feat, num_feat, 3, 1, 1)(input)
y = torch.reshape(x, (-1, img_size*img_size*in_c))
n = CustomLayer((img_size,img_size), in_c)(y)
x = nn.Conv2d(x+n)
</code></pre>
<p>Thank you again and i’ll try to provide you a compact and executable code snippet with the error as soon as possible.</p>

ptrblck: <p>I don’t know what your <code>CustomLayer</code> exactly does, but in case it contains registered parameters and buffers, note that you are re-initializing it in your <code>forward</code> method (same as the first <code>nn.Conv2d</code> layer while the second should raise an error).</p>
<p>Take a look at <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html">this tutorial</a> to see how <code>nn.Module</code>s are used. The standard approach is to initialize and register all modules in the <code>__init__</code> method and just use them in the <code>forward</code>.</p>