I'm not sure how to write a custom autograd.Function

ymin2570: <p>Through the response in <a href="https://discuss.pytorch.org/t/runtimeerror-element-0-of-variables-does-not-require-grad-and-does-not-have-a-grad-fn/11074/70">this post</a>, I’ve identified the issue in my code.</p>
<p>The error <code>RuntimeError: element 0 of variables does not require grad and does not have a grad_fn</code> occurred because I was recreating a tensor “out” in my custom function, detached from the computational graph.<br>
My model and implemented function are structured as follows:</p>
<pre><code>  def forward(self, x):
    x = self.head(x)

    res = self.body(x)
    res += x

    x = self.tail(res)                   # x.grad_fn returns valid value
    out = custom_function(x)             # out.grad_fn returns invalid value (None)
    out = out.cuda()

    return out
</code></pre>
<pre><code class="lang-auto">def custom_function(im):      #custom upsample function
       new_im = custom_module1(im)
       upsampled_im = new_im.repeat_interleave(2, dim=2).repeat_interleave(2, dim=3)
       # calculation part using new_im
       out = custom_module2(upsampled_im)  #Similar behavior with custom_module1
       return out
</code></pre>
<pre><code class="lang-auto">def custom_module1(im):
    b, _, h, w = im.shape
    new_im = torch.zeros((b, 3, h, w), dtype = torch.float32)  #recreating a tensor
    # calculation part using im
    return new_im 
</code></pre>
<p>However, due to my conditions, I cannot alter that behavior.<br>
In this case, I need to write custom <code>autograd.Function</code>. However, I’m having trouble understanding the example code provided <a>for this</a>. Specifically, I’m unsure of how to go about it.</p>
<p>In my case, the function performs upsampling, and it’s challenging to formulate the differentiation function.</p>
<p>Can anyone help me with this?</p>

ptrblck: <aside class="quote no-group" data-username="ymin2570" data-post="1" data-topic="202341">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/a587f6/48.png" class="avatar"> ymin2570:</div>
<blockquote>
<p>However, due to my conditions, I cannot alter that behavior.</p>
</blockquote>
</aside>
<p>If you are creating a new tensor it does not have a gradient history and previous operations don’t affect this tensor in any way.</p>
<aside class="quote no-group" data-username="ymin2570" data-post="1" data-topic="202341">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/a587f6/48.png" class="avatar"> ymin2570:</div>
<blockquote>
<p>In my case, the function performs upsampling, and it’s challenging to formulate the differentiation function.</p>
</blockquote>
</aside>
<p>I don’t think the actual <code>backward</code> computation is the issue in your use case but the lack of a computation graph. Since the tensor is created as a new leaf tensor not via upsampling, no gradients will flow back.</p>

ymin2570: <p>Then my custom_function won’t be usable for network training…</p>
<p>Thank you for your response. It was very helpful.</p>

ymin2570: <p>Hello, ptrblck,</p>
<pre><code class="lang-auto">def custom_function(im):      #custom upsample function, same as before.
       im = custom_module1(im)
       upsampled_im = im.repeat_interleave(2, dim=2).repeat_interleave(2, dim=3)
       # calculation part using im
       out = custom_module2(upsampled_im)  #Similar behavior with custom_module1
       return out
</code></pre>
<pre><code class="lang-auto">def custom_module1(im):
    im [:, 0, :, :] = transfer_table[im[:, 0, :, :].to(torch.int64), 0]
    return im 
</code></pre>
<p>I attempted to train network by modifying my <code>custom_module1</code> code as above, but encountered another issue where the network isn’t learning properly.</p>
<p>It seems like the problem may have arisen due to the transformation of values caused by the transfer_table which is <strong>nonlinear mapping.</strong></p>
<p>I think the deep learning network might not be able to flow back through the gradient due to mapping via this transfer_table.</p>
<p>I’m curious whether it’s possible to train a deep learning network using such a look-up table, or if there are any references using this approach.</p>

ptrblck: <p>You can train the lookup table itself, since it’s also done in embedding layers, but not the input since you are detaching it by transforming it to an integer type. Integer types are not usefully differentiable since their gradient would be zero everywhere and undefined (or <code>Inf</code>) at the rounding points.</p>