When I used a numpy-based upsampling function in the network, the training didn't proceed properly

ymin2570: <p>I want to use a numpy-based upsampling function in training a deep learning network.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/593bfea81d4b0e584eb72550af2ce4327755322c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/593bfea81d4b0e584eb72550af2ce4327755322c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/593bfea81d4b0e584eb72550af2ce4327755322c_2_345x218.jpeg" alt="image" data-base62-sha1="cJp22Z8dPguOsKg7eeIxMg0GxPu" width="345" height="218" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/593bfea81d4b0e584eb72550af2ce4327755322c_2_345x218.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/593bfea81d4b0e584eb72550af2ce4327755322c_2_517x327.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/593bfea81d4b0e584eb72550af2ce4327755322c_2_690x436.jpeg 2x" data-dominant-color="E5DEDA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">800×506 68.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I want to use the loss function between the upsampling result and the GT to train the network to preprocess the image before upsampling.</p>
<pre><code>def forward(self, x):
    x = self.head(x)

    res = self.body(x)
    res += x

    x = self.tail(res)
    images = []
    for batch in range(x.shape[0]):
        single_img = x[batch, :, :, :].squeeze(0)
        np_single = tensor_to_numpy(single_img.cpu().detach())
        upsampled = numpy_based_upsample(np_single)
        up_tensor = numpy_to_tensor(upsampled).unsqueeze(0)
        images.append(up_tensor)
    out = torch.cat(images, dim=0).cuda()
    return x, out
</code></pre>
<p>I wrote the code as described above, and I encountered error message<br>
“RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”<br>
So, I changed the code as follow<br>
<code>out = torch.cat(images, dim=0).cuda().requires_grad_(True)</code>, but upon inspecting the results, it seems that the network hasn’t learned properly. In such a case, how should I resolve this problem?</p>
<p>(The reason why I wrote the code as above is that the function based on numpy is for images of (H, W, C), But the tensor image of the network is the shape of (B, C, H, W).)</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/ymin2570">@ymin2570</a>,</p>
<p>When you detach a tensor and pass it through a <code>numpy</code> function, you delete the gradient history the operation (which sets its <code>requires_grad</code> flag to <code>False</code>. When you re-cast it to a PyTorch tensor, it will only have the history from the re-casting to the computing of the loss.</p>
<aside class="quote no-group" data-username="ymin2570" data-post="1" data-topic="202252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ymin2570/48/60786_2.png" class="avatar"> ymin2570:</div>
<blockquote>
<pre><code class="lang-auto">        np_single = tensor_to_numpy(single_img.cpu().detach())
</code></pre>
</blockquote>
</aside>
<p>This is why you get,</p>
<aside class="quote no-group" data-username="ymin2570" data-post="1" data-topic="202252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ymin2570/48/60786_2.png" class="avatar"> ymin2570:</div>
<blockquote>
<p>“RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”</p>
</blockquote>
</aside>
<p>When you manually add the <code>requires_grad_(True)</code> to be True, you’re not solving the problem but hiding it, because autograd will track gradients up to,</p>
<aside class="quote no-group" data-username="ymin2570" data-post="1" data-topic="202252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ymin2570/48/60786_2.png" class="avatar"> ymin2570:</div>
<blockquote>
<pre><code class="lang-auto">    out = torch.cat(images, dim=0).cuda()
</code></pre>
</blockquote>
</aside>
<p>but it can’t track the gradient through the numpy function.</p>
<p>In order to solve this solution, you’ll need to use a pytorch function (which will track the gradient), or you will need to create your own <code>torch.autograd.Function</code> with its own forward and backward calculations defined manually (assuming you can define the gradient of an up-sampling operation).</p>
<p>An example <code>torch.autograd.Function</code> can be found in the documentation here: <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch: Defining New autograd Functions — PyTorch Tutorials 2.3.0+cu121 documentation</a></p>