Autograd issue with F.grid_sample()

mowolf: <p>I need the gradient of this warping operation.</p>
<pre><code class="lang-auto">// - tensor   :  output of my neural network 
//               tensor.requires_grad = True

warped_tensor = F.grid_sample(tensor, 
                              grid,
                              align_corners=True,
                              mode='bilinear',
                              padding_mode='zeros').
</code></pre>
<p>This operation returns a gradient, however it seems to be not correct. I used the warped_tensor and just the tensor for my loss and with the warped_tensor my network does not correctly optimise the weights.</p>
<p>Is this an autograd issue or is there some other issue I am not seeing here?</p>
<hr>
<p>The same happens when I use <a class="hashtag" href="/c/vision/kornia/19">#<span>vision:kornia</span></a></p>
<pre><code class="lang-auto">// - tensor   :  output of my neural network 
//               tensor.requires_grad = True
// - M      : transformation Matrix prev created with kornia

warped_tensor: torch.tensor = kornia.warp_affine(tensor, M, dsize=(h_original, w_original))
</code></pre>

albanD: <p>Hi,</p>
<p>This is most likely due to your network. This function is quite widely used and tested so I think it is correct.<br>
You can double check by using <code>torch.autograd.gradcheck.gradcheck</code> function wil double typed inputs to verify that the computed gradients are correct.</p>

mowolf: <p>Will check my network with <code>torch.autograd.gradcheck.gradcheck</code>.</p>
<p>However isn’t it weird that my network works when I use <code>loss(tensor, truth)</code> but not with <code>loss(warped_tensor, warped_truth)</code>  where <code>warped_truth</code> was applied the exact same transformation.</p>
<p>I will try to provide you with some executable code soon.</p>

albanD: <p>Depends on the warping you apply.<br>
You can loose precision when you warp if you collapse multiple input pixels into a single output one?</p>
<p>I can take a look if you have a small code sample yes !</p>

mowolf: <p>This makes sense! I am quite certain now that my error lies here.</p>
<p>Here is a gist using <code>kornia.warp_affine</code>  for the transformation as they use <code>F.grid_sample</code> under the hood.  (<a href="https://github.com/kornia/kornia/blob/20a9acca2636512522116601ae09c6be0408b486/kornia/geometry/transform/imgwarp.py#L126" rel="nofollow noopener">source for <code>kornia.warp_affine</code></a>)</p>
<p>Here is my code:</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/mowolf/fa1b0c327ddd3add758bac5d990a4638" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/mowolf/fa1b0c327ddd3add758bac5d990a4638" target="_blank" rel="nofollow noopener">https://gist.github.com/mowolf/fa1b0c327ddd3add758bac5d990a4638</a></h4>
<h5>mock_transformation_network.py</h5>
<pre><code class="Python">import kornia
import numpy as np
import torch.nn.functional as F


def get_M():
    '''
    Provide mock trafo Matrix and original image sizes
    :return: w_original, h_original, M 
    '''</code></pre>
This file has been truncated. <a href="https://gist.github.com/mowolf/fa1b0c327ddd3add758bac5d990a4638" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>How can I now make sure that my gradient is not loosing precision? Must the size of the resulting trafo tensor just be big enough? The interpolation with <code>nearest</code> might also have this problem. Would <code>bilinear</code> fix that or should I make sure that I dont scale down?</p>

albanD: <p>I don’t have kornia at hand to try it but yes these sound like good things to try out.</p>

gedoensmax: <p>Has there been any update on this ? My first guesses were too that <code>nearest</code> mode might not work but that did not help in my case.</p>