[SOLVED]Torch.grid_sample?

Petro_Key: <p>Hi, thank you always for your support.<br>
I cannot understand the output torch.grid_sampler at the end or the torch.nn.functional.grid_sample.<br>
Before watching the code, I expected that the grid_sample function calculate an output torch tensor sampled from the input tensor.<br>
What does the grid_sample do?</p>
<p>Thank you in advance:)</p>

ptrblck: <p>The method samples the output from the input using the specified grid.<br>
Have a look at this example:</p>
<pre><code class="lang-python">input = torch.arange(4*4).view(1, 1, 4, 4).float()
print(input)
&gt; tensor([[[[ 0.,  1.,  2.,  3.],
          [ 4.,  5.,  6.,  7.],
          [ 8.,  9., 10., 11.],
          [12., 13., 14., 15.]]]])

# Create grid to upsample input
d = torch.linspace(-1, 1, 8)
meshx, meshy = torch.meshgrid((d, d))
grid = torch.stack((meshy, meshx), 2)
grid = grid.unsqueeze(0) # add batch dim

output = torch.nn.functional.grid_sample(input, grid)
print(output)
&gt; tensor([[[[ 0.0000,  0.4286,  0.8571,  1.2857,  1.7143,  2.1429,  2.5714,
            3.0000],
          [ 1.7143,  2.1429,  2.5714,  3.0000,  3.4286,  3.8571,  4.2857,
            4.7143],
          [ 3.4286,  3.8571,  4.2857,  4.7143,  5.1429,  5.5714,  6.0000,
            6.4286],
          [ 5.1429,  5.5714,  6.0000,  6.4286,  6.8571,  7.2857,  7.7143,
            8.1429],
          [ 6.8571,  7.2857,  7.7143,  8.1429,  8.5714,  9.0000,  9.4286,
            9.8571],
          [ 8.5714,  9.0000,  9.4286,  9.8571, 10.2857, 10.7143, 11.1429,
           11.5714],
          [10.2857, 10.7143, 11.1429, 11.5714, 12.0000, 12.4286, 12.8571,
           13.2857],
          [12.0000, 12.4286, 12.8571, 13.2857, 13.7143, 14.1429, 14.5714,
           15.0000]]]])
</code></pre>

Petro_Key: <p>Thank you always for your kind reply, ptrblck.<br>
Your provided sample code is quite understandable and helpful for me.<br>
I now see what torch.grid_sample does.<br>
I am grateful for your continuous support!</p>

Hmrishav_Bandyopadhy: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> that was a nice explanation. However, I would like to know the intuition behind selecting the linspace range and interval.<br>
Also, if I wanted to do a downsample, or an affine transform using this, how can i proceed? How will I create the grid if, say, I want a piecewise affine transform and I have some source and destination points. For the sake of example, let’s assume src points as [[1,1],[2,2],[2,3],[1,2]] and dst points as [[1,1],[1,2],[2,2],[1,2]] ( transforming a trapezium to a square). How to start with this?</p>

ptrblck: <p>That’s an interesting use case and I’m not sure at the moment, how you could easily create the grid for these transformation. So far I’ve stuck to the matrix operations as seen e.g. <a href="https://github.com/ptrblck/pytorch_misc/blob/master/image_rotation_with_matrix.py">here</a> for a rotation.</p>

bnehoran: <p><a class="mention" href="/u/hmrishav_bandyopadhy">@Hmrishav_Bandyopadhy</a> If you want to do an affine transformation, then you can use code similar to what ptrblck shows above, but instead of using <code>linspace</code>/<code>meshgrid</code> to produce the grid, use <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.affine_grid" rel="nofollow noopener"><code>F.affine_grid()</code></a>.<br>
You just have to pass it a matrix <code>theta</code> which contains the affine parameters for your transformation.<br>
Then pass the resulting grid to <code>grid_sample</code>.</p>
<p>For upsampling/downsampling specifically, I wouldn’t recommend using <code>grid_sample</code> (although, of course, you could if you really wanted to). For this, you should probably use <a href="https://pytorch.org/docs/stable/nn.functional.html#interpolate" rel="nofollow noopener"><code>F.interpolate()</code></a>, which is specifically for upsampling and downsampling.</p>
<p>If you’re wondering about the range of coordinates in the grid passed to grid sample, they range from -1 to 1, where -1 refers to either the top or left edges of the sampled image (in the x and y axes, respectively), while +1 refers to the bottom or right edges of the sampled image. So a grid point containing (0, 0), for example, would sample from the center of the image.</p>

Hmrishav_Bandyopadhy: <p><a class="mention" href="/u/bnehoran">@bnehoran</a> Thanks for the detailed answer. Affine transform (The function in PyTorch) works nice for stuff like rotation, zoom in and zoom out. How is a skew operation, or an operation like image deformation, or a Piecewise affine Transform possible with F.affine_grid() ?</p>
<p>It seems that theta has too small dimensions in order to completely form the affine grid for something other than a rotation, or a zoom-in (which is probably its intended use) – which is why i resorted to directly computing the grid and applying the F.grid_sample() operation instead of going through the F.affine_grid() operation</p>

bnehoran: <p>Yeah, you are right. If you want to use more complex transformations that are not affine, then you need to generate your own custom grid to pass to <code>grid_sample</code>.</p>
<p>If you want, you can always start by calling <code>affine_grid</code> and then modifying the grid that you get to your liking before passing it to <code>grid_sample</code>, but that’s entirely up to you.</p>
<p>Just keep in mind that if you generate your own grid by hand, you need to worry about things like which way you are setting the <code>align_corners</code> parameter in order to make sure that the grid you are producing has the right values to get your intended effect.</p>

Hmrishav_Bandyopadhy: <p><a class="mention" href="/u/bnehoran">@bnehoran</a> can you elaborate on how a grid can be generated by hand ( like the constraints involved) ? I have been opening up issues and looking up the docs, but have not found a single reference to it <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
I have been trying to “learn” a grid by keeping it as an output of a layer. However, since I know nothing of the constraints, its impossible to create the model ! In fact whatever models I have had created to get some insight to the grid have failed miserably.</p>

Hmrishav_Bandyopadhy: <p>Okay–so i am posting part of the solution to my question just above here, as I have found it through some calculations and observing the results from <code>F.grid_sample()</code></p>
<p>Let us say grid sample is working with the help of a <em>grid</em> which has, say,  dimensions 1x64x64x2. The final dimensions of the image is 64x64. Now, say we are “filling” up the final image with values coming from the initial image. The first pixel (0,0) will have value as <code>img[grid[0,0,0],grid[0,0,1]]</code> where img is the original image and grid is the grid being used. The rest of the pixels will have values in a similar fashion.</p>
<p>Now, an important thing is the grid is a float tensor, and, thus cant be used for indexing. So, my question here is, do we average the pixel values? Say when we have the grid pixel at (0,0,0) as 41.5, do we do <code>((img[41,y]+img[42,y])/2.0)</code>  or not?–this method of pixel averaging somehow seems to be a crude approach, hence I am unsure.</p>

lyx0208: <p>I still fail to understand, can you explain in detail?<br>
Thanks in advance!</p>

Wesley_Neill: <p>Sorry to dig up an old question with an accepted answer. I want to get an intuitive/geometric feel for this function. The docs mention a flow field grid, which I googled and understand in the sense of motion, but not generally.</p>
<p>I’d like to restate your example in terms of a single-channel image.</p>
<p>Let’s say I’ve got a single channel image with resolution (4, 4). Then I want to find the color at some spatial point (2.5, 2.5), which is not captured by the resolution of the image. This seems like a case for upsampling to me.</p>
<p>Now in your example, your “flow field grid” is (1, 8, 8, 2), where it seems that <code>grid[:, i, j, :]</code> gives me something like a spatial coordinate (not index) of some unit box.</p>
<p>So somehow, I pass my input (4, 4) image in with a grid that has the resolution that I want and I get out an upsampled image.</p>
<p>Here’s what I don’t understand:</p>
<p><strong>What is this grid and what do the contained values signify.?</strong></p>
<p>I see that its dimensions make it a “flow field” but how does that connect to upsampling?</p>

ptrblck: <p>The grid contains the normalized coordinates which should be used to interpolate the image.<br>
They are normalized in <code>[-1, 1]</code> and these values are mapped to the “corners” of the input (the “corner” definition depends on the <code>align_corners</code> argument as well).<br>
The <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html?highlight=grid_sample#torch.nn.functional.grid_sample">docs</a> explain it as:</p>
<blockquote>
<p>For each output location <code>output[n, :, h, w]</code>, the size-2 vector <code>grid[n, h, w]</code> specifies <code>input</code> pixel locations <code>x</code> and <code>y</code>, which are used to interpolate the output value <code>output[n, :, h, w]</code>. In the case of 5D inputs, <code>grid[n, d, h, w]</code> specifies the <code>x</code>, <code>y</code>, <code>z</code> pixel locations for interpolating <code>output[n, :, d, h, w]</code>. <code>mode</code> argument specifies <code>nearest</code> or <code>bilinear</code> interpolation method to sample the input pixels.</p>
</blockquote>
<blockquote>
<p><code>grid</code> specifies the sampling pixel locations normalized by the <code>input</code> spatial dimensions. Therefore, it should have most values in the range of <code>[-1, 1]</code>. For example, values <code>x = -1, y = -1</code> is the left-top pixel of <code>input</code>, and values <code>x = 1, y = 1</code> is the right-bottom pixel of <code>input</code>.</p>
</blockquote>
<blockquote>
<p>If <code>grid</code> has values outside the range of <code>[-1, 1]</code>, the corresponding outputs are handled as defined by <code>padding_mode</code>.</p>
</blockquote>

Wesley_Neill: <p>I’m sorry, that doesn’t answer the question I’m trying to ask. Let me clarify my current understanding and give context for the question:</p>
<p>The input is a single-channel (for simplicity) image.</p>
<p>Then we have the “grid”, which is a stack of two maps such that: <code>destination(x, y) = stack(xmap(x, y), ymap(x, y))</code>.</p>
<p>This grid tells a pixel of the “input” where to go from its original location <code>(x, y)</code> to some <code>(x_hat, y_hat)</code> in the transformed (upsampled or downsampled). For upsampling, the mapping of any point/vector would have to be one-to-many and there’d have to be some (linear?) combination of pixels in the source image.</p>
<p>My problem is framed in a slightly different context, and the intent of my question is to connect those two contexts. Here’s my use-case:</p>
<p>I sample points on a 3d mesh with a geodetic coordinate system. Via a special transform, I am able to locate the (row/column) associated with that 3d point in several different views of the same object.</p>
<p><strong>The returned row/column may not be an integer value but something in between pixels.</strong></p>
<p>In this case, the <code>input</code> is the image and the <code>grid</code> is a set of <code>(row, column)</code> coordinates that originate from the sampled 3d mesh.</p>
<pre><code class="lang-auto">colors = F.grid_sample(image, sample_coordinates)
</code></pre>
<p><code>sample_coordinates</code> is not a grid. It is a set of coordinates with shape:</p>
<pre><code class="lang-auto">(1 [batch], num_samples, 1 [dummy], 2 [row, column])
</code></pre>
<p>such that the output <code>colors</code> is the interpolated colors at those sample points of those non-integer row/columns (which is the <code>grid</code> argument).</p>
<p>This is already done in our code base and works fine. I am inheriting this code and trying to make a connection in my mind between.</p>
<ol>
<li>input image pixels and “flow <code>grid</code>” .</li>
<li>input image pixels and sample coordinates (occupying the <code>grid</code> variable).</li>
</ol>
<p>The mental model difference between these two contexts is that in</p>
<ol>
<li><code>grid</code> is a map and <code>input</code> is the object being mapped to a higher or lower resolution.</li>
<li><code>grid</code> is set of continuous coordinates associated with discrete scalar field of colors (the <code>input</code> image).</li>
</ol>
<p>So rephrasing my original question:</p>
<p><strong>What is the mathematical interaction of the input argument and the grid argument?</strong></p>
<p>This will not only help me understand the shape of inputs, but also the unclear (to me) normalization requirements specified here:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c0ecc96ade044b0ec1b34e677eef450e95823f8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/c0ecc96ade044b0ec1b34e677eef450e95823f8a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c0ecc96ade044b0ec1b34e677eef450e95823f8a.png" alt="image" data-base62-sha1="rwGZeDci7eSlLRf8MNSQ2OOrDqq" width="690" height="70" data-dominant-color="F0EFF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">998×102 13 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Wesley_Neill: <p>I think that writing out that novel helped me get an understanding:</p>
<p>In both cases <code>input</code> is the reference and <code>grid</code> is the set of coordinates that say (I want the value of ‘input’ at these coordinates)</p>
<p>In context 1, we want an entirely resampled image. So we just create an evenly spaced grid of normalized coordinates. In my context, I only want a few points at a time but the concept is the same.</p>
<p>Note:</p>
<p>Documenting this function in terms of a “flow grid”  sounds like someone trying to be smart instead of helpful. I’ve spent hours trying to figure this out because of PyTorch’s convoluted docs.</p>
<p>My entire misunderstanding was centered on the relationship between a flow grid and an input image. Kornia’s docs didn’t help much by using “map” instead of “flow grid” because a flow grid is a map and the word map adds nothing.</p>
<p>The only thing we are mapping is coordinates to pixel values. Those coordinates are framed as the map by the documentation, rather than the thing that is being mapped. The relationship is exactly opposite of what is described in the documents.</p>

InnovArul: <p>Just for the completeness, <code>grid_sample()</code>, <code>affine_grid()</code> are inspired from the paper:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1506.02025">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1506.02025" target="_blank" rel="noopener">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/1506.02025" target="_blank" rel="noopener">Spatial Transformer Networks</a></h3>

  <p>Convolutional Neural Networks define an exceptionally powerful class of
models, but are still limited by the lack of ability to be spatially invariant
to the input data in a computationally and parameter efficient manner. In this
work we introduce a...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Ishaan: <p>Hi Wesley,</p>
<p>Thanks for this post! Your advice on viewing the “grid” as a set of coordinates instead of an upsampled/downsampled/deformed version of the grid the input was (re-)sampled helped me understand code that I was having trouble understanding! This viewpoint is indeed more generally applicable, and the documentation would be greatly improved if it framed the usage of “grid_sample” (and its arguments) in these terms.</p>
<p>Cheers,<br>
Ishaan</p>

torch_kim: <p>This solution was incredible helpful for my understanding (thank you) but I could not reproduce it with the current implementation. So - if anyone else is wondering in 2023 - the default has changed to <code>align_corners = False</code>. I find the term <code>align_corners</code> a bit funny, because it is more straight forward to align the corners of the input and the target grid when set to false, as the outer borders fall onto -1 and  1 and the centroids fall within the range. For <code>align_corners = True</code> the grid does not refer directly to the centroids in my understanding. The documentation states: “For example, values <code>x = -1, y = -1</code> is the left-top pixel of <code>input</code> , and values <code>x = 1, y = 1</code> is the right-bottom pixel of <code>input</code> .” This confused me because in the default case the centre points of left-top pixel and the right-bottom pixel are not as indicated. Reproducing a given grid with grid_sample() validated this.</p>
<p>I am reproducing the above example with both indexing variants of torch.meshgrid(). I like to keep the notation of xy and ij separate.</p>
<pre><code class="lang-auto">input = torch.arange(4*4).view(1, 1, 4, 4).float()
print(input)
&gt; tensor([[[[ 0.,  1.,  2.,  3.],
          [ 4.,  5.,  6.,  7.],
          [ 8.,  9., 10., 11.],
          [12., 13., 14., 15.]]]])

# Create grid to upsample input
d = torch.linspace(-1, 1, 8)

### Using Cartesian xy indexing ###
# both grid axis are the same so order does not matter in this case.
xx, yy = torch.meshgrid((d, d), indexing = "xy")
# x,y order of last two dimensions of grid
grid = torch.stack((xx, yy), 2)
grid = grid.unsqueeze(0) # add batch dim

# 09/2022 default is align_corners = False
output = torch.nn.functional.grid_sample(input, grid, align_corners = True)
print(output)
&gt; tensor([[[[ 0.0000,  0.4286,  0.8571,  1.2857,  1.7143,  2.1429,  2.5714,
            3.0000],
          [ 1.7143,  2.1429,  2.5714,  3.0000,  3.4286,  3.8571,  4.2857,
            4.7143],
          [ 3.4286,  3.8571,  4.2857,  4.7143,  5.1429,  5.5714,  6.0000,
            6.4286],
          [ 5.1429,  5.5714,  6.0000,  6.4286,  6.8571,  7.2857,  7.7143,
            8.1429],
          [ 6.8571,  7.2857,  7.7143,  8.1429,  8.5714,  9.0000,  9.4286,
            9.8571],
          [ 8.5714,  9.0000,  9.4286,  9.8571, 10.2857, 10.7143, 11.1429,
           11.5714],
          [10.2857, 10.7143, 11.1429, 11.5714, 12.0000, 12.4286, 12.8571,
           13.2857],
          [12.0000, 12.4286, 12.8571, 13.2857, 13.7143, 14.1429, 14.5714,
           15.0000]]]])

### Using Matrix ij indexing ###
# This is the default and was used in the referance example I am trying to reporduce
# ii corresponds to rows in matrix indexing, so y-axis in Cartesian indexing
ii, jj = torch.meshgrid((d, d), indexing = "ij")
# Same: ii, jj = torch.meshgrid((d, d))
# Reverse order to match x, y Cartesian ordering: jj == xx, ii == yy in previous example
grid = torch.stack((jj, ii), 2)
grid = grid.unsqueeze(0) # add batch dim

# 09/2023 default is align_corners = False
output = torch.nn.functional.grid_sample(input, grid, align_corners = True)
print(output)
&gt; tensor([[[[ 0.0000,  0.4286,  0.8571,  1.2857,  1.7143,  2.1429,  2.5714,
            3.0000],
          [ 1.7143,  2.1429,  2.5714,  3.0000,  3.4286,  3.8571,  4.2857,
            4.7143],
          [ 3.4286,  3.8571,  4.2857,  4.7143,  5.1429,  5.5714,  6.0000,
            6.4286],
          [ 5.1429,  5.5714,  6.0000,  6.4286,  6.8571,  7.2857,  7.7143,
            8.1429],
          [ 6.8571,  7.2857,  7.7143,  8.1429,  8.5714,  9.0000,  9.4286,
            9.8571],
          [ 8.5714,  9.0000,  9.4286,  9.8571, 10.2857, 10.7143, 11.1429,
           11.5714],
          [10.2857, 10.7143, 11.1429, 11.5714, 12.0000, 12.4286, 12.8571,
           13.2857],
          [12.0000, 12.4286, 12.8571, 13.2857, 13.7143, 14.1429, 14.5714,
           15.0000]]]])
</code></pre>

ruiluwang2rylonw: <p>For torch 1.12.1, to get the same output, you should add “align_corners = True”</p>
<pre><code class="lang-auto">output = torch.nn.functional.grid_sample(input, grid, align_corners = True)
print(output)
tensor([[[[ 0.0000,  0.4286,  0.8571,  1.2857,  1.7143,  2.1429,  2.5714,
            3.0000],
          [ 1.7143,  2.1429,  2.5714,  3.0000,  3.4286,  3.8571,  4.2857,
            4.7143],
          [ 3.4286,  3.8571,  4.2857,  4.7143,  5.1429,  5.5714,  6.0000,
            6.4286],
          [ 5.1429,  5.5714,  6.0000,  6.4286,  6.8571,  7.2857,  7.7143,
            8.1429],
          [ 6.8571,  7.2857,  7.7143,  8.1429,  8.5714,  9.0000,  9.4286,
            9.8571],
          [ 8.5714,  9.0000,  9.4286,  9.8571, 10.2857, 10.7143, 11.1429,
           11.5714],
          [10.2857, 10.7143, 11.1429, 11.5714, 12.0000, 12.4286, 12.8571,
           13.2857],
          [12.0000, 12.4286, 12.8571, 13.2857, 13.7143, 14.1429, 14.5714,
           15.0000]]]])
</code></pre>

Udayraj123: <p>Great discussion on this thread. <a class="mention" href="/u/hmrishav_bandyopadhy">@Hmrishav_Bandyopadhy</a> - I was wondering if you managed to solve the problem later. If yes, can you share what approach worked?</p>
<p>Here to add my 2 cents from my searching on <strong>dewarping a document with skew/curvature</strong>.</p>
<ul>
<li>Found <a href="https://github.com/FelixHertlein/inv3d-generator/blob/94310ab4500363e12d337a1bd009a5cc27efb440/src/inv3d_generator/supplementary/warped_angle.py#L141" rel="noopener nofollow ugc">this code</a> which seems to create a grid using a uv map. They have also provided a demo <a href="https://felixhertlein.de/docrefine/home" rel="noopener nofollow ugc">here</a></li>
</ul>