How does grid_sample(x, grid) work?

dingwang: <p>I don’t know how grid_sample() compute the output base on the grid and input?Does any one give me an example？Thank you.</p>

albanD: <p>Hi,<br>
The doc on <a href="http://pytorch.org/docs/0.3.1/nn.html#torch.nn.functional.grid_sample">grid_sample</a> contains details on how the output is computed. Let us know if you need more precisions.</p>

dingwang: <p>This function just passes parameters.The core is written in C. So I’m confused. Thank you for your apply！</p>

albanD: <p>Ho,<br>
the implementation is in <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c">this</a> file. In particular, the forward method (that computed the output) is <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L41">this</a> one.<br>
Here is a quick description of what the C code is doing:</p>
<ul>
<li>
<a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L48">here</a> check that the input is valid.</li>
<li>
<a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L49-L60">here</a> initialize stuff and make the output have the right size.</li>
<li>Then it iterates over the <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L62">batch</a> then <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L63">height</a> then <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L64">width</a>.</li>
<li>For each of these, it gets the input coordinated <code>ix</code> and <code>iy</code> <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L66-L71">here</a>.</li>
<li>
<a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L73-L81">here</a> it computes the 4 points around it that will be used for the interpolation.</li>
<li>
<a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L83-L87">here</a> it computes the weights for each of these 4 points.</li>
<li>
<a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L89-L99">here</a> it clips stuff if you asked for it.</li>
<li>Then it iterates over the <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L102">channels</a>.</li>
<li>It <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L105-L108">get</a> the 4 points values, <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L109">compute</a> the interpolation value then <a href="https://github.com/pytorch/pytorch/blob/f064c5aa33483061a48994608d890b968ae53fb5/aten/src/THNN/generic/SpatialGridSamplerBilinear.c#L110">save</a> it in the output tensor.</li>
</ul>

dingwang: <p>That’s exactly what I want ! Many thanks !<img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=5" title=":+1:" class="emoji" alt=":+1:"></p>

liangbright: <p>it seems grid_sample supports 2D and 3D images.</p>

Eurus-Holmes: <p>So how to implement  <code>padding mode</code> ?</p>

bnehoran: <p>Are you trying to <em>use</em> the <code>padding_mode</code> option of <code>grid_sample()</code>? If so, you can find it documented with an explanation <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.grid_sample" rel="nofollow noopener">here</a>, and if there is something in the explanation you don’t understand, feel free to ask.</p>
<p>Or are you trying to implement a new type of padding for <code>grid_sample()</code>? That is, other than the existing <code>zeros</code>, <code>border</code>, and <code>reflection</code>. If so, then that’s a completely different matter.</p>
<p>It’s not clear from your question which one you need help with.</p>

Eurus-Holmes: <p>Actually, I want to implement padding mode=zeros in tensorflow, so I want to know how to implement padding mode in pytorch source code…</p>

bnehoran: <p>Oh, okay, so here is the deal:<br>
PyTorch actually currently has 3 different underlying implementations of <code>grid_sample()</code> (a vectorized cpu 2D version, a nonvectorized cpu 3D version, and a CUDA implementation for both 2D and 3D), but their behavior is essentially supposed to be the same.</p>
<p>In my opinion, the easiest of the three to understand, if you just want to get the basic idea, is the CUDA version, which you can find <a href="https://github.com/pytorch/pytorch/blob/d3f6d5885d68d47d38a475d4fc9cf1611c91920c/aten/src/ATen/native/cuda/GridSampler.cu#L124" rel="nofollow noopener">here</a> for the 2D case.</p>
<p>The important lines in terms of <code>zero</code> padding are <a href="https://github.com/pytorch/pytorch/blob/d3f6d5885d68d47d38a475d4fc9cf1611c91920c/aten/src/ATen/native/cuda/GridSampler.cu#L199-L210" rel="nofollow noopener">these</a>, which implicitly perform the <code>zero</code> padding by calling the <code>within_bounds_2d()</code> to only add each term to the bilinear interpolation if it is in bounds. Any out-of-bounds grid points will get 0, of course, since nothing will then be added to the 0 from line <a href="https://github.com/pytorch/pytorch/blob/d3f6d5885d68d47d38a475d4fc9cf1611c91920c/aten/src/ATen/native/cuda/GridSampler.cu#L198" rel="nofollow noopener">198</a>.</p>
<p>Note that this out-of-bounds check does not affect the <code>border</code> and <code>reflection</code> padding modes, since in those cases, the grid points will have previously been brought in-bounds by the clip and reflect operations <a href="https://github.com/pytorch/pytorch/blob/d3f6d5885d68d47d38a475d4fc9cf1611c91920c/aten/src/ATen/native/cuda/GridSampler.cu#L164-L172" rel="nofollow noopener">here</a>.</p>
<p>Now, if you want to implement a similar zero padding behavior in TensorFlow, here’s how I would do it:<br>
Take a look at this <a href="https://github.com/tensorflow/models/blob/62184a96e740a3ecd65018acda42a11062b6d43d/research/transformer/spatial_transformer.py#L63" rel="nofollow noopener">_interpolate()</a> function. First of all, I should note that this function is not quite the same as the PyTorch <code>grid_sample()</code> in two ways:</p>
<ol>
<li>
<p>It is not meant to be called externally, only as part of <code>transformer()</code>, and so it actually takes the grid as two flattened tensors <code>x</code> and <code>y</code>. Of course, if you follow the code on <a href="https://github.com/tensorflow/models/blob/62184a96e740a3ecd65018acda42a11062b6d43d/research/transformer/spatial_transformer.py#L160-L170" rel="nofollow noopener">these</a> lines, you can figure out how to reformat your grid this way. (No need to <a href="https://github.com/tensorflow/models/blob/62184a96e740a3ecd65018acda42a11062b6d43d/research/transformer/spatial_transformer.py#L166" rel="nofollow noopener">multiply by an affine matrix</a> if you already have the grid you want to use. This just produces an affine grid.)</p>
</li>
<li>
<p>It miscalculates the conversion from the [-1,+1] of the grid to pixel indices, and is in fact off by half a pixel. If half a pixel doesn’t bother you, then great. If it does, fixing it is a bit more involved, but also possible.</p>
</li>
</ol>
<p>So supposing you can get over these two hurdles, how do you implement zero padding? First thing you have to do is remove the coordinate clipping on <a href="https://github.com/tensorflow/models/blob/master/research/transformer/spatial_transformer.py#L91-L94" rel="nofollow noopener">these</a> lines, since you want to know when a grid point is out-of-bounds. Second, you have to add a check for out-of-bounds grid points and either zero out their values <a href="https://github.com/tensorflow/models/blob/62184a96e740a3ecd65018acda42a11062b6d43d/research/transformer/spatial_transformer.py#L109-L112" rel="nofollow noopener">here</a> or their interpolation weights <a href="https://github.com/tensorflow/models/blob/62184a96e740a3ecd65018acda42a11062b6d43d/research/transformer/spatial_transformer.py#L119-L122" rel="nofollow noopener">here</a>.</p>
<p>That should be more than enough if you’re running this on GPU. If you’re running it on CPU, then you need to be careful <a href="https://github.com/tensorflow/models/blob/62184a96e740a3ecd65018acda42a11062b6d43d/research/transformer/spatial_transformer.py#L109-L112" rel="nofollow noopener">here</a> that you’re not going out of bound <em>in the flat 1D tensor</em>. That’s because otherwise, you will <a href="https://www.tensorflow.org/api_docs/python/tf/gather" rel="nofollow noopener">probably get an out-of-bounds error</a>.</p>
<p>Hopefully this is enough to get you where you need.</p>

Eurus-Holmes: <p>Oh, I see, thank you so much!</p>

TG_N: <p>Sorry to bother you, I have some doubts about the creation of the parameter “grid” in “grid_sample()”. Can I use CNN to generate the grid of N<em>H</em>W*2 directly, and get a valid grid through training? Is this idea feasible?</p>

harryhan618: <p><a class="mention" href="/u/bnehoran">@bnehoran</a> Thank you for your detailed explanation!<br>
I have a question about the API. Why <code>grid</code>, i.e. the coordinates to sample from source tensor, has to be in range [-1, 1]? Since in the c++ / cuda  implementation, the [-1, 1] coordinate still has to be converted to the scale of source tensor size, why not require the value <code>grid</code> to be real coordinate rather than scaled one?</p>

bnehoran: <p><a class="mention" href="/u/tg_n">@TG_N</a> Yes! That is a great idea!<br>
In fact, this is how a number of convolutional neutral nets for image registration and optical flow are structured.<br>
The CNN predicts a flow/grid/mapping/displacement field which is then passed to <code>grid_sample()</code> or the like (you may have to do some conversions to fit the conventions), and then backpropogate from the result.</p>

bnehoran: <p><a class="mention" href="/u/harryhan618">@harryhan618</a> that’s a really good question.</p>
<p>The convention of representing the image space using [-1, 1] comes from the original 2015 <a href="https://arxiv.org/abs/1506.02025" rel="nofollow noopener">paper</a> that introduced Spatial Transformer Networks (after which PyTorch’s <code>grid_sample()</code> and other similar functions in other ML frameworks are based).</p>
<p>A reasonable justification of this convention is that it allows coordinates to be specified without respect to the resolution of the underlying image. That is, the same image at different resolutions should still be sampled at the same location.<br>
So it’s a sort of resolution-agnostic representation.</p>
<p>That said, I agree that there should be an option to pass in the grid in unnormalized pixel units. I’ve pushed for such an option in the past, and maybe if I have extra time at some point, I’ll go through and implement it.</p>

gfotedar: <p>Is there a way I can pad with a custom value which is not zero? Let’s say I want to pad with 1?</p>

bnehoran: <p><a class="mention" href="/u/gfotedar">@gfotedar</a> There’s no direct way of padding with a non-zero value, but there are some workarounds you can try. Of course, it depends on your use case whether these workarounds are good enough for you.</p>
<p>I think the easiest one is this:<br>
Suppose you want to use a padding value of <code>val</code>.  Start by subtracting <code>val</code> from the input image you’re intending to sample. Then pass to <code>grid_sample()</code> with <code>padding_mode="zeros"</code>. Then just add back <code>val</code> to the result.</p>
<p>There are other, more complicated ways, like adding a border to your image and using <code>"border"</code> padding, but I would highly recommend <em>not</em> doing this since you’d have to adjust the grid to match, and it’s just very easy to get it wrong.</p>
<p>What is it specifically that you’re trying to acomplish, by the way?</p>

GtxMiracle: <p>Hi, I want to interpolate the value from a discreted image. Speciafically, I have a grid, a mask and an input. Only the points whose mask is 1 of the input will contribute the interpolation. Is there any way to achieve this with grid_sample function?</p>

hqabcxyxz: <p>= =||| I am confused about example of <code>reflection</code> <code>padding_mode</code>…</p>
<p>In doc:</p>
<p><code>padding_mode="reflection"</code> : use values at locations reflected by the border for out-of-bound grid locations. For location far away from the border, it will keep being reflected until becoming in bound, e.g., (normalized) pixel location <code>x = -3.5</code> reflects by border <code>-1</code> and becomes <code>x' = 1.5</code> , then reflects by border <code>1</code> and becomes <code>x'' = -0.5</code> .</p>
<p>but why reflects by border<code>1</code> becomes <code>x'' = -0.5</code>  not <code>x'' = 0.5</code>?</p>
<p>if <code>x = 6.5</code>,it does mean that <code>x' = -4.5</code>,<code>x'' = 2.5</code>,<code>x''' = -0.5</code> or <code>x''' = 0.5</code>?</p>
<p>Looking forward to your reply and thx for your help</p>

draco: <p>I’m not sure-<br>
does grid_sample actually perform forward warping or backward warping?</p>
<p>maybe it’s better to ask about the combination of:<br>
grid = F.affine_grid(theta, x.size())<br>
x = F.grid_sample(x, grid)</p>
<p>does it look at every pixel location in the dst image and asks- what source pixels should I place here? (backward)<br>
or does it simply moves the src image to the dst image using theta (and hence might leave holes)? (forward)</p>
<p>many thanks!</p>