Affine_grid and grid_sample: why is my image not rotated in the right direction?

username: Laurent1
cooked: <p>Hi,</p>
<p>In PyTorch 1.8, I want to break down the constituents of a Spatial Transformer, in particular see how <code>affine_grid</code> and <code>grid_sample</code> work. So I came up with the following piece of code where I apply a rotation of angle pi/8 to an image:</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt
from matplotlib.image import imread
import numpy as np
import torch
import torch.nn.functional as F

img = imread("food.jpg")
w, h, _ = img.shape
img = img / 255.
print("image shape: ", img.shape)
x = torch.from_numpy(img)
x = x.type(torch.FloatTensor)
# add a batch size N=1
x = torch.unsqueeze(x,0)
# x is of size NxHxWx3, we need Nx3xHxW 
x = x.permute(0,3,1,2)
# affine transformation, size Nx2x3
angle = np.pi/8
theta = [ [ np.cos(angle)*w/h, -np.sin(angle)    , 0. ],
          [ np.sin(angle),      np.cos(angle)*w/h, 0. ] ]
theta = torch.Tensor(theta).unsqueeze(0)
# grid is of size NxHxWx2
grid = F.affine_grid(theta, x.size(), align_corners=False)
x = F.grid_sample(x, grid, align_corners=False)
# x is of size Nx3xHxW, we need HxWx3 
x = x.squeeze(0)
x = x.permute(1,2,0)

fig = plt.figure(figsize=(16,8), facecolor='white')
ax = fig.add_subplot(1,3,1)
ax.imshow(img)
ax.axis('off')
ax.set_title("Source (U)", fontsize=16)
ax = fig.add_subplot(1,3,2)
ax.imshow(x)
ax.axis('off')
ax.set_title("Target (V)", fontsize=16)
fig.set_tight_layout(True)
fig.savefig('foo.png')
ax = fig.add_subplot(1,3,3)
ax.set_title("Grid", fontsize=16)
grid = grid.squeeze(0)
grid = grid.permute(2,0,1)
# grid is now of size 2xHxW
grid = grid.reshape(2,-1).numpy()
ax.scatter(grid[0,::100],grid[1,::100], s=0.01) 
plt.show()
</code></pre>
<p>and I got</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d5046f3be18f55e5145a59bde922eef0d3bf09a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1d5046f3be18f55e5145a59bde922eef0d3bf09a" title="b"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d5046f3be18f55e5145a59bde922eef0d3bf09a_2_690x345.jpeg" alt="b" data-base62-sha1="4bjQ3E1GhVnUDRqPsyynufoZ5bA" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d5046f3be18f55e5145a59bde922eef0d3bf09a_2_690x345.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d5046f3be18f55e5145a59bde922eef0d3bf09a_2_1035x517.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d5046f3be18f55e5145a59bde922eef0d3bf09a_2_1380x690.jpeg 2x" data-dominant-color="7E7B77"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">b</span><span class="informations">1600×800 348 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, if I understand correctly, the grid (on the right) is put on the lattice of the input feature maps (on the left) and afterwards a bilinear interpolation is carried out. If so, I would expect the bowl to be rotated in the opposite direction (i.e. tilted to the right downwards). Why isn’t it so?</p>
<p>thanks for the help.</p>

username: Sylvain55
cooked: <p>Hello !</p>
<p>On your scatter plot of the grid you need to add:</p>
<pre><code class="lang-auto">ax.invert_yaxis()
</code></pre>
<p>Because (-1, -1) is the top left corner, and (1,1) the bottom right corner so y axis is inverted.</p>
<p>See grid_sample documentation: <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html#torch.nn.functional.grid_sample" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.grid_sample — PyTorch 2.1 documentation</a></p>
<blockquote>
<p><code>grid</code> specifies the sampling pixel locations normalized by the <code>input</code> spatial dimensions. Therefore, it should have most values in the range of <code>[-1, 1]</code> . For example, values <code>x = -1, y = -1</code> is the left-top pixel of <code>input</code> , and values <code>x = 1, y = 1</code> is the right-bottom pixel of <code>input</code> .</p>
</blockquote>