Rotating non-square images using affine_grid

AndreiBarsan: <p>I am working on an architecture which requires applying a rigid transformation to a non-square image.</p>
<p>To this end, I am using a spatial transformer module. However, applying a (rigid) rotation to a non-square image inevitable produces distortion, as can be seen in this image:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3754af84c5ae9e45ea14bc09946afc7ade6d977b.png" data-download-href="https://discuss.pytorch.org/uploads/default/3754af84c5ae9e45ea14bc09946afc7ade6d977b" title="Screenshot%20from%202018-07-23%2019%3A55%3A02"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3754af84c5ae9e45ea14bc09946afc7ade6d977b_2_308x500.png" alt="Screenshot%20from%202018-07-23%2019%3A55%3A02" data-base62-sha1="7TtHkS3maxTT3rB9J9AdLahzKEr" width="308" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3754af84c5ae9e45ea14bc09946afc7ade6d977b_2_308x500.png, https://discuss.pytorch.org/uploads/default/original/2X/3/3754af84c5ae9e45ea14bc09946afc7ade6d977b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/3754af84c5ae9e45ea14bc09946afc7ade6d977b.png 2x" data-dominant-color="81537B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202018-07-23%2019%3A55%3A02</span><span class="informations">401×649 25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is it possible to avoid this issue without explicitly padding the input to make it square, and then cropping the result? Are there any parameters of, e.g., <code>grid_sample</code> I am missing, which could help get this job done?</p>
<p>Example of a correct rotation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f3c0c0ce86a1e2b9ba11f568604ca42708d3ad3e.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3c0c0ce86a1e2b9ba11f568604ca42708d3ad3e" title="Screenshot%20from%202018-07-23%2019%3A53%3A11"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f3c0c0ce86a1e2b9ba11f568604ca42708d3ad3e_2_347x500.png" alt="Screenshot%20from%202018-07-23%2019%3A53%3A11" data-base62-sha1="yMkZOCQwrkQe9KfNrpgDnHVGtrw" width="347" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f3c0c0ce86a1e2b9ba11f568604ca42708d3ad3e_2_347x500.png, https://discuss.pytorch.org/uploads/default/original/2X/f/f3c0c0ce86a1e2b9ba11f568604ca42708d3ad3e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f3c0c0ce86a1e2b9ba11f568604ca42708d3ad3e.png 2x" data-dominant-color="8E6689"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202018-07-23%2019%3A53%3A11</span><span class="informations">456×656 23.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is a self-contained example of the problem I am facing. The rotated output of the network ends up distorted, whereas I would prefer it to get transformed in a rigid manner, even if means some clipping occurs:</p>
<pre><code class="lang-auto">from math import sin, cos, pi

import matplotlib.pyplot as plt
import torch
import torch.nn.functional as F

# Enable this if inside notebook
#%matplotlib inline

DEG_TO_RAD = math.pi / 180.0

angle_deg = 45
angle_rad = angle_deg * DEG_TO_RAD
image_size = (1, 1, 480, 640)

rotation = torch.tensor([
    [ cos(angle_rad), sin(angle_rad), 0],
    [-sin(angle_rad), cos(angle_rad), 0],
]).unsqueeze(0).cuda()

image = torch.zeros(image_size).float().cuda()
image[:, :, :, 280:340] = 255

# Uncomment this to get the right result
# image_pad = torch.zeros((1, 1, 640, 640)).float().cuda()
# image_pad[:, :, 80:-80, :] = image
image_pad=image

grid = F.affine_grid(rotation, size=image_pad.size())

rotated = F.grid_sample(image_pad, grid)

plt.figure()
plt.imshow(image[0, 0].data.cpu().numpy())
plt.title("Original")

plt.figure()
rotated = rotated[:, :, 80:-80, :]
plt.imshow(rotated[0, 0].data.cpu().numpy())
plt.title("Rotated by {} degrees".format(angle_deg))
</code></pre>
<p>Bottom line: Can you rotate a non-square image without distorting it using the existing STN functionality without having to pad the image with zeros, or  implementing a custom version of <code>grid_sample</code>?</p>

cychai: <pre><code class="lang-auto">h,w=480, 640
rotation = torch.tensor([
    [ cos(angle_rad), sin(angle_rad)*h/w, 0],
    [-sin(angle_rad)*w/h, cos(angle_rad), 0],
]).unsqueeze(0).cuda()
</code></pre>

Yuhang_Lu: <p>It works for me, thank you! But I don’t understand why. Is it a bug of the affine_grid function?</p>

wuxiaji: <p>As for translation:</p>
<pre><code class="lang-python">translation_x = 100*2/w
translation_y = 10*2/h

rotation_yaw = torch.tensor([
    [ cos(angle_rad), sin(angle_rad)*h/w, translation_x],
    [-sin(angle_rad)*w/h, cos(angle_rad), translation_y],
]).unsqueeze(0).cuda()
</code></pre>

zhixuanli: <p>Thanks so much.<br>
Could you explain the reason?</p>