Affine transformation matrix paramters conversion

amirid: <p>Hi all,</p>
<p>I want to rotate an image about a specific point. First I create the Transformation matrices for moving the center point to the origin, rotating and then moving back to the first point, then apply the transform using <code>affine_grid</code> and <code>grid_sample</code> functions. But the resulting image is not what it should be. Once I tested these parameters by applying them on the image using scipy and it works.</p>
<pre><code class="lang-python">import torch.nn.functional as F
mat_move = torch.eye(3)
mat_move[0,2] = -x_center*2/my_image.shape[2]
mat_move[1,2] = -y_center*2/my_image.shape[3]
mat_rotate = torch.eye(3)
mat_rotate[0, 0] = cos_theta[0][0]
mat_rotate[0, 1] = -sin_theta[0][0]
mat_rotate[1, 0] = sin_theta[0][0]
mat_rotate[1, 1] = cos_theta[0][0]
mat_move_back = torch.eye(3)
mat_move_back[0,2] = x_center*2/my_image.shape[2]
mat_move_back[1,2] = y_center*2/my_image.shape[3]
rigid_transform = torch.mm(mat_move_back, torch.mm(mat_rotate, mat_move))
M = Variable(torch.zeros([1, 2, 3])).cuda()
M[0, 0, 0] = rigid_transform[0, 0]
M[0, 0, 1] = rigid_transform[0, 1]
M[0, 0, 2] = rigid_transform[0, 2]
M[0, 1, 0] = rigid_transform[1, 0]
M[0, 1, 1] = rigid_transform[1, 1]
M[0, 1, 2] = rigid_transform[1, 2]
grid = F.affine_grid(M, vertebrae.size())
vertebrae = F.grid_sample(vertebrae.float(), grid)
</code></pre>

ptrblck: <p>How are you comparing the results?<br>
It seems <code>scipy.ndimage.affine_transform</code> uses pixel values for the translation part, while <code>F.affine_grid</code> seems to want values in the range <code>[-1, 1]</code> (which you already provided).</p>
<p>This code tries to rotate and translate a line:</p>
<pre><code class="lang-auto">x = torch.eye(10).view(1, 1, 10, 10)
theta = torch.zeros(1, 2, 3)
angle = np.pi/2.
theta[:, :, :2] = torch.tensor([[np.cos(angle), -1.0*np.sin(angle)],
                                [np.sin(angle), np.cos(angle)]])
theta[:, :, 2] = 0.5

grid = F.affine_grid(theta, x.size())
x_trans = F.grid_sample(x, grid)

plt.imshow(x.squeeze().numpy())
plt.imshow(x_trans.squeeze().numpy())
</code></pre>
<p>Based on the values of <code>grid</code> the operation should work.<br>
The visualizations however look a bit strange, but this might be due to some interpolation,<br>
Maybe someone knows this better.</p>

amirid: <p>For example, the following code compares the results of the the same operation using pytorch and scipy. The results are not the same.</p>
<pre><code class="lang-auto">x = torch.eye(10).view(1, 1, 10, 10)
theta = torch.zeros(1, 2, 3)
angle = np.pi/2.
theta[:, :, :2] = torch.tensor([[np.cos(angle), -1.0*np.sin(angle)],
                                [np.sin(angle), np.cos(angle)]])
theta[:, :, 2] = 0.5

grid = F.affine_grid(theta, x.size())
x_trans = F.grid_sample(x, grid)

plt.imshow(x.squeeze().numpy())
plt.figure()
plt.imshow(x_trans.squeeze().numpy())

x = np.eye(10)
theta = np.zeros([2, 3])
angle = np.pi/2.

theta = np.float32([[np.cos(angle),-1.0*np.sin(angle),5],[np.sin(angle),np.cos(angle),5]]) # or 7.5 instead of 5

x_trans = scipy.ndimage.affine_transform(x, theta, order=1)

plt.figure()
plt.imshow(x)
plt.figure()
plt.imshow(x_trans)
plt.show()
</code></pre>

ljqcava: <p>Hi, I also meet this problem. I just want to know how to convert an affine transformation matrix described in <code>scipy/skimage.transform/opencv</code> into a right argument <code>theta</code> in <code>torch.nn.functioal.affine_grid(theta,size)</code>?</p>
<p>Now suppose we want to apply an affine transormation <img src="https://latex.codecogs.com/gif.latex?T" alt width="13" height="12"> on an image <img src="https://latex.codecogs.com/gif.latex?I" alt width="9" height="12"> with shape=(H,W,3), where<br>
<img src="https://latex.codecogs.com/gif.latex?T%3D%5Cbegin%7Bbmatrix%7D%20a_1%20%26%20a_2%20%26%20t_x%5C%5C%20a_3%20%26%20a_4%20%26%20t_y%5C%5C%200%20%26%200%20%26%201%20%5Cend%7Bbmatrix%7D" alt width="135" height="67"></p>
<p>What’s the right <code>theta</code> which should be used in <code>torch.nn.functioal.affine_grid(theta,size)</code> ?</p>

xiang: <p>@ <a href="https://discuss.pytorch.org/u/amirid">amirid</a><br>
Hi, i have solved it.Code is below.</p>
<p>import cv2<br>
import torch.nn.functional as F<br>
import skimage.transform as trans<br>
import numpy as np</p>
<p>def convert_image_np(inp):<br>
“”“Convert a Tensor to numpy image.”""<br>
inp = inp.numpy().transpose((1, 2, 0))<br>
inp = (inp*255).astype(np.uint8)<br>
return inp</p>
<p>def param2theta(param, w, h):<br>
param = np.linalg.inv(param)<br>
theta = np.zeros([2,3])<br>
theta[0,0] = param[0,0]<br>
theta[0,1] = param[0,1]*h/w<br>
theta[0,2] = param[0,2]*2/w + param[0,0] + param[0,1] - 1<br>
theta[1,0] = param[1,0]*w/h<br>
theta[1,1] = param[1,1]<br>
theta[1,2] = param[1,2]*2/h + param[1,0] + param[1,1] - 1<br>
return theta<br>
tr = trans.estimate_transform(‘affine’, src=src, dst=dst)<br>
M = tr.params[0:2,:]<br>
img = cv2.warpAffine(image,M,(w,h))<br>
theta = param2theta(tr.params, w, h)<br>
image = Variable(image).unsqueeze(0).cuda()<br>
theta = Variable(theta).unsqueeze(0).cuda()<br>
grid = F.affine_grid(theta,image.size())<br>
img_ = F.grid_sample(image,grid)<br>
img_ = convert_image_np(img_.data.cpu().squeeze(0))</p>
<p>The visualizations of img and img_ should be same. Hope it can help you!</p>

jiangxiluning: <p><a class="mention" href="/u/xiang">@xiang</a> hi, the iii I transformed using cv2.warpAffine is not always  the same with x using grid_sample, any idea? thanks</p>
<pre><code class="lang-auto">    def forward(self, feature_map, boxes, mapping):
        '''

        :param feature_map:  N * 128 * 128 * 32
        :param boxes: M * 8
        :param mapping: mapping for image
        :return: N * H * W * C
        '''

        max_width = 0
        boxes_width = []
        cropped_images = []
        matrixes = []
        images = []

        for img_index, box in zip(mapping, boxes):
            feature = feature_map[img_index]  # B * H * W * C
            images.append(feature)

            x1, y1, x2, y2, x3, y3, x4, y4 = box / 4  # 521 -&gt; 128


            # show_box(feature, box / 4, 'ffffff', isFeaturemap=True)

            rotated_rect = cv2.minAreaRect(np.array([[x1, y1], [x2, y2], [x3, y3], [x4, y4]]))
            box_w, box_h = rotated_rect[1][0], rotated_rect[1][1]

            width = feature.shape[2]
            height = feature.shape[1]

            if box_w &lt;= box_h:
                box_w, box_h = box_h, box_w

            mapped_x1, mapped_y1 = (0, 0)
            mapped_x4, mapped_y4 = (0, self.height)

            width_box = math.ceil(self.height * box_w / box_h)
            max_width = width_box if width_box &gt; max_width else max_width

            mapped_x2, mapped_y2 = (width_box, 0)

            # affine_matrix = cv2.getAffineTransform(np.float32([(x1, y1), (x2, y2), (x4, y4)]), np.float32([
            #     (mapped_x1, mapped_y1), (mapped_x2, mapped_y2), (mapped_x4, mapped_y4)
            # ]))

            affine_matrix = trans.estimate_transform('affine', np.float32([(x1, y1), (x2, y2), (x4, y4)]), np.float32([
                (mapped_x1, mapped_y1), (mapped_x2, mapped_y2), (mapped_x4, mapped_y4)
            ]))

            affine_matrix = affine_matrix.params[0:2, :]
            iii = cv2.warpAffine(feature.permute(1,2,0).cpu().numpy().astype(np.uint8),
                                 affine_matrix, (width, height))

            cv2.imshow('img', iii)
            cv2.waitKey()

            affine_matrix = self.param2theta(affine_matrix, width, height)

            grid = torch.nn.functional.affine_grid(torch.tensor(affine_matrix[np.newaxis]), feature[np.newaxis].size())
            x = torch.nn.functional.grid_sample(feature[np.newaxis], grid)
            x = x[0].permute(1, 2, 0).detach().cpu().numpy()
            x = x.astype(np.uint8)

            cv2.imshow('img', x)
            cv2.waitKey()

            matrixes.append(torch.tensor(affine_matrix, device=feature.device))
            boxes_width.append(width_box)

        matrixes = torch.stack(matrixes)
        images = torch.stack(images)
        grid = nn.functional.affine_grid(matrixes, images.size())
        feature_rotated = nn.functional.grid_sample(images, grid)


        channels = feature_rotated.shape[1]
        cropped_images_padded = torch.zeros((len(feature_rotated), channels, self.height, max_width),
                                            dtype=feature_rotated.dtype,
                                            device=feature_rotated.device)
</code></pre>

xiang: <p>Hi, i can not upload the code now. qq 6553947</p>

amirid: <p>In</p>
<pre><code class="lang-auto">def param2theta(param, w, h):
    param = np.linalg.inv(param)
</code></pre>
<p>How do you invert a non square matrix?</p>

jiangxiluning: <p>just append [0,0,1] horizontally</p>

jiangxiluning: <p>this is wrong</p>
<pre><code class="lang-auto">def param2theta(param, w, h):
        param = np.linalg.inv(param)
        theta = np.zeros([2,3])
        theta[0,0] = param[0,0]
        theta[0,1] = param[0,1]*h/w
        theta[0,2] = param[0,2]*2/w + param[0,0] + param[0,1] - 1
        theta[1,0] = param[1,0]*w/h
        theta[1,1] = param[1,1]
        theta[1,2] = param[1,2]*2/h + param[1,0] + param[1,1] - 1
        return theta
</code></pre>
<p>below is correct</p>
<pre><code class="lang-auto">def param2theta(param, w, h):
        param = np.linalg.inv(param)
        theta = np.zeros([2,3])
        theta[0,0] = param[0,0]
        theta[0,1] = param[0,1]*h/w
        theta[0,2] = param[0,2]*2/w + theta[0,0] + theta[0,1] - 1
        theta[1,0] = param[1,0]*w/h
        theta[1,1] = param[1,1]
        theta[1,2] = param[1,2]*2/h + theta[1,0] + theta[1,1] - 1
        return theta
</code></pre>

ljqcava: <p><a class="mention" href="/u/jiangxiluning">@jiangxiluning</a><br>
Yes. I have already solved this problem with the same method. But I don’t have further idea why it works. Could you share your explanations?</p>

Budel: <p>The param2theta function corresponds to this equation<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/6/669c364eeeb7efab6c9ee7dd20401779c65d5d08.gif" alt="eq" data-base62-sha1="eDJnRLRPCP8exvsWF8NZwQS83R6" width="235" height="75"> ,<br>
where you resize, so that the width and height equate to 2 and shift by -1, to put it in the range of [-1, 1] for each dimension.</p>
<p>see<br>
<a href="https://www.wolframalpha.com/input/?i=" class="inline-onebox" rel="noopener nofollow ugc">- Wolfram|Alpha</a>{{2/w, 0, -1}, {0, 2/h, -1}, {0, 0, 1}} * {{a, b, c}, {d, e, f}, {0, 0, 1}} * {{2/w, 0, -1}, {0, 2/h, -1}, {0, 0, 1}}^-1</p>

xiang: <p>Demo code can be found,<a href="https://github.com/wuneng/WarpAffine2GridSample" rel="nofollow noopener">https://github.com/wuneng/WarpAffine2GridSample</a>.</p>

jiangxiluning: <p>hi, <a class="mention" href="/u/budel">@Budel</a> could you explain how do you get the equation?<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>

edgarriba: <p>to mimic cv2.warpAffine in pytorch you can use kornia’s version: <a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.warp_affine" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.warp_affine</a></p>

xpngzhng: <p><a class="mention" href="/u/jiangxiluning">@jiangxiluning</a> actually, it is not difficult to derive, as long as you know pytorch uses normalized coordinates, which can be learn from other discussion in this forum, below actually shows theta to param, the inverse matrix will be param to theta<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/267e2ed3c929cab6ff7df90e051628e7468f45b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/267e2ed3c929cab6ff7df90e051628e7468f45b7" title="affine"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267e2ed3c929cab6ff7df90e051628e7468f45b7_2_469x499.png" alt="affine" data-base62-sha1="5uwulLKPEuO4CSrNsoutcXmhf6f" width="469" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267e2ed3c929cab6ff7df90e051628e7468f45b7_2_469x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267e2ed3c929cab6ff7df90e051628e7468f45b7_2_703x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267e2ed3c929cab6ff7df90e051628e7468f45b7_2_938x998.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">affine</span><span class="informations">1752×1864 172 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Giammarco07: <p>Hi, great code! I went through the various steps and managed to get to the final transformation matrix as <a class="mention" href="/u/budel">@Budel</a> and <a class="mention" href="/u/xpngzhng">@xpngzhng</a> did, and according to my mathematical calculations I shouldn’t do the inverse of “param” before. Obviously applying the transformation without the inverse I don’t get the correct result as yours… Could you please explain to me why it should be done? Why do you invert the matrix before making the transformation (change of basis)?</p>

theRealSuperMario: <p>Okay, so to drive this point completely home, here is how you convert front and back between <code>opencv</code> and <code>torch.F.affine_grid</code></p>
<pre><code class="lang-python">
def get_N(W, H):
    """N that maps from unnormalized to normalized coordinates"""
    N = np.zeros((3, 3), dtype=np.float64)
    N[0, 0] = 2.0 / W
    N[0, 1] = 0
    N[1, 1] = 2.0 / H
    N[1, 0] = 0
    N[0, -1] = -1.0
    N[1, -1] = -1.0
    N[-1, -1] = 1.0
    return N


def get_N_inv(W, H):
    """N that maps from normalized to unnormalized coordinates"""
    # TODO: do this analytically maybe?
    N = get_N(W, H)
    return np.linalg.inv(N)


def cvt_MToTheta(M, w, h):
    """convert affine warp matrix `M` compatible with `opencv.warpAffine` to `theta` matrix
    compatible with `torch.F.affine_grid`

    Parameters
    ----------
    M : np.ndarray
        affine warp matrix shaped [2, 3]
    w : int
        width of image
    h : int
        height of image

    Returns
    -------
    np.ndarray
        theta tensor for `torch.F.affine_grid`, shaped [2, 3]
    """
    M_aug = np.concatenate([M, np.zeros((1, 3))], axis=0)
    M_aug[-1, -1] = 1.0
    N = get_N(w, h)
    N_inv = get_N_inv(w, h)
    theta = N @ M_aug @ N_inv
    theta = np.linalg.inv(theta)
    return theta[:2, :]


def cvt_ThetaToM(theta, w, h, return_inv=False):
    """convert theta matrix compatible with `torch.F.affine_grid` to affine warp matrix `M`
    compatible with `opencv.warpAffine`.

    Note:
    M works with `opencv.warpAffine`.
    To transform a set of bounding box corner points using `opencv.perspectiveTransform`, M^-1 is required

    Parameters
    ----------
    theta : np.ndarray
        theta tensor for `torch.F.affine_grid`, shaped [2, 3]
    w : int
        width of image
    h : int
        height of image
    return_inv : False
        return M^-1 instead of M.

    Returns
    -------
    np.ndarray
        affine warp matrix `M` shaped [2, 3]
    """
    theta_aug = np.concatenate([theta, np.zeros((1, 3))], axis=0)
    theta_aug[-1, -1] = 1.0
    N = get_N(w, h)
    N_inv = get_N_inv(w, h)
    M = np.linalg.inv(theta_aug)
    M = N_inv @ M @ N
    if return_inv:
        M_inv = np.linalg.inv(M)
        return M_inv[:2, :]
    return M[:2, :]

# small test
theta = cvt_MToTheta(M, w, h)
M2 = cvt_ThetaToM(theta, w, h)
assert np.allclose(M, M2)


</code></pre>
<p>I will also make a PR to the example proposed <a class="mention" href="/u/xiang">@xiang</a> [<a href="https://github.com/wuneng/WarpAffine2GridSample/pull/4" rel="nofollow noopener">here</a>]</p>

maze09: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>how can I obtain the new coordinates of the corners of the transformed image, especially if I scale and rotate the original image.</p>
<p>Best Matthias</p>

ptrblck: <p>If you are using an affine transformation (e.g. by applying a transformation matrix), you could apply this transformation to the original edge coordinates to get the transformed coordinates.</p>