How to convert an affine transform matrix into "theta" to use torch.nn.functional.affine_grid()?

ljqcava: <p>Hi, all!<br>
I met a problem when using <code>torch.nn.functioal.affine_grid(theta,size)</code>.<br>
I’d like to know how to convert an affine transformation matrix described in <code>scipy/skimage.transform/opencv</code> into a right argument <code>theta</code> in <code>torch.nn.functioal.affine_grid(theta,size)</code>?</p>
<p>Now suppose we want to apply an affine transormation <img src="https://latex.codecogs.com/gif.latex?T" alt width="13" height="12"> on an image <img src="https://latex.codecogs.com/gif.latex?I" alt width="9" height="12"> with shape=(H,W,3), where<br>
<img src="https://latex.codecogs.com/gif.latex?T%3D%5Cbegin%7Bbmatrix%7D%20a_1%20%26%20a_2%20%26%20t_x%5C%5C%20a_3%20%26%20a_4%20%26%20t_y%5C%5C%200%20%26%200%20%26%201%20%5Cend%7Bbmatrix%7D" alt width="135" height="67"></p>
<p>What’s the right <code>theta</code> which should be used in <code>torch.nn.functioal.affine_grid(theta,size)</code> ?</p>

xiang: <p>Hi, I met same problem, Have you solved it?</p>

xiang: <p>Hi, i have solved it.Code is below.</p>
<p>import cv2<br>
import torch.nn.functional as F<br>
import skimage.transform as trans<br>
import numpy as np</p>
<p>def convert_image_np(inp):<br>
“”“Convert a Tensor to numpy image.”""<br>
inp = inp.numpy().transpose((1, 2, 0))<br>
inp = (inp*255).astype(np.uint8)<br>
return inp</p>
<p>def param2theta(param, w, h):<br>
param = np.linalg.inv(param)<br>
theta = np.zeros([2,3])<br>
theta[0,0] = param[0,0]<br>
theta[0,1] = param[0,1]*h/w<br>
theta[0,2] = param[0,2]*2/w + param[0,0] + param[0,1] - 1<br>
theta[1,0] = param[1,0]*w/h<br>
theta[1,1] = param[1,1]<br>
theta[1,2] = param[1,2]*2/h + param[1,0] + param[1,1] - 1<br>
return theta<br>
tr = trans.estimate_transform(‘affine’, src=src, dst=dst)<br>
M = tr.params[0:2,:]<br>
img = cv2.warpAffine(image,M,(w,h))<br>
theta = param2theta(tr.params, w, h)<br>
image = Variable(image).unsqueeze(0).cuda()<br>
theta = Variable(theta).unsqueeze(0).cuda()<br>
grid = F.affine_grid(theta,image.size())<br>
img_ = F.grid_sample(image,grid)<br>
img_ = convert_image_np(img_.data.cpu().squeeze(0))</p>
<p>The visualizations of img and img_ should be same. Hope it can help you!</p>

ljqcava: <p>Thanks for your sharing.</p>
<pre><code class="lang-python">theta[0,0] = param[0,0]
theta[0,1] = param[0,1]*h/w
theta[0,2] = param[0,2]*2/w + theta[0,0] + theta[0,1] - 1
theta[1,0] = param[1,0]*w/h
theta[1,1] = param[1,1]
theta[1,2] = param[1,2]*2/h + theta[1,0] + theta[1,1] - 1
</code></pre>
<p>I have solved this issue by the above code. Maybe your code only works under some non-trival cases.</p>

xiang: <p>I think it works under general case, not some non-trival cases.</p>

HectorAnadon: <p>My recomendation is using Kornia <a href="https://torchgeometry.readthedocs.io/en/latest/geometry.transform.html?highlight=warp_affine#kornia.geometry.transform.warp_affine" rel="nofollow noopener">https://torchgeometry.readthedocs.io/en/latest/geometry.transform.html?highlight=warp_affine#kornia.geometry.transform.warp_affine</a></p>
<p>Example: <a href="https://torchgeometry.readthedocs.io/en/latest/warp_affine.html" rel="nofollow noopener">https://torchgeometry.readthedocs.io/en/latest/warp_affine.html</a></p>

oldriver-wang: <p>I have a similar problem.<br>
I already use ITK-snap to process an affine registration : 3D volume image A to B.<br>
Then I get a matrix:</p>
<p>0.97464 0.0364447 0.00781475 1.95549<br>
0.00375859 0.922498 0.0297265 -8.87417<br>
-0.00727863 -0.0145295 0.961844 -1.10425<br>
0 0 0 1</p>
<p>I think it’s incompatibal in F.affine_grid theta.<br>
The shape of affine_grid shape is (N, 3, 4)<br>
likes:  R | T<br>
[<br>
[a, b, c,  t0],<br>
[e, f,  g,  t1],<br>
[h, i,   j,   t2]<br>
]<br>
I think the anterior three rows of the matrix is theta, but it’s wrong.<br>
the R is likely equal to theta[:3, :3],(I am not sure)<br>
but the T (t0, t1, t2) must be incompatible.</p>
<p>How can I get a right theta?<br>
By the way, how can I know what does the torch.nn.functioal.affine_grid really do ?</p>

ljqcava: <p>Hi <a class="mention" href="/u/oldriver-wang">@oldriver-wang</a></p>
<p>It seems that the current PyTorch API doesn’t support 3D affine transformation. Up to now, <code>affine_grid()</code>  and <code>grid_sample()</code> can only support 2D affine transformation (especially, 2D perspective transformation is not supported yet).</p>
<p>As the advices from <a class="mention" href="/u/hectoranadon">@HectorAnadon</a>, to implement complicated geometric transformations, you can try <a href="https://github.com/kornia/kornia" rel="nofollow noopener">Kornia</a>.</p>

oldriver-wang: <p>Hi <a class="mention" href="/u/ljqcava">@ljqcava</a><br>
Thanks for your advice!<br>
It seems that the 3d affine_grid is available in torch 1.5.1 but not in 1.2.0.</p>

ljqcava: <p><a class="mention" href="/u/oldriver-wang">@oldriver-wang</a> Oh, you are right. <code>affine_grid()</code> begin to support 3D transformation. It’s good news and glad to see that! However, it doesn’t support perspective transformation.</p>

Odium: <p><a class="mention" href="/u/oldriver-wang">@oldriver-wang</a> Have you solved this problem? Faced the same problem, can not solve it</p>

hm77: <p>Still having this issue in 2025. Kornia was a life-saver. Some of the documentation links in this thread are broken, so here’s what worked for me.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://kornia.readthedocs.io/en/stable/geometry.transform.html#kornia.geometry.transform.Affine">
  <header class="source">
      <img src="https://kornia.readthedocs.io/en/stable/_static/kornia_logo_favicon.png" class="site-icon" width="42" height="42">

      <a href="https://kornia.readthedocs.io/en/stable/geometry.transform.html#kornia.geometry.transform.Affine" target="_blank" rel="noopener nofollow ugc">kornia.readthedocs.io</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://kornia.readthedocs.io/en/stable/geometry.transform.html#kornia.geometry.transform.Affine" target="_blank" rel="noopener nofollow ugc">kornia.geometry.transform - Kornia</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>