Cv2 remap in pytorch?

thehappyidiot: <p>Is there a functionality close to cv2’s remap? pytorch’s grid_sample has deviating behaviors for the things I’ve tried with it.</p>

albanD: <p>Hi,</p>
<p>From quick look at cv’s doc, the two should be doing the same thing no? What kind of issues do you have with <code>grid_sample</code>?</p>

thehappyidiot: <p>I’ve tried optical flow warping to generate the next frame from the previous, but regardless of the flo file associated with the test images or one created using cv2’s farneback dense optical flow, the grid sample warped frame has visible differences.</p>
<aside class="quote quote-modified" data-post="1" data-topic="99298">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thehappyidiot/48/29769_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/image-warping-for-backward-flow-using-forward-flow-matrix-optical-flow/99298">Image warping for backward flow using forward flow matrix- optical flow</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi 
The code I’m using to warp the image is 

B, C, H, W = image_B.size()

xx = torch.arange(0, W).view(1,-1).repeat(H,1)

yy = torch.arange(0, H).view(-1,1).repeat(1,W)

xx = xx.view(1,1,H,W).repeat(B,1,1,1)

yy = yy.view(1,1,H,W).repeat(B,1,1,1)

grid = torch.cat((xx,yy),1).float()

vgrid = grid + flow_AB

vgrid[:,0,:,:] = 2.0*vgrid[:,0,:,:].clone() / max(W-1,1)-1.0

vgrid[:,1,:,:] = 2.0*vgrid[:,1,:,:].clone() / max(H-1,1)-1.0

warped_image = torch.nn.functional.grid_sample(image_B, vgrid.perm…
  </blockquote>
</aside>
<p>
(from a previous question of mine for the details)</p>
<p>If I use cv2’s remap, I get a perfect frame, so I thought their functionality might be different, but the documentation seems to read as if they’re basically the same</p>

albanD: <p>One notable difference maybe is that grid_sample does not take coordinates as input but values in [-1, 1] that tell you where to read in the input image.<br>
Are you warping your grid values properly to match this?</p>

thehappyidiot: <p>Yep, I took care of that. Since the flow was just subtracted, the result could be normalized easily to that range without it affecting the differences between the two functions, implementation wise (at least there).</p>

albanD: <p>Sounds good.<br>
And is remap also doing bilinear interpolation? Maybe that can lead to surprising results?</p>

thehappyidiot: <p>I’ve tested out nearest and bilinear with it, but the differences in results for both of them are negligible</p>

Ramiro451: <p>I have also same issue anyone have suggestion for this so please reply<a href="https://www.skylightpaycard.biz/" rel="noopener nofollow ugc">.</a></p>

Felix_Lessange: <p>Good news : I believe you can now do this in pytorch 1.11.0</p>
<pre><code class="lang-auto">
import torch

def remap_values(remapping, x):
    index = torch.bucketize(x.ravel(), remapping[0])
    return remapping[1][index].reshape(x.shape)


remapping = torch.arange(0, 256).cuda(), torch.randperm(256).cuda()
images_batch = torch.randint(0, 256, (16, 224, 224, 3)).cuda()
remapped_batch = remap_values(remapping, images_batch)
</code></pre>

Maghoumi: <p><a class="mention" href="/u/thehappyidiot">@thehappyidiot</a> Any updates on this?</p>
<p>I also had the same observation recently. Basically, the output from <code>grid_sample()</code> looks different from what <code>cv2.remap()</code> produces in that the former appears “scaled” compared to the latter. Best I can describe this scaled effect is what <a href="https://discuss.pytorch.org/t/maintain-aspect-ratio-during-grid-sample/101742">this question has mentioned.</a>. I see that depending on the inputs, the output from <code>cv2.remap()</code> leaves the invalid areas of pixel values unpopulated, whereas <code>grid_sample()</code> appears have scaled the image in a way that these areas do not exist.</p>
<p>Not sure how to replicate the behavior of <code>cv2.remap()</code> using <code>grid_sample()</code>. Maybe this difference is intended and there’s no way around it…?</p>

doubleZ: <p>I have also met the translation problem, here is my code in <code>cv2.remap()</code> and <code>torch.nn.functional.grid_sample()</code>, it is just suitable for my task.</p>
<p>My mission is to project the <code>ref_img</code> and <code>ref_depth</code> from a reference view to another source view.</p>
<p>The code in <code>Numpy</code> and <code>cv2</code> style:</p>
<pre><code class="lang-python">def reproject_with_depth(img_ref, depth_ref, intrinsics_ref, extrinsics_ref, intrinsics_src, extrinsics_src):
    width, height = depth_ref.shape[1], depth_ref.shape[0]

    x_ref, y_ref = np.meshgrid(np.arange(0, width), np.arange(0, height))
    x_ref, y_ref = x_ref.reshape([-1]), y_ref.reshape([-1])
    
    xyz_ref = np.matmul(np.linalg.inv(intrinsics_src),
                        np.vstack((x_ref, y_ref, np.ones_like(x_ref))) * depth_ref.reshape([-1]))    
   
    xyz_src = np.matmul(np.matmul(extrinsics_ref, np.linalg.inv(extrinsics_src)),
                        np.vstack((xyz_ref, np.ones_like(x_ref))))[:3]
    
    K_xyz_src = np.matmul(intrinsics_ref, xyz_src)
    xy_src = K_xyz_src[:2] / K_xyz_src[2:3]

    x_src = xy_src[0].reshape([height, width]).astype(np.float32)
    y_src = xy_src[1].reshape([height, width]).astype(np.float32)
    
    sampled_depth_src = cv2.remap(depth_ref, x_src, y_src, interpolation=cv2.INTER_LINEAR)
    sampled_img_src = cv2.remap(img_ref, x_src, y_src, interpolation=cv2.INTER_LINEAR)

    return sampled_depth_src, sampled_img_src
</code></pre>
<p>And this is the translation into <code>torch</code> style:</p>
<pre><code class="lang-python">def reproject_with_depth(img_ref, depth_ref, intrinsics_ref, extrinsics_ref, intrinsics_src, extrinsics_src):
    B, width, height = depth_ref.shape[0], depth_ref.shape[2], depth_ref.shape[1]

    y_ref, x_ref = torch.meshgrid([torch.arange(0, height, dtype=torch.float32, device=depth_ref.device), torch.arange(0, width, dtype=torch.float32, device=depth_ref.device)])
    y_ref, x_ref = y_ref.contiguous(), x_ref.contiguous()
    x_ref, y_ref = x_ref.reshape([-1]), y_ref.reshape([-1])
    # reference 3D space
    xyz_ref = torch.matmul(torch.inverse(intrinsics_src),
                        torch.stack((x_ref, y_ref, torch.ones_like(x_ref))).unsqueeze(0).repeat(B, 1, 1) * depth_ref.reshape([B, 1, -1]))
    
    xyz_src = torch.matmul(torch.matmul(extrinsics_ref, torch.inverse(extrinsics_src)),
                        torch.cat([xyz_ref, torch.ones_like(x_ref).unsqueeze(0).repeat(B,1,1)], dim=1))[:,:3]
    
    K_xyz_src = torch.matmul(intrinsics_ref, xyz_src)
    xy_src = K_xyz_src[:, :2] / K_xyz_src[:, 2:3]

    x_src = xy_src[:, 0].reshape([B, height, width]).float()
    y_src = xy_src[:, 1].reshape([B, height, width]).float()

    grid = torch.stack((x_src/((width-1)/2)-1, y_src/((height-1)/2)-1), dim=3)
    sampled_depth_src = F.grid_sample(depth_ref.unsqueeze(1), grid.view(B, height, width, 2), mode='bilinear', padding_mode='zeros').squeeze(1)
    sampled_img_src = F.grid_sample(img_ref, grid.view(B, height, width, 2), mode='bilinear', padding_mode='zeros')

    return sampled_depth_src, sampled_img_src
</code></pre>
<p>The <strong>essential translation principles</strong> you should obey I think:</p>
<ul>
<li>You should consider the batch channel <code>B</code> in torch style. And the <code>device</code> parameter is also the case.</li>
<li>The position sampling in the torch should be mapped into [-1, 1], hence, you should use <code>/((width-1)/2)-1</code> for x-axis and <code>/((height-1)/2)-1)</code> for y-axis.</li>
</ul>
<p>I hope it will be helpful to you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

edgarriba: <p>Remap is implemented in Kornia</p>
<p><a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.remap" class="onebox" target="_blank" rel="noopener nofollow ugc">https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.remap</a></p>
<p>You also have all the utils needed for the depth stuff</p>

Fade_Outisname: <p>Thank you very much! It’s very helpful.</p>