How to warp the image with optical flow and grid_sample

username: hunters0813
cooked: <p>Hello, guys. I’ve some questions of image warping with optical flows.</p>
<p>I would like to test warp the images by grid_sample, and I’ve got samples(two images with flow) from PWC-Net as follows:<br>
input image from → <a href="https://github.com/NVlabs/PWC-Net/tree/master/PyTorch/data" class="inline-onebox" rel="noopener nofollow ugc">PWC-Net/PyTorch/data at master · NVlabs/PWC-Net · GitHub</a><br>
flow file from → <a href="https://github.com/NVlabs/PWC-Net/tree/master/PyTorch/tmp" class="inline-onebox" rel="noopener nofollow ugc">PWC-Net/PyTorch/tmp at master · NVlabs/PWC-Net · GitHub</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ec86da6346cb21f12943ec8832a49aaf65d6c88.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3ec86da6346cb21f12943ec8832a49aaf65d6c88" title="test3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ec86da6346cb21f12943ec8832a49aaf65d6c88_2_690x293.jpeg" alt="test3" data-base62-sha1="8Xp1audzOxBV8QtFcR3eMWWDk8w" width="690" height="293" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ec86da6346cb21f12943ec8832a49aaf65d6c88_2_690x293.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ec86da6346cb21f12943ec8832a49aaf65d6c88.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ec86da6346cb21f12943ec8832a49aaf65d6c88.jpeg 2x" data-dominant-color="444444"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test3</span><span class="informations">1024×436 459 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>def warp(x, flo):<br>
“”"<br>
warp an image/tensor (im2) back to im1, according to the optical flow</p>
<pre><code>x: [B, C, H, W] (im2)
flo: [B, 2, H, W] flow

"""
B, C, H, W = x.size()
# mesh grid
xx = torch.arange(0, W).view(1 ,-1).repeat(H ,1)
yy = torch.arange(0, H).view(-1 ,1).repeat(1 ,W)
xx = xx.view(1 ,1 ,H ,W).repeat(B ,1 ,1 ,1)
yy = yy.view(1 ,1 ,H ,W).repeat(B ,1 ,1 ,1)
grid = torch.cat((xx ,yy) ,1).float()

if x.is_cuda:
    grid = grid.cuda()
vgrid = Variable(grid) + flo

# scale grid to [-1,1]
vgrid[: ,0 ,: ,:] = 2.0 *vgrid[: ,0 ,: ,:].clone() / max( W -1 ,1 ) -1.0
vgrid[: ,1 ,: ,:] = 2.0 *vgrid[: ,1 ,: ,:].clone() / max( H -1 ,1 ) -1.0

vgrid = vgrid.permute(0 ,2 ,3 ,1)
flo = flo.permute(0 ,2 ,3 ,1)
output = F.grid_sample(x, vgrid)
mask = torch.autograd.Variable(torch.ones(x.size())).cuda()
mask = F.grid_sample(mask, vgrid)

mask[mask &lt;0.9999] = 0
mask[mask &gt;0] = 1

return output*mask
</code></pre>
<p>test = cv2.readOpticalFlow(‘reference_frame_0010.flo’)<br>
test = torch.from_numpy(test)<br>
test = test.view(1, 2, H, W)<br>
test = Variable(test).cuda()</p>
<p>test_img2 = cv2.imread(‘frame_0011.png’)<br>
test_img2 = torch.from_numpy(test_img2)<br>
test_img2 = test_img2.view(1, 3, H, W)<br>
test_img2 = Variable(test_img2).cuda()</p>
<p>out = warp(test_img2, test)<br>
out = out.view(H, W, 3)<br>
out = out.cpu()<br>
out = np.float32(out)<br>
cv2.imwrite(‘test3.png’, out*255)</p>
<p>I’ve run the code above and got a result image. As you can see the last image, it is not I expected.<br>
It seems like several part of the images with grid.<br>
Is it right to get a warped target image with optical flow?<br>
If not, how can I get whole image looks like the input color images?</p>
<p>Please, help me to solve this problem.</p>

username: hunters0813
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8bc182b5f0cff75ad939fb4b76e7207576f6ba58.png" data-download-href="https://discuss.pytorch.org/uploads/default/8bc182b5f0cff75ad939fb4b76e7207576f6ba58" title="frame_0010"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8bc182b5f0cff75ad939fb4b76e7207576f6ba58_2_690x293.png" alt="frame_0010" data-base62-sha1="jWkXHrZjLtzcPEdhOoKjeo2MfmE" width="690" height="293" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8bc182b5f0cff75ad939fb4b76e7207576f6ba58_2_690x293.png, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8bc182b5f0cff75ad939fb4b76e7207576f6ba58.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8bc182b5f0cff75ad939fb4b76e7207576f6ba58.png 2x" data-dominant-color="424545"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">frame_0010</span><span class="informations">1024×436 488 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This image is one of the input.</p>

username: ptrblck
cooked: <p>You would have to call <code>transpose</code> on the numpy array or <code>.permute</code> on the tensor to change the dimension order.<br>
The <code>view</code> operation (<code>test.view(1, 2, H, W)</code>) will interleave the pixel values and thus create the wrong image.</p>

username: hunters0813
cooked: <p>I’ve changed all view operations to .permute. It works very well.<br>
Thanks a lot!   : )</p>

username: LiUzHiAn
cooked: <aside class="quote no-group" data-username="hunters0813" data-post="1" data-topic="71531">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hunters0813/48/21149_2.png" class="avatar"> hunters0813:</div>
<blockquote>
<p>cv2.imwrite(‘test3.png’, out*255)</p>
</blockquote>
</aside>
<p>I executed the warp funcion to get back img1 as you post, but the warped image quality was not so good.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8a3013d18b84e9189ee7026d775bbc631366fff.png" data-download-href="https://discuss.pytorch.org/uploads/default/b8a3013d18b84e9189ee7026d775bbc631366fff" title="warped"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8a3013d18b84e9189ee7026d775bbc631366fff_2_690x293.png" alt="warped" data-base62-sha1="qln5D6Ag43b4FJptxwWougZjMLB" width="690" height="293" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8a3013d18b84e9189ee7026d775bbc631366fff_2_690x293.png, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8a3013d18b84e9189ee7026d775bbc631366fff.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8a3013d18b84e9189ee7026d775bbc631366fff.png 2x" data-dominant-color="414343"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">warped</span><span class="informations">1024×436 452 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>. Pay attention to the weapon held by the girl and the shoulder part of the man. Is it because the optical flow was not estimated well?</p>

username: Yiran_Xu
cooked: <p>Hey, did you solve this issue? I think I met the same problem as yours</p>

username: Yiran_Xu
cooked: <p>Hey <a class="mention" href="/u/liuzhian">@LiUzHiAn</a> , did you solve this issue? I think I met the same problem as yours</p>

username: LiUzHiAn
cooked: <p><a class="mention" href="/u/yiran_xu">@Yiran_Xu</a> Not yet.Have you tried different optical flows?</p>

username: Yiran_Xu
cooked: <p>Yep, I tried different flows and found this issue happened to all of them</p>