Warp image using optical flow

username: brinapingu
cooked: <p>Hi, I’m trying to warp an image using a flow map(calculated using FlowNet2). I have read other similar posts in the forum which suggested using torch.nn.functional.grid_sample. I’ve never used this function before so I tried doing one based on how PWCNet used it <a href="https://github.com/NVlabs/PWC-Net/blob/master/PyTorch/models/PWCNet.py#L139" class="inline-onebox" rel="noopener nofollow ugc">PWC-Net/PyTorch/models/PWCNet.py at master · NVlabs/PWC-Net · GitHub</a></p>
<p>I modified it a bit to fit my data dimension but they’re basically the same implementation.</p>
<pre><code class="lang-auto">
def warp(x,flo):
     B,H,W,C = x.size()
     # mesh grid
     xx = torch.arange(0,W).view(1,-1).repeat(H,1)
     yy = torch.arange(0,H).view(-1,1).repeat(1,W)
     
     xx = xx.view(1,H,W,1).repeat(B,1,1,1)
     yy = yy.view(1,H,W,1).repeat(B,1,1,1)
     
     
     grid = torch.cat((xx,yy),3).float()
    
     if x.is_cuda:
         grid = grid.cuda()

     vgrid = Variable(grid) + flo
    
     ## scale grid to [-1,1]
     vgrid[:,:,:,0] = 2.0*vgrid[:,:,:,0].clone()/max(W-1,1)-1.0
     vgrid[:,:,:,1] = 2.0*vgrid[:,:,:,1].clone()/max(H-1,1)-1.0
     
     x = x.permute(0,3,1,2)

     output = torch.nn.functional.grid_sample(x,vgrid)
     mask = torch.autograd.Variable(torch.ones(x.size()))
     mask = torch.nn.functional.grid_sample(mask,vgrid)
    
     mask[mask&lt;0.9999]=0
     mask[mask&gt;0]=1
    
     return output*mask
</code></pre>
<p>I ran the FlowNet2 to get the optical flow from the first image to the second image and then warped them using the function above. (Apparently, new users can only post one image so I can’t post the picture I’m using but it’s from the Berkeley Deep Drive Dataset)</p>
<pre><code class="lang-auto">def show(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg,(1,2,0)),interpolation='nearest')
    plt.show()

img0 = cv2.imread("val/picture0013.png").astype(np.float32)
img0 = np.array([img0])

flow = read_flow("work/inference/run.epoch-0-flow-field/000000.flo")
flow = np.array([flow])

img0 = torch.from_numpy(img0)
flow = torch.from_numpy(flow)

result2 = warp(img0,flow)
show(result2[0])
</code></pre>
<p>This is what I got</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/1f19385035c82a42cac3199ed3deb479dda0f810.png" alt="Screenshot%20from%202019-09-12%2017-14-18" data-base62-sha1="4r6PMeA35QuwwLdq7dFwWwbQ9yw" width="599" height="384"></p>
<p>Am I misunderstanding what grid_sample does??</p>

username: LiUzHiAn
cooked: <p>Hi, <a class="mention" href="/u/brinapingu">@brinapingu</a>,</p>
<p>The reason you get a poor warped image is that the function <code>grid_sample()</code> needs an input tensor in the range [0,1] rather than [0,255] (which is the default range of <code>cv2</code>). So you just have to normalize your <code>img0</code>. BTW, you have to be careful when warping an RGB image (cv2 reads as BRG order), <code>img=img[:,:,::-1]</code> may help assume the <code>img</code> is a numpy array with shape <code>[H,W,3]</code>.</p>
<p>But I had the another question. Can anyone explain the functionality of the <code>mask</code> in the <code>warp()</code> function? In my understanding, the result of <code>grid_sample </code> is exactly the warped result.</p>
<p>Thanks in advance.</p>

username: BingfengHan-BIT
cooked: <p>Hi, i’m facing a problem when i try to warp img0 to img1, there are many black areas around the img, I’m wondering what these area mean. And another problem is, my warped image has different color area with the ground-truth img. Any idea? thanks in advance!</p>