[Solved]How to do the interpolating of optical flow?

Alex_Hex: <p>I used tensorflow before. I just learn pytorch a few days ago.<br>
I want to use pytorch to do something I did with tensorflow before. But I meet a problem, I do not know how to do the interpolating of optical flow.</p>
<p>That is mean, the optical flow has 2 channels such as (Batch_size, 2, height, width).  One channel for delta x, another for delta y, using the base grid to add the flow and then sample from origin image.</p>
<p>By the way, in the 2017 CVPR, there is also a paper mention it. Deep feature flow.</p>
<p>Maybe I just start to learn pytorch, it is a bit difficult for me to do interpolating with pytorch. Can I directly use for loop to handle each pixel? Or If you would like to help me, can you show me some demos or examples.</p>
<p>Thanks a lot.</p>

chsasank: <p>Are you looking for <a href="http://pytorch.org/docs/master/nn.html?highlight=upsample#torch.nn.functional.upsample" rel="nofollow noopener">Upsample</a>?</p>

Alex_Hex: <p>Yes. I find what I want: <strong>torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’).</strong><br>
You help me a lot, thanks!</p>

Alex_Hex: <p>…I find this API <strong>torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’).</strong> , but not in pytorch version 0.1.12…</p>

chsasank: <p>That’s probably because it’s in master branch but not released yet. You can build pytorch from source to get the master version. Read <a href="http://Readme.md" rel="nofollow noopener">Readme.md</a> in pytorch/pytorch repo for instructions on how to install from source.</p>

Alex_Hex: <p>Ok. I solve it. Thanks!</p>

zhuyi490: <p>Hi Alex,</p>
<p>I have the same question here, I want to use optical flow to warp images. Should I directly use torch.nn.functional.grid_sample(input, grid, mode=‘bilinear’) and nothing else?</p>
<p>I saw some people said that the grid(which in my case, is the optical flow) should be in the range of [-1,1]. Is that true? Because the range of my optical flow is like [-20,20]. So should I divide the flow by 20 before I feed it to the grid_sample function? And is there anything else to notice about? Thank you very much.</p>

lxtGH: <p>Hi zhuyi490<br>
I have the same question here, do you know how to do it now ?</p>

IanYeung: <p>Hi zhuyi490,</p>
<p>Have you solved this problem now?</p>

sniklaus: <p>You could have a look at the following.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/sniklaus/pytorch-spynet/blob/master/run.py#L135" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/sniklaus/pytorch-spynet/blob/master/run.py#L135" target="_blank" rel="nofollow noopener">sniklaus/pytorch-spynet/blob/master/run.py#L135</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="125" style="counter-reset: li-counter 124 ;">
<li>			def forward(self, tensorInput, tensorFlow):</li>
<li>				if hasattr(self, 'tensorGrid') == False or self.tensorGrid.size(0) != tensorFlow.size(0) or self.tensorGrid.size(2) != tensorFlow.size(2) or self.tensorGrid.size(3) != tensorFlow.size(3):</li>
<li>					torchHorizontal = torch.linspace(-1.0, 1.0, tensorFlow.size(3)).view(1, 1, 1, tensorFlow.size(3)).expand(tensorFlow.size(0), 1, tensorFlow.size(2), tensorFlow.size(3))</li>
<li>					torchVertical = torch.linspace(-1.0, 1.0, tensorFlow.size(2)).view(1, 1, tensorFlow.size(2), 1).expand(tensorFlow.size(0), 1, tensorFlow.size(2), tensorFlow.size(3))</li>
<li>
</li>
<li>					self.tensorGrid = torch.cat([ torchHorizontal, torchVertical ], 1).cuda()</li>
<li>				# end</li>
<li>
</li>
<li>				tensorFlow = torch.cat([ tensorFlow[:, 0:1, :, :] / ((tensorInput.size(3) - 1.0) / 2.0), tensorFlow[:, 1:2, :, :] / ((tensorInput.size(2) - 1.0) / 2.0) ], 1)</li>
<li>
</li>
<li class="selected">				return torch.nn.functional.grid_sample(input=tensorInput, grid=(self.tensorGrid + tensorFlow).permute(0, 2, 3, 1), mode='bilinear', padding_mode='border')</li>
<li>			# end</li>
<li>		# end</li>
<li>
</li>
<li>		self.modulePreprocess = Preprocess()</li>
<li>
</li>
<li>		self.moduleBasic = torch.nn.ModuleList([ Basic(intLevel) for intLevel in range(6) ])</li>
<li>
</li>
<li>		self.moduleBackward = Backward()</li>
<li>	# end</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


lxtGH: <p>Thanks for your code,  I want to make sure one thing.<br>
if I want to warp image1(source) to image2(target), the input optical flow is from image2 to image1 rather than image1 to image2 ?? Why?</p>

Qing_En: <p>hi <a class="mention" href="/u/lxtgh">@lxtGH</a>. If I want to warp image1 to image2, the input optical flow is computed from image2 to image1. And should I input this optical flow and image1 together to get image2?</p>

lxtGH: <p>Yes, according to the previous code, the source is from image2<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/NVlabs/PWC-Net/blob/master/PyTorch/models/PWCNet.py#L139" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/NVlabs/PWC-Net/blob/master/PyTorch/models/PWCNet.py#L139" target="_blank" rel="nofollow noopener">NVlabs/PWC-Net/blob/master/PyTorch/models/PWCNet.py#L139</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="129" style="counter-reset: li-counter 128 ;">
<li>    self.dc_conv6 = conv(64,       32,  kernel_size=3, stride=1, padding=1,  dilation=1)</li>
<li>    self.dc_conv7 = predict_flow(32)</li>
<li>
</li>
<li>    for m in self.modules():</li>
<li>        if isinstance(m, nn.Conv2d) or isinstance(m, nn.ConvTranspose2d):</li>
<li>            nn.init.kaiming_normal(m.weight.data, mode='fan_in')</li>
<li>            if m.bias is not None:</li>
<li>                m.bias.data.zero_()</li>
<li>
</li>
<li>
</li>
<li class="selected">def warp(self, x, flo):</li>
<li>    """</li>
<li>    warp an image/tensor (im2) back to im1, according to the optical flow</li>
<li>
</li>
<li>    x: [B, C, H, W] (im2)</li>
<li>    flo: [B, 2, H, W] flow</li>
<li>
</li>
<li>    """</li>
<li>    B, C, H, W = x.size()</li>
<li>    # mesh grid </li>
<li>    xx = torch.arange(0, W).view(1,-1).repeat(H,1)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

983511703: <p>How to do forward warping using img1 and forward flow?</p>

jiahaogeng: <p>Hi, have you solved this problem? If you have, could you share your solution with me?</p>

Benedict_K: <p>Just invert the flow and then warp from img1 to to img2. The flow is a 2d vector field so just multiplying by -1 would invert it. Magnitude remains, just directions is turned by 180 deg.</p>

Arnaud_Roussel: <p>This only works if your flow is uniform. If you use - flow with the forward flow applied on image1 in the grid sampler you will move the wrong pixels. It’s easier to see it in 1D. If you have [0, 0, 1, 0, 0] and a flow of [0, 0, +1, 0, 0] then the warp image is [0, 0, ?? , 1, 0] where ?? being occlusion.</p>
<p>But what the sampler does is to sample from a source given the coordinates in a grid. Your grid with -flow would be:<br>
[0, 1, 2-1 = 1, 3, 4]<br>
After sampling you get:<br>
[0, 0, 0, 0, 0]<br>
because it moved pixel at index 1 instead of pixel at index 2.</p>
<p>That is why the code above is suggesting to use the backward flow instead. In our case that flow would be [0, 0, 0, -1, 0] and then with grid [0, 1, 2, 3-1 = 2, 0]. When using grid sampler from im1 with that grid it will yield:<br>
[0, 0, 1, 1, 0, ]<br>
Which is what we should have without an occlusion mask which should be at index 2.</p>
<p>It sometimes works when you have some general movement in the frame but I prefer to use backward flow instead.</p>

soulslicer: <p>Hi all, can i know what is the units that your flow fields are in for the above function. I tried using pixel values directly but I get some extremely warped results</p>

YOGESH_SANKU: <p>Hi, I am trying to implement Deep feature flow in PyTorch. Do you have any resources?. I have gone through Scalsol and MM implementation but, if there is any other simple implementation it would be helpful. Thanks in advance.</p>

lizhihao6: <p>You could try this one. ( I am still updating it with bp). <a href="https://github.com/lizhihao6/Forward-Warp" class="inline-onebox" rel="noopener nofollow ugc">GitHub - lizhihao6/Forward-Warp: An optical flow forward warp's lib with backpropagation using pytorch.</a></p>