Block Matching algorithm?

tuvovan: <p>Hi, I’m trying to implement the block matching algorithm using torch.</p>
<p>The basic idea is to take a patch say 8 by 8, then define a search window say 50 by 50 and we need to find the top 10 most similar patches inside that search window.</p>
<p>I had idea to search for top 10 most similar patches in the whole image using unfold + calculate distance + sort function but not quite sure how to do if there is a search window.</p>
<p>Thanks!</p>

InnovArul: <p>This might be of some help:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/ClementPinard/Pytorch-Correlation-extension">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/ClementPinard/Pytorch-Correlation-extension" target="_blank" rel="noopener">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/bbb0f77b8198961684284730d4c8f8572ca2477f81cf000ba63ebed242c05bc3/ClementPinard/Pytorch-Correlation-extension" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/ClementPinard/Pytorch-Correlation-extension" target="_blank" rel="noopener">GitHub - ClementPinard/Pytorch-Correlation-extension: Custom implementation...</a></h3>

  <p>Custom implementation of Corrleation Module. Contribute to ClementPinard/Pytorch-Correlation-extension development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


tuvovan: <p>you mean this might help with the distance step?</p>

InnovArul: <p>Yes. This package may help to match patches between feature maps.<br>
I assumed the matching should happen at every pixel, where you consider the <code>8x8</code> patch at every pixel position and match it within the <code>50x50</code> window around it. Maybe this is not the case?</p>
<p>Not sure of your use case. If its matching one <code>8x8</code> patch with a particular <code>50x50</code> window, you could achieve it by convolving (<code>F.conv2d</code>?) this <code>8x8</code> patch over the <code>50x50</code> window, then sort the responses.</p>

tuvovan: <p>yeah right! the matching should happen at every pixel.</p>
<p>Let say there is a 256x256 image, for each pixel of that image, take a patch of 8x8 around it and start looking for similar patches inside a search window of 50x50.<br>
2 nested for loops might work, not sure if there is a better solution using vector or tensor…</p>

InnovArul: <p>Then my assumption was correct. Thanks for clarifying.<br>
The <code>Pytorch-Correlation-extension</code> package I linked above provides the necessary functionality.<br>
You can look at the examples provided in the Github and see if it works for you.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/ClementPinard/Pytorch-Correlation-extension#example">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/ClementPinard/Pytorch-Correlation-extension#example" target="_blank" rel="noopener">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/b83d96e07786cd9be7946fcb2bf60c15c7a27265e936f9713d086b8a1f5d6ce9/ClementPinard/Pytorch-Correlation-extension" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/ClementPinard/Pytorch-Correlation-extension#example" target="_blank" rel="noopener">GitHub - ClementPinard/Pytorch-Correlation-extension: Custom implementation...</a></h3>

  <p>Custom implementation of Corrleation Module. Contribute to ClementPinard/Pytorch-Correlation-extension development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


J_Johnson: <p>Here is one way you could approach the problem in a parallelized way. This assumes you want to compare “patches” with l1loss(though you could easily substitute MSE).</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

def l1loss(patch1, patch2, dim=1):
    return torch.mean(torch.abs((patch1-patch2)), dim=dim)

def get_patches(images, kernel_size=(8,8)):
    return F.unfold(images, kernel_size) # output returns size (batch, flattened patch, patches)

images=torch.randn((1,1,50, 50))

patches=get_patches(images)
b, hw, p = patches.shape
patches_exp=patches.unsqueeze(3).expand(b, hw, p, p)

z=torch.triu_indices(p,p,1) #mapping of the triangular upper matrix

losses = l1loss(patches_exp.triu(), patches_exp.rot90(k=1, dims=[2,3]).triu())
losses=losses[losses!=0]
print(losses.size()) #size should be p*((p-1)/2) which represents non-zeros for triu when diagonals are zero
values, indices = torch.topk(losses, k=10, largest=False) # top 10 values and their indices
print(values, indices)
index=0
x_val=z[0][indices[index]] # get the indices from the triu_indices mapping at the selected topk index
y_val=z[1][indices[index]]

print(x_val, y_val, values[index]) #check the indices and value
print(l1loss(patches[:,:,x_val],patches[:,:,y_val])) #check that the loss matches when the indices are applied to the original unfolded patches
</code></pre>
<p><code>.triu()</code> is used to eliminate duplicate calculations or getting the losses between the same patch.</p>
<p>Updated to correct an error and include a usage example with <code>topk()</code>.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.topk.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.topk.html</a></p>
<p>Note that with larger images, you’re going to run into some major memory problems and may need to split up the operation into an iterable.</p>