How to decide patch_size/kernel size for nn.unfold

sparshgarg23: <p>So,I am looking into the aspect of patchifying an image in VIT using nn.unfold</p>
<p>One of the parameters for unfolding is the kernel size/patch size.</p>
<p>As such what is the ideal value for the patch size so as to get a proper patched resutt similar to sliding windows.</p>
<p>When I try it on the image shown below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c21838484f48c4e12837328ee40c41b91531771f.png" data-download-href="https://discuss.pytorch.org/uploads/default/c21838484f48c4e12837328ee40c41b91531771f" title="0000000001"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c21838484f48c4e12837328ee40c41b91531771f_2_690x208.png" alt="0000000001" data-base62-sha1="rH2wf7IMbL4mpvbrXVG0JsSCi3J" width="690" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c21838484f48c4e12837328ee40c41b91531771f_2_690x208.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c21838484f48c4e12837328ee40c41b91531771f_2_1035x312.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c21838484f48c4e12837328ee40c41b91531771f.png 2x" data-dominant-color="5C5C5C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">0000000001</span><span class="informations">1242×375 258 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>my result comes out as this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e43f2f56f257fad564eba6ce4743f844e2817ecb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e43f2f56f257fad564eba6ce4743f844e2817ecb" title="download"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e43f2f56f257fad564eba6ce4743f844e2817ecb_2_500x500.jpeg" alt="download" data-base62-sha1="wzaeEs5cctlKc7CAi4H0UZRQoKT" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e43f2f56f257fad564eba6ce4743f844e2817ecb_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e43f2f56f257fad564eba6ce4743f844e2817ecb_2_750x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e43f2f56f257fad564eba6ce4743f844e2817ecb_2_1000x1000.jpeg 2x" data-dominant-color="717171"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">download</span><span class="informations">1252×1252 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Clearly there is something going wrong here because the  patches seem to be mixed and they don’t resmbel the original picture.If i choose a smaller patch size only the top region of the orginal image gets patched.</p>
<p>My code is as follows</p>
<pre><code class="lang-auto">import torch
from torch import nn
import cv2
import numpy as np
import matplotlib.pyplot as plt

class Patchify(nn.Module):
    def __init__(self, patch_size=56):
        super().__init__()
        self.p = patch_size
        self.unfold = torch.nn.Unfold(kernel_size=patch_size, stride=patch_size)

    def forward(self, x):
        # x -&gt; B c h w
        bs, c, h, w = x.shape
        
        x = self.unfold(x)
        # x -&gt; B (c*p*p) L
        
        # Reshaping into the shape we want
        a = x.view(bs, c, self.p, self.p, -1).permute(0, 4, 1, 2, 3)
        # a -&gt; ( B no.of patches c p p )
        return a

#Read image
patch=Patchify(patch_size=75)
image_path='/content/0000000001.png'
img_src = image_path
image = cv2.imread(img_src)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image = image.astype('float32') / 255.0  # Normalize to [0, 1]
image = torch.from_numpy(image)
image = image.permute(2,0,1)
image = image.unsqueeze(0) #to add the batch dimensio

#Patchify
p=patch(image)
X=p.squeeze()
X.shape

#Display patches
from mpl_toolkits.axes_grid1 import ImageGrid
def plot_patches(tensor):
    fig = plt.figure(figsize=(16, 16))
    grid = ImageGrid(fig, 111, nrows_ncols=(8, 8), axes_pad=0.1)

    for i, ax in enumerate(grid):
        patch = tensor[i].permute(1, 2, 0).numpy() 
        ax.imshow(patch)
        ax.axis('off')

    plt.show()

plot_patches(X)
</code></pre>
<p>As such is it an issue with how I am viewing the result or should I try to use larger patch size.<br>
thanks</p>