How to extract patches from an image

tmc: <p>Given an image, I need to extract patches with stride of 32, after changing the smaller dimension of the image to 227.</p>
<p>How can I do that?</p>

ptrblck: <p>You could use <code>unfold</code> as described in <a href="https://discuss.pytorch.org/t/creating-nonoverlapping-patches-from-3d-data-and-reshape-them-back-to-the-image/51210/6">this post</a>.</p>

tmc: <p>I get this error:</p>
<pre><code class="lang-auto">RuntimeError: maximum size for tensor at dimension 3 is 3 but size is 32

</code></pre>
<p>when I have this code :</p>
<pre><code class="lang-auto">S = 128 # channel dim
W = 227 # width
H = 227 # height
batch_size = 10

x = image_new.unsqueeze(0)

size = 32 # patch size
stride = 32 # patch stride
patches = x.unfold(1, size, stride).unfold(2, size, stride).unfold(3, size, stride)
print(patches.shape)
</code></pre>

ptrblck: <p>The following code works for me:</p>
<pre><code class="lang-python">S = 128 # channel dim
W = 227 # width
H = 227 # height
batch_size = 10

x = torch.randn(batch_size, S, H, W)

size = 32 # patch size
stride = 32 # patch stride
patches = x.unfold(1, size, stride).unfold(2, size, stride).unfold(3, size, stride)
print(patches.shape)
</code></pre>
<p>What shape does <code>x</code> have after the <code>unsqueeze</code> operation?</p>

tmc: <p>I first loaded the image like so:</p>
<pre><code class="lang-auto">import numpy as np
import cv2

img = cv2.imread('cat.jpg',1)
image_new=torch.from_numpy(img)
</code></pre>
<p>and then I used the code above</p>
<p>and the size of the x is:</p>
<pre><code class="lang-auto">torch.Size([1, 576, 1024, 3])

</code></pre>

ptrblck: <p><code>dim3</code> has only a size of 3, so you cannot unfold it with a kernel size of 32.<br>
I guess you would like to unfold only in dim1 and dim2?</p>
<p>Unrelated to this, but note that PyTorch expects image tensors in the shape <code>[batch_size, channels, height, width]</code>.</p>

tmc: <p>So the method of loading the image using opencv and then convert it to tensor is wrong? Is there another way to convert the image to tensor so that it outputs a shape with the dimensions you specified?</p>
<p>also when I only do dim1 and dim2 I get a size of this :</p>
<pre><code class="lang-auto">torch.Size([1, 18, 32, 3, 32, 32])

</code></pre>
<p>and when I show the image using plt, there’s still something wrong</p>

ptrblck: <p>You could use <code>PIL</code> to load the image and then <code>torch.from_numpy</code> to create the tensor or alternatively use OpenCV, transform the image from BGR to RGB, and permute the dimensions via <code>image = image.permute(2, 0, 1)</code>.</p>

tmc: <p>The first solution gave me the same dimensions as before, also I had to use a different code like so :</p>
<pre><code class="lang-auto">image=torch.as_tensor(np.array(image).astype('float'))
</code></pre>
<p>when I use the code you posted like so :</p>
<pre><code class="lang-auto">from PIL import Image 
  
# open method used to open different extension image file 
im = Image.open(r"cat.jpg")  
image_new = torch.from_numpy(im)
</code></pre>
<p>it gives me an error :</p>
<pre><code class="lang-auto">TypeError: expected np.ndarray (got JpegImageFile)
</code></pre>
<p>So the first solution doesn’t work.</p>
<p>The second solution I got this error when I did permute</p>
<pre><code class="lang-auto">AttributeError: 'numpy.ndarray' object has no attribute 'permute'

</code></pre>
<p>The code was like so :</p>
<pre><code class="lang-auto">image = cv2.imread('cat.jpg',1)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) #transfrom image from BGR to RGB
image = image.permute(2, 0, 1)
</code></pre>

Lukman_Hakim: <p>I want to make patches with size 32x32 with stride=16(overlap). How to reshape/reconstruct  this patches to the original image?</p>

ptrblck: <p>Sorry for the confusion.<br>
This should work:</p>
<pre><code class="lang-python">img = PIL.Image.open(path)
x = torch.from_numpy(np.array(img))
x = x.permute(2, 0, 1)

# or
y = transforms.ToTensor()(img) # will permute and normalize to [0, 1]
</code></pre>

ptrblck: <p><code>nn.Unfold</code> and <code>nn.Fold</code> will give you the ability to recreate the input, but note that the overlapping pixels will be summed.</p>

tmc: <p>Now it’s giving me this error:</p>
<pre><code class="lang-auto">RuntimeError: maximum size for tensor at dimension 1 is 3 but size is 128
</code></pre>
<p>and the shape of x is :</p>
<pre><code class="lang-auto">torch.Size([1, 3, 576, 1024])
</code></pre>
<p>and my code is :</p>
<pre><code class="lang-auto">from PIL import Image 

img = Image.open("cat.jpg")
x = torch.from_numpy(np.array(img))
x = x.permute(2, 0, 1)

x = x.unsqueeze(0)

size = 128 # patch size
stride = 32 # patch stride
patches = x.unfold(1, size, stride).unfold(2, size, stride).unfold(3, size, stride)
print(patches.shape)
</code></pre>

tmc: <p>why do you have 128 channels anyways? aren’t they 3? red, green and blue?</p>

ptrblck: <p>I used your code snippet from <a href="https://discuss.pytorch.org/t/how-to-extract-patches-from-an-image/79923/3">this post</a>. If you are not dealing with 128 channels, then you should change it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Regarding the error message: you cannot use a kernel size of 128 for 3 channels.</p>

edgarriba: <p>You have a small api to extract/combine patches in Kornia</p>
<p><a href="https://kornia.readthedocs.io/en/latest/contrib.html#image-patches" class="onebox" target="_blank" rel="noopener nofollow ugc">https://kornia.readthedocs.io/en/latest/contrib.html#image-patches</a></p>
<p>The exact function that you are looking for is <a href="https://kornia.readthedocs.io/en/latest/contrib.html#kornia.contrib.extract_tensor_patches" rel="noopener nofollow ugc">extract_tensor_patches</a></p>