Extract image patches from image

dome272: <p>Disclaimer: I know there are a couple questions already like this, however none of them seems to get my problem covered.</p>
<p>My goal is simply to extract image patches. The following should make it clear. Consider the following sketch.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25c3beaf1854c01bda498e8cbbd62f85e14b1577.png" data-download-href="https://discuss.pytorch.org/uploads/default/25c3beaf1854c01bda498e8cbbd62f85e14b1577" title="patches"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25c3beaf1854c01bda498e8cbbd62f85e14b1577_2_690x299.png" alt="patches" data-base62-sha1="5o52XS1TH1ujuXYcqlqE8fM6Qfl" width="690" height="299" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25c3beaf1854c01bda498e8cbbd62f85e14b1577_2_690x299.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25c3beaf1854c01bda498e8cbbd62f85e14b1577_2_1035x448.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25c3beaf1854c01bda498e8cbbd62f85e14b1577_2_1380x598.png 2x" data-dominant-color="F9F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">patches</span><span class="informations">1705×741 7.36 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to seperate my image into e.g. 4 patches and have each patch have all the channels in it.<br>
For example given an image 3x8x8 I want to have the patches tensor to be 4x(4 * 4 * 3) → 4x48</p>
<p>Any ideas for a general applicable code?</p>

ptrblck: <p><code>tensor.unfold</code> should work as seen in this minimal example:</p>
<pre><code class="lang-python">x = torch.randn(3, 8, 8)
out = x.unfold(2, 4, 4).unfold(1, 4, 4)
print(out.shape)
# torch.Size([3, 2, 2, 4, 4])
out = out.permute(1, 2, 0, 3, 4)
print(out.shape)
# torch.Size([2, 2, 3, 4, 4])
out = out.contiguous().view(out.size(0)*out.size(1), -1)
print(out.shape)
# torch.Size([4, 48])
</code></pre>

dome272: <p>Thanks for the quick reply. Though it gets the shapes correct it sort of doesnt get the ordering of each sample correct.<br>
Consider this code:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.arange(3* 8* 8).reshape((3,8,8))
&gt;&gt;&gt; p = 4
&gt;&gt;&gt; res = []
&gt;&gt;&gt; for i in range(1, 3):
...     for j in range(1,3):
...             res.append(x[:, p*(i-1):p*i, p*(j-1):p*j].flatten())
&gt;&gt;&gt; res[0]
tensor([  0,   1,   2,   3,   8,   9,  10,  11,  16,  17,  18,  19,  24,  25,
         26,  27,  64,  65,  66,  67,  72,  73,  74,  75,  80,  81,  82,  83,
         88,  89,  90,  91, 128, 129, 130, 131, 136, 137, 138, 139, 144, 145,
        146, 147, 152, 153, 154, 155])
</code></pre>
<p>Whereas in yor case it looks like:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; out[0]
tensor([  0,   8,  16,  24,   1,   9,  17,  25,   2,  10,  18,  26,   3,  11,
         19,  27,  64,  72,  80,  88,  65,  73,  81,  89,  66,  74,  82,  90,
         67,  75,  83,  91, 128, 136, 144, 152, 129, 137, 145, 153, 130, 138,
        146, 154, 131, 139, 147, 155])
</code></pre>
<p>Any idea how to fix this?</p>

ptrblck: <p>I’m not sure, if you would need to extract patches in your expected output is just the flattened tensor.<br>
In my approach the first intermediate output would keep the patches as:</p>
<pre><code class="lang-python">x = torch.arange(3* 8* 8).reshape((3,8,8))
out = x.unfold(2, 4, 4).unfold(1, 4, 4)
out
print(out.shape)
# torch.Size([3, 2, 2, 4, 4])
out = out.permute(1, 2, 0, 3, 4)
print(out)
tensor([[[[[  0,   8,  16,  24],
           [  1,   9,  17,  25],
           [  2,  10,  18,  26],
           [  3,  11,  19,  27]],

          [[ 64,  72,  80,  88],
           [ 65,  73,  81,  89],
           [ 66,  74,  82,  90],
           [ 67,  75,  83,  91]],
           ...
</code></pre>
<p>If you want to get the flattened output wouldn’t a <code>view</code> operation just work?</p>

dome272: <p>Yes, I guess a view() would work. Could you help with that? I don’t seem to get my desired output <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"></p>

ptrblck: <p>I think I would need to see your (slow) loop approach or any other references.<br>
The previous post:</p>
<pre><code class="lang-python">&gt;&gt;&gt; res[0]
tensor([  0,   1,   2,   3,   8,   9,  10,  11,  16,  17,  18,  19,  24,  25,
         26,  27,  64,  65,  66,  67,  72,  73,  74,  75,  80,  81,  82,  83,
         88,  89,  90,  91, 128, 129, 130, 131, 136, 137, 138, 139, 144, 145,
        146, 147, 152, 153, 154, 155])
</code></pre>
<p>would just work, if you use:</p>
<pre><code class="lang-python">x = torch.arange(3* 8* 8).reshape((3,8,8))
out = x.view(-1)
print(out)
</code></pre>
<p>although your output is missing some values.</p>

dome272: <p>I misunderstood your question " If you want to get the flattened output wouldn’t a <code>view</code> operation just work?"<br>
My intention is to first extract the patches and then flatten each patch. The first code which you sent was almost already perfect for my needs, just that it messed up the order of the patch values.</p>
<p>Update:<br>
I guess I found the missing modification to your code:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.arange(3* 8* 8).reshape((3,8,8))
&gt;&gt;&gt; out = x.unfold(2, 4, 4).unfold(1, 4, 4)
&gt;&gt;&gt; out = torch.transpose(out, 3,4) &lt;---- transposing it gives me my desired output
&gt;&gt;&gt; out = out.permute(1, 2, 0, 3, 4)
&gt;&gt;&gt; out = out.contiguous().view(out.size(0)*out.size(1), -1)
&gt;&gt;&gt; out[0]
tensor([  0,   1,   2,   3,   8,   9,  10,  11,  16,  17,  18,  19,  24,  25,
         26,  27,  64,  65,  66,  67,  72,  73,  74,  75,  80,  81,  82,  83,
         88,  89,  90,  91, 128, 129, 130, 131, 136, 137, 138, 139, 144, 145,
        146, 147, 152, 153, 154, 155])
</code></pre>
<p>which is equivalent to my before posted code</p>
<pre><code class="lang-auto">&gt;&gt;&gt; res[0]
tensor([  0,   1,   2,   3,   8,   9,  10,  11,  16,  17,  18,  19,  24,  25,
         26,  27,  64,  65,  66,  67,  72,  73,  74,  75,  80,  81,  82,  83,
         88,  89,  90,  91, 128, 129, 130, 131, 136, 137, 138, 139, 144, 145,
        146, 147, 152, 153, 154, 155])
</code></pre>

ptrblck: <p>Oh sorry, I’m blind. I didn’t notice that your reference output contained a different ordering.<br>
Good to hear you’ve solved it and sorry for the confusion.</p>

dome272: <p>No problem holy PyTorch Keanu Reeves!</p>

abolfazl_saheban: <p>Hello,<br>
If I use this process of unfolding a feature map during training of a neural network when all of my computations are on a gpu, is this unfolding and permutation execute on gpu or it halts operation on gpu for memory allocation in cpu an ram?<br>
thanks</p>

ptrblck: <p>The execution depends on the placement of the data. I.e. if you call <code>tensor.unfold</code>, the device will be used where <code>tensor</code> resides.</p>

abolfazl_saheban: <p>Hello again and thanks for your reply<br>
so if I put my model for training on a gpu, and in the network structure, I use the unfolding and the folding operation to create patches from temporary variables the whole process of forward propagation will be executed on gpu? or it might execute on cpu?</p>

ptrblck: <p>If the tensor was moved the the GPU, the <code>unfold</code> operation will be performed on the GPU.<br>
Check the <code>.device</code> attribute of the corresponding tensor to double check which device will be used.</p>