How to split tensors with overlap and then reconstruct the original tensor?

PytorchBeginner: <p>I encountered a problem. My network is trained with tensors of size BxCx128x128, but I need to verify its image reconstruction performance with images of size 1024x1024. To make the reconstruction smooth, I need to split my input of size BxCx1024x1024 into BxCx128x128  tensors with overlap, which are then fed to the network for reconstruction. Then, the reconstructed tensors of size BxCx128x128 should be used for reconstructing the tensor of size BxCx1024x1024 by averaging the overlaping elements. Note that (Size(in)-128)/stride may be not a integer. How to use padding strategy to ensure that every element is cropped at least for one time? How to implement the process of recovering the BxCx1024x1024 tensor from overlapping BxCx128x128 tensors? Could anyone give me some suggestions? Thanks advance for your consideration.</p>

ptrblck: <p><code>fold</code> should work in your use case.<br>
Here is a small example creating the expected input shape step by step:</p>
<pre><code class="lang-python">B, C, W, H = 2, 3, 1024, 1024
x = torch.randn(B, C, H, W)

kernel_size = 128
stride = 64
patches = x.unfold(3, kernel_size, stride).unfold(2, kernel_size, stride)
print(patches.shape) # [B, C, nb_patches_h, nb_patches_w, kernel_size, kernel_size]

# perform the operations on each patch
# ...

# reshape output to match F.fold input
patches = patches.contiguous().view(B, C, -1, kernel_size*kernel_size)
print(patches.shape) # [B, C, nb_patches_all, kernel_size*kernel_size]
patches = patches.permute(0, 1, 3, 2) 
print(patches.shape) # [B, C, kernel_size*kernel_size, nb_patches_all]
patches = patches.contiguous().view(B, C*kernel_size*kernel_size, -1)
print(patches.shape) # [B, C*prod(kernel_size), L] as expected by Fold
# https://pytorch.org/docs/stable/nn.html#torch.nn.Fold

output = F.fold(
    patches, output_size=(H, W), kernel_size=kernel_size, stride=stride)
print(output.shape) # [B, C, H, W]
</code></pre>
<p>Let me know, if this works for you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

PytorchBeginner: <p>Thanks for your reply. In fact, I have noticed these two functions in other related posts (e.g., <a href="https://discuss.pytorch.org/t/patch-making-does-pytorch-have-anything-to-offer/33850/11">Patch Making Does Pytorch have Anything to Offer?</a>), but there are still several questions.</p>
<ol>
<li>In the example, the stride is set to 64, which satisfies (Size(in)-128)/164 to be an integer. But how to handle arbirary strides that may not satisfy this (e.g. stride=40) with padding or other strategies?</li>
<li>When we get patches with size being [B,C,nb_patches_h,nb_patches_w,kernel_size,kernel_size], we can surely perform the operation on each patch by looping over patches in dimension 0 and 1. Is there a way that we can perform the operation on all the patch one-time?</li>
<li>Is the spatial neighborhood information preserved in unfold and fold process? The reconstructed large tensor should average the overlapping elements in tensor patches, but I did not see any signs of this process.</li>
</ol>
<p>Many thanks for your help.</p>

ptrblck: <ol>
<li>
<p>Padding should work fine.</p>
</li>
<li>
<p>Depending on the operation you could use e.g. a single multiplication as seen <a href="https://discuss.pytorch.org/t/how-to-implement-a-convolutional-layer/68211/7">here</a>.</p>
</li>
<li>
<p>Yes, the overlapping elements are however summed so you might need to normalize them afterwards with e.g. a mask.</p>
</li>
</ol>

PytorchBeginner: <p>Thanks for your reply.</p>
<ol>
<li>How to automatically use padding in unfold? Or just padding before unfolding operation?</li>
<li>In your posted example, why do we need to permute patches with parameters (0,1,3,2)? What is the difference between</li>
</ol>
<pre><code class="lang-auto">patches = patches.contiguous().view(B, C, -1, kernel_size*kernel_size)
print(patches.shape) # [B, C, nb_patches_all, kernel_size*kernel_size]
patches = patches.permute(0, 1, 3, 2) 
print(patches.shape) # [B, C, kernel_size*kernel_size, nb_patches_all]
patches = patches.contiguous().view(B, C*kernel_size*kernel_size, -1)
print(patches.shape) # [B, C*prod(kernel_size), L] as expected by Fold
</code></pre>
<p>and</p>
<pre><code class="lang-auto">patches2=patches.contiguous().view(B,C,-1,kernel_size*kernel_size)
print(patches2.shape)
patches2=patches2.view(B,C*kernel_size*kernel_size,-1)
print(patches2.shape)
</code></pre>
<p>Thanks a lot!</p>

ptrblck: <ol>
<li>
<p>Pad the data before unfolding.</p>
</li>
<li>
<p>If you are <code>viewing</code> collapsing non-neighboring dimensions, you are interleaving the pixels. You could try it on an image tensor and your result should be interleaved.</p>
</li>
</ol>

PytorchBeginner: <p>Thanks for your reply.<br>
I am a pytorch beginner. Could you please give a function including padding that can handle arbitrary strides and recover the original tensor?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/creating-nonoverlapping-patches-from-3d-data-and-reshape-them-back-to-the-image/51210/6">This post</a> is using padding so that the original input tensor can be restored.</p>

PytorchBeginner: <p>Many thanks to your reply. You are so warm-hearted!</p>

eburling: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>My aim is the same as the OP: unfold a large image into overlapping tiles, then fold them back together, averaging values where there was overlap. I’ve been testing this example with dummy image data, but it doesn’t seem to be returning the overlapped sum.</p>
<pre><code class="lang-auto">from skimage.data import astronaut

def tensor2im(input_image, imtype=np.uint8):
    """"Converts a Tensor array into a numpy image array.
    Parameters:
        input_image (tensor) --  the input image tensor array
        imtype (type)        --  the desired type of the converted numpy array
    """
    if not isinstance(input_image, np.ndarray):
        if isinstance(input_image, torch.Tensor):  # get the data from a variable
            image_tensor = input_image.data
        else:
            return input_image
        image_numpy = image_tensor[0].cpu().float().numpy()  # convert it into a numpy array
        if image_numpy.shape[0] == 1:  # grayscale to RGB
            image_numpy = np.tile(image_numpy, (3, 1, 1))
        image_numpy = (image_numpy + 1) / 2.0 * 255.0  # post-processing: tranpose and scaling

    else:  # if it is a numpy array, do nothing
        image_numpy = input_image
    return image_numpy.astype(imtype)


tpi = transforms.ToPILImage()
tform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.5, 0.5, 0.5],
                                 std=[0.5, 0.5, 0.5])
        ])

x = tform(astronaut()).unsqueeze(0)
B, C, W, H = x.shape

kernel_size = 64
stride = 32
patches = x.unfold(3, kernel_size, stride).unfold(2, kernel_size, stride)
print(patches.shape) # [B, C, nb_patches_h, nb_patches_w, kernel_size, kernel_size]

# perform the operations on each patch
# ...

# reshape output to match F.fold input
patches = patches.contiguous().view(B, C, -1, kernel_size*kernel_size)
print(patches.shape) # [B, C, nb_patches_all, kernel_size*kernel_size]
patches = patches.permute(0, 1, 3, 2) 
print(patches.shape) # [B, C, kernel_size*kernel_size, nb_patches_all]
patches = patches.contiguous().view(B, C*kernel_size*kernel_size, -1)
print(patches.shape) # [B, C*prod(kernel_size), L] as expected by Fold
# https://pytorch.org/docs/stable/nn.html#torch.nn.Fold

output = F.fold(
    patches, output_size=(H, W), kernel_size=kernel_size, stride=stride)
print(output.shape) # [B, C, H, W]
</code></pre>
<pre><code class="lang-auto"># Take a look at the input
tpi(tensor2im(x).transpose(1,2,0))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2af0e66c1777e2e9f657b3673e111899e897220.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e2af0e66c1777e2e9f657b3673e111899e897220" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2af0e66c1777e2e9f657b3673e111899e897220_2_375x375.jpeg" alt="image" data-base62-sha1="wlkXNZIrGu96SxeFOf9L7lYouVW" width="375" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2af0e66c1777e2e9f657b3673e111899e897220_2_375x375.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2af0e66c1777e2e9f657b3673e111899e897220.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2af0e66c1777e2e9f657b3673e111899e897220.jpeg 2x" data-dominant-color="8F6A60"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">512×512 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto"># Take a look at the output
tpi(tensor2im(output).transpose(1,2,0))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e58e9a7bde84693e1f906902c725b3ab033c217e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e58e9a7bde84693e1f906902c725b3ab033c217e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e58e9a7bde84693e1f906902c725b3ab033c217e_2_375x375.jpeg" alt="image" data-base62-sha1="wKKRLB22rfmz1DclcfFGKWCkWaG" width="375" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e58e9a7bde84693e1f906902c725b3ab033c217e_2_375x375.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/5/e58e9a7bde84693e1f906902c725b3ab033c217e.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/5/e58e9a7bde84693e1f906902c725b3ab033c217e.jpeg 2x" data-dominant-color="858082"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">512×512 380 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The overflow artifacts are expected here and are easily corrected by division with a mask generated by running a tensor of ones through the unfold/fold operation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1d024f10932628892693e5d08a90dca815a9d5a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c1d024f10932628892693e5d08a90dca815a9d5a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1d024f10932628892693e5d08a90dca815a9d5a_2_375x375.jpeg" alt="image" data-base62-sha1="rEy68cMjrMWKIqpbu3Cqp3Xa6kG" width="375" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1d024f10932628892693e5d08a90dca815a9d5a_2_375x375.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1d024f10932628892693e5d08a90dca815a9d5a.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1d024f10932628892693e5d08a90dca815a9d5a.jpeg 2x" data-dominant-color="8F6A60"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">512×512 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
…but what we can see is that the patches have not been reassembled as expected. We do get the same shape, but there is substantial scrambling.</p>
<p>Any help on this problem would be much appreciated. Cheers!</p>

eburling: <p>Just figured it out. Had to permute the patches:</p>
<p><code>patches = x.unfold(3, kernel_size, stride).unfold(2, kernel_size, stride)</code><br>
becomes<br>
<code>patches = x.unfold(3, kernel_size, stride).unfold(2, kernel_size, stride).permute(0,1,2,3,5,4)</code></p>

whuang7000: <p>Hi eburling,</p>
<p>I am trying to do the same thing (unfold a large image into overlapping tiles, then fold them back together, averaging values where there was overlap). I see you figured out how to unfold and fold the image back together. How did you take care of the averaging overlapping values part?</p>

eburling: <p>Hi <a class="mention" href="/u/whuang7000">@whuang7000</a>,</p>
<p>I generated a normalization mask by unfolding/refolding a torch.ones tensor of equal size, then divided my output image by the normalization mask. Probably a better way out there, but it worked for me.</p>

NGH: <p>I also want to do the same, but I also have a target image, which means I need to split input and target, so for each input patch create its corresponding target patch. how can I do that? also is there way to split the input into unsquare patch, since my original image are large  when I gave them to the super resolution model I got run out of memory, so I want to divide input by half and reconstruct each sub image and then stick them to reconstruct the whole image.<br>
Thanks</p>

ptrblck: <p>You could use <code>tensor.unfold</code> on both, the input and target tensors (or alternatively the <code>nn.Unfold</code> module).<br>
In both approaches you can specify the kernel size and could use non-square shapes for it.</p>

NGH: <p>Thanks for your answer. But my input and target are not the same size, the target image is larger than input with specefice scale factor.</p>

ptrblck: <p>This would mean that you would either end up with more patches in the target or with bigger patches (larger kernel size in the <code>unfold</code> operation). Based on your initial description it seems you would like to create corresponding target patches for each input patch, so scaling up the kernel size (and stride) should probably work.</p>

NGH: <p>Thanks for your help I did that but still  I have a problem. What I want to do is that I have an input image let’s note it as LR, and the target image takes it as HR, I fed the LR image into my model(Super-resolution reconstruction model)and the output of the model is SR image. SR and HR have the same scale and they are larger than LR by scale factor m. As my LR images are large if I directly fed the  LR image into the model I get out of memory, that’s why I want to divide it into the smaller patches and reconstruct its corresponding SR, and then combine them to create the final SR image, the result in this way should be the same as when I directly input whole LR into the model. But after applying fold on SR_Patches, the SR image result is messy. Do you have any suggestions on how can solve this issue?<br>
Thanks.</p>
<p>LR_Image.shape :<br>
torch.Size([3, 702, 1020])<br>
HR_Image.shape :<br>
torch.Size([ 3, 1404, 2040])<br>
scale=2<br>
lr_kernel_size= (351,510)<br>
hr_kernel_size= (702,1020)<br>
lr_stride =lr_kernel_size<br>
hr_stride =hr_kernel_size * scale<br>
fold_params = dict(kernel_size=lr_kernel_size, stride=lr_stride)<br>
unfoldlr = nn.Unfold(**fold_params)<br>
lr_patches=unfoldlr(LR_Image)<br>
lr_patches:<br>
torch.Size([1, 537030, 4])<br>
unfoldhr = nn.Unfold(**dict(kernel_size=hr_kernel_size, stride=hr_stride))<br>
hr_patches = unfoldhr(HR_Image)<br>
hr_patches:<br>
torch.Size([1, 2148120, 4])<br>
sr_patches= self.model(lr_patches, idx_scale)<br>
sr_patches: torch.Size([1, 2148120, 4])<br>
fold = nn.Fold(( 1404, 2040), **dict(kernel_size=hr_kernel_size, stride=hr_stride))<br>
sr = fold(sr_patches)<br>
sr.shape:<br>
torch.Size([1, 3, 1404, 2040])</p>

ptrblck: <aside class="quote no-group" data-username="NGH" data-post="20" data-topic="70261">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngh/48/26804_2.png" class="avatar"> NGH:</div>
<blockquote>
<p>the result in this way should be the same as when I directly input whole LR into the model.</p>
</blockquote>
</aside>
<p>That wouldn’t be guaranteed, as your model only sees patches and could thus create artifacts on the borders of these patches.<br>
You could try to use some overlapping (and try to sum these overlapping pixels in the output or use another reduction), but I’m unsure which approach would work the best.</p>

Alexandre-Delplanque: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I would like to get the maximum value instead of averaging in overlapping areas, do you have a simple and effective idea to do that?</p>
<p>Thanks in advance for your help!</p>