Unable to reconstruct unfolded image

username: sakkosjo
cooked: <p>I’m having difficulty reconstructing an image from patches after running it through my model. This same approach worked with only one channel input image size (1460,1936), but seems to scramble the order with two channel input and one channel output. I’m sure it’s something very simple that I’m missing, but I can’t seem to find the error in my code.</p>
<p>Example code:</p>
<pre><code class="lang-auto">in_channels = 2
temp = np.random.rand(in_channels, 1460, 1936)
normalized = np.zeros([in_channels,temp.shape[1],temp.shape[2]])
for i in range(in_channels):
    normalized[i,:,:] = (temp[i,:,:] - np.mean(temp[i,:,:]))/(np.max(temp[i,:,:]) - np.min(temp[i,:,:]))
x = torch.from_numpy(normalized)
# kernel size
k = 256 
# stride
d = 256
#hpadding
hpad = (k-x.size(1)%k) // 2 
#wpadding
wpad = (k-x.size(2)%k) // 2 
#pad x
x = F.pad(x,(wpad,wpad,hpad,hpad)) #shape torch.Size([2, 1536, 2048])

patches = x.unfold(1, k, d).unfold(2, k, d) #shape torch.Size([2, 6, 8, 256, 256])
unfold_shape = patches.size()
#reshape to (batch,1,h,w)
patches = patches.contiguous().view(-1, in_channels,k, k) #shape torch.Size([48, 2, 256, 256])
temp = torch.empty(patches.shape[0],k,k) #torch.Size([48, 256, 256])
#loop over all patches, feed model predictions back into storage tensor
for i,patch in enumerate(patches):
    temp[i,:,:] = model(patch.view(-1,in_channels,k,k).to(device, dtype = torch.float)).cpu().detach()[0][1]
# Reshape back
patches_orig = temp.view(unfold_shape[1],unfold_shape[2],unfold_shape[3],unfold_shape[4])
output_h = unfold_shape[1] * unfold_shape[3]
output_w = unfold_shape[2] * unfold_shape[4]
patches_orig = patches_orig.permute(0,2,1,3).contiguous()
patches_orig = patches_orig.view(output_h, output_w)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12fc8b0ab5097c42a9dfa300e7f70eef4b3226db.png" data-download-href="https://discuss.pytorch.org/uploads/default/12fc8b0ab5097c42a9dfa300e7f70eef4b3226db" title="test"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12fc8b0ab5097c42a9dfa300e7f70eef4b3226db_2_690x345.png" alt="test" data-base62-sha1="2HXFgH2VdFPZxkAeNYQzL7ABjgT" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12fc8b0ab5097c42a9dfa300e7f70eef4b3226db_2_690x345.png, https://discuss.pytorch.org/uploads/default/original/3X/1/2/12fc8b0ab5097c42a9dfa300e7f70eef4b3226db.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/2/12fc8b0ab5097c42a9dfa300e7f70eef4b3226db.png 2x" data-dominant-color="B9BBB5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test</span><span class="informations">864×432 145 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your code is unfortunately not executable, so I can just make a few suggestions.<br>
I assume this line of code is wrong:</p>
<pre><code class="lang-python">patches = patches.contiguous().view(-1, in_channels,k, k)
</code></pre>
<p><code>patches</code> should have the shape <code>[2, 6, 8, 256, 256]</code> (<code>[channels, patch_h, patch_w, h, w]</code>), so the <code>view</code> call would interleave the data.<br>
Probably <code>patches.permute(1, 2, 0, 3, 4).view(-1, in_channels, k, k)</code> would work.</p>

username: sakkosjo
cooked: <p>Thanks! Your suggestion worked, with the addition of <code>.contiguous()</code></p>
<pre><code class="lang-auto">patches.permute(1, 2, 0, 3, 4).contiguous().view(-1, in_channels, k, k)
</code></pre>