Deconvolution for not square images

username: smu226
cooked: <p>Hello! I am trying to build a network that reconstructs some images from a 1D vector. However the images are not squared (they are 15x27). What is the best way to do that? ConvTranspose2d seems to be able to return just squared images. Thank you!</p>

username: ptrblck
cooked: <p>This should not be the case:</p>
<pre><code class="lang-python">conv = nn.ConvTranspose2d(1, 3, 4, 2)
x = torch.randn(1, 1, 15, 27)
out = conv(x)
print(out.shape)
&gt; torch.Size([1, 3, 32, 56])
</code></pre>
<p>You could also define the parameters of your transposed convolution for each spatial dimension separately, if needed.</p>

username: smu226
cooked: <p>I am sorry if I explained that the wrong way. I start with a 1D vector and I want to end up with a 15x27 image. I don’t start with that image. So my input is (32,5), (the 32 is just the batch size) and the output should be (32, 2, 15, 27) so a batch of images with 2 channels and size 15x27.</p>

username: ptrblck
cooked: <p>There are of course multiple ways of achieving the desired output shape and this would be one:</p>
<pre><code class="lang-python">x = torch.randn(32, 5)
x = x.unsqueeze(2).unsqueeze(2)

model = nn.Sequential(
    nn.ConvTranspose2d(5, 2, (2, 2), 2),
    nn.ConvTranspose2d(2, 2, (2, 4), 2),
    nn.ConvTranspose2d(2, 2, (2, 3), 2),
    nn.ConvTranspose2d(2, 2, (1, 3), 2),
)

out = model(x)
print(out.shape)
&gt; torch.Size([32, 2, 15, 27])
</code></pre>

username: smu226
cooked: <p>Thanks a lot for this! I tried it and I do get the right shape and the NN is training, however the images look a bit weird (I attached an example of input/output after the NN is trained). Using a similar architecture (which is basically an autoencoder), I get almost perfect reconstruction for square images. Also the error seems to be systematic i.e. if I train the NN with different initializations I get the same strides in the output, for all the images. Do you know what could cause this (I haven’t used non-square images before). Could it be because of the way the deconvolution is done (it could just as well be a bug in my code, but it is worth asking here)? Thank you!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b046249d5da5b5b866332b731a8cb7542518666.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b046249d5da5b5b866332b731a8cb7542518666" title="inp"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b046249d5da5b5b866332b731a8cb7542518666_2_690x406.png" alt="inp" data-base62-sha1="hyg7wHLoVm78X2JqpCHFfFNwsrY" width="690" height="406" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b046249d5da5b5b866332b731a8cb7542518666_2_690x406.png, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b046249d5da5b5b866332b731a8cb7542518666.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b046249d5da5b5b866332b731a8cb7542518666.png 2x" data-dominant-color="957278"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">inp</span><span class="informations">782×461 13.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b211145421b756620969bb61980e93b72b429171.png" data-download-href="https://discuss.pytorch.org/uploads/default/b211145421b756620969bb61980e93b72b429171" title="out"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b211145421b756620969bb61980e93b72b429171_2_690x398.png" alt="out" data-base62-sha1="ppfA5ZC01V0H1lmHTkVwRmISte9" width="690" height="398" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b211145421b756620969bb61980e93b72b429171_2_690x398.png, https://discuss.pytorch.org/uploads/default/original/3X/b/2/b211145421b756620969bb61980e93b72b429171.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/2/b211145421b756620969bb61980e93b72b429171.png 2x" data-dominant-color="796D86"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">out</span><span class="informations">765×442 14.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Thanks for getting back!<br>
I guess the pattern is created by the last <code>nn.ConvTranspose2d</code> layer, which uses a kernel height of 1 and a stride of 2 in both spatial dimensions.<br>
This will skip every second row. Sorry for missing this.</p>
<p>You could of course change the overall architecture and try to use overlapping or neighboring kernels.<br>
This is a quick and dirty fix using the current architecture and increasing the last kernel dimension:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.ConvTranspose2d(5, 2, (2, 2), 2),
    nn.ConvTranspose2d(2, 2, (2, 4), 2),
    nn.ConvTranspose2d(2, 2, (2, 3), 2),
    nn.ConvTranspose2d(2, 2, (2, 3), stride=2, padding=(1, 0), output_padding=(1, 0)),
)
</code></pre>
<p>Let me know, if that works better.</p>

username: smu226
cooked: <p>Thanks a lot! It works perfectly now!</p>

username: Angry_potato
cooked: <p>Hi,<br>
I would like to know how you are training your network.Are you using GAN type of approach?</p>

username: smu226
cooked: <p>I am doing a VAE approach actually.</p>

username: danielgonzalezcortes
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> how do you calculate the structure to reach that output?, for example, how can I modify your example to reach an output of  torch.Size([32, 1, 101, 27]) or<br>
torch.Size([32, 1, 101, 35])</p>
<p>What is the math/equations behind this?</p>
<p>Thank you very much!</p>

username: ptrblck
cooked: <p>The formula to calculate the output shape is given in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html#torch.nn.ConvTranspose2d">docs</a> of <code>nn.ConvTranspose2d</code>.<br>
You could calculate the approx. architecture manually (e.g. start with layers, which would double the spatial size) and “fine-tune” the layers later in case the final output size doesn’t match.<br>
Note that you can add <code>print</code> statements into the <code>forward</code> method of your model to get the output shapes of intermediate tensors, which would make creating the model a bit easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: danielgonzalezcortes
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a> , helps me a lot.</p>