Autoencoder - help with nn.sequential

username: chaslie
cooked: <p>Morning all,</p>
<p>I am hoping that someone will be able to help me as I am at the end of my tether…</p>
<p>I am trying to write the autoencoder for  my network. having hit a problem i started experiementing with a know working decoder (shown below):<br>
class Decoder(nn.Module):</p>
<pre><code>def __init__(self, z_dim):
    super(Decoder, self).__init__()
    self.fc1 = nn.Linear(z_dim, 672)
    self.conv1 = nn.ConvTranspose1d(32, 32, 8, 2, padding=3)
    self.conv2 = nn.ConvTranspose1d(32, 32, 8, 2, padding=3)
    self.conv3 = nn.ConvTranspose1d(32, 16, 8, 2, padding=3)
    self.conv4 = nn.ConvTranspose1d(16, 16, 8, 2, padding=3)
    self.conv5 = nn.ConvTranspose1d(16, 1, 7, 1, padding=3)
    self.bn1 = nn.BatchNorm1d(32)
    self.bn2 = nn.BatchNorm1d(32)
    self.bn3 = nn.BatchNorm1d(16)
    self.bn4 = nn.BatchNorm1d(16)
    self.relu = nn.ReLU()
</code></pre>
<h1></h1>
<pre><code>def forward(self, z):
    z = self.relu(self.fc1(z))
    print(z)
    print(z.shape)
   #z = F.dropout(z, 0.3)
    z = z.view(-1, 32, 21)
    z = self.relu(self.conv1(z))
    z = self.bn1(z)
    #z = F.dropout(z, 0.3)
    z = self.relu(self.conv2(z))
    z = self.bn2(z)
    #z = F.dropout(z, 0.3)
    z = self.relu(self.conv3(z))
    z = self.bn3(z)
    #z = F.dropout(z, 0.3)
    z = self.relu(self.conv4(z))
    z = self.bn4(z)
    #z = F.dropout(z, 0.3)
    z = self.conv5(z)
    recon = torch.sigmoid(z)
    return recon
</code></pre>
<p>summary(vae.decoder, (1, 2))</p>
<p>this gives the following results:</p>
<pre><code class="lang-auto">**tensor([[[0.0000, 0.3415, 0.6484,  ..., 0.2840, 0.0000, 0.3873]],**

**        [[0.0000, 0.4126, 0.6034,  ..., 0.2224, 0.0000, 0.3813]]],**
**       device='cuda:0', grad_fn=&lt;ReluBackward0&gt;)**
**torch.Size([2, 1, 672])**
**----------------------------------------------------------------**
**        Layer (type)               Output Shape         Param #**
**================================================================**
**            Linear-1               [-1, 1, 672]           2,016**
**              ReLU-2               [-1, 1, 672]               0**
**   ConvTranspose1d-3               [-1, 32, 42]           8,224**
**              ReLU-4               [-1, 32, 42]               0**
**       BatchNorm1d-5               [-1, 32, 42]              64**
**   ConvTranspose1d-6               [-1, 32, 84]           8,224**
**              ReLU-7               [-1, 32, 84]               0**
**       BatchNorm1d-8               [-1, 32, 84]              64**
**   ConvTranspose1d-9              [-1, 16, 168]           4,112**
**             ReLU-10              [-1, 16, 168]               0**
**      BatchNorm1d-11              [-1, 16, 168]              32**
**  ConvTranspose1d-12              [-1, 16, 336]           2,064**
**             ReLU-13              [-1, 16, 336]               0**
**      BatchNorm1d-14              [-1, 16, 336]              32**
**  ConvTranspose1d-15               [-1, 1, 336]             113**
**================================================================**
**Total params: 24,945**
**Trainable params: 24,945**
**Non-trainable params: 0**
**----------------------------------------------------------------**
**Input size (MB): 0.00**
**Forward/backward pass size (MB): 0.29**
**Params size (MB): 0.10**
**Estimated Total Size (MB): 0.38**
**----------------------------------------------------------------**
**which is expected.**
</code></pre>
<p>however changing the decoder to be included in a nn.sequential module the summary no longer works (see below).</p>
<pre><code class="lang-auto">class Decoder(nn.Module):
    def __init__(self, z_dim):
        super(Decoder, self).__init__()
        self.decode = nn.Sequential(OrderedDict([
            ('fc1',nn.Linear(z_dim, 672)),
            ('conv1',nn.ConvTranspose1d(32, 32, 8, 2, padding=3)),
            ('conv2',nn.ConvTranspose1d(32, 32, 8, 2, padding=3)),
            ('conv3',nn.ConvTranspose1d(32, 16, 8, 2, padding=3)),
            ('conv4',nn.ConvTranspose1d(16, 16, 8, 2, padding=3)),
            ('conv5',nn.ConvTranspose1d(16, 1, 7, 1, padding=3)),
            ('bn1',nn.BatchNorm1d(32)),
            ('bn2',nn.BatchNorm1d(32)),
            ('bn3',nn.BatchNorm1d(16)),
            ('bn4',nn.BatchNorm1d(16)),
            ('relu',nn.ReLU())
        ]))
        
        
    def forward(self, z):
        z = self.decode.relu(self.decode.fc1(z))
        print(z)
        print(z.shape)
        z = z.view(-1, 32, 31)
        z = self.decode(z)
        #z = self.relu(self.fc1(z))
        #z = F.dropout(z, 0.3)
        recon = torch.sigmoid(z)
        return recon
summary(vae.decoder, (1, 2))
</code></pre>
<p>This gives the following error:</p>
<pre><code class="lang-auto">tensor([[[0.6222, 0.0000, 0.0000,  ..., 0.0180, 0.3997, 0.3615]],

        [[0.7606, 0.0000, 0.0000,  ..., 0.0000, 0.2021, 0.2654]]],
       device='cuda:0', grad_fn=&lt;ReluBackward0&gt;)
torch.Size([2, 1, 672])
Traceback (most recent call last):

  File "G:/decoder-test.py", line 30, in forward
    z = z.view(-1, 32, 31)

RuntimeError: shape '[-1, 32, 31]' is invalid for input of size 1344
</code></pre>
<p>I know I have done something silly, but I cannot for the life of me see what (removing the         z = self.decode.relu(self.decode.fc1(z)) line just changes the size from 1344 to 4)…</p>
<p>I’m hopng that someone can point me in the correct direction…</p>
<p>Chaslie</p>
<pre><code class="lang-auto"></code></pre>

username: Oli
cooked: <aside class="quote no-group" data-username="chaslie" data-post="1" data-topic="50092">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>z = z.view(-1, 32, 31)</p>
</blockquote>
</aside>
<p>Did you mean to do <code>z = z.view(-1, 32, 21)</code>?</p>
<p>But, I’m not sure the code will work on the next line where you do <code>z = self.decode(z)</code> anyways. But one step at a time <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: chaslie
cooked: <p>hi Olaf,</p>
<p>thanks for spotting that, yes i did mean z.view(-1,32,21), but know i haave the following error</p>
<pre><code>output = input.matmul(weight.t())
</code></pre>
<p>RuntimeError: size mismatch, m1: [64 x 21], m2: [2 x 672] at C:/w/1/s/tmp_conda_3.7_044431/conda/conda-bld/pytorch_1556686009173/work/aten/src\THC/generic/THCTensorMathBlas.cu:268</p>
<p>commenting out “z = self.decode.relu(self.decode.fc1(z))” (as i am double accounting)</p>
<p>gives this error</p>
<p>File “G:/decoder-test.py”, line 30, in forward<br>
z = z.view(-1, 32, 21)</p>
<p>RuntimeError: shape ‘[-1, 32, 21]’ is invalid for input of size 4</p>

username: Oli
cooked: <p>I think you should look further into how the nn.Sequential model works. From my understanding it is used when you want a series of layers to always execute after one another, but it has to be put into the sequential in that order. Honestly, I think your first code was fine. Why did you want to put it into sequential in the first place?</p>
<p>Example of sequential, but without the dropout (which you should probably not use if you use batchnorm)</p>
<pre><code class="lang-auto">self.decode = nn.Sequential(OrderedDict([
            ('conv1',nn.ConvTranspose1d(32, 32, 8, 2, padding=3)),
            ('bn1',nn.BatchNorm1d(32)),
            ('conv2',nn.ConvTranspose1d(32, 32, 8, 2, padding=3)),
            ('bn2',nn.BatchNorm1d(32)),
            ...
        ]))
</code></pre>

username: chaslie
cooked: <p>Hi Olaf,</p>
<p>Sorry for the delay. The reason I chose to use sequential is that i understood that it ran quicker and was more efficient…</p>
<p>I’m just trying your suggestion know, I will let you know if solves the problem.</p>
<p>chaslie</p>

username: chaslie
cooked: <p>hi Olaf,</p>
<p>thanks for the suggestion but that didn’t get over the error.</p>
<pre><code class="lang-auto">        self.decode = nn.Sequential(OrderedDict([
            ('fc1',nn.Linear(z_dim, 672)),
            ('relu1',nn.ReLU()),
            ('conv1',nn.ConvTranspose1d(32, 32, 8, 2, padding=3)),
            ('bn1',nn.BatchNorm1d(32)),
            ('conv2',nn.ConvTranspose1d(32, 32, 8, 2, padding=3)),
            ('bn2',nn.BatchNorm1d(32)),
            ('conv3',nn.ConvTranspose1d(32, 16, 8, 2, padding=3)),
            ('bn3',nn.BatchNorm1d(32)),
            ('conv4',nn.ConvTranspose1d(16, 16, 8, 2, padding=3)),
            ('bn4',nn.BatchNorm1d(32)),
            ('conv5',nn.ConvTranspose1d(16, 1, 7, 1, padding=3)),
        ]))
        
        
    def forward(self, z):
        z = self.decode.relu1(self.decode.fc1(z))
        print(z)
        print(z.shape)
        z = z.view(-1, 32, 21)
        print(z)
        print(z.shape)
        z = self.decode(z)
        print(z)
        print(z.shape)
        #z = self.relu(self.fc1(z))
        #z = F.dropout(z, 0.3)
        recon = torch.sigmoid(z)
        return recon
</code></pre>
<p>gives the following error:</p>
<pre><code class="lang-auto">tensor([[[0.3054, 0.0000, 0.0000,  ..., 0.0000, 0.4342, 0.0000]],

        [[0.4361, 0.0000, 0.0000,  ..., 0.0206, 0.5194, 0.0000]]],
       device='cuda:0', grad_fn=&lt;ReluBackward0&gt;)
torch.Size([2, 1, 672])
tensor([[[0.3054, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
         [0.4938, 0.0000, 0.0000,  ..., 0.0000, 0.2481, 0.9124],
         [0.5702, 0.0000, 0.0000,  ..., 0.4131, 0.1147, 0.0842],
         ...,
         [0.0000, 0.1210, 0.5313,  ..., 0.6218, 0.3895, 0.0000],
         [0.0000, 0.0000, 0.0000,  ..., 0.0850, 0.5332, 0.1528],
         [0.2276, 0.0021, 0.0000,  ..., 0.0000, 0.4342, 0.0000]],

        [[0.4361, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
         [0.3129, 0.0000, 0.0000,  ..., 0.0000, 0.3701, 1.3570],
         [0.8684, 0.0000, 0.0000,  ..., 0.2555, 0.4145, 0.0000],
         ...,
         [0.0000, 0.4524, 0.8189,  ..., 0.8206, 0.2301, 0.0000],
         [0.0000, 0.0000, 0.0000,  ..., 0.4690, 0.3480, 0.0000],
         [0.0000, 0.0000, 0.0000,  ..., 0.0206, 0.5194, 0.0000]]],
       device='cuda:0', grad_fn=&lt;ViewBackward&gt;)
torch.Size([2, 32, 21])
Traceback (most recent call last):

RuntimeError: size mismatch, m1: [64 x 21], m2: [2 x 672] at C:/w/1/s/tmp_conda_3.7_044431/conda/conda-bld/pytorch_1556686009173/work/aten/src\THC/generic/THCTensorMathBlas.cu:268

</code></pre>

username: Oli
cooked: <p>I don’t think you can include a view into your sequential. Saw a post where they tried to make it into a layer but I wouldn’t bother. You’re not going to see that much speedup anyways. I suggest that you do something like this</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from collections import OrderedDict

class MyModel(nn.Module):
  def __init__(self, zdim):
    super().__init__()
    self.fc1 = nn.Linear(zdim, 672)
    self.decode = nn.Sequential(OrderedDict([
            ('conv1',nn.ConvTranspose1d(32, 32, 8, 2, padding=3)),
            ('relu1',nn.ReLU()),
            ('bn1',nn.BatchNorm1d(32)),
            ('conv2',nn.ConvTranspose1d(32, 32, 8, 2, padding=3)),
            ('relu2',nn.ReLU()),
            ('bn2',nn.BatchNorm1d(32)),
            ('conv3',nn.ConvTranspose1d(32, 16, 8, 2, padding=3)),
            ('relu3',nn.ReLU()),
            ('bn3',nn.BatchNorm1d(16)),
        ]))
        
        
  def forward(self, z):
    z = F.relu(self.fc1(z))
    print(z)
    print(z.shape)
    z = z.view(-1, 32, 21)
    z = self.decode(z)
    print(z.shape)


zdim = 10
model = MyModel(zdim)
inputs = torch.randn(2, zdim)
outputs = model(inputs)
</code></pre>

username: chaslie
cooked: <p>Oli,</p>
<p>thanks for this you are a life saver. I tried putting the fc1 layer into a another sequential slot but that didn’t work.</p>
<p>Your solution is neat and works…</p>
<p>Chaslie</p>