Extracting reduced dimension data from autoencoder in pytorch

guitarman: <p>Hi!</p>
<p>I have defined my autoencoder in pytorch as following:</p>
<pre><code class="lang-auto">        self.encoder = nn.Sequential(
            nn.Conv2d(input_shape[0], 32, kernel_size=1, stride=1),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=1, stride=1),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=1, stride=1),
            nn.ReLU()
        )

        self.decoder = nn.Sequential(
            nn.Conv2d(64, 64, kernel_size=1, stride=1),
            nn.ReLU(),
            nn.Conv2d(64, 32, kernel_size=1, stride=1),
            nn.ReLU(),
            nn.Conv2d(32, input_shape[0], kernel_size=1, stride=1),
            nn.ReLU(),
            nn.Sigmoid()
        )
</code></pre>
<p>Everything works fine, I can train it.</p>
<p>But, what I need is to get a reduced dimension encoding which requires creating a new linear layer of the dimension N much lower than the image dimension so that I can extract the activations.</p>
<p>If anybody can help me with fitting a linear layer in the decoder part I would appreciate (i know how to Flatten() the data, but I guess I need to “unflatten” it again to interface with the Conv2d layer again)</p>
<p>Thank you in advance.</p>

guitarman: <p>After doing some research on pytorch I have gotten the following</p>
<pre><code class="lang-auto">self.encoder = nn.Sequential(
    nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Conv2d(32, 64, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(64, 8, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.MaxPool2d(7, stride=1)
)

self.decoder = nn.Sequential(
    nn.ConvTranspose2d(8, 64, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 32, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(32, input_shape[0], kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Sigmoid()
)
</code></pre>
<p>This gives me a 8-dimensional bottleneck at the output of the encoder which works fine torch.Size([1, 8, 1, 1]) . Just the way I wanted to in the “bottleneck” layer.</p>
<p>What I cannot do is train the autoencoder with:</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
</code></pre>
<p>The decoder gives me an error:</p>
<p><code>Calculated padded input size per channel: (3 x 3). Kernel size: (4 x 4). Kernel size can't be greater than actual input size</code></p>

ptrblck: <p>The output of your first <code>nn.ConvTranspose2d</code> layer will be <code>[batch_size, 64, 3, 3]</code>, which is too small for the next one, which uses a kernel size of 4.<br>
You could use a kernel size of 4 for the first <code>nn.ConvTranpose2d</code> layer.<br>
This, however, will yield an error in the last one, since you are using <code>nn.Conv2d</code> layers afterwards in your decoder. Is that a typo or do you really want to lower the spatial size again?</p>

guitarman: <p>Thank you for the fast reply.</p>
<p>I have managed to get my autoencoder “right” <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> , I have a bottleneck layer of 2x2x2 which gives me 8 clusters like this:</p>
<pre><code class="lang-auto">        self.encoder = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(64, 2, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.MaxPool2d(6, stride=1)
        )

        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(2, 64, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.ConvTranspose2d(64, 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.ConvTranspose2d(32, 1, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x
</code></pre>
<p>and my training is simple, the loss is how much the prediction differs from the original image:</p>
<pre><code class="lang-auto">
    distance = nn.MSELoss()

    loss = distance(states,output)
</code></pre>
<p>but now the problem is that the network is not learning (the loss is constant at 10k which is bad)</p>
<p>even with a high learning rate:</p>
<p><code>optimizer_encoder = optim.SGD(model2.parameters(), lr=0.005)</code></p>
<p>I have no idea why <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

guitarman: <p>I have removed</p>
<pre><code>       nn.ReLU(),
       nn.Sigmoid()
</code></pre>
<p>on the encoder end and now the loss is in rather normal range and decreasing.</p>
<p>If anyone has the same problem or is creating an unsupervised autoencoder <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>