Autoencoder + classifier - chance level learning

AlephG: <p>I am implementing a neural net model that has the following architecture:</p>
<pre><code class="lang-auto">        # Encoder
        self.encoder = nn.Sequential(encoder_config)

        # Decoder
        self.decoder = nn.Sequential(decoder_config)

        # Classifier
        self.classifier = nn.Sequential(classifier_config)
</code></pre>
<p>The idea is to first train the autoencoder part of the model then train the classifier on the encoded representations. I have tried many different layer configurations and it turns out the autoencoder portion never learns anything. Before, training the autoencoder, I freeze the classifier as follows:</p>
<pre><code class="lang-auto">    def freeze(self, network):

        for p in network.parameters():
            p.requires_grad = False
</code></pre>
<p>This is done by calling <code>model.freeze(model.classifier)</code>.</p>
<p>Here is an example of the latest layer configuration I used:</p>
<pre><code class="lang-auto">#Neural net layer config
encoder_config = OrderedDict({'lin1_encoder':nn.Linear(256, 216), 'norm1_encoder':nn.BatchNorm1d(216),
                              'sig1_encoder':nn.Sigmoid(), 'lin2_encoder':nn.Linear(216, 196),
                              'norm2_encoder':nn.BatchNorm1d(196), 'sig2_encoder':nn.Sigmoid()})
decoder_config = OrderedDict({'lin1_decoder':nn.Linear(196, 216),'norm1_decoder':nn.BatchNorm1d(216),
                              'sig2_decoder':nn.Sigmoid(), 'lin2_decoder':nn.Linear(216, 256),
                              'norm2_decoder':nn.BatchNorm1d(256), 'sig2_decoder':nn.Sigmoid()})
classifier_config = OrderedDict({'lin1_classifier':nn.Linear(196,128), 'norm1_classifier':nn.BatchNorm1d(128),
                                 'sig1_classifier':nn.Sigmoid(), 'lin2_classifier':nn.Linear(128, 32),
                                 'norm2_classifier':nn.BatchNorm1d(32), 'sig2_classifier':nn.Sigmoid(),
                                 'lin3_classifier':nn.Linear(32,2), 'norm3_classifier':nn.BatchNorm1d(2), 
                                 'sig3_classifier':nn.Sigmoid()})
</code></pre>
<p>I then train the autoencoder as follows (using data that has added Gaussian noise but even with 0 noise the model does not learn):</p>
<pre><code class="lang-auto">    def train_autoencoder(self, num_epochs, stimuli, batch_size, noise_factor, optimizer, criterion, scheduler, conv=False, verbose=False):

        running_loss = []
        test_loss = []
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

        # Load test and training datasets
        testloader = load_dataset_AE(stimuli=stimuli, batch_size=batch_size, noise_factor=0.0, conv=conv)
        trainloader = load_dataset_AE(stimuli=stimuli, batch_size=batch_size, noise_factor=noise_factor, conv=conv)

        for epoch in range(num_epochs):
            cur_running_loss = 0.0
            for i, (stimuli, target) in enumerate(trainloader):
                stimuli = stimuli.to(device)
                target = target.to(device)
                # Train mode
                self.train()
                # Forward pass
                x = self.forward(stimuli, decode=True)
                loss = criterion(x, target)
                # Backward pass
                optimizer.zero_grad()
                loss.backward(retain_graph=True)
                optimizer.step()
                # Add loss of current pass
                cur_running_loss += loss.item()

            cur_running_loss /= len(trainloader)
            running_loss.append(cur_running_loss)

            # Eval loss
            eval_loss = self.evaluate_AE(testloader, criterion)
            test_loss.append(eval_loss)

            if verbose:
                print('Autoencoder --&gt; Running Loss: {} \t Eval loss: {}'.format(cur_running_loss, eval_loss))

            # Scheduler
            scheduler.step(eval_loss)

        return running_loss, test_loss
</code></pre>
<p>However I configure the encoder portion, it learns [0.500, 0.500, 0.500, … , 0.500] and decoder seems to learn to output the approximately the zero vector. The training data are binary vectors. I use <code>torch.MSELoss()</code> and <code>torch.optim.Adam()</code> as loss and optimizer. Weirdly enough, for vectors of length 256, I get a very low loss when training the autoencoder (~0.002). Given that the encoder learns nothing, the classifier categorizes at chance. Lastly, an SVM can learn to classify this data with near perfect accuracy so I doubt this is a problem with my data. Thanks for your help!</p>