Autoencoder for classification

Berny: <p>Hi. I’m implementing an autoencoder and a FC network in parallel to make predictions on a dataset.<br>
The following is a piece of code I used:</p>
<p>for epoch in range(epochs):<br>
loss = 0<br>
if <strong>name</strong> == ‘<strong>main</strong>’:<br>
for (batch_features, _) in train_loader:<br>
# reshape mini-batch data to [N, 2105] matrix<br>
# load it to the active device<br>
batch_features = batch_features.view(-1, 2105).to(device)<br>
#batch_features=torch.tensor(batch_features)</p>
<pre><code>        # reset the gradients back to zero
        # PyTorch accumulates gradients on subsequent backward passes
        optimizer.zero_grad()
    
        # compute reconstructions
        outputs = model(batch_features)

        # compute training reconstruction loss
        train_loss = criterion(outputs, batch_features)
    
        # compute accumulated gradients
        train_loss.backward()
    
        # perform parameter update based on current gradients
        optimizer.step()
    
        # add the mini-batch training loss to epoch loss
        loss += train_loss.item()

        # compute the epoch training loss
        loss = loss / len(train_loader)

        # display the epoch training loss
        print("epoch : {}/{}, loss = {:.6f}".format(epoch + 1, epochs, loss))
</code></pre>
<p>I get the following error corresponding to the line  train_loss = criterion(outputs, batch_features):  ‘tuple’ object has no attribute ‘size’ .<br>
Could someone help me to fix it?</p>

Andrei_Cristea: <p>Can you print this out right before the problematic line:</p>
<pre><code class="lang-auto">print(type(outputs)) 
</code></pre>
<p>If that’s a <code>tuple</code>, you need to make it into a <code>Tensor</code> before passing to <code>criterion</code>.</p>

Berny: <p>Thank you for your answer Andrei. Yes, it seems to be a tuple, so I tried with: outputs=torch.Tensor(outputs)<br>
but have the error ‘only one element tensors can be converted to Python scalars’. Maybe, is there a more appropriate way to transform it to a Tensor?</p>

Andrei_Cristea: <p>Sounds like <code>outputs</code> is a tuple of Tensors. I can’t give a generic correct answer without knowing what exactly your model outputs, since that’s a function of your model design.</p>
<p>Can you share the <code>forward</code> function of your <code>model</code> so we can see what the output consists of?</p>
<p>Mechanically something like this might work, but we should be first sure that we understand what <code>outputs</code> really is before hammering it in:</p>
<pre><code class="lang-auto">outputs = torch.stack(outputs)
</code></pre>

Berny: <p>The forward function is:</p>
<p>def forward(self, input):<br>
activation = self.encoder_hidden_layer(input)<br>
activation = torch.relu(activation)<br>
code = self.encoder_output_layer(activation)<br>
code = torch.relu(code)<br>
x = self.decoder_hidden_layer(code)<br>
x = torch.relu(x)<br>
x = self.decoder_output_layer(x)<br>
x = torch.relu(x)<br>
y = F.relu(self.fc1(code))<br>
y = F.relu(self.fc2(y))<br>
y = torch.sigmoid(self.fc3(y))<br>
return x,y</p>
<p>x represents the reconstructed input data, while y should be the predicted output (to perform in parallel with x).</p>

Andrei_Cristea: <p>Ok so in that case I think you want to take only the first element for the loss function, which is the <code>x</code> inside your <code>forward</code> function, since that’s what should be the reconstructed input data:</p>
<pre><code class="lang-auto">train_loss = criterion(outputs[0], batch_features)
</code></pre>
<p>By the way you can use 3 back quotes around your code to format it in a more readable way.</p>

Berny: <p>Now the code works, thank you very much!</p>

Berny: <p>Hello, now I want to make predictions (y) and use a softmax function (instead of sigmoid) as last layer such as:</p>
<p>‘’‘y = F.softmax(self.fc3(y),dim=1)’‘’</p>
<p>It returns 6 probability values. How to consider also the second element (y) for the loss function and combine it with the first one? I tried with:</p>
<p>‘’‘train_loss1 = criterion(outputs[0], batch_features)’‘’<br>
‘’‘train_loss2 = criterion(outputs[1], labels )’‘’<br>
‘’‘train_loss = train_loss1 + train_loss2’‘’</p>
<p>But the problem is that labels is 961 dimension and outputs[1] only 6, how to fix it?</p>

Andrei_Cristea: <p>Hi, it should return a tensor shaped like <code>n_batch</code> x <code>n_label</code>, where the sum of the values within each batch is 1. Labels should be shaped like <code>n_batch</code>. If you’re using something like <code>CrossEntropyLoss</code>, that should work. However, please note that <code>CrossEntropyLoss</code> does the <code>Softmax</code> for you, all you need to return is the raw values without the Softmax:<br>
<code>y = self.fc3(y)</code></p>
<p>Also, you may want to use a different loss function (criterion) for the reconstruction loss and the classification loss.</p>

Berny: <p>Yes sounds good, I’ll try. Superthanks!</p>