VAE with Supervised Multilabel classification

Anuj_Daga: <p>Hello All,<br>
I am trying to train a Variational Auto Encoder along with supervised multilabel Classification.<br>
As my Binary Cross-Entropy loss and Loss of a VAE both are performing well, I wonder where I am doing badly.<br>
One More quick question, the latent representation vector is represented is Z or mean?</p>
<pre><code class="lang-auto">class GCNModelVAE(nn.Module):
    def __init__(self, input_feat_dim, hidden_dim1, hidden_dim2,num_classes, dropout):
        super(GCNModelVAE, self).__init__()
        self.gc1 = GraphConvolution(input_feat_dim, hidden_dim1, dropout, act=F.relu)
        self.gc2 = GraphConvolution(hidden_dim1, hidden_dim2, dropout, act=lambda x: x)
        self.gc3 = GraphConvolution(hidden_dim1, hidden_dim2, dropout, act=lambda x: x)
        self.dc = InnerProductDecoder(dropout, act=lambda x: x)
        self.sigmoid = nn.Sigmoid()
        self.classifier = Classifier(hidden_dim2,hidden_dim1,num_classes,p=dropout)

    def encode(self, x, adj):
        hidden1 = self.gc1(x, adj)
        return self.gc2(hidden1, adj), self.gc3(hidden1, adj)

    def reparameterize(self, mu, logvar):
        if self.training:
            std = torch.exp(logvar)
            eps = torch.randn_like(std)
            return eps.mul(std).add_(mu)
        else:
            return mu

    def forward(self, x, adj):
        mu, logvar = self.encode(x, adj)
        z = self.reparameterize(mu, logvar)
        return self.dc(z), mu, logvar, mu, mu

def loss_function(preds, labels, mu, logvar,pred_labels, true_labels, n_nodes, norm, pos_weight):
    cost2 = norm * F.binary_cross_entropy_with_logits(pred_labels, true_labels)
    cost1 = norm * F.binary_cross_entropy_with_logits(preds, labels, pos_weight=pos_weight)
    # see Appendix B from VAE paper:
    # Kingma and Welling. Auto-Encoding Variational Bayes. ICLR, 2014
    # https://arxiv.org/abs/1312.6114
    # 0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
    KLD = -0.5 / n_nodes * torch.mean(torch.sum(1 + 2 * logvar - mu.pow(2) - logvar.exp().pow(2), 1))
    return KLD+cost1+cost2
</code></pre>
<p>cost2 represents my loss for Supervision.<br>
Lets suppose predicts_label=[-0.0013, -0.0006, -0.0017,  0.0011, -0.0030,  0.0052] and true_labels=[1 0 0 1 1 0]<br>
Thanks again,I wish to see some nice clusters for the latent space representation.<br>
I look forward to hearing from you soon.</p>

Anuj_Daga: <p>For example, I am supposed to get overlapping clusters as seen in an image in left in ideal case ,still maintaining a position in a latent space. But the one I am getting are shown in right image with red colour, where we can’t distinguish between the specific area/position of a cluster highlighted even after I train it over their supervised Labels <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3faa25bd0dad1a764fccc64f0a6d5bf7f9708b3a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3faa25bd0dad1a764fccc64f0a6d5bf7f9708b3a" title="Proposed vs Real"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3faa25bd0dad1a764fccc64f0a6d5bf7f9708b3a_2_690x212.jpeg" alt="Proposed vs Real" data-base62-sha1="95cCquGJW4liiMDs6LK0ZGRYwdA" width="690" height="212" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3faa25bd0dad1a764fccc64f0a6d5bf7f9708b3a_2_690x212.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3faa25bd0dad1a764fccc64f0a6d5bf7f9708b3a_2_1035x318.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3faa25bd0dad1a764fccc64f0a6d5bf7f9708b3a_2_1380x424.jpeg 2x" data-dominant-color="FDFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Proposed vs Real</span><span class="informations">1924×592 87.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thanks again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>