Autoencoder and Classification inside the same model

Petrucio: <p>Hello everyone,</p>
<p>I am new to PyTorch . I would like to train a simple autoencoder and use the encoded layer as an input for a classification task (ideally inside the same model). This is my implementation:</p>
<pre><code class="lang-auto">class Mixed(nn.Module):
    def __init__(self, n_embedded):
        super(Mixed, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(X_train.shape[1], n_embedded),
            nn.ReLU()                 
        )
        self.decoder = nn.Sequential(
            nn.Linear(n_embedded, X_train.shape[1])
        )
        self.classifier = nn.Sequential(
            nn.Linear(n_embedded, 1),
            nn.Dropout(0.2),
            nn.Sigmoid()
        )
    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        out = self.classifier(encoded)
        return decoded, out
        
model = Mixed(40)
            
criterion1 = nn.MSELoss()
criterion2 = nn.BCELoss()
            
optimizer = optim.Adam(model.parameters(), lr=0.01)

epochs = 150
   
for epoch in range(epochs): 
    for inputs, labels in train_loader:
        inputs = Variable(inputs)
        labels = Variable(labels)       
        optimizer.zero_grad()
        decoded, out = model(inputs)
       
        loss1 = criterion1(decoded, inputs) 
        loss2 = criterion2(out, labels)
        loss = loss1 + loss2
        loss.backward()
        optimizer.step()
</code></pre>
<p>When I run the model, it generates a meaningful result but I just want to make sure that the architecture is correct.</p>
<p>Thank you for your time.</p>

ptrblck: <p>The code looks fine.<br>
Just a small side note: <code>Variables</code> are deprecated since PyTorch <code>0.4.0</code> so you can just use <code>tensors</code> instead in newer versions. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

Petrucio: <p>Thank you very much for your help!</p>

rzhang63: <p>Hi ptrblck,</p>
<p>I’m trying to implement a similar thing here, but</p>
<ol>
<li>use a pretrained network (e.g. mobilenetV2) as the classifier. So shall I just set <code>self.classifier=models.mobilenet_v2(pretrained = pretrained)</code>? Or is this not the correct way to do it?</li>
<li>I have two image datasets. I wish to use dataset 1 in both the training of the AE and the classifier, but use dataset 2 only in the training of the AE (I don’t have the labels for dataset 2). Is it still possible to implement these inside one model?</li>
</ol>
<p>Thank you in advance!</p>

ptrblck: <ol>
<li>
<p>A “classifier” in some CNNs such as VGG, ResNet etc. contains a few linear layers with activation functions between them. I’m not sure, if your <code>self.classifier</code> is supposed to work like that and to use the extracted features to output the predictions. If that’s the case, I wouldn’t use a complete pretrained model at this point.</p>
</li>
<li>
<p>This should be possible and you could use e.g. a flag inside the <code>forward</code> method to switch between different branches. However, you should also consider the validation or deployment use case, where you might now have the information where the sample is coming from.</p>
</li>
</ol>

ptrblck: <p>An SVM would also be a classifier, so I assume “instead of a classifier” means the <code>model.classifier</code> module.</p>
<p>It’s possible to feed your model outputs to other classifiers and the overall workflow depends what should be trained and which part of the model would be frozen.</p>
<p>E.g. if you do not want to train the PyTorch model anymore, but just use it’s features to train an SVM or another model, you could transform the outputs to numpy arrays and use e.g. scikit-learn or any other Python library.</p>
<p>However, if you want to train the PyTorch model, this won’t work out of the box, since Autograd won’t be able to track the numpy operations in the scikit-learn model and you would thus detach the computation graph by leaving PyTorch.</p>
<p>You could try to write the <code>backward</code> functions for all numpy operations manually via custom <code>torch.autograd.Function</code>s as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a> or use <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiMarginLoss.html#multimarginloss">MultiMarginLoss</a>, which should be similar to the HingeLoss used in SVMs (I haven’t looked deeply into potential differences, but if I recall correctly, other users were using this criterion for the “SVM loss”).</p>

ziva: <p>Thank You so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

kimback: <p>Are you able to see accuracy in a supervised manner?</p>

anu: <p>hi <a class="mention" href="/u/petrucio">@Petrucio</a> &amp; <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>is it necessary to add the losses, and then backward? I was also wondering if I could go through separate backward to evaluate them separately?</p>
<p>thanks,</p>

ptrblck: <p>You can separately call <code>backward</code> on different losses (and would need to use <code>loss1.backward(retain_graph=True)</code> to keep the intermediate tensors for the second backward call if necessary), which would accumulate the gradients in all used parameters.</p>

anu: <p>but setting <code>retain_graph=True</code> would connect the loss1 and loss2, right? I should set that attribute to True only if I want to connect them which summing them up before calling <code>backward</code> makes much more sense in that case. Additionally, is <code>retain_graph=True</code> necessary for the loss2 as well?</p>
<p>thanks!</p>

ptrblck: <p><code>retain_graph</code> doesn’t connect the losses, but keeps the intermediate activations after a <code>backward</code> call. This is necessary, if any other loss or output calling <code>.backward()</code> would need these activations from the forward pass to calculate the gradients.<br>
The “connection” between the losses is defined by the forward pass.<br>
E.g. if both losses were created using the same model (and thus parameters), the backward passes would calculate the gradients w.r.t. the same parameters as well.</p>

krishna511: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> sir<br>
I am using a bit similar concept , found this question relevant so did not raise  another one. I am trying  to train to separate models using convolutional autoencoders and then will use both the encoded outputs for final classification, There is a paper on this concept. one of My model is this</p>
<pre><code class="lang-auto">
     
        
        
class CNN1D(torch.nn.Module):
     def __init__(self, num_classes=7):
        super(CNN1D, self).__init__()     
        self.cn1=nn.Conv1d(1,32,5,1)
        self.mp1=nn.MaxPool1d(4,return_indices=True)
        self.cn2=nn.Conv1d(32,32,5,1)
        self.mp2=nn.MaxPool1d(4,return_indices=True)
        self.cn3=nn.Conv1d(32,64,5,1)
        self.mp3=nn.MaxPool1d(4,return_indices=True)
        self.cn4=nn.Conv1d(64,64,5,1)
        self.mp4=nn.MaxPool1d(2,return_indices=True)
        self.cn5=nn.Conv1d(64,128,5,1)
        self.mp5=nn.MaxPool1d(2,return_indices=True)
        self.cn6=nn.Conv1d(128,128,5,1)
        self.mp6=nn.MaxPool1d(2,return_indices=True)
        self.relu=nn.ReLU()
        
        
        self.cnt1=nn.ConvTranspose1d(128,128,5,1)
        self.mt1=nn.MaxUnpool1d(2)
        self.cnt2=nn.ConvTranspose1d(128,64,5,1)
        self.mt2=nn.MaxUnpool1d(2)  
        self.cnt3=nn.ConvTranspose1d(64,64,5,1)
        self.mt3=nn.MaxUnpool1d(2)
        self.cnt4=nn.ConvTranspose1d(64,32,5,1)
        self.mt4=nn.MaxUnpool1d(4)
        self.cnt4=nn.ConvTranspose1d(32,32,5,1)
        self.mt5=nn.MaxUnpool1d(4)
        self.cnt4=nn.ConvTranspose1d(32,1,5,1)
        self.mt6=nn.MaxUnpool1d(4)
        self.fc1=nn.Linear(32,1024)
    
     def forward(self,x):
         out=F.relu(self.cn1(x))
         out,ind1= self.mp1(out)
         out=F.relu(self.cn2(out))
         out,ind2=self.mp2(out)
         out=F.relu(self.cn3(out))
         out,ind3=self.mp3(out)
         out=F.relu(self.cn4(out))
         out,ind4=self.mp4(out)
         out=F.relu(self.cn5(out))
         out,ind5=self.mp5(out)
         out=F.relu(self.cn6(out))
         out,ind6=self.mp6(out)
         encoded=out
        
         out=self.mt1(out,ind6)
         out=F.relu(self.cnt1(out))
         out=self.mt2(out,ind5)
         out=F.relu(self.cnt2(out))
         out=self.mt3(out,ind4)
         out=F.relu(self.cnt3(out))
         out=self.mt4(out,ind3)
         out=F.relu(self.cnt4(out))
         out=self.mt5(out,ind2)
         out=F.relu(self.cnt5(out))
         out=self.mt6(out,ind1)
         out=F.relu(self.cnt6(out))
        
         
         
         return encoded, out
</code></pre>
<p>The second one is also similar , I am stuck here dont know how to proceed, individually i can train both these networks , but how to extract these two encoded faetues and finally feed them to a classifier, Plz  guide</p>

ptrblck: <p>Your code looks correct and you could feed both outputs to a new classifier.<br>
I’m not familiar with the wanted workflow, but assume you would like to stack these features somehow before processing them further?</p>

krishna511: <p>Yes sir exactly. I need to use these encoded features, from both the models. I wanted to ask what  should be the flow of model? Like do i need to train these models individually?<br>
and then somehow need to store these features ? and then through third model final classification or, is there any way to do all this in a single code?<br>
regards</p>

ptrblck: <p>You could add the classifier into the same model or use it as a separate one and I think it depends more on your coding style or which approach would be more convenient to use in the long run.</p>
<p>Anyway, assuming you want to use these two features and a single classifier on top, you could use something like:</p>
<pre><code class="lang-python">class CNN1D(torch.nn.Module):
     def __init__(self, num_classes=7):
        super(CNN1D, self).__init__()
        ...
        self.classifier = nn.Linear(in_features, nb_classes)
        
    def forward(self, x):
        ...
        encoded = out
        
        ...
        out=F.relu(self.cnt6(out))
        # flatten encoded
        encoded = encoded.view(encoded.size(0), -1)
        # concatenate both features in the feature dimension
        out = torch.cat((encoded, out), dim=1)
        out = self.classifier(out)
        return out
</code></pre>
<p>Note that I’ve added the flattening operation to <code>encoded</code>, as I assume it’s a 3-dimensional tensor.<br>
Also, you would have to check the <code>in_features</code> e.g. by adding a print statement in the <code>forward</code> and use this number in the <code>self.classifier</code> <code>in_features</code>.<br>
You could of course use a “better” classifier by e.g. using <code>nn.Sequential</code> with more layers.</p>

krishna511: <p>Thank you so much for this answer sir.<br>
Please correct me if m wrong…what I am  getting here is<br>
encoded is the bottleneck  or reduced representation of the input data,<br>
out is reconstructed representation of the data right?<br>
if yes then why to concatenate these two?</p>
<p>what is my actual question is I have two such models different preprocessing for both of them , I need to train both the models for this bottleneck features and finally concatenate them for final classification plz find the attached picture.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6ec7394fde2681ad9de7d565d055b0052cf752c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b6ec7394fde2681ad9de7d565d055b0052cf752c" title="forum"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6ec7394fde2681ad9de7d565d055b0052cf752c_2_690x291.jpeg" alt="forum" data-base62-sha1="q6duFgMeQUHj8460pjp77h5Qpgo" width="690" height="291" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6ec7394fde2681ad9de7d565d055b0052cf752c_2_690x291.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6ec7394fde2681ad9de7d565d055b0052cf752c_2_1035x436.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6ec7394fde2681ad9de7d565d055b0052cf752c.jpeg 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">forum</span><span class="informations">1216×513 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  Looking forward for your guidance</p>

shivammehta007: <p>As both your networks outputs a fully connected layer, I think the work is much simplified for you, you don’t even need to flatten outputs or anything just concatenating them and passing through a linear layer should do the job for you</p>
<pre><code class="lang-auto">class MergerModel(nn.Module):
	def __init__(self, model1, model2):
		super(MergerModel, self).__init__()
		self.model1 = model1
		self.model2 = model2

		self.fc2 = nn.Linear(output_length_of_model1 + output_length_of_model2, nb_classes)

	def forward(self, x):
		x1 = self.model1(x)
		x2 = self.model2(x)

		x = torch.cat((x1, x2), dim=1)

		final_output = F.softmax(self.fc2(x))

		return x
</code></pre>
<p>Something like this should solve your problem</p>

krishna511: <p>Thank you <a class="mention" href="/u/shivammehta007">@shivammehta007</a><br>
But how to call these two models?  I am using this function for training</p>
<pre><code class="lang-auto">Losses, Accuracies = fp.fit_sm(model=model, optimizer=optimizer, epochs=100,                                    trainloader=tr_loader, validloader=ts_loader,
                                   criterion=nn.CrossEntropyLoss(),
                                   device=device, verbose=True)

</code></pre>
<p>Here fp.fit_sm is a function for model training,<br>
I got these three models , model 1 , model 2 and merged model, but which model should i call here in this above snippet,I  am blank here ,how to call these function here plz guide.</p>

ptrblck: <p>I assume you are using a higher-level API, which provides this <code>fit_sm</code> method.<br>
If so, it should internally execute the forward and backward passes, as well as the optimization via:</p>
<pre><code class="lang-python">optimizer.zero_grad()
output = model(data)
loss = criterion(output, target)
loss.backward()
optimizer.step()
</code></pre>
<p>I don’t know what else is performed in this method, but these parts should be there at least.</p>
<p>Assuming you’ve floowed the suggestions and added the classifier to the entire model, you should be able to use it directly.<br>
Note that I don’t see how the data is passed to this <code>fit_sm</code> method, but I guess that it might be part of the <code>fp</code> object.</p>