Sector Vector Machine Classifier

Berny: <p>Hi. I have to implement an autoencoder network with a SVM as classifier. The input of the classifier is the latent space generated by the autoencoder. Also, I have to sum the loss of the AE and that of the SVM. I used this code:</p>
<pre><code>    for epoch in range(NUM_EPOCHS):
    for batch_tr, (XTrain, YTrain) in enumerate(train_loader):
        XTrain = XTrain.to(device)
        optimizer.zero_grad()
        recon_batch, mu, logvar, latent_space = model(XTrain)
        predictions = svm_model(latent_space)
        loss_svm_train = svm_loss_criteria(predictions, YTrain)
        loss = loss_mse(recon_batch, XTrain, mu, logvar) + loss_svm_train
        svm_optimizer.zero_grad()
        svm_optimizer.step()
        loss.backward()
</code></pre>
<p>It generates the following error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 2]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. How can I fix it?</p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce the issue, please?</p>

Berny: <p>This is what I have for the svm:</p>
<pre><code>class SVMLoss(nn.Module):
    def __init__(self):
        super(SVMLoss, self).__init__()

   def forward(self, predictions, targets):
        return torch.sum(torch.clamp(1 - predictions.t()*targets, min=0))/TRAIN_BATCH_SIZE
</code></pre>
<p>Then, inside the main I used this:</p>
<pre><code>optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE, 
weight_decay=L2_REGULARIZATION)
loss_mse = CustomLoss(1.5)
train_loss = []
svm_model = nn.Linear(LATENT_FEATURES, NUM_CLASSES)
svm_loss_criteria = SVMLoss()
svm_optimizer = torch.optim.SGD(svm_model.parameters(), lr=LEARNING_RATE, 
momentum=MOMENTUM)
</code></pre>
<p>I think the problem is to do the gradient backpropagation of two different classes.</p>

ptrblck: <p>Your code is unfortunately not executable, but the posted code snippets work fine:</p>
<pre><code class="lang-python">class SVMLoss(nn.Module):
    def __init__(self):
        super(SVMLoss, self).__init__()

    def forward(self, predictions, targets):
        return torch.sum(torch.clamp(1 - predictions.t()*targets, min=0))/TRAIN_BATCH_SIZE
    
TRAIN_BATCH_SIZE = 2
LEARNING_RATE = 1e-3
L2_REGULARIZATION = 0.1
LATENT_FEATURES = 2
NUM_CLASSES = 2

model = nn.Linear(LATENT_FEATURES, NUM_CLASSES)
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE, weight_decay=L2_REGULARIZATION)
loss_mse = nn.MSELoss()
svm_loss_criteria = SVMLoss()

x = torch.randn(TRAIN_BATCH_SIZE, LATENT_FEATURES)
y = torch.randn(TRAIN_BATCH_SIZE, NUM_CLASSES)

for epoch in range(1000):
    optimizer.zero_grad()
    out = model(x)
    loss = loss_mse(out, y) + svm_loss_criteria(out, y)
    loss.backward()
    optimizer.step()
    print("epoch {}, loss {}".format(epoch, loss.item()))
</code></pre>

Berny: <p>The code now works, thank you very much, but you removed the  autoencoder optimizer:</p>
<pre><code>optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE, 
weight_decay=L2_REGULARIZATION)
</code></pre>
<p>is there a way to mantain it together with the svm optimizer? Substantially, I’d like to train both the AE and the SVM.</p>

ptrblck: <p>Yes, I removed it as it was undefined in your code snippets, but it should be possible to use both optimizers with both models.</p>

Berny: <p>Sorry, I forgot the autoencoder model:</p>
<pre><code>model = Autoencoder(1.5).to(device)
</code></pre>
<p>However, I solved by keeping only an optimizer for both the models:</p>
<pre><code>params = list(model.parameters()) + list(svm_model.parameters())
optimizer = optim.Adam(params, lr=LEARNING_RATE, weight_decay=L2_REGULARIZATION)
</code></pre>
<p>Can be a good method?</p>

ptrblck: <p>Sure, using a single optimizer should also work assuming both models should be updated with <code>Adam</code> now.</p>