Layer weights in custom Loss results in autograd problem

Leroy_Freyler: <p>Hello,</p>
<p>I want to introduce a kind of total variation penalty.</p>
<p>I want to realize this by the fact, that the neural network is penalized for it, if the neighboured weights differ strongly.</p>
<p>But I want to apply this only in one layer. So I have added this additionally in the loss function.</p>
<p>My code parts are therefore as follows:</p>
<pre><code class="lang-python">class CNN(nn.Module):
   def __init__(self):
       super(CNN, self).__init__()

       # decoder 
       self.conv1 = nn.Conv1d(in_channels=1, out_channels=1, kernel_size=50, bias=False)
       self.conv2 = nn.BatchNorm1d(1)
       self.conv3 = nn.Conv1d(in_channels=1,out_channels=1, kernel_size=30, bias=False)
       self.conv4 = nn.BatchNorm1d(1)
       self.conv5 = nn.Conv1d(in_channels=1,out_channels=1, kernel_size=11, bias=False)
       self.conv6 = nn.BatchNorm1d(1)
       self.ASC = nn.Softmax(dim=1)
       # encoder 
       self.DEC = nn.Linear(in_features=3,out_features=91)

   def forward(self, x):
       x = F.leaky_relu(self.conv1(x))
       x = F.leaky_relu(self.conv2(x))
       x = F.leaky_relu(self.conv3(x))
       x = F.leaky_relu(self.conv4(x))
       x = F.leaky_relu(self.conv5(x))
       x = x*3.5
       x = F.leaky_relu(self.conv6(x))
       a = self.ASC(x)
       x = F.leaky_relu(self.DEC(a))
       return x, a
net = CNN()
</code></pre>
<pre><code class="lang-python">def train(net, trainloader, NUM_EPOCHS, cur_epoch=0):
    train_loss = []
    for epoch in range(cur_epoch, NUM_EPOCHS):
        running_loss = 0.0
        for data in trainloader:
            inputData = data
            inputData = inputData.to(device)
            optimizer.zero_grad()
            outputData ,a = net(inputData)
            loss_MSE = criterion(outputData, inputData)
            loss_TVPenalty = TV_Penalty(net.DEC)
            loss = loss_MSE + loss_TVPenalty
            loss.backward()
            optimizer.step()
            net.DEC.weight.data.clamp_(min=0)
            running_loss += loss.item()
        loss = running_loss / len(trainloader)
        train_loss.append(loss)
    return train_loss
</code></pre>
<pre><code class="lang-python">def TV_Penalty(Layer):
    loss = torch.pow((Layer.weight.data[1:]-Layer.weight.data[:-1]), 2).sum()*TV_PENALTY_CONSTANT
    return loss

</code></pre>
<p>If I use it this way, the autograde function of Pytorch is not applied to the <code>python def TV_Penalty(Layer): </code> function.</p>
<p><strong>How to use the autograd in Pytocht on the structure of the weights of the network itself? So then this is considered in an additional loss function.</strong></p>

ptrblck: <p>You should be able to use the <code>.weight</code> parameters directly, instead of their <code>.data</code> attribute, which Autograd cannot track and which should thus be avoided. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Leroy_Freyler: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="102137">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>.weight</code> parameters directly, instead of their <code>.data</code> attribute, which Autograd cannot track and which should thus be avoided. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Hihih, many thanks, works like a charm. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>