How to create compound loss MSE + L1-norm regularization

com: <p>Hello!</p>
<p>I am trying to create a compound loss function where the first part is MSELoss and the second part is the L1-norm regularization of the model’s parameters</p>
<p>The fest part is simple</p>
<p>MSEloss = nn.MSELoss()<br>
loss = MSEloss(rec_x, x)</p>
<p>But how to attach the second part?<br>
I appreciate your help!</p>

Crazyai: <p>You can use <code>nn.L1Loss</code>. See this post <a href="https://discuss.pytorch.org/t/simple-l2-regularization/139/2">Simple L2 regularization?</a>.</p>

com: <p>Thank you for your answer! I didn’t mean to use L1 loss (where predicted and actual values are in use) I need to use L1 norm regularization. My goal is to fine high weights of the model.</p>

Crazyai: <p>You can compute L1 regularization manually like this:</p>
<pre><code class="lang-auto">regularization_loss = 0
for param in model.parameters():
    regularization_loss += torch.sum(torch.abs(param))
</code></pre>

com: <p>But should not it be back-propagated?</p>
<p>Because, MSELoss for sure is backpropagated in terms of weights, so similarly L1-norm regularization should be backpropagated.</p>

Crazyai: <p>It will do backward propagation. If you are using pytorch 0.4.0, you can check it like this:</p>
<pre><code class="lang-auto">## x in range [0, 1]
x = torch.rand(3,2,requires_grad=True)
loss = torch.sum(torch.abs(x))
loss.backward()
## gradient should be all one
x.grad
</code></pre>

com: <p>Thanks for your reply!</p>
<p>The problem is I am obligated to use pytorch 0.3.0 and it seems this version doesn’t support requires_grad on Tensors.</p>
<p>So I think I should somehow wrap the data into Variable <a href="http://pytorch.org/docs/0.3.0/autograd.html" rel="nofollow noopener">http://pytorch.org/docs/0.3.0/autograd.html</a><br>
But how to do so I don’t know.</p>

Crazyai: <p>Try this if you are using version 0.3.X.</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
## x in range [0, 1]
x = torch.rand(3,2)
x = Variable(x, requires_grad=True)
loss = torch.sum(torch.abs(x))
loss.backward()
## gradient should be all one
x.grad.data
</code></pre>

com: <p>So let’s get back to my tasks. I have to compose MSE loss with L1-norm regularization (among all layers’ weights)</p>
<p>I know how to iterate over all layers</p>
<pre><code class="lang-auto">    for name, W in model.named_parameters():
            l1 = W.norm(p=1)
</code></pre>
<p>But how to add all weights to Variable</p>
<p>The following methods don’t work</p>
<blockquote>
<pre><code>    x = Variable(self.enc_linear1, requires_grad=True)
    x = Variable(W, requires_grad=True)
</code></pre>
</blockquote>

com: <p>Does it look right?</p>
<blockquote>
<pre><code> l1_reg = Variable( torch.FloatTensor(1), requires_grad=True)

 for name, W in self.named_parameters():
    l1_reg = l1_reg + W.norm(1)
</code></pre>
</blockquote>

MrGreensjtu: <p>I am trying to do the same thing as your question, so i write the following code, but it don’t work. Have you find any solution?</p>
<pre><code class="lang-auto">loss_func = t.nn.MSELoss()
optimizer = t.optim.SGD(net.parameters(), lr)

# train the neural network
for epoch in range(EPOCH):
    for i, data in enumerate(train_loader):
        inputs, labels = data
        inputs, labels = Variable(inputs), Variable(labels)
        prediction = net(inputs)
        loss = loss_func(prediction, labels)
        for name, param in net.named_parameters():
            if 'weight' in name:
                L1_1 = Variable(param, requires_grad=True)
                L1_2 = t.norm(L1_1, 1)
                L1_3 = L1_lambda * L1_2
                loss = loss + L1_3
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>

Daniel_Ortega: <p>I’m trying to do the same thing you did, could you solve the problem?</p>