L1-regularization for a single matrix

username: Gabrer
cooked: <p>I have a hierarchical model with many components.<br>
I want one particular matrix to be sparse and to do so, I am trying to applying L1 regularization to only this matrix involved in my architecture.<br>
So far, I have found discussions about applying L1 reg. to the final loss function, but in this way, I would force the L1 on the overall model, while I want just one matrix to be sparse.</p>
<p>Is there any way to do so?</p>

username: ptrblck
cooked: <p>You can add the L1 regularization to your loss and call <code>backward</code> on the sum of both.<br>
Here is a small example using the weight matrix of the first linear layer to apply the L1 reg:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 2)
)

x = torch.randn(10, 10)
target = torch.empty(10, dtype=torch.long).random_(2)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3)

for epoch in range(1000):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, target)
    l1_norm = torch.norm(model[0].weight, p=1)
    loss += l1_norm
    loss.backward()
    optimizer.step()
    
    print('Epoch {}, loss {}, mat0 norm {}'.format(
        epoch, loss.item(), l1_norm.item()))
</code></pre>
<p>If you comment out <code>loss += l1_norm</code> you’ll see, that the norm won’t necessarily be decreased.</p>

username: Gabrer
cooked: <p>Thank you, looks what I was looking for!<br>
In this way, even the weights in the second first layer will be affected by the L1 reg, right?</p>

username: Deepak
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Here you have shown how to apply l1 regularization on a single layer, using <code>torch.norm(model[0].weight, p=1)</code>.</p>
<p>What if I have, say 10 layers and want to apply l1 regularization on all of them. Can I do it in a one-shot, or do I need to iterate over each layer using <code>model.parameters()</code>?</p>

username: ptrblck
cooked: <p>You would have to iterate the parameters as shown in <a href="https://discuss.pytorch.org/t/how-does-one-implement-weight-regularization-l1-or-l2-manually-without-optimum/7951/2">this example</a>.</p>

username: ahmadreza9
cooked: <p>Thank you for your answer. Now I want to know why your solution</p>
<pre><code class="lang-auto">torch.norm(model[0].weight, p=1)
</code></pre>
<p>doesn’t work for below network?</p>
<pre><code class="lang-auto"> l1_norm = torch.norm(model[2].weight, p=2)
TypeError: 'Net_test' object is not subscriptable
</code></pre>
<pre><code class="lang-auto">class Net_test(nn.Module):

    def __init__(self):
        super(Net_test, self).__init__()
        self.fc1 = nn.Linear(10, 10)
        self.fc2 = nn.Linear(10, 2)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x


model = Net_test()

x = torch.randn(10, 10)
target = torch.empty(10, dtype=torch.long).random_(2)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3)

for epoch in range(10):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, target)
    l1_norm = torch.norm(model[2].weight, p=1)
    loss += l1_norm
    loss.backward()
    optimizer.step()

    print('Epoch {}, loss {}, mat0 norm {}'.format(
        epoch, loss.item(), l1_norm.item()))

</code></pre>

username: ahmadreza9
cooked: <p>I found the solution <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> (I was careless about it)</p>
<pre><code class="lang-auto">l1_norm = torch.norm(model.fc1.weight, p=1)
</code></pre>

username: rasoulnorouzi
cooked: <p>See This:</p>
<aside class="quote" data-post="7" data-topic="2646">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasoulnorouzi/48/43293_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-implement-custom-regularization-losses-on-the-weights/2646/7">How to Implement Custom Regularization Losses on the Weights?</a> 
  </div>
  <blockquote>
    This what I do: 
class YourModel(nn.Module):
    
    def __init__(self,x,y):
        super(MF, self).__init__()
         pass

    def init_weights(self):
        pass

    def regularization(self):
        return your regularization term 
        # example: return torch.sum(torch.pow(self.x.weight, 2)) + torch.sum(torch.pow(self.y.weight, 2))

    def forward(self):
        pass

and for Loss: 
loss = criterion(pred, target) + lambda*(model.regularization())
  </blockquote>
</aside>