How to Implement Custom Regularization Losses on the Weights?

username: Carson_McNeil
cooked: <p>Hello! I’ve been searching quite a bit, but I’m having trouble finding the proper way to implement a custom regularization loss on the weights. (Say I wanted to implement L3Loss, but only on a particular layer. Or a Laplacian (2nd/derivative)  loss on a subset of weight tensors along certain dimensions?) I’m interested in losses that are easily implemented using only torch operations on Variables. (i.e. no custom CUDA code, no implementing autograd)</p>
<p>In Tensorflow, you have access to the weight tensors, and there’s really no difference between a loss on the network output, and a regularization loss on some of the weights as far as the framework is concerned. What I really want to be able to do is something like the following:</p>
<pre><code>first_layer = nn.Conv2d(...)
my_regularization = my_custom_loss_function(first_layer.weight)
model = nn.Sequential(first_layer, nn.Conv2d(...), nn.Conv2d(...), ...)
loss = my_regularization + some_other_loss(model.forward())
model.backward()
optimizer.step()
</code></pre>
<p>However, the above code doesn’t work at all. Do I need to define my loss as a Module? If so, how do I get at the weights for only a certain layer? Thanks!</p>

username: jekbradbury
cooked: <p>That code looks very tensorflow – you have to do initialization things at init time and runtime things in your training loop. So lines 2, 4, 5, and 6 should be in your training loop while 1 and 3 are in your initialization.</p>

username: Carson_McNeil
cooked: <p>Yea, I’ve figured it out. Thank you, and you’re right, I wasn’t really grasping all of the framework differences yet (and probably still am not). It would be wonderful if there were a “PyTorch for Tensorflow users” tutorial XD.</p>

username: Lustr
cooked: <p>Have you solved your problem? I have the same problem.<br>
how do I get at the weights for only a certain layer? and how to Implement Custom Regularization Losses on the Weights?</p>

username: Eta_C
cooked: <p>Assume we want to customize L1 Loss Regularization.</p>
<p>An example for one parameter.</p><aside class="quote quote-modified" data-post="2" data-topic="28088">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/l1-regularization-for-a-single-matrix/28088/2">L1-regularization for a single matrix</a> 
  </div>
  <blockquote>
    You can add the L1 regularization to your loss and call backward on the sum of both. 
Here is a small example using the weight matrix of the first linear layer to apply the L1 reg: 
model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 2)
)

x = torch.randn(10, 10)
target = torch.empty(10, dtype=torch.long).random_(2)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3)

for epoch in range(1000):
    optimizer.zero_grad()
    output = mode…
  </blockquote>
</aside>

<p>An example for each layer.</p><aside class="quote quote-modified" data-post="2" data-topic="139">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/48/58_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/simple-l2-regularization/139/2">Simple L2 regularization?</a> 
  </div>
  <blockquote>
    Hi, 
The L2 regularization on the parameters of the model is already included in most optimizers, including optim.SGD and can be controlled with the weight_decay parameter as can be seen in <a href="http://pytorch.org/docs/optim.html#torch.optim.SGD" rel="noopener nofollow ugc">the SGD documentation</a>. 
L1 regularization is not included by default in the optimizers, but could be added by including an extra loss nn.L1Loss in the weights of the model. 
l1_crit = nn.L1Loss(size_average=False)
reg_loss = 0
for param in model.parameters():
    reg_loss += l1_crit(param)

factor = 0.0005
lo…
  </blockquote>
</aside>


username: rasoulnorouzi
cooked: <p>This what I do:</p>
<pre><code class="lang-auto">class YourModel(nn.Module):
    
    def __init__(self,x,y):
        super(MF, self).__init__()
         pass

    def init_weights(self):
        pass

    def regularization(self):
        return your regularization term 
        # example: return torch.sum(torch.pow(self.x.weight, 2)) + torch.sum(torch.pow(self.y.weight, 2))

    def forward(self):
        pass
</code></pre>
<p>and for Loss:</p>
<p><code>loss = criterion(pred, target) + lambda*(model.regularization())</code></p>