How does one implement Weight regularization (l1 or l2) manually without optimum?

Brando_Miranda: <p>I wanted to do it manually so I implemented it as follows:</p>
<pre><code class="lang-python">reg_lambda=1.0
l2_reg=0
for W in mdl.parameters():
    l2_reg += *W.norm(2)
batch_loss = (1/N_train)*(y_pred - batch_ys).pow(2).sum() + reg_lambda*l2_reg
## BACKARD PASS
batch_loss.backward() # Use autograd to compute the backward pass. Now w will have gradients
</code></pre>
<p>is this correct? the key part I care about is that the SGD update works correctly. i.e.:</p>
<pre><code class="lang-python">## SGD update
for W in mdl.parameters():
    delta = eta*W.grad.data
    W.data.copy_(W.data - delta) # W - eta*g + A*gdl_eps
</code></pre>
<p>has the <code>2w</code> term in the SGD update.</p>
<p>here is a related question: <a href="https://discuss.pytorch.org/t/simple-l2-regularization/139">Simple L2 regularization?</a></p>

smth: <p>it’s almost correct.</p>
<p><code>l2_reg</code> here is a python scalar, so operations done on it are not recorded for the autograd backward().<br>
Instead, you should make l2_reg to be an autograd Variable.</p>
<pre><code class="lang-python">l2_reg = None
for W in mdl.parameters():
    if l2_reg is None:
        l2_reg = W.norm(2)
    else:
        l2_reg = l2_reg + W.norm(2)
batch_loss = (1/N_train)*(y_pred - batch_ys).pow(2).sum() + l2_reg * reg_lambda
batch_loss.backward()
</code></pre>

Brando_Miranda: <p>Why not:</p>
<pre><code>    l2_reg = Variable( torch.FloatTensor(1), requires_grad=True)
    for W in mdl.parameters():
        l2_reg = l2_reg + W.norm(2)
</code></pre>
<p>(not sure if <code>l2_reg</code> should be 1x1 or 1).</p>

smth: <p>yes you can do this too.</p>

jaiabhayk: <p>I get the following error while trying this</p>
<p>File “…/lib/python3.6/site-packages/torch/autograd/variable.py”, line 146, in backward<br>
self._execution_engine.run_backward((self,), (gradient,), retain_variables)<br>
File “…/lib/python3.6/site-packages/torch/autograd/_functions/reduce.py”, line 200, in backward<br>
return input.mul(grad_output[0] / self.norm)<br>
ZeroDivisionError: float division by zero</p>
<p>Any suggestions ? (I guess its somehow getting self.norm as 0).</p>

jaiabhayk: <p>As a workaround, I just ensure the l2 norm of the weights is not 0 after initialization (which should be handled in the code I think).</p>

yibo: <p>Since the parameters are Variables, won’t l2_reg be automatically converted to a Variable at the end? I’m using l2_reg=0 and it seems to work.<br>
Also I’m not sure if OP’s formula for L2 reg is correct. You need the sum of every parameter element squared.</p>

Brando_Miranda: <p>W.norm(2) should be W.norm(2)**2, no?</p>

Brando_Miranda: <p>I Im missing a square, W.norm(2) should be W.norm(2)**2, no?</p>

yibo: <p>Yeah; but it’s probably more efficient to just do <code>torch.pow(W, 2).sum()</code>. Also you may want to multiply by 0.5 as a standard convention.</p>

nbansal90: <p>Further Printing the the value returned by the function, it surely seems to be going towards INF. But almost the same logic for custom regularization, works in tensorflow. Am I doing something wrong code wise.</p>
<p>Epoch: [0][0/391]       Time 6.739 (6.739)      Loss 3157501.5000 (3157501.5000)        Prec@1 10.938 (10.938)<br>
Variable containing:<br>
2.7529e+08<br>
[torch.cuda.FloatTensor of size 1 (GPU 0)]</p>
<p>Variable containing:<br>
4.1735e+12<br>
[torch.cuda.FloatTensor of size 1 (GPU 0)]</p>
<p>Variable containing:<br>
5.0968e+27<br>
[torch.cuda.FloatTensor of size 1 (GPU 0)]<br>
…</p>
<p>Then the error.</p>
<p>Thanks!</p>

szhang: <p>I was looking for how to add a L2 norm of a parameter to the loss function. And I did as suggested above, however,  there comes a bug:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>How should I solve this problem? Thank you very much.</p>

SimonW: <p>You are backproping through the same graph multiple times. Make sure that it is desired because most of the cases you don’t need to. If so specify the flag <code>retain_graph</code>. If not, find out where you are backproping more than once and fix it.</p>

com: <p>Is it possible to replace</p>
<blockquote>
<pre><code>batch_loss = (1/N_train)*(y_pred - batch_ys).pow(2).sum() + l2_reg * reg_lambda
</code></pre>
</blockquote>
<p>with</p>
<blockquote>
<p>batch_loss = MSEloss(y_pred, batch_ys) + l2_reg * reg_lambda</p>
</blockquote>
<p>Thanks!</p>

Jared_77: <p>Won’t this penalize the bias terms as well?</p>

smth: <p>yes, it will penalize the bias terms. if you want it to not penalize the bias terms, you can easily filter them out by using the <code>model.named_parameters()</code> call, and not invoking the regularizer for the <code>bias</code> named terms.</p>

torayeff: <p>I noticed that scaling by 0.5 is also necessary</p>
<pre><code class="lang-auto">reg_loss = None
    for param in model.parameters():
        if reg_loss is None:
            reg_loss = 0.5 * torch.sum(param**2)
        else:
            reg_loss = reg_loss + 0.5 * param.norm(2)**2

    loss += lmbd * reg_loss
</code></pre>
<p>Full code:</p>
<pre><code class="lang-auto">import torch

torch.manual_seed(1)

N, D_in, H, D_out = 10, 5, 5, 1
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

criterion = torch.nn.MSELoss()
lr = 1e-4
weight_decay = 0  # for torch.optim.SGD
lmbd = 0.9  # for custom L2 regularization

optimizer = torch.optim.SGD(model.parameters(), lr=lr, weight_decay=weight_decay)

for t in range(100):
    y_pred = model(x)

    # Compute and print loss.
    loss = criterion(y_pred, y)

    optimizer.zero_grad()

    reg_loss = None
    for param in model.parameters():
        if reg_loss is None:
            reg_loss = 0.5 * torch.sum(param**2)
        else:
            reg_loss = reg_loss + 0.5 * param.norm(2)**2

    loss += lmbd * reg_loss

    loss.backward()

    optimizer.step()

for name, param in model.named_parameters():
    print(name, param)

</code></pre>

brijeshiitg: <p>Can I do weight normailzation for conv layers as follows?:</p>
<p>conv.weight+=lamda*(conv.weight**2)</p>

Esteban_Lanter: <p>Can you provide some intuition as to why we should scale by 0.5? As it is stated now it is creating some confusion.</p>

torayeff: <p>It is not necessary in general, I noticed that this way it will be compatible with Pytorch implementation.</p>