Adding L1/L2 regularization in a Convolutional Networks in PyTorch?

hugozanini: <p>I am new to pytorch and would like to add an L1 regularization after a layer of a convolutional network. However, I do not know how to do that.</p>
<p>The architecture of my network is defined as follows:</p>
<pre><code>downconv = nn.Conv2d(outer_nc, inner_nc, kernel_size=4,
                         stride=2, padding=1, bias=use_bias)
downrelu = nn.LeakyReLU(0.2, True)
downnorm = norm_layer(inner_nc)
uprelu = nn.ReLU(True)
upnorm = norm_layer(outer_nc)
upconv = nn.ConvTranspose2d(inner_nc, outer_nc,
                                    kernel_size=4, stride=2,
                                    padding=1, bias=use_bias)
down = [downrelu, downconv]
up = [uprelu, upconv, upnorm]
model = down + up
</code></pre>
<p>And I used the following code to implement the regularizer:</p>
<pre><code>xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
target = Variable(torch.from_numpy(np.zeros((3,3))))
l1_crit = nn.L1Loss()
l1_crit(xx, target)

l1_crit = nn.L1Loss(size_average=False)
reg_loss = 0
for param in model:
    print "PARAM: ", param
 reg_loss += l1_crit(param)

model = down + [submodule] + up + [nn.Dropout(0.5)]
</code></pre>
<p>But I get the following error:<br>
TypeError: forward() takes exactly 3 arguments (2 given)</p>
<p>I think the problem is in my code that implements the L1 regularization. Can someone help me?</p>

Qing_En: <p>You just input param and size_average in reg_loss+=l1_crit(param) without target.</p>

ptrblck: <p>You could implement L! regularization using something like <a href="https://discuss.pytorch.org/t/how-does-one-implement-weight-regularization-l1-or-l2-manually-without-optimum/7951/2?u=ptrblck">example of L2 regularization</a>.<br>
For L1 regularization, you should change <code>W.norm(2)</code> to <code>W.norm(p=1)</code>.</p>

blade: <p>Since the L1 regularizer is not differentiable everywhere, what does PyTorch do when it encounters differentiating this functions? A simple example shows PyTorch returns zero.</p>
<pre><code class="lang-auto">import torch

x = torch.linspace(-1.0, 1.0, 5, requires_grad=True)
y = torch.abs(x)
y[2].backward()

print(x.grad)
</code></pre>
<blockquote>
<p><code>tensor([-0., -0., 0., 0., 0.])</code></p>
</blockquote>
<p>Why is this the case?</p>

ptrblck: <p>I think a zero gradient is expected to be returned for zero inputs and would fit the idea of a regularizer since no penalty should be added to a value which is already at zero. All other values will get valid gradients:</p>
<pre><code class="lang-python">x = torch.linspace(-1.0, 1.0, 5, requires_grad=True)
y = torch.norm(x, 1)
y.backward()

print(x)
# tensor([-1.0000, -0.5000,  0.0000,  0.5000,  1.0000], requires_grad=True)
print(x.grad)
# tensor([-1., -1.,  0.,  1.,  1.])
</code></pre>

blade: <p>That makes sense. Is PyTorch using a specific algorithm to compute the gradient at this non-differentiable point? Is there an academic reference that discusses this behaviour that I can read?</p>

ptrblck: <p>The derivative is implemented <a href="https://github.com/pytorch/pytorch/blob/d41b5d7c145f3e09c7223c2b707933266241ec9b/tools/autograd/derivatives.yaml#L205">here</a> as:</p>
<pre><code class="lang-python">self: grad * self.sgn()
</code></pre>
<p>and with:</p>
<pre><code class="lang-python">torch.tensor(0.).sgn()
# tensor(0.)
torch.tensor(1.).sgn()
# tensor(1.)
torch.tensor(-1.).sgn()
# tensor(-1.)
</code></pre>
<p>would then return a zero.<br>
I don’t know how it was defined, but based on e.g. <a href="https://discuss.pytorch.org/t/how-does-autograd-deal-with-non-differentiable-opponents-such-as-abs-and-max/34538/2">this answer</a> the zero output might be “convenient” for users.</p>