Convolution operation with L2 normalized weights

numpee: <p>Hi all,<br>
Is there a way normalize (L2) the weights of a convolution kernel before performing the convolution?<br>
For a fully connected layer, I’d go about doing something like:</p>
<pre><code class="lang-auto"># __init__()
weights = nn.Parameter(torch.Tensor(in_size, out_size))

# forward
forward(x):
    w = F.normalize(weights)
    out = torch.mm(x, w)
    return out
</code></pre>
<p>But I’m not exactly sure how this would work on convolutions, even if I’m using 1x1 convolution kernels.</p>
<p>Any ideas?</p>

ptrblck: <p>You could use the same approach, but call <code>F.conv2d</code> instead of <code>torch.mm</code> to perform a convolution.<br>
Also, the <code>input</code> and <code>weight</code> shapes would be different, since <code>F.conv2d</code> expects an input of <code>[batch_size, channels, height, width]</code> and a <code>weight</code> of <code>[out_channels, in_channels, height, width]</code>.</p>

numpee: <p>I had originally implemented it by dividing the output by the norm of the convolution weights. However, this caused some NaN issues since <code>torch.norm()</code> may return a almost-zero value.</p>
<p>I’ve implemented a new method with your suggestion, and it seems to be working well now. Thanks!</p>

Ashima_Garg: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, what if instead of using <code>nn.Parameter</code> in the below code in the <code>init()</code> function, I use <code>nn.utils.weight_norm</code>?<br>
Will that first reparameterize the weights then followed by L2-normalization before each forward pass?</p>
<h1>
<a name="h-__init__-1" class="anchor" href="#h-__init__-1"></a><strong>init</strong>()</h1>
<p>weights = nn.Parameter(torch.Tensor(in_size, out_size))</p>
<h1>
<a name="forward-2" class="anchor" href="#forward-2"></a>forward</h1>
<p>forward(x):<br>
w = F.normalize(weights)<br>
out = torch.mm(x, w)<br>
return out</p>
<p>Looking for help with PyTorch. Thanks!</p>

ptrblck: <p><code>weight_norm</code> is applied on modules not parameters, so it won’t work in your code snippet.<br>
Internally, it will register new parameters as seen <a href="https://github.com/pytorch/pytorch/blob/d9106116aa5e399f7d63feeb7fc77f92a076dd93/torch/nn/utils/weight_norm.py#L26-L54">here</a> so you could try to adapt this code to your use case and manipulate your parameter manually.</p>

Ashima_Garg: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, Thanks for your response. I used the following two implementations. With Implementation 2, I got better results on accuracy.</p>
<p>But I am not clear of how <code>nn.utils.weight_norm</code> will change the performance. The PyTorch documentation reads that <code>nn.utils.weight_norm</code> is just used to decouple the norm vector and the angle. Then why is there difference in the numerical value?</p>
<p>Implementation 1</p>
<pre><code class="lang-auto">def __init__(self):
    super(MyModel, self).__init__()
    self.linear = nn.Linear(2, 2)

def forward(self, x):
    weight = F.normalize(self.linear.weight)
    out = torch.mm(x, weight.t()) + self.linear.bias
    return out
</code></pre>
<p>Implementation 2</p>
<pre><code class="lang-auto">def __init__(self):
    super(MyModel, self).__init__()
    self.linear = nn.utils.weight_norm(nn.Linear(2, 2))

def forward(self, x):
    self.linear.weight = F.normalize(self.linear.weight)
    out = self.linear(x)
    return out
</code></pre>