Kernel Constraint similar to the one implemented in Keras

username: ilyes
cooked: <p>Hi,</p>
<p>I want to add a constraint (max_norm) to my 2D convolutional layer’s weights. For example we can do that easily in Keras using:</p>
<p><code>keras.layers.Conv2D(8, (3, 2), activation='relu', kernel_constraint=max_norm(1.))</code></p>
<p>which makes a convolutional layer with 8 kernels each one has a size of (3, 2).</p>
<p>Is there a way to do the same in Pytorch? i searched in the forum but can’t find something relevant.</p>
<p>Thank you</p>

username: tom
cooked: <p>The usual way to do this is to use the functional interface to redefine the forward</p>
<pre><code class="lang-auto">class ConstrainedConv2d(nn.Conv2d):
    def forward(self, input):
        return F.conv2d(input, self.weight.clamp(min=-1.0, max=1.0), self.bias, self.stride,
                        self.padding, self.dilation, self.groups)
</code></pre>
<p>The important bit here being the <code>.clamp</code>. For spectral norm, there is a utility in <code>torch.nn.utils</code>.<br>
In IPython, you can get the forward of <code>Conv2d</code> using <code>?? nn.Conv2d</code> to ensure feature parity (e.g. I left out the <code>padding_mode</code>).</p>
<p>Best regards</p>
<p>Thomas</p>

username: ilyes
cooked: <p>Thank you! still have a problem with the exact implementation. Let’s say I have a code as follows:</p>
<pre><code class="lang-auto">class Model (nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.H  = 200
        self.W = 200
        self.conv = nn.Conv2d(1, 8, (1, 101), padding = (0, 50), bias = False)
        self.fc1 = nn.Linear(self.H * self.W , 1)        
    def forward(self, x):
      
        x = self.conv(x)
        x = F.relu(x)
        x = x.view(-1, self.H * self.W)
        x = torch.sigmoid(self.fc1(x))
        return x
</code></pre>
<p>Now Do I have to implement the ConstrainedConv2d like you did</p>
<pre><code class="lang-auto">class ConstrainedConv2d(nn.Conv2d):
    def forward(self, input):
        return F.conv2d(input, self.weight.clamp(min=-1.0, max=1.0), self.bias, self.stride,
                        self.padding, self.dilation, self.groups)
</code></pre>
<p>and replace my conv2D defined in the <strong>init</strong> by ConstrainedConv2D ? and if so how do I pass the the desired parameters to ConstrainedConv2D (input channel, outputChannel, kernel size …)</p>

username: tom
cooked: <p><code>ConstrainedConv2d</code> inherits it’s <code>__init__</code> from <code>nn.Conv2d</code>, so you can use it as a drop-in replacement.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ilyes
cooked: <p>Got it! Thank you Thomas</p>

username: lkc
cooked: <p>Wouldn’t this constrain the weights to be between <code>-1.0</code> and <code>1.0</code>? In Keras the <code>max_norm</code> constraint is on the norm itself and weights are adjusted proportionally. Something along the following lines</p>
<pre><code class="lang-auto">    def _max_norm(self, w):
        norm = w.norm(2, dim=0, keepdim=True)
        desired = torch.clamp(norm, 0, self._max_norm_val)
        return w * (desired / (self._eps + norm))
</code></pre>
<p>This is taken from this <a href="https://github.com/kevinzakka/pytorch-goodies#max-norm-constraint" rel="nofollow noopener">post</a>.</p>
<h3>Max Norm Constraint</h3>
<p><em>If a hidden unit’s weight vector’s L2 norm  <code>L</code>  ever gets bigger than a certain max value  <code>c</code> , multiply the weight vector by  <code>c/L</code> . Enforce it immediately after each weight vector update or after every  <code>X</code>  gradient update.</em></p>
<p><em>This constraint is another form of regularization. While L2 penalizes high weights using the loss function, “max norm” acts directly on the weights. L2 exerts a constant pressure to move the weights near zero which could throw away useful information when the loss function doesn’t provide incentive for the weights to remain far from zero. On the other hand, “max norm” never drives the weights to near zero. As long as the norm is less than the constraint value, the constraint has no effect.</em></p>
<p>Since the OP accepted the given answer, the weight constraint probably solved their problem. But Just for future references, I think <code>max_norm</code> is different than clamping the weight directly. Let me know if I missed something here.</p>