Set constraints on parameters or layers

Shen: <p>Hi, are there any ways in Pytorch to set the range of parameters or values in each layer? For example, is it able to constrain the range of the linear product Y = WX to [-1, 1]? If not, how about limiting the range of the weight?</p>
<p>I noticed in Karas, user can define this by setting <strong><em>constraints</em></strong>. Any equivalence in Pytorch?<br>
Thanks!</p>

Ranahanocka: <p>You can just clip the weights of the parameters after each optimization update.</p>
<pre><code class="lang-auto">class WeightClipper(object):

    def __init__(self, frequency=5):
        self.frequency = frequency

    def __call__(self, module):
        # filter the variables to get the ones you want
        if hasattr(module, 'weight'):
            w = module.weight.data
            w = w.clamp(-1,1)


model = Net()
clipper = WeightClipper()
model.apply(clipper)
</code></pre>
<p>created with inspiration from <a href="https://discuss.pytorch.org/t/restrict-range-of-variable-during-gradient-descent/1933/3">this post</a></p>

eshgovil: <p>I was trying to place explicit weight constraints on my network layers using your suggested approach, I found that nothing was changing until I added this line (correct me if I’m wrong):</p>
<pre><code>def __call__(self, module):
    # filter the variables to get the ones you want
    if hasattr(module, 'weight'):
        w = module.weight.data
        w = w.clamp(-1,1)
        module.weight.data = w
</code></pre>
<p>Just a heads up for anyone trying to implement this in the future!</p>

bf_Lee: <p>Have you solved it? How to place the explicit constraints only for the weights of a specified layer?</p>

bf_Lee: <p>Hi, I have the similar problem. Your code is to add constraints to the whole net, but I want to add the same constraint for the parameters of a specified layer(e.g., the last layer). Can you give me some suggestions?</p>

Shen: <p>Hi, I tried the method you suggested. However, it seems <em><strong>module</strong></em> doesn’t have the attribute <em><strong>weight</strong></em> (<em><strong>module.weight</strong></em> returns a <em><strong>None</strong></em>). Have you encountered this issue? Thx</p>

anantguptadbl: <aside class="quote no-group quote-modified" data-username="eshgovil" data-post="3" data-topic="23620">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eshgovil/48/10065_2.png" class="avatar"> eshgovil:</div>
<blockquote>
<p>if hasattr(module, ‘weight’): w = module.weight.data w = w.clamp(-1,1) module.weight.data = w</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/bf_lee">@bf_Lee</a></p>
<p>I will take an example</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.l1=nn.Linear(100,50)
        self.l2=nn.Linear(50,10)
        self.l3=nn.Linear(10,1)
        self.sig=nn.Sigmoid()
    
    def forward(self,x):
        x=self.l1(x)
        x=self.l2(x)
        x=self.l3(x)
        x=self.sig(x)
        return(x)
</code></pre>
<pre><code class="lang-auto">class weightConstraint(object):
    def __init__(self):
        pass
    
    def __call__(self,module):
        if hasattr(module,'weight'):
            print("Entered")
            w=module.weight.data
            w=w.clamp(0.5,0.7)
            module.weight.data=w
</code></pre>
<pre><code class="lang-auto"># Applying the constraints to only the last layer
constraints=weightConstraint()
model=Model()
model._modules['l3'].apply(constraints)
</code></pre>
<p>Hope this helps</p>

Debajyoti_Majumdar: <p>Is there a way to restrict the values for torch.nn.Parameter?<br>
Since we cannot use apply(fn) for this kind of object.</p>

Aryan_Asadian: <p>You can limit your parameter by feed it as input to a function, e.g., sigmoid.</p>
<pre><code class="lang-auto">my_param= nn.Parameter(torch.empty(1).cuda(), requires_grad=True)
my_param_limited = torch.sigmoid(my_param)
</code></pre>
<p>Note the difference in the names of the parameters, since using a single name, changes the computation graph, and make backpropagation impossible.</p>

ptrblck: <p>The output of <code>torch.sigmoid</code> will create a non-leaf tensor and you will use the <code>nn.Parameter</code> property, so I would recommend to apply the sigmoid on the tensor before wrapping it into the <code>nn.Parameter</code> (unless you want exactly this behavior).</p>
<p>Nit: <code>torch.empty</code> will use uninitialized memory and the tensor might thus contain invalid values such as NaNs/Infs. <code>torch.sigmoid(NaN)</code> would also output a NaN value, so you should initialize it somehow e.g. using <code>rand(n)</code>.</p>

Aryan_Asadian: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your reply. But When I apply sigmoid before wrapping my variable to <code>nn.Parameter</code>, after a few epochs, my parameter violates its range, i.e., [0,1]. How can I handle that?</p>
<pre><code class="lang-auto">my_param= nn.Parameter(torch.sigmoid(torch.rand(1)).cuda(), requires_grad=True)
</code></pre>

googlebot: <p>It is impossible to <strong>declare</strong> a constrained parameter in pytorch. So, in <code>__init__</code> an unconstained parameter is declared, e.g.:<br>
self.my_param = nn.Parameter(torch.zeros(1))</p>
<p>And in forward(), you do the transformation:<br>
my_param_limited = torch.sigmoid(my_param)</p>
<p>this is a dynamically created tensor stored in a local variable</p>

Aryan_Asadian: <p>Thank you for your reply. the transformation that you did in <code>forward()</code> as <a class="mention" href="/u/ptrblck">@ptrblck</a> mentioned, makes it a non-leaf tensor. I am wondering how can we learn a parameter which should be constrained. For example, we have a custom loss function which is the combination of cross-entropy and MSE loss.</p>
<pre><code class="lang-auto">def __init__():
           my_param= nn.Parameter(torch.empty(1).cuda(), requires_grad=True)

def forward():
          gamma = torch.sigmoid(my_param)

         total_loss = gamma * cross_entropy_loss + (1- gamma) * MSE()
         return total_loss
</code></pre>
<p>So in this way, <code>my_param</code> can not be trained meaningfully. right?</p>

googlebot: <p>Built-in optimizers have no idea about constraints, so any other solution would be more cumbersome.</p>
<p>You can improve things a bit:</p>
<pre><code class="lang-auto">def __init__(self):
    self.my_param= nn.Parameter(torch.logit(torch.tensor([0.5])) #inverse of sigmoid
@property
def gamma(self):
    return self.my_param.sigmoid()
</code></pre>
<p>though I don’t like uncached <code>@property</code> there</p>
<p>PS: cuda() can be omitted and done on a module, and requires_grad=True is unnecessary</p>

Y_Y: <p>Could use  module.weight.data.clamp_(-clip,clip)</p>

Jonas_ar: <p>I want to do the same with a specific layer for the MobileNetV3. But I stuggle with finding out the right module name, where to apply my defined constraints.</p>
<p>As an example:<br>
This is the first Inverted Residual block from the MobileNetV3, I only want to set constraints for the Conv2D layer.</p>
<blockquote>
<p>(1): InvertedResidual(<br>
(block): Sequential(<br>
(0): ConvNormActivation(<br>
(0): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)<br>
(1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)<br>
(2): ReLU(inplace=True)<br>
)<br>
(1): ConvNormActivation(<br>
(0): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)</p>
</blockquote>
<p>So basically my “model._modules[‘l3’].apply()” in your example should say something like<br>
model._modules[‘conv2d layer in (block)sequential in  (0)ConvNormActivation’].apply().</p>
<p>I tried to use:</p>
<blockquote>
<p>for name, module in model.named_modules():<br>
print(name)</p>
</blockquote>
<p>for the module name. But the module name “features.1.block.0.0” for the weights I consequently want to constrain (Weightname:"features.1.block.0.0.weight ") throws a “KeyError”.</p>
<blockquote>
<p>model._modules[‘features.1.block.0.0’].apply()</p>
</blockquote>
<p>I don’t want to constrain all weights within the model to the same boundaries, since the value range is quite different in some layers and therefore I would cut of some weights after training if I would not use different boundaries for different layers.</p>
<p>For reference, I am injecting a BitFlip into the weights and therefore want to constrain the weights to the maximum and minimum values after training. So if a bitflip occurs the value does not go beyond the max and min value, and relies on the inherit fault resilience of DNN models for small value changes.</p>
<p>Any help would be greatly appreciated!</p>

Jonas_ar: <p>New update in that regard, accessing the first features “features.0” works with</p>
<blockquote>
<p>print(model._modules[‘features’][0])</p>
</blockquote>
<p>with the output:</p>
<blockquote>
<p>ConvNormActivation(<br>
(0): Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)<br>
(2): Hardswish()<br>
)</p>
</blockquote>
<p>where when I use</p>
<blockquote>
<p>print(model._modules[‘features’][0][0])</p>
</blockquote>
<p>only accesses the Conv2D, which is what I want.</p>
<p>But for the second feature “features.1” the same syntax does not work since this feature is a InvertedResidual and I get the error “‘InvertedResidual’ object” is not subscriptable.</p>
<blockquote>
<p>print(model._modules[‘features’][1])</p>
</blockquote>
<blockquote>
<p>InvertedResidual(<br>
(block): Sequential(<br>
(0): ConvNormActivation(<br>
(0): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)<br>
(1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)<br>
(2): ReLU(inplace=True)<br>
)<br>
(1): ConvNormActivation(<br>
(0): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)<br>
)<br>
)<br>
)</p>
</blockquote>