How to implement a customize weight

111179: <p>Hi, I would like to implement tanh(weight) instead of weight, for example:<br>
I am not sure if there is a problem with my code，so thank you so much for your help.</p>
<pre><code class="lang-auto">class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1, 6, 3, 1, 2,bias=False),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(6, 16, 5,bias=False),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        self.fc1 = nn.Linear(16 * 5 * 5, 120,bias=False)
        self.fc2 = nn.Linear(120, 10,bias=False)
        self.conv1.weight =torch.tanh(self.conv1.weight)
        self.conv2.weight =torch.tanh(self.conv2.weight)
        self.fc1.weight =torch.tanh(self.fc1.weight)
        self.fc2.weight =torch.tanh(self.fc2.weight)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size()[0], -1)
        x = self.fc1(x)
        x= F.relu(x)
        x = self.fc2(x)
        return x
</code></pre>

Eta_C: <p>Do you mean <code>weight</code> is learnable and calculate <code>y = tanh(weight) * x</code> when learning?</p>
<pre><code class="lang-python">class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1, 6, 3, 1, 2,bias=False),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(6, 16, 5,bias=False),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        self.fc1 = nn.Linear(16 * 5 * 5, 120,bias=False)
        self.fc2 = nn.Linear(120, 10,bias=False)

    def forward(self, x):
        with torch.no_grad():
            self.conv1[0].weight[:] = torch.tanh(self.conv1[0].weight[:])
            self.conv2[0].weight[:] = torch.tanh(self.conv2[0].weight[:])
            self.fc1.weight[:] = torch.tanh(self.fc1.weight[:])
            self.fc2.weight[:] = torch.tanh(self.fc2.weight[:])
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size()[0], -1)
        x = self.fc1(x)
        x= F.relu(x)
        x = self.fc2(x)
        return x
</code></pre>
<p>I am not sure <code>with torch.no_grad</code> is needed or not.</p>

111179: <p>Thank you for your kind reply. Yes, weight is a learnable parameter, and use tanh(weight) to limit the weight range.<br>
Apart from <code>with torch.no_grad</code>, what is the difference between putting <code>self.conv1[0].weight[:] = torch.tanh(self.conv1[0].weight[:])</code> before <code>def forward(self, x):</code> and inside <code>def forward(self, x):</code>.</p>
<p>Thank you again for your time.<br>
Best regards.</p>

Eta_C: <p><code>__init__</code> only work when you init your network.<br>
If you want to limit it, inside <code>forward</code><br>
.</p>

111179: <p>oh! Thank you so much!</p>

Shisho_Sama: <p>If all you want is to limit the values to be in a range, why not using the <a href="https://pytorch.org/docs/stable/generated/torch.clamp.html" rel="noopener nofollow ugc">torch.clamp</a>?<br>
I just want to make sure you use the right tools here, <code>tanh</code> is an activation function, and it transforms your input to be in that range, depending on the task it might not give you what you desire.<br>
So if you know what you are doing then fine, in case you didn’t know, check <code>torch.clamp</code> out, maybe this is what you were looking for?</p>

Eta_C: <p>Yes, if <strong>ONLY CLIP</strong>, check here</p><aside class="quote quote-modified" data-post="3" data-topic="1933">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/8baadc/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/restrict-range-of-variable-during-gradient-descent/1933/3">Restrict range of variable during gradient descent</a> 
  </div>
  <blockquote>
    For your example (constraining variables to be between 0 and 1), there’s no difference between what you’re suggesting – clipping the gradient update – versus letting that gradient update take place in full and then clipping the weights afterwards. Clipping the weights, however, is much easier than modifying the optimizer. 
Here’s a simple example of a UnitNorm clipper: 
class UnitNormClipper(object):

    def __init__(self, frequency=5):
        self.frequency = frequency

    def __call__(self,…
  </blockquote>
</aside>


111179: <p>Hi, thank you for your suggestion. I have tried<code> torch.clamp</code>, besides, I also want to use <code>tanh (weight)</code> for <code>weight regularizer</code> for loss function, for example:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d179ad092a6c05819baf58b5ce762c76749119a.png" alt="image" data-base62-sha1="49mq0QJkLCFEkA0PZ46lmzPNV1M" width="680" height="99"></p>
<p>Thank you so much for all.</p>