Update Weights With Learnable Parameters

username: Lillian_Li
cooked: <p>Hi all,</p>
<p>I am trying to implement a convolutional neural network where one of the layers’ weights are constrained by a couple of parameters, which are optimized during backpropagation. Another way to put it: the weights themselves are not directly updated, but the parameters that determine the weight matrices should be. Here is a simplified version of the code:</p>
<pre><code class="lang-auto">import torch.nn.functional as F
import torch
from torch import nn


class GaborConv2d(nn.Module):
    def __init__(
        self,
        in_channels,
        out_channels,
        kernel_size,
        stride=1,
    ):
        super().__init__()

        self.is_calculated = False

        self.conv_layer = nn.Conv2d(
            in_channels,
            out_channels,
            kernel_size,
            stride=stride,
        )

        self.kernel_size = self.conv_layer.kernel_size
        self.test = nn.Parameter((torch.Tensor([2.0])), requires_grad=True)
        self.register_parameter("test", self.test)

    def forward(self, input_tensor):
        if self.training:
            self.calculate_weights()
            self.is_calculated = False
        if not self.training:
            if not self.is_calculated:
                self.calculate_weights()
                self.is_calculated = True

        return self.conv_layer(input_tensor)

    def calculate_weights(self):
        self.conv_layer.weight.data += self.test


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = GaborConv2d(1, 1, 3)
        self.fc = nn.Linear(9, 3)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)
        x = F.relu(x)
        return x

if __name__ == '__main__':
    model = Net()
    model.optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9) # only the weights and biases of the fully connected layer are learned
    model.loss = nn.CrossEntropyLoss()
    for i in range(100):
        random_data = torch.rand((1, 1, 5, 5))
        model.optimizer.zero_grad()
        output = model(random_data)
        l = model.loss(output, torch.tensor([0]))
        l.backward(retain_graph=True)
        model.optimizer.step()
</code></pre>
<p>The “test” variable is the parameter that controls the weights. In the optimizer, I do see test listed in the params group. However, its grad is None, and it is not updated during backpropagation. Any input would be appreciated! Thanks!</p>

username: AlphaBetaGamma96
cooked: <p>Hi,</p>
<p>I don’t think you need to re-register <code>test</code> after declaring it as an <code>nn.Parameter</code> object. So, delete the line,</p>
<pre><code class="lang-auto">self.register_parameter("test", self.test)
</code></pre>
<p>Secondly, within your <code>calculate_weights</code> method there are 2 problems. Firstly, don’t use the <code>.data</code> attribute as it’s deprecated and can lead to unexpected behaviour. And, you’re updating <code>conv_weight.weight</code> via an inplace operator (<code>+=</code>). So try,</p>
<pre><code class="lang-auto">self.conv_layer.weight = self.conv_layer.weight + self.test 
</code></pre>

username: Lillian_Li
cooked: <p>Thank you for your input!</p>
<p><code>self.conv_layer.weight = self.conv_layer.weight + self.test</code> gave me this error:<br>
TypeError: cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected), so I tried<br>
<code>self.conv_layer.weight = nn.Parameter(self.conv_layer.weight + self.test)</code> and<br>
<code> self.conv_layer.weight = nn.Parameter(self.conv_layer.weight + self.test, requires_grad=False)</code>. In both cases, the grad’s and grad_fn’s for “test” are still None. I also added a line before backward() to retain test’s gradient: <code>model.conv1.test.retain_grad()</code>. The issues persist.</p>

username: KFrank
cooked: <p>Hi Lillian!</p>
<aside class="quote no-group" data-username="Lillian_Li" data-post="1" data-topic="146982" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lillian_li/48/31552_2.png" class="avatar"> Lillian_Li:</div>
<blockquote>
<p>the weights themselves are not directly updated, but the parameters that determine the weight matrices should be.</p>
</blockquote>
</aside>
<p>Could pytorch’s <a href="https://pytorch.org/tutorials/intermediate/parametrizations.html" rel="noopener nofollow ugc">parametrizations</a> facility be applied to your use case?</p>
<p>Best.</p>
<p>K. Frank</p>