Why doesnt these variables get updated?

username: Shisho_Sama
cooked: <p>Hello all, I created a simple network where a convolutional layers weight matrix is altered by a custom function.<br>
I came up with this  :</p>
<pre><code class="lang-auto">
class snet(nn.Module):
    def __init__(self, num_classes=3):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 2, 1, 0)
        shape = self.conv1.weight.shape
        self.var1 = nn.Parameter(torch.ones(shape))
        self.var2 = nn.Parameter(torch.ones(shape))

        self.conv2 = nn.Conv2d(6, 6, 5, 1, 0)
        self.fc = nn.Linear(6*11*11, num_classes)

    def some_method(self):
        """
            Suppose this is a custom method, tasked with
            producing values for each entry in the weight matrix
            of a convolutional layer. for simplicity we used 
            addition here
        """
        return self.var1 + self.var2

    def forward(self, input):
        self.conv1.weight = nn.Parameter(self.some_method())
        output = self.conv1(input)
       # or using the functional api and using the weight matrix directly is no different
      # output = F.conv2d(input, self.some_method())
        output = self.conv2(output)
        output = output.view(input.size(0), -1)
        output = self.fc(output)
        return output


n = snet(num_classes=3)
fake_dataset = torchvision.datasets.FakeData(100,
                                             image_size=(3, 16, 16),
                                             num_classes=3,
                                             transform=transforms.ToTensor())
fake_dataloader = torch.utils.data.DataLoader(fake_dataset,
                                              batch_size=20)
criterion = nn.CrossEntropyLoss()


opt = torch.optim.Adam(n.parameters(), lr=0.01)
for imgs, labels in fake_dataloader:
    p = n(imgs)
    loss = criterion(p, labels)
    opt.zero_grad()
    loss.backward()
    opt.step()
    print(loss.item())
</code></pre>
<p>Apparently this is wrong as nothing happens! the parameters are added to the module and they show up in the parameters list. however, the gradient is always zero!<br>
I noticed the <code>grad_fn</code> property for both variables/parameters are None! where as it must have been the addition right?<br>
Based on the <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="nofollow noopener">autograd tutorial</a>, when one variable in an operation has <code>requires_grad = True</code>, the output also will have its<code>requires_grad = True</code> and thus  the gradient should flow back to those with<code>requires_grad</code> set to True.!<br>
Since the <code>nn.Parameter()</code> sets this property implicitly to True, this should work yet it does not!<br>
whats wrong here? what am I missing here?<br>
Any help is greatly appreciated.</p>

username: Krish
cooked: <p><strong>var1</strong> and <strong>var2</strong> have no grad_fn because you are not performing any operation that changes their values.</p>
<p>I believe that the conv1 weight doesn’t have a grad_fn because <em>some_function()</em> <strong>(or any python function)</strong> returns the <strong>value rather than the reference</strong>. So although the value gets updated, the lack of reference means Pytorch can’t know what changed the value.</p>

username: Shisho_Sama
cooked: <p>Thanks a lot for your response, but I thought since they are being used in an operation, so they must have a gradient in any case right?<br>
If not, how am I supposed to introduce a dependent variables?<br>
For example in my case, what should I be doing ? I’m really lost here!</p>

username: Krish
cooked: <p>They do take part in an operation but Pytorch doesn’t know what operation it is, because it just gets the value from the function.<br>
What you can try in this case is to perform the computation of <em>some_method</em> inside the <em><strong>init</strong></em> or <em>forward</em> method.<br>
I will try this out and let you know if that worked.</p>

username: Shisho_Sama
cooked: <p>Thanks a lot , its greatly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Krish
cooked: <p><code>nn.Parameter</code> doesn’t seem to transfer any kind of grad information. One workaround that I could get to work was to use the <code>torch.nn.functional</code> API and passing the kernel through there.</p>
<pre><code class="lang-auto">def some_function(self):
    self.kernel = self.var1 + self.var2

def forward(self, x):
    self.some_function()
    output  = F.conv2d(x, self.kernel, padding = 0, stride = 1)
</code></pre>
<p>Here grad_fn works as usual.<br>
You do have to figure out the kernel size by yourself though.</p>

username: Shisho_Sama
cooked: <p>Thanks but this only acts as a one time initialization , its as if I write :</p>
<pre><code class="lang-auto">def __init__(self):
    super().__init__()
    self.conv1 = nn.Conv2d(3, 6, 3) 
    shape = self.conv1.weight.shape
    self.conv1.weight = nn.Parameter(torch.rand(shape)) + nn.Parameter(torch.rand(shape))
</code></pre>
<p>and use self.conv in the forward pass!<br>
The difference is that <strong>here</strong> <strong>the values</strong> of the <strong>self.kernel</strong> will <strong>get optimized</strong>, but what I’m after is that I want the values of <strong>self.var</strong>1 and <strong>self.var2</strong> to get <strong>changed</strong> so that <strong>their interactions result in a  set of filters/kernels</strong> ( the resulting weight matrix) that <strong>minimize</strong> the loss.<br>
In other words, in your self.kernel case, there are n entries that get learned, but in my case, there are 2x more variables that are used to create  a kernel.<br>
The method call is necessary since, each time its called in the forward pass, it runs the underlying operation that creates a new set of filters, then these filters are used, some losses will be produced and ultimately a set of gradients will be created, I want these gradients to update <strong>not the filters values</strong> <strong>themselves</strong>, <strong>but</strong> the <strong>self.var1</strong> and <strong>self.var2</strong> values that <strong>create the filters values</strong></p>

username: Krish
cooked: <p>Had a missing line of code above. This</p>
<pre><code class="lang-auto">def forward(self, x):
    self.some_function()
...
</code></pre>
<p>The values of kernel <strong>won’t be optimised</strong> directly, instead the <code>optimizer</code> will optimise values of <code>var1</code> and <code>var2</code> only as they are the parameters.<br>
You can check that out by tracing the grad_fn backwards <em>(tedious process, although).</em></p>

username: Krish
cooked: <aside class="quote no-group" data-username="Shisho_Sama" data-post="7" data-topic="51187">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shisho_sama/48/6926_2.png" class="avatar"> Shisho_Sama:</div>
<blockquote>
<p>in your self.kernel case, there are n entries that get learned, but in my case, there are 2x more variables that are used to create a kernel.</p>
</blockquote>
</aside>
<p>I don’t clearly understand what you are saying. Can you explain it?</p>

username: Shisho_Sama
cooked: <p>Thanks, That actually worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Thanks a gazillion times sir <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Also I noticed there is no need to use a class attribute for this and simply returning the result like this would also work :</p>
<pre><code class="lang-auto">    def some_method(self):
        """
            Suppose this is a custom method, tasked with
            producing values for each entry in the weight matrix
            of a convolutional layer. for simplicity we used 
            addition here
        """
        result = self.var1 + self.var2
        return result

    def forward(self, inputs):
        weight = self.some_method()
        output = F.conv2d(inputs, weight )

        print('var1 grad',self.var1.grad)
        print('weight grad',weight.grad)
        
        return output
</code></pre>
<p>When running this the <code>self.var1.grad</code> actually has values!<br>
The catch here seems to be using the functional form for  this to work only! (Why is that? I’d really would like to know why the first wouldn’t work?)</p>
<pre><code class="lang-auto">var1 grad None
weight grad None

var1 grad tensor([[[[ 0.0295,  0.0308],
          [ 0.0284,  0.0288]],

         [[ 0.0157,  0.0115],
          [ 0.0275,  0.0370]],

         [[ 0.0231,  0.0077],
          [ 0.0269,  0.0438]]],


        [[[ 0.0812,  0.0750],
          [ 0.0818,  0.0876]],

         [[ 0.0944,  0.0916],
          [ 0.0720,  0.0938]],

         [[ 0.0702,  0.0918],
          [ 0.0966,  0.0842]]],


        [[[-0.0011,  0.0065],
          [ 0.0024,  0.0097]],

         [[ 0.0100, -0.0002],
          [ 0.0133,  0.0061]],

         [[ 0.0084,  0.0118],
          [ 0.0030,  0.0073]]],


        [[[ 0.0204,  0.0375],
          [ 0.0376,  0.0324]],

         [[ 0.0149,  0.0122],
          [ 0.0230,  0.0258]],

         [[ 0.0214,  0.0505],
          [ 0.0280,  0.0071]]],


        [[[-0.0798, -0.0690],
          [-0.0813, -0.0650]],

         [[-0.0819, -0.0741],
          [-0.0805, -0.0698]],

         [[-0.0827, -0.0689],
          [-0.0654, -0.0692]]],


        [[[ 0.0201,  0.0045],
          [ 0.0195,  0.0160]],

         [[ 0.0128,  0.0298],
          [ 0.0173,  0.0195]],

         [[ 0.0271,  0.0173],
          [ 0.0270,  0.0049]]]])
weight grad None
var1 grad tensor([[[[ 0.0686,  0.0713],
          [ 0.0695,  0.0971]],

         [[ 0.0705,  0.0810],
          [ 0.0792,  0.0667]],

         [[ 0.0934,  0.0621],
          [ 0.0784,  0.0773]]],


        [[[ 0.4964,  0.4827],
          [ 0.4911,  0.4820]],

         [[ 0.5122,  0.5005],
          [ 0.5119,  0.4927]],

         [[ 0.4984,  0.5238],
          [ 0.4832,  0.5231]]],


        [[[-0.2394, -0.2491],
          [-0.2389, -0.2550]],

         [[-0.2355, -0.2570],
          [-0.2466, -0.2347]],

         [[-0.2434, -0.2351],
          [-0.2315, -0.2475]]],


        [[[ 0.1815,  0.1805],
          [ 0.2223,  0.2162]],

         [[ 0.2135,  0.2039],
          [ 0.1949,  0.2067]],

         [[ 0.2072,  0.2130],
          [ 0.2181,  0.1701]]],


        [[[-0.1945, -0.1760],
          [-0.2120, -0.2003]],

         [[-0.1972, -0.1877],
          [-0.2123, -0.1866]],

         [[-0.1686, -0.2197],
          [-0.1895, -0.2136]]],


        [[[ 0.0689,  0.0675],
          [ 0.0926,  0.0759]],

         [[ 0.0856,  0.0696],
          [ 0.0806,  0.0503]],

         [[ 0.0815,  0.0521],
          [ 0.0534,  0.0843]]]])
...
</code></pre>
<p>The weird thing is that although we have gradients, yet <code>grad_fn</code> is <code>None</code> for both <code>self.var1</code> and <code>self.var2</code></p>
<p>and concerning :</p>
<blockquote>
<p>I don’t clearly understand what you are saying. Can you explain it?</p>
</blockquote>
<p>for the sake of simplicity, lets assume we a kernel of size 2x2 .<br>
this simply means we have 4 variables (separate values) that together form a kernel we know and use.<br>
in the forward pass, what happens is that these 4 variables are used in different operations (multiplication, sum , etc) and then in the backward pass, each of them get a gradient with respect to how they contributed to the end loss.<br>
Now, when I said, we have two variables for each entry, I simply meant suppose we don’t have this kernel of 2x2 (4 variables), instead suppose we have an empty 2x2 grid and we need to fill its entries.<br>
in the first case, a variable is allocated to each entry. thus a grid of 2x2 has 4 variables that each represent a single entry! (the entry they occupy !). now suppose we want to fill this grid not with one variable but 2. meaning 2 variables with be used to create a value for a single entry in our grid.<br>
2 variables per each entry. in other words, suppose we have a grid1 of 2x2 and a grid2 of 2x2 and we use these two grids to (for example add them together element wise), get the values for our empty grid i.e. kernel values. here this very kernel, is not treated as our variables, it simply a matrix of values, the real variables are gridl1 and grid2 respectively. I hope this makes it a bit more clear! Although I got what I was after thanks to you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>By the way, What puzzles me now is that we have gradients for our <code>var</code>s, yet their <code>grad_fn</code> attribute is None!!?<br>
Why is that?</p>

username: Krish
cooked: <aside class="quote no-group" data-username="Shisho_Sama" data-post="10" data-topic="51187">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shisho_sama/48/6926_2.png" class="avatar"> Shisho_Sama:</div>
<blockquote>
<p>we have gradients for our <code>var</code> s, yet their <code>grad_fn</code> attribute is None!!?<br>
Why is that?</p>
</blockquote>
</aside>
<p>As I said earlier, a variable has a <code>grad_fn</code> only when an operation changes its value (except when the code is wrapped in <code>with torch.no_grad():</code>). Here no operation changes the value of var1 or var2.</p>
<p>Something you can try to understand is to run this code:</p>
<pre><code class="lang-auto">x = torch.ones(2).requires_grad_(True)
y = x + 2
print (y.grad_fn, x.grad_fn)
</code></pre>
<p>The output should be: <code>AddBackward    None</code></p>
<p>The gradients store how much the variable affects the loss. You can see it for yourself by running along with the above code:</p>
<pre><code class="lang-auto">y.sum()
y.backward()
print(x.grad)
</code></pre>

username: Shisho_Sama
cooked: <p>Thanks a lot again got it this  time <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Now my only question that still persists, is why it doesn’t work on setting the conv layer weights directly using the non-functional form?</p>

username: Krish
cooked: <p>That I would have to find out as well <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>