Model parameters are not being updated?

kazem: <p>I made a simple example of a cnn layer where convolutional weights are defined as linear combination of predefined filters. The goal is to train the coefficients of linear combination while keeping predefined filters fixed. This can be easily achieved in tensorflow using tf.nn.conv2d. However here model.weight is not changing and model.weight.grad=None?<br>
Can anyone please have a look? Thanks.</p>
<pre><code class="lang-python">from __future__ import print_function
import torch.nn.functional as F
# from torch.autograd import Variable
# import copy
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms


class ConvNet(nn.Module):
    def __init__(self, device='cuda'):

        super(ConvNet, self).__init__()
        self.device = device
        self.filter = torch.Tensor(
            [[[0.06,  0,      0],
              [0.1,   0,    0.2],
              [0.06,  0.1,   0]],
             [[0.1,   0,      0],
             [0.2,    0,      0],
             [0.1,    0,      0]]]).to(self.device)
        # print(self.filter.shape)
        self.weight = nn.Parameter(torch.Tensor(1, 1, 2).to(self.device))
        self.bias = nn.Parameter(torch.Tensor(1)).to(self.device)
        nn.init.xavier_uniform_(self.weight)
        self.bias.data.uniform_(-1, 1)
        self.kernel = nn.Parameter(
            torch.einsum("ijk, klm -&gt; ijlm", 
            self.weight, self.filter).to(self.device), 
            requires_grad=False)
        # print(self.kernel.shape)
        self.bn1 = nn.BatchNorm2d(1)
        self.fc = nn.Linear(1*13*13, 10)

    def forward(self, x):
        out = F.conv2d(input=x, weight=self.kernel, bias=self.bias)
        out = self.bn1(out)
        out = nn.ReLU()(out)
        out = nn.MaxPool2d(kernel_size=2, stride=2)(out)
        # print("out.shape: ", out.shape)
        out = out.reshape(out.size(0), -1)
        # print("out.shape: ", out.shape)
        out = self.fc(out)
        return out


def main(argv=None):
    # Device configuration
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    # Hyper parameters
    num_epochs = 1
    batch_size = 1024
    learning_rate = 0.001
    log_interval = 10

    # MNIST dataset
    train_dataset = datasets.MNIST(
    root='../../data/', 
    train=True, 
    transform=transforms.ToTensor(), 
    download=True)
    train_loader = torch.utils.data.DataLoader(
    dataset=train_dataset, 
    batch_size=batch_size, 
    shuffle=True)

    model = ConvNet(device=device)
    model.to(device)
    for name, param in model.named_parameters():
        print(name, '\t\t', param.shape)
    optimizer = optim.SGD(model.parameters(), lr=learning_rate)
    loss_fn = nn.CrossEntropyLoss()
    for epoch in range(1, num_epochs + 1):
        model.train()
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = data.to(device), target.to(device)
            optimizer.zero_grad()
            output = model(data)
            loss = loss_fn(output, target)
            # print(model.weight)
            # print(model.kernel)
            # print(model.weight.grad)
            a = list(model.parameters())[0].clone()
            loss.backward()
            optimizer.step()
            b = list(model.parameters())[0].clone()
            print(torch.equal(a.data, b.data))
            if batch_idx % log_interval == 0:
                _, predicted = torch.max(output.data, 1)
                total = target.size(0)
                correct = (predicted == target).sum().item()
                print('batch Accuracy: {} %'.format(100 * correct / total))


if __name__ == '__main__':
    main()


</code></pre>

ptrblck: <p>I tried to play around with your code to see, what might be wrong, but get an error running it:</p>
<pre><code class="lang-python">RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>for PyTorch <code>1.0.0.dev20190207</code>.<br>
Is the code running fine on your machine?</p>

kazem: <p>Thank you so much for your reply. Yes it runs fine on my ubuntu 16.04 with pytorch 1.0.0 cuda version, python 3.5. Also there is really no need to use the gpu (the device). Since even on cpu it was having the same issue.</p>

ptrblck: <p>Your code is working on the CPU, if you specify <code>requires_grad=True</code> for <code>self.kernel</code> or just remove the <code>requires_grad</code> argument, since the default is set to <code>True</code> for <code>nn.Parameter</code>.<br>
It’s still strange, that your code threw an error on my machine using the GPU.</p>

kazem: <p>I understand but the problem is when i set <code>requires_grad=True</code> for <code>self.kernel</code> the algorithm just initializes the weights based on the linear combination. Afterwards, all the coordinate of <code>self.kernel</code> start changing. That is not the behavior i want. I want only the coefficients to be trained. So  <code>self.filter</code> has to stay fixed while <code>self.weight</code> is changing. Therefore, it has to be the case that for <code>self.kernel</code> <code>requires_grad=False</code>.<br>
An easy way to observe the right behavior during training is the following: coordinates <code>0x1</code>, <code>0x2</code>, <code>1x1</code>, and <code>2x2</code> of the <code>self.kernel</code> has to stay zero during the entire training process. Hope that helps.</p>
<p>Now, my question is, is it an internal limitation of pytorch that by design you cannot define the <code>weight</code> of <code>nn.conv2d</code> or <code>F.conv2d</code> to be non-trainable? Do i need to change the <code>F.conv2d</code> function internally in pytorch?<br>
Thank you for your help.</p>

kazem: <p>I’ve had no problem implementing this idea in tensorflow. The reason i want to move to pytorch is:<br>
It’s awesome dataset api and easy multi-gpu training.<br>
I can also share a tensorflow example if that helps you understand the algorithm better.<br>
I would be most grateful if you could help me solve this issue.</p>

InnovArul: <aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/InnovArul/ca0db578769186bbaf094c5f8416a8bc" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/InnovArul/ca0db578769186bbaf094c5f8416a8bc" target="_blank" rel="nofollow noopener">https://gist.github.com/InnovArul/ca0db578769186bbaf094c5f8416a8bc</a></h4>
<h5>filter_learning.py</h5>
<pre><code class="Python">from __future__ import print_function
import torch.nn.functional as F
# from torch.autograd import Variable
# import copy
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
</code></pre>
This file has been truncated. <a href="https://gist.github.com/InnovArul/ca0db578769186bbaf094c5f8416a8bc" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<ol>
<li>
<p>Remember that pytorch is based on dynamic computation graphs, where as tensorflow is based on static computation graphs. Hence, you need to calculate <code>filter</code> in <code>forward()</code> rather than <code>__init__()</code>.</p>
</li>
<li>
<p>You do not need to move individual parameters to cuda. Rather try to move whole model to cuda.</p>
</li>
</ol>

kazem: <p>Hi InnovArul! Thank you for your response. Well, it looks like your example is working:</p>
<p><code>self.weight</code> is changing.<br>
<code>self.weight.grad</code> is not <code>None</code> and changing as well.<br>
<code>self.kernel</code> itself is changing as expected as well upon closer inspection.</p>
<p>Thank you so much.</p>

kazem: <p>InnovArul and ptrblck.<br>
Thank you so much for your help.<br>
I just wanted to ask why should I use this line <code>self.register_buffer("filter", filter)</code>? and not something like <code>self.filter = nn.Parameter(filter, requires_grad=False)</code>? Why is the latter not correct?<br>
since I looked at a bunch of posts to learn about <code>register_buffer</code> usage but I am new to pytorch and still want to understand deeper:<br>
<a href="https://discuss.pytorch.org/t/solved-register-parameter-vs-register-buffer-vs-nn-parameter/31953">https://discuss.pytorch.org/t/solved-register-parameter-vs-register-buffer-vs-nn-parameter/31953</a><br>
<a href="https://discuss.pytorch.org/t/what-is-the-difference-between-register-buffer-and-register-parameter-of-nn-module/32723/2">https://discuss.pytorch.org/t/what-is-the-difference-between-register-buffer-and-register-parameter-of-nn-module/32723/2</a><br>
<a href="https://discuss.pytorch.org/t/use-and-abuse-of-register-buffer/4128">https://discuss.pytorch.org/t/use-and-abuse-of-register-buffer/4128</a></p>

YodaEmbedding: <p><code>requires_grad=False</code> by default, and not saved to checkpoint:</p>
<pre><code class="lang-python">self.filter = torch.tensor(...)
</code></pre>
<p><code>requires_grad=True</code> by default, and saved to checkpoint, and shows up in <code>module.parameters()</code>:</p>
<pre><code class="lang-python">self.filter = nn.Parameter(torch.tensor(...))
</code></pre>
<p><code>requires_grad=False</code> by default, and saved to checkpoint:</p>
<pre><code class="lang-python">self.filter = self.register_buffer("filter", torch.tensor(...))
</code></pre>
<hr>
<p>If you’re dealing with a constant tensor, you don’t want it showing up in <code>model.parameters()</code> since that makes the following include the constant tensor in the optimizer:</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
</code></pre>
<p>…this is perhaps not so problematic if you manually also set <code>requires_grad=False</code>, since parameters that have had <code>requires_grad=False</code> set since the beginning of time will be skipped. But it’s nicer to keep <code>buffers</code> separated from <code>parameters</code>.</p>