Module.zero_grad() with requires_grad=False for some Parameter?

FrancoisFleuret: It seems that Module.zero_grad() does not like parameters with no grad (source below crashes), but what is the proper way to have model parameters which should not be touched by the backprop, but should benefit from Module’s comfort (cuda() etc.)? <SCODE>import torch

from torch import Tensor
from torch.nn.parameter import Parameter
from torch.nn import Module

class Blah(Module):
    def __init__(self, dim):
        super(Blah, self).__init__()
        self.s = Parameter(torch.rand(1, dim), requires_grad = False)
        self.t = Parameter(torch.rand(1, dim))

blah = Blah(10)

blah.zero_grad()<ECODE>

Atcold: This should do the trick. <SCODE>import torch

from torch import Tensor
from torch.nn.parameter import Parameter
from torch.autograd import Variable
from torch.nn import Module

class Blah(Module):
    def __init__(self, dim):
        super(Blah, self).__init__()
        self.s = Variable(torch.rand(1, dim))
        self.t = Parameter(torch.rand(1, dim))

blah = Blah(10)

blah.zero_grad()
<ECODE>

FrancoisFleuret: Using a Variable was my first choice, but then Module.cuda() does not propagate to it. With the use of Variable you suggest, blah.cuda() will convert blah.t.data to torch.cuda.FloatTensor as expected, but will leave blah.s.data unchanged. Wouldn’t it be more consistent that Module.zero_grad() deal with requires_grad=False? <SCODE>def zero_grad(self):
    """Sets gradients of all model parameters to zero."""
    for p in self.parameters():
        if hasattr(p, 'grad'): p.grad.data.zero_()<ECODE>

ypxie: you can overlod the cuda function and call default cuda method inside it. <SCODE>def cuda(self):
    super(Blah, self).cuda()
    self.s.cuda()
<ECODE>

FrancoisFleuret: But ain’t there other functionalities that I will have to fix also ? Persistence in particular ?

Atcold: 

FrancoisFleuret: I must be missing something, I thought registe_buffer was to specify persistent fields, not the ones to be “cudaified”. In any case, I presume it should be self.register(‘s’, self.s) ? The following , with s a Variable <SCODE>import torch

from torch import Tensor
from torch.nn.parameter import Parameter
from torch.nn import Module
from torch.autograd import Variable

class Blah(Module):
    def __init__(self, dim):
        super(Blah, self).__init__()
        # self.s = Parameter(torch.rand(dim), requires_grad = False)
        self.s = Variable(torch.rand(dim))
        self.register_buffer('s', self.s)
        self.t = Parameter(torch.rand(dim))

blah = Blah(10)

blah.zero_grad()

print('s', type(blah.s.data))
print('t', type(blah.t.data))

if torch.cuda.is_available():
    blah.cuda()
    print('s', type(blah.s.data))
    print('t', type(blah.t.data))
<ECODE> does print <SCODE>s <class 'torch.FloatTensor'>
t <class 'torch.FloatTensor'>
s <class 'torch.FloatTensor'>
t <class 'torch.cuda.FloatTensor'>
<ECODE> And making s a Tensor instead of a Variable does not help.

Atcold: What about this. <SCODE>import torch

from torch import Tensor
from torch.nn.parameter import Parameter
from torch.nn import Module
from torch.autograd import Variable

class Blah(Module):
    def __init__(self, dim):
        super(Blah, self).__init__()
        self.register_buffer('s', torch.rand(dim))
        self.t = Parameter(torch.rand(dim))

blah = Blah(10)

blah.zero_grad()

print('s', type(blah.s))
print('t', type(blah.t.data))

if torch.cuda.is_available():
    blah.cuda()
    print('s', type(blah.s))
    print('t', type(blah.t.data))
<ECODE> Which outputs the following. <SCODE>/home/atcold/anaconda3/bin/python /home/atcold/Work/buffer.py
s <class 'torch.FloatTensor'>
t <class 'torch.FloatTensor'>
s <class 'torch.cuda.FloatTensor'>
t <class 'torch.cuda.FloatTensor'>

Process finished with exit code 0
<ECODE>

csarofeen: Shouldn’t you use ?

apaszke: I’d also recommend using a buffer for that. However it is a bug, and it should be fixed anyway.