All learnable parameters are not listed in model.parameters

Gkv: <p>Hi all,<br>
I created my model as follows. self.outv is a learnable parameter.</p>
<pre><code class="lang-auto">class model(nn.Module):
    def __init__(self):
        super(model,self).__init__()
        self.lstm1=nn.LSTM(300,1024,num_layers=1)
        self.batch_size=1
        self.tanh=nn.Tanh()
        self.softmax=nn.Softmax()
        self.fc1=nn.Linear(1024,512)
        self.drop=nn.Dropout(0.5)
        self.outv=nn.Parameter(torch.randn(512))
        if torch.cuda.is_available:
            self.lstm1=self.lstm1.cuda(gpu_device)
            self.fc1=self.fc1.cuda(gpu_device)
            self.outv=self.outv.cuda(gpu_device)
    def forward(self,features):
        emb1=self.get_embedding(features)
        out1=self.fc1(emb1)
        out1=self.drop(out1)
        out2=torch.dot(self.outv,out1)
        return out2
    def get_embedding(self,featues):
        hidden = (autograd.Variable(torch.randn(1, 1, 1024)).cuda(gpu_device),
          autograd.Variable(torch.randn((1, 1, 1024))).cuda(gpu_device)) #initialization of hidden state: creates two variables each contain a 1x1x3 tensor
        #passing data
        for w_embed in features:
            self.lstm1.flatten_parameters()
            out,hidden=self.lstm1(Variable(torch.from_numpy(np.array(w_embed)),volatile=True).view(1,1,300).cuda(gpu_device),hidden)
        return out
 
m=model()
m=m.cuda(gpu_device)
print(m.parameters)
#print output
&lt;bound method Module.parameters of model(
  (lstm1): LSTM(300, 1024)
  (tanh): Tanh()
  (softmax): Softmax()
  (fc1): Linear(in_features=1024, out_features=512, bias=True)
  (drop): Dropout(p=0.5)
)&gt;
</code></pre>
<p>I have few doubts.</p>
<ol>
<li>Why self.outv is  not listed in m.parameters. Is there any problem in my code?</li>
<li>How can I move the model to multiple gpus?</li>
</ol>
<p>Thanks</p>

crcrpar: <p>Hi,</p>
<ol>
<li>
<p>If you want to register parameter to module, <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L117" rel="nofollow noopener"><code>self.register_parameter(name, parameter)</code></a> would be preferable.</p>
</li>
<li>
<p>I’m sorry. I do not have any answer now.</p>
</li>
</ol>

ptrblck: <p>If you just have <code>Parameter</code>s in your <code>__init__</code>, you don’t have to handle cuda assignments yourself.<br>
Just remove the <code>if torch.cuda.is_available</code> code and your parameters will be registered correctly.<br>
Also, <code>.parameters()</code> is a method, so you should call it like:</p>
<pre><code>print(list(m.parameters()))
</code></pre>
<p>or</p>
<pre><code>print(list(m.named_parameters()))</code></pre>

Gkv: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thanks a lot. The parameters part is clear now. So if I do <strong>loss.backward()</strong>, all the parameters listed will update right?. What are the situations that I have to handle cuda assignments myself? (can you please share some links which explain this).</p>
<p>How can I run this model in multiple GPUs, do the following code work?</p>
<pre><code class="lang-auto">if torch.cuda.is_available:
    m=m.cuda()
    m = nn.DataParallel(m, device_ids=None)
</code></pre>
<p>If so, what modification I need to do in the <strong>get_embedding</strong> function to work in multiple gpus? (I mean what i need to put in place of .<strong>cuda(gpu_device)</strong>. )</p>
<p>Thanks</p>

ptrblck: <p>You could get the device from your <code>self.lstm1</code>:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.lstm1 = nn.LSTM(300, 1024, num_layers=1)
        
    def forward(self, x):
        x = self.get_embedding(x)
        return x
        
    def get_embedding(self, x):
        device = 'cpu'
        if next(self.lstm1.parameters()).is_cuda:
            device = 'cuda:' + str(next(self.lstm1.parameters()).get_device())
        print('Using device: {}'.format(device))
        hidden = (torch.randn(1, 1, 1024).to(device),
                  torch.randn(1, 1, 1024).to(device))
        for w_embed in x:
            self.lstm1.flatten_parameters()
            out, hidden = self.lstm1(w_embed.detach().view(1, 1, 300).to(device), hidden)
        return out


model = MyModel()
model = model.to(0)
model = nn.DataParallel(model, device_ids=[0, 1])

x = torch.randn(2, 300)
output = model(x)
print(output)
</code></pre>

Gkv: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Is the to(device) is  analogous to .cuda(device)? When I ran your code it says,MyModel object has  no attribute  <strong>to</strong>. I am using pytorch 0.3.</p>
<p>Thanks</p>

ptrblck: <p>It’s the same syntax, if you push your model to the GPU, but not if it should stay on the CPU.<br>
If you are sure to use in on the GPU, you can just change it to <code>.cuda(device)</code>.<br>
Generally I would recommend updating to the current stable release (<code>0.4.0</code>).</p>

Jaideep_Valani: <p>hi ptrblck<br>
I had one doubt regarding model.parameters<br>
Assuming my model has got sequence which in turn is list of modules<br>
0 Sequential(<br>
(0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=True)<br>
(1): Relu()<br>
(2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
1<br>
Sequential(<br>
(0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=True)<br>
(1): Relu()<br>
(2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)</p>
<p>In this case if  i do<br>
for i in model[0].parameters():<br>
print(i)</p>
<p>How many time loop would be running  3 times (weight and bias of conv ,Batch norm parameters) or 4 times or more</p>

ptrblck: <p>If <code>model[0]</code> refers to the first <code>nn.Sequential</code> module, you’ll get 4 parameters:</p>
<ul>
<li><code>conv.weight</code></li>
<li><code>conv.bias</code></li>
<li><code>bn.weight</code></li>
<li><code>bn.bias</code></li>
</ul>

Jaideep_Valani: <p>Thanks so this would pull off only those which has got attr weights or bias ?<br>
so We dont need to explicitly check using hasttr(m,‘weight’)?</p>
<ol start="2">
<li>secondly i had got one more q with regard to the weights ,what is diff between m.weights and m.weights.data ?</li>
</ol>

ptrblck: <p>No, <code>model.parameters()</code> will list all registered parameters. E.g. if you are using a custom module and assign a parameter as</p>
<pre><code class="lang-python">self.my_param = nn.Parameter(torch.randn(1))
</code></pre>
<p>it will also be listed in <code>model.parameters()</code>.</p>
<p>What is your use case that you would like to check, if the module has <code>weight</code> as a parameter?</p>
<p><code>weight.data</code> refers to the underlying data tensor. If you manipulate it, Autograd won’t be able to track this operation, which might result in wrong calculations. Therefore I would recommend to avoid using the <code>.data</code> attribute.</p>

Alpha: <p>What will be registered in the <code>model.parameters()</code>.<br>
As far as now, what I know are as belows:</p>
<pre><code class="lang-auto">conv: weight, bias
bn: weight bias
nn.Parameter()
</code></pre>
<p>are there something else?<br>
Thank you in advance.</p>

111179: <p>Hello, Mr.Ptrblck.<br>
When I used <code>print(list(m.named_parameters()))</code> in my code, I still can not get the learnable parameters: <code>threshold</code>. Could you give me any suggestion? Thanks a lot.<br>
Below is my code:</p>
<pre><code class="lang-auto">class Surrogate_BP_Function(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        out = torch.zeros_like(input).cuda()
        out[input &gt; 0] = 1.0
        return out

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad = grad_input * 0.3 * F.threshold(1.0 - torch.abs(input), 0, 0)
        return grad
</code></pre>
<pre><code class="lang-auto">class SNN_VGG9_BNTT(nn.Module):
    def __init__(self, num_steps, leak_mem=0.95, img_size=32,  num_cls=10):
        super(SNN_VGG9_BNTT, self).__init__()

        self.img_size = img_size
        self.num_cls = num_cls
        self.num_steps = num_steps
        self.spike_fn = Surrogate_BP_Function.apply
        self.leak_mem = leak_mem
        self.batch_num = self.num_steps

        print ("&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; VGG 9 &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;")
        print ("***** time step per batchnorm".format(self.batch_num))
        print ("&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;")

        affine_flag = True
        bias_flag = False

        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=bias_flag)
        self.bntt1 = nn.ModuleList([nn.BatchNorm2d(64, eps=1e-4, momentum=0.1, affine=affine_flag) for i in range(self.batch_num)])
        self.conv2 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1, bias=bias_flag)
        self.bntt2 = nn.ModuleList([nn.BatchNorm2d(64, eps=1e-4, momentum=0.1, affine=affine_flag) for i in range(self.batch_num)])
        self.pool1 = nn.AvgPool2d(kernel_size=2)

        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1, bias=bias_flag)
        self.bntt3 = nn.ModuleList([nn.BatchNorm2d(128, eps=1e-4, momentum=0.1, affine=affine_flag) for i in range(self.batch_num)])
        self.conv4 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=bias_flag)
        self.bntt4 = nn.ModuleList([nn.BatchNorm2d(128, eps=1e-4, momentum=0.1, affine=affine_flag) for i in range(self.batch_num)])
        self.pool2 = nn.AvgPool2d(kernel_size=2)

        self.conv5 = nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1, bias=bias_flag)
        self.bntt5 = nn.ModuleList([nn.BatchNorm2d(256, eps=1e-4, momentum=0.1, affine=affine_flag) for i in range(self.batch_num)])
        self.conv6 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=bias_flag)
        self.bntt6 = nn.ModuleList([nn.BatchNorm2d(256, eps=1e-4, momentum=0.1, affine=affine_flag) for i in range(self.batch_num)])
        self.conv7 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=bias_flag)
        self.bntt7 = nn.ModuleList([nn.BatchNorm2d(256, eps=1e-4, momentum=0.1, affine=affine_flag) for i in range(self.batch_num)])
        self.pool3 = nn.AvgPool2d(kernel_size=2)


        self.fc1 = nn.Linear((self.img_size//8)*(self.img_size//8)*256, 1024, bias=bias_flag)
        self.bntt_fc = nn.ModuleList([nn.BatchNorm1d(1024, eps=1e-4, momentum=0.1, affine=affine_flag) for i in range(self.batch_num)])
        self.fc2 = nn.Linear(1024, self.num_cls, bias=bias_flag)

        self.conv_list = [self.conv1, self.conv2, self.conv3, self.conv4, self.conv5, self.conv6, self.conv7]
        self.bntt_list = [self.bntt1, self.bntt2, self.bntt3, self.bntt4, self.bntt5, self.bntt6, self.bntt7, self.bntt_fc]
        self.pool_list = [False, self.pool1, False, self.pool2, False, False, self.pool3]

        # Turn off bias of BNTT
        for bn_list in self.bntt_list:
            for bn_temp in bn_list:
                bn_temp.bias = None


        # Initialize the firing thresholds of all the layers
        for m in self.modules():
            if (isinstance(m, nn.Conv2d)):
                m.threshold = 1.0
                torch.nn.init.xavier_uniform_(m.weight, gain=2)
            elif (isinstance(m, nn.Linear)):
                m.threshold = 1.0
                torch.nn.init.xavier_uniform_(m.weight, gain=2)

    def forward(self, inp):
        batch_size = inp.size(0)
        mem_conv1 = torch.zeros(batch_size, 64, self.img_size, self.img_size).cuda()
        mem_conv2 = torch.zeros(batch_size, 64, self.img_size, self.img_size).cuda()
        mem_conv3 = torch.zeros(batch_size, 128, self.img_size//2, self.img_size//2).cuda()
        mem_conv4 = torch.zeros(batch_size, 128, self.img_size//2, self.img_size//2).cuda()
        mem_conv5 = torch.zeros(batch_size, 256, self.img_size//4, self.img_size//4).cuda()
        mem_conv6 = torch.zeros(batch_size, 256, self.img_size//4, self.img_size//4).cuda()
        mem_conv7 = torch.zeros(batch_size, 256, self.img_size//4, self.img_size//4).cuda()
        mem_conv_list = [mem_conv1, mem_conv2, mem_conv3, mem_conv4, mem_conv5, mem_conv6, mem_conv7]

        mem_fc1 = torch.zeros(batch_size, 1024).cuda()
        mem_fc2 = torch.zeros(batch_size, self.num_cls).cuda()
        for t in range(self.num_steps):

            for i in range(len(self.conv_list)):
                mem_conv_list[i] = self.leak_mem * mem_conv_list[i] + self.bntt_list[i][t](self.conv_list[i](inp))
                mem_thr = (mem_conv_list[i] / self.conv_list[i].threshold) - 1.0
                out = self.spike_fn(mem_thr)
                rst = torch.zeros_like(mem_conv_list[i]).cuda()
                rst[mem_thr &gt; 0] = self.conv_list[i].threshold
                mem_conv_list[i] = mem_conv_list[i] - rst
                out_prev = out.clone()
                if self.pool_list[i] is not False:
                    out = self.pool_list[i](out_prev)
                    out_prev = out.clone()
            out_prev = out_prev.reshape(batch_size, -1)
            mem_fc1 = self.leak_mem * mem_fc1 + self.bntt_fc[t](self.fc1(out_prev))
            mem_thr = (mem_fc1 / self.fc1.threshold) - 1.0
            out = self.spike_fn(mem_thr)
            rst = torch.zeros_like(mem_fc1).cuda()
            rst[mem_thr &gt; 0] = self.fc1.threshold
            mem_fc1 = mem_fc1 - rst
            out_prev = out.clone()
            mem_fc2 = mem_fc2 + self.fc2(out_prev)
        out_voltage = mem_fc2 / self.num_steps
        return out_voltage
</code></pre>

ptrblck: <p><code>threshold</code> is used from the functional API via <code>F.threshold</code>, which is a function and is thus not trainable, and also assigned as an attribute to some modules as a scalar, which isn’t a trainable <code>nn.Parameter</code> either.<br>
You would thus have to define it as an <code>nn.Parameter</code>, so that it’ll be returned by <code>model.parameters()</code>.</p>

111179: <p>Thank you so much for your kind reply. If <code>threshold</code> is not converted to nn.module, is there any other way to print <code>threshold</code>? Or have any other way to load the model’s state_dict?</p>
<p>Best regards.</p>

ptrblck: <p>If you want to register this tensor inside the module, so that it would also be added to the <code>state_dict</code>, but would remain constant, you could register it as a buffer via:</p>
<pre><code class="lang-python">m.register_buffer('threshold', torch.tensor([1.]))
</code></pre>

111179: <p>Thank you so much. I try it.</p>