How to make a tensor part of model parameters?

John_Deterious: <p>I have a parameter that is learnable, I want the model to update it. Here is how I attached it to the model:</p>
<pre><code class="lang-auto">class Dan(nn.Module):
    def __init__(self):
        super(Dan, self).__init__()
blah blah blah
        self.alpha = t.tensor(0.5, requires_grad=True).cuda()
</code></pre>
<p>It is alpha. However, after training, I find its value unchanged. It must hven’t been passed to optimizer when I asked for model.parameters. Why isn’t included in that method, and how to include it?</p>

Mazhar_Shaikh: <p>Hi John,<br>
alpha needs to be added to the model as a Parameter.</p>
<pre><code class="lang-auto">self.alpha = t.nn.Parameter(t.tensor(0.5), requires_grad=True).cuda()
</code></pre>
<p>The documentation for this is available <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Parameter" rel="nofollow noopener">here</a></p>

John_Deterious: <p>Hi, thanks for suggestion. It is not working however, still not being trained.</p>

Paulo_Mann: <p>Are you sure that you send this param to the optimizer? Would be better to help if you provide more code, especially for the optimizer creation and <code>forward()</code> function to see where you use the <code>self.alpha</code></p>

John_Deterious: <p>Hi Paulo. Good question. Am I sure that I sent this to the optimizer? Well, not explicitly, I just passed the entire model parameters as is conventional, and I assume that alpha was one of them, because after all, it was in the init method, so it should be part of the model. Can I pass it explicitly to the optimizer? How?</p>
<p>The other good question, how was alpha used in the code? I sent it as a paramter to a function that needed it, which is this:</p>
<pre><code class="lang-auto">def modrelu(re, im, alpha):
    abs_ = t.sqrt(re**2 + im**2)
    ang = t.atan2(im, re)
    abs_ = nn.functional.relu(abs_ + alpha)
    return abs_ * t.cos(ang), abs_ * t.sin(ang)

</code></pre>
<p>So, inside the forward of my neural net, at some point I call this function and I go like:</p>
<pre><code class="lang-auto">blah  =  modrelu( blah, blah, self.alpha )
</code></pre>
<p>Thanks.</p>

ptrblck: <p>Although the tensor was defined in the <code>__init__</code> method, it won’t show in the interal parameters:</p>
<pre><code class="lang-python">class Dan(nn.Module):
    def __init__(self):
        super(Dan, self).__init__()
        self.alpha = torch.tensor(0.5, requires_grad=True)

model = Dan()
print(list(model.parameters()))
&gt; []
</code></pre>
<p>As <a class="mention" href="/u/mazhar_shaikh">@Mazhar_Shaikh</a> said, you should register this tensor as an <code>nn.Parameter</code>:</p>
<pre><code class="lang-python">class Dan(nn.Module):
    def __init__(self):
        super(Dan, self).__init__()
        self.alpha = nn.Parameter(torch.tensor(0.5, requires_grad=True))

model = Dan()
print(list(model.parameters()))
&gt; [Parameter containing:
tensor(0.5000, requires_grad=True)]
</code></pre>
<p>which will make sure it’ll be passed to the optimizer.</p>
<p>The <code>requires_grad</code> attribute will be set to <code>True</code> by default for <code>nn.Parameters</code>, so you don’t have to set it manually.</p>
<p>The usage of  <code>alpha</code> in <code>modrelu</code> looks good and will yield valid gradients for it.</p>

John_Deterious: <p>It worked! Thank you ptrblck</p>

yingjiao_liu: <p>Hi Ptrblck! I got the same problem that the parameters are not updating. I am using torch.nn.Parameters as you suggested, but still does not work. At the last line of <strong>init</strong>, I created the model parameter named <code>combine_weight</code> in order to obtain trainable weights.</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.GRU = nn.GRU(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)
        self.combine_weight = nn.Parameter(torch.ones(12) / 12, requires_grad=True).to(device)
</code></pre>
<p>And I checked whether the <code>combine_weight</code> has existed in model.parameters and optimizer.param_groups. And yes the print results show that the parameter does exist.</p>
<pre><code class="lang-auto">down_model_optimizer = get_optimizer(config, downstream_model, given_learning_rate=config.PER_learning_rate)
i_list = [down_model_optimizer.param_groups[0]['params']]
print(i_list)

print results &gt;&gt;&gt;[[Parameter containing:
tensor([0.0833, 0.0833, 0.0833, 0.0833, 0.0833, 0.0833, 0.0833, 0.0833, 0.0833,
        0.0833, 0.0833, 0.0833], requires_grad=True)
</code></pre>
<p>Although this created parameter is in model and optimizer, the number is still 0.0833 without updating.<br>
what other places should I also check? Do you think what may cause this problem? Thank you so much!</p>

ptrblck: <p>By calling the <code>to()</code> operation on <code>nn.Parameter</code> you could create a non-leaf parameter.<br>
Move the <code>device</code> transfer into the tensor creation:</p>
<pre><code class="lang-python">self.combine_weight = nn.Parameter(torch.ones(12).to(device) / 12, requires_grad=True)
</code></pre>
<p>Let me know, if that helps.</p>

Jaideep_Valani: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<pre><code class="lang-auto">class loss(torch.nn.Module):
    def __init__(self):
        super(qloss, self).__init__()
        self.w=torch.nn.Parameter( torch.tensor([0.2,0.5,0.8]))
</code></pre>
<p>I have defined loss as Module and put some learnable paramters their  as you see above.<br>
Will these parameters be updated ,optimizers i suppose takes care of updating only model .parameters</p>

ptrblck: <p>Yes, the optimizer will update the <code>w</code> parameter, if you pass the <code>loss</code> parameters to it (as is done with any other module):</p>
<pre><code class="lang-python">l = loss()
optimizer = optim.SGD(l.parameters(), lr=1.)
</code></pre>

Jaideep_Valani: <p>how would i have both of them updated … i tried passing it as list  but i got an error module.Parameters not accepted or so…<br>
i tried some thing like thiis<br>
<code>optim.SGD([model.parameters(),loss.parameters())</code></p>

ptrblck: <p>You would have to pass all parameters as a list e.g. via:</p>
<pre><code class="lang-python">optim.SGD(list(model.parameters()) + list(l.parameters()))
</code></pre>

mathematics: <p>Hi<br>
what is difference between two of them<br>
<code>[p for p in model.parameters() if p.requires_grad]</code> or <code>model.parameters()</code><br>
while using with optimizer, i used to use second , Is there any difference when initializing them in optimizer or both  work fine</p>

ptrblck: <p>Both would yield the same result. However, in the second approach the optimizer will internally iterate all parameters and skip the ones which don’t require gradients.<br>
This can have a tiny overhead, as this check is not needed in the first approach.</p>

ptrblck: <p>I don’t know when and where you are calling <code>mem_update</code>. Note that you should create the trainable <code>nn.Parameter</code>s, pass them to the optimizer, and use them later. Currently you are creating the <code>threshold</code> parameter inside <code>mem_update</code> so if you are repeatedly calling this method, move the parameter creation outside of this method and pass it as an argument to it.</p>

111179: <p>Thank you for your replying. The <code>mem_update</code> is called in</p>
<pre><code class="lang-auto">class neural network(nn.Module):
   def __init__(self, inplanes, planes,  batch_size, stride=1, option='A'):
        super(neural network, self).__init__()
        self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn1 = BatchNorm(planes)
        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn2 = BatchNorm(planes)
        self.drop2 = nn.Dropout(0.2)
        self.planes = planes
        self.stride = stride
  def forward(self, x, c1_mem, c1_spike, c2_mem, c2_spike):
        out = self.bn1(self.conv1(x))
        c1_mem, c1_spike = mem_update(out, c1_mem, c1_spike)
        out = self.bn2(self.conv2(c1_spike))
        out += self.shortcut(x)
        c2_mem, c2_spike = mem_update(out, c2_mem, c2_spike)
        c2_spike = self.drop2(c2_spike)
        return c2_spike, c1_mem, c1_spike, c2_mem, c2_spike
</code></pre>
<p>I have moved the <code>threshold</code> parameter outside of the <code>mem_update,</code> but this goes against the <code>learnable threshold</code> parameter I want to implement.</p>

ptrblck: <p>Why do you think it goes against the learnable <code>threshold</code> parameter?<br>
I don’t see in the current code snippet where <code>threshold</code> is created, but in your previous code you were recreating the <code>threshold</code> in each <code>forward</code> pass, so no training was performed on this parameter.</p>

111179: <p>Thank you for your replying.<br>
Based on your suggestions, I moved the threshold to the <code>nn.Module</code>, like:</p>
<pre><code class="lang-auto">class SpikingBasicBlock(nn.Module):
    expansion = 1
    def __init__(self, inplanes, planes, image_size, batch_size, stride=1, option='A',init_threshold=1.0):
        super(SpikingBasicBlock, self).__init__()
        self.threshold = nn.Parameter(torch.tensor(init_threshold, dtype=torch.float))
        self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn1 = BatchNorm(planes)
        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn2 = BatchNorm(planes)
        self.drop2 = nn.Dropout(0.2)
        self.planes = planes
        self.stride = stride 
    def forward(self, x, c1_mem, c1_spike, c2_mem, c2_spike):
        out = self.bn1(self.conv1(x))
        c1_mem, c1_spike = mem_update(out, c1_mem, c1_spike,self.threshold)     
        out = self.bn2(self.conv2(c1_spike))
        out += self.shortcut(x)
        c2_mem, c2_spike = mem_update(out, c2_mem, c2_spike,self.threshold)
        c2_spike = self.drop2(c2_spike)
        return c2_spike, c1_mem, c1_spike, c2_mem, c2_spike
</code></pre>
<p>Up to now, the threshold can update by <code>forward</code> pass, but its changes are minimal. Is it because  the gradient of <code>self. threshold</code> is 0 everywhere?</p>
<p>Thank you again for your support.<br>
Best regards.</p>

ptrblck: <p>Yes, if the gradient is calculated as zeros for <code>self.threshold</code>, it won’t be updated (unless the gradient was non zero before and you are using an optimizer with running estimates, weight decay etc.).</p>