Make nn.Parameter recognizable by optimizer

Ashwin_Raju: <p>Here is my model.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = ...
        self.conv3 = ...
        self.custom_param = torch.nn.Parameter(torch.zeros(100, 50, requires_grad=True)).cuda()

model = Model()
</code></pre>
<p>When i send the <code>mode.parameters()</code> to my optimizer I am not able to view my <code>custom_param</code> parameter in <code>optimizer.param_groups[0]['params']</code> this makes the custom_param weights not trainable. How do I make this parameter recognizable by optimizer ?</p>

ptrblck: <p>Could you try to call <code>.cuda()</code> on the tensor instead of the <code>nn.Parameter</code>?<br>
Most likely <code>custom_param</code> isn’t a leaf variable anymore as it was created by <code>.cuda()</code>.<br>
Have a look at <a href="https://discuss.pytorch.org/t/valueerror-cant-optimize-a-non-leaf-tensor/21751/2?u=ptrblck">this post</a> for more information.</p>

sprakashdash: <p>I have a similar problem and I have specified <code>.to(device)</code> instead of <code>.cuda()</code>. But I am getting error that: <code>AttributeError: 'Tensor' object has no attribute 'parameters'</code></p>

ptrblck: <p>I guess you are calling <code>.parameters()</code> on a tensor instead of an <code>nn.Module</code>.<br>
Make sure to call this method on the model itself and not (internal) tensors.</p>

sprakashdash: <p>Yes I am calling the parameters of that particular tensor for specifying the learning rate something like this:<br>
<code>optimizer = optim.Adam([{'params':model.custom_param.parameters(), 'lr':1e-3}, ... and so on])</code></p>
<p>How can I do that?</p>

ptrblck: <p>A tensor doesn’t have the <code>.parameters()</code> method, so either call it on an <code>nn.Module</code> object or pass the <code>nn.Parameter</code> to the optimizer instead.</p>

sprakashdash: <p>So inside the model class if I have defined the layer something like this: <code>self.actor_logstd = nn.Parameter(torch.zeros(1, np.prod(envs.single_action_space.shape)))</code>, then I can directly call <code>optimizer = optim.Adam([{'params':model.actor_logstd, 'lr':1e-3}])</code> without adding the <code>.parameters()</code> method?</p>

ptrblck: <aside class="quote no-group" data-username="sprakashdash" data-post="7" data-topic="33910">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sprakashdash/48/48780_2.png" class="avatar"> sprakashdash:</div>
<blockquote>
<p>I have defined the layer something like this</p>
</blockquote>
</aside>
<p>You are defining a parameter, not a layer.</p>
<aside class="quote no-group" data-username="sprakashdash" data-post="7" data-topic="33910">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sprakashdash/48/48780_2.png" class="avatar"> sprakashdash:</div>
<blockquote>
<p>then I can directly call</p>
</blockquote>
</aside>
<p>Yes, you can pass parameters like this to the optimizer.</p>