Passing 'model.parameters() + other_parms' to optimizer

username: Skinish
cooked: <p>I am using <code>nn.AdaptiveLogSoftmaxWithLoss</code>. the way I am building my model, the loss is outside of my <code>nn.Module</code>.</p>
<p>How can I pass the weights included in this loss for them to appear in my <code>model.parameters()</code> and <code>model.modules()</code>? Or at least, how can I join both the parameters/modules of my model with the one sin the loss function?</p>

username: MariosOreo
cooked: <p>Hi,</p>
<p><code>model.parameters()</code> and <code>model.modules()</code> are both generator, firstly you could get the list of parameters and modules by <code>list(model.parameters())</code> and then passing the weights and the loss module in a <strong>append to list</strong> method.</p>
<p>But <code>model.modules()</code> get submodules in a iteration way, so there will be something difficult.</p>

username: alex.veuthey
cooked: <p><a href="https://discuss.pytorch.org/t/different-learning-rate-for-different-type-of-module/8999/3">This answer</a> is pretty much all you need!</p>
<p>In the SGD example of the answer, you would only need to change the <code>model.base</code> by <code>your_model_name</code> and <code>model.classifier</code> by <code>your_loss_name</code>. If you wrote your loss module properly (with registered <code>nn.Parameter</code>s and not just tensors), it should work.</p>

username: Skinish
cooked: <p>Yes, you are both right. Since the losses in PyTorch are <code>nn.Module</code>, it’s no problem, they have <code>parameters</code> and <code>modules</code> methods. But what if I wanted to add an extra tensor to the optimizer (with grad set to True)?</p>

username: MariosOreo
cooked: <p>Firstly, the extra tensor should be trainable (with requires_grad=True) and included in the computational graph.<br>
Secondly, you could add the extra tensor into optimizer.params_group like <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options" rel="nofollow noopener">this.</a></p>

username: Skinish
cooked: <p>Can you give a code example please? I don’t find that <code>optimizer.params_group</code>in the link that you mentioned</p>

username: MariosOreo
cooked: <p>I meant you could add extra tensor into optimizer by <code>optimizer.add_param_group</code><br>
or initialize your optimizer as follows:</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>

username: Skinish
cooked: <p>but a tensor does not have <code>.paramaters()</code>, what do you mean?</p>

username: alex.veuthey
cooked: <p>I don’t think you actually can do that with Tensors, which is the whole point of <code>torch.nn.Parameter</code>.</p>
<p>My understanding is that <code>Parameter</code> was added specifically to avoid computing gradients for normal Tensors.</p>
<p>What is your use case for Tensors which are not part of the model?</p>

username: Skinish
cooked: <p>yeah, <code>Parameter</code> works, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Suraj
cooked: <p>Hi,</p>
<p>So in my case I have pretrained a model and after pre-training, in the main training loop, I have applied a few functions (tensor operations) on top of the pre-trained model’s output. This, applying a new function has some parameters that I need to update at each iteration.</p>
<p>Do you recommend making a nn.Module sub-class out of these new functions and parameter where I can define the new parameter using nn.Parameter ??</p>
<p>The problem is, I will not be using any pytorch’s nn layer in this new model (model on top of pre-trained model). So, is it even necessary or the best practice to add additional parameters like this? What do you guys suggest? Please let me know If I am being unclear.</p>
<p>Would appreciate your help, Thanks!</p>

username: alex.veuthey
cooked: <p>I think even if you don’t use PyTorch layers you should use Parameters for your learnable weights/params. This will probably make it easier for you.</p>
<p>Using a subclass of nn.Module will ensure that all the backend work for computing gradients will work out of the box, and you can use your functions (or let’s call it a module, then) like another module of pytorch layers…</p>