How to put different groups of trainable parameters into optimizer in Pytorch

xiong_xiong: <p>My problem is how to train different groups of parameters with an optimizer in PyTorch.<br>
I have a basic fully-connected neural network with trainable weights and biases, and I have two special trainable parameters(lambda_1 and lambda_2), then I can get two groups of parameters. Group one includes weights and biases,  group two includes<br>
weights, biases, and lambda_1 and lambda_2.  My goal is to put these two groups of parameters into an optimizer, respectively. I tried to implement it but failed. Here is My code.</p>
<pre><code class="lang-auto">print("group one: weight, biase")
for p in model.parameter_wb():
  print(p)

print("\n")
print("group two: weight, biase and lambda")
for p in model.parameter_wb_lambda():
  print(p)
</code></pre>
<p>Why model.parameter_wb() and model.parameter_wb_lambda() produce the same result(both of them produce parameters of group two. I hope to get two different groups of trainable parameters.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

from collections import OrderedDict

class FNN(torch.nn.Module):
  def __init__(self, layers):
    super(FNN, self).__init__()
    
    # parameters
    self.depth = len(layers) - 1
    
    # set up layer order dict
    #  torch.nn.Tanhshrink/torch.nn.Tanh
    # torch.nn.functional.tanh
    self.activation = torch.nn.Tanh
    
    layer_list = list()
    for i in range(self.depth - 1): 
        layer_list.append(
            ('layer_%d' % i, torch.nn.Linear(layers[i], layers[i+1]))
        )
        layer_list.append(('activation_%d' % i, self.activation()))
        
    layer_list.append(
        ('layer_%d' % (self.depth - 1), torch.nn.Linear(layers[-2], layers[-1]))
    )
    layerDict = OrderedDict(layer_list)
    
    # deploy layers
    self.layers = torch.nn.Sequential(layerDict)
    
  def forward(self, x):
      out = self.layers(x)
      return out

class Model():
  def __init__(self,  layers, nn):   
    self.nn = nn
    # deep neural networks
    self.layers = layers
    
    if self.nn == "FNN":
      self.multi_task_model = FNN(self.layers)

    # group 1: weight and biase
    self.parameter_wb = self.multi_task_model.parameters

    # another two trainable parameters
    self.lambda_1 = torch.tensor([0.0], requires_grad=True)
    self.lambda_2 = torch.tensor([-6.0], requires_grad=True)
    
    self.lambda_1 = torch.nn.Parameter(self.lambda_1)
    self.lambda_2 = torch.nn.Parameter(self.lambda_2)
    
    # register parameter
    self.multi_task_model.register_parameter('lambda_1', self.lambda_1)
    self.multi_task_model.register_parameter('lambda_2', self.lambda_2)

    # group 2: weight, biase and lambda
    self.parameter_wb_lambda = self.multi_task_model.parameters

layers = [1,3,3,2]

nn = "FNN"
model = Model(layers, nn)

print("print trainable parameter: weight, biase")
for p in model.parameter_wb():
  print(p)

print("\n")
print("print trainable parameter: weight, biase and lambda")
for p in model.parameter_wb_lambda():
  print(p)

</code></pre>

KFrank: <p>Hi Xiong!</p>
<aside class="quote no-group quote-modified" data-username="xiong_xiong" data-post="1" data-topic="153685" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xiong_xiong/48/31163_2.png" class="avatar"> xiong_xiong:</div>
<blockquote>
<p>Why model.parameter_wb() and model.parameter_wb_lambda() produce the same result</p>
<pre><code class="lang-auto">    # group 1: weight and biase
    self.parameter_wb = self.multi_task_model.parameters
...
    # register parameter
    self.multi_task_model.register_parameter('lambda_1', self.lambda_1)
    self.multi_task_model.register_parameter('lambda_2', self.lambda_2)

    # group 2: weight, biase and lambda
    self.parameter_wb_lambda = self.multi_task_model.parameters
</code></pre>
</blockquote>
</aside>
<p>Your problem is that:</p>
<pre data-code-wrap="python"><code class="lang-python">    self.parameter_wb = self.multi_task_model.parameters
</code></pre>
<p>assigns the <code>parameters</code> method of your <code>multi_task_model</code> to<br>
<code>parameter_wb</code>, but doesn’t <em>evaluate</em> that method.</p>
<p>Then:</p>
<pre data-code-wrap="python"><code class="lang-python">    self.parameter_wb_lambda = self.multi_task_model.parameters
</code></pre>
<p>assigns that <em>same</em> method to <code>parameter_wb_lambda</code>, again without<br>
evaluating it.  It doesn’t matter that you registered two more <code>Parameter</code>s<br>
in between the two assignments.</p>
<p>It’s only when you later call:</p>
<pre data-code-wrap="python"><code class="lang-python">for p in model.parameter_wb():
# and
for p in model.parameter_wb_lambda():
</code></pre>
<p>that the <code>parameters</code> method is actually evaluated and it returns, both<br>
times, the (generator for the) list of all of the <code>Parameter</code>s that have been<br>
registered <em>at the time</em> of evaluation.</p>
<p>Best.</p>
<p>K. Frank</p>

xiong_xiong: <p>Hi, Frank, thank you so much for your nice answer. I understand the reason for the same result produced by</p>
<pre><code class="lang-auto">for p in model.parameter_wb():
# and
for p in model.parameter_wb_lambda():
</code></pre>
<p>I hope to pass different groups of parameters into an optimizer. I hope that self.parameter_wb() will produce weights and biases, and self.parameter_wb_var() will produce weights, biases, and lambda_1 and lambda_2. Then I can get two different optimizers. I have no idea how to implement this. Could you please give some advice?<br>
Thank you so much anyway.</p>
<pre><code class="lang-auto">self.optimizer_1 = optim.Adam(self.parameter_wb(), lr=0.001)
self.optimizer_2 = optim.Adam(self.parameter_wb_var(), lr=0.001)
</code></pre>