"bool value of Tensor with more than one value is ambiguous" when trying set different lr to different layers

hsiangyu: <p>Hello everyone, recently I have met with a tough issue (at least for me) when I am tring to set different learning rate to different layers.<br>
My segmentation model consists of a pretrained encoder and a decoder, so I want to set a lower lr to pretrained encoder. The encoder consists of a set of modules which have similar names: conv_encoder1, conv_encoder2, …<br>
So I have written some code like this:</p>
<pre><code class="lang-auto">encode_params = []
for i in range(5):
    module = eval('model.conv_encode{}'.format(i+1))
    params = list(map(id, module.parameters()))
    encode_params += params
decode_params = filter(lambda p: id(p) not in encode_params,
                     model.parameters())
optimizer = torch.optim.SGD([{'params': decode_params},
                             {'params': encode_params, 'lr': base_lr / 10}],
                            lr=base_lr, momentum=0.9)
</code></pre>
<p>However when running my code I get error in the topic: RuntimeError: bool value of Tensor with more than one value is ambiguous.<br>
Also when I am running a single line of</p>
<pre><code class="lang-auto">list(decode_params)
</code></pre>
<p>I get the same error as this. I have no idea what goes wrong and I will appreciate it if anyone would point out the mistakes I have made in my code.<br>
This is the first time I post a topic in PyTorch forum and please forgive me for any inappropriate language.</p>

mariosasko: <p>This is not a good place to use <code>eval</code>. Furthermore, you should use sets to separate the encoder parameters from the decoder parameters instead of <code>id</code>. With this in mind, a better way to write the first part is:</p>
<pre><code class="lang-python">encode_params = set()
for i in range(5):
     module = getattr(model, 'conv_encode{}'.format(i+1))
     encode_params.update(module.parameters())

model_params = set(model.parameters())
decode_params = model_params - encode_params
 
optimizer = torch.optim.SGD([{'params': list(decode_params)},
                             {'params': list(encode_params), 'lr': base_lr / 10}], 
                            lr=base_lr, momentum=0.9)
</code></pre>
<p>Try to print everything that you suspect may cause the issue.</p>
<p>If you can’t solve the problem yourself, please provide some additional information so it’s easier for us to find a solution (the error’s stack trace would be helpful, and printing <code>decode_params</code>).</p>

hsiangyu: <p>It works! I am really grateful for your help, but I am still a little bit confused of why substituting set for <code>id</code> would work in my case.</p>

mariosasko: <p>One issue is that you were passing ids instead of tensors to the optimizer (<code>{'params': encode_params}</code>), so this would raise an error eventually but not the one you mention. Other than that, it’s not obvious what’s the problem in your original code, so please provide the <strong>entire</strong> stack trace (in both cases) to make it easier for us.</p>

hsiangyu: <p>I have tried my initial versions of code and they got error like this:</p>
<ol>
<li>when running code below:</li>
</ol>
<pre><code class="lang-auto">  ...: base_lr = 1e-4
  ...: encode_params = []
  ...: for i in range(5):
  ...:     module = eval('model.conv_encode{}'.format(i+1))
  ...:     params = list(module.parameters())
  ...:     encode_params += params
  ...: decode_params = filter(lambda p: p not in encode_params,
  ...:                      model.parameters())
  ...: optimizer = torch.optim.SGD([{'params': decode_params},
  ...:                              {'params': encode_params, 'lr': base_lr / 10}],
  ...:                             lr=base_lr, momentum=0.9)
Traceback (most recent call last):
  File "D:\Anaconda3\envs\torch\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "&lt;ipython-input-2-36fa8cf864b8&gt;", line 57, in &lt;module&gt;
    lr=base_lr, momentum=0.9)
  File "D:\Anaconda3\envs\torch\lib\site-packages\torch\optim\sgd.py", line 64, in __init__
    super(SGD, self).__init__(params, defaults)
  File "D:\Anaconda3\envs\torch\lib\site-packages\torch\optim\optimizer.py", line 51, in __init__
    self.add_param_group(param_group)
  File "D:\Anaconda3\envs\torch\lib\site-packages\torch\optim\optimizer.py", line 195, in add_param_group
    param_group['params'] = list(params)
  File "&lt;ipython-input-2-36fa8cf864b8&gt;", line 53, in &lt;lambda&gt;
    decode_params = filter(lambda p: p not in encode_params,
RuntimeError: The size of tensor a (64) must match the size of tensor b (7) at non-singleton dimension 3
</code></pre>
<ol start="2">
<li>when running code below (I think this version may be entirely wrong)</li>
</ol>
<pre><code class="lang-auto">  ...: base_lr = 1e-4
  ...: encode_params = []
  ...: for i in range(5):
  ...:     module = eval('model.conv_encode{}'.format(i+1))
  ...:     params = list(module.parameters())
  ...:     encode_params += params
  ...: decode_params = filter(lambda p: id(p) not in encode_params,
  ...:                      model.parameters())
  ...: optimizer = torch.optim.SGD([{'params': decode_params},
  ...:                              {'params': encode_params, 'lr': base_lr / 10}],
  ...:                             lr=base_lr, momentum=0.9)
Traceback (most recent call last):
  File "D:\Anaconda3\envs\torch\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "&lt;ipython-input-5-91e3d95a8795&gt;", line 11, in &lt;module&gt;
    lr=base_lr, momentum=0.9)
  File "D:\Anaconda3\envs\torch\lib\site-packages\torch\optim\sgd.py", line 64, in __init__
    super(SGD, self).__init__(params, defaults)
  File "D:\Anaconda3\envs\torch\lib\site-packages\torch\optim\optimizer.py", line 51, in __init__
    self.add_param_group(param_group)
  File "D:\Anaconda3\envs\torch\lib\site-packages\torch\optim\optimizer.py", line 195, in add_param_group
    param_group['params'] = list(params)
  File "&lt;ipython-input-5-91e3d95a8795&gt;", line 7, in &lt;lambda&gt;
    decode_params = filter(lambda p: id(p) not in encode_params,
RuntimeError: bool value of Tensor with more than one value is ambiguous
</code></pre>

hsiangyu: <p>The initial version of my code is rather popular in many Chinese blogs and forums about PyTorch, but I am not sure why almost nobody has reported issues about these mistaken code in Chinses forums; it will be really nice of you if you can help identify the mistakes in the code (especially the reason why using list rather set can raise an error) and I will send the hyperlink and translated version of this issue there to help people not make similar mistakes.</p>

mariosasko: <p>Now I can see the error clearly. When the optimizer starts consuming the <code>filter</code> iterator, the membership test <code>id(p) not in encode_params</code> throws an error.</p>
<p>This snippet reproduces the error:</p>
<pre><code class="lang-python">ls = [torch.randn(3), torch.randn(2, 3)]
ls[0] in ls # returns True
torch.randn(4) in ls # error
id(torch.randn(3)) in ls # error again
</code></pre>
<p>You can think of the membership test looking like this:</p>
<pre><code class="lang-python">if obj is value:
   return True
else:
   return bool(obj == value)
</code></pre>
<p>This explains why the first example returns <code>True</code> and the other two throw an error.<br>
And this is why you should use sets instead of lists to solve this issue.</p>