TypeError: optimizer can only optimize Tensors, but one of the params is list

username: tianle-BigRice
cooked: <p>I needed to optimize my own loss using the Optimizer, but I ran into this problem</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 138, in &lt;module&gt;
    fire.Fire()
  File "D:\Anaconda3\lib\site-packages\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "D:\Anaconda3\lib\site-packages\fire\core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "D:\Anaconda3\lib\site-packages\fire\core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "train.py", line 73, in train
    trainer = FasterRCNNTrainer(faster_rcnn).cuda()
  File "D:\python\monoSAR_rcnn\trainer.py", line 63, in __init__
    self.optimizer = self.faster_rcnn.get_optimizer()
  File "D:\python\monoSAR_rcnn\model\faster_rcnn.py", line 307, in get_optimizer
    self.optimizer = t.optim.SGD([params, {'params': self.awl.parameters(), 'weight_decay': 0}], momentum=0.9)
  File "D:\Anaconda3\lib\site-packages\torch\optim\sgd.py", line 68, in __init__
    super(SGD, self).__init__(params, defaults)
  File "D:\Anaconda3\lib\site-packages\torch\optim\optimizer.py", line 52, in __init__
    self.add_param_group(param_group)
  File "D:\Anaconda3\lib\site-packages\torch\optim\optimizer.py", line 230, in add_param_group
    raise TypeError("optimizer can only optimize Tensors, "
TypeError: optimizer can only optimize Tensors, but one of the params is list
</code></pre>
<p>this is my get_optimizer</p>
<pre><code class="lang-auto">    def get_optimizer(self):
        """
        return optimizer, It could be overwriten if you want to specify 
        special optimizer
        """
        lr = opt.lr
        params = []
        for key, value in dict(self.named_parameters()).items():
            if value.requires_grad:
                if 'bias' in key:
                    params += [{'params': [value], 'lr': lr * 2, 'weight_decay': 0}]
                else:
                    params += [{'params': [value], 'lr': lr, 'weight_decay': opt.weight_decay}]
        print(type(params))
        if opt.use_adam:
            self.optimizer = t.optim.Adam([params, {'params': self.awl.parameters(), 'weight_decay': 0}])
        else:
            self.optimizer = t.optim.SGD([params, {'params': self.awl.parameters(), 'weight_decay': 0}], momentum=0.9)
        return self.optimizer
</code></pre>
<p>this is my loss function   (self.awl = AutomaticWeightedLoss(2))</p>
<pre><code class="lang-auto">class AutomaticWeightedLoss(nn.Module):
    """automatically weighted multi-task loss
    Params：
        num: int，the number of loss
        x: multi-task loss
    Examples：
        loss1=1
        loss2=2
        awl = AutomaticWeightedLoss(2)
        loss_sum = awl(loss1, loss2)
    """
    def __init__(self, num=2):
        super(AutomaticWeightedLoss, self).__init__()
        params = torch.ones(num, requires_grad=True)
        self.params = torch.nn.Parameter(params) 

    def forward(self, *x):
        loss_sum = 0
        for i, loss in enumerate(x):
            loss_sum += 0.5 / (self.params[i] ** 2) * loss + torch.log(1 + self.params[i] ** 2)

        return loss_sum
</code></pre>
<p>How can I solve this problem?</p>

username: KFrank
cooked: <p>Hi Tianle!</p>
<aside class="quote no-group quote-modified" data-username="tianle-BigRice" data-post="1" data-topic="127138" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/5daacb/48.png" class="avatar"> tianle-BigRice:</div>
<blockquote>
<pre><code class="lang-auto">TypeError: optimizer can only optimize Tensors, but one of the params is list
</code></pre>
<pre><code class="lang-auto">        params = []
...
        print(type(params))
        if opt.use_adam:
            self.optimizer = t.optim.Adam([params, {'params': self.awl.parameters(), 'weight_decay': 0}])
</code></pre>
</blockquote>
</aside>
<p>Your problem is that you passing your optimizer a <code>list</code> that consists<br>
of a <code>list</code> and a <code>dict</code>.  Quoting from the <a href="https://pytorch.org/docs/1.9.0/optim.html" rel="noopener nofollow ugc">torch.optim</a> documentation:</p>
<blockquote>
<h3>Per-parameter options</h3>
<p><a href="https://pytorch.org/docs/1.9.0/optim.html#torch.optim.Optimizer" rel="noopener nofollow ugc"><code>Optimizer</code></a> s also support specifying per-parameter options. To do this, instead of passing an iterable of <code>Variable</code> s, pass in an iterable of <a href="https://docs.python.org/3/library/stdtypes.html#dict" rel="noopener nofollow ugc"><code>dict</code></a> s. Each of them will define a separate parameter group, and should contain a <code>params</code> key, containing a list of parameters belonging to it.</p>
</blockquote>
<p>(Note, this documentation should be updated to reflect that <code>Variable</code><br>
is deprecated; the term <code>Parameter</code> should be used instead.)</p>
<p>That is, you can pass in  a <code>list</code> of <code>Parameter</code>s or a <code>list</code> of <code>dict</code>s<br>
(that contain <code>list</code>s of <code>Parameter</code>s), but you can’t pass in a <code>list</code><br>
of <code>list</code>s.</p>
<p>I think this would work for you:</p>
<pre data-code-wrap="python"><code class="lang-python">        print(type(params))   # at this point params is a list of dicts
        if opt.use_adam:
            params += [{'params': self.awl.parameters(), 'weight_decay': 0}]
            self.optimizer = t.optim.Adam (params)   # still a list of dicts
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: tianle-BigRice
cooked: <p>Hi KFrank!<br>
Thank you very much for your reply，I tried what you said</p>
<pre><code class="lang-auto">        if opt.use_adam:
            params += [{'params': self.awl.parameters(), 'weight_decay': 0}]
            self.optimizer = t.optim.Adam(params)
        else:
            params += [{'params': self.awl.parameters(), 'weight_decay': 0}]
            self.optimizer = t.optim.SGD(params, momentum=0.9)
</code></pre>
<p>But the error happened :</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 138, in &lt;module&gt;
    fire.Fire()
  File "D:\Anaconda3\lib\site-packages\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "D:\Anaconda3\lib\site-packages\fire\core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "D:\Anaconda3\lib\site-packages\fire\core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "train.py", line 73, in train
    trainer = FasterRCNNTrainer(faster_rcnn).cuda()
  File "D:\python\monoSAR_rcnn\trainer.py", line 63, in __init__
    self.optimizer = self.faster_rcnn.get_optimizer()
  File "D:\python\monoSAR_rcnn\model\faster_rcnn.py", line 310, in get_optimizer
    self.optimizer = t.optim.SGD(params, momentum=0.9)
  File "D:\Anaconda3\lib\site-packages\torch\optim\sgd.py", line 68, in __init__
    super(SGD, self).__init__(params, defaults)
  File "D:\Anaconda3\lib\site-packages\torch\optim\optimizer.py", line 52, in __init__
    self.add_param_group(param_group)
  File "D:\Anaconda3\lib\site-packages\torch\optim\optimizer.py", line 237, in add_param_group
    raise ValueError("parameter group didn't specify a value of required optimization parameter " +
ValueError: parameter group didn't specify a value of required optimization parameter lr
</code></pre>
<p>Then I tried to print params，  here’s what print in the last part of this dict:</p>
<pre><code class="lang-auto"> {'params': [Parameter containing:
tensor([1., 1.], requires_grad=True)], 'lr': 0.001, 'weight_decay': 0.0005}, 
(this is loss.params){'params': &lt;generator object Module.parameters at 0x00000226C29AEC10&gt;, 'weight_decay': 0}]
</code></pre>
<p>So I wonder if this is the problem with loss.params?</p>
<p>Best.</p>
<p>tianle</p>

username: KFrank
cooked: <p>Hi Tianle!</p>
<aside class="quote no-group" data-username="tianle-BigRice" data-post="3" data-topic="127138" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/5daacb/48.png" class="avatar"> tianle-BigRice:</div>
<blockquote>
<pre><code class="lang-auto">ValueError: parameter group didn't specify a value of required optimization parameter lr
</code></pre>
</blockquote>
</aside>
<p>Do what the error message says:  Include a value for <code>lr</code> when you<br>
append <code>self.awl.parameters()</code> to your <code>params</code> list of parameter<br>
groups (analogously to how you initially build <code>params</code> from<br>
<code>self.named_parameters()</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

username: tianle-BigRice
cooked: <p>Hi KFrank!<br>
Thank you very much for your reply，I tried what you said</p>
<pre><code class="lang-auto">params += [{'params': self.awl.parameters(), 'lr': lr, 'weight_decay': 0}]
</code></pre>
<p>But the error happened :</p>
<pre><code class="lang-auto">ValueError: some parameters appear in more than one parameter group
</code></pre>
<p>Then I tried to print params， here’s what print in the last part of this dict:</p>
<pre><code class="lang-auto">{'params': [Parameter containing:
tensor([1., 1.], requires_grad=True)], 'lr': 0.001, 'weight_decay': 0.0005},
 {'params': &lt;generator object Module.parameters at 0x000001F771AEFBA0&gt;, 'lr': 0.001, 'weight_decay': 0}]
</code></pre>
<p>That means I get an error if I add, and an error if I don’t <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"><br>
But if I don’t add loss.params the code will be fine…</p>
<p>Best.</p>
<p>tianle</p>

username: KFrank
cooked: <p>Hi Tianle!</p>
<aside class="quote no-group" data-username="tianle-BigRice" data-post="5" data-topic="127138" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/5daacb/48.png" class="avatar"> tianle-BigRice:</div>
<blockquote>
<pre><code class="lang-auto">ValueError: some parameters appear in more than one parameter group
</code></pre>
</blockquote>
</aside>
<p>What do you think this error message might be telling you?</p>
<p>If you were debugging somebody else’s code, what issues might<br>
this error message suggest you take a look at?</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">{'params': [Parameter containing:
tensor([1., 1.], requires_grad=True)], 'lr': 0.001, 'weight_decay': 0.0005},
</code></pre>
</blockquote>
</aside>
<p>This entry in your <code>params</code> list is a dictionary that contains a <code>Parameter</code>.<br>
Does this make sense?</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto"> {'params': &lt;generator object Module.parameters at 0x000001F771AEFBA0&gt;, 'lr': 0.001, 'weight_decay': 0}]
</code></pre>
</blockquote>
</aside>
<p>But this entry in your <code>params</code> list is a dictionary that contains<br>
something called a “generator object.”  Is this okay?  Is it legitimate<br>
that your <code>params</code> list contains two different kinds of things?  (And<br>
what is a “generator object?”)</p>
<p>What could you do to explore further what might be going on here?</p>
<p>Best.</p>
<p>K. Frank</p>