Optimising model.parameters and custom learnable parameter together using torch.optim gives non-leaf tensor error

username: saquibmazhar
cooked: <p>Framework: <strong>PyTorch</strong></p>
<p>I am trying to optimise a custom <strong>nn.parameter</strong>(Temperature) used in <strong>softmax</strong> calculation along with the model parameters using a single <strong>Adam</strong> optimiser while model training. But doing so gives the following error:</p>
<p><strong>ValueError: can’t optimize a non-leaf Tensor</strong></p>
<p>Here is my custom loss function:</p>
<pre><code>class CrossEntropyLoss2d(torch.nn.Module):
    def __init__(self, weight=None):
	    super().__init__()
	    self.temperature = torch.nn.Parameter(torch.ones(1, requires_grad=True, device=device))
	    self.loss = torch.nn.NLLLoss(weight)
	    self.loss.to(device)
	
    def forward(self, outputs, targets):
	    T_logits = self.temp_scale(outputs)
	    return self.loss(torch.nn.functional.log_softmax(T_logits, dim=1), targets)

    def temp_scale(self, logits):
	    temp = self.temperature.unsqueeze(1).expand(logits.size(1), logits.size(2), logits.size(3))
	    return logits/temp 
</code></pre>
<p>.<br>
.<br>
.<br>
.<br>
.<br>
.<br>
Here is the part of training code:</p>
<pre><code>criterion = CrossEntropyLoss2d(weight)
params = list(model.parameters()) +list(criterion.temperature)
optimizer = Adam(params, 5e-4, (0.9, 0.999),  eps=1e-08, weight_decay=1e-4)
</code></pre>
<p>Error:</p>
<pre><code>File "train_my_net_city.py", line 270, in train
optimizer = Adam(params, 5e-4, (0.9, 0.999),  eps=1e-08, weight_decay=1e-4)
File "/home/saquib/anaconda3/lib/python3.8/site-packages/torch/optim/adam.py", line 48, in __init__
super(Adam, self).__init__(params, defaults)
File "/home/saquib/anaconda3/lib/python3.8/site-packages/torch/optim/optimizer.py", line 54, in __init__
self.add_param_group(param_group)
File "/home/saquib/anaconda3/lib/python3.8/site-packages/torch/optim/optimizer.py", line 257, in add_param_group
raise ValueError("can't optimize a non-leaf Tensor")
ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>Checking the variable for leaf gives true:</p>
<pre><code>print(criterion.temperature.is_leaf)
True
</code></pre>
<p>The error arises due to the <strong>criterion.temperature</strong> parameter and not due to <strong>model.parameters</strong>.<br>
<strong>Kindly help!</strong></p>

username: ptrblck
cooked: <p>It seems that <code>list(criterion.temperature)</code> creates a non-leaf tensor with <code>UnbindBackward</code> as its <code>grad_fn</code>. Use <code>[criterion.temperature]</code> instead for now (I’m unsure if this is expected or not).</p>

username: tom
cooked: <p>list(tensor) takes the first dimension of the tensor as an iterable and makes a list of it.</p>
<pre><code class="lang-python">&gt;&gt;&gt; list(torch.randn(3))
[tensor(-1.0266), tensor(1.7809), tensor(0.0680)]
</code></pre>
<p>Note how there are three scalar tensors here that have been “computed” from the original 3-element tensor. To autograd, this iteration, splitting the tensor along the first axis, is computation:</p>
<p>The <code>[tensor]</code> sticks tensor (the original object) into a new list, so the original object is actually in the list:</p>
<pre><code class="lang-python">&gt;&gt;&gt; [torch.randn(3)]
[tensor([ 0.9664, -0.7144,  1.3371])]
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: ptrblck
cooked: <p>This makes total sense, thanks!<br>
I didn’t see the issue as I was using a single value in my test <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

username: saquibmazhar
cooked: <p>Dear <a class="mention" href="/u/tom">@tom</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a> thank you so much for your reply. It was still showing an error. But I managed to get it work by doing the following:</p>
<pre><code>  params = list(model.parameters())
  params.append(criterion.temperature)
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji only-emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>