Replacing value of a loss - custom loss

mbehzad: <p>Hi, how can I replace the loss value of the loss with another value?</p>
<p>Let’s say I have computed the loss as <code>loss = criterion(outputs, labels)</code>. The <code>print(loss.item())</code> gives me a value of let’s say 2 which I want to replace with let’s say 5. How can I do that?</p>
<p>I tried the following:</p>
<pre><code class="lang-auto">loss = criterion(outputs, labels)
print(loss)
loss_value = 5
loss.set_(torch.Tensor([loss_value]).to(device)[0])
print(loss)
</code></pre>
<p>but in this case, the grad_function gets disabled. Following is the output of the loss function before and after the new loss value:</p>
<pre><code class="lang-auto">tensor(2., device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
tensor(5., device='cuda:0', grad_fn=&lt;NotImplemented&gt;)
</code></pre>

Bmichele: <p>Hi,<br>
just a “replacement” of the loss value is probably not possible, as how should it be backpropagated ? (as the backpropagation is based on the calculation of the loss value and then take the derivative). If you think about making larger/smaller steps in the optimization, perhaps looking at the learning rate could help you.</p>

mbehzad: <p>Thanks for your reply. The loss would be of course back propagated after it is updated first. the next instructions are <code>loss.backward()</code> and then <code>optimizer.step()</code>. The above code will look as follows:</p>
<pre><code class="lang-auto">loss = criterion(outputs, labels)
loss_value = 5
#update the loss value
loss.set_(torch.Tensor([loss_value]).to(device)[0])

#compute gradients by back-propagation and update the parameters
loss.backward()
optimizer.step()
</code></pre>

Bmichele: <p>The problem is, that you need the gradient of the loss value w.r.t. the different weights of the different weights of your network, to update the weights like you do. But if you just set a number for the loss value, you are missing all these gradients. You can think of it, as setting a number (without any rule) is not a differentiable function, so you can not calculate the gradients.<br>
Why do you want to set the loss value to a certain number ?</p>

mbehzad: <p>There is a rule on why am I assigning a certain number to the loss but I have just simplified my problem for understanding. I am working on a custom loss value that I think that would help the network in better learning. The loss in each iteration would be different based on a rule so I think the gradients would get better each iteration. I think that is another topic to discuss what the gradients would be, but for now, I am interested in changing the loss value to another value.</p>

Bmichele: <p>I am not sure if I understand correctly what you want to do. You have a loss value, and you want to change this loss value due to a rule, and then update the weights of the layers.<br>
The update of the weights of the layers should be based on this new loss value ?<br>
If this “rule” is a differentiable function it is no problem, especially if it is based on functions already implemented in pytorch. I would then suggest to just write your own custom loss function.</p>

mbehzad: <p>Yes you understood right. So I actually have two networks with loss1 from net1 and a loss2 from net2. I want to back-propagate loss1+loss2 to both networks. Do I still need to write custom loss function or could just sum their loss values somehow and compute the gradients based on that?</p>

nivesh_gadipudi: <pre><code class="lang-auto">Loss = loss1 + loss2
Loss.backward()
Opmtimizer.step()
</code></pre>
<p>Yes, This will work. Pytorch’s gradient engine is intelligent enough to back propagate and update all the parameters connected to the Loss.</p>

mbehzad: <p>Thank you <a class="mention" href="/u/nivesh_gadipudi">@nivesh_gadipudi</a>. Would this compute gradients for both net1 and net2? How about if I want to let’s say compute the gradients for net1 only based on this new loss?</p>

nivesh_gadipudi: <p>You can use <code>required_grad = False</code> For the parameters in the network you don’t wanted to train.</p>