How to combine multiple criterions to a loss function?

robromijnders: <p>I;m building recursive neural nets, so I’ll have a variable list of criterions. I want to sum them up and backpropagate error.<br>
All the errors are single float values and of same scale. So the total loss is the sum of individual losses</p>
<p>To understand this, I crystallized this problem. It’s for another classification project.<br>
I wrote this code and it works</p>
<pre><code>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()
  output= model(data)
  final = output[-1,:,:]
  loss = criterion(final,targets)
  return loss
</code></pre>
<p>Now I want to know how I can make a list of criterions and add them for backpropagation.<br>
I tried these, but they don;t work</p>
<pre><code>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()
  output= model(data)
  final = output[-1,:,:]
  loss = 0.0
  for b in range(batch_size):
    loss += criterion(final[b], targets[b])
  return loss
</code></pre>
<p>I doesn;t return errors, but the network simply won;t train.</p>
<p>This alternative results an error</p>
<pre><code>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()   
  output= model(data)
  final = output[-1,:,:]
  loss = []
  for b in range(batch_size):
    loss.append(criterion(final[b], targets[b]))
  loss = torch.sum(loss)
  return loss
</code></pre>
<p>Note, this is a dummy example. If I understand how to fix this, I can apply that to the recursive neural nets.</p>
<p>For the final project, the sequence of losses have arbitrary length. For example, sometimes it’s adding 4 losses, other times 6. So it’s not an option to pre-allocate a Tensor and save the losses to the Tensor</p>

apaszke: <p>Yeah, you can optimize a variable number of losses without a problem. I’ve had an application like that, and I just used <code>total_loss = sum(losses)</code> where <code>losses</code> is a list of all costs. Calling <code>.backward()</code> on that should do it. Note that you can’t expect <code>torch.sum</code> to work with lists - it’s a method for Tensors. As I pointed out above you can use <code>sum</code> Python builtin (it will just call the <code>+</code> operator on all the elements, effectively adding up all the losses into a single one).</p>
<p>Iterating over a batch and summing up the losses should work too, but is unnecessary, since the criterions already support batched inputs. It probably doesn’t work for you, because criterions <strong>average the loss</strong> over the batch, instead of summing them (this can be changed using the <code>size_average</code> constructor argument). So if you have very large batches, your gradients will get multiplied by the batch size, and will likely blow up your network. Just divide the <code>loss</code> by the batch size and it should be fine.</p>

robromijnders: <p>Thank you. That helped a lot.</p>
<p>I know I can batch this criterion. But iI’m doing it to understand how PyTorch works.<br>
And this explained a lot to me. So thank you very much</p>

meetshah1995: <p><a class="mention" href="/u/apaszke">@apaszke</a></p>
<p>Is it possible to add losses of multiple types provided by <code>torch.nn</code>?</p>
<p>For eg. I have a network with two outputs and corresponding losses - crossentropy2d and crossentropy. I tried the following but it throws an error about incompatible types. Is there a way around?</p>
<pre><code class="lang-auto">b = cross_entropy2d(output_x, x_labels)
a = nn.CrossEntropyLoss(output_y, y_labels)
loss = a + b
</code></pre>
<p>Additional question - Is it possible to weigh the losses ie. <code>total_loss = a + alpha * b</code> ?</p>

jekbradbury: <p>You should be using <code>F.cross_entropy_loss</code>, not the <code>nn.Module</code> form (or you can write your second line as <code>a = nn.CrossEntropyLoss()(output_y, y_labels)</code>). Right now you’re just creating a module, not actually computing the loss.</p>

meetshah1995: <p><a class="mention" href="/u/jekbradbury">@jekbradbury</a></p>
<p>Thanks a lot.</p>

Bixqu: <p>still the question remains, can one do what <a class="mention" href="/u/meetshah1995">@meetshah1995</a> asked with 2 different loss criteria?</p>
<pre><code>b = nn.MSELoss(output_x, x_labels)
a = nn.CrossEntropyLoss(output_y, y_labels)
loss = a + b</code></pre>

Jordan_Campbell: <p>Doing that is fine, it would be:</p>
<pre><code>b = nn.MSELoss()(output_x, x_labels)
a = nn.CrossEntropyLoss()(output_y, y_labels)
loss = a + b

loss.backward()
</code></pre>
<p>Note the additional parentheses, as James mentioned above.</p>
<p>This is equivalent to:</p>
<pre><code>b = nn.MSELoss()
a = nn.CrossEntropyLoss()

loss_a = a(output_x, x_labels)
loss_b = b(output_y, y_labels)

loss = loss_a + loss_b

loss.backward()</code></pre>

D-X-Y: <p>Hi, I have another question.<br>
I want to sum multiple losses, which are all the same loss functions.</p>
<hr>
<h2>1:<br>
criterion = nn.MSELoss().cuda()<br>
loss = criterion(a, label) + criterion(c, label)<br>
2:<br>
criterion1, criterion2 = nn.MSELoss().cuda(), nn.MSELoss().cuda()<br>
loss = criterion1(a, label) + criterion2(c, label)</h2>
<p>which way should I take? Thanks.</p>

smth: <p>both give you same result. I’d say (1) is simpler.</p>

BikashgG: <p>I did this but got the following error :</p>
<p>add received an invalid combination of arguments - got (torch.FloatTensor), but expected one of:</p>
<ul>
<li>(float value)<br>
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)</li>
<li>(torch.cuda.FloatTensor other)<br>
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)</li>
<li>(torch.cuda.sparse.FloatTensor other)<br>
didn’t match because some of the arguments have invalid types: (torch.FloatTensor)</li>
<li>(float value, torch.cuda.FloatTensor other)</li>
<li>(float value, torch.cuda.sparse.FloatTensor other)</li>
</ul>
<p>Both my loss are of the same type - nn.CrossEntropyLoss() &lt;class ‘torch.autograd.variable.Variable’&gt;</p>
<p>Any suggestion on how to resolve this please?</p>

tom: <p>If you stare down the error message, you see that you have one cuda and one not.</p>
<p>Best regards</p>
<p>Thomas</p>

BikashgG: <p>Yes, thank you <a class="mention" href="/u/tom">@tom</a>. I am very novice here.</p>

platero: <p>Is it possible to weight the losses when multiple losses are used and how? Is it correct by using the following code?</p>
<pre><code class="lang-python">    mse_loss = nn.MSELoss(size_average=True)
    a = weight1 * mse_loss(inp, target1)
    b = weight2 * mse_loss(inp, target2)
    loss = a + b

    loss.backward()
</code></pre>

achaiah: <p>Yup, you can certainly weigh the losses however you see fit. What you’re doing in your example should be fine.</p>

ElleryL: <p>what if my second loss function requires some computed value from first loss (or even the grad of first loss?) in that case I can’t add two loss together; they must be gradients respectively; and retain_graph=True gives wrong results as well as the intermediate grads not correct</p>
<p>see this one</p><aside class="quote quote-modified" data-post="1" data-topic="18416">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elleryl/48/11861_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/backward-w-r-t-to-different-parameters-with-different-loss-function/18416">Backward() w.r.t. to different parameters with different loss function</a> 
  </div>
  <blockquote>
    Consider 
I have Variable x 

y = f(x) 
z = Q(y) # Q here is a neural net 

Step(1): gradient w.r.t. x. 

z.backward(retain_graph = True) 
x.grad 
g = x.grad.clone() 
x.grad.data.zero_() 

Step(2): have another function that take the gradients we just compute 

L(g) 

I want to take gradient of it w.r.t. to the weights of neural net Q 
as following 

var_opt = torch.optim.Adam(Q.parameters(), lr=lr)
while not converge:
    var_opt.zero_grad()
    variance_loss = torch.mean(L(g))
    variance_los…
  </blockquote>
</aside>


PeterXiaoGuo: <p>thank you very much Jordan! your answer help me to combine my custom loss function with nn.Loss.</p>

PeterXiaoGuo: <p>Hi BikashgG, I think you could use .type() to change tensor’s type from torch.FloatTensor to torch.LongTensor.</p>
<p>I face a similar problem when I use this CrossEntropyLoss().</p>
<p>Check the official document on this loss function, there is a requirement on the type of feed-in tensor.</p>
<p>Hope it helps,</p>
<p>Peter</p>

tarun: <p>What if the losses are computed over different parts of the network, say loss1 is for first 3 layers and loss2 is for first 7 layers (incl. the first 3)? Wouldn’t the sum of losses method also backprop loss1 through layers 4-7? Would calling loss1.backward() and loss2.backward() separately be recommended in that case?</p>

sid87: <aside class="quote" data-post="6" data-topic="2489">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wang5566/48/6246_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-implement-multiple-loss-function-at-diffent-layer/2489/6">How to implement multiple loss function at diffent layer</a> 
  </div>
  <blockquote>
    Does torch.autograd.back([l1,l2]) mean this two loss backward for separate node? such as softmaxloss updates fc2 and layers before fc2, custom_loss updates fc1 and layers before fc1? 
And if I use L = l1 + l2, L.backward(), 
is L will update fc2 and fc1 separately?
  </blockquote>
</aside>

<p>Hope it helps!</p>