How to combine multiple criterions to a loss function?

robromijnders: <SCODE>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()
  output= model(data)
  final = output[-1,:,:]
  loss = criterion(final,targets)
  return loss
<ECODE> <SCODE>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()
  output= model(data)
  final = output[-1,:,:]
  loss = 0.0
  for b in range(batch_size):
    loss += criterion(final[b], targets[b])
  return loss
<ECODE> I doesn;t return errors, but the network simply won;t train. This alternative results an error <SCODE>def loss_calc(data,targets):
  data = Variable(torch.FloatTensor(data)).cuda()
  targets = Variable(torch.LongTensor(targets)).cuda()   
  output= model(data)
  final = output[-1,:,:]
  loss = []
  for b in range(batch_size):
    loss.append(criterion(final[b], targets[b]))
  loss = torch.sum(loss)
  return loss
<ECODE> Note, this is a dummy example. If I understand how to fix this, I can apply that to the recursive neural nets. For the final project, the sequence of losses have arbitrary length. For example, sometimes it’s adding 4 losses, other times 6. So it’s not an option to pre-allocate a Tensor and save the losses to the Tensor

apaszke: 

robromijnders: Thank you. That helped a lot.

meetshah1995: @apaszke For eg. I have a network with two outputs and corresponding losses - crossentropy2d and crossentropy. I tried the following but it throws an error about incompatible types. Is there a way around? <SCODE>b = cross_entropy2d(output_x, x_labels)
a = nn.CrossEntropyLoss(output_y, y_labels)
loss = a + b
<ECODE>

jekbradbury: 

meetshah1995: @jekbradbury Thanks a lot.

Bixqu: <SCODE>b = nn.MSELoss(output_x, x_labels)
a = nn.CrossEntropyLoss(output_y, y_labels)
loss = a + b<ECODE>

Jordan_Campbell: Doing that is fine, it would be: <SCODE>b = nn.MSELoss()(output_x, x_labels)
a = nn.CrossEntropyLoss()(output_y, y_labels)
loss = a + b

loss.backward()
<ECODE> Note the additional parentheses, as James mentioned above. This is equivalent to: <SCODE>b = nn.MSELoss()
a = nn.CrossEntropyLoss()

loss_a = a(output_x, x_labels)
loss_b = b(output_y, y_labels)

loss = loss_a + loss_b

loss.backward()<ECODE>

D-X-Y: which way should I take? Thanks.

smth: both give you same result. I’d say (1) is simpler.

BikashgG: I did this but got the following error : add received an invalid combination of arguments - got (torch.FloatTensor), but expected one of: (float value)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor) (torch.cuda.FloatTensor other)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor) (torch.cuda.sparse.FloatTensor other)
didn’t match because some of the arguments have invalid types: (torch.FloatTensor) (float value, torch.cuda.FloatTensor other) (float value, torch.cuda.sparse.FloatTensor other) Both my loss are of the same type - nn.CrossEntropyLoss() <class ‘torch.autograd.variable.Variable’> Any suggestion on how to resolve this please?

tom: If you stare down the error message, you see that you have one cuda and one not. Best regards Thomas

BikashgG: 

platero: Is it possible to weight the losses when multiple losses are used and how? Is it correct by using the following code? <SCODE>    mse_loss = nn.MSELoss(size_average=True)
    a = weight1 * mse_loss(inp, target1)
    b = weight2 * mse_loss(inp, target2)
    loss = a + b

    loss.backward()
<ECODE>

achaiah: Yup, you can certainly weigh the losses however you see fit. What you’re doing in your example should be fine.

ElleryL: what if my second loss function requires some computed value from first loss (or even the grad of first loss?) in that case I can’t add two loss together; they must be gradients respectively; and retain_graph=True gives wrong results as well as the intermediate grads not correct

PeterXiaoGuo: thank you very much Jordan! your answer help me to combine my custom loss function with nn.Loss.

PeterXiaoGuo: Hi BikashgG, I think you could use .type() to change tensor’s type from torch.FloatTensor to torch.LongTensor. I face a similar problem when I use this CrossEntropyLoss(). Check the official document on this loss function, there is a requirement on the type of feed-in tensor. Hope it helps, Peter

tarun: What if the losses are computed over different parts of the network, say loss1 is for first 3 layers and loss2 is for first 7 layers (incl. the first 3)? Wouldn’t the sum of losses method also backprop loss1 through layers 4-7? Would calling loss1.backward() and loss2.backward() separately be recommended in that case?

sid87: Hope it helps!