Custom loss functions

Jean_Da_Rolt: <p>Hi,</p>
<p>I’m implementing a custom loss function in Pytorch 0.4. Reading the docs and the forums, it seems that there are two ways to define a custom loss function:</p>
<ul>
<li>Extending Function and implementing forward and backward methods.</li>
<li>Extending Module and implementing only the forward method.</li>
</ul>
<p>With that in mind, my questions are:</p>
<ul>
<li>Can I write a python function that takes my model outputs as inputs and use torch.* functions to compute my loss function (without extending Function or Module)? If not, why?</li>
</ul>
<p>Regards,</p>

ptrblck: <p>Sure, as long as you use PyTorch operations, you should be fine.<br>
Here is a dummy implementation of <code>nn.MSELoss</code> using the mean:</p>
<pre><code class="lang-python">def my_loss(output, target):
    loss = torch.mean((output - target)**2)
    return loss

model = nn.Linear(2, 2)
x = torch.randn(1, 2)
target = torch.randn(1, 2)
output = model(x)
loss = my_loss(output, target)
loss.backward()
print(model.weight.grad)
</code></pre>

YingLaiLin: <p>This is a quite simple implementation of custom loss functions while there are not extra parameters. Could you please share some solutions to fix this problem?<br>
For example, in keras, you can implement weighted loss by following:</p>
<blockquote>
<p>def label_depend_loss(alpha):<br>
&nbsp;&nbsp;&nbsp;&nbsp; def label_depend(output, target):<br>
&nbsp;&nbsp;&nbsp;&nbsp; pos_loss = something<br>
&nbsp;&nbsp;&nbsp;&nbsp; neg_loss = something<br>
&nbsp;&nbsp;&nbsp;&nbsp;  return pos_loss  + alpha * neg_loss</p>
</blockquote>
<p>Besides, BCELoss may doesn’t suit this case.</p>

ptrblck: <p>Depending on your loss function, you could just multiply the positive and negative losses with your weights.<br>
Maybe <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="nofollow noopener">nn.BCEWithLogitsLoss</a> might fit your use case providing <code>pos_weight</code>.</p>

YingLaiLin: <p>Yes, Thanks for you advise. I can split it into two kinds of loss function and just sum it up weightly. Much thanks!</p>

MariosOreo: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Could you further explain the <code>weight</code> in <code>nn.CrossentropyLoss()</code> and <code>nn.BCELoss</code>, <code>pos_weight</code> in <code>nn.BCEWithLogitsLoss()</code>?</p>
<ul>
<li>
<code>weight</code> in CrossentropyLoss is a Tensor of size <code>C</code>, but why does it should have the size of <code>nbatch</code> in <code>nn.BCELoss()</code>? And it seems that <code>weight</code> in BCELoss does not work for unbalanced data, right? ( because the weight is related to nbatch )</li>
<li>Does <code>pos_weight</code> has the same effect with <code>weight</code> in <code>nn.CrossentropyLoss</code>?</li>
</ul>
<p>Thanks in advance.</p>

ptrblck: <p>The <code>weight</code> argument in <code>nn.BCE(WithLogits)Loss</code> has the shape of the input batch, since the loss functions take floating point targets, which does not correspond to a class weighting schema. <code>pos_weight</code> on the other side is closer to a class weighting, as it only weights the positive examples. Furthermore, you can balance the recall and precision changing the <code>pos_weight</code> argument.</p>

msrdinesh: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> can you share, a similar dummy function to cross entropy loss. It would be helpful to me. Thanks in Advance!</p>

ptrblck: <p>Sure, here is the simple version without weighting, different reduction types etc:</p>
<pre><code class="lang-python">def my_cross_entropy(x, y):
    log_prob = -1.0 * F.log_softmax(x, 1)
    loss = log_prob.gather(1, y.unsqueeze(1))
    loss = loss.mean()
    return loss


criterion = nn.CrossEntropyLoss()

batch_size = 5
nb_classes = 10
x = torch.randn(batch_size, nb_classes, requires_grad=True)
y = torch.randint(0, nb_classes, (batch_size,))

loss_reference = criterion(x, y)
loss = my_cross_entropy(x, y)

print(loss_reference - loss)
&gt; tensor(0., grad_fn=&lt;SubBackward0&gt;)
</code></pre>

netaglazer: <p>but the loss inputs are tensors, how does it work?</p>

ptrblck: <p>Could you add some more details to your question please?<br>
What is confusing about input tensors to a loss function? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

netaglazer: <p>i mean, for each batch the input of the loss function is a list of all predictions and labels in the current batch, and the loss is built for input of only one prediction and label</p>
<p>so how it should be implement?</p>
<p>and another thing - how the backward() of costume function should be implemented?</p>

netaglazer: <p>i mean, for each batch the input of the loss function is a list of all predictions and labels in the current batch, and the loss is built for input of only one prediction and label</p>
<p>so how it should be implement?</p>
<p>and another thing - how the backward() of costume function should be implemented?</p>

ptrblck: <p>I think you could index your output and target at the desired location and pass it to your criterion:</p>
<pre><code class="lang-python">model = models.resnet18()
output = model(torch.randn(10, 3, 224, 224))
target = torch.randint(0, 1000, (10,))

criterion = nn.CrossEntropyLoss()

loss = criterion(output[0:1], target[0:1])
loss.backward()
</code></pre>

ashj: <p><a class="mention" href="/u/netaglazer">@netaglazer</a><br>
I believe if you are worried about the first dimension being the Batch index, pytorch automatically extracts the individual predictions and accumulated the loss as batch loss. So, you can write your loss function assuming your batch has only one sample.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> could you please correct me if my understanding about loss function above is wrong?</p>

Rachel_Copperman: <p>Hi, Could you please help with some code I have?</p>

ptrblck: <p>Feel free to create a new topic with your problem description as well as your code, so that we can have a look. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Rachel_Copperman: <p>opened</p>
<aside class="quote quote-modified" data-post="1" data-topic="70900">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rachel_copperman/48/12810_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/loss-function-with-small-amount-of-positives/70900">Loss function with small amount of positives</a> 
  </div>
  <blockquote>
    Hello, 
I have a binary classification model. My data is having 1 million samples, and only 100 of the are tagged with 1 (rest tagged with 0). 
My model now is a simple fully-connected. 
I’m having some problems with finding a loss function that will be able to overcome the small amount of true positives. 
I saw BCEWithLogitsLoss, but had some problems with it, so I used the next loss function I run into (which actually makes the same, I think) : 
def weighted_binary_cross_entropy(output, target…
  </blockquote>
</aside>


lycaenidae: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="29387">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>loss = my_cross_entropy(x, y)</p>
</blockquote>
</aside>
<p>Do we need to implement forward pass for my_cross_entropy function?</p>

ptrblck: <p><code>my_cross_entropy</code> is implemented as a simple function so you can just call it.<br>
You could of course wrap it in an <code>nn.Module</code> and put the operations in the <code>forward</code> method, if that’s more convenient or if you need to store some internal states.</p>