Pytorch custom criterion depending on target

norhther: <p>Im doing a research project where I want to create a custom loss function depending on the targets. I.e. I want to penalize with <code>BCEWithLogitsLoss</code> plus adding a hyperparameter <code>lambda</code>. I only want to add this hyperparameter if the model is not correctly detecting a class.</p>
<p>With more detail, I have a pretrained model that I want to retrain freezing some of the layers. This model detects faces in images with some probability. I want to penalize certain kind of images if they are incorrectly classified with a factor lambda (suppose that the images that need that penalization have a special character in the name or so)</p>
<p>From the source code of <code>pytorch</code>:</p>
<pre><code>import torch.nn.modules.loss as l

class CustomBCEWithLogitsLoss(l._Loss):
    def __init__(self, weight: Optional[Tensor] = None, size_average=None, reduce=None, reduction: str = 'mean',
                 pos_weight: Optional[Tensor] = None) -&gt; None:
        super(BCEWithLogitsLoss, self).__init__(size_average, reduce, reduction)
        self.register_buffer('weight', weight)
        self.register_buffer('pos_weight', pos_weight)
        self.weight: Optional[Tensor]
        self.pos_weight: Optional[Tensor]

    def forward(self, input: Tensor, target: Tensor) -&gt; Tensor:
        return F.binary_cross_entropy_with_logits(input, target,
                                                  self.weight,
                                                  pos_weight=self.pos_weight,
                                                  reduction=self.reduction)
</code></pre>
<p>Here, forward has two tensors as inputs, so I dont know how to add here the class of the images that I want to penalize with lambda. Adding lambda to the constructor is ok, but how to do the forward pass if it only allows tensors?</p>
<p>To clarify the question, Suppose that I have a training/testing folder with the images. The files with the character <code>@</code> in the filename are the ones that I want to classify correctly way more than the files without the character, with a factor <code>lambda</code>.</p>
<p>How can I tell in the regular fashion of <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="noopener nofollow ugc">training a model in pytorch</a>, that those files have to use a <code>lambda</code> penalization (let’s say that the loss function is lambda * BCEWithLogitLoss) but the other ones not? I’m using <code>DataLoader</code>.</p>

ptrblck: <aside class="quote no-group" data-username="norhther" data-post="1" data-topic="153714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/norhther/48/16747_2.png" class="avatar"> norhther:</div>
<blockquote>
<p>let’s say that the loss function is lambda * BCEWithLogitLoss</p>
</blockquote>
</aside>
<p>You could most likely scale the unreduced loss and add the reduction afterwards.<br>
To avoid the default <code>"mean"</code> reduction, use <code>nn.BCEWithLogitsLoss(reduction="none")</code> which will output the loss values for each sample. Scale the loss with your <code>lambda</code> weigths afterwards, and finally reduce the loss. You might also want to normalize the weight with the used weights to avoid creating a different loss ranges which would depend on the current class distribution.</p>

norhther: <p>I see. Could you post a small snippet?<br>
Also, how would you access each sample in the batch with a data loader?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/does-weightedrandomsampler-increase-interations/51514/16">Here</a> is an example using <code>nn.CrossEntropyLoss</code> showing how the loss weighting is applied on an unreduced loss and normalized afterwards.</p>
<aside class="quote no-group" data-username="norhther" data-post="3" data-topic="153714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/norhther/48/16747_2.png" class="avatar"> norhther:</div>
<blockquote>
<p>how would you access each sample in the batch with a data loader?</p>
</blockquote>
</aside>
<p>The <code>DataLoader</code> returns a batch containing samples in <code>dim0</code> (the batch dimension) so you can directly access them.</p>