Weighted Cross Entropy for each sample in batch

Uwais_Iqbal: <p>I’m working on a problem that requires cross entropy loss in the form of a reconstruction loss. I would like to weight the loss for each sample in the mini-batch differently. The current API for cross entropy loss only allows weights of shape <code>C</code>. I would like to pass in a weight matrix of shape <code>batch_size , C</code> so that each sample is weighted differently.</p>
<p>Is there a way to do this? The only workaround I can think of is to evaluate the loss for each sample in the mini-batch and pass in a new set of weights each time.</p>

Henry_Chibueze: <p>Sorry I don’t really understand your question.<br>
Are you saying that for a batch N, you want to get the loss between each data point in N and it’s corresponding prediction?</p>

Uwais_Iqbal: <p>I would like to provide custom weights for each sample that I evaluate the loss for in a batch.</p>
<p>At the moment I can do:</p>
<pre><code class="lang-python">F.cross_entropy(y_pred, y_true, weight=weight)
</code></pre>
<p>and this will use a single set of weights for the entire batch.</p>
<p>What I would like to do is pass in a set of weights for each sample in the batch</p>

Henry_Chibueze: <p>A set of weights for each sample?</p>
<p>If you are giving a set of weights for each sample in a batch, then what is the point of generalizing?</p>
<p>I mean you could always just set your batch size to one but I doubt the model will perform well at inference.</p>

Uwais_Iqbal: <p>I’m using a denoising autoencoder and wanted to use a weighted loss function to emphasise the loss for the corrupted components of the data. A batch size of 1 could work but the model would perform poorly.</p>
<p>The only solution I can think of is to do something like:</p>
<pre><code class="lang-python">loss = []
for i, element in enumerate(batch):
    # calculate weights
    l = F.cross_entropy(y_pred[i], y_true[i], weight=weights[i])
    loss.append(l)
</code></pre>

Henry_Chibueze: <p>If each sample had its own weight, then ur model won’t be able to generalize properly on data that wasn’t part of the training data.</p>
<p>It’s like having a data points in a graph that make up a straight line. You would want to get the slope (m) of the straight line after making the plot of the entire data points, rather than finding m for each and every single point.</p>

Henry_Chibueze: <p>Well this could work.<br>
Have you tried it ?</p>
<p>but the weights[ i ] might give an error because the weights are meant to be common for each point</p>

Alexey_Demyanchuk: <p>Hi. You can probably do with reduction ‘none’. Something like this should probably work.<br>
<code>criterion = CrossEntropyLoss(reduction='none')</code><br>
<code>loss = criterion(network_out, label) --&gt; shape batch_size x 1</code><br>
<code>loss = loss * batch_weights</code><br>
<code>loss = torcn.mean(loss)</code></p>

Uwais_Iqbal: <p>That seems like a much more elegant solution!</p>

Oli: <p>Did you see improvements doing this? I don’t… Made this thread - <a href="https://discuss.pytorch.org/t/clip-limit-the-loss-for-outlier-samples-in-a-batch/125825" class="inline-onebox">Clip/limit the loss for outlier samples in a batch</a></p>