Dealing with imbalanced datasets in pytorch

Skinish: <p>I am trying to find a way to deal with imbalanced data in pytorch. I was used to Keras’ <code>class_weight</code>, although I am not sure what it really did (I think it was a matter of penalizing more or less certain classes).</p>
<p>The only solution that I find in pytorch is by using <code>WeightedRandomSampler</code>with <code>DataLoader</code>, that is simply a way to take more or less the same number of samples per each class (and maybe duplicate the samples of some classes if needed?). However, I am looking for another another alternative like the one provided in Keras that does not involve repeating some samples. Is it possible to do it in pytorch? Thanks in advance.</p>

ptrblck: <p>You can also apply class weighting using the <code>weight</code> argument for a lot of loss functions.<br>
<code>nn.NLLLoss</code> or <code>nn.CrossEntropyLoss</code> both include this argument.<br>
You can find all loss functions <a href="https://pytorch.org/docs/stable/nn.html#loss-functions">here</a>.</p>

Skinish: <p>You are right, did not realize that! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Let me be a bit more specific then. I am building a hierarchical classifier that combines 3 losses (3 levels of classification). It is a multi-label problem and I am using BCELoss at the 3 different levels, then summing all and backpropagating the errors. In this case, should I instantiate 3 different BCELoss and pass them a list containing the weights for the classes corresponding to the hierarchy level?</p>

ptrblck: <p>Could you explain a bit more about how you use the <code>BCELoss</code> at 3 different levels?<br>
Do you assign something like this:</p>
<pre><code class="lang-python">0.0 - class0
0.5 - class1
1.0 - class2
</code></pre>
<p>?<br>
If so, I would recommend to weight the different predictions:</p>
<pre><code class="lang-python">batch_size = 5
nb_classes = 3
output = torch.randn(batch_size, nb_classes)
target = torch.empty(batch_size, nb_classes).random_(2)
weight = torch.tensor([1.0, 2.0, 1.0])

criterion = nn.BCEWithLogitsLoss(reduction='none')
loss = criterion(output, target)
loss = loss * weight
loss = loss.mean()
</code></pre>
<p>Would that work for you?</p>

Skinish: <p>Thank you for helping me out <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"></p>
<p>This is how I update my weights, each <code>target</code> and <code>y_pred</code> are binary vectors.</p>
<pre><code class="lang-auto"> # Compute loss and update parameters for all levels
loss1 = criterion(y_pred1.cuda(), target1)
loss2 = criterion(y_pred2.cuda(), target2)
loss3 = criterion(y_pred3.cuda(), target3)
loss = sum([loss1, loss2, loss3])  # combine all losses
loss.backward()
</code></pre>
<p>My loss is <code>criterion = nn.BCELoss()</code>. My idea was to initialize 3 different loss functions instead (criterion1, criterion2 and criterion3) so that I could pass the weight vector right away. Something like</p>
<pre><code class="lang-auto">criterion1 = nn.BCELoss(weights1)
criterion2 = nn.BCELoss(weights2)
criterion3 = nn.BCELoss(weights3)
</code></pre>

ptrblck: <p>Should also work fine.<br>
One side note: why do you call <code>.cuda()</code> on the predictions?<br>
They should already be on the GPU if your input and model are pushed to a CUDA device.</p>

Skinish: <p>You are right, it makes no difference to call <code>.cuda()</code>.</p>

Skinish: <p>It seems like with weights the loss function needs to be on GPU, is that true?</p>

ptrblck: <p>Yes, <code>weight</code> should be on the same device as your output and target.</p>

Skinish: <p>Thank you. One last question: what should be the rule of thumb for assigning weights to the labels? I am assigning according to the following rule: <code>weight_label_l = max(nr_samples_per_label) / nr_samples_label_l</code></p>

ptrblck: <p>Sounds reasonable. You could probably also try <code>sum(nr_samples_per_label)</code>.<br>
I’m not sure if there is a general rule of thumb as you might want to balance your per-class accuracies manually.</p>

Deeply: <p>Shouldn’t the weights  be like:</p>
<pre><code class="lang-auto">weight_label_i =  nr_samples_of_label_i  / total_number_of_samples

i = 1, 2, ...
</code></pre>
<p>??</p>

ptrblck: <p>This would weight the majority classes higher, while we would like to weight the loss of minority classes higher or am I mistaken?</p>

Deeply: <p>You are right. The BCELoss is given by:<br>
<code>ℓ(x,y)=L={l1,…, ln,....,lN}⊤, ln = −wn[yn⋅logxn+(1−yn)⋅log(1−xn)]</code></p>
<p>in this case we invert the above:<br>
<code>weight_label_i = total_number_of_samples / nr_samples_of_label_i</code></p>

Deeply: <p>The documentation also says:</p>
<p><em>weight (Tensor, optional) – a manual rescaling weight given to <strong>the loss of each batch element</strong>. If given, has to be a Tensor of size “nbatch”</em></p>
<p>So, the weight vector works on the batch level and not the per-class samples level, or, I am missing something here?</p>

ptrblck: <p>Yeah, you are right! Thanks for pointing this out. I tried not to mix up both new threads about weighting and thought we are dealing with some classification loss like <code>nn.NLLLoss</code>.</p>
<p><a class="mention" href="/u/skinish">@Skinish</a> have a look at <a href="https://discuss.pytorch.org/t/about-bcewithlogitslosss-pos-weights/22567/4">this thread</a> to see how to apply <code>pos_weight</code> instead.</p>

Deeply: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="22596">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>BCEWithLogitsLoss</p>
</blockquote>
</aside>
<p>I am not sure of the <code>pos_weight</code> will do, even <code>BCELoss</code> has this <code>pos_weight</code> <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">input argument</a>, and this does not seem to fulfill what is needed (I could be mistaken). The way I see it to resolve this issue in a simple manner is to dot product the weight vector by the output and target/label-vector, during finding the loss of the training, as follows:</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss()
...
...
loss = criterion(W*output.float(), W*target.float())
</code></pre>
<p>For more flexibility and ease, the (balance) weight vector could be generated within the <code>dataset class</code>.</p>

Deeply: <p>I was trying to implement this weight multiplication, not sure if this is the best way to do it as I had to use <code>torch.transpose</code> twice, here it goes:</p>
<p><code># 10 is the batch size, so each sample has a weight value</code></p>
<pre><code class="lang-auto">ipdb&gt;weight.shape
torch.Size([10])  
ipdb&gt;output.shape
torch.Size([10, 644])
ipdb&gt;weight
tensor([ 0.9987,  0.9997,  0.9997,  0.9992,  0.9997,  0.9985,  0.9905,
         0.9911,  0.9476,  0.9944], dtype=torch.float64, device='cuda:0')

ipdb&gt; output = torch.mul(weight, torch.transpose(output, 0, 1) )
ipdb&gt; output = torch.transpose(output, 0,1)
ipdb&gt;output.shape
torch.Size([644, 10])

</code></pre>
<p>NB. I have added the weights to the <code>Dataset class</code></p>
<p>The whole thing will look like this:</p>
<pre><code class="lang-auto">if cf.use_weight_to_balance_data:
       weight = weight.to(device)
       output = torch.mul(weight, torch.transpose(output.double(), 0, 1) )
       output = torch.transpose(output, 0, 1)
       target = torch.mul(weight, torch.transpose(target.double(), 0, 1) )
       target = torch.transpose(target, 0, 1)
</code></pre>

Skinish: <p><a class="mention" href="/u/deeply">@Deeply</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for all your help. Should the weight multiplication be the viable solution? I did not quite understand what would be wrong with passing the weights vector to the loss function (to the <code>pos_weight</code> argument), although the weight assignment that I said should be changed.</p>
<p>What would the effect of passing a vector to <code>weight</code> then?</p>

ptrblck: <p><a class="mention" href="/u/deeply">@Deeply</a> I’m not sure it’s a good idea to multiply the output and target directly with the weight.<br>
Both will pass a criterion and maybe a <code>sigmoid</code>, so I would rather multiply the loss.</p>
<p><a class="mention" href="/u/skinish">@Skinish</a> <a href="https://github.com/pytorch/pytorch/issues/5660">here</a> is a github issue discussing the introduction of <code>pos_weight</code> and a comparison between <code>weight</code> and <code>pos_weight</code>.<br>
As you have different criteria for your targets and predictions, I think using <code>pos_weight</code> would work.<br>
Let me know, if that works for you.</p>