Is it okay to match the ratio of labels in the mini-batch from the extremely imbalanced data?

songsong0425: <p>Hi, thank you for your effort in establishing PyTorch.<br>
I have a general question about the sampling strategy in the <code>DataLoader</code>.</p>
<p>When I have a very imbalanced dataset with binary classes for the link prediction task, the basic sampler returns the randomly sampled cases while preserving the original ratio of classes.<br>
(In detail, I have 138,015 edges with label 0 and 8,791 with label 1)<br>
Before making the model training, I checked the ratio of labels in the mini-batch and found that most labels were set to 0. (i.e., [0,0,0,0,0,0,0,0] when batch_size=8)</p>
<p>Although I checked that there are several ways to solve this problem (e.g., <code>WeightedRandomSampler</code> in <code>torch.utils.data</code>, <code>RandomUnderSampler</code> in <code>imblearn.under_sampling</code>, …), I considered matching the ratio of pos(1) and neg(0) labels in each batch by randomly choosing the samples from the major class (i.e., [1,1,1,1,0,0,0,0] when batch_size=8).<br>
But is it the right way to overcome the data imbalance?<br>
As the general meaning of <code>negative sampling</code> in the graph is <code>sampling the edges from the unconnected edges</code>, I’m not sure how to deal with the minor class: is it okay to repeat it during iteration?</p>
<p>Thank you for reading my question.</p>

KFrank: <p>Hi Songyeon!</p>
<aside class="quote no-group" data-username="songsong0425" data-post="1" data-topic="198872" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/songsong0425/48/67200_2.png" class="avatar"> songsong0425:</div>
<blockquote>
<p>I considered matching the ratio of pos(1) and neg(0) labels in each batch by randomly choosing the samples from the major class (i.e., [1,1,1,1,0,0,0,0] when batch_size=8).<br>
But is it the right way to overcome the data imbalance?</p>
</blockquote>
</aside>
<p>Yes, forcing each batch to contain the same number of negative samples as<br>
positive samples is a perfectly legitimate way to sample data when you have<br>
unbalanced binary classes.</p>
<p>(You could also reweight the samples in your loss function, for example,<br>
by using <code>BCEWithLogitsLoss</code>’s <code>pos_weight</code> constructor argument.  It is,<br>
however, my theoretical preference to sample the minority class more<br>
heavily, as you propose doing, rather than reweight the samples in the<br>
loss function, unless the number of minority samples in your training set is<br>
so small that batches would often contain duplicates of the same sample.)</p>
<p>It’s worth noting that if you use something like <code>WeightedRandomSampler</code><br>
so that your batches of eight samples contain <em>on average</em> four positive and<br>
four negative samples, it is unlikely that a given batch would contain no<br>
positive or no negative samples and it is rather unlikely that a given batch<br>
would contain only one sample from a class.  So you would still get batches<br>
that are usually reasonably well balanced even if you don’t force each batch<br>
to have exactly four positive and four negative samples.</p>
<p>Best.</p>
<p>K. Frank</p>

songsong0425: <p>Thank you for your kind and prompt reply!<br>
It’s very helpful to my task. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>