Extremely Imbalanced Data

username: mmf
cooked: <p>Hi, everyone. I’m struggling with a very challenging issue. I have a binary database of 500K images. Of those 500K, only 10K are from one class and the rest are from a different class. I can’t use synthetic data.</p>
<p>I’m using a weighted loss, but that doesn’t change the fact that I’ll have more than 7800 steps in an epoch, considering batch=64.  I was wondering if it makes sense to combine a Weighted Loss + sampler setting a fixed number of samples, or even Weighted loss + Weighted sampler.<br>
Does that make sense? Is it conceptually correct?</p>
<p>The augmentations and loss I’m using now:</p>
<pre><code class="lang-auto">        aug = (
            [
                T.RandomHorizontalFlip(),
                T.RandomVerticalFlip(),
                T.RandomResizedCrop(img_size, scale=(0.75, 1.0)),
                T.RandomRotation(45),
                T.ColorJitter(hue=0.2),
            ]

</code></pre>
<pre><code class="lang-auto">#Loss
class_weights = class_weights.to(device)
self.loss_fn = lambda logits, labels: F.cross_entropy(logits, labels, weight=class_weights)
</code></pre>

username: Arunprakash-A
cooked: <p>If you are working on a binary classification problem, having 10K images per class is generally sufficient for training. A straightforward approach is to balance the dataset, since you already have enough samples for both classes.</p>
<p>You can achieve this by randomly sampling 10K images from the remaining 490K images of the other classes. This will result in a balanced dataset with a total of 20K images.</p>

username: mmf
cooked: <p>Thanks for your reply. But it seems that the model isn’t learning anything. 490k samples from one class vs 10k from another class is too much imbalance.<br>
the results show a super overfited model</p>

username: mmf
cooked: <p>Or are you saying I should only do the augmentations in the minority class?</p>

username: Arunprakash-A
cooked: <p>Assuming you have 10K images for class-0 and 490K images for class-1, I suggest selecting 10K samples from the 490K available for class-1. This will give you a <strong>balanced dataset</strong> with 10K samples for each class.</p>
<p>You can then train your model on this new, balanced dataset. You won’t need to use the <code>class_weights</code> as the dataset is <strong>balanced.</strong></p>

username: mmf
cooked: <p>Thats a good idea, thanks. But I have another dataset with 0 - 100899 and 1 - 54.</p>
<p>What do I do in that case? The first one I agree with you, I think its a good approach but the other one, the version with 54 images is killing me.</p>
<p>I cannot combine this set with others. ):</p>

username: Arunprakash-A
cooked: <aside class="quote no-group" data-username="mmf" data-post="6" data-topic="223065">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/8e8cbc/48.png" class="avatar"> mmf:</div>
<blockquote>
<p>I have another dataset with 0 - 100899 and 1 - 54.</p>
</blockquote>
</aside>
<p>There is no mention of the domain you are working in. You might consider using pre-trained models (typically trained on ImageNet-1K), which require far fewer samples (often, even 100 can be sufficient) for convergence. You can also augment class-1 to increase the number of samples to at least 100.</p>

username: mmf
cooked: <p>I am working with medical images, with different devices and diagnoses. And according to the rules of the problem, I cannot mix them</p>