Does Pytorch have an under sampler?

username: skyunyoo
cooked: <p>I am dealing with extremely imbalance data (neg 99%, pos 1%).</p>
<p>I used a <code>WeightedRandomSampler</code> to solve this problem.<br>
Below is the WRS code I used.</p>
<pre><code class="lang-auto">        neg=0.0
        pos=0.0
        targets = []
        for _, target in traindataset:
            targets.append(target.max())
            if target.max()==1:
                pos+=1
            else:
                neg+=1
        weights = neg/pos
        pos_weight = torch.tensor(weights)
        
        targets = torch.stack(targets).long()
        class_sample_count = [neg,pos]
        weights = 1/torch.tensor(class_sample_count, dtype=torch.float)
        samples_weights = [weights[t] for t in targets]
        sampler = torch.utils.data.sampler.WeightedRandomSampler(weights=samples_weights, num_samples=len(samples_weights), replacement=True)
        data_loader = torch.utils.data.DataLoader(traindataset, batch_size = batch_size, shuffle=False,sampler=sampler, num_workers=0, pin_memory=False)
</code></pre>
<p>When using WRS, the train accuracy exceeded 90% in 4 epochs, but the valid accuracy did not exceed 10%.</p>
<p>In order to know the cause, I checked the train image with WRS applied in each batch, and confirmed that the duplicate of pos data was serious because it was extremely imbalanced.</p>
<p>Because of this overfitting, I wonder if there is a smapler that undersampling neg data, leaving pos data intact.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="skyunyoo" data-post="1" data-topic="88950">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>In order to know the cause, I checked the train image with WRS applied in each batch, and confirmed that the duplicate of pos data was serious because it was extremely imbalanced.</p>
</blockquote>
</aside>
<p>Was the batch even imbalanced after using the <code>WeightedRandomSampler</code> or do I misunderstand the sentence?</p>
<aside class="quote no-group" data-username="skyunyoo" data-post="1" data-topic="88950">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/2acd7d/48.png" class="avatar"> skyunyoo:</div>
<blockquote>
<p>Because of this overfitting, I wonder if there is a smapler that undersampling neg data, leaving pos data intact.</p>
</blockquote>
</aside>
<p>Under- or oversampling can be done by changing the weights for each sample.<br>
E.g. if you are using <code>1/class_count_X</code> for classX, you would try to balance the batch.<br>
However, you could of course scale these weights to force some undersampling (and reducing the <code>num_samples</code> argument).</p>

username: skyunyoo
cooked: <ol>
<li>
</li></ol>
<p>It meant that a lot of duplicates were made because there was too little data to oversampling.<br>
With WRS, the class imbalance problem was solved well!</p>
<ol start="2">
<li>
</li></ol>
<pre><code class="lang-auto">num_sample
</code></pre>
<p>I was able to undersampling as much as I wanted through this part!</p>
<p>Thanks a lot!</p>