Clarification on WeightedRandomSampler

Pyroka: <p>Hi,</p>
<p>so i have an unbalanced dataset of 2 classes. I searched for solutions and came across the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler" rel="noopener nofollow ugc">WeightedRandomSampler</a> which many people seem to use.</p>
<p>However i do not fully understand what it is used for: As far as i understand the WeightedRandomSampler makes sure (if correctly used), that in each Batch there is approximatly the same amount of samples for each Class.</p>
<p>Lets assume i have 80 samples of class “0” and 20 samples of class “1”.<br>
I now use 10 Batches of size 10 and want to use the WeightedRandomSampler. That means i use ~50 samples from class “0” and class “1” meaning 30 of those (from class “1”) are duplicates, which leads to biased learning.</p>
<p>Now i can set “replacement” to false, But then im stuck with 20 Samples of class “1”. Meaning after 4 Batches (with each having 5 samples from class 1), the remaining Batches have no samples of class “1” at all. So im not sure how that is supposed to help at all.</p>
<p>The only sense i can make out of this feature is, to combine data augmentation and WeightedRandomSampler, to only augment data if it’s from class “1”, via random transforms. But that also means it is possible to have the same exact image of class “1” multiple times in my training leading to an at least slightly biased learning.</p>
<p>Am i missing something? Or do i missunderstand the functionality of the WeightedRandomSampler ?</p>

ptrblck: <aside class="quote no-group" data-username="Pyroka" data-post="1" data-topic="108508">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pyroka/48/32391_2.png" class="avatar"> Pyroka:</div>
<blockquote>
<p>As far as i understand the WeightedRandomSampler makes sure (if correctly used), that in each Batch there is approximatly the same amount of samples for each Class.</p>
</blockquote>
</aside>
<p>That’s one use case, but I wouldn’t say that it’s the only correct one. As the name suggests this sampler is used for weighting the sampling strategy, not only balancing it.</p>
<aside class="quote no-group quote-modified" data-username="Pyroka" data-post="1" data-topic="108508">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pyroka/48/32391_2.png" class="avatar"> Pyroka:</div>
<blockquote>
<p>Now i can set “replacement” to false, […]</p>
</blockquote>
</aside>
<p>Yes, your description is correct. If you want to have balanced batches, you should not use <code>replacement=False</code>.</p>
<aside class="quote no-group" data-username="Pyroka" data-post="1" data-topic="108508">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pyroka/48/32391_2.png" class="avatar"> Pyroka:</div>
<blockquote>
<p>But that also means it is possible to have the same exact image of class “1” multiple times in my training leading to an at least slightly biased learning.</p>
</blockquote>
</aside>
<p>Yes, that might be the case, but could still perform better than the imbalanced dataset.<br>
You would have to apply the sampler on your use case and compare the model performance metric to see how e.g. the confusion matrix changes and what the acceptable result would be.</p>

thesekyi: <p>I have the same dilema. In my case, I will like to combine the <strong>WeightedRandomSampler</strong> with random data augmentation for the minority class. Is there any example of how to achieve this? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>In the <code>__getitem__</code> of your <code>Dataset</code>, you could check the current <code>target</code> value and apply (additional) transformations to the data for the minority classes.</p>