Use Weighted Random Sampler for Imbalanced Class

username: DJ_1992
cooked: <p>Hi,<br>
I am trying to use WeightedRandomSampler in this way</p>
<blockquote>
<p>class_sample_count = [39736,949, 7807]<br>
weights = 1 / torch.Tensor(class_sample_count)<br>
weights = weights.double()<br>
sampler = torch.utils.data.sampler.WeightedRandomSampler(<br>
weights=weights,<br>
num_samples=?,<br>
replacement=False)<br>
dataloaders = {x: torch.utils.data.DataLoader(image_datasets<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>, drop_last=True, sampler = sampler, batch_size=32) for x in [‘train’, ‘valid’]}</p>
</blockquote>
<p>I want the minority class samples atleast once … what num_samples should I use… also am I using it the right way as I am seeing all the samples from only the minority class? Thank You in advance …</p>

username: ptrblck
cooked: <p>The <code>weights</code> tensor should contain a weight for each sample, not the class weights.<br>
Have a look at <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">this post</a> for an example.</p>

username: DJ_1992
cooked: <p>Okay I understand,<br>
Thanks again <a class="mention" href="/u/ptrblck">@ptrblck</a>… Your Sample code in the link helped alot … <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Puneesh_Khanna
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  - I m using weighted random sampler for an imbalanced class problem. I have a doubt regarding usage of <strong>replacement</strong> parameter. Does passing it as False ensure that there are no repetitions of samples within the batch size. All my class samples are more than batch size.</p>

username: ptrblck
cooked: <p><code>replacement=False</code> will not draw the same samples during the entire epoch, so while the first batches might have a balanced class distribution, the latter ones will yield more samples of the majority classes.<br>
Thus, I don’t think using <code>replacement=False</code> is a proper way to balance the data batches.</p>

username: ramazan
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Isn’t it important to state that with <code>replacement=False</code>, the sampler doesn’t consider weights as one might expect?</p>
<p>For example, I have a dataset with 1000 samples: 300 cats and 700 dogs. I assign weights as <code>[1/300, 1/700]</code> and set the dataset size to twice the size of the minority class (300*2=600). Using <code>replacement=False</code>, I would expect to get a balanced subset each epoch containing approximately 300 cats and 300 dogs. However, I don’t actually get a balanced subset.</p>
<p>While I can set <code>replacement=True</code>, it leads to duplicating samples within an epoch, which I want to avoid.</p>

username: ptrblck
cooked: <p>Using <code>replacement=False</code> allows you to undersample the dataset (you would need to stop iterating the dataset in this case) since the weights will still be used, but the sampler will be forced to pick imbalanced samples towards the end of the dataset. Without duplicating samples it will be impossible to create the same number of batches (as the full dataset will yield) in a balanced way.</p>

username: ramazan
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="84280">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>but the sampler will be forced to pick imbalanced samples towards the end of the dataset</p>
</blockquote>
</aside>
<p>After reviewing the source code, I understand why the imbalance occurs at the end of the dataset. This is due to the behavior of <code>torch.multinomial</code>.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="84280">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p> Without duplicating samples </p>
</blockquote>
</aside>
<p>Yes, that’s why I’m setting <code>num_samples=2*num_minor_class</code>. This theoretically allows for sampling balanced classes without duplicates. Initially, I thought this would happen based on the sampler’s <a href="https://pytorch.org/docs/stable/data.html#:~:text=%5BSOURCE%5D-,Samples,-elements%20from%20%5B0" rel="noopener nofollow ugc">documentation</a>.</p>
<p>I recommend adding a note near the <code>weights</code> argument indicating that weights may not be used as expected when <code>replacement=False</code>.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ramazan" data-post="8" data-topic="84280">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/e0b2c6/48.png" class="avatar"> ramazan:</div>
<blockquote>
<p>I recommend adding a note near the <code>weights</code> argument indicating that weights may not be used as expected when <code>replacement=False</code>.</p>
</blockquote>
</aside>
<p>I believe they are used as expected. However, when no replacement is allowed, the used sample cannot be picked anymore. Higher weights will still indicate a higher likelihood to be selcted.</p>