How does the WeightedRandomSampler works?

username: Ishal_Garg
cooked: <p>I have a binary classification problem and I have a unbalanced dataset. I decided to use  <strong>WeightedRandomSampler</strong>  from  <strong>torch.utils.data</strong></p>
<p><code>class_weights = [0.35,0.65]</code><br>
<code>class_weights_initialize = []</code><br>
<code>for i in train_readyframe['target']:</code><br>
<code>class_weights_initialize.append(class_weights[i])</code><br>
<code>weighted_sampler=WeightedRandomSampler(weights=class_weights_initialize,num_samples=len(class_weights_initiaze),replacement=True)</code></p>
<p>I have given a weight of 0.35 to the 0th class and 0.65 to the other.<br>
<strong>Does it means that the DATALOADER will select 65% of 1st class and 35% of 0th class in a single batch of training data?</strong></p>

username: KFrank
cooked: <p>Hello Ishal!</p>
<aside class="quote no-group" data-username="Ishal_Garg" data-post="1" data-topic="88863" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ishal_garg/48/23953_2.png" class="avatar"> Ishal_Garg:</div>
<blockquote>
<p>I have given a weight of 0.35 to the 0th class and 0.65 to the other.<br>
<strong>Does it means that the DATALOADER will select 65% of 1st class and 35% of 0th class in a single batch of training data?</strong></p>
</blockquote>
</aside>
<p>No, not exactly.  To take an extreme example, let’s say that your<br>
dataset consists of 100 samples, <em>all</em> from your 1st class.  Then<br>
each batch you generate will only contain samples from your<br>
1st class.</p>
<p>Instead let’s say your dataset consists of 65 samples from your 0th<br>
class and 35 samples from your 1st class.  Now each batch you<br>
generate will be <em>on average</em> equally weighted between the two<br>
classes, but any given batch will have random fluctuations away<br>
from being exactly equally weighted.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Ishal_Garg
cooked: <p>Thanks for the reply KFrank!<br>
But what do you mean by " each batch you<br>
generate will be  <em>on average</em>  equally weighted between the two<br>
classes"<br>
Sorry for the noob question. I am new to data science.</p>

username: KFrank
cooked: <p>Hi Ishal!</p>
<aside class="quote no-group" data-username="Ishal_Garg" data-post="3" data-topic="88863" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ishal_garg/48/23953_2.png" class="avatar"> Ishal_Garg:</div>
<blockquote>
<p>But what do you mean by " each batch you<br>
generate will be  <em>on average</em>  equally weighted between the two<br>
classes"</p>
</blockquote>
</aside>
<p>Using <code>WeightedRandomSampler</code> with a dataloader will build<br>
batches by randomly sampling from your training set.</p>
<p>Let’s say you have six samples in your dataset, with items 1, 2,<br>
and 3 being from class 0, and items 4, 5, and 6 being from class<br>
1.</p>
<p>Let’s build a batch of four items by rolling a six-sided die four<br>
times.</p>
<p>For one batch you roll 3, 6, 4, 6.  This batch will have one class-0<br>
sample, and three class-1 samples – not fifty-fifty.  Then you roll<br>
5, 2, 4, 3.  This batch will be fifty-fifty.</p>
<p>It’s random whether any specific batch is fifty-fifty, but on average<br>
your batches will consist of 50% class-0 samples and 50% class-1<br>
samples.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Ishal_Garg
cooked: <p>Nice explanation <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"><br>
It will randomly generate 50-50, when the class weights are [0.5,0.5] or it depends on the dataset distribution?</p>

username: KFrank
cooked: <p>Hi Ishal!</p>
<aside class="quote no-group" data-username="Ishal_Garg" data-post="5" data-topic="88863" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ishal_garg/48/23953_2.png" class="avatar"> Ishal_Garg:</div>
<blockquote>
<p>It will randomly generate 50-50, when the class weights are [0.5,0.5] or it depends on the dataset distribution?</p>
</blockquote>
</aside>
<p>As the example in my first reply shows, it clearly must depend<br>
on the dataset distribution.</p>
<p>In short, the probability of drawing a sample of a given class is<br>
proportional to the fraction of that class in your dataset times<br>
the weight you assign to that class.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Ishal_Garg
cooked: <p>Thnaks of the explanation KFrank.<br>
So if I have a dataset that contains only 2-3% of <strong>1 class</strong>, then i should prefer giving a higher class weight than the other.</p>